Integrity check, content validation and encryption of files over ICS FTP adapter

Oracle Integration Cloud Service (ICS), an industry-leading iPaaS offering, provides rich orchestration and mapping capabilities over FTP connections.   ICS FTP adapter supports connections over secure or non-secure channels.  Files can be transferred and processed by ICS without validating content upon transfer, which is the default behavior.  Files transferred over secured FTP benefit from inherent integrity checks built into the SFTP protocol.  For more information on ICS FTP adapter, see the link to a guide in references section.

Some users might have advanced requirements to validate content of the files during transfer or check for integrity of file explicitly. In this article, let’s take a look at steps to enable content validation against a schema during transfer, to perform integrity checks using MD5 checksum and to implement PGP encryption to secure files over secure or plain FTP connections.

Note that code archives provided with this article do not come with warranty from Oracle. Use discretion and additional testing to reuse code for production. Instructions in this article are applicable to release 17.1.3 of ICS.

Main article

This article is split into three sections, in order to keep them easier to follow. Click on one of the titles below to jump to a section.

Validating MD5 checksum of content

Validating content against a schema

PGP Encryption/Decryption of content

Validating MD5 checksum of content

Some integration use cases might require that the data file’s integrity be verified after transfer and before further processing.  This could be achieved by storing MD5 checksum of the original content, transfer it along with the file, compute checksum after transfer and compare with the checksum provided along with the file.  If they match, then content has not been altered or corrupt during or after transfer.

In the sample orchestration, a data file and a file containing its MD5 checksum are in a .zip file. ICS computes checksum upon unpacking files into ICS.  The checksum computed at source is compared with the checksum of the data file provided with file listing. If they match, orchestration proceeds with further processing.image021

The source file, datafile.zip, has datafile.csv and checksum.txt. checksum.txt has MD5 checksum of datafile.csv.  FTP connection is configured to unpack the .zip file into an ICS folder upon transfer.

Validating content against a schema

FTP connections in ICS allow definition of a schema for content being transferred. The content could be plain text and comma separated.  By default, FTP adapter reads the content and parses the records into structure defined by the schema, but does not validate content. For example, a column in the file could be specified as date, but the value could contain an invalid date string, something like ‘2017-12-32’. This condition could be handled within the orchestration, to reject the record with invalid date.

Optionally, a tweak to the integration archive could apply validation upon transfer and reject the file with invalid content. Note that such validation will be resource intensive for large files and might not be suitable for all scenarios.

Here is a simple orchestration that reads a .csv file via FTP connection and parses the file into records and fields.

image011

This is the schema definition page of the FTP connection. Note that the type for each column.

image012

This orchestration runs without errors even if EMPID has an alphabet or DATEOFHIRE has invalid month. To enable content validation, the integration needs to be exported, a tweak to be made and the archive be imported into ICS.  This change should be made to JCA file for FTP connection, at this location of unpacked archive.

./icspackage/project/FTP_VALIDATION_SCHEMA_02.00.0000/resources/application_256/inbound_996/resourcegroup_298/READ_VALIDATE_REQUEST.jca

Note that the schema is HTML encoded, so locate “<xsd:schema” and insert  “nxsd:validation="true"” as an attribute to the element.

Re-pack the archive into an .iar file and import in ICS.  Now FTP connection will validate content and fail with error upon invalid content. Note that the orchestration will stop at the FTP connection so any further logic will not be executed until content is fixed.

PGP Encryption/Decryption of content

PGP encryption is the best way to secure content transferred via FTP, if higher level of security than SFTP protocol is desired. Let’s look at enabling PGP encryption via ICS FTP adapter using a sample orchestration.

First, let’s quickly cover the basics of PGP encryption, which is performed using key pairs from the sending and receiving parties.  Each party shares the public key to the other. Content encrypted using the public key can only be decrypted by the associated private key, which is a secret known only to the possessing party.  Each key pair is associated with an email address name or both, of a recipient.  Files are encrypted by specifying the recipient, whose public key is already imported.

There are plenty of free utilities that help generate key pairs and to encrypt or decrypt content, OpenPGP and GPG are the most popular. For the sample orchestration, GPG was used to generate key pair.  Here are some useful commands for Linux.

To generate key pair:

gpg --gen-key

To export public and private keys of a key pair:

gpg --export -a "Test.User@invaliddomain.com" > public.key
gpg --export-secret-key -a "Test.User@invaliddomain.com" > private.key

To encrypt a file:

gpg  --verbose -e -r "Test.User@invaliddomain.com" datafile.zip

In the sample orchestration, PGP-related settings are configured in the connection page. Public key , Private key along with password, cipher (typically AES256) are relevant inputs for configuration.  PGP configuration used for sample orchestration is shown here. image031

The sample orchestration decrypts a file, named datafile.encrypted, upon transfer of the file to a local ICS folder and lists contents of ICS folder. Orchestration fails if the decryption fails. Here is the orchestration.image032

Summary

This article explored ways to check the integrity of files transferred over FTP connections and to enable validation of content being transferred. The solutions implemented in the 3 sample orchestrations can be used individually or combined for a more robust solution.  Note that the instructions are specific to release 17.1.3.

Artifacts

Download artifacts at this link Artifacts

References

ICS FTP adapter: https://docs.oracle.com/en/cloud/paas/integration-cloud-service/icsft/using-ftp-adapter.pdf

PGP tools: http://openpgp.org/ , https://gnupg.org/

 

Add Your Comment