ICS File handling options and tips.

Introduction

Oracle Integration Cloud Service (ICS) provides file handling capabilities through technology adapters and other application-specific adapters. This post focuses on the most commonly used file handling methods in ICS, using FTP, File and REST adapters and some techniques to tackle advanced file handling requirements. Information in this post is applicable to release 17.2.5 of ICS.

File handling in ICS

FTP adapter is widely used for transferring and process files. It provides ability to read, download, write and delete files at an FTP/SFTP location.  Size limits apply to files read via FTP, 2MB for “Read” operation and 10MB for “Download to ICS” operation. Files could be decrypted, unzipped upon Read and zipped, encrypted during Write operation. Refer to the guide about using FTP adapter for more information.

File adapter allows file polling, read and write operations on an on-premise location, using ICS connectivity agent.  The target and source directories are locally accessible from the server where ICS connectivity agent is deployed.  Refer to the guide about using File adapter for more information.

REST adapter allows receiving files as HTTP multipart attachments, and sending file as attachments, given that the target endpoint is capable of processing files sent as attachments. Incoming HTTP requests are limited to 10MB in size. In an upcoming release (currently 17.2.5) , REST adapter will be capable of receiving and sending files in raw form in HTTP body using content type “application/octet-stream”. This will allow uploading to and downloading files to content servers, such as Microsoft SharePoint.

Finally, StageFile action, allows reading, writing, listing, zipping and unzipping of files within ICS. It allows reading and processing files previously downloaded into ICS or created by StageFile write action.  Note that the files handled by StageFile are temporary in nature, accessible only during execution of an integration instance.

Base64 encoding and decoding

File content that’s binary, not readable or non-conforming (to XML schema, for example) need to be converted to base64-encoded string so that web services can handle them. Likewise, web services return such content as base64-encoded string that need to be converted to files. ICS FTP adapters allow this through a schema shown below.

<schema targetNamespace="http://xmlns.oracle.com/pcbpel/adapter/opaque/" xmlns="http://www.w3.org/2001/XMLSchema">
<element name="opaqueElement" type="base64Binary"/>
</schema>

In FTP invoke or StageFile action read the file contents are read into variable opaqueElement as base64binary. In write operations, conversely, the base64binary value assigned to opaqueElement is converted to native format. Note that ICS also provide functions to base64 encode or decode contents of variables.

Validation, encryption and decryption

Occasionally, files might need to be validated upon reading through FTP. Note that such validation will result in rejection of entire file. FTP adapter also allows PGP encryption and decryption. Refer to this post for more information and instructions.

Chunked file reading

StageFile action in ICS allows processing for large file in chunks of pre-determined number of records, between 20 and 200 in each chunk. This reduces the amount of memory used during processing of large files within ICS. See screenshot below.

image1

Parallel processing

Parallel processing, although not specific to file processing, is nevertheless very useful in reducing processing time for large set of records, typically read from large files.

image2

Summary

This post provided and overview of common file processing options in ICS and addressed some advanced processing techniques.

References

FTP adapter:

https://docs.oracle.com/en/cloud/paas/integration-cloud…/using-ftp-adapter.pdf

 

File adapter:

https://docs.oracle.com/en/cloud/paas/integration-cloud-service/icsfi/file-adapter-capabilities.html

Integrity check and validation:

http://www.ateam-oracle.com/integrity-check-content-validation-and-encryption-of-files-over-ics-ftp-adapter/

 

Comments

  1. Paula Yang says:

    Hi Mani,

    I have a question,

    “StageFile action in ICS allows processing for large file in chunks of pre-determined number of records, between 20 and 200 in each chunk. ” Here you are talking about # of records.

    But in ICS document, link https://docs.oracle.com/en/cloud/paas/integration-cloud-service/icsug/creating-integrations.html#GUID-8673224A-42C5-4C10-BCF2-66FCA7AB9DEF
    Table 2-3, it said “Segment Size: Specify a file chunking size between 10 and 200 MB”.

    I guess you are right, is it the case?

    Thank you!
    Paula

Add Your Comment