Troubleshooting tips when hitting rate limits for Oracle Integration Cloud File Server and Oracle Identity Cloud Services

Introduction

OIC File Server is an embedded SFTP server within Oracle Integration Cloud. However, what could happen when OIC cannot connect to the OIC File Server? Is it a File Server issue? Is it an OIC issue? Is it an authentication issue?

The rate limits for the collection of integrated applications depend on the individual applications and their associated rate limits. When these rate limits are exceeded, the error message passed to the end user will rely on the application that exceeded its limits and the associated pipeline that will surface the error. When customers get the error "Unable to connect to the OIC File Server. Please check connectivity" in the response payload, this message will lead the user to check the OIC setup, the File Server configuration, or the connectivity between OIC and the File Server. Additional information from the File Server may point to "No Data to process" or "File to be processed was not found or unavailable or has no content."

Data check on the File Server may point out that the data is actually in the file directory, and a subsequent try of the very same integration may yield a successful processing for this integration instance. So why did it fail in the first place? Are you running into an intermittent issue?
A configuration issue will either work or not work 100% of the time. When sometimes, it works; occasionally, it doesn't, may point to runtime issues. Rate limits may impact solutions at runtime if the allowed capacity is reached. 

OIC File Server Service Limits

The OIC File Server concurrent connections limit allows 50 connections per service instance for OIC Gen 2 or Gen 3. Even though File Server metrics are not exposed to the end user, integration metrics are readily available in the OIC Console. A glance into integrations that use file-based operations will reveal the concurrency count. If this count is getting close to 50 for several minutes in a row, you might need to open a Service Request and have an Oracle Support engineer assist you in taking a closer look into File Server metrics. Most customers do not get close to this maximum concurrency of 50 connections. See below documentation for OIC Service Limits.

OIC Gen 2
https://docs.oracle.com/en/cloud/paas/integration-cloud/oracle-integration-oci/service-limits.html#GUID-EB64C32A-BAEE-4ABC-BF2D-0E8B3FD5B848__2

OIC Gen 3
https://docs.oracle.com/en/cloud/paas/application-integration/oracle-integration-oci/service-limits.html#GUID-EB64C32A-BAEE-4ABC-BF2D-0E8B3FD5B848__DDD

API Rate Limits for different identity domain types

If the issue is neither with the File Server configuration nor the File Server rate limit, opening a Service Request and having an Oracle Support engineer assist you in taking a closer look into File Server logs might be needed.
In the OIC File Server logs, Support Engineers may find "Error in searchUsers. Response code: 500. Internal Server Error: Too Many Requests" messages for the integration erroring intermittently. This will point out an IDCS rate-limiting issue, as the internal error is related to too many requests when searching for users.

https://docs.oracle.com/en-us/iaas/Content/Identity/sku/overview.htm#api-rate-limits

Oracle APIs are subject to rate limiting to protect the API service usage for all of Oracle's customers. If you reach the API limit for the identity domain type, IAM returns a 429 error code. Therefore, one would expect a 429 error in the integration response payload if we hit the current IDCS rate limit. 

However, we had a customer who was using the free licenses for IDCS and was reaching the allowed rate limit. Whenever the load from the Integration File Server exceeded this limit, the File Server could not access the resources, and an OIC File Server connection error was raised. The rate limits were no longer hit by opting for a higher-limit IDCS licensing model, and the integration instances using the File Server could run successfully.

Conclusion

You may run into IDCS rate limits even if you do not see the 429 error message on the integration response payload. You might need to look into IDCS if the issue is related to authentication/authorization. Ensure every rate limit is checked and your solution is tested for peak performance, given your business's use case.