X

Best Practices from Oracle Development's A‑Team

Setting Up ODI Marketplace Agents in High Availability with ATP

Matthieu Lombard
Consulting Solution Architect

 Background

This blog walks though how to configure Oracle Data Integrator (ODI) on Oracle Cloud Marketplace Agents in High Availability (HA) Mode with the Autonomous Transaction Processing (ATP) Database to hold the ODI repository. 


To allow for a true application HA, the agents configured throughout this blog will be J2EE agents deployed in a WebLogic domain cluster of managed servers . For a reminder on the different types of agents that can be configure with ODI, please see this blog.

Suggested Prerequisite Reading Material:

Using Oracle Cloud Marketplace

* Configuring High Availability for ODI

Overall prerequisites

* Follow the steps in the first blog for instructions on:

Policies Assignment (Step 1)

SSH key-pair Generation (Step 4-b and 4-c)

ODI Marketplace Provisioning (Step 2)

User/Policy/Key Requirements:

* It is strongly recommend to configure OCI and ODI with a local (non-Federated) user.

 Architecture and High Level Process

 Architecture

The following diagram describes the different components of the ODI Marketplace HA architecture :

Figure 1: ODI Agents HA Architecture

* The OCI foundations, including the tenancy; a compartment within the tenancy, the network elements such as a Virtual Cloud Network (VNC), a subnet with the proper security lists (Egress/Ingress rules) and routing table, and finally the public load balancer (primary and standby) with its listener, back end server set and health check rules;


* The ATP database to support the ODI Repository and the WebLogic domain schemas : the Metadata Services schema (MDS), the Audit Services schemas (IAU), the Oracle Platform Security Services schema (OPSS), the WebLogic Services schema (WLS), the ODI Repository schemas (ODI Master and Work) and the Common Infrastructure Services (STB) schema . For more details on the WebLogic domain schema, please refer to Understanding Repository Creation Utility Schemas, IDs and TableSpace;


* The two ODI Marketplace compute instances, provisioned in two different availability domains and  that will host the two J2EE agents. Both ODI instances are provisioned within a public subnet with both private and public IPs. By default, the ODI Marketplace instance firewall is enabled and configured to block all traffic. Therefore additional firewall configuration is required;


* The WebLogic domain that consists of two WebLogic Managed Servers instances, one on each of the ODI Marketplace compute instances. The Managed Servers are grouped in a WebLogic cluster, in which the ODI J2EE agents application are deployed. They are managed with a single Administration server running on the first ODI Marketplace compute instance. The WebLogic domain here will also host the Coherence cluster. To avoid duplicate ODI schedule processing, only one of the ODI agents behaves as a scheduler. A Coherence cache is used to handle scheduler service uniqueness and migration;


* The two Node Managers java utilities running on each ODI Marketplace compute instance as a separate process from the Managed or Admin Servers. The Node Managers allows to perform common operations tasks for a Managed Server (start, stop,…) regardless of its location with respect to its Admin server. These utilities are not represented explicitly on the diagram though.

 High Level Process

Setting up HA ODI Agents on a Marketplace context can be summarized in twelves high level steps:

  1. Configuring Network to create the Virtual Cloud Network , subnet and security list as described in the OCI Network Documentation and ODI HA documentation ;
  2. Provisioning an ATP database instance, saving the JDBC connection properties details and downloading the ATP wallet to allow client connections (via SQL Developer for example);
  3. Provisioning the first ODI Marketplace instance (referred to as node 1 later on in the blog) to create the ODI repositories on the ATP database provisioned in step 2;
  4. Provisioning the second ODI Marketplace instance (referred to as node 2 later on in the blog) using an existing ODI Repository on the ATP database where the ODI Repositories were created in step 3;
  5. Configuring the firewall on both node 1 and node 2 to allow HTTP traffic between the two instance, to allow the common required WebLogic ports to be opened,…;
  6. Configuring an HTTP server on both node 1 and node 2 to test and validate the Load Balancer configuration later on. For details on configuring  an HTTP server on an OCI compute instance running on Linux, please see here ;
  7. Creating the WebLogic domain on node 1 by executing the ./config.sh script;
  8. Configuring the Node Manager and starting the Admin / Managed servers on node 1;
  9. Packing the domain on node 1 and unpacking the domain on node 2;
  10. Configuring the Node Manager and starting the Managed server on node 2;
  11. Creating and configuring the Load Balancer, its listeners, back end servers set and health check rule. For details on configuring  a public Load Balancer an OCI, please see here ;
  12. Updating the ODI Topology to create the physical agent using the load balancer public IP and HTTP port and testing the configuration by running an ODI integration against the load balanced J2EE agents.

 Some Details Steps, Common Potholes and Things to Watch Out For


For a successful set up, some of the twelves steps above need some more granular level of description. There are indeed some common blockers that one can face during the process. This section will go over five of these main common potholes:


Updating the security lists and firewall rules to support the Coherence Death Detect via TCP port 7 in step 1 and 5 above

When starting the Managed Servers, one may hit the waring below, pertaining to Coherence Death detect port being closed. 

####<Apr 14, 2020 8:25:17,456 PM GMT> <Warning> <com.oracle.coherence> <oracle-odi-inst-oewo.app.odiu6pmpcnx.oraclevcn.com> <ODI_server2> <Logger@526962079 12.2.1.4.0> <<anonymous>> <> <64568a5f-5798-46f2-9b2e-bf961a395ccd-00000007> <1586895917456> <[severity-value: 16] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000000> <2020-04-14 20:25:17.456/121.718 Oracle Coherence GE 12.2.1.4.0 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; IpMonitor failed to verify the reachability of senior Member(Id=1, Timestamp=2020-04-13 18:54:41.545, Address=10.0.17.7:32783, MachineId=7318, Location=site:app.odiu6pmpcnx.oraclevcn.com,machine:odi-node1,process:19998,member:ODI_server1, Role=ODI_cluster1)  via /10.0.17.8; if this persists it is likely the result of a local or remote firewall rule blocking either ICMP pings, or connections to TCP port 7>

 

Having this port closed may create subsequent issues, specifically it may prevent the Coherence cache to handle scheduler service uniqueness and migration correctly.


Therefore, TCP Port 7 needs to be open in the security list (Ingress rules) as described below

 

Stateless

Source

IP Protocol

Source Port Range

Destination Port Range

Type and Code

Allows

Comments

No

0.0.0.0/0

TCP

All

7

Nil

TCP traffic for ports : 7 ECHO

Coherence TcpRing/IpMonitor death detection feature.

Table 1: TCP port 7 ingress rule

 

Also, the firewall rules needs to be amended with the rule below, on each compute instance hosting the ODI Agent Managed Servers.

sudo firewall-cmd --permanent --new-service=odicoherencedeathdetect
sudo firewall-cmd --permanent --service=odicoherencedeathdetect --set-description="ODI Coherence Cluster TCP Ring"
sudo firewall-cmd --permanent --service=odicoherencedeathdetect --add-port=7/tcp
sudo firewall-cmd --permanent --add-service=odicoherencedeathdetect
sudo firewall-cmd --reload
sudo firewall-cmd --list-all

 

Updating the security lists and firewall rules to support HTTP traffic between the two nodes in step 1 and 5 above

To allow the load balancing to work properly, HTTP Traffic needs to be enabled between the two ODI Marketplace compute instances. This translate into updating the security list of the VCN (Ingress rules) and the firewall rules on each of ODI Marketplace compute instances, as described respectively in the table and command below.

Stateless

Source

IP Protocol

Source Port Range

Destination Port Range

Type and Code

Allows

Comments

No

0.0.0.0/24

TCP

All

80

Nil

TCP traffic for ports : 80

OCI Load balancer

No

<private IP range of the VCN>/24

TCP

All

80

Nil

TCP traffic for ports : 80

OCI Load balancer

Table 2: TCP port 80 ingress rule

sudo firewall-cmd --permanent --zone=public --add-service=http
sudo firewall-cmd --reload

 

Providing the ATP specific JDBC connection properties when configuring the WebLogic Domain during step 7

When running the configuration script to create the ODI domain, special care needs to be applied when entering the JDBC connection properties for the different RCU schemas.


When filling the STB Schema details, make sure to only check the LocalSvcTBl Schema. Then Connection Properties. The Connection Properties screen displays the following details:

 

oracle.net.authentication_service TCPS
oracle.net.ssl_server_dn_match false
javax.net.ssl.trustStore /u01/oracle/mwh/wallet/cwallet.sso (Path of the wallet location)
javax.net.ssl.keyStoreType SSO
javax.net.ssl.keyStore /u01/oracle/mwh/wallet/cwallet.sso(Path of the wallet location)
javax.net.ssl.trustStoreType SSO

Figure 2: ODI domain LocalSvcTBl Schema JDBC connection properties

Then all the other schemas can be selected before clicking Connection Properties. Then the connection properties described above can be filled it at once for all schemas but the LocalSvcTBl Schema.

Figure 3: ODI domain other schemas JDBC connection properties

At the time of writing this blog, these properties are described in a different section than the HA section in the ODI Marketplace documentation. For further details, please see here.

Properly Configuring the Load Balancer Health Check Rule

A common mistake when creating the health check rule at the load balancer level is to forget the forward slash “/” at the end of the URL.

Figure 4: Load balancer Health Check URL Path missing “/”

Doing so will result in getting 502 Bad Gateway Errors when accessing the load balanced ODI Agent URL.

Figure 5: Bad Gateway error

So when updating the load balancer health check, please make sure to add the forward slash “/” at the end of the URL Path, as depicted in the below figure.

Figure 6: Load balancer Health Check URL Path with “/”

Ensuring non default port are used when configuring WebLogic Admin server


The OCI Threat and Vulnerability Management Team (TVM) was notified that there have been successful exploits against Oracle WebLogic Server leveraging Java Deserialization issue (CVE-2020-2883). TVM will identify the IPs in a tenancy using T3 protocol that are vulnerable to this CVE. 


To address this issue, TVM recommends to identify all instances within your organization which may be using the T3 protocol and close the affected ports (Default ports 7001 and 7002). 


If one would like to continue using T3 protocol, the recommendation is to block ports 7001, 7002 and any other ports using T3 protocol from external access and make sure they are accessible via Oracle VPN only using T3 over SSL (T3S).

 

 Next Considerations

In a next blog, ATP Roadmap for HA and Disaster recovery will be discussed, as well as configuring HTTPS and SSL configuration with a private load balancers, ODI Marketplace instances in a private network and ATP with Private Endpoint, to fulfill ODI Marketplace HA Enterprise Deployments.

 Want to Learn More?

Click here for more A-Team Oracle Data Integrator (ODI) Blogs

Click here to sign up to the RSS feed to receive notifications for when new A-team blogs are published

Click here to access the ODI Marketplace documentation library 

 Summary

This article walked through the steps to configure Oracle Data Integrator (ODI) on Oracle Cloud Marketplace Agents in High Availability (HA) Mode with the Autonomous Transaction Processing (ATP) Database to hold the ODI repository. 

Stay tuned for more to come!

Bookmark this post to stay up-to-date on changes made to this blog as our products evolve.

 

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha