Last validated on September 30, 2020

Introduction

The latest releases of Oracle Data Integrator (ODI) are now available on Oracle Cloud Marketplace.

From the Using Oracle Cloud Marketplace website:

"Oracle Cloud Marketplace is an online store—a one-stop shop—selling hundreds of business apps and professional services that complement your existing Oracle Cloud implementation." 

This offering includes a full instance of ODI with a pre-configured repository. This repository may optionally be deployed on Oracle's Autonomous Databases.

This post is a step-by-step guide to installing and configuring ODI Marketplace in a public subnet on Oracle Cloud Infrastructure (OCI) using an Autonomous Database for the pre-configured repository.  It is a companion blog to Deploying Oracle Data Integrator Marketplace in a Private Subnet with Autonomous Database

The official documentation is Using Oracle Data Integrator on Oracle Cloud Marketplace

Validations

Published on December 24, 2019

Versions

• Oracle Data Integrator 12.2.1.4.0

• MacBook Pro OS 10.13.6 

• Linux 7 Compute Instance

• Autonomous Database 

Topics

• Before You Begin

• Preparing the ODI Marketplace Stack

• Applying the ODI Marketplace Stack

• Validating the ODI Configuration

 Before You Begin

Deploying ODI from the Oracle Marketplace requires these items. Links to the relevant documentation are provided. Managing these items require Administrator privileges and are assumed to exist prior to beginning the steps in this post.

An Oracle Cloud Tenancy in Oracle Cloud Infrastructure (OCI) 

When an OCI account is created, Oracle creates a tenancy, which is a secure and isolated partition within Oracle Cloud Infrastructure where you can create, organize, and administer cloud resources.

A Tenancy User Account within an Identity Provider

A tenancy is federated with Oracle Identity Cloud Service (IDCS) as the identity provider. OCI's native Identity and Access Management (IAM) service may also be used as the identity provider. More advanced options like federating Active Directory are outside the scope of this post. For more information on federated users, see Managing Oracle Identity Cloud Service Users and Groups in the Oracle Cloud Infrastructure Console.

This post uses a federated IDCS user account named Lisa.Jones

A Group Membership within IDCS

The user account needs to belong to an IAM group. OCI privileges are granted at the IAM group level. For an IDCS user to be assigned to an IAM group, the user first needs to be associated with an IDCS Group. See the link above for additional IDCS group information. 

This post uses an IDCS group named dcarley_compartment_ODI_Group

A Group Membership within IAM

An IAM group is what receives permissions to act on OCI resources. For more information on IAM groups see Managing Groups 

This post uses an IAM group named dcarley_compartment_ODI_group

A Mapping of the IDCS Group to the IAM group

This mapping grants an IDCS user membership in an IAM group. For more information see Map an IDCS group to an IAM group

A Compartment to Contain ODI resources

Compartments are logical groupings of OCI resources used for organizing and managing permissions. For more information on compartments see Here.

This post uses a compartment named ODI_Compartment

Note: With the exception of using an existing Virtual Cloud Network, all ODI resources used or created in the provisioning of ODI marketplace must reside in the same compartment.

A Dynamic Group for ODI Instances

Dynamic Groups allow you to group Oracle Cloud Infrastructure computer instances as "principal" actors (similar to user groups). Permissions can then be granted to these groups to manage and use OCI resources. For more information on dynamic groups see Here.

This post uses a dynamic group named dcarley_ODI_dyn_grp

Dynamic Groups require Matching Rules that identify which instances belong to the group. A rule that gives membership to all instances in a compartment i.e. ODI_Compartment is written as:

ALL {instance.compartment.id = 'ocid1.compartment.oc1..aaaaaaaaashjhg2nim567cpqjjazrwtcuqgupj77jj5ur2k7obk2v7suss6a'}

Policies to Use and Manage ODI resources

Policies are used to grant permissions on ODI resources. For more information see Here and Here. These ODI resources include the Virtual Cloud Network (VCN), the Compute instance hosting the ODI application and the Database hosting the ODI repository.

A typical example noted in the OCI documentation is for users to "Have full permissions on all resources, but only in a specific compartment".

This post uses the above example with a policy created in the ODI Compartment named dcarley_compartment_ODI_policy.

The policy has two rules: 

• Allow group dcarley_compartment_ODI_group to manage all-resources in COMPARTMENT ODI_Compartment

• Allow dynamic-group dcarley_ODI_dyn_grp to manage all-resources in COMPARTMENT ODI_Compartment

These rules allow the creation of the ODI compute instance as well as the use or creation of a VCN and the use of a Autonomous Repository Database.

A Provisioned Autonomous Transaction Processing (ATP) Database

This post uses an ATP database for the ODI repository. This database needs to be provisioned prior to preparing the ODI Marketplace Stack described below.

The user who provisions ODI and who has the privileges described above can provision an ATP database in the compartment designated for ODI resources. For more information see Provision Autonomous Transaction Processing

After these prerequisites are in place ODI can be provisioned from the Marketplace.

 Preparing the ODI Marketplace Stack

This section is performed by the example provisioning IDCS user Lisa.Jones. All the information required for the ODI provisioning is collected and stored in an OCI Resource Manager Stack. A stack is defined as a "collection of Oracle Cloud Infrastructure resources corresponding to a given Terraform configuration" For more information on see Overview of Resource Manager.

It is important to note that creating the Stack does not create the ODI resources. The resources are created in the next section.

Log into the OCI Console

Log into the OCI Cloud Console using your credentials. This post uses IDCS credentials for Lisa.Jones. 

Select the OCI Region

Select the desired region from the drop-down on the menu bar.

Navigate to the OCI Marketplace

Click on the hamburger icon in the top left portion of the menu bar to expand the list of services. Click on Marketplace.

Search for and click on Oracle Data Integrator

Use Oracle for the Publisher and Data Integration for the Category. Then click on Oracle Data Integrator.

Launch the Stack

Select the compartment i.e. ODI_Compartment where your privileges are granted, check the Terms box and click Launch Stack.

Complete the Stack Dialogs

Optionally enter a NAME and DESCRIPTION for the stack. 

Click Next

General Settings

There are two choices for NETWORKING OPTIONS. You can have the Stack create a VCN for you or use an existing VCN. The Stack creates a VCN for the ODI instance in the same compartment specified for the Stack. If using an existing VCN in a different compartment, additional policy privileges may be required.

This post chooses to use an existing VCN.

There are three options for the ODI REPOSITORY LOCATION.

1. Use an existing Repository created by an RCU utility.

2. Create a new ODI Repository in an existing Autonomous database.

3. Create a new ODI Repository in a new MySQL database embedded into the ODI compute instance.

This post chooses to create a new repository in an existing ATP database.

Optionally provide a PREFIX for all new resources.

Select the TARGET COMPARTMENT. This is the target compartment where your privileges are granted. If you are using an existing Autonomous Database, it needs to be in this compartment also.

Network Configuration

If using an existing VCN, choose the VCN COMPARTMENT, VCN, SUBNET COMPARTMENT, and SUBNET from the drop-downs. Check the ASSIGN PUBLIC IP ADDRESS box as it is needed to SSH into the ODI instance.

If creating a new VCN, accept the default or specify the VIRTUAL NETWORK CIDR notation and check the CREATE A PUBLIC ODI SUBNET box if you want one to be provisioned.

ODI Instance Settings

Accept the default for the ODI NODE SHAPE or choose a larger shape from the dropdown.

Generate or locate an SSH Key pair. The public key should have a .pub extension. To generate a pair run the ssh-keygen utility. Open the .pub file with a text editor and copy the entire text. Paste it into the SSH PUBLIC KEY box.

Select an ODI AVAILABILITY DOMAIN from the dropdown.

Enter an ODI VNC PASSWORD to use with ODI Studio.

New ODI Metadata Repository

For the New Repository in an Existing ATP option:

Select an AUTONOMOUS DATABASE INSTANCE from the dropdown.

Enter the AUTONOMOUS DATABASE INSTANCE ADMIN password.

Enter an ODI SUPERVISOR PASSWORD for the new repository.

Enter a SCHEMA PREFIX for the new ATP repository schemas.

Enter a SCHEMA PASSWORD for the new ATP repository schemas.

Click Next above to complete the Stack dialogs.

Review and Save the Stack

Verify your entries and click Save Changes.

 Applying the ODI Marketplace Stack

After the Stack has been created above it appears in the Resource Manager service in an Active state. 

Apply the Stack

To use and create the resources identified in the stack, the stack needs to be applied. This is done via a Resource Manager Job. To run the job, choose Apply from the Terraform Actions dropdown.

When the Job succeeds, the ODI resources have been created and are ready to use. 

Below is a sample diagram of the OCI topology after a successful deployment.

 Validating the ODI Configuration

This section describes connecting to the ODI instance, running ODI Studio and connecting to the pre-configured repository in ATP.

Connecting to the ODI instance VNC Server

Provisioning of the Linux compute instance hosting ODI includes a VNC server running on Port 5901. You can connect to it directly by adding an ingress rule to the VCN security list opening port 5901. The best practice is to use an SSH tunnel and connect to it via the SSH port 22. Port 22 is open by default in the VCN security list.

Opening the SSH Tunnel

An SSH tunnel can be created via the PuTTY utility on Windows or via the ssh command on Linux and Mac. This post uses the ssh command. An example is below:

ssh -I < Path to Private Key > -L 5901:localhost:5901 opc@< ODI instance public IP >

Running the above command opens an SSH session and creates the tunnel.

Connecting to the VNC Server 

The next step is to connect to the VNC server via a VNC viewer. An example viewer on Windows is TigerVNC. The Mac OS contains a VNC viewer which is used by this post. To use the MAC viewer:

• Start the Finder application

• From the menu bar, click Connect to Server from the Go dropdown

• In the Server Address box, enter vnc://localhost:5901 and click Connect

• Enter the VNC password supplied to the ODI Stack and click Connect

The VNC window appears. If it is the first time using it, click through the language and privacy settings.

Starting ODI Studio

To start ODI Studio, click on ODI Studio from the Programming tab in the Applications dropdown. 

If it is the first use, Click No for Confirm Import Preferences 

Connecting to the ATP Repository

From the ODI Studio home page, click on Connect to Repository.

For the New Wallet Password dialog, check the box for the choice of secure wallet usage and click OK. This post uses a development-only method of not using a secure wallet.

The Oracle Data Integrator Login dialog has been pre-populated and uses the SUPERVISOR password supplied to the ODI Stack. Click OK.

The ODI Initialization Dialog may take a while the first time. When it finishes you are connected to the Repository and the Studio.

 Summary

This post described installing and configuring ODI Marketplace in a public subnet on Oracle Cloud Infrastructure (OCI) using an Autonomous Database (ATP) for the pre-configured repository.

For other posts relating to analytics and data integration visit http://www.ateam-oracle.com/dayne-carley