On November 30th, OCI Data Integration introduced a new feature enabling users to seamlessly export and import a range of elements—projects, folders, tasks, data flows, pipelines, and applications—across different workspaces. With this new feature, development in OCI Data Integration is became easier and more resilient.
In this blog post, we will delve deeper into this topic, exploring a few use cases for exporting and importing objects. Additionally, there is a corresponding video (below) that complements the content discussed here.
Backup and Version Control
The first use case that comes to mind is the ability to create backups of the development work. By exporting an object to a safe location, users can later on restore that object if it was deleted, or overwrite the current version if some changes introduced a computation error. Pushing the idea a further, users can apply version control to their OCI Data Integration development by exporting regularly their objects in Git, Subversion or a similar tool. This allows developers to compare changes between specific versions and restore a version if needed.
Moreover, users now have the ability to monitor the progression of business rules. By keeping all version of objects, organizations can find which business rules where used to compute a metric at a specific point of time even if the data flow implementing it has changed since then. It’s particularly important for organizations who are subject to audits; for instance in the financial or medical industries.
Backups or version control can be automated thanks to the OCI SDK, OCI CLI or REST APIs. This enables the scheduling of frequent backups and optionally commit them into a version control system. By following this approach, users can be assured to have a regular snapshot of their objects, should they restore it later.
Objects promotion
Central to the development lifecycle is the migration of an object from one environment to another. Previously, users were able to replicate an application from one workspace to another and subsequently synchronise the duplicate when changes in the source are ready for promotion. As an application solely encompasses tasks, this suffices for production environments, which typically steer clear of developmental objects.
Within a hotfix environment, created to mirror the production setup for rapid development of fixes, it is necessary to also include data flows and pipelines as well. This is made possible by exporting from one workspace and importing it into another. And once again, automation can be used to align environments on a regular basis.
It is now possible to establish sandbox environments for testing modifications or training staff. By importing a subset of objects in a new environments it’s now possible to play with the objects without breaking anything in the main development environment. Sandboxes can also be used for objects that are created programmatically through the OCI SDK or the OCI CLI. They can be tested in the sandbox before being exported and imported in the main development workspace.
Migration and disaster recovery
Export and import also makes it possible to move our data integration projects from one tenancy to another. We have seen many organizations having several tenancies for various reasons. There can be tenancies owned by different departments, or by different companies merging together. By exporting from one workspace in a tenancy and importing into another workspace in a different tenancy, it’s now possible to move or consolidate data integration projects.
For certain organizations, establishing a new workspace in a different region for disaster recovery purposes is another essential use case.
Sharing Objects
The Export/Import functionality is also beneficial for collaboration on data integration projects. It is now possible to export objects and share it with others in the community. A useful task or pipeline can be shared online as a zip file and used by other users having a similar use case.
Our partners can now develop and deploy solutions tailored to specific industries. The option to share entire projects or applications with customers not only facilitates collaboration but also unlocks new avenues for business opportunities.
The introduction of the Export/Import functionality in OCI Data Integration has significantly enhanced the development, deployment, and governance of data integration projects. It provides a versatile tool for creating backups, promoting objects between environments, migrating projects across tenancies, and fostering collaboration within the community. This makes OCI Data Integration an even more compelling choice for organizations seeking robust and scalable data integration solutions.