Migrate Database backup from AWS s3 to OCI Object storage using rclone and restore it with help of Oracle storage gateway

Introduction –

It is a common customer requirement to have Database backup migrated from non oracle cloud to the Oracle Cloud Infrastructure. The primary reason could be to save on the cost or to have Oracle on Oracle. Organizations moving to Oracle Cloud will have the requirement of migrating legacy / on-premise / third-party cloud data into Oracle cloud. In this blog,  we will review a use case where we will migrate the database backup from AWS-S3 bucket to the Oracle Object storage. The object storage backup will then be then restored on the OCI Compute VM Database

Scenario

• Transfer backup stored in AWS-S3 bucket to OCI object storage bucket.
• Install storage gateway on OCI.
• Present the objects within object storage directly to the Compute VM that will host the database, as a filesystem.

In order to transfer the backup stored in S3 or azure blob storage, we will use an open source utility called rclone.

Rclone is a great command-line utility that synchronizes files and directories between a local file system and a variety of cloud back-ends. It is very easy to setup to do the data transfer.

How to install rclone ?

Here is how you can do a system-Wide Installation On Oracle Linux 7:

$ sudo yum install -y rclone

On other Linux distributions, you can use the installation script provided by rclone. Download and run the script to install the latest version of rclone system-wide:

$ curl https://rclone.org/install.sh | sudo bash

Here is the screenshot of rclone installation –

[root@node2 ~]# curl https://rclone.org/install.sh | sudo bash

Once rclone is installed, we need to configure source and target for the rclone. This will inform the rclone binary, where to copy objects from and where to store them. Here is the example of source and target configuration in my test case.

Here are links on how to collect the AWS and OCI Access key and secret access key for this configuration.

Once the files are transferred, we can see them on the OCI object storage, in our case DBSMBK bucket. Refer below -

Now that we have the backup transferred to OCI storage, we can leverage either service gateway for connectivity to object storage endpoint or we can use the storage gateway to directly publish this backup, in our case compute VM, where it needs to be restored.

For this use case we will use storage gateway. Storage Gateway exposes an NFS mount point that can be mounted to any host that supports an NFSv4 client. Moreover storage gateway is a free service to all OCI customers.

Here is how we can install and use storage gateway. Refer the document for the Storage gateway installation. The high level steps are -

• Download the storage gateway software, install/configure it on a compute instance.
• Configure the management console and create a file-system to map to the OCI bucket.
• Publish the file-system to the NFS client (DB node where backup needs to be restored)

 Here are some screenshots of the installation –

Once installed, configure the management console and create the file-system to refer the the OCI object store bucket.

As you can see here with the storage gateway we are able to mount the OCI bucket directly onto the Compute VM. All the objects from bucket are visible on the server.

Once the bucket is mounted as a NFS filesystem, we can restore directly. Refer below snippet -

and the Database is restored and open -

The limitations for this method are –

• Internet speed will determine the transfer speed. ( Typical speed observed is ~ 5 hours for 1 TB)
• Source Buckets endpoints need to be made read only momentarily, for the period of the transfer.
• If the Database backup is too large ( > 20 TB) its best to use data transfer appliance.

Summary –

By using rclone we are able to transfer the data across in a multi cloud scenario. This data can be transferred directly to the OCI object storage. The object storage bucket is a regional service and not tied to any specific compute instance.

Storage Gateway can be leveraged to publish OCI object storage objects via NFS mount to any desired compute instance.

By using rclone and storage gateway, we can conveniently transfer the Database backup from any other cloud to OCI cloud by saving it on object storage. The database can be restored by mounting this backup on any NFS client, though the storage gateway.