Overview
As we ride and lead in multi-cloud journey and make is easier for customers to use multiple clouds, we keep an eye on third party cloud releases that affect OCI customers. One such instance is a recent update in AWS cli related to S3 buckets integrity check.
Many customers utilize the AWS S3 API, AWS Command Line Interface (CLI), or the Boto3 client library to access and retrieve files across multiple cloud platforms, including Oracle Cloud. The AWS S3 API provides a standardized way to interact with object storage, enabling seamless data retrieval. The AWS CLI facilitates command-line-based operations, allowing users to fetch files efficiently. The Boto3 client library, designed for Python, simplifies programmatic access to S3-compatible storage. Oracle Cloud offers S3-compatible object storage, allowing integration with these tools, making multi-cloud file management more flexible and efficient for businesses handling large-scale data operations.
If you use updated AWS cli (version 2.23.5 or later), you will not be able to use create or update objects in OCI object storage.
% aws s3api put-object --bucket tests31 --endpoint-url https://$NAMESPACE.compat.objectstorage.$REGION.oraclecloud.com --body test.txt --key test.txt
An error occurred (MissingContentLength) when calling the PutObject operation: The Content-Length header is required
You can fix that by adding AWS_REQUEST_CHECKSUM_CALCULATION, AWS_RESPONSE_CHECKSUM_VALIDATION environment variables. The value is ‘when_required’. After you set the environment variables, then you can use AWS cli with OCI buckets.
% export AWS_REQUEST_CHECKSUM_CALCULATION=when_required
% export AWS_RESPONSE_CHECKSUM_VALIDATION=when_required
% aws s3api put-object --bucket tests31 --endpoint-url https://$NAMESPACE.compat.objectstorage.$REGION.oraclecloud.com --body test.txt --key test.txt
{ "ETag": "\"638b89c0ff34a19614372cb390664bf4\"",
"VersionId": "cb5b74a6-266a-4f29-a840-c6acfc3e54b0" }
If you are using python boto3 library, the library also requires those environment variables as mentioned in the below code.
import boto3
import os # Set the environment variable for the Oracle Cloud Infrastructure (OCI) config file
os.environ['AWS_REQUEST_CHECKSUM_CALCULATION'] = 'when_required'
os.environ['AWS_RESPONSE_CHECKSUM_VALIDATION'] = 'when_required'
s3 = boto3.client(
's3',
aws_access_key_id="$OCI_ACCESS_KEY_ID",
aws_secret_access_key="$OCI_ACCESS_KEY",
region_name="us-phoenix-1", # Region name here that matches the endpoint
endpoint_url="https://$NAMESPACE.compat.objectstorage.$REGION.oraclecloud.com" # Include your namespace in the URL
)
# Create a new object in the bucket
s3.put_object( Bucket='tests31', Key='hello', Body='Hello World!' )