A First Look at the Oracle Cloud Metering API

February 27, 2020 | 8 minute read
Stefan Hinker
Text Size 100%:

Anything the OCI Console displays is the result of REST calls to one of the various APIs through which all OCI services are available. After all, the Console is just as much a client to these APIs as the OCI CLI, the Python SDK or the Terraform provider. However, there is one area where the Console provides data not available via any of these other API clients: Cost Analysis. In fact, looking over the list of the OCI APIs , none seem to address this topic. The Budgets API (see concepts) is the only one touching the area of cost, but it provides what is available in the OCI Console under Account Management/Budgets, where you can configure budgets and alerts. So where is the source for the data displayed on the Cost Analysis screen?

Console Cost Analysis

The answer can be found in the Oracle Cloud Account Metering REST API. This blog post will introduce you to the API, how to authenticate to it and how to understand the data returned.

The first thing to understand about this API is that it is not a native OCI API. It was created along with the first version of Oracle Cloud, now called OCI Classic. However, unlike the more technical APIs for compute or networking, this API needs to report usage and cost for a cloud account – something that usually relates to both versions of Oracle Cloud. Essentially, the data provided by the Metering API is all about account usage and cost and not so much about monitoring or configuring cloud artifacts like instances or subnets. So when the second generation of Oracle Cloud came around, the API backend was updated to report on both the new and the old infrastructure. It is helpful to keep this in mind when using this API.

Features and Limitations

The Metering API is focused on reporting cost for a tenancy. It provides methods to group and structure cost using cost tracking tags and compartments. This organizational view can be very useful when combined with a suitable design of cost tracking tags and compartments. However, the API is limited with regards to cost resolution – it will only report on resource types, but not on individual resource location or instances. For example, the API will report on the cost of all compute resources in a specified compartment. But there is no way to report on those resources grouped by region or to report cost for a single compute instance only. This level of usage detail is only available in the Usage Reports. However, these only report OCI native services and do not currently include cost information.

Authentication

Unlike the OCI API authentication, which is based on request signatures, the Metering API uses basic authentication with username and password. (Alternatively, you can also use token based authentication.) To select the correct account, your identity domain ID must be provided in the header of each request, and your cloud account or domain is part of the request URL.

To get the cloud account number (also called domain), go to the Service User Console available in the OCI Console User Profile menu. This used to be called MyServices Console in OCI Classic. The cloud account number is part of the URL for the Services Console, starting with “cacct-”, for example: “https://myservices-cacct-<account_id>.console.oraclecloud.com/mycloud/cloudportal/cloudHome”.

The identity domain ID (IDCS ID) can be found by selecting the Identity Admin Console from the Services dashboard. There, click About from the User Profile menu to see the Instance GUID, for example: “idcs-<instance_id>”.  See here for a three step guide.

Once all four parts of the authentication are available, we can begin to construct a first, simple API call. To get the details of your cloud account, we’d use the following basic call:

Method: GET
Path: /metering/api/v1/cloudbucks/{accountId}

The IDCS ID is provided in the header, along with the basic username and password information. The full call (using curl) would look like this:

curl -X GET 
     -u user@example.com:<password>
     -H "X-ID-TENANT-NAME:idcs-<instance_id>" 
     "https://itra.oraclecloud.com/metering/api/v1/cloudbucks/cacct-<account_id>"

You can use this simple example, taken from the API documentation, to confirm that authentication works.

Data Returned by the API

Let’s take a more interesting call to look at the data returned by the API and how it is structured. To query total cost for a specific period of time, we use the following API request:

/metering/api/v1/usagecost/{accountId}

Required parameters are

  • startTime
  • endTime
to specify the time window for the query.

The parameter usageType defaults to TOTAL, which will report a single line item for every resource with total cost for the time period. If, instead, you set this to HOURLY or DAILY, each resource will be reported as many times as there are hours or days in the requested time interval, with usage and cost for that hour or day.

A full example to query the total cost for January 2020 with curl would look like this:

curl -X GET 
     -u username@example.com:SomePassword!2
     -H "X-ID-TENANT-NAME:idcs-<instance_id>" 
      "https://itra.oraclecloud.com/metering/api/v1/usagecost/
       cacct-<account_id>?
       startTime=2020-01-01T00:00:00.000&
       endTime=2020-01-31T23:59:59.999000.000"

The returned data is formatted in JSON and looks like this:

{
    "accountId": "cacct-<account_id>",
    "items": [
        {
            "subscriptionId": "<subscription_id>",
            "subscriptionType": "PRODUCTION",
            "serviceName": "ADWC",
            "resourceName": "ADWC_PAAS_BYOL_OCPU_HOUR",
            "currency": "USD",
            "gsiProductId": "B89039",
            "startTimeUtc": "2020-01-01T00:00:00.000",
            "endTimeUtc": "2020-01-31T23:59:59.999",
            "serviceEntitlementId": "<entitlement_id>",
            "costs": [
                {
                    "computedQuantity": 153,
                    "computedAmount": 74.0367,
                    "unitPrice": 0.4839,
                    "overagesFlag": "Y"
                }
            ]
        },
        {
            "subscriptionId": "<subscription_id>",
            "subscriptionType": "PRODUCTION",
            "serviceName": "ADWC",
            "resourceName": "ADWC_PAAS_EP_OCPU_HOUR",
            "currency": "USD",
            "gsiProductId": "B89040",
            "startTimeUtc": "2020-01-01T00:00:00.000",
            "endTimeUtc": "2020-01-31T23:59:59.999",
            "serviceEntitlementId": "<entitlement_id>",
            "costs": [
                {
                    "computedQuantity": 60,
                    "computedAmount": 151.212,
                    "unitPrice": 2.5202,
                    "overagesFlag": "Y"
                }
            ]
        },

It will continue for more items. You can use your favorite JSON parser to extract the fields of interest. Note the currency, which will show the currency you chose when creating the tenancy.

Cost data is structured in two sub-categories: Services (denoted by serviceName) and resources (resourceName), where each service can include multiple resources. There will be one item for each combination of serviceName and resourceName that has non-zero usage or cost. Each resource reports its consumed quantity and the associated price and cost for the period of time. So in order to digest the full result, you will need to act on each service and its resource costs. Since the returned data depends on what has been consumed, your processing should be able to deal with varying services and resources. In the most simple approach, just adding up all the costs (computedAmount) for all combinations of service and resource would return the total cost for the time period queried.

Cost Tracking Tags and Compartments

One nice feature of OCI are cost tracking tags. Once defined, you can use them to group resources for financial reporting. For example, you could have a tag namespace Billing with a tag key department and the two possible values HR and Finance. You can then report on the cost of all resources tagged with these tags. Of course, this works best if you automate the tagging of resources in some way.

Once defined, all cost tracking tags are visible via the Metering API. You can query it for all known tags. Since cost tracking tags are a native OCI feature, the API (being pre-OCI) will report them as NON_NATIVE. The API call for them is:

https://itra.oraclecloud.com/metering/api/v1/usage/{domain}/tags?tagType=NON_NATIVE

In our example, it would return

{
    "tag": "Billing:department=HR",
    "tagType": "NON_NATIVE",
    "createdOn": "2019-03-11T20:02:15.119"
},
{
    "tag": "Billing:department=Finance",
    "tagType": "NON_NATIVE",
    "createdOn": "2019-03-13T16:25:09.220"
}

So if we wanted to know the cost for January of all resources tagged with HR, the API call would be this:

curl -X GET  \
     -u <username>:<password> \
     -H "X-ID-TENANT-NAME:idcs-<instance_id>"  \
     "https://itra.oraclecloud.com/metering/api/v1/usagecost/cacct-<account_id>/tagged?
         startTime=2020-01-01T00:00:00.000&
         endTime=2020-01-31T23:59:59.999000.000&
         usageType=TOTAL&
         tags=Billing:department=HR"

Note the “/tagged” at the end of the request URL.

For OCI tenancies created after December 17, 2019, a tag default automatically adds a Oracle-Tag:CreatedBy tag. You can use this to track the cost of resources by creator, or add some simple automation using this tag to populate other tags as might be useful for your use case.

In OCI, compartments are an important concept within IAM to structure both user access and resource groupings (see References at the end of this page). All compartments are automatically injected into the Metering API’s database as cost tracking tags. Therefore, you can report on the cost of a single compartment using the call from the example above, just with a different string for the tag. The additional advantage is that all OCI resources are a member of exactly one compartment, so no extra automation is required to structure your resources in this way. For example, if you have all your database resources in a Database compartment, the call to retrieve cost for January for this compartment would be:

curl -X GET  \
     -u <username>:<password> \
     -H "X-ID-TENANT-NAME:idcs-<instance_id>"  \
     "https://itra.oraclecloud.com/metering/api/v1/usagecost/cacct-<account_id>/tagged?
         startTime=2020-01-01T00:00:00.000&
         endTime=2020-01-31T23:59:59.999000.000&
         usageType=TOTAL&
         tags=ORCL:OCICompartmentName=Database"

Note that compartment names need not be unique – you could have two compartments named Database as sub-compartments of Production and Test. I therefore recommend to use the compartment OCID for this call, which is always unique. The tag string in the request then changes to:

tags=ORCL:OCICompartment=<compartment_ocid>

If you intend to report on all of your compartments, don’t forget to also include the root compartment in your queries. The OCID of the root compartment is the OCID of your OCI tenancy.

It is important to understand that the API returns cost for all resources in the given compartment, but not in any sub-compartments. The API has no understanding of the hierarchy of compartments and treats compartments just like any other cost tracking tag. You can add multiple tags parameters to a single call. However, the response will not show any association between compartment and cost, so it is often desirable to query each compartment separately in order to retain the cost association.

Cost of Classic Services

As you explore the cost structure of your tenancy using cost tracking tags and compartments, you might notice that there is a difference between the aggregate cost of all your compartments (including the root compartment) and the total cost reported for your tenancy. This will be the case whenever you use certain services that are non-native to OCI. Some common examples are IDCS or CASB. Those services, as of the writing of this blog, are integrated from the previous release of Oracle Cloud. Since this Classic cloud does not have a notion of compartments, these services are not associated with any compartment. Therefore, they are only reported for the whole tenancy.

One way to make this very apparent is to collect the tenancy cost (with a call to the account usagecost API call) and then deduct all cost elements (services and their respective resources) for all compartments in the tenancy from these. The remaining, non-zero services and resources are those that are not associated with any compartment.

Working with the Data

Finally, here are some short examples how to process the data returned by the API. As we’ve seen above, each line item is returned in an item structure with usage, pricing and cost information for the item. Parsing this with jq we can process the data in multiple ways:

  • Convert it to CSV for easy importing into a spreadsheet: 
    	jq -r '.items[]|[.serviceName,.resourceName,.costs[].computedAmount]|@csv'
  • Listing all resources for a specific service 
    	jq -r '.items[]|select (.serviceName == "COMPUTEBAREMETAL")|.resourceName'
  • Adding up the cost for a specific service 
    	jq -r '.items[]|select (.serviceName == "COMPUTEBAREMETAL") |
           .costs[].computedAmount'|awk '{sum+=$0} END {print sum}'

Of course, this can also be done in many other ways, but this would need to be covered in another blog entry.

Summary

In this blog, we discussed the scope and functionality of the Oracle Cloud Metering API with its features and limitations. We saw a few examples for calling it and looked at the data returned. Cost tracking tags and compartments are the key to structuring cost data. We saw how to query cost using these tags and how to process the data. Based on this, you should now be able to write your own queries to get a better understanding of the cost of your organization’s resources.

References

Stefan Hinker

After around 20 years working on SPARC and Solaris, I am now a member of A-Team, focusing on infrastructure on Oracle Cloud.


Previous Post

Part 2: Implementing Desktop SSO using Azure AD, IDCS, and App Gateway

Tim Melander | 9 min read

Next Post

Collecting diagnostics for troubleshooting of UI performance issues in Fusion Apps

Ingolf Loboda | 14 min read
Resources for
Why Oracle
Learn
What's New
Contact Us