AboutHelp CenterDeveloper SupportDoiT console
Guides
Start typing to search…

DataHub API

Ingest data from various sources for contextualized analysis of your spend.

The DataHub API allows you to import data from internal and external sources into the DoiT Console. You can leverage this functionality to monitor your internal resource usage, optimize resource allocation and manage operational expenses.

Operations

Data formats

The DataHub API supports two data formats:

Data format

Media type

Limits

JSON

application/json

Up to 255 events per request

CSV file:

  • uncompressed
  • compressed in ZIP or GZ format

multipart/form-data

Maximum file size: 30MB

If you choose CSV, be aware that:

  • The DataHub API only accepts files no larger than 30MB. If your CSV is beyond 30MB, compress it using ZIP or GZ.

  • The Data Hub API doesn't accept ZIP or GZ archives that contain more than one CSV.

See also rate limits.

Data latency

Once you receive a success response (HTTP 201 OK), the ingested data will be available in the DoiT console within 15 minutes, regardless of the data formats.

JSON format

Events schema

Property

Type

Description

provider (required)

string

This field is used to identify the data provider. Choose a human-readable value so you can easily recognize the dataset. For example, Datadog, Databricks, Snowflake, etc. The default value is ingestion-api.

Allowed characters: alphanumeric (0-9, a-z, A-Z), underscore (_), space, dash (-)

id

string

A unique identifier for the event. If not set, a UUIDv4 is generated at ingestion time.
DataHub uses the event ID to handle duplicate data (newer data overwrites existing ones). You also need the event ID to delete an incorrect ingestion (an event can be deleted 90 minutes after ingestion).

dimensions.type

string

Possible values: fixed, label, project_label, system_label

dimensions.key

string

If the type is fixed, the key must be one of the allowed keys for fixed dimensions.

dimensions.value

string

The value of the dimension.

time (required)

string (RFC3339)

The time that the event occurred. For example, 2024-03-10T23:00:00Z.

metrics.type

string

For integration with billing data. Possible values: cost, usage, or custom metric.

metrics.value

number

Value for the metric.

Allowed keys for fixed dimensions

Event dimension keyReport field
billing_account_idBilling Account ID
countryCountry
is_marketplaceMarketplace
operationOperation
pricing_unitUnit
project_idProject/Account ID
project_nameProject/Account Name
project_numberProject/Account Number
regionRegion
resource_idResource
resource_global_idGlobal Resource
service_descriptionService
service_idService ID
sku_descriptionSKU
sku_idSKU ID
zoneZone
👍

Note

See Standard dimensions for details about the report fields.

Sample request: Publish events

📘

YOUR_API_KEY

Replace "YOUR_API_KEY" with your actual API key as explained at Get Started

curl --request POST \
--url https://api.doit.com/datahub/v1/events \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {YOUR_API_KEY}' \
--data '{
"events": [
    {
      "provider": "Datadog",
      "id": "12345678-1234-1234-1234-123412341234",
      "dimensions": [
        {
          "key": "service_description",
          "value": "Devices",
          "type": "fixed"
        },
        {
          "key": "region",
          "value": "us-east-1b",
          "type": "fixed"
        },
        {
          "key": "zone",
          "value": "us-east-1b",
          "type": "fixed",
        },
        {
         "key": "env",
         "value": "prod",
         "type": "label",
        },
        {
         "key": "team",
         "value": "alpha",
         "type": "label",
        },
        {
         "key": "department",
         "value": "rnd",
         "type": "project_label",
        }
       ],
      "time": "2024-02-23T12:00:00.00Z",
      "metrics": [
        {
          "value": 352,
          "type": "cost",
         },
         {
          "value": 36,
          "type": "usage",
         }
        ]
      }
  ]
}
'

CSV file

See CSV ingestion for the syntax, conventions, and limitations when sending data using CSV.

📘

YOUR_API_KEY

Replace "YOUR_API_KEY" with your actual API key as explained at Get Started

Sample: Publish events using a CSV compressed in ZIP

curl --request POST \
--url https://api.doit.com/datahub/v1/csv/upload \
--header ‘Content-Type: multipart/form-data’ \
--header 'Authorization: Bearer {YOUR_API_KEY}' \
--form 'provider="mysuperdataset"' \
--form 'file=@“/Users/me/test_csv.zip”’
{
    "batch": "test_csv.zip_1730885309695",
    "ingestedRows": 2
}

Updated about 1 month ago