Guides
  1. Data ingestion

DataHub API

The DataHub API allows you to import data from internal and external sources into the DoiT Console. You can leverage this functionality to monitor your internal resource usage, optimize resource allocation and manage operational expenses.

Operations

Data formats

The DataHub API supports two data formats:

Data formatMedia typeLimits
JSONapplication/jsonUp to 255 events per request
CSV file:
  • uncompressed
  • compressed in ZIP or GZ format
multipart/form-dataMaximum file size: 30MB

If you choose CSV, be aware that:

  • The DataHub API only accepts files no larger than 30MB. If your CSV is beyond 30MB, compress it using ZIP or GZ.

  • The Data Hub API doesn't accept ZIP or GZ archives that contain more than one CSV.

See also rate limits.

Data latency

Once you receive a successful response (HTTP 201 OK), the ingested data will be available in the DoiT console within 15 minutes, regardless of the data formats.

JSON format

Events schema

PropertyTypeDescription
provider (required)string

This field is used to identify the data provider. Choose a human-readable value to easily identify the dataset. For example, Datadog, Databricks, Snowflake, etc. The default value is ingestion-api.

Allowed characters: alphanumeric (0-9, a-z, A-Z), underscore (_), space, dash (-)

idstringA unique identifier for the event. If not set, a UUIDv4 is generated at ingestion time.
DataHub uses the event ID to handle duplicate data (newer data overwrites existing ones). You also need the event ID to delete an incorrect ingestion (an event can be deleted 90 minutes after ingestion).
dimensions.typestringPossible values: fixed, label, project_label, system_label
dimensions.keystringIf the type is fixed, the key must be one of the allowed keys for fixed dimensions.
dimensions.valuestringThe value of the dimension.
time (required)string (RFC3339)The time that the event occurred. For example, 2024-03-10T23:00:00Z.
metrics.typestringThe type of metric that maps to the predefined or user-defined metrics. Possible values: cost, usage, savings, or custom metric. This allows you to consolidate your DataHub data together with your regular billing data.
metrics.valuenumberValue for the metric (e.g., cost in dollars, usage in GB, custom metric count).

Allowed keys for fixed dimensions

Event dimension keyReport field
billing_account_idBilling Account ID
countryCountry
is_marketplaceMarketplace
operationOperation
pricing_unitUnit
project_idProject/Account ID
project_nameProject/Account Name
project_numberProject/Account Number
regionRegion
resource_idResource
resource_global_idGlobal Resource
service_descriptionService
service_idService ID
sku_descriptionSKU
sku_idSKU ID
zoneZone
👍

Note

See Standard dimensions for details about the report fields.

Sample request: Publish events

📘

YOUR_API_KEY

Replace "YOUR_API_KEY" with your actual API key as explained at Get Started

curl --request POST \
--url https://api.doit.com/datahub/v1/events \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {YOUR_API_KEY}' \
--data '{
"events": [
    {
      "provider": "Datadog",
      "id": "12345678-1234-1234-1234-123412341234",
      "dimensions": [
        {
          "key": "service_description",
          "value": "Devices",
          "type": "fixed"
        },
        {
          "key": "region",
          "value": "us-east-1b",
          "type": "fixed"
        },
        {
          "key": "zone",
          "value": "us-east-1b",
          "type": "fixed",
        },
        {
         "key": "env",
         "value": "prod",
         "type": "label",
        },
        {
         "key": "team",
         "value": "alpha",
         "type": "label",
        },
        {
         "key": "department",
         "value": "rnd",
         "type": "project_label",
        }
       ],
      "time": "2024-02-23T12:00:00.00Z",
      "metrics": [
        {
          "value": 352,
          "type": "cost",
         },
         {
          "value": 36,
          "type": "usage",
         }
        ]
      }
  ]
}
'

CSV file

See CSV ingestion for the syntax, conventions, and limitations when sending data using CSV.

📘

YOUR_API_KEY

Replace "YOUR_API_KEY" with your actual API key as explained at Get Started

Sample: Publish events using a CSV compressed in ZIP

curl --request POST \
--url https://api.doit.com/datahub/v1/csv/upload \
--header ‘Content-Type: multipart/form-data’ \
--header 'Authorization: Bearer {YOUR_API_KEY}' \
--form 'provider="mysuperdataset"' \
--form 'file=@“/Users/me/test_csv.zip”’
{
    "batch": "test_csv.zip_1730885309695",
    "ingestedRows": 2
}