Meter Ingestion - Event schema
The default ingest event schema supported by Amberflo is as follows:
{
"customerId": string,
"meterApiName": string,
"meterValue": number,
"meterTimeInMillis": number,
"uniqueId": string
"dimensions": map<string, string>
}
Amberflo supports the following input formats
- JSON
- NDJSON
- CSV
Amberflo also supports defining a custom schema, which transforms the input into Amberflo ingest events. This guide explains how to use custom schema for ingesting meter events.
JSON format
Sample Input in JSON format.
[{
"customerId": "customer-123",
"meterApiName": "ComputeHours",
"meterValue": 5,
"meterTimeInMillis": 1619445706909,
"dimensions": {
"region": "us-west-2",
"az": "az1"
}
}, {
...
}]
NDJSON format
Amberflo also supports the NDJSON format, where each record is separated by a newline.
Sample Input in NDJSON format.
{ "customerId": "customer-123", "meterApiName": "ComputeHours", "meterValue": 5, "meterTimeInMillis": 1619445706909 }
{ "customerId": "customer-321", "meterApiName": "ComputeHours", "meterValue": 4, "meterTimeInMillis": 1619445712341 }
{ "customerId": "customer-123", "meterApiName": "ComputeHours", "meterValue": 1, "meterTimeInMillis": 1619445783456 }
CSV format
Sample Input in CSV format.
meterApiName,meterValue,meterTimeInMillis,customerId,region
ComputeHours,5,1619445706909,customer-123,us-west-2
ComputeHours,4,1619445712341,customer-321,us-east-1
ComputeHours,1,1619445783456,customer-123,us-west-2
Difference in behavior for CSV and JSON inputs
If custom schema is not used, then for CSV input the columns not part of Amberflo default schema will be converted to dimensions, whereas for JSON input keys which are not part of Amberflo default schema are excluded.
Lets take a look at the below table to understand this.
Input | Transformed Input | |||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
CSV |
| { | ||||||||||
JSON | { | { |
As we can see that the region
column/key in CSV and JSON inputs does not belong to default Amberflo schema. So we can see in the transformed input that the region
column has been transformed into dimensions for CSV input whereas it is excluded for the JSON input.
If a custom schema is used then only the columns/key mapped in the schema sub components will be considered rest others will be excluded for both CSV and JSON.
Defining custom ingest schema
The Amberflo Custom Ingest Schema API lets you define your own custom schema based on your events.
This custom schema can be used with both CSV and JSON inputs. The schema has sub components which define the mapping of a particular source column-name/json-key to the amberflo ingest event parameter.
Note
- Each schema should contain at least sub components for
meter api name
,meter value
,customer id
andevent time
. - Each sub component should have a different source name, multiple sub components cannot have same source name.
For instance consider the following custom schema:
{
"id": "custom-schema-id",
"schema": {
"type": "explode",
"subComponents": [
{
"type": "meter_api_name",
"sourceName": "met"
},
{
"type": "meter_value",
"sourceName": "val"
},
{
"type": "event_time",
"timeFormat": "EPOCH_SECONDS",
"sourceName": "time"
},
{
"type": "customer_id",
"sourceName": "c_id"
}
]
}
}
In this custom schema you have 4 sub components which map 4 different columns/keys to ingest event parameters.
CSV example
Sample CSV Input
met | val | time | c_id |
---|---|---|---|
m1 | 1 | 1714070079 | c1 |
m1 | 1 | 1714070079 | c2 |
Using the custom schema the above CSV input will be transformed to the following ingest payload
[
{
"meterApiName": "m1",
"customerId": "c1",
"meterValue": 1,
"meterTimeInMillis": 1714070079000
},
{
"meterApiName": "m1",
"customerId": "c2",
"meterValue": 1,
"meterTimeInMillis": 1714070079000
}
]
JSON example
Sample JSON input
[
{"met": "m2", "c_id": "c3", "val": 10, "time": 1714073679},
{"met": "m2", "c_id": "c4", "val": 10, "time": 1714073679}
]
Using the custom schema the above json input will be transformed to the following ingest payload
[
{
"meterApiName": "m2",
"customerId": "c3",
"meterValue": 10,
"meterTimeInMillis": 1714073679000
},
{
"meterApiName": "m2",
"customerId": "c3",
"meterValue": 10,
"meterTimeInMillis": 1714073679000
}
]
Using custom ingest schema
Once you have created the custom ingest schema, you can use this schema to parse your meter events.
Ingesting through API
To use the custom schema for parsing ingest payload
- Get the id of the schema that you want to use for parsing.
- Provide the schema id as a query string parameter named
ingestSchemaId
.
Example
https://app.amberflo.io/ingest?ingestSchemaId=<schema-id>
Sample curl request to ingest events using the custom schema defined in previous section
curl --request POST \
--url 'https://app.amberflo.io/ingest?ingestSchemaId=custom-schema-id' \
--header 'accept: application/json' \
--header 'content-type: application/json' \
--header 'x-api-key: <replace-with-api-key>' \
--data '
{
"c_id": "c3",
"met": "m2",
"val": 1,
"time": 1714070079
}
'
Ingesting through AWS S3
To use custom schema when ingesting events using AWS S3 the ingestSchemaId
should be provided in the s3 file key.
S3 key format for using custom schema
/ingest/amberdata/06-07-2022/ingestSchemaId=<schema-id>/0000a2e4-e6ad-11ec-8293-6a8da1c4f9f0
Updated 6 months ago