Skip to main content

Push data to Timeplus via ingest REST API

As a generic solution, you can call the ingestion REST API to push data to Timeplus, with any preferred languages. With the recent enhancements of the ingest API, in many cases, you can configure other systems to push data directly to Timeplus via webhook, without writing code.

Please check https://docs.timeplus.com/rest for the detailed API documentations.

Create a stream in Timeplus

First you need to create a stream in Timeplus, either using the web UI or via REST API. Columns with proper names and types should be set. In the following section, we assume the stream name is foo.

Set API Key in HTTP Header

Please generate an API Key for a workspace and set the API Key in the HTTP Header, with the name: X-Api-Key

info

If you would like to leverage a 3rd party system/tool to push data to Timeplus but it doesn't allow custom HTTP Header, then you can use the standard Authorization Header, with value ApiKey $KEY.

For Timeplus Enterprise self-hosted deployments, you need to set HTTP Authorization header to be Basic [Base64 encoded user:password].

Push data to Timeplus

Endpoint

The endpoint for real-time data ingestion is https://us.timeplus.cloud/{workspace-id}/api/v1beta2/streams/{name}/ingest

info

Make sure you are using the workspace-id, instead of workspace-name. The workspace id is a random string with 8 characters. You can get it from the browser address bar: https://us.timeplus.cloud/<workspace-id>/console. The workspace name is a friendly name you set while you create your workspace. Currently this name is read only but we will make it editable in the future.

You need to send POST request to this endpoint, e.g. https://us.timeplus.cloud/ws123456/api/v1beta2/streams/foo/ingest

Options

Depending on your use cases, there are many options to push data to Timeplus via REST API. You can set different Content-Type in the HTTP Header, and add the format query parameter in the URL.

Here are a list of different use cases to push data to Timeplus:

Use CasesSample POST bodyContent-TypeURLColumns in the target stream
1) Push JSON objects. Each JSON is an event.{"key1": "value11", "key2": "value12", ...}
{"key1": "value21", "key2": "value22", ...}
application/x-ndjsoningest?format=streamingmultiple columns, e.g. key1, key2
2) Push a single JSON or a long text. Single event.{"key1": "value11", "key2": "value12", ...}text/plainingest?format=rawsingle column, named raw
3) Push a batch of events. Each line is an event.event1
event2
text/plainingest?format=linessingle column, named raw
4) Push a special JSON with multiple events, without repeating the column name{
"columns": ["key1","key2"],
"data": [
["value11","value12"],
["value21","value22"],
]
}
application/jsoningest?format=compact or just ingestmultiple columns, e.g. key1, key2

1) Push JSON objects directly

Request samples

curl -s -X POST -H "X-Api-Key: your_api_key" \
-H "Content-Type: application/x-ndjson" \
https://us.timeplus.cloud/ws123456/api/v1beta2/streams/foo/ingest?format=streaming \
-d '
{"key1": "value11", "key2": "value12"}
{"key1": "value21", "key2": "value22"}
'

You can push Newline Delimited JSON (http://ndjson.org/) to the endpoint. Make sure you set the HTTP Header as one of these:

  • application/x-ndjson
  • application/vnd.timeplus+json;format=streaming
info

If you would like to leverage a 3rd party system/tool to push data to Timeplus but it doesn't allow custom content type, then you can use the standard application/json, and send POST request to /api/v1beta2/streams/$STREAM_NAME/ingest?format=streaming. This will ensure the Timeplus API server treats the POST data as NDJSON.

The request body is just a stream of JSON objects. e.g.

{"key1": "value11", "key2": "value12", ...}
{"key1": "value21", "key2": "value22", ...}
...

Each object does not have to be in a single line. For example:

{
"key1": "value11",
"key2": "value12", ...
}
{
"key1": "value21",
"key2": "value22", ...
}
...

They don’t have to be separated by newline either:

{"key1": "valueA", ...}{"key1": "valueB", ...}{"key1": "valueC", ...,
}...

Just make sure all columns in the target stream are specified with proper value in the request body.

2) Push a single JSON or string to a single column stream

Request samples

curl -s -X POST -H "X-Api-Key: your_api_key" \
-H "Content-Type: text/plain" \
https://us.timeplus.cloud/ws123456/api/v1beta2/streams/foo/ingest?format=raw \
-d '
{"key1": "value11", "key2": "value12"}
'

It's a common practice to create a stream in Timeplus with a single string column, called raw. You can put JSON objects in this column then extract values (such as select raw:key1), or put the raw log message in this column.

When you set Content-Type header to text/plain, and add format=raw to the ingestion endpoint, the entire body in the POST request will be put in the raw column.

3) Push multiple JSON or text to a single column stream. Each line is an event

Request samples

curl -s -X POST -H "X-Api-Key: your_api_key" \
-H "Content-Type: text/plain" \
https://us.timeplus.cloud/ws123456/api/v1beta2/streams/foo/ingest?format=lines \
-d '{"key1": "value11", "key2": "value12"}
{"key1": "value21", "key2": "value22"}
'

When you set Content-Type header to text/plain, and add format=lines to the ingestion endpoint, each line in the POST body will be put in the raw column.

4) Push multiple events in a batch without repeating the columns

Request samples

curl -s -X POST -H "X-Api-Key: your_api_key" \
-H "Content-Type: application/json" \
https://us.timeplus.cloud/ws123456/api/v1beta2/streams/foo/ingest \
-d '
{
"columns": ["key1","key2"],
"data": [
["value11","value12"],
["value21","value22"],
]
}
'

The above method should work very well for most system integration. However, the column names will be repeatedly mentioned in the requested body.

We also provide a more performant solution to only list the column names once.

Same endpoint URL: https://us.timeplus.cloud/{workspace-id}/api/v1beta2/streams/{name}/ingest

But you need to set the HTTP Header to one of these:

  • application/json
  • application/vnd.timeplus+json
  • application/vnd.timeplus+json;format=compact

The request body is this format:

{
"columns": [..],
"data": [
[..],
[..],
]
}

Note:

  • the columns is an array of string, with the column names
  • the data is an array of arrays. Each nested array represents a row of data. The value order must match the exact same order in the columns.

For example:

{
"columns": ["key1", "key2"],
"data": [
["value11", "value12"],
["value21", "value22"]
]
}

You can also use one of our SDKs to ingest data without handling the low level details of REST API.