Timeplus Cloud v1 and Proton
This page summarizes changes for each major update in Proton and Timeplus Cloud, including new features and important bug fixes.
Jun 24, 2024
Timeplus Core Engine (Proton v1.5.10):
- Avro-Encoded Messages: Previously, Schema Registry must be enabled to read Avro-encoded messages. Now, SQL can be used to define the Avro schema and read these messages. Learn more
- Improved Proton Client:
-h 127.0.0.1
is no longer used when launching the proton-client. Proton v1.5.10 listens on both IPv4 and IPv6 ports.
May 28, 2024
Timeplus Core Engine (Proton v1.5.9):
- You can now run table function on Kafka External Streams. This will read all existing data in the Kafka topic, e.g.
select * from table(topic) where condition=true
. - To get the number of messages in a Kafka topic, you can run
select count() from table(topic)
. This will return the message count efficiently in less than 1 second, without scanning each message body. - A new advanced setting is available to control the speed of historical data backfill with a certain column. 。
select * from test_stream where _tp_time > earliest_timestamp() settings replay_speed=1, replay_time_column='time_col'
. - New SQL functions: parse_datetime and parse_datetime_in_joda_syntax. For example
parse_datetime('2021-01-04+23:00:00', '%Y-%m-%d+%H:%i:%s')
returns a value indatetime
type, for '2021-01-04 23:00:00'.
Timeplus Console:
- The number of External Tables is now shown in the "Workspace at a Glance" section on the Homepage.
- Refined the SQL editor's height adjustment and lock height behavior.
- Workspace owners can now see who created API keys.
May 13, 2024
Timeplus Core Engine (Proton v1.5.8):
- A new setting in
config.yaml
:max_consumers_per_stream
defines how many consumers can be created for each Kafka external stream. When a SELECT query on a Kafka external stream is executed, a consumer will be created for that query. This means no more thanmax_consumers_per_stream
SELECT queries can be executed in parallel on the same Kafka external stream. - New example in the Timeplus Proton repo /examples folder: Analyzing Nginx Access Logs, contributed by Timeplus Community member Saïd Abiola
Timeplus Console:
- Send us questions or comments via the new “Talk to Us” button in the top header.
- "External Tables" resource list now available from the left navigation menu.
- To see resource schema in the SQL editor, mouseover on the underlined resource and press Ctrl + click to open the details side panel.
- For new charts, the default color will be pink (color scheme will be
Dawn
). - In charts, the legend will be hidden when the series count exceeds 30.
Apr 29, 2024
Timeplus Core Engine (Proton v1.5.7):
- You can now join multiple versioned streams using
LEFT JOIN
and by assigning primary key(s). Results will be emitted whenever there are updates to either side of the JOIN. Learn more - New examples in the Timeplus Proton repo /examples folder:
- One Billion Rows Challenge (1BRC), contributed by Timeplus Community member Saïd Abiola
- Real-time retrieval-augmented generation (RAG)
Timeplus Console:
- External tables configuration: A new wizard is available to integrate with ClickHouse.
- New "Save As" button on the Query Page: Save SQL as a view, materialized view, or bookmark.
- In the SQL editor, click the name of the stream, view, or materialized view to show the resource's schema.
- After creating a new source or external stream, you will now be redirected to the SQL Console page.
- View and edit resource details: Removed "Edit" from high-level actions in resource lists – it's now available in the side panel after clicking the "View" icon.
Apr 15, 2024
Timeplus Core Engine (Proton v1.5.5 and v1.5.6):
- Added two new Apache Kafka external stream settings for Kafka broker certifications:
skip_ssl_cert_check
allows you to skip verifying the server's certificationssl_ca_pem
allows you to specify the Certificate Authority (CA) content, without needing to upload the CA file
- Added functions for datetime:
to_unix_timestamp64_milli
returns the UNIX timestamp with millisecond of the datetime64to_unix_timestamp64_micro
returns with microsecondto_unix_timestamp64_nano
returns with nanosecond
- Newly created streams will have streaming data retention set to 1 day storage by default.
- On MacOS,
sudo proton install
will install proton to /usr/local/bin instead of /usr/bin.
Timeplus Cloud and Timeplus Enterprise:
- Added workspace templates: you can create a new workspace from scratch, or with a demo template (Market Data or Stream Processing).
- New layout for Data Ingestion page, with labels to show
Source
orExternal Stream
. - In Data Lineage, we've optimized the layout for when there are large amounts of resources in the workspace. Additionally, zooming out will show the icon and color, and added a button to zoom out to show all.
- Added TLS settings and Certificate Authority (CA) input field for Apache Kafka external streams and sinks.
Apr 1, 2024
Timeplus Proton:
- Added support for selecting message key from Apache Kafka external streams, via
SELECT _message_key, * FROM external_stream
. - Streaming processing now supports nullable data type.
- External Table: ClickHouse external tables with names containing special characters (such as dashes) are now supported. Simply set
table='test-a-b'
in theCREATE EXTERNAL TABLE
DDL. - External Stream: Error handling and connection pooling/retry for Kafka external streams have been greatly improved.
- Materialized View: Added option to skip dirty/unexpected data. If you set
SETTINGS recovery_policy='best_effort'
, Timeplus will try up to 3 times, then skip dirty data and continue processing the rest of the data.
Timeplus Cloud and Timeplus Enterprise:
- Added built-in support for Redpanda Serverless. See our how-to guide
- Ingesting Apache Kafka data: We changed the implementation of Kafka integration from Benthos-based implementations to external stream-based. Our configuration wizard in the web console remains the same.
- New left menu item for “Sources”. The “Data Ingestion” page will now only show options to ingest data, and not list added sources.
- Based on user feedback, we enhanced the built-in OHLC chart to show live updates even when the current window of OHLC is not closed.
- Updated layout for Home page, with new stats on existing resources, and number of currently running queries in the top right corner. If you don't have a certain resource in your workspace, we will now hide the stat box, instead of showing 0.
- Enhancements to the 3 demos in our Timeplus Demo workspace. In the Stream Processing demo, we've added a dashboard to show how to calculate standard deviation with live data and streaming SQL, and use query variables.
Mar 18, 2024
Introducing three new demos with live data in our Timeplus Demo workspace: Stream Processing, Market Data, and ksqlDB Alternative. Choose a demo via the dropdown in the top header. You can explore the demos in read-only mode, or take a quick guided tour.
Timeplus Cloud:
- Data Definition Language (DDL) is supported in the SQL Console (previously called the Query page). You can modify resources using commands such as
CREATE
andDROP
. - In Data Lineage, after clicking a resource tile, you can make edits to it directly from the details side panel. Please note that only certain fields may be edited for a particular resource type.
- Also in Data Lineage, ClickHouse external tables are now included. Click the tile to see more details, such as the address and username.
- In addition to streams, you can now set external streams and external tables as the target for materialized views.
Mar 4, 2024
Proton:
- Proton can now natively integrate with ClickHouse, available for both ClickHouse Cloud or local/self-managed versions of ClickHouse. Learn more
- Bulk CSV import is enhanced, in Proton 1.5.2. You can load billions of rows in multiple CSV files via a single SQL. Learn more
- Kafka Schema Registry is supported with Protobuf and Avro format (Proton 1.5.2). Learn more
- Self-signed HTTPS certification for Schema Registry is supported (Proton 1.5.3).
- Proton now can be compiled on SUSE Linux.
Timeplus Cloud:
- In Data Lineage side panels, more details about the resources are added. Click on a tile to view.
- Data types such as
float
orinteger
are depreciated. The UI will show precise data types such asint8
,uint16
, etc. - In dashboard charts, the legend is maintained when rendering. Click to show or hide series.
Feb 20, 2024
Proton:
- In Proton v1.5.1, we introduced more streaming result emit polices, based on feedback from ksqlDB users. Most notably, when you run tumble/hop/session window aggregations, before the window closes, the intermediate aggregation results can be emitted at a certain interval or when the value changes. Learn more in our docs
- You can now install Proton as a single binary via
curl https://install.timeplus.com | sh
. - Besides GitHub Container Registry (GHCR), you can also pull Proton Docker via
docker pull public.ecr.aws/timeplus/proton
. - A first version of the Rust Client for Proton is now available at https://crates.io/crates/proton_client, contributed by Marvin Hansen (Director of Emet-Labs).
Timeplus Cloud:
- For NATS data source, we’ve added the option in the UI to choose either JWT or NKey file content for authentication.
- When you add a Confluent Cloud data source with Avro Schema Registry, the UI will suggest a set of column names for the new stream, but you’ll need to choose the proper data types. In the future, we will enhance it to load the data types from the schema registry.
- In an upcoming release, you will be able to run any SQL in Timeplus Cloud, including
CREATE EXTERNAL TABLE
and other Data Definition Language (DDL). Contact us for early access to this new feature.