# Introduction | Timeplus > Timeplus is a simple, powerful, and cost-efficient stream processing platform. ## Alert - [Alert | Timeplus](https://docs.timeplus.com/alert): Timeplus alerts enable you to monitor your streaming data and automatically trigger actions when specific conditions are met. When your streaming queries detect events of interest, alerts can notify stakeholders via email or Slack, send data to downstream systems like Apache Kafka, or execute custom Python functions for automated responses. - Core Features - Alert # Alert Timeplus alerts enable you to monitor your streaming data and automatically trigger actions when specific conditions are met. When your streaming queries detect events of interest, alerts can notify stakeholders via email or Slack, send data to downstream systems like Apache Kafka, or execute custom Python functions for automated responses. Starting with Timeplus Enterprise v2.9, the alert feature has been moved from the application server to the core engine for improved performance, stability, and SQL-based management. The previous application-level alerting feature will be deprecated in future releases. ## Create New Alert​ ### Syntax​ For example: ### Limitations​ - The alerts only run on the metadata leader node. - The return value of the Python UDF is ignored. - The select query cannot include any aggregation or JOIN. You can create a materialized view with complex JOIN or aggregation logic to cache the alert events and SELECT the target stream of the materialized view in the alert definition. - Check system.stream_state_log for the alert states or logs. - The checkpoints of the alerts are available in system.alert_ckpt_log stream with the _tp_sn column. ### Python UDF​ You can import Python libraries and build the custom alert action via Python UDF. The return value doesn't matter. Here is an example to send events to a specific Slack channel via Slack webhook: Please note, similar to regular Python UDF, the input parameter of the Python UDF is an array, instead of a single event. The return value can be anything but you can return the input value, so that you can test the Python UDF via SELECT udf_name(input). ## List Alerts​ Without SETTINGS verbose=true, it lists the alert name and its UUID. With SETTINGS verbose=true, the following columns are added: - version - last_modified - last_modified_by - created - created_by ## Show Alert Definition​ ## Drop Alerts​ - Create New AlertSyntaxLimitationsPython UDF - Syntax - Limitations - Python UDF - List Alerts - Show Alert Definition - Drop Alerts - Syntax - Limitations - Python UDF ## Append-stream - [Append Stream | Timeplus](https://docs.timeplus.com/append-stream): By default, the streams in Timeplus are Append Streams: - Core Features - Streams - Append Stream # Append Stream By default, the streams in Timeplus are Append Streams: - They are designed to handle a continuous flow of data, where new events are added to the end of the stream. - The data is saved in columnar storage, optimized for high throughput and low latency read and write. - Older data can be purged automatically by setting a retention policy, which helps manage storage costs and keeps the stream size manageable. - Limited capabilities to update or delete existing data, as the primary focus is on appending new data. ## Architecture - [Architecture | Timeplus](https://docs.timeplus.com/architecture): High Level Architecture - Overview - Architecture # Architecture ## High Level Architecture​ The following diagram depicts the high level architecture of Timeplus SQL engine, starting from a single node deployment. All of the components / functionalities are built into one single binary. ## Data Storage​ Users can create a stream by using CREATE STREAM ... DDL SQL. Every stream has 2 parts at storage layer by default: 1. the real-time streaming data part, backed by Timeplus NativeLog 2. the historical data part, backed by ClickHouse historical data store. Fundamentally, a stream in Proton is a regular database table with a replicated Write-Ahead-Log (WAL) in front but is streaming queryable. ## Data Ingestion​ When users INSERT INTO ... data to Proton, the data always first lands in NativeLog which is immediately queryable. Since NativeLog is in essence a replicated Write-Ahead-Log (WAL) and is append-only, it can support high frequent, low latency and large concurrent data ingestion work loads. In background, there is a separate thread tailing the delta data from NativeLog and commits the data in bigger batch to the historical data store. Since Proton leverages ClickHouse for the historical part, its historical query processing is blazing fast as well. ## External Stream​ In quite lots of scenarios, data is already in Kafka / Redpanda or other streaming data hubs, users can create external streams to point to the streaming data hub and do streaming query processing directly and then either materialize them in Proton or send the query results to external systems. ## Learn More​ Interested users can refer How Timeplus Unifies Streaming and Historical Data Processing blog for more details regarding its academic foundation and latest industry developments. You can also check the video below from Kris Jenkins's Developer Voices podcast. Jove shared our key decision choices, how Timeplus manages data and state, and how Timeplus achieves high performance with single binary. - High Level Architecture - Data Storage - Data Ingestion - External Stream - Learn More ## Automq-kafka-source - [Load streaming data from AutoMQ | Timeplus](https://docs.timeplus.com/automq-kafka-source): AutoMQ is a cloud-native version of Kafka redesigned for cloud environments. AutoMQ is open source and fully compatible with the Kafka protocol, fully leveraging cloud benefits. Compared to self-managed Apache Kafka, AutoMQ, with its cloud-native architecture, offers features like capacity auto-scaling, self-balancing of network traffic, moving partition in seconds. These features contribute to a significantly lower Total Cost of Ownership (TCO) for users. # Load streaming data from AutoMQ AutoMQ is a cloud-native version of Kafka redesigned for cloud environments. AutoMQ is open source and fully compatible with the Kafka protocol, fully leveraging cloud benefits. Compared to self-managed Apache Kafka, AutoMQ, with its cloud-native architecture, offers features like capacity auto-scaling, self-balancing of network traffic, moving partition in seconds. These features contribute to a significantly lower Total Cost of Ownership (TCO) for users. This article will guide you on how to load data from AutoMQ into Timeplus using the Timeplus Console. Since AutoMQ is 100% compatible with the Apache Kafka protocol, you can also create an external stream to analyze data in AutoMQ without moving it. To prepare your AutoMQ environment and test data, follow the AutoMQ Quick Start guide to deploy your AutoMQ Kafka cluster. For self-hosting deployments, you can install AutoMQ and Timeplus Enterprise on Linux or Mac with the following commands: For SaaS deployments, please ensure that Timeplus can directly connect to your AutoMQ server. You can use tools like ngrok to securely expose your local AutoMQ Kafka proxy on the internet, allowing Timeplus to connect to it. For more details, see the blog. Access the web console of Timeplus Enterprise and go the "Data Collection" page. Click on the AutoMQ tile. Set the broker(s) for AutoMQ. For example, for standalone deployment of AutoMQ on your local host, the hosts are localhost:9094,localhost:9095. Disable authentication and TLS if needed. Click Next. Timeplus will connect to the server and list all topics. Choose one topic and set whether you want to read it as JSON or Text. In the next step, confirm the schema of the Timeplus stream and specify a name. At the end of the wizard, an external stream will be created in Timeplus. You can query data or even write data to the AutoMQ topic with SQL. See also: - Kafka External Stream - Tutorial: Streaming ETL from Kafka to ClickHouse ## Bare-metal-install - [Deploy on Bare Metal | Timeplus](https://docs.timeplus.com/bare-metal-install): Timeplus Enterprise can be easily installed bare metal Linux or MacOS, as a single node or multi-node cluster. - Deployment & Operations - Timeplus Enterprise Self-hosted - Deploy on Bare Metal # Deploy on Bare Metal Timeplus Enterprise can be easily installed bare metal Linux or MacOS, as a single node or multi-node cluster. ## Single Node Install​ ### Install Script​ If your server or computer is running Linux or MacOS, you can run the following command to download the package and start Timeplus Enterprise without any other dependencies. For Windows users, please follow our guide for running Timeplus Enterprise with Docker. This script will download the latest release (based on your operating system and CPU architecture) to the current folder. Uncompress the package and start the Timeplus Enterprise server. If you'd like to download the package for a certain feature release, you can run the following command: Replace 2.8 with the desired version number. ### Manual Install​ You can also download packages manually with the following links: - Linux (Intel/AMD chip, ARM chip) - MacOS (Intel chip, Apple Silicon chip) - Windows (Please install via Docker) After you download the package, put it in a desired folder, uncompress the file, and you can run Timeplus Enterprise with preconfigured settings: Then change directory to the bin folder and run ### Timeplus Enterprise Processes​ This will start Timeplus Enterprise with a few key processes: - timeplusd: The core SQL engine, severing at port 8463 (TCP, for timeplus client) and 3218 (HTTP, for JDBC/ODBC drivers - timeplus_appserver: The application server, severing at HTTP port 8000 - timeplus_web: The web UI, managed by timeplus_appserver - timeplus_connector: The service to provide extra sources and sinks, managed by timeplus_appserver Access the Timeplus Enterprise web console via http://localhost:8000. On your first login, please create an account with a password to start the 30-day free trial. It is also possible to only start/stop single process by running timeplus start -s service_name. For example, when you only want to startup Timeplus Core (SQL Engine), run timeplus start -s timeplusd. For more information, please check the CLI Reference. ### Upgrade​ To upgrade Timeplus Enterprise, run timeplus stop to stop all the services. Then replace all the binaries to the higher version of Timeplus Enterprise release and then run timeplus start. ### Uninstall​ Timeplus Enterprise has no external dependencies. Just run timeplus stop then delete the timeplus folder. ## Docker Install​ Alternatively, run the following command to start Timeplus Enterprise with Docker: A few optional parameters: - Add --name timeplus to set a name for the container - Add -d to run the container in the background - Add -p 8463:8463 -p 3218:3218 if you want to run SQL with timeplus client or JDBC/ODBC drivers - Add -v "$(pwd)"/data:/timeplus/data -v "$(pwd)"/logs:/timeplus/logs if you want to mount the data and log files to your host ## Quickstart with Docker Compose​ To try our demo kit, you can use Docker Compose to start Timeplus Enterprise, together with Redpanda (A Kafka API compatible message bus), ClickHouse and data generator. For Linux or Mac users, please run the command: For Windows users, you can download the package with Docker Compose file and bootstrap scripts here. Access the Timeplus demo kit web console at https://localhost:8000. This stack demonstrates how to run streaming ETL, getting data from Kafka API, applying filter or aggregation, then sending to another Kafka topic or ClickHouse tables. For more details, please see: - Tutorial – Streaming ETL: Kafka to Kafka - Tutorial – Streaming ETL: Kafka to ClickHouse ## Cluster Install​ Timeplus Enterprise can be installed in multi-node cluster mode for high availability and horizontal scalability. This page shares how to install Timeplus cluster on bare metal. Please refer to this guide to deploy Timeplus Enterprise on Kubernetes. ### Multi-Node Cluster​ There are multiple ways to setup a cluster without Kubernetes. One easy solution is to run all components in one node, and the rest of nodes running the timeplusd only. For other deployment options, please contact support or message us in our Slack Community. Choose one node as the lead node, say its hostname is timeplus-server1. Stop all services via timeplus stop command. Then configure environment variables. Then run timeplus start to start all services, including timeplusd, timeplus_web, timeplus_appserver and timeplus_connector. On the second node, first make sure all services are stopped via timeplus stop. Then configure environment variables. Then run timeplus start -s timeplusd to only start timeplusd services. Similarly on the third node, set export ADVERTISED_HOST=timeplus-server3 and the same METADATA_NODE_QUORUM and only start timeplusd. ### Single-Host Cluster​ Starting from Timeplus Enterprise v2.7, you can also easily setup multiple timeplusd processes on the same host by running the timeplusd server with node-index option. This is useful for testing multi-node cluster. Timeplusd will automatically bind to different ports for each node. You can run timeplusd client to connect to one node and check the status of the cluster via: ## License Management​ When you start Timeplus Enterprise and access the web console for the first time, the 30-day free trial starts. When it ends, the software stops working. Please check the guide to update licenses. - Single Node InstallInstall ScriptManual InstallTimeplus Enterprise ProcessesUpgradeUninstall - Install Script - Manual Install - Timeplus Enterprise Processes - Upgrade - Uninstall - Docker Install - Quickstart with Docker Compose - Cluster InstallMulti-Node ClusterSingle-Host Cluster - Multi-Node Cluster - Single-Host Cluster - License Management - Install Script - Manual Install - Timeplus Enterprise Processes - Upgrade - Uninstall - Multi-Node Cluster - Single-Host Cluster ## Category - [Third-party Tools | Timeplus](https://docs.timeplus.com/category/tools): Integrate Timeplus to your tool stacks. - Integrations - Third-party Tools # Third-party Tools Integrate Timeplus to your tool stacks. ## 📄️ Integration with Grafana Grafana has long been one of the most popular tools for real-time monitoring and data visualization, helping organizations track metrics and analyze trends through a single, user-friendly interface. For self-hosted Timeplus Enterprise or Timeplus Proton, you can try the Grafana plugin for Timeplus with the source code at GitHub. This plugin was designed to leverage Grafana’s new Grafana Live capability, allowing users to keep their SQL query results up-to-date without the need to refresh their dashboards. Check out here for sample setup. ## 📄️ Load MySQL/Postgres/CSV/S3 to Timeplus via Sling CLI While Kafka and Debezium provide robust Change Data Capture (CDC) capabilities for streaming data from MySQL or PostgreSQL to other systems, they may introduce unnecessary complexity for historical data migration scenarios. The infrastructure requirements for Kafka clusters and JVM-based deployments can present significant operational overhead. Sling offers a lightweight alternative for data transfer operations to and from Timeplus, utilizing YAML-based configuration files and scheduled execution through cron jobs. ## 📄️ Push data to Timeplus via Kafka Connect Kafka Connect is a framework for connecting Kafka with external systems such as databases, key-value stores, search indexes, and file systems, using so-called Connectors. ## 📄️ DevOps with Flyway Redgate Flyway extends DevOps to databases to accelerate software delivery and ensure quality code. From version control to continuous delivery, Flyway builds on application delivery processes to automate database deployments. - [SQL Statements | Timeplus](https://docs.timeplus.com/category/commands): Overview of the SQL commands supported by Timeplus. - SQL Reference - Statements # SQL Statements Overview of the SQL commands supported by Timeplus. ## 🔗 SELECT ## 📄️ ALTER STREAM You can modify the retention policy for historical store via MODIFY TTL and modify the retention policy for streaming storage via MODIFY SETTING. For mutable streams, you can also run MODIFY SETTING to change the RocksDB settings. ## 📄️ ALTER VIEW You can use this SQL to change a view or a materialized view. Today only the settings can be changed. To change the SQL query behinds the view, you have to drop and re-create it. ## 📄️ CREATE ALERT Starting from Timeplus Enterprise 2.9, you can create alerts to monitor your streaming data and automatically trigger actions when specific conditions are met. ## 📄️ CREATE DATABASE Starting from Timeplus Enterprise 2.6, you can create a database to organize your streams, materialized views and other resources. The database is a logical container to help you to manage and query the data more efficiently. ## 📄️ CREATE DICTIONARY Starting from Timeplus Enterprise 2.7, you can create dictionaries to store key-value pairs in memory, with data from various sources, such as files, MySQL/ClickHouse databases, or streams in Timeplus. ## 📄️ CREATE DISK By default, Timeplus only created a default disk for local storage. ## 📄️ CREATE EXTERNAL STREAM External stream for Kafka is official supported. The external stream for local log files is at technical preview. In Timeplus Enterprise, it also supports another type of External Stream to read/write data for a remote Timeplus Enterprise. ## 📄️ CREATE EXTERNAL TABLE Timeplus supports 3 types of external tables: ClickHouse, MySQL, and S3. Reading data from external tables are bounded, which means the query will end when all the data is read. Writing data to external tables are unbounded, which means the query will keep running until you cancel it. ## 📄️ CREATE FORMAT Timeplus supports reading or writing messages in Protobuf or Avro format. This document covers how to process data without a Schema Registry. Check this page if your Kafka topics are associated with a Schema Registry. ## 📄️ CREATE FUNCTION Timeplus supports four ways to develop/register UDF. Please check UDF page for the overview. ## 📄️ CREATE MATERIALIZED VIEW The difference between a materialized view and a regular view is that the materialized view is running in background after creation and the resulting stream is physically written to internal storage (hence it's called materialized). ## 📄️ CREATE MUTABLE STREAM Regular streams in Timeplus are immutable, and stored in columnar format. Mutable streams are stored in row format (implemented via RocksDB), and can be updated or deleted. Please check this page for details. ## 📄️ CREATE RANDOM STREAM You may use this special stream to generate random data for tests. For example: ## 📄️ CREATE REMOTE FUNCTION At Timeplus, we leverage SQL to make powerful streaming analytics more accessible to a broad range of users. Without SQL, you have to learn and call low-level programming API, then compile/package/deploy them to get analytics results. This is a repetitive and tedious process, even for small changes. ## 📄️ CREATE STREAM Stream is a key concept in Timeplus. All data lives in streams, no matter static data or data in motion. We don't recommend you to create or manage TABLE in Timeplus. ## 📄️ CREATE VIEW There are two types of views in Timeplus: logical view (or just view ) and materialized view. ## 📄️ DELETE Starting from Timeplus Enterprise 2.7, you can delete data from the streams. ## 📄️ DROP ALERT Starting from Timeplus Enterprise 2.9, you can create alerts to monitor your streaming data and automatically trigger actions when specific conditions are met. ## 📄️ DROP DATABASE Starting from Timeplus Enterprise 2.6, you can create a database to organize your streams, materialized views and other resources. The database is a logical container to help you to manage and query the data more efficiently. ## 📄️ DROP DICTIONARY Starting from Timeplus Enterprise 2.7, you can create dictionaries to store key-value pairs in memory, with data from various sources, such as files, MySQL/ClickHouse databases, or streams in Timeplus. ## 📄️ DROP EXTERNAL TABLE Run the following SQL to drop an external table from Timeplus, while keeping the data in external database intact. ## 📄️ DROP FORMAT SCHEMA ## 📄️ DROP FUNCTION No matter UDF or UDAF, you can remove the function via DROP FUNCTION ## 📄️ DROP STREAM Run the following SQL to drop a stream or an external stream, with all data in streaming storage and historical storage. ## 📄️ DROP VIEW Run the following SQL to drop a view or a materialized view. ## 📄️ EXPLAIN The execution of a query in Timeplus is decomposed into many steps. Each step of the query execution can be analyzed and troubleshooted using the corresponding EXPLAIN query. These steps are summarized in the chart below: ## 📄️ OPTIMIZE Run the following SQL to initiate an unscheduled merge of a mutable stream or a table with MergeTree engine. This will reorganize the data in the stream and optimize the performance of the stream. Note that we generally recommend against using OPTIMIZE STREAM ... FINAL as its use case is meant for administration, not for daily operations. ## 📄️ RENAME STREAM You can rename a stream using the RENAME STREAM statement. ## 📄️ SHOW ALERTS Starting from Timeplus Enterprise 2.9, you can create alerts to monitor your streaming data and automatically trigger actions when specific conditions are met. ## 📄️ SHOW CREATE Show Details For A Stream ## 📄️ SHOW DATABASES Starting from Timeplus Enterprise 2.6, you can create a database to organize your streams, materialized views and other resources. The database is a logical container to help you to manage and query the data more efficiently. ## 📄️ SHOW DICTIONARIES Starting from Timeplus Enterprise 2.7, you can create dictionaries to store key-value pairs in memory, with data from various sources, such as files, MySQL/ClickHouse databases, or streams in Timeplus. ## 📄️ SHOW DISKS Starting from Timeplus Enterprise 2.8, you can create S3 disks for tiered storage or autoscaling materialized views. ## 📄️ SHOW FORMAT SCHEMAS List schemas in the current Timeplus deployment: ## 📄️ SHOW FUNCTIONS List available User-Defined Functions. ## 📄️ SHOW STREAMS List Streams From Current Database ## 📄️ SYSTEM PAUSE SYSTEM PAUSE MATERIALIZED VIEW ## 📄️ SYSTEM RECOVER This is an advanced feature, for debugging purpose. ## 📄️ SYSTEM RESUME SYSTEM RESUME MATERIALIZED VIEW ## 📄️ SYSTEM SET LOG LEVEL ## 📄️ SYSTEM TRANSFER LEADER Transfer the leader of a materialized view to another node in the cluster. ## 📄️ TRUNCATE STREAM Run the following SQL to remove the historical data from a stream, keeping the schema and the stream itself. ## 📄️ USE Starting from Timeplus Enterprise 2.6, you can create a database to organize Timeplus streams, materialized views, or connect to other systems such as MySQL and Iceberg without creating multiple external tables repeating the connection information. - [Regular Functions | Timeplus](https://docs.timeplus.com/category/functions): Integrate Timeplus to your tool stacks. - SQL Reference - Functions - Regular Functions # Regular Functions Integrate Timeplus to your tool stacks. ## 📄️ Type Conversion to_time ## 📄️ Arrays, Maps, Tuples Arrays ## 📄️ Process Date and Time year ## 📄️ Process URL and IP protocol ## 📄️ Process JSON Various functions are provided to extract values from JSON documents. You can also use the shortcut :true,"b"1}}. Then you can convert it to other data types using to_int() or ::int shortcut. ## 📄️ Process Text lower ## 📄️ Hash Hash functions can be used for the deterministic pseudo-random shuffling of elements. ## 📄️ Random Data Generation You can use the following functions to generate random data in Timeplus. ## 📄️ Logic if ## 📄️ Math abs ## 📄️ Financial xirr ## 📄️ Geo Location pointinpolygon ## 📄️ Dictionaries Functions for working with dictionaries. ## Changelog-stream - [Changelog Key Value Stream | Timeplus](https://docs.timeplus.com/changelog-stream): When you create a stream with the mode changelog_kv, the data in the stream is no longer append-only. When you query the stream directly, only the latest version for the same primary key(s) will be shown. Data can be updated or deleted. You can use Changelog Stream in JOIN either on the left or on the right. Timeplus will automatically choose the latest version. - Core Features - Streams - Changelog Key Value Stream # Changelog Key Value Stream When you create a stream with the mode changelog_kv, the data in the stream is no longer append-only. When you query the stream directly, only the latest version for the same primary key(s) will be shown. Data can be updated or deleted. You can use Changelog Stream in JOIN either on the left or on the right. Timeplus will automatically choose the latest version. Here are some examples: ## Create the Stream​ In this example, you create a stream dim_products in changelog_kv mode with the following columns: The rest of this page assumes you are using Timeplus Console. If you are using Proton, you can create the stream with DDL. ## Query Single Stream​ If you don't add any data, query SELECT * FROM dim_products will return no results and keep waiting for the new results. ### Add data​ Keep the query running and add a few more rows to the stream (via REST API or create a new browser tab and add rows to the streams directly). The query console will show those 2 rows automatically. ### Delete data​ Somehow, you don't want to list iPhone14_Plus any more. All you need is to add a row with _tp_delta=-1: Then cancel the query and run it again, you will only get 1 row, not 3 rows. The reason for that is the 2nd row and 3rd row are with the same primary id but with opposite _tp_delta, so Timeplus merges them. This process is called "compaction". ### Update data​ Now if you want to change the price of iPhone14, you need to add two rows: Cancel the query SELECT * FROM dim_products and run again, you will get only 1 row in the product list: As you can imagine, you can keep adding new rows. If _tp_delta is 1 and the primary key is new, then you will get a new row in the query result. If _tp_delta is -1 and the primary key exists already, then the previous row is deleted. You can update the value by adding a new row with the primary key. In fact, you can assign an expression as the primary key. For example you can use first_name||' '||last_name to use the combined full name as the primary key, instead of using a single column. ### Show aggregated results​ If you run queries like select count(1), sum(price) from dim_products , this streaming SQL will always give you latest results: ## Use Changelog Stream in JOIN as lookup​ In the above examples, you always get the latest version of the event with the same primary key. This is very useful when such streams acts as the "lookup-table" for the JOIN. Imagine you have an append-only stream for the orders: The current dim_products stream is: Now start a streaming SQL: Then add 2 rows: In the query console, you will see 2 rows one by one: Then you can change the price of iPhone14 to 800, by adding 2 new rows in dim_products Also add a new row in orders You will get the 3rd row in the previous streaming SQL: It shows that the latest price of iPhone14 is applied to the JOIN of new event. ## Use Changelog Stream in JOIN as left table​ You can also use Changelog Stream on the left side of the JOIN. Let's create a new stream order2 in Changelog Stream mode: You can add/update/delete orders by adding the rows with proper _tp_delta value. When you run streaming SQL like: Timeplus will use the latest version of order records to join with the lookup table. What's more useful is that if you run aggregations such as: you will get the correct numbers whenever the order is added, updated, or deleted. ## Setup CDC with Changelog Stream​ CDC, or Change Data Capture, is a critical part of Modern Data Stack. Most of the modern databases support CDC to sync the data changes to other systems at real-time. One popular open-source solution is Debezium. Timeplus Changelog Stream can work with Debezium or other CDC solutions. It can also work without them, if your applications can generate events with the proper _tp_delta flag (1 for adding data, -1 for deleting data). For example, you created 2 tables in PostgreSQL 14: ### Setup PostgreSQL Tables​ In order to capture the before values for UPDATE or DELETE, you also need to set REPLICA IDENTIFY to FULL. Check the Debezium documentation for more details. ### Setup Debezium​ Now start Debezium with Kafka Connect in your preferred way. You also need a local or remote Kafka/Redpanda as message brokers to receive CDC data. In this example, we are going to use Redpanda Cloud as message broker, and a local docker image for the Kafka Connect with built-in Debezium. The docker-compose file: You can put the credentials in a .env file in the same folder: Optionally, you can add Redpanda Console in the docker compose file. It provides a nice UI for you to add/review topics and manage connectors. ### Add PostgreSQL Connector​ Start the docker-compose with docker-compose up. It will load the images for Debezium and Redpanda Console. You can access the Redpanda Console via http://localhost:8080 and the Kafka Connect REST API endpoint is http://localhost:8083 You may add the Debezium connector via the web UI. But the following command line just works fine: This will add a new connector with the name pg-connector, connecting to a PostgreSQL database in a remote server (in this case, Aiven Cloud). A few notes to the configuration items: - publication.autocreate.mode is filtered, and table.include.list is a list of tables you want to apply CDC. Sometimes you don't have permissions to enable publications for all tables. So filtered is recommended. - plugin.name is pgoutput. This works well with new versions of PostgreSQL (10+). The default value is decoderbufs, which is required to be installed separately. - topic.prefix is set to doc. As the name implies, it will be the prefix for Kafka topics. Since the schema is public, the topics to be used will be doc.public.dim_products and doc.public.orders Make sure you create those 2 topics in Kafka/Redpanda. Then in a few seconds, new messages should be available in the topics. You can try INSERT/UPDATE/DELETE data in PostgreSQL and check the generated JSON messages. ### Sample CDC Data​ #### INSERT​ SQL: CDC Data: #### UPDATE​ SQL: CDC Data: #### DELETE​ SQL: CDC Data: #### For existing rows​ Debezium also read all existing rows and generate messages like this ### Load data to Timeplus​ You can follow this guide to add 2 external streams to load data from Kafka or Redpanda. For example: - Data source name s1 to load data from topic doc.public.dim_products and put in a new stream rawcdc_dim_products - Data source name s2 to load data from topic doc.public.orders and put in a new stream rawcdc_orders Then run the following SQL in query page: This SQL query may look a bit overwhelming. At the top level, there are 3 subqueries to union the result: - one subquery is to check if the op is either c or r. It will insert a new row to the changelog stream with _tp_detla=1 - one subquery is to check if the op is d. It will delete the row in the changelog stream with _tp_detla=-1 - the last subquery is to check if the op is u. It will send a row with _tp_detla=-1 first, followed by a _tp_delta=1. Since union operation in SQL won't guarantee the order of emit, so we created an array for [-1,1] and use array_join to turn 1 row to 2 rows, with expected order. Click the Send as Sink button and choose Timeplus type, to send the results to an existing stream dim_products . - Create the Stream - Query Single StreamAdd dataDelete dataUpdate dataShow aggregated results - Add data - Delete data - Update data - Show aggregated results - Use Changelog Stream in JOIN as lookup - Use Changelog Stream in JOIN as left table - Setup CDC with Changelog StreamSetup PostgreSQL TablesSetup DebeziumAdd PostgreSQL ConnectorSample CDC DataINSERTUPDATEDELETEFor existing rowsLoad data to Timeplus - Setup PostgreSQL Tables - Setup Debezium - Add PostgreSQL Connector - Sample CDC DataINSERTUPDATEDELETEFor existing rows - INSERT - UPDATE - DELETE - For existing rows - Load data to Timeplus - Add data - Delete data - Update data - Show aggregated results - Setup PostgreSQL Tables - Setup Debezium - Add PostgreSQL Connector - Sample CDC DataINSERTUPDATEDELETEFor existing rows - INSERT - UPDATE - DELETE - For existing rows - Load data to Timeplus - INSERT - UPDATE - DELETE - For existing rows ## Checkpoint-settings - [Checkpoint Settings | Timeplus](https://docs.timeplus.com/checkpoint-settings): For materialized views, checkpoint settings control how and where checkpoints are created, as well as whether they are incremental or asynchronous. This can be specified in the checkpoint_settings section of the materialized view settings. - Core Features - Materialized Views - Checkpoint Settings # Checkpoint Settings For materialized views, checkpoint settings control how and where checkpoints are created, as well as whether they are incremental or asynchronous. This can be specified in the checkpoint_settings section of the materialized view settings. ## checkpoint_settings​ You can set key-value pairs in checkpoint_settings. ### type​ Definition: Defines which checkpoint type to use. Possible Values: - auto (default) - Automatically determine whether to use file or rocks checkpoint based on the query’s state type. - file - Native file format. You can explicitly use the local file system for the checkpoint storage, even for some materialized views, using rocksdb is recommended. ### replication_type Definition: Specifies where checkpoints will be stored. Possible Values: - auto (default) - Automatically determine whether to store in local_file_system or nativelog - local_file_system - Stored in local file system for a single instance environment - nativelog - Stored in nativelog, and ensure cluster synchronization through raft (Only valid in clusters) - shared_disk - Shared storage like S3, it must be bound ### async​ Definition: Determines whether checkpoints are created asynchronously. Possible Values: - true (default) - Asynchronous checkpoint replication - false ### incremental​ Definition: Indicates whether checkpoints are incremental (saving only changes since the last checkpoint) or full. Possible Values: - false (default) - true - Only enabled when using a hybrid hash table (Recommended for large states with low update frequency) ### interval​ Definition: Specifies the time interval in seconds between checkpoint operations. Possible Values: - Any integer (default 0 means use a configured value) This is also configurable via the global configuration file. ### shared_disk Definition: Specifies a disk name, which can be created through sqlcreate disk {disk_name} ..., which is used with a shared checkpoint storage (i.e. S3) ## checkpoint_interval​ In some cases, you may want to adjust the checkpoint interval after the materialized view is created. You can do this by modifying the checkpoint_settings parameter in the ALTER VIEW statement. If you don't want to repeat the other key-value pairs in the checkpoint_settings, you can use the top-level checkpoint_interval parameter, e.g. Set checkpoint_interval to a negative integer value to disable checkpointing. Set it to 0 to use the global checkpoint interval. Set it to a positive integer value to specify the checkpoint interval in seconds. ## Examples​ For some scenarios with large states and low update frequency: For some scenarios with S3 checkpoint storage: - checkpoint_settingstypereplication_typeasyncincrementalintervalshared_disk - type - replication_type - async - incremental - interval - shared_disk - checkpoint_interval - Examples - type - replication_type - async - incremental - interval - shared_disk ## Cli-backup - [timeplus backup | Timeplus](https://docs.timeplus.com/cli-backup): Create a Timeplus Enterprise backup file. - Integrations - CLI, APIs & SDKs - timeplus (CLI) - timeplus backup # timeplus backup Create a Timeplus Enterprise backup file. ## timeplus backup​ Open a terminal window and change directory to the timeplus/bin folder, and run timeplus backup without extra parameters, it will back up all data and configuration files and create a timeplus.bak in the bin folder. You can also customize the file name with -f or --backup-file flag. If your working directory is not the bin folder, you can also specify the Timeplus Enterprise base folder via -d or --timeplus-home flag. ## See Also​ timeplus restore - timeplus backup - See Also ## Cli-diag - [timeplus diag | Timeplus](https://docs.timeplus.com/cli-diag): Run diagnostics of Timeplus Enterprise services. - Integrations - CLI, APIs & SDKs - timeplus (CLI) - timeplus diag # timeplus diag Run diagnostics of Timeplus Enterprise services. This command need be ran when the Timeplus Enterprise services are running or stopped. It will generate a file in the "logs" folder with ".diag" as the file extension and "timeplus_" as the prefix, followed by the datetime. When you contact Timeplus support team, please run this command and compress the entire logs folder with current logs and the diag file, and send the compressed file to support team. This will help the support engineers better understand how Timeplus Enterprise is deployed. ## diag file​ The diag file, such as "timeplus/logs/timeplus_20240712100051.diag", is a JSON document without sensitive customer data. The key content of the diag file is: - versions for Timeplus Enterprise services. - cpu/memory/os information for the node. - how the Timeplus Enterprise cluster is setup. - schema for Timeplus streams, views, or external tables. No sample data will be captured in the diag file. - license information. ## system.stream_state_log​ Starting from Timeplus Enterprise 2.6, the system.stream_state_log table is exported as a CSV file in the logs folder. This file contains the state changes of streams, views, and external tables. The file is named as "timeplus/logs/timeplus_system_stream_state_log_timestamp.csv". - diag file - system.stream_state_log ## Cli-help - [timeplus help | Timeplus](https://docs.timeplus.com/cli-help): timeplus help - Integrations - CLI, APIs & SDKs - timeplus (CLI) - timeplus help # timeplus help ## timeplus help​ When you run timeplus help without extra parameters, it will list all available commands, e.g. ## timeplus help [command]​ You can add the command name after help to see the available flags, e.g. - timeplus help - timeplus help [command] ## Cli-license - [timeplus license | Timeplus](https://docs.timeplus.com/cli-license): Manage Timeplus Enterprise licenses via CLI. - Integrations - CLI, APIs & SDKs - timeplus (CLI) - timeplus license # timeplus license Manage Timeplus Enterprise licenses via CLI. You can also manage licenses via the web console (workspace settings -> license -> add new license). ## timeplus license show​ Show the current license. For a newly-installed Timeplus Enterprise, the output can be For a paid license, the output can be ## timeplus license import -f [file] -k [key]​ Once you get the license key and file from Timeplus team, you can use web console or the CLI to import the license. For example: - timeplus license show - timeplus license import -f [file] -k [key] ## Cli-migrate - [timeplus migrate | Timeplus](https://docs.timeplus.com/cli-migrate): Migrate data and resources between Timeplus deployments, including: - Integrations - CLI, APIs & SDKs - timeplus (CLI) - timeplus migrate # timeplus migrate Migrate data and resources between Timeplus deployments, including: - Migrate the data and configuration from a single-node Timeplus Proton to self-hosted Timeplus Enterprise. - Migrate the data and configuration among Timeplus Enterprise deployments, even there are breaking changes in the data format. This tool can also be used to apply changes to production deployments after verifying changes in the staging deployments. ## How It Works​ The migration is done via capturing the SQL DDL from the source deployment and rerunning those SQL DDL in the target deployment. Data are read from source Timeplus via Timeplus External Streams and write to the target Timeplus via INSERT INTO .. SELECT .. FROM table(tp_ext_stream). The data files won't be copied among the source and target Timeplus, but you need to ensure the target Timeplus can access to the source Timeplus, so that it can read data via Timeplus External Streams. ## Supported Resources​ The migrate command will migrate the following resources: - streams, external streams, external tables - views, materialized views - UDFs, UDAFs, schemas - sinks, sources, dashboards, saved queries(a.k.a. bookmarks) The following resources are not supported yet: - alerts - users, roles, grants - databases - API keys Query history in the SQL Console could be exported but won't be migrated to the target Timeplus. ## Preparation​ To best ensure the data integrity, it's recommended to stop sending requests to ingestion REST API, stop running INSERT SQL. If in your source deployment, there are materialized views reading data from external Kafka or Pulsar system, you can pause them. Expose the 8463 TCP port from the source deployment. Making sure the target deployment can access to the port. ## Migrate Workflow​ 1. Create a configuration YAML file to specify the connections to the source and target deployments. 2. Run timeplus migrate plan command with this configuration file. This will capture the DDL and other configurations in your source deployment. A migration plan will be generated. 3. (Optional), you can review and modify the migration plan file. For example, you can remove some of the streams or views which you don't want to create in the target deployment 4. Run timeplus migrate run command to create the streams and other resources in the target deployment. Data will be replicated via Timeplus External Streams. Create a configuration YAML file to specify the connections to the source and target deployments. Run timeplus migrate plan command with this configuration file. This will capture the DDL and other configurations in your source deployment. A migration plan will be generated. (Optional), you can review and modify the migration plan file. For example, you can remove some of the streams or views which you don't want to create in the target deployment Run timeplus migrate run command to create the streams and other resources in the target deployment. Data will be replicated via Timeplus External Streams. ## timeplus migrate plan -f [cfg_file] -p [plan_file]​ Run timeplus migrate plan -f config-file-path -p plan_file_path to generate the migration plan. Here is a sample configuration file: Settings in the configuration file: - source. The connection to the source Timeplus. - target. The connection to the target Timeplus. - mode. By default the generated file includes DDL to create resources in the target Timeplus. Set mode to resource_only to only dump the resource definitions in the source Timeplus. - show_query_history. Default to false. Set it to true if you want to include the SQL query history in the generated plan file. But those query history won't be migrated to the target Timeplus. - stop_data_ingestion_on_source. Default to false. Set it to true to stop all sources on source Timeplus. - stop_data_ingestion_on_target. Default to false. Set it to true to stop all sources on target Timeplus. - max_insert_threads Default to 8. The number of insert threads. - drop_migration_external_stream Default false. Timeplus External Streams will be created to migrate data. Set to true if you want to delete them after migration. - max_block_size Default to 10. For some streams with large size records, you can set a proper value to control how many rows to be inserted in a batch. - use_timeplusd_for_data_migration Default to false. Set it to true to use timeplusd TCP port (instead of timeplus_appserver HTTP port) to run the migration SQL. This is recommended for Timeplus deployments with large volume of historical data. ## timeplus migrate run -p [plan_file]​ You can review the generated plan file and make changes if necessary. After the review, you can run this command to start the migration. ## timeplus migrate clean -p [plan_file]​ If there is anything wrong during the migration and you want to try it again, this clean sub-command will delete the migrated resource in target Timeplus, so that you can recreate them in the next timeplus migrate run. ## timeplus migrate kv --host [host] -p [password]​ Since Timeplus Enterprise v2.8, the application metadata is stored in mutable streams, instead of a special key-value subsystem as part of timeplusd. If you previously deployed Timeplus Enterprise v2.7 or earlier, and want to keep the metadata after upgrading to v2.8 or later, you need to run the following migration tool: If you don't run the migration script, the previously saved query bookmarks, dashboards or alerts won't be visible in the new deployment. ## Limitations​ 1. There is no checkpoint or error recovering for now. While migrating streams with lots of data, it is possible to experience network interruption. In this case, you can turn on use_timeplusd_for_data_migration setting and use timeplusd instead of the application server to run the migration script. You can also run migrate clean command to clean up the half-done resources and retry. 2. The owner for the newly created resources in the target Timeplus are the API user in configuration file, not necessarily the original owner in source Timeplus. - How It Works - Supported Resources - Preparation - Migrate Workflow - timeplus migrate plan -f [cfg_file] -p [plan_file] - timeplus migrate run -p [plan_file] - timeplus migrate clean -p [plan_file] - timeplus migrate kv --host [host] -p [password] - Limitations ## Cli-reference - [CLI Reference | Timeplus](https://docs.timeplus.com/cli-reference): Besides timeplusd-client, Timeplus also provide the timeplus command line interface to help you manage the self-hosted deployments. - Integrations - CLI, APIs & SDKs - timeplus (CLI) # CLI Reference Besides timeplusd-client, Timeplus also provide the timeplus command line interface to help you manage the self-hosted deployments. For bare metal installation, the timeplus binary is available in /bin folder. For Kubernetes deployments, please add the following section to values.yaml : Then upgrade the helm chart via: Once timeplus-cli pod is up and running, you can run kubectl exec -n $NS -it timeplus-cli -- /bin/bash to run commands in the pod. ## Commands​ The following table displays the available top-level timeplus commands. ## Common Flags​ Most of the commands support the following flags: - -v, or --verbose: run the command in the verbose mode - -h, or --help: show the help message for the command, with sub-commands and flags. - Commands - Common Flags ## Cli-restart - [timeplus restart | Timeplus](https://docs.timeplus.com/cli-restart): Restart the Timeplus Enterprise services in the current node. - Integrations - CLI, APIs & SDKs - timeplus (CLI) - timeplus restart # timeplus restart Restart the Timeplus Enterprise services in the current node. ## timeplus restart​ When you run timeplus restart without extra parameters, it will restart all services, e.g. ## timeplus restart -s [service]​ You can also use the -s or --service flag to restart a specific service, e.g. ## See Also​ timeplus start timeplus stop timeplus status - timeplus restart - timeplus restart -s [service] - See Also ## Cli-restore - [timeplus restore | Timeplus](https://docs.timeplus.com/cli-restore): Restore from a Timeplus Enterprise backup file. - Integrations - CLI, APIs & SDKs - timeplus (CLI) - timeplus restore # timeplus restore Restore from a Timeplus Enterprise backup file. ## timeplus restore -f [file] -d [home]​ This will load the backup file and restores to the Timeplus Enterprise base folder. It is recommended to stop the Timeplus Enterprise services, restore the data and configuration, then start the service. ## See Also​ timeplus backup timeplus stop timeplus start - timeplus restore -f [file] -d [home] - See Also ## Cli-service - [timeplus service | Timeplus](https://docs.timeplus.com/cli-service): Add Timeplus Enterprise services to Linux systemd control. This feature is not available for macOS. - Integrations - CLI, APIs & SDKs - timeplus (CLI) - timeplus service # timeplus service Add Timeplus Enterprise services to Linux systemd control. This feature is not available for macOS. ## timeplus service enable​ This command needs to be ran by the root user. Please get the current user and user group via command lines. Then use sudo to enable the service with the right user and group. Then you can use systemctl to manage the service, e.g. - sudo systemctl status timeplus.service - sudo systemctl start timeplus.service - sudo systemctl stop timeplus.service - sudo systemctl restart timeplus.service To remove the service, run this command: - timeplus service enable ## Cli-start - [timeplus start | Timeplus](https://docs.timeplus.com/cli-start): Start the Timeplus Enterprise services in the current node. - Integrations - CLI, APIs & SDKs - timeplus (CLI) - timeplus start # timeplus start Start the Timeplus Enterprise services in the current node. ## timeplus start​ When you run timeplus start without extra parameters, it will start all services, e.g. ## timeplus start -s [service]​ You can also use the -s or --service flag to start a specific service, e.g. ## timeplus start --start-trial​ By default, when you start Timeplus Enterprise for the first time, you need to login to the Web Console to finish the user onboarding and create an account. You can also automate this by adding --start-trial in the timeplus start command. Starting from Timeplus Enterprise 2.6, you can disable the anonymous usage data collection by adding --telemetry false in the timeplus start command. ## Note for macOS users​ If you are using macOS and fail to start Timeplus Enterprise, please check the "Privacy & Security" settings and see whether there is any warning for "timeplusd". If so, choose "Allow Anyway" to trust timeplusd and start Timeplus Enterprise again. ## See Also​ timeplus stop timeplus restart timeplus status - timeplus start - timeplus start -s [service] - timeplus start --start-trial - Note for macOS users - See Also ## Cli-status - [timeplus status | Timeplus](https://docs.timeplus.com/cli-status): Show the status for Timeplus Enterprise services. - Integrations - CLI, APIs & SDKs - timeplus (CLI) - timeplus status # timeplus status Show the status for Timeplus Enterprise services. ## timeplus status​ When you run timeplus status without extra parameters, it will list the status for all services, e.g. Or: The "status" can be either "running" or "stopped", by checking the pid (process id). The "healthy" value is determined by checking the HTTP endpoint of each service. For stopped service, the "healthy" value ought to be "false". ## timeplus status -v​ -v or --verbose flag can show extra information, such as the pid (process id). ## See Also​ timeplus start timeplus stop timeplus restart - timeplus status - timeplus status -v - See Also ## Cli-stop - [timeplus stop | Timeplus](https://docs.timeplus.com/cli-stop): Stop the Timeplus Enterprise services in the current node. - Integrations - CLI, APIs & SDKs - timeplus (CLI) - timeplus stop # timeplus stop Stop the Timeplus Enterprise services in the current node. ## timeplus stop​ When you run timeplus stop without extra parameters, it will stop all services, e.g. Usually all services will be stopped immediately, except timeplusd, which may take a few seconds or minutes, depending on the server workload. ## timeplus stop -s [service]​ You can also use the -s or --service flag to stop a specific service, e.g. ## See Also​ timeplus start timeplus restart timeplus status - timeplus stop - timeplus stop -s [service] - See Also ## Cli-sync - [timeplus sync | Timeplus](https://docs.timeplus.com/cli-sync): Synchronizes resources to Timeplus Enterprise. - Integrations - CLI, APIs & SDKs - timeplus (CLI) - timeplus sync # timeplus sync Synchronizes resources to Timeplus Enterprise. It works like a database migration tool except that it does not store the migration states, instead, users are responsible for deciding which migration scripts to run. Features: - Runs migration scripts in dictionary order (by script file names). - Supports Go template. All migration scripts will be rendered as Go templates. - Function for extracting UDF/A Javascript from Javascript files. ## timeplus sync apply [file_or_folder]​ For example: This command will run the path/to/your/migration-script.sql SQL script (the file MUST use .sql as its extension name) for the current running Timeplus server. Sample output: You can also specify a folder, for example: In this case, all migration scripts (files with .sql extension name) will be applied one by one in the A-to-Z dictionary order. All other files will be ignored. For example, given that migration-scripts folder contains the following files: The scripts will be executed in the following order: 1. 01.sql 2. 02.sql 3. 03.sql 4. 04.sql udf_function.js and README.md are skipped because they do not have .sql extension. If a directory has sub-directories, migration scripts in the sub-directories will be executed too, but they will be executed after the scripts in the parent directory. It's also possible to specify multiple paths to the apply command. For example: ### Templates and Variables​ All migration scripts are treated as Go template. This command uses the sprig library, thus all functions provided by sprig are all available to use. Also, a special data .values is defined to allow users to define their own values to be used in the migrations scripts. For example, given a migration script contains the following SQL statement: The statement does not hard-code the Kafka brokers and topic, so that it can be easily shared in different environments. There are several ways to specify the values. #### Set variable value in the command​ Use command line argument --set. You can set multiple values with comma as the separator, e.g. Or, specify the values in their own --set: #### Set variable values in a file​ Create a file called values.yaml (or any name you like to use), and write the following content in it: Or with hierarchy, e.g. Then you can run the command: ## timeplus sync apply --dry-run​ Dry-run mode is supported. It prints what SQL statements will be executed without actually executing them. For example the SQL file is: - timeplus sync apply [file_or_folder]Templates and VariablesSet variable value in the commandSet variable values in a file - Templates and VariablesSet variable value in the commandSet variable values in a file - Set variable value in the command - Set variable values in a file - timeplus sync apply --dry-run - Templates and VariablesSet variable value in the commandSet variable values in a file - Set variable value in the command - Set variable values in a file - Set variable value in the command - Set variable values in a file ## Cli-user - Integrations - CLI, APIs & SDKs - timeplus (CLI) - timeplus user ## timeplus user​ When you run timeplus user without extra parameters, it will list all available sub-commmands, e.g. ## timeplus user list​ You need to add --verbose to this command to get the user list. This is fixed in Timeplus Enterprise 2.4.16 This will list all users in the cluster. Sample output: For each new deployment, by default the default and proton users will be provisioned and you can create more users via the web console or CLI. ## timeplus user update --user [user] --password [password]​ Update the password for the specified user. ## timeplus user update --user [user] --role [role]​ Change the role for the specified user. The valid value of role can be either admin or read_only. ## timeplus user create --user [user] --password [password]​ Create a new user with the specified password. By default, the admin role will be assigned. To create a user with read-only permission, add --role read_only to the end. ## timeplus user delete --user [user]​ Delete the specified user. - timeplus user - timeplus user list - timeplus user update --user [user] --password [password] - timeplus user update --user [user] --role [role] - timeplus user create --user [user] --password [password] - timeplus user delete --user [user] ## Cli-version - [timeplus version | Timeplus](https://docs.timeplus.com/cli-version): Show Timeplus Enterprise version and component versions. - Integrations - CLI, APIs & SDKs - timeplus (CLI) - timeplus version # timeplus version Show Timeplus Enterprise version and component versions. ## timeplus version​ This will show the overall version for the Timeplus Enterprise release. Sample output: ## timeplus version -v​ This will run the command in the verbose mode and show the component versions and JSON output of each service. - timeplus version - timeplus version -v ## Compare - [Timeplus Proton vs. Timeplus Enterprise | Timeplus](https://docs.timeplus.com/compare): Timeplus Proton powers unified streaming and data processing on a single database node. Its commercial counterpart, Timeplus Enterprise, supports advanced deployment strategy and includes enterprise-ready features. - Timeplus Proton (OSS) - vs. Timeplus Enterprise # Timeplus Proton vs. Timeplus Enterprise Timeplus Proton powers unified streaming and data processing on a single database node. Its commercial counterpart, Timeplus Enterprise, supports advanced deployment strategy and includes enterprise-ready features. - Single-node Docker image - Single binary on Mac/Linux - Single node - Multi-node Cluster for high availability and horizontal scalability, with data replication and distributed query processing - Kubernetes Helm Chart - Streaming SQL - Historical Data Processing - Append Stream, Random streams, Mutable Stream v1 (Versioned Stream) - Streaming transformation, join, aggregation, tumble/hop/session windows - User-Defined Function: JavaScript, Remote, SQL - Everything in Timeplus Proton - Auto-scaling Materialized View - Mutable Stream v2 with row-based storage for 3x performance and efficiency, also support coalesced and better high cardinality data mutations - Support more EMIT strategies and spill-to-disk capabilities when memory is scarce - Python UDF - Dynamic Dictionary based on MySQL/Postgres or Mutable streams - Tiered Storage using S3 or HDD - Just-In-Time compilation - External streams to Apache Kafka, Confluent Cloud, Redpanda, Apache Pulsar, and Remote Timeplus - Streaming ingestion via REST API (compact mode only) - External table to ClickHouse - Everything in Timeplus Proton - External streams to HTTP API, External tables to MySQL, PostgreSQL, MongoDB, Amazon S3, Apache Iceberg - Hundreds of connectors from Redpanda Connect, e.g. WebSocket, HTTP Stream, NATS - CSV upload - Streaming ingestion via REST API (with API key and flexible modes) - RBAC, Pipeline Wizard, SQL Exploration, Data Lineage, Cluster Monitoring, Troubleshooting and Manageability - Community support from GitHub and Slack - Enterprise support via email, Slack, and Zoom, with a SLA These details are subject to change, but we'll do our best to make sure they accurately represent the latest roadmaps for Timeplus Proton and Timeplus Enterprise. Contact us for more details or schedule a demo. ## Credits - [Credits | Timeplus](https://docs.timeplus.com/credits): Parts of the Timeplus product are powered by open source projects. # Credits Parts of the Timeplus product are powered by open source projects. - Core engine (Proton, Apache License 2.0) uses code from ClickHouse (Apache License 2.0) for SQL functions and the historical storage. - Sources and sinks are powered by Redpanda Connect (Apache License 2.0) and Benthos Framework (MIT License). - JavaScript UDF is powered by V8 (BSD License) - Other third-party libraries introduced by ClickHouse, which can be obtained by the following query: ## Apache License 2.0​ ## Benthos MIT License​ ## V8 BSD-style License​ - Apache License 2.0 - Benthos MIT License - V8 BSD-style License ## Datatypes - [Data Types | Timeplus](https://docs.timeplus.com/datatypes): Like many analytics systems, the following common types are supported. - SQL Reference - Data Types # Data Types Like many analytics systems, the following common types are supported. ## Destination - [Sending Data Out | Timeplus](https://docs.timeplus.com/destination): With Timeplus Console, you can easily explore and analyze streaming data, with intuitive UI, standard SQL and streaming charts. But you won't stop here. Timeplus enables you to setup real-time data pipelines to send data to other systems, or notify individuals or power up downstream applications. - Core Features - Sending Data Out # Sending Data Out With Timeplus Console, you can easily explore and analyze streaming data, with intuitive UI, standard SQL and streaming charts. But you won't stop here. Timeplus enables you to setup real-time data pipelines to send data to other systems, or notify individuals or power up downstream applications. ## Overview​ Timeplus supports various systems as the downstreams: - Send data to Kafka topics - Send data to Pulsar topics - Send data to ClickHouse tables - Send data to another Timeplus deployment - Send data to Webhook endpoints - Notify others via Slack - Send data to other systems via Redpanda Connect ## Send data to Kafka​ You can leverage Timeplus for various streaming analysis, such as - Downsample the data from iot devices and get min/max/avg values every 5 second - Identify any outlier based on the past pattern - transform the data by removing sensitive information, remove duplication, or apply lookup with dimension tables The transformed data or outlier events can be sent to a Kafka topic for other systems to further process. To send data to Kafka, you can submit a streaming SQL in SQL Console, and click the "Save As" button and choose "Sink". Then choose "Apache Kafka". The following parameters are required: - Kafka broker(s) URL - Topic name: either an existing topic or specify the new topic name for Timeplus to create. - Authentication Please refer to the this page for details of the parameters. You can send data to Confluent Cloud, Confluent Platform, or custom managed Apache Kafka. ## Trigger actions via Webhook​ You can also add automation to trigger other systems to take actions when Timeplus finds any real-time insights. Simply choose the Webhook as the action type and optionally set a message body (by default, the entire row will be encoded as a JSON document and sent to the webhook). You can use this approach to perform rule-based automation without human interaction, such as swapping an overheated equipment, scaling up to scaling down the server farm, or reminder users on slack, etc. Please check this blog for real-world examples. ## Notify others via Slack​ After you start running a streaming query, you can click the icon to send real-time results to other systems. You need to create a Slack incoming webhook so that Timeplus can send a slack message in the specific channel for each result. Please follow the Slack documentation for the instructions. Once you've got the Slack webhook URL, you can specify it in the dialog and set a message body. You can refer to the column name via the {{.column}} expression. For instance, assume the output of the query is You can set the message body to be The sensor data at {{.time}} is {{.number}}, with note: {{.note}} ## Send data to other systems via Redpanda Connect​ Starting from Timeplus Enterprise 2.5, you can send data to various systems by editing a Redpanda Connect yaml file. Redpanda Connect is a declarative data streaming service that solves a wide range of data engineering problems with simple, chained, stateless processing steps. You can submit a streaming SQL in SQL Console, and click the "Save As" button and choose "Sink". Expand the "Send data via Redpanda Connect" section or type a keyword to search for suitable components. Please refer to Redpanda Connect Documentations for Outputs for how to edit the configuration. - Overview - Send data to Kafka - Trigger actions via Webhook - Notify others via Slack - Send data to other systems via Redpanda Connect ## Enterprise-v2.3 - [Timeplus Enterprise 2.3 | Timeplus](https://docs.timeplus.com/enterprise-v2.3): This page lists the release history of Timeplus Enterprise. - Release Notes - Older 2.x Releases - Timeplus Enterprise 2.3 # Timeplus Enterprise 2.3 This page lists the release history of Timeplus Enterprise. Each release of Timeplus Enterprise includes the following components: - timeplusd: The core SQL engine - timeplus_appserver: The application server to provide web console access and REST API - timeplus_web: The web console static resources, managed by timeplus_appserver - timeplus_connector: The service to provide extra sources and sinks, managed by timeplus_appserver - timeplus: The CLI (Command Line Interface) to start/stop/manage the deployment. Each component tracks their changes with own version numbers. The version number for each Timeplus Enterprise release is a verified combination of Timeplus components. ## Key Highlights​ Key highlights of this release: - KV Stream, later on renamed to Mutable Streams - Able to modify materialized view settings - In a cluster, you can set those memory-consuming materialized views with a memory_weight setting, so that those materialized views will be evenly scheduled to run on different nodes. - New timeplus CLI commands to manage users ## Stable Releases​ ### 2.3.5​ Built on 07-02-2024. You can get it via: - Bare metal packages - For Docker users (not for production): docker run -p 8000:8000 timeplus/timeplus-enterprise:2.3.5 Component versions: - timeplusd 2.2.8 - timeplus_appserver 1.4.34 - timeplus_web 1.4.18 - timeplus_connector 1.5.3 - timeplus cli 1.0.9 Changelog (comparing to 2.3.0): - timeplusd feat: new setting allow_independent_shard_processing, default false. When data is already sharded correctly on file system and if the aggregation is on the sharding key, set it to true to avoid re-shuffle the data. feat: support modify materialized view query settings, e.g. alter stream mv_with_inner_stream modify query setting checkpoint_interval=600 feat: in a cluster, you can set those memory-consuming materialized views with a memory_weight setting, so that those materialized views will be evenly scheduled to run on different nodes. - feat: new setting allow_independent_shard_processing, default false. When data is already sharded correctly on file system and if the aggregation is on the sharding key, set it to true to avoid re-shuffle the data. - feat: support modify materialized view query settings, e.g. alter stream mv_with_inner_stream modify query setting checkpoint_interval=600 - feat: in a cluster, you can set those memory-consuming materialized views with a memory_weight setting, so that those materialized views will be evenly scheduled to run on different nodes. - timeplus_appserver feat: configuration items renaming, avoid using codenames feat: disabled user provision on timeplusd cluster feat: global metrics and dependencies feat: improved mv stats chore: updated onprem image name to timeplus-appserver - feat: configuration items renaming, avoid using codenames - feat: disabled user provision on timeplusd cluster - feat: global metrics and dependencies - feat: improved mv stats - chore: updated onprem image name to timeplus-appserver - timeplus_web fix(query): timeplusd resources do not refresh after DDL feat(datalineage): show resources stats on datalineage feat: do not provision user on timeplusd cluster - fix(query): timeplusd resources do not refresh after DDL - feat(datalineage): show resources stats on datalineage - feat: do not provision user on timeplusd cluster - cli feat: use more formal product name in command line messages feat: use more formal product name in command line messages feat: commands for user management feat: commands for cluster management - feat: use more formal product name in command line messages - feat: use more formal product name in command line messages - feat: commands for user management - feat: commands for cluster management - feat: new setting allow_independent_shard_processing, default false. When data is already sharded correctly on file system and if the aggregation is on the sharding key, set it to true to avoid re-shuffle the data. - feat: support modify materialized view query settings, e.g. alter stream mv_with_inner_stream modify query setting checkpoint_interval=600 - feat: in a cluster, you can set those memory-consuming materialized views with a memory_weight setting, so that those materialized views will be evenly scheduled to run on different nodes. - feat: configuration items renaming, avoid using codenames - feat: disabled user provision on timeplusd cluster - feat: global metrics and dependencies - feat: improved mv stats - chore: updated onprem image name to timeplus-appserver - fix(query): timeplusd resources do not refresh after DDL - feat(datalineage): show resources stats on datalineage - feat: do not provision user on timeplusd cluster - feat: use more formal product name in command line messages - feat: use more formal product name in command line messages - feat: commands for user management - feat: commands for cluster management ## Other Releases​ ### 2.3.0​ Built on 06-20-2024. You can get it via: - Bare metal packages - For Docker users (not for production): docker run -p 8000:8000 timeplus/timeplus-enterprise:2.3.0 Components: - timeplusd 2.2.7 - timeplus_appserver 1.4.32 - timeplus_web 1.4.17 - timeplus_connector 1.5.3 - timeplus cli 1.0.4 - Key Highlights - Stable Releases2.3.5 - 2.3.5 - Other Releases2.3.0 - 2.3.0 - 2.3.5 - 2.3.0 ## Enterprise-v2.4 - [Timeplus Enterprise 2.4 | Timeplus](https://docs.timeplus.com/enterprise-v2.4): Each release of Timeplus Enterprise includes the following components: - Release Notes - Older 2.x Releases - Timeplus Enterprise 2.4 # Timeplus Enterprise 2.4 Each release of Timeplus Enterprise includes the following components: - timeplusd: The core SQL engine - timeplus_appserver: The application server to provide web console access and REST API - timeplus_web: The web console static resources, managed by timeplus_appserver - timeplus_connector: The service to provide extra sources and sinks, managed by timeplus_appserver - timeplus: The CLI (Command Line Interface) to start/stop/manage the deployment. Each component tracks their changes with own version numbers. The version number for each Timeplus Enterprise release is a verified combination of Timeplus components. ## Key Highlights​ Key highlights of this release: - Distributed Mutable Streams for high performance query and UPSERT (UPDATE or INSERT), with primary keys, secondary keys, column families, sorting columns, parallel full scan and many more - External Streams to query or write to remote Timeplus, designed for data migration or hybrid deployment - Built-in system observability. Your workspace now comes with a system dashboard to monitor your cluster, including charts for running nodes and failed nodes, read/write throughput and EPS, used disk storage, and more. See additional metrics for resources in the details side panel, accessed via the data lineage or resource list pages, including status and any last errors - Kafka schema registry support for Avro output format - Read/write Kafka message keys via _tp_message_key column - More performance enhancements, including: Concurrent and idempotent data ingestion Memory efficiency improvement for window processing Independent shard query processing Async query state and materialized view checkpointing JOIN performance enhancements - Concurrent and idempotent data ingestion - Memory efficiency improvement for window processing - Independent shard query processing - Async query state and materialized view checkpointing - JOIN performance enhancements - After running a streaming query in the SQL console, you can check out the its query pipeline, or execution plan. Click "View Query Pipeline" under the SQL editor, and mouseover on the throughput boxes to see processing times - Timeplus Native JDBC with streaming SQL and fast insert - Concurrent and idempotent data ingestion - Memory efficiency improvement for window processing - Independent shard query processing - Async query state and materialized view checkpointing - JOIN performance enhancements ## Releases​ Please use the stable releases for production deployment, while we also provide latest engineering builds for testing and evaluation. ### 2.4.26 (Stable)​ Built on 02-18-2025. You can install via: - For Linux or Mac users: curl https://install.timeplus.com/2.4 | sh Downloads - For Kubernetes users: helm install timeplus/timeplus-enterprise --version v3.0.10 .. - For Docker users (not for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.4.26 Component versions: - timeplusd 2.3.34 - timeplus_appserver 1.4.44 - timeplus_web 1.4.33 - timeplus_connector 1.5.5 - timeplus cli 1.0.19 #### Changelog​ Compared to the 2.4.25 release: - timeplusd 2.3.32 -> 2.3.34 gracefully handle unsupported metadata commands improve garbage collection for NativeLog on clusters fix a bug during versioned schema fetch for inner storage of materialized views - gracefully handle unsupported metadata commands - improve garbage collection for NativeLog on clusters - fix a bug during versioned schema fetch for inner storage of materialized views - gracefully handle unsupported metadata commands - improve garbage collection for NativeLog on clusters - fix a bug during versioned schema fetch for inner storage of materialized views #### Known issues​ 1. If you have deployed one of the 2.3.x releases, you cannot reuse the data and configuration directly. Please have a clean installation of 2.4.x release, then use tools like timeplus sync CLI or Timeplus External Stream for migration. ### 2.4.25 (Stable)​ Built on 01-31-2025. You can install via: - For Linux or Mac users: curl https://install.timeplus.com/2.4 | sh Downloads - For Kubernetes users: helm install timeplus/timeplus-enterprise --version v3.0.9 .. - For Docker users (not for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.4.25 Component versions: - timeplusd 2.3.32 - timeplus_appserver 1.4.44 - timeplus_web 1.4.33 - timeplus_connector 1.5.5 - timeplus cli 1.0.19 #### Changelog​ Compared to the 2.4.23 release: - timeplusd 2.3.30 -> 2.3.32 fix potential corruption for a stream when it's altered multiple times better data recovery for file corruption due to power loss set mutable streams' default logstore retention policy from keeping forever to automatic - fix potential corruption for a stream when it's altered multiple times - better data recovery for file corruption due to power loss - set mutable streams' default logstore retention policy from keeping forever to automatic - fix potential corruption for a stream when it's altered multiple times - better data recovery for file corruption due to power loss - set mutable streams' default logstore retention policy from keeping forever to automatic #### Known issues​ 1. If you have deployed one of the 2.3.x releases, you cannot reuse the data and configuration directly. Please have a clean installation of 2.4.x release, then use tools like timeplus sync CLI or Timeplus External Stream for migration. ### 2.4.23 (Stable)​ Built on 08-22-2024. You can install via: - Bare metal packages - For Kubernetes users: helm install timeplus/timeplus-enterprise --version v3.0.7 .. - For Docker users (not for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.4.23 Component versions: - timeplusd 2.3.30 - timeplus_appserver 1.4.44 - timeplus_web 1.4.33 - timeplus_connector 1.5.5 - timeplus cli 1.0.19 #### Changelog​ Compared to the 2.4.19 release: - timeplusd 2.3.25 -> 2.3.30 support dropping partitions on cluster add additional query_type in sql analyzer, fixing the known issue enhanced historical asof joins with a performance improvement of over 30% bugfixes and performance enhancements - support dropping partitions on cluster - add additional query_type in sql analyzer, fixing the known issue - enhanced historical asof joins with a performance improvement of over 30% - bugfixes and performance enhancements - support dropping partitions on cluster - add additional query_type in sql analyzer, fixing the known issue - enhanced historical asof joins with a performance improvement of over 30% - bugfixes and performance enhancements #### Known issues​ 1. If you have deployed one of the 2.3.x releases, you cannot reuse the data and configuration directly. Please have a clean installation of 2.4.x release, then use tools like timeplus sync CLI or Timeplus External Stream for migration. ### 2.4.19​ Built on 08-15-2024. You can install via: - Bare metal packages - For Kubernetes users: helm install timeplus/timeplus-enterprise --version v3.0.3 .. - For Docker users (not for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.4.19 #### Changelog​ Compared to the 2.4.17 release: - timeplusd 2.3.23 -> 2.3.25 improvement: bug fixes and error handling - improvement: bug fixes and error handling - timeplus_web 1.4.31 -> 1.4.32 feat(ingest): use username:password for ingest API wizard - feat(ingest): use username:password for ingest API wizard - improvement: bug fixes and error handling - feat(ingest): use username:password for ingest API wizard #### Known issues​ 1. If you have deployed one of the 2.3.x releases, you cannot reuse the data and configuration directly. Please have a clean installation of 2.4.x release, then use tools like timeplus sync CLI or Timeplus External Stream for migration. 2. In Timeplus Console, no result will be shown for SQL SHOW FORMAT SCHEMAS or SHOW FUNCTIONS. This only impacts the web interface. Running such SQL via timeplusd client CLI or JDBC/ODBC will get the expected results. ### 2.4.17​ Built on 08-14-2024. You can install via: - Bare metal packages - For Docker users (not for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.4.17 #### Changelog​ Compared to the 2.4.16 release: - timeplusd 2.3.21 -> 2.3.23 Components: - timeplusd feat: support running table function on Timeplus External Stream improvement: track more stats: external_stream_read_failed, external_stream_written_failed, mv_recover_times, mv_memory_usage. improvement: better track memory usage in macOS and Docker container. feat: allow you to drop streams with force_drop_big_stream=true setting. improvement: default listen for 0.0.0.0 instead 127.1 (localhost) - feat: support running table function on Timeplus External Stream - improvement: track more stats: external_stream_read_failed, external_stream_written_failed, mv_recover_times, mv_memory_usage. - improvement: better track memory usage in macOS and Docker container. - feat: allow you to drop streams with force_drop_big_stream=true setting. - improvement: default listen for 0.0.0.0 instead 127.1 (localhost) - feat: support running table function on Timeplus External Stream - improvement: track more stats: external_stream_read_failed, external_stream_written_failed, mv_recover_times, mv_memory_usage. - improvement: better track memory usage in macOS and Docker container. - feat: allow you to drop streams with force_drop_big_stream=true setting. - improvement: default listen for 0.0.0.0 instead 127.1 (localhost) #### Known issues​ ... (content truncated) ## Enterprise-v2.5 - [Timeplus Enterprise 2.5 | Timeplus](https://docs.timeplus.com/enterprise-v2.5): Each release of Timeplus Enterprise includes the following components: - Release Notes - Timeplus Enterprise 2.5 # Timeplus Enterprise 2.5 Each release of Timeplus Enterprise includes the following components: - timeplusd: The core SQL engine - timeplus_appserver: The application server to provide web console access and REST API - timeplus_web: The web console static resources, managed by timeplus_appserver - timeplus_connector: The service to provide extra sources and sinks, managed by timeplus_appserver - timeplus: The CLI (Command Line Interface) to start/stop/manage the deployment. Each component tracks their changes with own version numbers. The version number for each Timeplus Enterprise release is a verified combination of Timeplus components. ## Key Highlights​ Key highlights of this release: - Reading or writing data in Apache Pulsar or StreamNative via External Stream. Learn more. - Connecting to various input or output systems via Redpanda Connect. Learn more. - Creating and managing users in the Web Console. You can change the password and assign the user either Administrator or Read-only role. - New migrate subcommand in timeplus CLI for data migration and backup/restore. - Materialized views auto-rebalancing in the cluster mode. Learn more. - Approximately 30% faster data ingestion and replication in the cluster mode. - Performance improvement for ASOF JOIN and EMIT ON UPDATE. ## Supported OS​ ## Releases​ Please use the stable releases for production deployment, while we also provide latest engineering builds for testing and evaluation. ### 2.5.13 (Public GA)​ Built on 07-08-2025. You can install via: - For Linux or Mac users: curl https://install.timeplus.com/2.5 | sh Downloads - For Kubernetes users: helm install timeplus/timeplus-enterprise --version v4.0.12 .. - For Docker users (not for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.5.13 Component versions: - timeplusd 2.4.30 - timeplus_web 2.0.6 - timeplus_appserver 2.0.9 - timeplus_connector 2.0.3 - timeplus cli 1.2.8 #### Changelog​ Compared to the 2.5.12 release: - timeplusd 2.4.27 -> 2.4.30 Bug fixes without new features - Bug fixes without new features - Bug fixes without new features #### Known issues​ 1. If you have deployed one of the 2.4.x releases, you can reuse the data and configuration directly. However, if your current deployment is 2.3 or earlier, you cannot upgrade directly. Please have a clean installation of 2.5.x release, then use tools like timeplus sync CLI or Timeplus External Stream for migration. 2. Pulsar external streams are only available in Linux bare metal builds and Linux-based Docker images. This type of external stream is not available in macOS bare metal builds. ### 2.5.12 (Public GA)​ Built on 01-09-2025. You can install via: - For Linux or Mac users: curl https://install.timeplus.com/2.5 | sh Downloads - For Kubernetes users: helm install timeplus/timeplus-enterprise --version v4.0.11 .. - For Docker users (not for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.5.12 Component versions: - timeplusd 2.4.27 - timeplus_web 2.0.6 - timeplus_appserver 2.0.9 - timeplus_connector 2.0.3 - timeplus cli 1.2.8 #### Changelog​ Compared to the 2.5.11 release: - timeplusd 2.4.26 -> 2.4.27 Able to drop malformed UDFs with DROP FUNCTION udf_name SETTINGS force=true. - Able to drop malformed UDFs with DROP FUNCTION udf_name SETTINGS force=true. - Able to drop malformed UDFs with DROP FUNCTION udf_name SETTINGS force=true. #### Known issues​ 1. If you have deployed one of the 2.4.x releases, you can reuse the data and configuration directly. However, if your current deployment is 2.3 or earlier, you cannot upgrade directly. Please have a clean installation of 2.5.x release, then use tools like timeplus sync CLI or Timeplus External Stream for migration. 2. Pulsar external streams are only available in Linux bare metal builds and Linux-based Docker images. This type of external stream is not available in macOS bare metal builds. ### 2.5.11 (Public GA)​ Built on 12-01-2024. You can install via: - Bare metal packages - For Kubernetes users: helm install timeplus/timeplus-enterprise --version v4.0.10 .. - For Docker users (not for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.5.11 Component versions: - timeplusd 2.4.26 - timeplus_web 2.0.6 - timeplus_appserver 2.0.9 - timeplus_connector 2.0.3 - timeplus cli 1.2.8 #### Changelog​ Compared to the 2.5.10 release: - timeplusd 2.4.24 -> 2.4.26 enhancements for multi-raft clusters bugfix for data ingestion in Timeplus external streams - enhancements for multi-raft clusters - bugfix for data ingestion in Timeplus external streams - enhancements for multi-raft clusters - bugfix for data ingestion in Timeplus external streams You can upgrade a deployment of Timeplus Enterprise 2.4 to Timeplus Enterprise 2.5, by stopping the components and replacing the binary files, or reusing the Docker or Kubernetes volumes and update the image versions. #### Known issues​ 1. If you have deployed one of the 2.4.x releases, you can reuse the data and configuration directly. However, if your current deployment is 2.3 or earlier, you cannot upgrade directly. Please have a clean installation of 2.5.x release, then use tools like timeplus sync CLI or Timeplus External Stream for migration. 2. Pulsar external streams are only available in Linux bare metal builds and Linux-based Docker images. This type of external stream is not available in macOS bare metal builds. ### 2.5.10 (Controlled Release)​ Built on 11-21-2024. You can install via: - Bare metal packages - For Docker users (not for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.5.10 Component versions: - timeplusd 2.4.24 - timeplus_web 2.0.6 - timeplus_appserver 2.0.9 - timeplus_connector 2.0.3 - timeplus cli 1.2.8 #### Changelog​ Compared to the 2.5.9 release: - timeplusd 2.4.23 -> 2.4.24 refined the advanced settings for streaming SQL: asterisk_include_reserved_columns and asterisk_include_tp_sn_column. introduced an advanced settings for NativeLog append data timeout: insert_timeout_ms. - refined the advanced settings for streaming SQL: asterisk_include_reserved_columns and asterisk_include_tp_sn_column. - introduced an advanced settings for NativeLog append data timeout: insert_timeout_ms. - timeplus cli 1.2.7 -> 1.2.8 introduced an option for timeplus start CLI to start the free trial with a new admin account. - introduced an option for timeplus start CLI to start the free trial with a new admin account. - refined the advanced settings for streaming SQL: asterisk_include_reserved_columns and asterisk_include_tp_sn_column. - introduced an advanced settings for NativeLog append data timeout: insert_timeout_ms. - introduced an option for timeplus start CLI to start the free trial with a new admin account. You can upgrade a deployment of Timeplus Enterprise 2.4 to Timeplus Enterprise 2.5, by stopping the components and replacing the binary files, or reusing the Docker or Kubernetes volumes and update the image versions. #### Known issues​ 1. If you have deployed one of the 2.4.x releases, you can reuse the data and configuration directly. However, if your current deployment is 2.3 or earlier, you cannot upgrade directly. Please have a clean installation of 2.5.x release, then use tools like timeplus sync CLI or Timeplus External Stream for migration. 2. Pulsar external streams are only available in Linux bare metal builds and Linux-based Docker images. This type of external stream is not available in macOS bare metal builds. ### 2.5.9 (Controlled Release)​ Built on 11-15-2024. You can install via: - Bare metal packages - For Docker users (not for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.5.9 Component versions: - timeplusd 2.4.23 - timeplus_web 2.0.6 - timeplus_appserver 2.0.9 - timeplus_connector 2.0.3 - timeplus cli 1.2.7 #### Changelog​ Compared to the 2.4.23 release: - timeplusd 2.3.30 -> 2.4.23 new type of External Streams for Apache Pulsar. for bare metal installation, previously you can login with the username default with empty password. To improve the security, this user has been removed. enhancement for nullable data types in streaming and historical queries. Materialized views auto-rebalancing in the cluster mode.Learn more. Approximately 30% faster data ingestion and replication in the cluster mode. Performance improvement for ASOF JOIN and EMIT ON UPDATE. - new type of External Streams for Apache Pulsar. - for bare metal installation, previously you can login with the username default with empty password. To improve the security, this user has been removed. - enhancement for nullable data types in streaming and historical queries. - Materialized views auto-rebalancing in the cluster mode.Learn more. - Approximately 30% faster data ingestion and replication in the cluster mode. - Performance improvement for ASOF JOIN and EMIT ON UPDATE. - timeplus_web 1.4.33 -> 2.0.6 ... (content truncated) ## Enterprise-v2.6 - [Timeplus Enterprise 2.6 | Timeplus](https://docs.timeplus.com/enterprise-v2.6): Each release of Timeplus Enterprise includes the following components: - Release Notes - Timeplus Enterprise 2.6 # Timeplus Enterprise 2.6 Each release of Timeplus Enterprise includes the following components: - timeplusd: The core SQL engine - timeplus_appserver: The application server providing web console access and REST API - timeplus_web: The web console static resources, managed by timeplus_appserver - timeplus_connector: The service providing extra sources and sinks, managed by timeplus_appserver - timeplus: The CLI (Command Line Interface) to start/stop/manage the deployment. Each component maintains its own version numbers. The version number for each Timeplus Enterprise release represents a verified combination of these components. ## Key Highlights​ Key highlights of this release: - Revolutionary hybrid hash table technology. For streaming SQL with JOINs or aggregations, by default a memory based hash table is used. This is helpful for preventing the memory limits from being exceeded for large data streams with hundreds of GB of data. You can adjust the query setting to apply the new hybrid hash table, which uses both the memory and the local disk to store the internal state as a hash table. - Enhanced operational visibility. Gain complete transparency into your system's performance through comprehensive monitoring of materialized views and streams. Track state changes, errors, and throughput metrics via system.stream_state_log and system.stream_metric_log. - Advanced cross-deployment integration. Seamlessly write data to remote Timeplus deployments by configuring Timeplus external stream as targets in materialized views. - Improved data management capabilities. Add new columns to an existing stream. Truncate historical data for streams. Create new databases to organize your streams and materialized views. - Optimized ClickHouse integration. Significant performance improvements for read/write operations with ClickHouse external tables. - Enhanced user experience. New UI wizards for Coinbase data sources and Apache Pulsar external streams, alongside a redesigned SQL Console and SQL Helper interface for improved usability. Quick access to streams, dashboards, and common actions via Command+K (Mac) or Windows+K (PC) keyboard shortcuts. ## Supported OS​ ## Releases​ ### 2.6.8​ Released on 04-13-2025. Installation options: - For Linux or Mac users: curl https://install.timeplus.com/2.6 | sh Downloads - For Kubernetes users: helm install timeplus/timeplus-enterprise --version v5.0.13 .. - For Docker users (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.6.8 Component versions: - timeplusd 2.5.20 - timeplus_web 2.1.8 - timeplus_appserver 2.1.6 - timeplus_connector 2.1.1 - timeplus cli 1.2.11 #### Changelog​ Compared to the 2.6.7 release: - timeplusd 2.5.19 -> 2.5.20 fixed the issue for where 1=2 - fixed the issue for where 1=2 - fixed the issue for where 1=2 Upgrade Instructions: Users can upgrade from Timeplus Enterprise 2.5 to 2.6 by stopping components and replacing binary files, or by updating Docker/Kubernetes image versions while maintaining existing volumes. #### Known issues​ 1. Direct upgrades from version 2.3 or earlier are not supported. Please perform a clean installation of 2.6.x and utilize timeplus sync CLI or Timeplus External Stream for data migration. 2. Pulsar external stream functionality is limited to Linux bare metal builds and Linux-based Docker images, excluding macOS bare metal builds. 3. The timeplus_connector component may experience health issues on Ubuntu Linux with x86_64 chips, affecting Redpanda Connect functionality. This issue is specific to Ubuntu and does not affect other Linux distributions. ### 2.6.7​ Released on 03-26-2025. Installation options: - For Linux or Mac users: Downloads - For Kubernetes users: helm install timeplus/timeplus-enterprise --version v5.0.12 .. - For Docker users (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.6.7 Component versions: - timeplusd 2.5.19 - timeplus_web 2.1.8 - timeplus_appserver 2.1.6 - timeplus_connector 2.1.1 - timeplus cli 1.2.11 #### Changelog​ Compared to the 2.6.6 release: - timeplusd 2.5.17 -> 2.5.19 fixed aggregate calculation issues re-applied blocks for non-replicated streams in cluster settings - fixed aggregate calculation issues - re-applied blocks for non-replicated streams in cluster settings - fixed aggregate calculation issues - re-applied blocks for non-replicated streams in cluster settings Upgrade Instructions: Users can upgrade from Timeplus Enterprise 2.5 to 2.6 by stopping components and replacing binary files, or by updating Docker/Kubernetes image versions while maintaining existing volumes. #### Known issues​ 1. Direct upgrades from version 2.3 or earlier are not supported. Please perform a clean installation of 2.6.x and utilize timeplus sync CLI or Timeplus External Stream for data migration. 2. Pulsar external stream functionality is limited to Linux bare metal builds and Linux-based Docker images, excluding macOS bare metal builds. 3. The timeplus_connector component may experience health issues on Ubuntu Linux with x86_64 chips, affecting Redpanda Connect functionality. This issue is specific to Ubuntu and does not affect other Linux distributions. ### 2.6.6​ Released on 03-24-2025. Installation options: - For Linux or Mac users: Downloads - For Kubernetes users: helm install timeplus/timeplus-enterprise --version v5.0.11 .. - For Docker users (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.6.6 Component versions: - timeplusd 2.5.17 - timeplus_web 2.1.8 - timeplus_appserver 2.1.6 - timeplus_connector 2.1.1 - timeplus cli 1.2.11 #### Changelog​ Compared to the 2.6.5 release: - timeplusd 2.5.13 -> 2.5.17 fixed the issue for Pulsar external stream with compressed message types and partitions - fixed the issue for Pulsar external stream with compressed message types and partitions - fixed the issue for Pulsar external stream with compressed message types and partitions Upgrade Instructions: Users can upgrade from Timeplus Enterprise 2.5 to 2.6 by stopping components and replacing binary files, or by updating Docker/Kubernetes image versions while maintaining existing volumes. #### Known issues​ 1. Direct upgrades from version 2.3 or earlier are not supported. Please perform a clean installation of 2.6.x and utilize timeplus sync CLI or Timeplus External Stream for data migration. 2. Pulsar external stream functionality is limited to Linux bare metal builds and Linux-based Docker images, excluding macOS bare metal builds. 3. The timeplus_connector component may experience health issues on Ubuntu Linux with x86_64 chips, affecting Redpanda Connect functionality. This issue is specific to Ubuntu and does not affect other Linux distributions. ### 2.6.5​ Released on 03-18-2025. Installation options: - For Linux or Mac users: Downloads - For Kubernetes users: helm install timeplus/timeplus-enterprise --version v5.0.10 .. - For Docker users (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.6.5 Component versions: - timeplusd 2.5.16 - timeplus_web 2.1.8 - timeplus_appserver 2.1.6 - timeplus_connector 2.1.1 - timeplus cli 1.2.11 #### Changelog​ Compared to the 2.6.4 release: - timeplusd 2.5.13 -> 2.5.16 fixed several problems causing aggregation function misbehavior - fixed several problems causing aggregation function misbehavior - fixed several problems causing aggregation function misbehavior Upgrade Instructions: Users can upgrade from Timeplus Enterprise 2.5 to 2.6 by stopping components and replacing binary files, or by updating Docker/Kubernetes image versions while maintaining existing volumes. #### Known issues​ 1. Direct upgrades from version 2.3 or earlier are not supported. Please perform a clean installation of 2.6.x and utilize timeplus sync CLI or Timeplus External Stream for data migration. 2. Pulsar external stream functionality is limited to Linux bare metal builds and Linux-based Docker images, excluding macOS bare metal builds. 3. The timeplus_connector component may experience health issues on Ubuntu Linux with x86_64 chips, affecting Redpanda Connect functionality. This issue is specific to Ubuntu and does not affect other Linux distributions. ### 2.6.4​ Released on 03-06-2025. Installation options: - For Linux or Mac users: Downloads - For Kubernetes users: helm install timeplus/timeplus-enterprise --version v5.0.9 .. - For Docker users (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.6.4 Component versions: - timeplusd 2.5.13 - timeplus_web 2.1.8 - timeplus_appserver 2.1.6 - timeplus_connector 2.1.1 - timeplus cli 1.2.11 #### Changelog​ Compared to the 2.6.3 release: - timeplusd 2.5.12 -> 2.5.13 fixed bugs related to materialized views stuck - fixed bugs related to materialized views stuck - fixed bugs related to materialized views stuck Upgrade Instructions: Users can upgrade from Timeplus Enterprise 2.5 to 2.6 by stopping components and replacing binary files, or by updating Docker/Kubernetes image versions while maintaining existing volumes. #### Known issues​ 1. Direct upgrades from version 2.3 or earlier are not supported. Please perform a clean installation of 2.6.x and utilize timeplus sync CLI or Timeplus External Stream for data migration. 2. Pulsar external stream functionality is limited to Linux bare metal builds and Linux-based Docker images, excluding macOS bare metal builds. 3. The timeplus_connector component may experience health issues on Ubuntu Linux with x86_64 chips, affecting Redpanda Connect functionality. This issue is specific to Ubuntu and does not affect other Linux distributions. ### 2.6.3​ ... (content truncated) ## Enterprise-v2.7 - [Timeplus Enterprise 2.7 | Timeplus](https://docs.timeplus.com/enterprise-v2.7): Each release of Timeplus Enterprise includes the following components: - Release Notes - Timeplus Enterprise 2.7 # Timeplus Enterprise 2.7 Each release of Timeplus Enterprise includes the following components: - timeplusd: The core SQL engine - timeplus_appserver: The application server providing web console access and REST API - timeplus_web: The web console static resources, managed by timeplus_appserver - timeplus_connector: The service providing extra sources and sinks, managed by timeplus_appserver - timeplus: The CLI (Command Line Interface) to start/stop/manage the deployment. Each component maintains its own version numbers. The version number for each Timeplus Enterprise release represents a verified combination of these components. ## Key Highlights​ Key highlights of this release: - Stream processing for files in S3 buckets: With the new S3 external table, Timeplus Enterprise now supports writing stream processing results to S3 buckets, or reading files in S3. - Join the latest data from MySQL or ClickHouse via dictionary: You can now create a dictionary to store key-value pairs in memory or a mutable stream, with data from various sources, such as files, MySQL/ClickHouse databases, or streams in Timeplus. - PostgreSQL and MySQL CDC via Redpanda Connect: Timeplus Enterprise now supports CDC (Change Data Capture) for PostgreSQL and MySQL databases via Redpanda Connect. This feature enables real-time data ingestion from these databases into Timeplus. - Support IAM authentication for accessing Amazon MSK: Avoid storing static credentials in Kafka external streams by setting sasl_mechanism to AWS_MSK_IAM. - Load the credentials from HashiCorp Vault or local file system for all external streams or external tables. - Mutable stream delete: You can now delete data from mutable streams with the DELETE SQL command. - Cluster monitoring and materialized view troubleshooting: Significant improvements in web console for multi-node cluster monitoring and troubleshooting. - Python UDF: You can now create user-defined functions (UDFs) in Python to extend the functionality of Timeplus with rich ecosystem of Python. It's currently in technical preview for Linux x86_64 only. ## Supported OS​ ## Upgrade Guide​ 1. Direct upgrades from version 2.3 or earlier are not supported. Please perform a clean installation of 2.7.x and utilize timeplus sync CLI or Timeplus External Stream for data migration. 2. For bare metal users, you can upgrade from Timeplus Enterprise 2.6 to 2.7 by stopping components and replacing binary files. 3. For Kubernetes users, please follow the guide carefully since a few timeplusd built-in users are removed in the new helm chart, and you can configure ingress for Appserver and Timeplusd independently. ## Releases​ ### 2.7.9​ Released on 06-05-2025. Installation options: - For Linux or Mac users: curl https://install.timeplus.com/2.7 | sh Downloads - For Kubernetes users: helm install timeplus/timeplus-enterprise --version v6.0.18 .. - For Docker users (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.7.9 Component versions: - timeplusd 2.7.48 - timeplus_web 2.2.12 - timeplus_appserver 2.2.13 - timeplus_connector 2.2.8 - timeplus cli 1.2.12 #### Changelog​ Compared to the 2.7.8 release: - timeplusd 2.7.47 -> 2.7.48 enhanced on parsing Kafka Protobuf data enhanced metadata update exception handling fixed check size, depth, size etc when there is cyclic AST ref - enhanced on parsing Kafka Protobuf data - enhanced metadata update exception handling - fixed check size, depth, size etc when there is cyclic AST ref - enhanced on parsing Kafka Protobuf data - enhanced metadata update exception handling - fixed check size, depth, size etc when there is cyclic AST ref #### Known issues​ 1. Pulsar external stream functionality is limited to Linux bare metal builds and Linux-based Docker images, excluding macOS bare metal builds. 2. The timeplus_connector component may experience health issues on Ubuntu Linux with x86_64 chips, affecting Redpanda Connect functionality. This issue is specific to Ubuntu and does not affect other Linux distributions. 3. Python UDF support is limited to Linux x86_64 bare metal and Linux x86_64 Docker image, excluding macOS or ARM builds. ### 2.7.8​ Released on 05-30-2025. Installation options: - For Linux or Mac users: curl https://install.timeplus.com/2.7 | sh Downloads - For Kubernetes users: helm install timeplus/timeplus-enterprise --version v6.0.16 .. - For Docker users (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.7.8 Component versions: - timeplusd 2.7.47 - timeplus_web 2.2.12 - timeplus_appserver 2.2.13 - timeplus_connector 2.2.8 - timeplus cli 1.2.12 #### Changelog​ Compared to the 2.7.7 release: - timeplusd 2.7.46 -> 2.7.47 support renaming stream column name fixed known stabilization issues - support renaming stream column name - fixed known stabilization issues - support renaming stream column name - fixed known stabilization issues #### Known issues​ 1. Pulsar external stream functionality is limited to Linux bare metal builds and Linux-based Docker images, excluding macOS bare metal builds. 2. The timeplus_connector component may experience health issues on Ubuntu Linux with x86_64 chips, affecting Redpanda Connect functionality. This issue is specific to Ubuntu and does not affect other Linux distributions. 3. Python UDF support is limited to Linux x86_64 bare metal and Linux x86_64 Docker image, excluding macOS or ARM builds. ### 2.7.7​ Released on 05-06-2025. Installation options: - For Linux or Mac users: curl https://install.timeplus.com/2.7 | sh Downloads - For Kubernetes users: helm install timeplus/timeplus-enterprise --version v6.0.12 .. - For Docker users (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.7.7 Component versions: - timeplusd 2.7.46 - timeplus_web 2.2.12 - timeplus_appserver 2.2.13 - timeplus_connector 2.2.8 - timeplus cli 1.2.12 #### Changelog​ Compared to the 2.7.6 release: - timeplusd 2.7.45 -> 2.7.46 fixed Python UDF known stabilization issues - fixed Python UDF known stabilization issues - fixed Python UDF known stabilization issues #### Known issues​ 1. Pulsar external stream functionality is limited to Linux bare metal builds and Linux-based Docker images, excluding macOS bare metal builds. 2. The timeplus_connector component may experience health issues on Ubuntu Linux with x86_64 chips, affecting Redpanda Connect functionality. This issue is specific to Ubuntu and does not affect other Linux distributions. 3. Python UDF support is limited to Linux x86_64 bare metal and Linux x86_64 Docker image, excluding macOS or ARM builds. ### 2.7.6​ Released on 04-29-2025. Installation options: - For Linux or Mac users: curl https://install.timeplus.com/2.7 | sh Downloads - For Kubernetes users: helm install timeplus/timeplus-enterprise --version v6.0.11 .. - For Docker users (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.7.6 Component versions: - timeplusd 2.7.45 - timeplus_web 2.2.12 - timeplus_appserver 2.2.13 - timeplus_connector 2.2.8 - timeplus cli 1.2.12 #### Changelog​ Compared to the 2.7.5 release: - timeplusd 2.7.37 -> 2.7.45 added new setting mv_preferred_exec_node while creating materialized view added new EMIT policy EMIT ON UPDATE WITH DELAY. The SQL syntax for EMIT has been refactored. Learn more fixed global aggregation with EMIT ON UPDATE in multi-shard environments fixed concurrency issues in hybrid aggregation support incremental checkpoints for hybrid hash join fixed grouping key issues in hybrid aggregation - added new setting mv_preferred_exec_node while creating materialized view - added new EMIT policy EMIT ON UPDATE WITH DELAY. The SQL syntax for EMIT has been refactored. Learn more - fixed global aggregation with EMIT ON UPDATE in multi-shard environments - fixed concurrency issues in hybrid aggregation - support incremental checkpoints for hybrid hash join - fixed grouping key issues in hybrid aggregation - added new setting mv_preferred_exec_node while creating materialized view - added new EMIT policy EMIT ON UPDATE WITH DELAY. The SQL syntax for EMIT has been refactored. Learn more - fixed global aggregation with EMIT ON UPDATE in multi-shard environments - fixed concurrency issues in hybrid aggregation - support incremental checkpoints for hybrid hash join - fixed grouping key issues in hybrid aggregation #### Known issues​ 1. Pulsar external stream functionality is limited to Linux bare metal builds and Linux-based Docker images, excluding macOS bare metal builds. 2. The timeplus_connector component may experience health issues on Ubuntu Linux with x86_64 chips, affecting Redpanda Connect functionality. This issue is specific to Ubuntu and does not affect other Linux distributions. 3. Python UDF support is limited to Linux x86_64 bare metal and Linux x86_64 Docker image, excluding macOS or ARM builds. ### 2.7.5​ Released on 04-14-2025. Installation options: - For Linux or Mac users: Downloads - For Kubernetes users: helm install timeplus/timeplus-enterprise --version v6.0.9 .. - For Docker users (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.7.5 Component versions: - timeplusd 2.7.37 - timeplus_web 2.2.12 - timeplus_appserver 2.2.13 - timeplus_connector 2.2.8 - timeplus cli 1.2.12 #### Changelog​ Compared to the 2.7.4 release: - timeplusd 2.7.35 -> 2.7.37 ... (content truncated) ## Enterprise-v2.8 - [Timeplus Enterprise 2.8 | Timeplus](https://docs.timeplus.com/enterprise-v2.8): Each release of Timeplus Enterprise includes the following components: - Release Notes - Timeplus Enterprise 2.8 # Timeplus Enterprise 2.8 Each release of Timeplus Enterprise includes the following components: - timeplusd: The core SQL engine - timeplus_appserver: The application server providing web console access and REST API - timeplus_web: The web console static resources, managed by timeplus_appserver - timeplus_connector: The service providing extra sources and sinks, managed by timeplus_appserver - timeplus: The CLI (Command Line Interface) to start/stop/manage the deployment. Each component maintains its own version numbers. The version number for each Timeplus Enterprise release represents a verified combination of these components. ## Key Highlights​ Key highlights of this release: - New Compute Node server role to run materialized views elastically with checkpoints on S3 storage. - Timeplus can read or write data in Apache Iceberg tables. Learn more - Timeplus can read or write PostgreSQL tables directly via PostgreSQL External Table or look up data via dictionaries. - Use S3 as the tiered storage for streams. - New SQL command to rename streams or columns. - JavaScript UDFs benefit from multiple V8 instances, improving concurrency and isolation. - A new page to visualize nodes in a cluster. - New page to view the details of streams or materialized views. ## Supported OS​ ## Releases​ We recommend using stable releases for production deployment. Engineering builds are available for testing and evaluation purposes. ### 2.8.2 (Public GA)​ Released on 07-22-2025. Installation options: - For Linux or Mac users: curl https://install.timeplus.com/2.8 | sh Downloads - For Kubernetes users: helm install timeplus/timeplus-enterprise --version v7.0.13 .. - For Docker users (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.8.2 Component versions: - timeplusd 2.8.27 - timeplus_web 2.8.12 - timeplus_appserver 2.8.11 - timeplus_connector 2.8.1 - timeplus cli 2.8.0 #### Changelog​ Compared to the 2.8.1 release: - timeplusd 2.8.26 -> 2.8.27 Some new features and enhancements in 2.9 are ported to 2.8.2: New JSON Data Type & SQL Functions: Added a native JSON data type and SQL functions json_encode, json_cast, json_array_length, json_merge_patch for powerful JSON manipulation. Mutable Stream Enhancements Introduced coalesced mutable streams for optimized storage and querying of updatable data. You can now define Time-To-Live (TTL) for data in mutable streams, automatically managing data retention. Able to add new columns for an existing mutable stream. Able to add or drop secondary index for mutable streams. Able to set version_column to make sure only rows with higher value of the version_column will override the rows with same primary key. This setting can work with or without coalesced. Support the UUID data type for primary key columns. HTTP External Stream: Added a new type of external stream to send streaming data to external HTTP endpoints, such as Splunk, Open Search and Slack. MongoDB External Table: Added a new type of external table to send streaming data to MongoDB. Enhanced MySQL External Table to support replace_query and on_duplicate_clause settings. Enhanced Kafka External Stream allows to customize the partitioner property, e.g. settings properties='partitioner=murmur2'. Enhanced Kafka External Stream and Pulsar External Stream to support write message headers via _tp_message_headers. Support map_from_arrays and map_cast with 4 or more parameters. SHOW CREATE command supports show_multi_versions=true to get the history of the object. New query setting precise_float_parsing to precisely handle float numbers. JavaScript User Defined Aggregation Function supports null value as input. Support UUIDv7 functions. Support ULID functions. A set of views are provided in the system namespace that enable effective troubleshooting and monitoring of your streaming data operations. Improved the support for gRPC protocol. Support EMIT TIMEOUT for both global aggregations and window aggregations. Able to change log level during runtime via SYSTEM SET LOG LEVEL or REST API. - Some new features and enhancements in 2.9 are ported to 2.8.2: New JSON Data Type & SQL Functions: Added a native JSON data type and SQL functions json_encode, json_cast, json_array_length, json_merge_patch for powerful JSON manipulation. Mutable Stream Enhancements Introduced coalesced mutable streams for optimized storage and querying of updatable data. You can now define Time-To-Live (TTL) for data in mutable streams, automatically managing data retention. Able to add new columns for an existing mutable stream. Able to add or drop secondary index for mutable streams. Able to set version_column to make sure only rows with higher value of the version_column will override the rows with same primary key. This setting can work with or without coalesced. Support the UUID data type for primary key columns. HTTP External Stream: Added a new type of external stream to send streaming data to external HTTP endpoints, such as Splunk, Open Search and Slack. MongoDB External Table: Added a new type of external table to send streaming data to MongoDB. Enhanced MySQL External Table to support replace_query and on_duplicate_clause settings. Enhanced Kafka External Stream allows to customize the partitioner property, e.g. settings properties='partitioner=murmur2'. Enhanced Kafka External Stream and Pulsar External Stream to support write message headers via _tp_message_headers. Support map_from_arrays and map_cast with 4 or more parameters. SHOW CREATE command supports show_multi_versions=true to get the history of the object. New query setting precise_float_parsing to precisely handle float numbers. JavaScript User Defined Aggregation Function supports null value as input. Support UUIDv7 functions. Support ULID functions. A set of views are provided in the system namespace that enable effective troubleshooting and monitoring of your streaming data operations. Improved the support for gRPC protocol. Support EMIT TIMEOUT for both global aggregations and window aggregations. Able to change log level during runtime via SYSTEM SET LOG LEVEL or REST API. - New JSON Data Type & SQL Functions: Added a native JSON data type and SQL functions json_encode, json_cast, json_array_length, json_merge_patch for powerful JSON manipulation. - Mutable Stream Enhancements Introduced coalesced mutable streams for optimized storage and querying of updatable data. You can now define Time-To-Live (TTL) for data in mutable streams, automatically managing data retention. Able to add new columns for an existing mutable stream. Able to add or drop secondary index for mutable streams. Able to set version_column to make sure only rows with higher value of the version_column will override the rows with same primary key. This setting can work with or without coalesced. Support the UUID data type for primary key columns. - Introduced coalesced mutable streams for optimized storage and querying of updatable data. - You can now define Time-To-Live (TTL) for data in mutable streams, automatically managing data retention. - Able to add new columns for an existing mutable stream. - Able to add or drop secondary index for mutable streams. - Able to set version_column to make sure only rows with higher value of the version_column will override the rows with same primary key. This setting can work with or without coalesced. - Support the UUID data type for primary key columns. - HTTP External Stream: Added a new type of external stream to send streaming data to external HTTP endpoints, such as Splunk, Open Search and Slack. - MongoDB External Table: Added a new type of external table to send streaming data to MongoDB. - Enhanced MySQL External Table to support replace_query and on_duplicate_clause settings. - Enhanced Kafka External Stream allows to customize the partitioner property, e.g. settings properties='partitioner=murmur2'. - Enhanced Kafka External Stream and Pulsar External Stream to support write message headers via _tp_message_headers. - Support map_from_arrays and map_cast with 4 or more parameters. - SHOW CREATE command supports show_multi_versions=true to get the history of the object. - New query setting precise_float_parsing to precisely handle float numbers. - JavaScript User Defined Aggregation Function supports null value as input. - Support UUIDv7 functions. - Support ULID functions. - A set of views are provided in the system namespace that enable effective troubleshooting and monitoring of your streaming data operations. - Improved the support for gRPC protocol. - Support EMIT TIMEOUT for both global aggregations and window aggregations. - Able to change log level during runtime via SYSTEM SET LOG LEVEL or REST API. - timeplus_web 2.8.8 -> 2.8.12 Some new UI features and enhancements in 2.9 are ported to 2.8.2: Materialized Views (MVs): Added UI support for pausing and resuming materialized views. Introduced Dead Letter Queue (DLQ) support and UI for MVs. Improved MV details page, monitoring, and statistics display. Enabled modifying comments for MVs via DDL. Added ability to inspect MV status data by navigating to the query page. Improved Cluster Details Page: Enhanced the cluster details page with a top statistics bar, better data presentation, and improved node details view. Enhanced cluster data generation and retrieval for UI display. In stream and materialized view list page, the earliest and latest columns have been removed to improve performance. In the SQL Query page, side panel is simplified by removing the snippets and functions accordion. In database selector, the empty database is shown as dimmed. In the materialized listing page, the "Pause" button is moved to the detailed page. - Some new UI features and enhancements in 2.9 are ported to 2.8.2: Materialized Views (MVs): ... (content truncated) ## Enterprise-v2.9 - [Timeplus Enterprise 2.9 | Timeplus](https://docs.timeplus.com/enterprise-v2.9): Each release of Timeplus Enterprise includes the following components: - Release Notes - Timeplus Enterprise 2.9 # Timeplus Enterprise 2.9 Each release of Timeplus Enterprise includes the following components: - timeplusd: The core SQL engine - timeplus_appserver: The application server providing web console access and REST API - timeplus_web: The web console static resources. Starting from Timeplus Enterprise 2.9, this component is packaged into timeplus_appserver. No longer as a separate binary or container - timeplus_connector: The service providing extra sources and sinks, managed by timeplus_appserver - timeplus: The CLI (Command Line Interface) to start/stop/manage the deployment Each component maintains its own version numbers. The version number for each Timeplus Enterprise release represents a verified combination of these components. ## Key Highlights​ Key highlights of the Timeplus 2.9 release include: - Enhanced Mutable Streams: Introducing online schema evolution, versioning, coalesced storage, Time-To-Live (TTL), and secondary index management capabilities. - Native JSON Support: A new native JSON data type and powerful json_encode / json_cast functions simplify working with JSON. - Improved Data Integrity: Dead Letter Queue (DLQ) support for Materialized Views ensures robust data processing. - Expanded Connectivity: Native HTTP External Stream for seamless integration with systems like Splunk, Elasticsearch, and more. - Performance Boost: JIT (Just-In-Time) compilation for streaming queries delivers significant performance and efficiency improvements. Large cardinality sessionization. - Parameterized Views: Create Parameterized Views for more flexible and reusable query patterns. - Scalable Log Processing: Distributed LogStream enables efficient handling of large volumes of log data. - Broader UDF Support: Python UDFs now run on ARM CPUs (Linux/macOS), and JavaScript UDFs benefit from multiple V8 instances. - Refined Cluster UI: The web console offers an improved experience for visualizing and managing cluster nodes. ## Supported OS​ ## Releases​ We recommend using stable releases for production deployment. Engineering builds are available for testing and evaluation purposes. ### 2.9.0 (Preview 2)​ Released on 07-15-2025. Installation options: - For Linux or Mac users: curl https://install.timeplus.com/2.9 | sh Downloads - For Docker users (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.9.0-preview.2 - We will provide new Helm Charts for Kubernetes deployment when v2.9 is GA. Component versions: - timeplusd 2.9.9-rc.20 - timeplus_web 2.9.35 - timeplus_appserver 2.9.34 - timeplus_connector 2.9.1 - timeplus cli 2.9.0 #### Changelog​ Compared to the 2.8.1 release: - timeplusd 2.8.26 -> 2.9.9-rc.20 New Features: Parameterized Views: You can now create parameterized views, allowing for more dynamic and reusable view definitions. JIT Compilation for Queries: Introduced Just-In-Time (JIT) compilation for queries, potentially improving execution performance for certain query types. New JSON Data Type & SQL Functions: Added a native JSON data type and SQL functions json_encode, json_cast, json_array_length, json_merge_patch for powerful JSON manipulation. Mutable Stream TTL: You can now define Time-To-Live (TTL) for data in mutable streams, automatically managing data retention. Materialized View DLQ: Introduced Dead Letter Queue (DLQ) support for materialized views to handle data processing errors more robustly. HTTP External Stream: Added a new type of external stream to send streaming data to external HTTP endpoints, such as Splunk, Open Search and Slack. MongoDB External Table: Added a new type of external table to send streaming data to MongoDB. Enhanced MySQL External Table to support replace_query and on_duplicate_clause settings. Enhanced Kafka External Stream and Pulsar External Stream to support write message headers via _tp_message_headers Build and manage Alerts with SQL. Monitor your streaming data and automatically trigger actions when specific conditions are met. Python UDFs on ARM: Python User-Defined Functions (UDFs) are now supported on ARM-based architectures (Linux/macOS), expanding platform compatibility. Improved JavaScript UDFs: Enhanced JavaScript UDF execution with support for multiple V8 instances, improving concurrency and isolation (also available in 2.8.1 or above). JavaScript User Defined Aggregation Function supports null value as input. Log Stream Virtual Columns: Log streams now include _filepath and _filename virtual columns, providing richer context about the data source. UUID as Primary Key: Mutable streams now support the UUID data type for primary key columns. Support SQL UDF on cluster. Support UUIDv7 functions SQL and Data Model Enhancements: Advanced EMIT Clause: The EMIT clause for changelog generation now supports EMIT ON UPDATE WITH DELAY and EMIT AFTER KEY EXPIRE options for more granular control over streaming results. ALTER STREAM for Multiple Columns: You can now add or modify multiple columns in a single ALTER STREAM command. Modifying Comments: Added ALTER COMMENT support for streams, views, materialized views, KVStreams, and RandomStreams. Mutable Stream Schema Evolution: Support for adding new columns and dropping secondary indexes in mutable streams. Support writing to nested array of records Avro schemas Enhanced Kafka External Stream allows to customize the partitioner property, e.g. settings properties='partitioner=murmur2' New query setting precise_float_parsing to precisely handle float numbers. Added emit policy EMIT TIMEOUT and EMIT PER EVENT Added new functions array_partial_sort and array_partial_reverse_sort Performance and Scalability: Incremental Checkpointing: Implemented and enabled incremental checkpointing by default for substreams, hybrid hash joins, and Materialized Views, significantly reducing recovery time and resource usage during stateful operations. Optimized Connection Pooling: Refactored internal connection pooling for improved performance and resource management. Parallel Log Processing: Log files from sources can now be processed in parallel for faster data ingestion. Coalesced Mutable Streams: Introduced coalesced mutable streams for optimized storage and querying of updatable data. Distributed LogStream: Enhanced LogStream capabilities for distributed environments, improving scalability for log data processing. Monitoring and Management: New System Views: Introduced additional built-in system views (e.g., for troubleshooting distributed queries, MVs, query memory, and checkpoint status) for enhanced system observability. Versioned SHOW CREATE: The SHOW CREATE command now displays multi-versioned definitions for streams, views, MVs, UDFs, format schemas, and databases, aiding in tracking changes. Disk Metrics for Dictionaries: Added metrics to monitor disk size usage by dictionaries. MV Node Pinning & Placement: Added functionality to pin materialized view execution to specific cluster nodes and manage node placements for better resource control. Kafka External Stream Timeout: Added connection_timeout_ms setting for Kafka external streams. Dependency Checks for Storage Policies: Added checks for dependencies before allowing a storage policy or disk to be dropped. A set of views are provided in the system namespace that enable effective troubleshooting and monitoring of your streaming data operations External Data Integration: Kafka Enhancements: Added support for writing Kafka message timestamps and improved error handling for Kafka external streams with _tp_time and CSV format. Iceberg Integration: Provided various bug fixes and enhancements for interacting with Apache Iceberg tables. Pulsar Client Upgrade: Upgraded the Pulsar C++ client to version 3.7.0. Security Enhancements: Improved mechanisms for password propagation within clustered environments. Support for utilizing user information from HTTP URL parameters for authentication or context. - New Features: ... (content truncated) ## External-stream - [External Stream | Timeplus](https://docs.timeplus.com/external-stream): You can create External Streams in Timeplus to query data in the external systems without loading the data into Timeplus. The main benefit for doing so is to keep a single source of truth in the external systems (e.g. Apache Kafka), without duplicating them. In many cases, this can also achieve even lower latency to process Kafka or Pulsar data, because the data is read directly by Timeplus core engine, without other components, such as Redpanda Connect or Airbyte. - Core Features - External Streams & Tables - External Streams # External Stream You can create External Streams in Timeplus to query data in the external systems without loading the data into Timeplus. The main benefit for doing so is to keep a single source of truth in the external systems (e.g. Apache Kafka), without duplicating them. In many cases, this can also achieve even lower latency to process Kafka or Pulsar data, because the data is read directly by Timeplus core engine, without other components, such as Redpanda Connect or Airbyte. You can run streaming analytics with the external streams in the similar way as other streams, with some limitations. Timeplus supports 4 types of external streams: - Kafka External Stream - Pulsar External Stream - Log External Stream (experimental) Besides external streams, Timeplus also provides external tables to query data in ClickHouse, MySQL, Postgres or S3/Iceberg. The difference of external tables and external streams is that external tables are not real-time, and they are not designed for streaming analytics. You can use external tables to query data in the external systems, but you cannot run streaming SQL on them. Learn more about external tables. ## Faq - [Timeplus Enterprise FAQ | Timeplus](https://docs.timeplus.com/faq): This document provides answers to frequently asked questions about Timeplus Enterprise, including its features, usage, and troubleshooting. - FAQ # Timeplus Enterprise FAQ This document provides answers to frequently asked questions about Timeplus Enterprise, including its features, usage, and troubleshooting. ## How to read/write Kafka or Redpanda​ You use External Stream to read from Kafka topics or write data to the topics. We verified the integration with Apache Kafka, Confluent Cloud, Confluent Platform, Redpanda, WarpStream and many more. ## How to load data from PostgreSQL/MySQL/ClickHouse​ For PostgreSQL, MySQL or other OLTP databases, you can apply the CDC (Change Data Capture) technology to load realtime changes to Proton via Debezium and Kafka/Redpanda. Example configuration at the cdc folder of proton repo. This blog shows the Timeplus Enterprise UI but could be applied to Proton too. If you have data in local ClickHouse or ClickHouse Cloud, you can also use External Table to read data. ## How to read/write ClickHouse​ You use External Table to read from ClickHouse tables or write data to the ClickHouse tables. We verified the integration with self-hosted ClickHouse, ClickHouse Cloud, Aiven for ClickHouse and many more. ## How to handle UPSERT or DELETE​ By default, streams in Timeplus are append-only. But you can create a stream with versioned_kv or changelog_kv mode to support data mutation or deletion. The Versioned Stream supports UPSERT (Update or Insert) and Changelog Stream supports UPSERT and DELETE. You can use tools like Debezium to send CDC messages to Timeplus, or just use INSERT SQL to add data. Values with same primary key(s) will be overwritten. For more details, please check this video: ## How to work with JSON​ Proton supports powerful, yet easy-to-use JSON processing. You can save the entire JSON document as a raw column in string type. Then use JSON path as the shortcut to access those values as string. For example raw:a.b.c. If your data is in int/float/bool or other type, you can also use :: to convert them. For example raw:a.b.c::int. If you want to read JSON documents in Kafka topics, you can choose to read each JSON as a raw string, or read each top level key/value pairs as columns. Please check the doc for details. ## How to load CSV files​ If you only need to load a single CSV file, you can create a stream then use the INSERT INTO .. SELECT .. FROM file(..) syntax. For example, if there are 3 fields in the CSV file: timestamp, price, volume, you can create the stream via Please note there will be the 4th column in the stream, which is _tp_time as the Event Time. To import CSV content, use the file table function to set the file path and header and data types. Please note: 1. You need to specify the column names. Otherwise SELECT * will get 3 columns while there are 4 columns in the data stream. 2. For security reasons, Proton only read files under proton-data/user_files folder. If you install proton via proton install command on Linux servers, the folder will be /var/lib/proton/user_files. If you don't install proton and run proton binary directly via proton server start, the folder will be proton-data/user_files 3. We recommend to use max_insert_threads=8 to use multiple threads to maximize the ingestion performance. If your file system has high IOPS, you can create the stream with SETTINGS shards=3 and set a higher max_insert_threads value in the INSERT statement. If you need to import multiple CSV files to a single stream, you can do something similar. You can even add one more column to track the file path. ## How to visualize Timeplus query results with Grafana or Metabase​ The official Grafana plugin for Timeplus is available here. The source code is at https://github.com/timeplus-io/proton-grafana-source. You can run streaming SQL with the plugin and build live charts in Grafana, without having to refresh the dashboard. Check out here for sample setup. We also provide a plugin for Metabase: https://github.com/timeplus-io/metabase-proton-driver This is based on the Proton JDBC driver. ## How to access Timeplus Proton programmatically​ SQL is the main interface to work with Proton. The Ingest REST API allows you to push realtime data to Proton with any language. The following drivers are available: - https://github.com/timeplus-io/proton-java-driver JDBC and other Java clients - https://github.com/timeplus-io/proton-go-driver for Golang - https://github.com/timeplus-io/proton-python-driver for Python ## Get the number of failed materialized views via prometheus​ Follow the Prometheus Integration to access the metrics endpoint of timeplusd. You can use TimeplusdMaterializedView_QueryStatus metrics to check the status code of the materialized views. - How to read/write Kafka or Redpanda - How to load data from PostgreSQL/MySQL/ClickHouse - How to read/write ClickHouse - How to handle UPSERT or DELETE - How to work with JSON - How to load CSV files - How to visualize Timeplus query results with Grafana or Metabase - How to access Timeplus Proton programmatically - Get the number of failed materialized views via prometheus ## Flyway - [DevOps with Flyway | Timeplus](https://docs.timeplus.com/flyway): Redgate Flyway extends DevOps to databases to accelerate software delivery and ensure quality code. From version control to continuous delivery, Flyway builds on application delivery processes to automate database deployments. - Integrations - Third-party Tools - DevOps with Flyway # DevOps with Flyway Redgate Flyway extends DevOps to databases to accelerate software delivery and ensure quality code. From version control to continuous delivery, Flyway builds on application delivery processes to automate database deployments. A database extension for Timeplus is available at GitHub. If you already use Flyway in your DevOps toolchain, you can use Flyway to manage SQL resources with version control and continuous delivery. ## Demo​ Check out this short video that demonstrates how to create, update and clean up Timeplus resources with Flyway. ## Installation​ Timeplus team have submitted the plugin to Flyway community, before it's merged and available in flyway releases, please download the Timeplus extension as a JAR file from the GitHub Releases page. Flyway can be installed as a command-line tool or added to your Java project via Maven or Gradle build tools. Put the flyway-database-timeplus-VERSION_NUMBER.jar to the proper folder or the classpath of the Java project. You also need to add the Timeplus JDBC driver to the flyway folder or Java classpath. Download timeplus-native-jdbc-shaded-VERSION_NUMBER.jar from https://github.com/timeplus-io/timeplus-native-jdbc/releases. Taking flyway CLI on macOS as an example: - You can install it via brew install flyway - Download and put the extension to /opt/homebrew/Cellar/flyway/11.1.0/libexec/lib/flyway/flyway-database-timeplus-10.16.3.jar - Download and put the JDBC driver to /opt/homebrew/Cellar/flyway/11.1.0/libexec/drivers/timeplus-native-jdbc-shaded-2.0.7.jar ## Setup a sample project​ Create a new folder and create a flyway.toml file inside. Change the url, user and password value to match your Timeplus deployment. Create a new folder 'migrations' with a few SQL scripts: V1__Create_person_stream.sql: V2__Add_people.sql: V3__Create_Protobuf_Schema.sql: The folder structure is: ## Flyway Commands​ Run the flyway command from the folder where flyway.toml resides. - Run flyway info to validation the connection to Timeplus, as well as scanning the SQL scripts. - Run flyway migrate to run the SQL scripts. This will setup flyway_schema_history to track schema history if the table doesn't exist. - Run flyway clean to delete all streams and the flyway_schema_history table. - Run flyway repair if there is any failure in prior flyway migrate. This command will remove related rows from flyway_schema_history table. ## Limitations​ - flyway_schema_history is created as a MergeTree table. We plan to add deletion in Mutable Streams and create this as a Mutable Stream. If your deployment is a cluster version of Timeplus Enterprise, please configure flyway to talk to the same node, since the table won't be replicated among the cluster. - Demo - Installation - Setup a sample project - Flyway Commands - Limitations ## Functions - [SQL Functions | Timeplus](https://docs.timeplus.com/functions): Timeplus supports ANSI-SQL standard syntax. The following functions are provided for various use cases. Most of the functions in ClickHouse are available in Timeplus, with different naming conventions (for example arrayjoin instead of arrayJoin). New functions are added for streaming processing. Please contact us if you need more functions. - SQL Reference - Functions # SQL Functions Timeplus supports ANSI-SQL standard syntax. The following functions are provided for various use cases. Most of the functions in ClickHouse are available in Timeplus, with different naming conventions (for example array_join instead of arrayJoin). New functions are added for streaming processing. Please contact us if you need more functions. Please note, in Timeplus Proton 1.3.27 or the earlier versions, SQL functions were case-sensitive and always in lower-case. Since 1.3.28, function names are case-insensitive. For example count, COUNT and Count work in the same way. SQL Keywords such as SELECT or FROM are case-insensitive too. In the rest of the documentation, we use function names in lower case for demonstration. Type Conversion Arrays, Maps, Tuples Process Data and Time Process URL Process JSON Process Text Hash Aggregation Logic (if, multi_if) Math Financial Geo Location Streaming Processing ## Functions_for_agg - [Aggregation Functions | Timeplus](https://docs.timeplus.com/functions_for_agg): count - SQL Reference - Functions - Aggregation Functions # Aggregation Functions ### count​ count(*) to get the row number, or count(col) to get the number of rows when col is not NULL ### count_distinct​ count_distinct(col) to get the number of unique values for the col column. Same as count(distinct col) ### count_if​ count_if(condition) to apply a filter with condition and get the number of records. e.g. count_if(speed_kmh>80) ### distinct​ distinct(col)to get the distinct value for the col column. ### unique​ unique([, , ...]): Calculates the approximate number of different values of the columns. ### unique_exact​ unique_exact([, , ...])Calculates the exact number of different values of the columns. ### unique_exact_if​ unique_exact_if(col,condition) to apply a filter with condition and get the distinct count of col, e.g. to get the cars with high speed unique_exact_if(cid,speed_kmh>80) ### min​ min(): minimum value of a column. For String column, the comparison is lexicographic order. ### max​ max(): maximum value of a column. For String column, the comparison is lexicographic order. ### sum​ sum(): sum of the columns. Only works for numbers. ### avg​ avg(): average value of a column (sum(column) / count(column)). Only works for numeric columns. ### median​ median() Calculate median of a numeric data sample. ### quantile​ quantile(column,level)Calculate an approximate quantile of a numeric data sequence. e.g. quantile(a,0.9)to get the P90 for the column and quantile(a,0.5) to get the median number ### p90​ short for quantile(a,0.9) ### p95​ short for quantile(a,0.95) ### p99​ short for quantile(a,0.99) ### top_k​ top_k(,K): Top frequent K items in column_name. Return an array. e.g. top_k(cid, 3) may get ["c01","c02","c03"] If you need to get the event count, you can set true as the 3rd parameter, e.g. top_k(cid, 3, true) may get [("c01",1200,0),("c02",800,0),("c03",700,0)]. The 3rd element in the tuple is the probability of the error of the calculation, since it is an approximate calculation. 0 means no error. If you want to get the exact top K without approximate calculation, you can use top_k_exact function, e.g. top_k_exact(cid,3), which will be slower than top_k function. Read more on Top-N Query Pattern page. ### min_k​ min_k(,K [,context_column]): The least K items in column_name. Return an array. You can also add a list of columns to get more context of the values in same row, such as min_k(price,3,product_id,last_updated) This will return an array with each element as a tuple, such as [(5.12,'c42664'),(5.12,'c42664'),(15.36,'c84068')] Read more on Top-N Query Pattern page. ### max_k​ max_k(,K[,context_column]): The greatest K items in column_name. You can also add a list of columns to get more context of the values in same row, such as max_k(price,3,product_id,last_updated) Read more on Top-N Query Pattern page. ### arg_min​ arg_min(argument, value_column) Gets the value in the argument column for a minimal value in the value_column. If there are several different values of argument for minimal values of value_column, it returns the first of these values encountered. You can achieve the same query with min_k(value_column,1, argument)[1].2 . But this is much easier. ### arg_max​ arg_max(argument, value_column) Gets the value in the argument column for a maximum value in the value_column. If there are several different values of argument for maximum values of value_column, it returns the first of these values encountered. You can achieve the same query with max_k(value_column,1, argument)[1].2 . But this is much easier. ### group_array​ group_array() to combine the values of the specific column as an array. For example, if there are 3 rows and the values for these columns are "a","b","c". This function will generate a single row and single column with value ['a','b','c']. Starting from Timeplus Enterprise v2.7, you can set the second parameter to specify the maximum number of elements in the array. If the number of elements exceeds the specified value, the function will return an array with the first max_length elements. For example, group_array(a, 2) will return ['a','b'] if the original array is ['a','b','c']. ### group_array_last​ group_array_last(, max_size) to combine the values of the specific column as an array. For example, if there are 3 rows and the values for these columns are "a","b","c". group_array_last(col,2) will generate a single row and single column with value ['b','c']. ### group_array_sorted​ group_array_sorted() to combine the values of the specific column as an array, sorted in ascending order. For example, if there are 3 rows and the values for these columns are "c","b","a". This function will generate a single row and single column with value ['a','b','c']. ### group_array_sample​ group_array_sample(, ) to combine the values of the specific column as an array, sampled randomly. For example, if there are 3 rows and the values for these columns are "a","b","c". This function will generate a single row and single column with value ['a','b'] with group_array_sample(col,2). ### group_uniq_array​ group_uniq_array() to combine the values of the specific column as an array, making sure only unique values in it. For example, if there are 3 rows and the values for these columns are "a","a","c". This function will generate a single row and single column with value ['a','c']. Starting from Timeplus Enterprise v2.7, you can set the second parameter to specify the maximum number of elements in the array. If the number of elements exceeds the specified value, the function will return an array with the first max_length elements. For example, group_uniq_array(a, 2) will return ['a','b'] if the original array is ['a','b','c']. ### group_concat​ group_concat() to combine the values of the specific column as a string, separated by a comma. For example, if there are 3 rows and the values for these columns are "a","b","c". This function will generate a single row and single column with value 'abc'. To combine the values of the specific column as a string, separated by a semicolon, use group_concat(, ';'). ### moving_sum​ moving_sum(column) returns an array with the moving sum of the specified column. For example, select moving_sum(a) from(select 1 as a union select 2 as a union select 3 as a) will return [1,3,6]. ### any​ any(column) Selects the first encountered (non-NULL) value, unless all rows have NULL values in that column. The query can be executed in any order and even in a different order each time, so the result of this function is indeterminate. To get a determinate result, you can use the min or max function instead of any. ### first_value​ first_value(column) Selects the first encountered value. ### last_value​ last_value(column) Selects the last encountered value. ### stochastic_linear_regression_state​ stochastic_linear_regression_state(num, target, param1, param2) This function implements stochastic linear regression. It supports custom parameters for learning rate, L2 regularization coefficient, mini-batch size and has few methods for updating weights (Adam (used by default), simple SGD, Momentum, Nesterov). Learn more at ClickHouse docs. ### stochastic_logistic_regression​ stochastic_logistic_regression(num, num, num, string) This function implements stochastic logistic regression. It can be used for binary classification problem, supports the same custom parameters as stochasticLinearRegression and works the same way. Learn more at ClickHouse docs. ### largest_triangle_three_buckets​ largest_triangle_three_buckets(x, y, n) or lttb(x, y, n). x is the x coordinate. y is the y coordinate. n is the number of points in the resulting series. Applies the Largest-Triangle-Three-Buckets algorithm to the input data. The algorithm is used for downsampling time series data for visualization. It is designed to operate on series sorted by x coordinate. It works by dividing the sorted series into buckets and then finding the largest triangle in each bucket. The number of buckets is equal to the number of points in the resulting series. The function will sort data by x and then apply the downsampling algorithm to the sorted data. For example: ### lttb​ Alias for largest_triangle_three_buckets. ### avg_time_weighted​ avg_time_weighted(column, time_column) to calculate the time-weighted average of the column. The time column should be in the format of datetime,datetime64 or date. Optionally, you can add a third parameter to specify an end time for your analysis period. When you omit the third parameter, the calculation excludes the last value. If provided, the end time must match the timestamp column's data type, and the function uses the difference between the last time point and this end time as the weight for the final value. ### median_time_weighted​ median_time_weighted(column, time_column) to calculate the time-weighted median of the column. The time column should be in the format of datetime,datetime64 or date. This function also takes an optional third parameter to specify an end time for your analysis period. ### histogram​ histogram(column, bin_count) to calculate the histogram of the column. The bin count should be a positive integer. ### kurt_pop​ kurt_pop(expr) Computes the kurtosis of a sequence. ### kurt_samp​ ... (content truncated) ## Functions_for_comp - [Arrays, Maps, Tuples | Timeplus](https://docs.timeplus.com/functions_for_comp): Arrays - SQL Reference - Functions - Regular Functions - Arrays, Maps, Tuples # Arrays, Maps, Tuples ## Arrays​ ### array_cast​ array_cast(element1,element2,..) create a new array with the given elements, e.g. array_cast(1,2) will get [1,2] Please note, the elements should be in the same type, such as array_cast('a','n'), not array_cast('a',0) ### length​ length(array) Get the length of the array. ### array[index]​ You can easily access any element in the array, just using arrayName[index], such as topValues[2] The first element's index is 1, instead of 0. ### index_of​ index_of(arr,x) returns the index of x in the array arr. The first element's index is 1. Return 0 if x is not in the array. ### array_compact​ array_compact(arr) Removes consecutive duplicate elements from an array, e.g. array_compact([1,1,2,2,2,3,4,4,5])returns [1,2,3,4,5] ### array_concat​ array_concat(array1,array2) Concatenates two arrays into one. ### array_difference​ array_difference(arr) calculates the difference between adjacent array elements. Returns an array where the first element will be 0, the second is the difference between a[2] - a[1], etc. e.g. array_difference([1,2,3,5])returns [0,1,1,2] ### array_distinct​ array_distinct(arr) returns an array containing the distinct elements only. e.g. array_distinct([1,1,2,3,3,1])return [1,2,3], while array_compact([1,1,2,3,3,1])returns [1,2,3,1] ### array_flatten​ array_flatten(array1, array2,..)Converts an array of arrays to a flat array. e.g. array_flatten([[[1]], [[2], [3]]]) returns [1,2,3] ### array_string_concat​ array_string_concat(arr[, separator]) Concatenates string representations of values listed in the array with the separator. separator is an optional parameter: a constant string, set to an empty string by default. For example array_string_concat([1,2,3],'-') to get a string 1-2-3 ### array_join​ array_join(an_array) This is a special function. group_array(col) to group the column value from multiple rows to a single value in a row. array_join does the opposite: it can convert one row with an array value to multiple rows. For example select array_join([10,20]) as v, 'text' as t will get 2 rows ### array_pop_back​ array_pop_back(array) removes the last item from the array. e.g. array_pop_back([1,2,3]) returns [1,2] ### array_pop_front​ array_pop_front(array) removes the first item from the array. e.g. array_pop_front([1,2,3]) returns [2,3] ### array_push_back​ array_push_back(array, value) adds the value to the array as the last item. e.g. array_push_back([1,2,3],4) returns [1,2,3,4] ### array_push_front​ array_push_front(array, value) adds the value to the array as the first item. e.g. array_push_front([1,2,3],4) returns [4,1,2,3] ### array_product​ array_product(array) multiplies elements in the array. e.g. array_product([2,3,4]) returns 24 (2 x 3 x 4) ### array_resize​ array_resize(array, size [,extender]) changes the length of the array. If sizeis smaller than the current length of the array, the array is truncated. Otherwise, a new array with the specified size is created, filling value with the specified extender. e.g. array_resize([3,4],1) returns [3]. array_resize([3,4],4,5)returns [3,4,5,5] ### array_reverse​ array_reverse(arr) returns an array with the reversed order of the original array, e.g. array_reverse([1,2,3]) returns [3,2,1] ### array_slice​ array_slice(arr, offset [,length]) returns a slice of the array. If length is not specified, then slice to the end of the array, e.g. array_slice([1,2,3,4,5],2) returns [2,3,4,5]. If offset is greater than the array length, it returns an empty array []. If length is specified, this is the length of the new array, e.g. array_slice([1,2,3,4,5],2,3) returns [2,3,4] ### array_uniq​ array_uniq(arr) returns the number of unique values in the array, e.g. array_uniq([1,1,2,3]) returns 3 ### array_zip​ array_zip(arr1,arr2,.. arrN) group elements from different arrays to a new array of tuples. e.g. array_zip([1,2,3],['a','b','c']) returns [(1,'a'),(2,'b'),(3,'c')] ### array_all​ array_all([func,] array) returns 1(true) or 0(false) if all elements in the array meets the condition. For example, array_all([1,2]) return 1, and array_all([0,0])return 0. You can pass a lambda function to it as the first argument to customize the condition check, such as array_all(x->x%2=0,[2,4,6]) to check whether each element in the array is even. It returns 1. ### array_avg​ array_avg([func,] array) returns the average value in the array. For example, array_avg([2,6]) return 4. You can pass a lambda function to it as the first argument to apply on each element before calculating the average, such as array_avg(x->x*x,[2,6]) to get the average for 2*2 and 6*6, which is 20. ### array_count​ array_count([func,] array) returns the number of elements in the array meeting the condition. By default, check whether the value is not 0. e.g. array_count([0,0,1,2]) returns 2. You can pass a lambda function to it as the first argument to apply on each element before calculating the count, such as array_count(x->x>1,[1,2]) to get the number of numbers which is greater than 1, it returns 1. ### array_cum_sum​ array_cum_sum([func,] array) returns an array of partial sums of elements in the source array (a running sum). e.g. array_cum_sum([1,1,1]) returns [1,2,3]. You can pass a lambda function to it as the first argument to apply on each element before calculating the moving sum, such as array_cum_sum(x->x*x,[1,2]) to get [1,5] ### array_exists​ array_exists([func,] array) returns 1(true) or 0(false) if any element in the array meet the condition. For example, array_exists([0,1,2]) return 1, and array_exists([0,0])return 0. You can pass a lambda function to it as the first argument to customize the condition check, such as array_exists(x->x%2=0,[2,3,4]) to check whether any element in the array is even. It returns 1. To check whether all elements meet the condition, use array_all ### array_filter​ array_filter(func, array) returns an array containing only the element that matches the condition of the specified function. e.g. array_filter(x->x%2=0, [1,2,3,4])returns [2,4] ### array_first​ array_first(func, array) returns the first element that matches the condition of the specified function. e.g. array_first(x->x%2=0, [1,2,3,4])returns 2. ### array_first_index​ array_first_index(func, array) returns the index of the first element that matches the condition of the specified function. e.g. array_first_index(x->x%2=0, [1,2,3,4])returns 2. ### array_last​ array_last(func, array) returns the last element that matches the condition of the specified function. e.g. array_last(x->x%2=0, [1,2,3,4])returns 4. If nothing is found, it returns 0. ### array_last_index​ array_last_index(func, array) returns the index of the last element that matches the condition of the specified function. e.g. array_last_index(x->x%2=0, [1,2,3,4])returns 4. If nothing is found, it returns 0. ### array_map​ array_map(func, array) applies the function to every element in the array and returns a new array. e.g. array_map(x->x*x,[1,2])returns [1,4] ### array_max​ Overload 1: array_max(array) returns the maximum value in the array. e.g. array_max([1,2,3]) returns 3 Overload 2: array_max(func, array) apply the function to every element in the array and then returns the maximum value e.g. array_max(x->x*x,[1,2])returns 4 ### array_min​ Overload 1: array_min(array) returns the minimum value in the array. e.g. array_min([1,2,3]) returns 1 Overload 2: array_min(func, array) apply the function to every element in the array and then returns the minimum value e.g. array_min(x->x*x,[1,2])returns 1 ### array_sort​ array_sort([func,] array) sorts the array elements in ascending order. e.g. array_sort([3,2,5,4]) returns [2,3,4,5]. You can pass a lambda function to it as the first argument to apply the function before the sort, e.g. array_sort(x->-x,[3,2,5,4])returns [5,4,3,2] ### array_sum​ array_sum([func,] array) returns the sum value in the array. For example, array_sum([2,6]) return 8. You can pass a lambda function to it as the first argument to apply on each element before calculating the sum, such as array_sum(x->x*x,[2,6]) to get the sum for 2*2 and 6*6, which is 40. ### array_fold​ array_fold(lambda_function, arr1, arr2, ..., accumulator) applies a lambda function to one or more equally-sized arrays and collects the result in an accumulator. For example, array_fold(acc, x -> acc + (x * 2), [1, 2, 3, 4], to_int64(3)) will initialize the accumulator acc as value 3, then for each element in the [1, 2, 3, 4] array, assign it to x and accumulate them together with x*2. So the end result is 3+12+22+32+42=23. Make sure the return type of lambda function must be the same as the accumulator type. ## Maps​ ### map[key]​ You can easily access any element in the map, just using mapName[keyName], such as kv['key1'] ### map_cast​ map_cast(array1, array2) to generate a map with keys from array1 and values from array2 (these 2 arrays should be the same size). For example map_cast(['k1','k2'],[91,95]) will get {'k1':91,'k2':95} Alternatively, you can use map_cast(key1,value1,key2,value2..) Since Proton v1.4.2, a new extract_key_value_pairs function is added to extract key value pairs from a string to a map. ### map_from_arrays​ Alias for the above map_cast function. ## Tuples​ ### tuple_cast​ tuple_cast(item1,item2) to generate a tuple with these 2 elements. You can also use the shortcut syntax: (item1,item2) to create the tuple directly. ### untuple​ untuple(a_tuple) show elements in the tuple. The names of the result columns are implementation-specific and subject to change. Do not assume specific column names after untuple. ### tuple_element​ ... (content truncated) ## Functions_for_datetime - [Process Date and Time | Timeplus](https://docs.timeplus.com/functions_for_datetime): year - SQL Reference - Functions - Regular Functions - Process Date and Time # Process Date and Time ### year​ year(date) returns the year, for example year(today()) will return the current year. ### quarter​ quarter(date) returns the quarter, for example quarter(today()) will be 1 if it's currently in Q1. ### month​ month(date) returns the month, for example month(today()) will be 2 if it's currently Feb. ### day​ day(date) returns the day in the month. ### day_of_year​ day_of_year(date) returns the number of the day of the year (1-365, or 1-366 in a leap year). ### day_of_week​ day_of_week(date) returns the day of the week. e.g. Monday is 1, Sunday is 7. ### hour​ hour(datetime) returns the hour of the datetime. ### minute​ minute(datetime) returns the minute of the datetime. ### second​ second(datetime) returns the second of the datetime. ### from_unix_timestamp​ Convert a Unix timestamp number to a datetime value. For example from_unix_timestamp(1644272032). ### from_unix_timestamp64_milli​ Convert a Unix timestamp number to a datetime64(3) value. For example from_unix_timestamp64_milli(1712982826540). ### from_unix_timestamp64_micro​ Convert a Unix timestamp number to a datetime64(6) value. For example from_unix_timestamp64_micro(1712982905267202). ### from_unix_timestamp64_nano​ Convert a Unix timestamp number to a datetime64(9) value. For example from_unix_timestamp64_nano(1712983042242306000). ### to_unix_timestamp​ Returns the UNIX timestamp of the datetime, a number in uint32 For example to_unix_timestamp(now()) returns 1644272032 ### to_unix_timestamp64_milli​ Returns the UNIX timestamp with millisecond of the datetime64, a number in int64 For example to_unix_timestamp64_milli(now64()) returns 1712982826540 ### to_unix_timestamp64_micro​ Returns the UNIX timestamp with microsecond of the datetime64, a number in int64 For example to_unix_timestamp64_micro(now64(9)) returns 1712982905267202 ### to_unix_timestamp64_nano​ Returns the UNIX timestamp with nanosecond of the datetime64, a number in int64 For example to_unix_timestamp64_nano(now64(9)) returns 1712983042242306000 ### to_start_of_year​ to_start_of_year(date) rounds down a date or date with time to the first day of the year. Returns the date. ### to_start_of_quarter​ to_start_of_quarter(date) rounds down a date or date with time to the first day of the quarter. Returns the date. ### to_start_of_month​ to_start_of_month(date) rounds down a date or date with time to the first day of the month. Returns the date. ### to_start_of_week​ to_start_of_week(date) rounds down a date or date with time to the first day of the week. Returns the date. ### to_start_of_day​ to_start_of_day(date) rounds down a date with time to the start of the day. ### to_start_of_hour​ to_start_of_hour(datetime) rounds down a date or date with time to the start of the hour. ### to_start_of_minute​ to_start_of_minute(datetime) rounds down a date or date with time to the start of the minute. ### to_start_of_second​ to_start_of_second(datetime64) rounds down a date or date with time to the start of the second. Unlike other to_start_of_ functions, this function expects a datetime with millisecond, such as to_start_of_second(now64()) ### to_date​ to_date(string) converts a date string to a date type, e.g. to_date('1953-11-02') It can parse string 2023-09-19 05:31:34 but not 2023-09-19T05:31:34Z. Please use to_time function. ### to_datetime​ to_datetime(value) converts the value to a datetime type, e.g. to_datetime(1655265661) or to_datetime(today()) It can parse string 2023-09-19 05:31:34 but not 2023-09-19T05:31:34Z. Please use to_time function. ### to_time​ Please refer to to_time ### today​ today() returns the current date. ### yesterday​ yesterday() returns yesterday as date type. ### to_YYYYMM​ to_YYYYMM(date) returns a number. For example to_YYYYMM(today()) will return the number 202202 ### to_YYYYMMDD​ to_YYYYMMDD(date) returns a number. ### to_YYYYMMDDhhmmss​ to_YYYYMMDDhhmmss(date) returns a number. ### to_timezone​ to_timezone(datetime_in_a_timezone,target_timezone) converts the datetime from one timezone to the other. For the full list of possible timezones, please check "TZ database name" column in the wikipedia page. For the most common timezones, please check to_time For example, Output: ### format_datetime​ format_datetime(time,format,timezone) formats the datetime as a string. The 3rd argument is optional. The following placeholders are supported ### parse_datetime​ Coverts a string to a datetime. This function is the opposite operation of function format_datetime. parse_datetime(str, format [,timezone]). For example select parse_datetime('2021-01-04+23:00:00', '%Y-%m-%d+%H:%i:%s') returns '2021-01-04 23:00:00'. ### parse_datetime_in_joda_syntax​ Similar to parse_datetime, except that the format string is in Joda instead of MySQL syntax. parse_datetime_in_joda_syntax(str, format [,timezone]). For example select parse_datetime_in_joda_syntax('2023-02-24 14:53:31', 'yyyy-MM-dd HH:mm:ss') returns '2023-02-24 14:53:31'. ### date_diff​ date_diff(unit,begin,end) calculates the difference between begin and end and produce a number in unit. For example date_diff('second',window_start,window_end) Supported unit: - us: for microseconds. 1 second = 1,000,000 us - ms: for milliseconds. 1 second = 1,000 ms - s: for seconds - m: for minutes - h: for hours - d: for days ### date_diff_within​ date_diff_within(timegap,time1, time2) returns true or false. This function only works in stream-to-stream join. Check whether the gap between time1 and time2 are within the specific range. For example date_diff_within(10s,payment.time,notification.time) to check whether the payment time and notification time are within 10 seconds or less. ### date_trunc​ date_trunc(unit, value[, timezone]) truncates date and time data to the specified part of date. For example, date_trunc('month',now()) returns the datetime at the beginning of the current month. Possible unit values are: - year - quarter - month - day - hour - minute - second ### date_add​ It supports both date_add(unit, value, date) and a shortcut solution data_add(date,timeExpression) - date_add(HOUR, 2, now()) will get a new datetime in 2 hours, while date_add(HOUR, -2, now()) will get a new datetime 2 hours back. - date_add(now(),2h) and date_add(now(),-2h) also work ### date_sub​ It supports both date_sub(unit, value, date) and a shortcut solution data_sub(date,timeExpression) - date_sub(HOUR, 2, now()) will get a new datetime 2 hours back - date_sub(now(),2h) also work ### earliest_timestamp​ earliest_timestamp() returns "1970-1-1 00:00:00" ### earliest_ts​ earliest_ts() is a shortcut for earliest_timestamp() - year - quarter - month - day - day_of_year - day_of_week - hour - minute - second - from_unix_timestamp - from_unix_timestamp64_milli - from_unix_timestamp64_micro - from_unix_timestamp64_nano - to_unix_timestamp - to_unix_timestamp64_milli - to_unix_timestamp64_micro - to_unix_timestamp64_nano - to_start_of_year - to_start_of_quarter - to_start_of_month - to_start_of_week - to_start_of_day - to_start_of_hour - to_start_of_minute - to_start_of_second - to_date - to_datetime - to_time - today - yesterday - to_YYYYMM - to_YYYYMMDD - to_YYYYMMDDhhmmss - to_timezone - format_datetime - parse_datetime - parse_datetime_in_joda_syntax - date_diff - date_diff_within - date_trunc - date_add - date_sub - earliest_timestamp - earliest_ts ## Functions_for_dict - [Dictionaries | Timeplus](https://docs.timeplus.com/functions_for_dict): Functions for working with dictionaries. - SQL Reference - Functions - Regular Functions - Dictionaries # Dictionaries Functions for working with dictionaries. For dictionaries created with DDL queries, the dict_name parameter must be fully specified, like database.dict_name. Otherwise, the current database is used. ### dict_get​ dict_get(dict_name, attr_names, id_expr) returns the value(s) from the dictionary. For example dict_get('taxi_zone_dictionary', 'Borough', 132). Arguments: - dict_name - the name of the dictionary. If the dictionary is in a different database, you must specify the database name. - attr_names - the name of the attribute(s) to return. If multiple attributes are specified as a tuple, the function returns a tuple. For example dict_get('taxi_zone_dictionary', ('Borough', 'Zone'), 132) returns a named tuple, such as ('Queens','JFK Airport'). - id_expr - the key value to look up in the dictionary. If you define the dictionary with a composite key, you can pass a tuple. Timeplus returns the value in the same type as the specified dictionary attribute. For example, 'Borough' column is a string, so the function returns a string. If the key is not found, the function returns an empty string. ### dict_get_or_default​ dict_get_or_default(dict_name, attr_names, id_expr, default_value) returns the value(s) from the dictionary. The only difference from dict_get is that if the key is not found, the function returns the default value. The default_value should be of the same type as the dictionary attribute. If it's not, Timeplus will try to convert it to the attribute type. For example: ### dict_get_or_null​ dict_get_or_null(dict_name, attr_names, id_expr) returns the value(s) from the dictionary. The only difference from dict_get is that if the key is not found, the function returns null, instead of an empty string or 0, depending on the attribute type. For example: ### dict_has​ dict_has(dict_name, id_expr) returns 1 if the key exists in the dictionary, otherwise 0. - dict_get - dict_get_or_default - dict_get_or_null - dict_has ## Functions_for_fin - [Financial | Timeplus](https://docs.timeplus.com/functions_for_fin): xirr - SQL Reference - Functions - Regular Functions - Financial # Financial ### xirr​ Calculates the internal rate of return of an investment based on a specified series of potentially irregularly spaced cash flows. xirr(cashflow_column,date_column [, rate_guess]) - xirr ## Functions_for_geo - [Geo Location | Timeplus](https://docs.timeplus.com/functions_for_geo): pointinpolygon - SQL Reference - Functions - Regular Functions - Geo Location # Geo Location ### point_in_polygon​ Checks whether the point belongs to the polygon. point_in_polygon((x,y),[(a,b),(c,d)..]) e.g. SELECT point_in_polygon((3., 3.), [(6, 0), (8, 4), (5, 8), (0, 2)]) AS res returns 1 since the point (3,3) is in the defined polygon. ### geo_distance​ Calculates the distance on WGS-84 ellipsoid. geo_distance(lon1,lat1,lon2,lat2) - point_in_polygon - geo_distance ## Functions_for_hash - [Hash | Timeplus](https://docs.timeplus.com/functions_for_hash): Hash functions can be used for the deterministic pseudo-random shuffling of elements. - SQL Reference - Functions - Regular Functions - Hash # Hash Hash functions can be used for the deterministic pseudo-random shuffling of elements. ### md5​ md5(string) Calculates the MD5 from a string and returns the resulting set of bytes as fixed_string(16). If you want to get the same result as output by the md5sum utility, use lower(hex(md5(s))). ### md4​ md4(string) Calculates the MD4 from a string and returns the resulting set of bytes as fixed_string(16). ### half_md5​ half_md5(par1,..) Interprets all the input arguments strings and calculates the MD5 hash value for each of them. Then combines the MD5 hash values, takes the first 8 bytes of the result, and interprets them as uint64 in big-endian order. ### weak_hash32​ weak_hash32(data) Calculates a uint32 from any data type. ### kostik_consistent_hash​ kostikConsistentHash(input, n) An O(1) time and space consistent hash algorithm by Konstantin 'kostik' Oblakov. ### xx_hash32​ xx_hash32(str) Calculates xxHash from a string. ### xx_hash64​ xx_hash64(str) Calculates xxHash from a string. ### xxh3​ xxh3(expr) Produces a 64-bit xxh3 hash value. - md5 - md4 - half_md5 - weak_hash32 - kostik_consistent_hash - xx_hash32 - xx_hash64 - xxh3 ## Functions_for_json - [Process JSON | Timeplus](https://docs.timeplus.com/functions_for_json): Various functions are provided to extract values from JSON documents. You can also use the shortcut :true,"b"1}}. Then you can convert it to other data types using to_int() or ::int shortcut. - SQL Reference - Functions - Regular Functions - Process JSON # Process JSON Various functions are provided to extract values from JSON documents. You can also use the shortcut :: to extract the string value for specified JSON path, e.g. raw::b.c to get value "1" from {"a":true,"b":{"c":1}}. Then you can convert it to other data types using to_int() or ::int shortcut. ### json_extract_int​ json_extract_int(json, key) to get the integer value from the specified JSON document and key. For example json_extract_int('{"a":10,"b":3.13}','a') will get the number 10. You can also use the shortcut col:a::int. ### json_extract_float​ json_extract_float(json, key) to get the float value from the specified JSON document and key. For example json_extract_int('{"a":10,"b":3.13}','b') will get the float value 3.13. You can also use the shortcut col:a::float. ### json_extract_bool​ json_extract_bool(json, key) to get the boolean value from the specified JSON document and key. For example json_extract_bool('{"a":true}','a') will get the boolean value true or 1. You can also use the shortcut col:a::bool. ### json_extract_string​ json_extract_string(json, key)to get the string value from the specified JSON document and key. For example json_extract_string('{"a":true,"b":{"c":1}}','b') will get the string value {"c":1} and you can keep applying JSON functions to extract the values. You can also use the shortcut col:b to get the string value, or col:b.c::int to get the nested value. ### json_extract_array​ json_extract_array(json, key)to get the array value from the specified JSON document and key. For example json_extract_array('{"a": "hello", "b": [-100, 200.0, "hello"]}', 'b') will get the array value ['-100','200','"hello"'] If the entire JSON document is an array, the 2nd parameter key can be omitted to turn the json string as an array, e.g. json_extract_array(arrayString). You can also use the shortcut col:b[*]. ### json_extract_keys​ json_extract_keys(jsonStr) to parse the JSON string and extract the keys. e.g. select '{"system_diskio_name":"nvme0n1"}' as tags,json_extract_keys(tags) will get an array: [ "system_diskio_name" ] ### is_valid_json​ is_valid_json(str) to check whether the given string is a valid JSON or not. Return true(1) or false(0) ### json_has​ json_has(json, key) to check whether a specified key exists in the JSON document. For example json_has('{"a":10,"b":20}','c')returns 0(false). ### json_value​ json_value(json, path) allows you to access the nested JSON objects. For example, json_value('{"a":true,"b":{"c":1}}','$.b.c') will return the number 1 ### json_query​ json_query(json, path) allows you to access the nested JSON objects as JSON array or JSON object. If the value doesn't exist, an empty string will be returned. For example, json_query('{"a":true,"b":{"c":1}}','$.b.c') will return an array with 1 element [1] In a more complex example, json_query('{"records":[{"b":{"c":1}},{"b":{"c":2}},{"b":{"c":3}}]}','$.records[*].b.c') will get [1,2,3] ### json_encode​ This takes one or more parameters and return a json string. You can also turn all column values in the row as a json string via json_encode(*). ### json_cast​ This takes one or more parameters and return a json object. You can also turn all column values in the row as a json object via json_cast(*). ### json_array_length​ Get the length of the JSON array. For example, json_array_length('[3,4,5]') will return 3. ### json_merge_patch​ Merge multiple JSON documents into one. For example, json_merge_patch('{"a":1,"b":2}', '{"b":3,"c":4}') will return {"a":1,"b":3,"c":4}. If the key exists in both documents, the value from the second document will overwrite the first one. - json_extract_int - json_extract_float - json_extract_bool - json_extract_string - json_extract_array - json_extract_keys - is_valid_json - json_has - json_value - json_query - json_encode - json_cast - json_array_length - json_merge_patch ## Functions_for_logic - [Logic | Timeplus](https://docs.timeplus.com/functions_for_logic): if - SQL Reference - Functions - Regular Functions - Logic # Logic ### if​ if(condition,yesValue,noValue) Controls conditional branching. For example if(1=2,'a','b') will get b ### multi_if​ multi_if(condition1, then1, condition2, then2.. ,else) An easier way to write if/self or case/when. ### case_with_expression​ case_with_expression(expression, condition1, value1, condition2, value2, ..., default_value) Timeplus supports CASE WHEN .. THEN .. ELSE .. END operator. It is used to evaluate a series of conditions and return a value based on the first condition that evaluates to true. Timeplus supports two forms: - CASE WHEN .. THEN .. ELSE .. END - CASE .. WHEN .. THEN .. ELSE .. END For example: ### and​ and(val1, val2...) Calculates the logical conjunction of two or more values. ### or​ or(val1, val2...) Calculates the logical disjunction of two or more values. ### not​ not(val) Calculates the logical negation of a value. ### xor​ xor(val1, val2...) Calculates the logical exclusive disjunction of two or more values. For more than two input values, the function first xor-s the first two values, then xor-s the result with the third value etc. ### sleep​ sleep(seconds) Not necessary as a logic function, but it is a function that can be used to pause the execution for a specified number of seconds. For example select sleep(2) will pause the execution for 2 seconds. - if - multi_if - case_with_expression - and - or - not - xor - sleep ## Functions_for_math - [Math | Timeplus](https://docs.timeplus.com/functions_for_math): abs - SQL Reference - Functions - Regular Functions - Math # Math ### abs​ abs(value) returns the absolute value of the number. ### round​ round(x [,N]) Rounds a value to a specified number of decimal places. - If N is omitted, we consider N as 0 and the function rounds the value to the near integer, e.g. round(3.14)as 3 - If N >0, the function rounds the value to the right of the decimal point, e.g. round(3.14,1) as 3.1 - If N <0, the function rounds the value to the left of the decimal point. e.g. round(314.15,-2) as 300 ### e​ e() returns a float number that is close to the number e ### pi​ pi() returns a float number that is close to the number π ### exp​ exp(x) returns a float number that is close to the exponent of the argument x. ### exp2​ exp2(x) returns a float number that is close to 2 to the power of x. ### exp10​ exp10(x) returns a float number that is close to 10 to the power of x. ### log​ log(x) returns a float number that is close to the natural logarithm of the argument x. ### log2​ log2(x) returns a float number that is close to the binary logarithm of the argument x. ### log10​ log10(x) returns a float number that is close to the decimal logarithm of the argument x. ### sqrt​ sqrt(x) returns a float number that is close to the square root of the argument x. ### cbrt​ cbrt(x) returns a float number that is close to the cubic root of the argument x. ### lgamma​ lgamma(x) the logarithm of the gamma function. ### tgamma​ tgamma(x)the gamma function ### sin​ sin(x) the sine ### cos​ cos(x) the cosine ### tan​ tan(x) the tangent ### asin​ asin(x) the arc sine ### acos​ acos(x) the arc cosine ### atan​ atan(x) the arc tangent ### pow​ pow(x,y) returns a float number that is close to x to the power of y ### power​ power(x,y) returns a float number that is close to x to the power of y ### sign​ sign(x) returns the sign of the number x. If x<0, return -1. If x>0, return 1. Otherwise, return 0. ### degrees​ degrees(x) converts the input value in radians to degrees. E.g. degress(3.14) returns 180. ### radians​ radians(x) converts the input value in degrees to radians . E.g. radians(180) returns 3.14. ### is_finite​ is_finite(x) return 1 when the value x is not infinite and not a NaN, otherwise return 0. ### is_infinite​ is_infinite(x) to return 1 when the value x is infinite, otherwise return 0. ### is_nan​ is_nan(x) to return 1 if the x is Not-a-Number(NaN), otherwise return 0. - abs - round - e - pi - exp - exp2 - exp10 - log - log2 - log10 - sqrt - cbrt - lgamma - tgamma - sin - cos - tan - asin - acos - atan - pow - power - sign - degrees - radians - is_finite - is_infinite - is_nan ## Functions_for_random - [Random Data Generation | Timeplus](https://docs.timeplus.com/functions_for_random): You can use the following functions to generate random data in Timeplus. - SQL Reference - Functions - Regular Functions - Random Data Generation # Random Data Generation You can use the following functions to generate random data in Timeplus. ## Numbers​ ### rand​ rand() or rand32() Generates a random uint32 number. ### rand64​ rand64() Generates a random uint64 number. ### rand_uniform​ rand_uniform(min, max) Generates a random float64 drawn uniformly from interval [min, max]. ### rand_normal​ rand_normal(mean, variance) Generates a random float64 drawn uniformly from a [normal distribution])(https://en.wikipedia.org/wiki/Normal_distribution). ### rand_log_normal​ rand_log_normal(mean, variance) Generates a random float64 drawn uniformly from a [log-normal distribution])(https://en.wikipedia.org/wiki/Log-normal_distribution). ## Strings​ ### random_printable_ascii​ random_printable_ascii(length) Generates a string with a random set of ASCII characters. All characters are printable. ### random_string​ random_string(length) Generates a string of the specified length filled with random bytes (including zero bytes). Not all characters may be printable. ### random_fixed_string​ random_fixed_string(length) Generates a binary string of the specified length filled with random bytes (including zero bytes). Not all characters may be printable. ## Any date type, any logic​ Timeplus also provides a random_in_type function to generate random value in any data type, with any custom logic. ### random_in_type​ This function takes at least 1 parameter, can be 2, can be 3. Syntax: random_in_type(datatype [,max_value] [,generator_lambda]) When there is only one parameter specific, it generates a random value in that type, such as - random_in_type('int') returns a random int. - random_in_type('uint') returns a random uint(unsigned integer, positive number) - random_in_type('string') returns a random string - random_in_type('date') returns a random date - random_in_type('decimal32(3)') returns a random decimal - random_in_type('uuid') returns a random uuid - random_in_type('ipv4') returns a random ipv4 - random_in_type('ipv6') returns a random ipv6 You can set the 2nd parameter as the max value, as large as UINT64_MAX. Then the function will generate a random value in that type, smaller than the 2nd parameter, such as - random_in_type('int',3) returns a random int, either 0, 1, or 2. - random_in_type('date',30) returns a random date since 1970-01-01, no later than 1970-01-30. The 2nd parameter can be a lambda function to customize the generation logic, such as - random_in_type('int',x -> to_int(2*x)) returns a random int multiply with 2 You can also specify the maximum value and lambda together, such as - random_in_type('date32', 3, x -> to_date('2023-9-1') + interval x day) returns a random date since 2023-09-01, for the first 3 days. ## Generate multiple rows​ You can combine random functions with numbers table function to generate data with multiple columns and multiple rows. ### numbers​ numbers(N) – Returns a table with the single number column (uint64) that contains integers from 0 to N-1. numbers(N, M) - Returns a table with the single number column (uint64) that contains integers from N to (N + M - 1). Similar to the system.numbers table, it can be used for testing and generating successive values, numbers(N, M) more efficient than system.numbers. The following queries are equivalent: And the following queries are equivalent: Examples: - Numbersrandrand64rand_uniformrand_normalrand_log_normal - rand - rand64 - rand_uniform - rand_normal - rand_log_normal - Stringsrandom_printable_asciirandom_stringrandom_fixed_string - random_printable_ascii - random_string - random_fixed_string - Any date type, any logicrandom_in_type - random_in_type - Generate multiple rowsnumbers - numbers - rand - rand64 - rand_uniform - rand_normal - rand_log_normal - random_printable_ascii - random_string - random_fixed_string - random_in_type - numbers ## Functions_for_streaming - [Streaming Processing Functions | Timeplus](https://docs.timeplus.com/functions_for_streaming): The following functions are supported in streaming query, but not all of them support historical query. Please check the tag like this. - SQL Reference - Functions - Streaming Processing Functions # Streaming Processing Functions The following functions are supported in streaming query, but not all of them support historical query. Please check the tag like this. ✅ streaming query 🚫 historical query ### table​ table(stream) turns the unbounded data stream as a bounded table, and query its historical data. For example, you may load the clickstream data from a Kafka topic into the clicks stream in Timeplus. By default, if you run SELECT .. FROM clicks .. This is a streaming query with unbounded data. The query will keep sending you new results whenever it's available. If you only need to analyze the past data, you can put the stream into the table function. Taking a count as an example: - running select count(*) from clicks will show latest count every 2 seconds and never ends, until the query is canceled by the user - running select count(*) from table(clicks) will return immediately with the row count for the historical data for this data stream. You can create views such as create view histrical_view as select * from table(stream_name), if you want to query the data in the table mode more than once. This may work well for static data, such as lookup information(city names and their zip code). Please note, the table function also works in other types of streams: - mutable streams: get the historical data of the mutable stream. - Kafka external stream: read the existing data in the topic. select count(*) from table(kafka_ext_stream) is optimized to efficiently get the number of current message count from the Kafka topic. - Pulsar external stream: read the existing data in the topic. - Timeplus external stream: read the existing data for a stream in a remote Timeplus. - Random stream: generate a block of random data. The number of rows in the block is predefined and subject to change. The current value is 65409. For testing or demonstration purpose, you can create a random stream with multiple columns and use the table function to generate random data at once. Learn more about Non-streaming queries. ### tumble​ tumble(stream [,timeCol], windowSize) Create a tumble window view for the data stream, for example tumble(iot,5s) will create windows for every 5 seconds for the data stream iot . The SQL must end with group by with either window_start or window_end or both. ✅ streaming query ✅ historical query ### hop​ hop(stream [,timeCol], step, windowSize) Create a hopping window view for the data stream, for example hop(iot,1s,5s) will create windows for every 5 seconds for the data stream iot and move the windows forward every second. The SQL must end with group by with either window_start or window_end or both. ✅ streaming query 🚫 historical query ### session​ session(stream [,timeCol], idle, [maxLength,] [startCondition,endCondition] ) Create dynamic windows based on the activities in the data stream. Parameters: - stream a data stream, a view, or a CTE/subquery - timeCol optional, by default it will be _tp_time (the event time for the record) - idle how long the events will be automatically split to 2 session windows - maxLength the max length of the session window. Optional. Default value is the 5 times of idle - [startCondition, endCondition]Optional. If specified, the session window will start when the startConditionis met and will close when endCondition is met. You can use [expression1, expression2]to indicate start and end events will be included in the session, or (expression1, expression2] to indicate the ending events will be included but not the starting events. For example, if the car keeps sending data when it's moving and stops sending data when it's parked or waiting for the traffic light - session(car_live_data, 1m) partition by cid will create session windows for each car with 1 minute idle time. Meaning if the car is not moved within one minute, the window will be closed and a new session window will be created for future events. If the car keeps moving for more than 5 minutes, different windows will be created (every 5 minutes), so that as analysts, you can get near real-time results, without waiting too long for the car to be stopped. - session(car_live_data, 1m, [speed>50,speed<50)) partition by cid create session windows to detect when the car is speeding. The first event with speed over 50 will be included, and the last event with speed lower than 50 will not be included in the session window. - session(access_log, 5m, [action='login',action='logout']) partition by uid create session windows when the user logins the system and logout. If there is no activity within 5 minutes, the window will be closed automatically. ✅ streaming query 🚫 historical query ### dedup​ dedup(stream, column1 [,otherColumns..] [liveInSecond,limit]) Apply the deduplication at the given data stream with the specified column(s). Rows with same column value will only show once (only the first row is selected and others are omitted.) liveInSecond specifies how long the keys will be kept in the memory/state. By default forever. But if you only want to avoid duplicating within a certain time period, say 2 minutes, you can set 120s, e.g. dedup(subquery,myId,120s) The last parameter limit is optional which is 100000 by default. It limits the max unique keys maintained in the query engine. If the limit reaches, the system will recycle the earliest keys to maintain this limit. You can cascade this table function like tumble(dedup(table(.... and so far the wrapping order must in this sequence : tumble/hop/session -> dedup -> table. ✅ streaming query ✅ historical query When you use dedup function together with table() function to get the latest status for events with same ID, you can consider ordering the data by _tp_time in the reverse way, so that the latest event for same ID is kept. e.g. Otherwise, if you run queries with dedup(table(my_stream),id) the earliest event with same ID will be processed first, ignoring the rest of the updated status. In many cases, this is not what you expect. ### lag​ lag( [, ][, ]): Work for both streaming query and historical query. If you omit the offset the last row will be compared. E.g. lag(total) to get the value of total from the last row. lag(total, 12) to get the value from 12 rows ago. lag(total, 12, 0) to use 0 as the default value if the specified row is not available. ✅ streaming query 🚫 historical query ### lags​ lags(, begin_offset, end_offset [, ]) similar to lag function but can get a list of values. e.g. lags(total,1,3) will return an array for the last 1, last 2 and last 3 values. Starting from Timeplus Enterprise v2.7, you can set begin_offset as 0 to get the value in the current row. ✅ streaming query 🚫 historical query ### date_diff_within​ date_diff_within(timegap,time1, time2) returns true or false. This function only works in Range Bidirectional Join. Check whether the gap between time1 and time2 are within the specific range. For example date_diff_within(10s,payment.time,notification.time) to check whether the payment time and notification time are within 10 seconds or less. ✅ streaming query 🚫 historical query ### lag_behind​ lag_behind(offset) or lag_behind(offset,, ) It is designed for streaming JOIN. If you don't specify the column names, then it will use the processing time on the left stream and right stream to compare the timestamp difference. Example: ✅ streaming query 🚫 historical query ### latest​ latest() gets the latest value for a specific column, working with streaming aggregation with group by. ✅ streaming query 🚫 historical query ### earliest​ earliest() gets the earliest value for a specific column, working with streaming aggregation with group by. ✅ streaming query 🚫 historical query ### now​ now() Show the current date time, such as 2022-01-28 05:08:16 If the now() is used in a streaming query, no matter SELECT or WHERE or tumble/hop window, it will reflect the current time when the row is projected. ✅ streaming query ✅ historical query ### now64​ Similar to now() but with extra millisecond information, such as 2022-01-28 05:08:22.680 It can be also used in streaming queries to show the latest datetime with milliseconds. ✅ streaming query ✅ historical query ### emit_version​ emit_version() to show an auto-increasing number for each emit of streaming query result. It only works with streaming aggregation, not tail or filter. For example, if you run select emit_version(),count(*) from car_live_data the query will emit results every 2 seconds and the first result will be with emit_version=0, the second result with emit_version=1. This function is particularly helpful when there are multiple rows in each emit result. For example, you can run a tumble window aggregation with a group by. All results from the same aggregation window will be in the same emit_version. You can then show a chart with all rows in the same aggregation window. ✅ streaming query 🚫 historical query ### changelog​ changelog(stream[, [key_col1[,key_col2,[..]],version_column], drop_late_rows]) to convert a stream (no matter append-only stream or versioned stream) to a changelog stream with given primary keys. ... (content truncated) ## Functions_for_text - [Process Text | Timeplus](https://docs.timeplus.com/functions_for_text): lower - SQL Reference - Functions - Regular Functions - Process Text # Process Text ### lower​ lower(str) Converts ASCII Latin symbols in a string to lowercase. ### upper​ upper(str) Converts ASCII Latin symbols in a string to uppercase. ### format​ format(template,args) Formatting constant pattern with the string listed in the arguments. For example, format('{} {}', 'Hello', 'World')gets Hello World ### concat​ concat(str1,str2 [,str3]) Combine 2 or more strings as a single string. For example, concat('95','%') to get 95%. You can also use || as the shortcut syntax, e.g. '95' || '%' Each parameter in this function needs to be a string. You can use to_string function to convert them, for example to_string(95) || '%' ### substr​ substr(str,index [,length]) Returns the substring of str from index (starting from 1). length is optional. ### substring​ substring(str,index [,length]) Alias of substr. ### starts_with​ starts_with(str,prefix) Determines whether a string starts with a specified prefix. ### ends_with​ ends_with(str,suffix) Determines whether a string ends with a specified suffix. ### trim​ trim(string) Removes all specified characters from the start or end of a string. By default removes all consecutive occurrences of common whitespace (ASCII character 32) from both ends of a string. ### split_by_string​ split_by_string(sep,string) Splits a string into substrings separated by a string. It uses a constant string sep of multiple characters as the separator. If the string sep is empty, it will split the string string into an array of single characters. For example split_by_string('b','abcbxby')will get an array with string ['a','c','x','y'] ### match​ match(string,pattern) determines whether the string matches the given regular expression. For example, to check whether the text contains a sensitive AWS ARN, you can run match(text,'arn:aws:kms:us-east-1:\d{12}:key/.{36}') ### multi_search_any​ multi_search_any(text, array) determines whether the text contains any of the strings from the given array. For example, to check whether the text contains any sensitive keywords, you can run multi_search_any(text,['password','token','secret']) ### replace_one​ replace_one(string,pattern,replacement) Replace pattern with the 3rd argument replacement in string. For example replace_one('abca','a','z') will get zbca ### replace​ replace(string,pattern,replacement) Replace pattern with the 3rd argument replacement in string. For example replace('aabc','a','z') will get zzbc ### replace_regex​ replace_regex(string,pattern,replacement) Replaces all occurrences of the pattern. This can be used to mask data, e.g. to hide the full phone number, you can run replace_regex('604-123-4567','(\\d{3})-(\\d{3})-(\\d{4})','\\1-***-****') to get 604-***-**** ### extract​ Process plain text with regular expression and extract the content. For example, extract('key1=value1, key2=value2','key1=(\\w+)'), this will get “value1”. If the log lines are put into a single text column, you can create a view with the extracted fields, e.g. ### extract_all_groups​ extract_all_groups(haystack, pattern) Matches all groups of the haystack string using the pattern regular expression. Returns an array of arrays, where the first array includes keys and the second array includes all values. ### extract_all_groups_horizontal​ extract_all_groups_horizontal(haystack, pattern) Matches all groups of the haystack string using the pattern regular expression. Returns an array of arrays, where the first array includes all fragments matching the first group, the second array matching the second group, etc. ### extract_key_value_pairs​ extract_key_value_pairs(string) Extract key value pairs from the string and return a map. For example, extract_key_value_pairs('name:neymar, age:31 team:psg,nationality:brazil') will return a map with keys: name, age, team, ad nationality. For the advanced usage of the function, please check the doc. ### grok​ grok(string,pattern) Extract value from plan text without using regular expression. e.g. SELECT grok('My name is Jack. I am 23 years old.','My name is %{DATA:name}. I am %{INT:age} years old.') as m will get {"name":"Jack","age":"23"} as the m. Please note all keys and values in the returned map are in string type. You can convert them to other type, e.g. (m['age'])::int. Learn more about the Grok patterns. ### coalesce​ coalesce(value1, value2,..) Checks from left to right whether NULL arguments were passed and returns the first non-NULL argument. If you get error messages related to Nullable type, e.g. "Nested type array(string) cannot be inside Nullable type", you can use this function to turn the data into non-NULL For example json_extract_array(coalesce(raw:payload, '')) ### hex​ hex(argument)Returns a string containing the argument’s hexadecimal representation. argument can be any type. ### unhex​ unhex(string) Performs the opposite operation of hex. It interprets each pair of hexadecimal digits (in the argument) as a number and converts it to the byte represented by the number. The return value is a binary string (BLOB). ### uuid​ Alias of generate_uuidv4. uuid() or uuid(x) Generates a universally unique identifier (UUIDv4) which is a 16-byte number used to identify records. In order to generate multiple UUID in one row, pass a parameter in each function call, such as SELECT uuid(1) as a, uuid(2) as b Otherwise if there is no parameter while calling multiple uuid functions in one SQL statement, the same UUID value will be returned. ### generate_uuidv4​ Generates a universally unique identifier (UUIDv4) which is a 16-byte number used to identify records. ### uuid7​ Alias of generate_uuidv7. ### generate_uuidv7​ generate_uuidv7() Generates a universally unique identifier (UUIDv7), which contains the current Unix timestamp in milliseconds (48 bits), followed by version "7" (4 bits), a counter (42 bit) to distinguish UUIDs within a millisecond (including a variant field "2", 2 bit), and a random field (32 bits). For any given timestamp (unix_ts_ms), the counter starts at a random value and is incremented by 1 for each new UUID until the timestamp changes. In case the counter overflows, the timestamp field is incremented by 1 and the counter is reset to a random new start value. ### generate_uuidv7_thread_monotonic​ Generates a UUID version 7. The generated UUID contains the current Unix timestamp in milliseconds (48 bits), as generate_uuidv7 but gives no guarantee on counter monotony across different simultaneous requests. Monotonicity within one timestamp is guaranteed only within the same thread calling this function to generate UUIDs. ### generate_uuidv7_non_monotonic​ Generates a UUID version 7. This function is the fastest but it gives no monotonicity guarantees within a timestamp. ### uuid_to_num​ uuid_to_num(uuid) Extract bytes from a UUID and convert them to a number. The result is a 128-bit integer, which is the same as the UUID in binary format. The argument needs to be a UUIDv4 or UUIDv7, not a string. If you get a string, use uuid_to_num(to_uuid(uuid_string)) or uuid_string_to_num(uuid_string). ### uuid_string_to_num​ uuid_string_to_num(string[, variant = 1]) Accepts string containing 36 characters in the format xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, and returns a fixed_stringtring(16) as its binary representation, with its format optionally specified by variant (Big-endian by default). ### uuidv7_to_datetime​ uuidv7_to_datetime(uuid [, timezone]) Returns the timestamp component of a UUID version 7. ### to_uuid​ to_uuid(string) Converts a string in the format xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx to a UUID. The string must be 36 characters long and contain only hexadecimal digits and hyphens. ### generate_ulid​ generate_ulid([x]) generate a ULID. If you just need to generate one ULID, no need to specify the parameter. But if in one SQL, you need to generate different ULID, you can set different value as the parameter to make sure the function call won't be cached. For example SELECT generate_ulid(1),generate_ulid(2). ### ulid_string_to_datetime​ ulid_string_to_datetime(ulid [, timezone]) extracts the timestamp from a ULID. ### base64_encode​ base64_encode(string) Encodes a string or fixed_string as base64. For example base64_encode('hello') returns aGVsbG8= ### base64_decode​ base64_decode(string) Decode a base64 string to a string. For example base64_decode('aGVsbG8=') returns hello ### base58_encode​ base58_encode(string) Encodes a string or fixed_string as base58 in the "Bitcoin" alphabet. For example base58_encode('hello') returns Cn8eVZg ### base58_decode​ base58_decode(string) Decode a base58 string to a string. For example base58_decode('Cn8eVZg') returns hello ### format_readable_quantity​ format_readable_quantity(number) Returns a rounded number with suffix (thousand, million, billion, etc.) as string. For example, format_readable_quantity(10036) returns "10.04 thousand". ### format_readable_size​ format_readable_size(number) Returns a rounded number with suffix (KiB, GiB, etc.) as string. For example, format_readable_size(10036) returns "9.80 KiB". - lower - upper - format - concat - substr - substring - starts_with - ends_with - trim - split_by_string - match - multi_search_any - replace_one - replace - replace_regex - extract - extract_all_groups - extract_all_groups_horizontal - extract_key_value_pairs - grok - coalesce - hex - unhex - uuid - generate_uuidv4 - uuid7 - generate_uuidv7 - generate_uuidv7_thread_monotonic - generate_uuidv7_non_monotonic - uuid_to_num - uuid_string_to_num - uuidv7_to_datetime - to_uuid - generate_ulid - ulid_string_to_datetime - base64_encode - base64_decode - base58_encode - base58_decode - format_readable_quantity - format_readable_size ## Functions_for_type - [Type Conversion | Timeplus](https://docs.timeplus.com/functions_for_type): to_time - SQL Reference - Functions - Regular Functions - Type Conversion # Type Conversion ### to_time​ to_time(time_string [, default_time_zone] [,defaultValue]) to convert the string to a datetime64 value For example to_time('1/2/22') or to_time('1/2/22','America/New_York') Some of the common time zones are: - UTC: same as GMT(Greenwich Mean Time) - EST: US Eastern Time - MST: US Mountain Time - PST8PDT: US Pacific Time - America/New_York: same as EST - America/Los_Angeles: same as PST8PDT - America/Vancouver: same as PST8PDT - Asia/Shanghai: same as PRC For the full list of possible time zones, please check the "TZ database name" column in the wikipedia page. You can also convert the time between time zones via to_timezone It can parse various date time formats, such as 2023-09-19 05:31:34, 2023-09-19T05:31:34Z, 2023-09-19T05:31:34.026Z, 2023-09-19T05:31:34.302+00:00. This function is the alias to parse_datetime_best_effort. ### to_int​ to_int(string) Convert a string to an integer. ### to_float​ to_float(string) Convert a string to a float number, e.g. to_float('3.1415926') ### to_decimal​ to_decimal(number_or_string, scale) For example to_decimal('3.1415926',2) to get 3.14 ### to_string​ Convert any data type to a string, so that you can do other string operations, such as concat ### to_bool​ Convert the value to a bool type. e.g. select to_bool(1), to_bool(true),to_bool(True),to_bool('true') all return true. Please note you cannot run to_bool('True') ### cast​ Convert an input value to the specified data type. Three syntax variants are supported: - cast(x, T) - cast(x as t) - x::t While - x — A value to convert. Can be of any type. - T — The name of the target data type. String. - t — The target data type For example ### to_type_name​ to_type_name(x) to show the type name of the argument x. This is mainly for troubleshooting purposes to understand the date type for a function call. - to_time - to_int - to_float - to_decimal - to_string - to_bool - cast - to_type_name ## Functions_for_url - [Process URL and IP | Timeplus](https://docs.timeplus.com/functions_for_url): protocol - SQL Reference - Functions - Regular Functions - Process URL and IP # Process URL and IP ### protocol​ protocol(url) Extracts the protocol from a URL. For example protocol('https://docs.timeplus.com/functions')will return the string https ### domain​ domain(url) Extracts the domain from a URL. For example domain('https://docs.timeplus.com/functions')will return the string docs.timeplus.com ### port​ port(url) Extracts the port from a URL. If port is missing in the URL, it returns 0. For example port('https://docs.timeplus.com/functions') will return the integer 0 ### path​ path(url) Extracts the path from a URL, without the query string or fragment. For example path('https://docs.timeplus.com/functions') will return the string /functions ### path_all​ path_all(url) Extracts the path from a URL, including the query string or fragment. For example path_full('https://docs.timeplus.com/functions_for_logic#if') will return the string /functions_for_logic#if ### fragment​ fragment(url) Extracts the fragment from a URL. If there is no fragment, return an empty string. For example fragment('https://docs.timeplus.com/functions_for_logic#if') will return the string if ### query_string​ query_string(url) Extracts the query string from a URL. If there is no query string, return an empty string. For example query_string('https://a.com?k=v&key=value') will return the string k=v&key=value ### decode_url_component​ decode_url_component(url) Returns the decoded URL. ### encode_url_component​ encode_url_component(url) Returns the encoded URL. ### ipv4_num_to_string​ ipv4_num_to_string(ip) takes a ipv4 or uint32 value and returns a string containing the corresponding IPv4 address in the format A.B.C.d (dot-separated numbers in decimal form). For example. select ipv4_num_to_string(1823216871) returns '108.172.20.231' ### ipv4_string_to_num​ ipv4_string_to_num(string) takes a string value and returns a uint32 value. If the IPv4 address has an invalid format, it throws exception. For example. select ipv4_string_to_num('108.172.20.231') returns 1823216871 ### to_ipv4​ Alias of ipv4_string_to_num ### ipv4_num_to_string_class_c​ ipv4_num_to_string_class_c(ip). Similar to ipv4_num_to_string(ip), but using xxx instead of the last octet. For example. select ipv4_num_to_string_class_c(1823216871) returns '108.172.20.xxx' Since using ‘xxx’ is highly unusual, this may be changed in the future. We recommend that you do not rely on the exact format of this fragment. ### ipv6_num_to_string​ ipv6_num_to_string(ip) takes a fixed_string(16) containing the IPv6 address in binary format. Returns a string containing this address in text format. IPv6-mapped IPv4 addresses are output in the format ::ffff:111.222.33.44. For example. select ipv6_num_to_string(to_fixed_string(unhex('2A0206B8000000000000000000000011'),16)) returns '2a02:6b8::11' ### ipv6_string_to_num​ ipv6_string_to_num(string) takes a string value and returns a uint128 value. If the IPv6 address has an invalid format, it throws exception. If the input string contains a valid IPv4 address, returns its IPv6 equivalent. HEX can be uppercase or lowercase. For example. select hex(ipv6_string_to_num('2a02:2168:aaa:bbbb::2')) returns 2A0221680AAABBBB0000000000000002 select hex(ipv6_string_to_num('108.172.20.231')) returns 00000000000000000000FFFF6CAC14E7 ### to_ipv6​ Alias of ipv6_string_to_num ### ipv4_to_ipv6​ ipv4_to_ipv6(ip) Convert the ipv4 value to ipv6 For example: select ipv6_num_to_string(ipv4_to_ipv6(ipv4_string_to_num('192.168.0.1'))) returns '::ffff:192.168.0.1' ### ipv4_cidr_to_range​ ipv4_cidr_to_range(ipv4,number)Accepts an IPv4 and an uint8 value containing the CIDR. Return a tuple with two IPv4 containing the lower range and the higher range of the subnet. For example: select ipv4_cidr_to_range(to_ipv4('192.168.0.1'),16) returns ('192.168.0.0','192.168.255.255') ### ipv6_cidr_to_range​ ipv6_cidr_to_range(ipv6,number)Accepts an IPv6 and an uint128 value containing the CIDR. Return a tuple with two IPv6 containing the lower range and the higher range of the subnet. For example: select ipv6_cidr_to_range(to_ipv6('2001:0db8:0000:85a3:0000:0000:ac1f:8001'),32) returns ('2001:db8::','2001:db8:ffff:ffff:ffff:ffff:ffff:ffff') ### is_ipv4_string​ is_ipv4_string(string) returns 1 if it's an ipv4 string, otherwise 0. ### is_ipv6_string​ is_ipv6_string(string) returns 1 if it's an ipv6 string, otherwise 0. ### is_ip_address_in_range​ is_ip_address_in_range(address, prefix) Determines if an IP address is contained in a network represented in the CIDR notation. Returns true or false. For example, select is_ip_address_in_range('127.0.0.1', '127.0.0.0/8') returns true. select is_ip_address_in_range('127.0.0.1', 'ffff::/16') returns false ### geohash_encode​ geohash_encode(longitude, latitude, [precision]) Encodes latitude and longitude as a geohash string. Input values - longitude - longitude part of the coordinate you want to encode. Floating in range[-180°, 180°] - latitude - latitude part of the coordinate you want to encode. Floating in range [-90°, 90°] - precision - Optional, length of the resulting encoded string, defaults to 12. Integer in range [1, 12]. Any value less than 1 or greater than 12 is silently converted to 12. Returned values - alphanumeric String of encoded coordinate (modified version of the base32-encoding alphabet is used). Example Result ### geohash_decode​ Decodes any geohash-encoded string into longitude and latitude. Input values - encoded string - geohash-encoded string. Returned values - (longitude, latitude) - 2-tuple of float64 values of longitude and latitude. Example Result ### geohashes_in_box​ Returns an array of geohash-encoded strings of given precision that fall inside and intersect boundaries of given box, basically a 2D grid flattened into array. Syntax Arguments - longitude_min — Minimum longitude. Range: [-180°, 180°]. Type: float. - latitude_min — Minimum latitude. Range: [-90°, 90°]. Type: float. - longitude_max — Maximum longitude. Range: [-180°, 180°]. Type: float. - latitude_max — Maximum latitude. Range: [-90°, 90°]. Type: float. - precision — Geohash precision. Range: [1, 12]. Type: uint8. All coordinate parameters must be of the same type: either float32 or float64. Returned values - Array of precision-long strings of geohash-boxes covering provided area, you should not rely on order of items. - [] - Empty array if minimum latitude and longitude values aren’t less than corresponding maximum values. Type: Array(String). Function throws an exception if resulting array is over 10’000’000 items long. Example Query: Result: - protocol - domain - port - path - path_all - fragment - query_string - decode_url_component - encode_url_component - ipv4_num_to_string - ipv4_string_to_num - to_ipv4 - ipv4_num_to_string_class_c - ipv6_num_to_string - ipv6_string_to_num - to_ipv6 - ipv4_to_ipv6 - ipv4_cidr_to_range - ipv6_cidr_to_range - is_ipv4_string - is_ipv6_string - is_ip_address_in_range - geohash_encode - geohash_decode - geohashes_in_box ## Getting-help - [Getting Help | Timeplus](https://docs.timeplus.com/getting-help): If you have questions or get stuck, we are happy to help. - Getting Help # Getting Help If you have questions or get stuck, we are happy to help. ## Enterprise Support​ ### Create A Support Ticket​ Customers of Timeplus Enterprise can open support tickets by emailing support@timeplus.com or accessing the support web portal to open a new ticket. ### Diagnostic Bundle​ For self-hosted Timeplus Enterprise deployments, run timeplus diag from the timeplus/bin folder to collect diagnostic information when Timeplus Enterprise is running. A diagnostic file will be generated under the timeplus/logs directory. When creating a support ticket, you may package the logs directory as a zip and attach it in the support ticket. ## Community Support​ ### GitHub Discussions and Issues​ For Timeplus core engine (Proton), you can raise your questions on https://github.com/timeplus-io/proton/discussions, or report bugs or submit PR on https://github.com/timeplus-io/proton/issues. ### Community Slack​ Join our community Slack to chat with our Timeplus team and other users. Slack is a great place to ask questions, provide feedback, tell us what features you’d like to see, and learn more about using Timeplus. - Enterprise SupportCreate A Support TicketDiagnostic Bundle - Create A Support Ticket - Diagnostic Bundle - Community SupportGitHub Discussions and IssuesCommunity Slack - GitHub Discussions and Issues - Community Slack - Create A Support Ticket - Diagnostic Bundle - GitHub Discussions and Issues - Community Slack ## Glossary - [Key Terms and Concepts | Timeplus](https://docs.timeplus.com/glossary): This page lists key terms and concepts in Timeplus, from A to Z. - Overview - Key Terms and Concepts # Key Terms and Concepts This page lists key terms and concepts in Timeplus, from A to Z. ## bookmark​ Query bookmarks or SQL scripts, only available in Timeplus Enterprise, not in Timeplus Proton. You can save the common SQL statements as bookmarks. They can be run quickly in the web console by a single click. You can create, list, edit, remove bookmarks in the query page. Both bookmarks and views can help you easily re-run a query. However views are defined in the Timeplus SQL engine and you can query the view directly via select .. from .. But bookmarks are just UI shortcuts. When you click the bookmark, the original SQL statement will be pre-filled in the query console. You cannot run select .. from my_bookmark ## CTE​ A common table expression, or CTE, (in SQL) is a temporary named result set, derived from a simple query and defined within the execution scope of a SELECT, INSERT, UPDATE, or DELETE statement. CTEs can be thought of as alternatives to derived tables (subquery), views, and inline user-defined functions. ## dashboard​ Only available in Timeplus Enterprise, not in Timeplus Proton. You can create multiple dashboards, and add multiple charts to a dashboard. You can also add filters or Markdown (experimental). ## event time​ Each row in Timeplus streams contains a _tp_time column as the event time. Timeplus takes this attribute as one important identity of an event. Event time is used to identify when the event is generated, like a birthday to a human being. It can be the exact timestamp when the order is placed, when the user logins a system, when an error occurs, or when an IoT device reports its status. If there is no suitable timestamp attribute in the event, Timeplus will generate the event time based on the data ingestion time. By default, the _tp_time column is in datetime64(3, 'UTC') type with millisecond precision. You can also create it in datetime type with second precision. When you are about to create a new stream, please choose the right column as the event time. If no column is specified, then Timeplus will use the current timestamp as the value of _tp_time It's not recommended to rename a column as _tp_time at the query time, since it will lead to unexpected behaviour, specially for Time Travel. Event time is used almost everywhere in Timeplus data processing and analysis workflow: - while doing time window based aggregations, such as tumble or hop to get the downsampled data or outlier in each time window, Timeplus will use the event time to decide whether certain event belongs to a specific window - in such time sensitive analytics, event time is also used to identify out of order events or late events, and drop them in order to get timely streaming insights. - when one data stream is joined with the other, event time is the key to collate the data, without expecting two events to happen in exactly the same millisecond. - event time also plays an important role to device how long the data will be kept in the stream ### how to specify the event time​ #### Specify during data ingestion​ When you ingest data into Timeplus, you can specify an attribute in the data which best represents the event time. Even if the attribute is in String type, Timeplus will automatically convert it to a timestamp for further processing. If you don't choose an attribute in the wizard, then Timeplus will use the ingestion time to present the event time, i.e. when Timeplus receives the data. This may work well for most static or dimensional data, such as city names with zip codes. #### Specify during query​ The tumble or hop window functions take an optional parameter as the event time column. By default, we will use the event time in each data. However you can also specify a different column as the event time. Taking an example for taxi passengers. The data stream can be The data may come from a Kafka topic. When it's configured, we may set trip_end as the (default) event time. So that if we want to figure out how many passengers in each hour, we can run query like this This query uses trip_end , the default event time, to run the aggregation. If the passenger ends the trip on 00:01 at midnight, it will be included in the 00:00-00:59 time window. In some cases, you as the analyst, may want to focus on how many passengers get in the taxi, instead of leaving the taxi, in each hour, then you can set trip_start as the event time for the query via tumble(taxi_data,trip_start,1h) Full query: ## external stream​ You can create external streams to read or write data from/to external systems in the streaming way. Timeplus supports external streams to Apache Kafka, Apache Pulsar, and other streaming data platforms. Learn more: External Stream ## external table​ You can create external tables to read or write data from/to external systems in the non-streaming way. Timeplus supports external tables to ClickHouse, PostgreSQL, MySQL, etc. ## materialized view​ Real-time data pipelines are built via materialized views in Timeplus. The difference between a materialized view and a regular view is that the materialized view is running in background after creation and the resulting stream is physically written to internal storage (hence it's called materialized). Once the materialized view is created, Timeplus will run the query in the background continuously and incrementally emit the calculated results according to the semantics of its underlying streaming select. ## query​ Timeplus provides powerful streaming analytics capabilities through the enhanced SQL. By default, queries are unbounded and keep pushing the latest results to the client. The unbounded query can be converted to a bounded query by applying the function table(), when the user wants to ask the question about what has happened like the traditional SQL. Learn more: Streaming Query and Non-Streaming Query ## sink​ a.k.a. destination. Only available in Timeplus Enterprise, not in Timeplus Proton. Timeplus enables you to send real-time insights or transformed data to other systems, either to notify individuals or power up downstream applications. Learn more: Destination. ## source​ A source is a background job in Timeplus Enterprise to load data into a stream. For Kafka API compatible streaming data platform, you need to create Kafka external streams. Learn more: Data Collection ## stream​ Timeplus is a streaming analytics platform and data lives in streams. Timeplus streams are similar to tables in the traditional SQL databases. Both of them are essentially datasets. The key difference is that Timeplus stream is an append-only, unbounded, constantly changing events group. Learn more: Stream ## timestamp column​ When you create a source and preview the data, you can choose a column as the timestamp column. Timeplus will use this column as the event time and track the lifecycle of the event and process it for all time related computation/aggregation. ## view​ You can define reusable SQL statements as views, so that you can query them as if they are streams select .. from view1 .. By default, views don't take any extra computing or storage resources. They are expanded to the SQL definition when they are queried. You can also create materialized views to 'materialize' them (keeping running them in the background and saving the results to the disk). Learn more: View and Materialized View - bookmark - CTE - dashboard - event timehow to specify the event timeSpecify during data ingestionSpecify during query - how to specify the event timeSpecify during data ingestionSpecify during query - Specify during data ingestion - Specify during query - external stream - external table - materialized view - query - sink - source - stream - timestamp column - view - how to specify the event timeSpecify during data ingestionSpecify during query - Specify during data ingestion - Specify during query - Specify during data ingestion - Specify during query ## Grok - [Grok Patterns | Timeplus](https://docs.timeplus.com/grok): Timeplus supports grok function to extract structured data from unstructured text. - SQL Reference - Grok Patterns # Grok Patterns Timeplus supports grok function to extract structured data from unstructured text. Grok is a powerful tool for extracting structured data from unstructured text. Grok syntax is composed of reusable elements called Grok patterns that enable parsing for data such as timestamps, IP addresses, hostnames, log levels, and more. The prebuilt patterns make Grok easier to use than defining new regular expressions to extract structured data, especially for long text strings. ## Key Concepts for Using Grok​ Grok patterns follow the syntax: %{PATTERN:label}: - %{PATTERN}: the Grok pattern to match the text. - :label: the name of the field to assign the matched text. When this is omitted, the matched text won't be assigned to any field. This is useful for matching patterns without capturing the text. ## Examples of Using Grok​ ### A simple example​ Let's start with a simple example. Suppose you get a message like My name is Jack. I am 23 years old. and you want to extract the name and age. You can use the following Grok pattern: The last 2 columns will return "Jack Lee" and 23. ### Parsing timeplusd log​ Let's try to parse the log from timeplusd. Here is an example log: With the help of ChatGPT, you may use the following regular expression to extract the values But this is a bit hard to read and maintain. With Grok, you can use the following pattern: You setup data pipelines in Timeplus to parse the real-time logs with the following SQL: ## Grok Patterns​ Grok patterns are a set of predefined regular expressions that can be used to match text strings. They are defined in https://github.com/timeplus-io/proton/blob/develop/programs/server/grok-patterns. The following are some of the most commonly used Grok patterns: - Key Concepts for Using Grok - Examples of Using GrokA simple exampleParsing timeplusd log - A simple example - Parsing timeplusd log - Grok Patterns - A simple example - Parsing timeplusd log ## History - [Historical Data Processing | Timeplus](https://docs.timeplus.com/history): In addition to stream processing, Timeplus also store and serve for historical data, like many OLAP databases. By default, data are saved in Timeplus' columnar storage, with optional secondary indexes. For mutable streams, historical data are saved in row-based storage, for fast update and range queries. - Core Features - Stream Processing - Historical Data Processing # Historical Data Processing In addition to stream processing, Timeplus also store and serve for historical data, like many OLAP databases. By default, data are saved in Timeplus' columnar storage, with optional secondary indexes. For mutable streams, historical data are saved in row-based storage, for fast update and range queries. When you run SELECT .. FROM stream, it's in streaming mode by default, and the query is long-running. To run historical query and stop the query when results are returned, there are 3 ways to do it: ## query_mode='table'​ Run historical query by setting query_mode='table'. This mode is useful if there are multiple streams in the query and users like to do historical data processing as a whole in the query. ## table(stream)​ Run historical query per stream by wrapping stream with table function. This mode is flexible as you can query certain streams in the streaming way and other streams in batch way. For example enrich an append-only stream with a static dimension table join. ## 8123 or 7587 ports​ While using one of our SDK to connect to Timeplus, you can choose HTTP port 8123 or TCP port 7587. Timeplus will automatically turn on historical query mode while receiving queries in these 2 ports. Learn more. - query_mode='table' - table(stream) - 8123 or 7587 ports ## Http-external - [HTTP External Stream | Timeplus](https://docs.timeplus.com/http-external): Since Timeplus Enterprise v2.9 and v2.8.2, you can send data to HTTP endpoints via the HTTP External Stream. You can use this feature to trigger Slack notifications or send streaming data to downstream systems, such as Splunk, Datadog, Elasticsearch, Databricks, or any other HTTP-based service. - Core Features - External Streams & Tables - External Streams - HTTP Write # HTTP External Stream Since Timeplus Enterprise v2.9 and v2.8.2, you can send data to HTTP endpoints via the HTTP External Stream. You can use this feature to trigger Slack notifications or send streaming data to downstream systems, such as Splunk, Datadog, Elasticsearch, Databricks, or any other HTTP-based service. Currently, it only supports writing data to HTTP endpoints, but reading data from HTTP endpoints is not supported yet. ## CREATE EXTERNAL STREAM​ To create an external stream for HTTP endpoints, you can run the following DDL SQL: For the full list of settings, see the DDL Settings section. ### Examples​ #### Write to OpenSearch/ElasticSearch​ Assuming you have created an index students in a deployment of OpenSearch or ElasticSearch, you can create the following external stream to write data to the index. Then you can insert data via a materialized view or just #### Write to Splunk​ Follow the guide to set up and use HTTP Event Collector(HEC) in Splunk. Make sure you create a HEC token for the desired index and enable it. Create the HTTP external stream in Timeplus: Then you can insert data via a materialized view or just #### Write to Datadog​ Create or use an existing API key with the proper permission for sending data. Create the HTTP external stream in Timeplus: Then you can insert data via a materialized view or just #### Write to Algolia​ The Algolia Ingestion API accepts multiple rows in a single request in the following JSON payload: The objectID need to be unique for each record. Assuming the document contains 3 columns: firstname, lastname, and zip_code, you can create the HTTP external stream as this: Follow the documentation to create a "Push to Algolia" connector, and obtain the task ID, as well as application ID and API key. You can insert multiple rows to the algolia index via materialized views or INSERT command: #### Write to BigQuery​ Assume you have created a table in BigQuery with 2 columns: Follow the guide to choose the proper authentication to Google Cloud, such as via the gcloud CLI gcloud auth application-default print-access-token. Create the HTTP external stream in Timeplus: Replace the OAUTH_TOKEN with the output of gcloud auth application-default print-access-token or other secure way to obtain OAuth token. Replace PROJECT, DATASET and TABLE to match your BigQuery table path. Also change format_template_row_format to match the table schema. Then you can insert data via a materialized view or just via INSERT command: #### Write to Databricks​ Follow the guide to create an access token for your Databricks workspace. Assume you have created a table in Databricks SQL warehouse with 2 columns: Create the HTTP external stream in Timeplus: Replace the TOKEN, HOST, and WAREHOUSE_ID to match your Databricks settings. Also change format_template_row_format and format_template_row_format to match the table schema. Then you can insert data via a materialized view or just via INSERT command: This will insert one row per request. We plan to support batch insert and Databricks specific format to support different table schemas in the future. #### Trigger Slack Notifications​ You can follow the guide to configure an "incoming webhook" to send notifications to a Slack channel. Then you can insert data via a materialized view or just via INSERT command: Please follow Slack's text formats guide to add rich text to your messages. ### DDL Settings​ #### type​ The type of the external stream. The value must be http to send data to HTTP endpoints. #### config_file​ You can specify the path to a file that contains the configuration settings. The file should be in the format of key=value pairs, one pair per line. You can set the HTTP credentials or Authentication tokens in the file. Please follow the example in Kafka External Stream. #### url​ The endpoint of the HTTP service. Different services and different use cases may have different endpoints. For example, to send data to a specified OpenSearch index, you can use http://host:port/my_index/_bulk. To send data to multiple indexes (depending on the column in the streaming SQL), you can use http://host:port/_bulk and also specify the output_format_opensearch_index_column. #### data_format​ The data_format specifies how the HTTP POST body is constructed. We have built a special integration with OpenSearch or ElasticSearch, with data_format='OpenSearch' or data_format='ElasticSearch'. Other supported values for data_format are: - JSONEachRow: each row of the message as a single JSON document. The top level JSON key/value pairs will be parsed or constructed as the columns. If you need the POST body to be a JSON array for multiple rows in a batch, also set output_format_json_array_of_rows=1 in the DDL. If you need each POST body represents a single row as a JSON object, set one_message_per_row=true. - CSV: less commonly used. - TSV: similar to CSV but tab as the separator - ProtobufSingle: for single Protobuf message per message - Protobuf: there could be multiple Protobuf messages in a single message. - Avro - RawBLOB: the default value. Read/write message as plain text. - Template: create the payload with customized templates #### format_template_resultset_format​ When you set date_format to Template, you need to specify both format_template_resultset_format and format_template_row_format, and optionally format_template_rows_between_delimiter. format_template_resultset_format controls how all the rows will be combined to create the POST payload, and contains the following placeholders: - data is the rows with data in format_template_row_format, separated by format_template_rows_between_delimiter. - totals is the row with total values. - min is the row with minimum values. - max is the row with maximum values. - rows_before_limit is the minimal number of rows there would have been without LIMIT. - time is the request execution time in seconds. - rows_read is the number of rows has been read. - bytes_read is the number of bytes (uncompressed) has been read. For example: - To send markdown messages to Slack, the template is '{"blocks":[{"type":"section","text":{"type":"mrkdwn","text":"${data}"}}]}' - To send rows to BigQuery, the template is '{"rows":[${data}]}' #### format_template_row_format​ The setting specifies format strings for rows with the following syntax: For example: - To send markdown messages to Slack, the template is '${text:Raw}' - To send rows to BigQuery, the template is '{"json":{"num":${num:JSON},"str":${str:JSON}}}' (replace the column names per your table schema) The following escaping rules are supported: #### format_template_rows_between_delimiter​ The setting format_template_rows_between_delimiter setting specifies the delimiter between rows, which is printed (or expected) after every row except the last one (\n by default) #### http_header_Authorization​ You can set key/value pairs in HTTP header, using the format http_header_key=value. For example to set the Authorization header for Splunk HEC, you can use http_header_Authorization=Splunk hec_token. #### output_format_json_array_of_rows​ If you need the POST body to be a JSON array for multiple rows in a batch, set output_format_json_array_of_rows=1, e.g. ## DROP EXTERNAL STREAM​ - CREATE EXTERNAL STREAMExamplesWrite to OpenSearch/ElasticSearchWrite to SplunkWrite to DatadogWrite to AlgoliaWrite to BigQueryWrite to DatabricksTrigger Slack NotificationsDDL Settingstypeconfig_fileurldata_formatformat_template_resultset_formatformat_template_row_formatformat_template_rows_between_delimiterhttp_header_Authorizationoutput_format_json_array_of_rows - ExamplesWrite to OpenSearch/ElasticSearchWrite to SplunkWrite to DatadogWrite to AlgoliaWrite to BigQueryWrite to DatabricksTrigger Slack Notifications - Write to OpenSearch/ElasticSearch - Write to Splunk - Write to Datadog - Write to Algolia - Write to BigQuery - Write to Databricks - Trigger Slack Notifications - DDL Settingstypeconfig_fileurldata_formatformat_template_resultset_formatformat_template_row_formatformat_template_rows_between_delimiterhttp_header_Authorizationoutput_format_json_array_of_rows - type - config_file - url - data_format - format_template_resultset_format - format_template_row_format - format_template_rows_between_delimiter - http_header_Authorization - output_format_json_array_of_rows - DROP EXTERNAL STREAM - ExamplesWrite to OpenSearch/ElasticSearchWrite to SplunkWrite to DatadogWrite to AlgoliaWrite to BigQueryWrite to DatabricksTrigger Slack Notifications - Write to OpenSearch/ElasticSearch - Write to Splunk - Write to Datadog - Write to Algolia - Write to BigQuery - Write to Databricks - Trigger Slack Notifications - DDL Settingstypeconfig_fileurldata_formatformat_template_resultset_formatformat_template_row_formatformat_template_rows_between_delimiterhttp_header_Authorizationoutput_format_json_array_of_rows - type - config_file - url - data_format - format_template_resultset_format - format_template_row_format - format_template_rows_between_delimiter - http_header_Authorization - output_format_json_array_of_rows - Write to OpenSearch/ElasticSearch - Write to Splunk - Write to Datadog - Write to Algolia - Write to BigQuery - Write to Databricks - Trigger Slack Notifications - type - config_file - url - data_format - format_template_resultset_format - format_template_row_format - format_template_rows_between_delimiter - http_header_Authorization - output_format_json_array_of_rows ## Iceberg - [Iceberg Integration | Timeplus](https://docs.timeplus.com/iceberg): Apache Iceberg is an open table format for large-scale analytic datasets, designed for high performance and reliability. It provides an open, vendor-neutral solution that supports multiple engines, making it ideal for various analytics workloads. Initially, the Iceberg ecosystem was primarily built around Java, but with the increasing adoption of the REST catalog specification, Timeplus is among the first vendors to integrate with Iceberg purely in C++. This allows Timeplus users to stream data to Iceberg with a high performance, low memory footprint, and easy installation without relying on Java dependencies. - Core Features - External Streams & Tables - External Tables - Apache Iceberg # Iceberg Integration Apache Iceberg is an open table format for large-scale analytic datasets, designed for high performance and reliability. It provides an open, vendor-neutral solution that supports multiple engines, making it ideal for various analytics workloads. Initially, the Iceberg ecosystem was primarily built around Java, but with the increasing adoption of the REST catalog specification, Timeplus is among the first vendors to integrate with Iceberg purely in C++. This allows Timeplus users to stream data to Iceberg with a high performance, low memory footprint, and easy installation without relying on Java dependencies. Since Timeplus Proton 1.7(to be released soon) and Timeplus Enterprise 2.8, we provide native support for Apache Iceberg as a new database type. This allows you to read and write data using the Apache Iceberg open table format, with support for the Iceberg REST Catalog (IRC). In the initial release, we focused on writing data to Iceberg, with basic query optimization for reading data from Iceberg. The integration with Amazon S3, AWS Glue's Iceberg REST endpoint and the Apache Gravitino Iceberg REST Server are validated. More REST catalog implementations are planned. ## Key Benefits for Timeplus Iceberg Integration​ - Using Timeplus materialized views, users can continuously process and transform streaming data (from Apache Kafka for example) and write to the cost-effective object storage in Apache Iceberg open table format. - Apache Iceberg's open table format ensures you're never locked into a single vendor or query engine - Query your Iceberg tables with multiple engines including Timeplus, Apache Spark, Apache Flink, ClickHouse, DuckDB, and AWS Athena - Future-proof your data architecture with broad industry support and an active open-source community ## CREATE DATABASE​ To create an Iceberg database in Timeplus, use the following syntax: ### DDL Settings​ - type – Specifies the type of the database. Be sure to use iceberg for Iceberg tables. - catalog_uri – Specifies the URI of the Iceberg catalog. - catalog_type – Specifies the catalog type. Currently, only rest is supported in Timeplus. - warehouse – The Iceberg warehouse identifier where the table data is stored. - storage_endpoint – The S3-compatible endpoint where the data is stored. For AWS S3, use https://bucketname.s3.region.amazonaws.com. - rest_catalog_sigv4_enabled – Enables AWS SigV4 authentication for secure catalog communication. - rest_catalog_signing_region – AWS region used for signing the catalog requests. - rest_catalog_signing_name – The service name used in AWS SigV4 signing. - use_environment_credentials – Default to true, Timeplus will use environment-based credentials, useful for cases where Timeplus runs in an AWS EC2 instance with an assigned IAM role, or AWS credentials in environment variables as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. Setting this to false if you are using local minio or public S3 bucket. - credential – A unified credential (username:password format) that applies to both catalog and storage if they share the same authentication (e.g. AWS access key and secret key). - catalog_credential – If the catalog requires a separate credential, specify it here. - storage_credential – If the storage (e.g. S3) requires a different credential, specify it separately. ### Example: AWS Glue REST Catalog​ ### Example: AWS S3 Table REST Catalog​ If you want to create new Iceberg tables from Timeplus, you can also set storage_credential to 'https://s3tables.us-west-2.amazonaws.com/(bucket-name)'. ### Example: Apache Gravitino REST Catalog​ #### Gravitino Configuration​ Here is the sample configuration for Gravitino Iceberg REST Server 0.9.0: ## Creating and Writing to an Iceberg Table​ Once the Iceberg database is created in Timeplus, you can list existing tables in the database or create new table via Timeplus SQL: ## Writing to Iceberg via a Materialized View​ You can run INSERT INTO statements to write data to Iceberg tables, or set up a materialized view to continuously write data to Iceberg tables. ## Querying Iceberg Data with SparkSQL​ ### Using SQL in Timeplus​ You can query Iceberg data in Timeplus by: This will return all results and terminate the query. No streaming mode is supported for Iceberg tables yet. It's recommended to set LIMIT to a small value to avoid loading too much data from Iceberg to Timeplus. This query is optimized to return the count of rows in the specified Iceberg table with minimal scanning of metadata and data files. ### Using SparkSQL​ Depending on whether you setup the catalog via AWS Glue or Apache Gravitino, you can also start a SparkSQL session to query or insert data into Iceberg tables. ## DROP DATABASE​ Please note this won't delete the data in catalog or S3 storage. ## Limitations​ - As of March 2025, only the REST catalog is supported. - Verified to work with AWS S3 for data storage. Other S3-compatible storages may work but are unverified. - Key Benefits for Timeplus Iceberg Integration - CREATE DATABASEDDL SettingsExample: AWS Glue REST CatalogExample: AWS S3 Table REST CatalogExample: Apache Gravitino REST CatalogGravitino Configuration - DDL Settings - Example: AWS Glue REST Catalog - Example: AWS S3 Table REST Catalog - Example: Apache Gravitino REST CatalogGravitino Configuration - Gravitino Configuration - Creating and Writing to an Iceberg Table - Writing to Iceberg via a Materialized View - Querying Iceberg Data with SparkSQLUsing SQL in TimeplusUsing SparkSQL - Using SQL in Timeplus - Using SparkSQL - DROP DATABASE - Limitations - DDL Settings - Example: AWS Glue REST Catalog - Example: AWS S3 Table REST Catalog - Example: Apache Gravitino REST CatalogGravitino Configuration - Gravitino Configuration - Gravitino Configuration - Using SQL in Timeplus - Using SparkSQL ## Idempotent - [Insert Idempotency | Timeplus](https://docs.timeplus.com/idempotent): Since Timeplus Enterprise v2.4 there are new settings idempotentid and enableidempotent_processing: - Core Features - Data Ingestion - Insert Idempotency # Insert Idempotency Since Timeplus Enterprise v2.4 there are new settings idempotent_id and enable_idempotent_processing: - INSERT INTO .. SETTINGS idempotent_id='..' VALUES .. - SELECT .. FROM .. SETTINGS enable_idempotent_processing=true These settings allow you to define a unique ID for each batch INSERT. Sending the data with the same value of idempotent_id won't result in duplicated data in the target stream. You can retry safely with those settings. Here is an example. ## Create a stream for testing​ Let's create a stream with 2 columns: This is an append-only stream. You can insert duplicated data, e.g. ## Insert with idempotent_id​ Recreate the stream if you have inserted any data. You can run the following SQL multiple times: Then run You will get 2. If you remove SETTINGS idempotent_id = 'batch1' and run the SQL again, duplicated data will be inserted. If you push data to Timeplus via REST API, you can also set x-timeplus-idempotent-id in the HTTP header. ## Streaming SQL with enable_idempotent_processing​ When you insert data with idempotent_id, querying the stream with table function will retrieve the historical data of the stream, without duplication. However if you run: Every time you run the INSERT SQL, no matter with idempotent_id or not, a bigger count number will be returned. To enable idempotent processing, set the flag to true in the Streaming SQL, e.g. With this setting, your Streaming SQL will only emit results when it accepts new data. ## max_idempotent_ids​ Timeplus keeps a set of idempotent IDs in the memory and on the disk. By default the maximum number of idempotent IDs is 1000 (per stream). As an advanced setting, before you start the server for the first time, you can set max_idempotent_ids to a different value in the /server/config.yaml then start the server. Each node in the cluster should set the same value of max_idempotent_ids. - Create a stream for testing - Insert with idempotent_id - Streaming SQL with enable_idempotent_processing - max_idempotent_ids ## Ingest-api - [Push data to Timeplus via ingest REST API | Timeplus](https://docs.timeplus.com/ingest-api): As a generic solution, you can call the ingestion REST API to push data to Timeplus, with any preferred languages. In many cases, you can configure other systems or tools to push data directly to Timeplus via the ingest REST as the webhook endpoint, without writing code. - Integrations - CLI, APIs & SDKs - Push data to Timeplus via ingest REST API # Push data to Timeplus via ingest REST API As a generic solution, you can call the ingestion REST API to push data to Timeplus, with any preferred languages. In many cases, you can configure other systems or tools to push data directly to Timeplus via the ingest REST as the webhook endpoint, without writing code. This feature is available in Timeplus Enterprise. It leverages the low-level ingest REST API in timeplusd or Timeplus Proton. Please check https://docs.timeplus.com/rest for the detailed API documentations. ## Create a stream in Timeplus​ First you need to create a stream in Timeplus, either using the web UI or via REST API. Columns with proper names and types should be set. In the following section, we assume the stream name is foo. ## Set Authorization Header​ For Timeplus Enterprise self-hosted deployments, you need to set HTTP Authorization header to be Basic [Base64 encoded user:password]. For example, if the user name is hello and the password is world. The base64-encoded content for hello:world is aGVsbG86d29scmQ=. Then you need to set the HTTP header as Authorization: Basic aGVsbG86d29scmQ=. A new user will be created when you login Timeplus Enterprise for the first time. You can also add new users via web console or CLI. ## Push data to Timeplus​ ### Endpoint​ The endpoint for real-time data ingestion is https://host/{workspace-id}/api/v1beta2/streams/{name}/ingest. Please note: 1. If you self-host Timeplus Enterprise, depending on whether you enable HTTPS, choose http or https accordingly. The workspace ID is default. 2. You need to send POST request to this endpoint. ### format​ Depending on your use cases, there are many options to push data to Timeplus via REST API. You can set different Content-Type in the HTTP Header, and add the format query parameter in the URL. Here are a list of different use cases to push data to Timeplus: #### 1) Push JSON objects directly​ Request samples - Node.js - curl - Python - Java You can push Newline Delimited JSON (https://github.com/ndjson/ndjson-spec/) to the endpoint. Make sure you set the HTTP Header as one of these: - application/x-ndjson - application/vnd.timeplus+json;format=streaming If you would like to leverage a 3rd party system/tool to push data to Timeplus but it doesn't allow custom content type, then you can use the standard application/json, and send POST request to /api/v1beta2/streams/$STREAM_NAME/ingest?format=streaming. This will ensure the Timeplus API server treats the POST data as NDJSON. The request body is just a stream of JSON objects. e.g. Each object does not have to be in a single line. For example: They don’t have to be separated by newline either: Just make sure all columns in the target stream are specified with proper value in the request body. #### 2) Push a single JSON or string to a single column stream​ Request samples - Node.js - curl - Python - Java It's a common practice to create a stream in Timeplus with a single string column, called raw. You can put JSON objects in this column then extract values (such as select raw:key1), or put the raw log message in this column. When you set Content-Type header to text/plain, and add format=raw to the ingestion endpoint, the entire body in the POST request will be put in the raw column. #### 3) Push multiple JSON or text to a single column stream. Each line is an event​ Request samples - Node.js - curl - Python - Java When you set Content-Type header to text/plain, and add format=lines to the ingestion endpoint, each line in the POST body will be put in the raw column. #### 4) Push multiple events in a batch without repeating the columns​ Request samples - Node.js - curl - Python - Java The above method should work very well for most system integration. However, the column names will be repeatedly mentioned in the requested body. We also provide a more performant solution to only list the column names once. Same endpoint URL: https://host/{workspace-id}/api/v1beta2/streams/{name}/ingest But you need to set the HTTP Header to one of these: - application/json - application/vnd.timeplus+json - application/vnd.timeplus+json;format=compact The request body is this format: Note: - the columns is an array of string, with the column names - the data is an array of arrays. Each nested array represents a row of data. The value order must match the exact same order in the columns. For example: You can also use one of our SDKs to ingest data without handling the low level details of REST API. ### db​ If you need to pushd data to a stream other than default database, you can add the db parameter. For example ingest?format=lines&db=mydb. - Create a stream in Timeplus - Set Authorization Header - Push data to TimeplusEndpointformat1) Push JSON objects directly2) Push a single JSON or string to a single column stream3) Push multiple JSON or text to a single column stream. Each line is an event4) Push multiple events in a batch without repeating the columnsdb - Endpoint - format1) Push JSON objects directly2) Push a single JSON or string to a single column stream3) Push multiple JSON or text to a single column stream. Each line is an event4) Push multiple events in a batch without repeating the columns - 1) Push JSON objects directly - 2) Push a single JSON or string to a single column stream - 3) Push multiple JSON or text to a single column stream. Each line is an event - 4) Push multiple events in a batch without repeating the columns - db - Endpoint - format1) Push JSON objects directly2) Push a single JSON or string to a single column stream3) Push multiple JSON or text to a single column stream. Each line is an event4) Push multiple events in a batch without repeating the columns - 1) Push JSON objects directly - 2) Push a single JSON or string to a single column stream - 3) Push multiple JSON or text to a single column stream. Each line is an event - 4) Push multiple events in a batch without repeating the columns - db - 1) Push JSON objects directly - 2) Push a single JSON or string to a single column stream - 3) Push multiple JSON or text to a single column stream. Each line is an event - 4) Push multiple events in a batch without repeating the columns ## Ingestion - [Getting Data In | Timeplus](https://docs.timeplus.com/ingestion): Timeplus supports multiple ways to load data into the system, or access the external data without copying them in Timeplus: - Core Features - Data Ingestion # Getting Data In Timeplus supports multiple ways to load data into the system, or access the external data without copying them in Timeplus: - External Stream for Apache Kafka, Confluent, Redpanda, and other Kafka API compatible data streaming platform. This feature is also available in Timeplus Proton. - External Stream for Apache Pulsar is available in Timeplus Enterprise 2.5 and above. - Source for extra wide range of data sources. This is only available in Timeplus Enterprise. This integrates with Redpanda Connect, supporting 200+ connectors. - On Timeplus web console, you can also upload CSV files and import them into streams. - For Timeplus Enterprise, REST API and SDKs are provided to push data to Timeplus programmatically. - On top of the REST API and SDKs, Timeplus Enterprise adds integrations with Kafka Connect, AirByte, Sling, and seatunnel. - Last but not the least, if you are not ready to load your real data into Timeplus, or just want to play with the system, you can use the web console to create sample streaming data, or use SQL to create random streams. ## Add new data via web console​ Choose "Data Collection" from the navigation menu to setup data access to other systems. There are two categories: - Timeplus Connect: directly supported by Timeplus Inc, with easy-to-use setup wizards. Demo Stream: generate random data for various use cases. Learn more Timeplus: read data from another Timeplus deployment. Learn more Apache Kafka: setup external streams to read from Apache Kafka. Learn more Confluent Cloud: setup external streams to read from Confluent Cloud Redpanda: setup external streams to read from Redpanda Apache Pulsar: setup external streams to read from Apache Pulsar. Learn more ClickHouse: setup external tables to read from ClickHouse, without duplicating data in Timeplus. Learn more NATS: load data from NATS to Timeplus streams WebSocket: load data from WebSocket to Timeplus streams HTTP Stream: load data from HTTP stream to Timeplus streams Coinbase Exchange: connect to the Coinbase Exchange via WebSocket and load data into Timeplus streams CSV: choose a CSV file and send to Timeplus. Learn more Stream Ingestion: a wizard to guide you to push data to Timeplus via Ingest REST API. Learn more - Demo Stream: generate random data for various use cases. Learn more - Timeplus: read data from another Timeplus deployment. Learn more - Apache Kafka: setup external streams to read from Apache Kafka. Learn more - Confluent Cloud: setup external streams to read from Confluent Cloud - Redpanda: setup external streams to read from Redpanda - Apache Pulsar: setup external streams to read from Apache Pulsar. Learn more - ClickHouse: setup external tables to read from ClickHouse, without duplicating data in Timeplus. Learn more - NATS: load data from NATS to Timeplus streams - WebSocket: load data from WebSocket to Timeplus streams - HTTP Stream: load data from HTTP stream to Timeplus streams - Coinbase Exchange: connect to the Coinbase Exchange via WebSocket and load data into Timeplus streams - CSV: choose a CSV file and send to Timeplus. Learn more - Stream Ingestion: a wizard to guide you to push data to Timeplus via Ingest REST API. Learn more - Redpanda Connect: available since Timeplus Enterprise 2.5 or above. Set up data access to other systems by editing a YAML file. Powered by Redpanda Connect, supported by Redpanda Data Inc. or Redpanda Community. - Demo Stream: generate random data for various use cases. Learn more - Timeplus: read data from another Timeplus deployment. Learn more - Apache Kafka: setup external streams to read from Apache Kafka. Learn more - Confluent Cloud: setup external streams to read from Confluent Cloud - Redpanda: setup external streams to read from Redpanda - Apache Pulsar: setup external streams to read from Apache Pulsar. Learn more - ClickHouse: setup external tables to read from ClickHouse, without duplicating data in Timeplus. Learn more - NATS: load data from NATS to Timeplus streams - WebSocket: load data from WebSocket to Timeplus streams - HTTP Stream: load data from HTTP stream to Timeplus streams - Coinbase Exchange: connect to the Coinbase Exchange via WebSocket and load data into Timeplus streams - CSV: choose a CSV file and send to Timeplus. Learn more - Stream Ingestion: a wizard to guide you to push data to Timeplus via Ingest REST API. Learn more ### Load streaming data from Apache Kafka​ As of today, Kafka is the primary data integration for Timeplus. With our strong partnership with Confluent, you can load your real-time data from Confluent Cloud, Confluent Platform, or Apache Kafka into the Timeplus streaming engine. You can also create external streams to analyze data in Confluent/Kafka/Redpanda without moving data. Learn more. ### Load streaming data from Apache Pulsar​ Apache® Pulsar™ is a cloud-native, distributed, open source messaging and streaming platform for real-time workloads. Since Timeplus Enterprise 2.5, Pulsar External Streams can be created to read or write data for Pulsar. Learn more. ### Upload local files​ If you have some static dataset or lookup tables in the CSV format, you can upload the files directly to Timeplus. 1. Click the Add Data from the navigation menu. Then click Import from CSV button 2. Drag and drop a CSV file from your local file system to upload the file. (COMING SOON: if your file is downloadable from a URL or S3 bucket, you can create a source to have Timeplus server to load it. File formats other than CSV will be supported too.) 3. Choose whether the first row of the file is the column header. 4. Specify a name for the stream, and optionally provide a readable description. 5. Click the button to start uploading data and click the View Imported Data button to run a query to review imported data. ### Load sample streaming data​ If you are not ready to load your real data into Timeplus, or just want to play with the system, you can use this feature to load some sampling streaming data. We provide 3 typical streaming data. 1. iot will generate data for 3 devices(device_0, device_1 and device_2). The number value can be anything between 0 to 100. The time column is when the event is generated. 2. user_logins will generate data for 2 users(user1 and user2), from 2 possible city values: Shanghai or Beijing. The time column is when the event is generated. 3. devops will generate data for 3 hostname(host_0,host_1, and host_2), from 3 possible region(eu-central-1, us-west-1, and sa-east-1), 3 possible rack(1,2,3), a number usage_user from 0 to 100, usage_system from 0 to 100, and time column for the event time. You can load such sample data via the Add Data button and the Sample Dataset option. You can create new streams or choose existing streams for the data. ## Push data to Timeplus via REST or SDK​ Timeplus provides ingestion REST API, and related SDKs in different programming languages. Developers can leverage those REST API or SDK to push real-time data to Timeplus. Learn more. ## Load other data into Timeplus via 3rd party tools​ Timeplus works with the data ecosystems and can leverage various tools to load data or even do data transformation at ingestion time. ### Airbyte​ AirByte provides both OSS version and managed cloud to collect data, transform data and send to other destinations. At the high level 1. AirByte can grab data from many different data sources, including database/CDC, or infrastructure log, application logs, or even business apps(such as Salesforce) 2. The data can be normalized via AirByte built-in capabilities. Or it can be saved to the destination database first, then relies on dbt or other tools to apply transformations/materialization. 3. Data collected by AirByte can be sent to many destinations, including Timeplus. Just name a few data sources from Airbyte: - App marketplace such as Apple App Store - AWS Cloudtrail - Google BigQuery - Load file from S3/GCS/SFTP/local with Gzip/Zip/xz/Snappy compression, in CSV/JSON/XML/Excel/Parquet/etc - Github, GitLab, or JIRA activities - Google Ads - Instagram social media - Slack or Microsoft Teams - PostgreSQL, RedShift, Snowflake, MongoDB, MySQL, Microsoft SQL Server, etc ### Kafka Connect​ You can use Kafka Connectors to load data from popular data sources into Confluent Cloud, Confluent Platform, or Apache Kafka, then use Timeplus to load them into streams via the built-in Kafka Source. There are a few examples of data sources that can be ingested into Timeplus via Kafka Connectors. Please check https://www.confluent.io/product/confluent-connectors/ for more details. - Amazon CloudWatch Logs - Amazon Kinesis - Amazon S3 - Amazon SQS - Azure Blob Storage - Azure Event Hubs - CockroachDB CDC - Databricks - Google Cloud Pub/Sub - InfluxDB - JDBC - Microsoft SQL Server - MongoDB - MQTT - MySQL CDC - Oracle Database - PostgreSQL CDC - Salesforce - ServiceNow - SFTP - SNMP - Splunk ### Sling​ Sling is an open source is a powerful data integration CLI tool. Whether ingesting CSV or JSON files, transferring data between databases, or exporting a custom SQL query to a Parquet file — Sling is the solution that empowers you to achieve it effortlessly. Since from v1.2.14, Sling adds built-in support for Timeplus. You just need a single binary for your OS to load any data to Timeplus, with a simple command such as: Learn more. - Add new data via web consoleLoad streaming data from Apache KafkaLoad streaming data from Apache PulsarUpload local filesLoad sample streaming data - Load streaming data from Apache Kafka - Load streaming data from Apache Pulsar - Upload local files - Load sample streaming data - Push data to Timeplus via REST or SDK - Load other data into Timeplus via 3rd party toolsAirbyteKafka ConnectSling - Airbyte - Kafka Connect - Sling ... (content truncated) ## Integration-grafana - [Integration with Grafana | Timeplus](https://docs.timeplus.com/integration-grafana): Grafana has long been one of the most popular tools for real-time monitoring and data visualization, helping organizations track metrics and analyze trends through a single, user-friendly interface. For self-hosted Timeplus Enterprise or Timeplus Proton, you can try the Grafana plugin for Timeplus with the source code at GitHub. This plugin was designed to leverage Grafana’s new Grafana Live capability, allowing users to keep their SQL query results up-to-date without the need to refresh their dashboards. Check out here for sample setup. - Integrations - Third-party Tools - Integration with Grafana # Integration with Grafana Grafana has long been one of the most popular tools for real-time monitoring and data visualization, helping organizations track metrics and analyze trends through a single, user-friendly interface. For self-hosted Timeplus Enterprise or Timeplus Proton, you can try the Grafana plugin for Timeplus with the source code at GitHub. This plugin was designed to leverage Grafana’s new Grafana Live capability, allowing users to keep their SQL query results up-to-date without the need to refresh their dashboards. Check out here for sample setup. ## User Feedback​ > "Using Timeplus and Grafana together has been awesome to work with! Timeplus simplifies what was a delicate manual transform process into an automatic SQL-based process. Grafana+Timeplus turns repetitive full queries into streaming incremental updates which makes the Grafana dashboard experience much more immersive, without all the constant full dashboard updates." - Jason Patterson, Director of Network Architecture, RocNet Supply "Using Timeplus and Grafana together has been awesome to work with! Timeplus simplifies what was a delicate manual transform process into an automatic SQL-based process. Grafana+Timeplus turns repetitive full queries into streaming incremental updates which makes the Grafana dashboard experience much more immersive, without all the constant full dashboard updates." - Jason Patterson, Director of Network Architecture, RocNet Supply ## Key Use Cases for Timeplus Grafana Plugin​ Our Timeplus Grafana Plugin provides a flexible, powerful way to enhance your dashboards with real-time insights. Here are a few ways our users are already benefiting: - Real-Time Operational Monitoring: Track key metrics with high-frequency data, such as system load, application performance, or network traffic, to identify and resolve issues as they arise. - Customer Experience Insights: Monitor live customer interactions, such as usage patterns, in-app behavior, or user activity across different regions, to improve responsiveness and overall service quality. - Advanced Analytics on Streaming Data: Combine metrics from Grafana with Timeplus streaming SQL capabilities to perform complex analytics—filtering, aggregating, and transforming data in real time for deeper insights into business operations. ## How to Get Started with Timeplus Grafana Plugin​ Getting started with the latest version of the Timeplus Grafana Plugin is straightforward. ### Step 1: Set Up Grafana​ First, ensure that you have Grafana OSS or Grafana Enterprise installed. • For Mac users, you can install Grafana by running: brew install grafana • For Linux users, you can use the following commands, adjusting the download URL to match the latest version if necessary: sudo yum install -y https://dl.grafana.com/enterprise/release/grafana-enterprise-11.3.0-1.x86_64.rpm For additional installation instructions, please refer to the Grafana download page. ### Step 2: Install the Timeplus Plugin​ Next, download and install the Timeplus plugin in your Grafana deployment. For example, if you’re on Linux: After installing the plugin, restart Grafana to enable it: sudo /bin/systemctl restart grafana-server.service ### Step 3: Configure the Plugin in Grafana​ 1. Login to Grafana’s web UI and navigate to Connections -> Data Sources. 2. Locate the Timeplus Data Source by scrolling or searching for “Timeplus.” 3. Configure the Connection Parameters: - For Timeplus Enterprise, create an admin account when you start Timeplus for the first time and use those credentials here. - For Timeplus Proton, use the default username with no password. 1. Save and Test the Connection by clicking the Save & Test button to confirm the plugin is connected successfully. ### Step 4: Create a Dashboard and Visualize Data​ 1. Follow the “Build a Dashboard” link to start a new dashboard, and add a visualization. 2. Select the Timeplus data source you configured earlier. Now, you can enter SQL queries to retrieve and visualize data from Timeplus directly in Grafana. ### Step 5: Example - Monitoring Real-Time Bitcoin Price​ To showcase the power of real-time streaming with Grafana and Timeplus, let’s set up a feed for live Bitcoin pricing: 1. In Timeplus Enterprise, open the Data Collection Wizard and choose Coinbase Exchange as the data source. If you haven’t tried Timeplus, please download the package at https://www.timeplus.com/product . 2. Use the default settings to load BTC-USD data feed. 3. After the source is created. Click on the first button to run an ad-hoc query to review the data from the newly created data stream. SELECT * FROM coinbase You should see the live data with multiple columns. 4. Return to the Grafana UI and run the following SQL query on the new view: Grafana will now load the live data from Timeplus and display it as a line chart. To get a closer look at recent trends, adjust the time period from the default 6 hours to the last 5 minutes. With this setup, you can now use Grafana to monitor live prices and transaction data in the cryptocurrency market. Additionally, you can expand this dashboard by adding SQL-based alerts or logic to detect potential trading signals in real time. ## Change Log​ ### 2.1.3​ Released on 05-12-2025 - Fix the render issue for nullable columns ### 2.1.2​ Released on 02-04-2025 - Fix ref id for data frames ### 2.1.1​ Released on 01-22-2025 - Propagate the error message from the backend to the frontend - Set batch size to 1000 and batch interval to 100ms for the streaming query ### 2.1.0​ Released on 01-12-2025 - Support query variables and annotations - Updated Grafana Go SDK ### 2.0.0​ Released on 11-05-2024 - Updated Grafana Go SDK and Proton Go driver - Unlisted from Grafana marketplace - User Feedback - Key Use Cases for Timeplus Grafana Plugin - How to Get Started with Timeplus Grafana PluginStep 1: Set Up GrafanaStep 2: Install the Timeplus PluginStep 3: Configure the Plugin in GrafanaStep 4: Create a Dashboard and Visualize DataStep 5: Example - Monitoring Real-Time Bitcoin Price - Step 1: Set Up Grafana - Step 2: Install the Timeplus Plugin - Step 3: Configure the Plugin in Grafana - Step 4: Create a Dashboard and Visualize Data - Step 5: Example - Monitoring Real-Time Bitcoin Price - Change Log2.1.32.1.22.1.12.1.02.0.0 - 2.1.3 - 2.1.2 - 2.1.1 - 2.1.0 - 2.0.0 - Step 1: Set Up Grafana - Step 2: Install the Timeplus Plugin - Step 3: Configure the Plugin in Grafana - Step 4: Create a Dashboard and Visualize Data - Step 5: Example - Monitoring Real-Time Bitcoin Price - 2.1.3 - 2.1.2 - 2.1.1 - 2.1.0 - 2.0.0 ## Jdbc - [JDBC Drivers | Timeplus](https://docs.timeplus.com/jdbc): Timeplus provides 2 types of JDBC drivers: - Integrations - CLI, APIs & SDKs - JDBC Drivers # JDBC Drivers Timeplus provides 2 types of JDBC drivers: - com.timeplus:timeplus-native-jdbc is the JDBC driver over the native TCP port. It can run both streaming queries and batch queries, with high performance. It's recommended to use this driver with Timeplus Enterprise. - com.timeplus:proton-jdbc is the JDBC driver over HTTP, ideal for running batch queries. It supports more formats, but it's not as performant as the native jdbc driver. ## JDBC over TCP​ ### Use Case​ This JDBC driver is designed for running streaming queries and the performance is usually higher than the JDBC over HTTP, but with some limitations on data types and compression methods. This library is available on maven central repository: ### Maven​ ### Gradle​ ### Configuration​ - Driver class is com.timeplus.jdbc.TimeplusDriver - JDBC URL is jdbc:timeplus://localhost:8463 - For Timeplus Proton, the username is default and password is an empty string. For Timeplus Enterprise, the username and password can be customized. Please note, by default Timeplus' query behavior is streaming SQL, looking for new data in the future and never ends. The ResultSet.next() can always return true. ### Example​ ### DBeaver​ You can also connect to Timeplus from GUI tools that supports JDBC, such as DBeaver. First add the Timeplus Native JDBC driver to DBeaver. Taking DBeaver 23.2.3 as an example, choose "Driver Manager" from "Database" menu. Click the "New" button, and use the following settings: - Driver Name: Timeplus - Driver Type: Generic - Class Name: com.timeplus.jdbc.TimeplusDriver - URL Template: jdbc:timeplus://localhost:8463 In the "Libraries" tab, click "Add Artifact" and type com.timeplus:timeplus-native-jdbc:2.0.9. Click the "Find Class" button to load the class. Create a new database connection, choose "Timeplus" and accept the default settings. Click the "Test Connection.." to verify the connection is okay. Open a SQL script for this connection, type the sample SQL select 1 Ctrl+Enter to run the query and get the result. But it's more common to use this native JDBC driver in your Java program to process the results from streaming SQL. ## JDBC over HTTP​ ### Use Case​ This JDBC driver is designed for running batch queries and you can integrate it with DBeaver and other tools. This library is available on maven central repository: ### Maven​ ### Gradle​ ### Configuration​ - Driver class is com.timeplus.proton.jdbc.ProtonDriver - JDBC URL is jdbc:proton://localhost:8123 or jdbc:proton://localhost:8123/default - For Timeplus Proton, the username is default and password is an empty string. For Timeplus Enterprise, the username and password can be customized. Please note, by default Timeplus' query behavior is streaming SQL, looking for new data in the future and never ends. This can be considered as hang for JDBC client. You have 2 options: - Use the 8123 port. In this mode, all SQL are ran in batch mode. So select .. from car_live_data will read all existing data. - Use 3218 port. In this mode, by default all SQL are ran in streaming mode. Please use select .. from .. LIMIT 100 to stop the query at 100 events. Or use the table function to query historical data, such as select .. from table(car_live_data).. ### Example​ ### DBeaver​ You can also connect to Timeplus from GUI tools that supports JDBC, such as DBeaver. First add the Proton JDBC driver to DBeaver. Taking DBeaver 23.2.3 as an example, choose "Driver Manager" from "Database" menu. Click the "New" button, and use the following settings: - Driver Name: Timeplus Proton - Driver Type: Generic - Class Name: com.timeplus.proton.jdbc.ProtonDriver - URL Template: jdbc:proton://{host}:{port}/{database} - Default Port: 8123 - Default Database: default - Default User: default - Allow Empty Password In the "Libraries" tab, click "Add Artifact" and type com.timeplus:proton-jdbc:0.7.1. Click the "Find Class" button to load the class. Create a new database connection, choose "Timeplus Proton" and accept the default settings. Click the "Test Connection.." to verify the connection is okay. Open a SQL script for this connection, type the sample SQL select 1 Ctrl+Enter to run the query and get the result. - JDBC over TCPUse CaseMavenGradleConfigurationExampleDBeaver - Use Case - Maven - Gradle - Configuration - Example - DBeaver - JDBC over HTTPUse CaseMavenGradleConfigurationExampleDBeaver - Use Case - Maven - Gradle - Configuration - Example - DBeaver - Use Case - Maven - Gradle - Configuration - Example - DBeaver - Use Case - Maven - Gradle - Configuration - Example - DBeaver ## Jit - [Just-In-Time (JIT) compilation | Timeplus](https://docs.timeplus.com/jit): Starting from Timeplus Enterprise 2.9, the JIT compilation is enabled by default. For example, if you need to run the following SQL multiple times: - Core Features - Stream Processing - Just-In-Time (JIT) compilation # Just-In-Time (JIT) compilation Starting from Timeplus Enterprise 2.9, the JIT compilation is enabled by default. For example, if you need to run the following SQL multiple times: Timeplus will compile the complex SQL expression to machine code to improve the runtime performance. For more technical details of the implementation, please check the blog. ## Settings​ The following settings can be overridden in the query time using SET key=value. You can also query the current value and description via ### min_count_to_compile_expression​ Minimum count of executing same expression before it is get compiled. uint64 type. Default to 3. ### min_count_to_compile_aggregate_expression​ The minimum number of identical aggregate expressions to start JIT-compilation. Works only if the compile_aggregate_expressions setting is enabled. uint64 type. Default to 3. ### min_count_to_compile_sort_description​ The number of identical sort descriptions before they are JIT-compiled. uint64 type. Default to 3. ## Monitoring​ You can run the following query to check the counts of JIT events: - Settingsmin_count_to_compile_expressionmin_count_to_compile_aggregate_expressionmin_count_to_compile_sort_description - min_count_to_compile_expression - min_count_to_compile_aggregate_expression - min_count_to_compile_sort_description - Monitoring - min_count_to_compile_expression - min_count_to_compile_aggregate_expression - min_count_to_compile_sort_description ## Joins - [Streaming Joins | Timeplus](https://docs.timeplus.com/joins): JOIN is a key feature in Timeplus to combine data from different sources and freshness into a new stream. - Core Features - Stream Processing - Streaming Joins # Streaming Joins JOIN is a key feature in Timeplus to combine data from different sources and freshness into a new stream. ## Introduction​ Please refer to https://en.wikipedia.org/wiki/Join_(SQL) for general introduction. ### Join Types​ #### INNER JOIN (default)​ INNER JOIN is the most common JOIN to return data have matching values in both side of the JOIN. This is also the default behaviour if you just use JOIN. (Credit: https://dataschool.com/how-to-teach-people-sql/sql-join-types-explained-visually/) #### LEFT JOIN​ LEFT JOIN returns all rows from the left stream with matching rows in the right table. Some columns can be NULL when there is no match. (Credit: https://dataschool.com/how-to-teach-people-sql/sql-join-types-explained-visually/) #### FULL/OUTER JOIN​ OUTER JOIN combines the columns and rows from all tables and includes NULL when there is no match. (Credit: https://dataschool.com/how-to-teach-people-sql/sql-join-types-explained-visually/) #### Not Supported: RIGHT or CROSS​ Other types of JOINS are not supported in the current version of Timeplus. If you have good use cases for them, please contact us at support@timeplus.com. 1. RIGHT 2. CROSS ### Join Strictnesses​ #### ALL JOIN (default)​ One row from the left side of the JOIN could result in several join results if the right side table keeps multiple values for the same key. #### ASOF JOIN​ ASOF JOIN has an additional unequal join expression. This can work well if two streams have the same id, but not with exactly the same timestamps. It first tries to find a match on the regular join keys, and then finds the closest match on the unequal join expression since we can keep multiple versions of value for each key in the right hash table. In ASOF JOIN, at most there will be one joined row for a row from the left stream. #### LATEST JOIN​ Only the latest key/value pairs of the right table will be joined with the left table. ## Data to be Joined​ All the streams and tables in Timeplus can be JOINed. ### Append Streams​ This is the default storage type for streams. It is optimized for fast ingestion and low latency. #### External Streams​ External Streams, such as Kafka, are treated as append streams during the JOIN operation. ### Mutable Streams​ Mutable Stream is only available in Timeplus Enterprise, with primary key(s) and row-based storage for fast updates. In Timeplus Proton, the following 2 types of streams provide similar changelog semantics: #### Versioned Streams​ Versioned Stream with primary key(s) and multiple versions #### Changelog Streams​ Changelog Stream with primary key(s) and CDC semantic (data can be removed, or updated with old&new value). You can also use the changelog() function to convert an append-only stream to changelog stream. ### Static Tables​ The following types of Timeplus resources won't be changed during the JOIN operation: #### table(stream)​ If you call the table() function for a stream, the historical data of the stream will be available in the JOIN, but won't be changed. #### External Table​ External tables, such as MySQL, are treated as static data during the JOIN. ### Dictionary​ Dictionaries are locally-cached data, aiming to provide fast lookup for dimension tables. You can set a LIFETIME to control the cache expiration time. If the value of the dictionary is changed during the JOIN, new value will be used. ## JOIN Category​ At the high level, the JOIN syntax is By default, the strictness is ALL and the join kind is INNER. Since Timeplus supports many types of streams, join types, and join strictness, these factors can lead to different JOIN behaviors, with hundreds of combinations. Not all of them are meaningful or performant. Please read on for the supported combinations. We can categorize them into the following categories: - Bidirectional JOIN: Left and right streams are unbounded and new data on either side of the JOIN will be matched in real-time. Self-join is a special case. - Range Bidirectional JOIN: Similar to Bidirectional JOIN, but with a time range constraint. - Dynamic Enrichment JOIN: Enrich streaming data by connecting to a relative static dimension table. ### Bidirectional JOIN​ Bidirectional join needs to buffer all data for the left stream and right stream and build hash tables for both of them. Whenever there are new or updated rows appearing on either side, the other side’s hash table is probed to see if a join can be matched. #### Append INNER/LEFT ALL JOIN Append​ This may look like the most easy-to-understand scenario. You can try this type of join if you have 2 streams with incoming new data. However, this is designed to be exploration purpose only, not recommended to for production use. Because both sides of the data streams are unbounded, it will consume more and more resources in the system. Internally there is a setting (join_max_buffered_bytes) to control the maximum source data it can buffer. Once the query reaches the limit, the streaming query will be aborted. Example: #### Mutable INNER/LEFT/FULL ALL JOIN Mutable​ Both Mutable Streams and Versioned Streams can be joined together. This is production ready. We also provide Changelog Streams joining Changelog Stream as an experimental feature. Examples: #### Self Join​ You can also join a stream to itself. A typical use case is to check whether there is a certain pattern for the data in the same stream, for example, whether for the same credit card, within 2 minutes, there is a big purchase after a small purchase. This could be a pattern for fraud. ### Range Bidirectional JOIN​ #### Append JOIN Append (within time range)​ The above the bidirectional JOIN may buffer too much data, range bidirectional join tries to mitigate this problem by bucketing the stream data in time ranges and try to join the data bidirectionally in appropriate range buckets. This allows for more efficient processing and reduces the amount of data that needs to be buffered. It's common to use the date_diff_within function in the join condition: date_diff_within function by default uses the _tp_time column on both side of the streams to compare the time difference. You can also explicitly specify the columns to use: Actually we don’t even require a timestamp for the range, any integral columns are supposed to work. For instance, AND left_stream.sequence_number < rightstream.sequence_number + 10. ### Dynamic Enrichment JOIN​ There are typical cases that an unbounded data stream needs enrichment by connecting to a relatively static dimension table. Timeplus can do this in one single engine by storing streaming data and dimension tables in it, as well as accessing to external database or file systems. #### Append JOIN Static​ Examples: In the above example, data from device_utils is a stream and data from device_products_info is historical data since it is wrapped with table() function. For every (new) row from device_utils, it is continuously joined with rows from dimension table device_products_info and enriches the streaming data with product vendor information. #### Mutable JOIN Static​ You can also put mutable streams, versioned stream or changelog stream on the left side of the join. #### Static JOIN Static​ You can put static tables or external tables on the both side of the join. This will be identical to the common OLAP databases. #### Append INNER/LEFT ALL JOIN Mutable​ The right side can be a mutable stream, versioned stream or changelog stream. Timeplus won't buffer the left stream at all, and keep all versions for the join key for the right stream. #### Direct JOIN​ Direct join is used when the right side is from an external source (ClickHouse for example). It does not require loading all the content into memory; but just the required data to complete the join. Please note that direct join is not a SQL keyword. You enable this by adding SETTINGS join_algorithm = 'direct' in the streaming SQL. ##### Direct JOIN with Dictionary​ Direct JOIN looks up the dictionary with the exact join key value and gets the matched rows for join. If some of the keys are not found in dictionary layout (storage) or their TTL is expired, Dictionary will initiate requests to remote source and update the layout content. Example of an append stream joining with a dictionary from a MySQL table: Example of an append stream joining with a dictionary from a ClickHouse table, and using mutable stream as the cache: ##### Direct JOIN with Mutable Stream​ You can also use direct join with a mutable stream (not versioned stream or changelog stream). The JOIN can be based on the primary key or secondary index. Example: #### Append INNER/LEFT ASOF JOIN Append​ ASOF enrichment join keeps multiple versions(by default 3 versions) of values for the same join key in the hash table and the values are sorted by ASOF unequal join key. This can be customized by setting the keep_versions. #### Append INNER/LEFT ASOF JOIN Mutable​ Example: #### Append INNER/LEFT LATEST JOIN Append​ Similar to ALL JOIN above, but we only keep the latest version of value for each join key. The right side of the LATEST JOIN can be an append stream or mutable stream(including versioned stream). #### Append INNER/LEFT LATEST JOIN Mutable​ ## Contact us for help or more features​ JOIN in streaming processing / analytics could be more complex than traditional database join regarding their semantics. Feel free to reach out to us for help or more features. Join our community slack at https://timeplus.com/slack to discuss more. ... (content truncated) ## Js-udf - [JavaScript UDF | Timeplus](https://docs.timeplus.com/js-udf): Timeplus supports JavaScript-based UDF running in the SQL engine. You can develop User-defined scalar functions (UDFs) or User-defined aggregate functions (UDAFs) with modern JavaScript (powered by V8). No need to deploy extra server/service for the UDF. More languages will be supported in the future. - SQL Reference - Functions - User Defined Functions - JavaScript UDF # JavaScript UDF Timeplus supports JavaScript-based UDF running in the SQL engine. You can develop User-defined scalar functions (UDFs) or User-defined aggregate functions (UDAFs) with modern JavaScript (powered by V8). No need to deploy extra server/service for the UDF. More languages will be supported in the future. ## Register a JS UDF via SQL​ Please check CREATE FUNCTION page for the SQL syntax. ## Register a JS UDF via Web Console​ 1. Open "UDFs" from the navigation menu on the left, and click the 'New UDF' button. 2. Specify a function name, such as second_max. Make sure the name won't conflict with built-in functions or other UDF. Description is optional. 3. Choose the data type for input parameters and return value. 4. Choose "JavaScript" as the UDF type. 5. Specify whether the function is for aggregation or not. 6. Enter the JavaScript source for the UDF. (We will explain more how to write the code.) 7. Click Create button to register the function. ## Arguments​ Unlike Remote UDF, the argument names don't matter when you register a JS UDF. Make sure you the list of arguments matches the input parameter lists in your JavaScript function. The input data are in Timeplus data type. They will be converted to JavaScript data type. ## Returned value​ The JavaScript UDF can return the following data types and they will be converted back to the specified Timeplus data types. The supported return type are similar to argument types. The only difference is that if you return a complex data structure as an object, it will be converted to a named tuple in Timeplus. ## Develop a scalar function​ A scalar function is a function that returns one value per invocation; in most cases, you can think of this as returning one value per row. This contrasts with Aggregate Functions, which returns one value per group of rows. ### Scalar function with 1 argument​ For example, you would like to check whether the user sets a work email in their profile. Although this could be doable with plain SQL but it'll be nice if you can create a UDF to make the SQL more readable, e.g. You can use the following code to define a new function is_work_email with one input type string and return bool. Notes: 1. The first line defines a function with the exact same name as the UDF. The number of arguments should match what you specify in the UDF form. 2. Please note the input is actually a JavaScript list. For the sake of high performance, Timeplus will reduce the number of function calls by combining the arguments together. You need to return a list with the exact same length of the input. 3. values.map(..) creates a new array populated with the results of calling a provided function on every element in the calling array (doc). 4. email=>email.endsWith("@gmail.com") is the shortcut to return a bool by checking whether the email ends with "@gmail.com". You can add more complex logic, or write in multiple lines and end with return ... ### Scalar function with 2 or more arguments​ Let's enhance the previous example, by defining a list of email domains which won't be considered as work-related. e.g. Similar to the last tutorial, you create a new function called email_not_in. This time you specify two arguments in string. Note: currently JS UDF doesn't support complex data types, such as array(string). The following code implements this new function: ### Scalar function with no argument​ Currently we don't support JS UDF without arguments. As a workaround, you can define a single argument, e.g. The magic_number takes an int argument as a workaround. In this case, the function will return 42 no matter what parameter is specified. ## Develop an aggregate function​ An aggregate function returns one value per group of rows. When you register the UDF, make sure you turn on the option to indicate this is an aggregation function. Compared to scalar functions, the life cycle is a bit more complex. ### 3 required and 3 optional functions​ Let's take an example of a function to get the second maximum values from the group. ### Example: get second largest number​ The full source code for this JS UDAF is To register this function with Timeplus Console: choose JavaScript as UDF type, make sure to turn on 'is aggregation'. Set the function name say second_max (you don't need to repeat the function name in JS code). Add one argument in float type and set return type to float too. Please note, unlike JavaScript scalar function, you need to put all functions under an object {}. You can define internal private functions, as long as the name won't conflict with native functions in JavaScript, or in the UDF lifecycle. ### Advanced Example for Complex Event Processing​ User-Defined Aggregation Function can be used for Complex Event Processing (CEP). Here is an example to count the number of failed login attempts for the same user. If there are more than 5 failed logins, create an alert message. If there is a successful login, reset the counter. Assuming the stream name is logins , with timestamp, user, login_status_code, this SQL can continuously monitor the login attempts: The UDAF is registered in this way: There is an advanced setting has_customized_emit. When this is set to true: - initialize() is called to prepare a clean state for each function invocation. - Proton partitions the data according to group by keys and feeds the partitioned data to the JavaScript UDAF. process(..) is called to run the customized aggregation logic. If the return value of process(..) is 0, no result will be emitted. If a none-zero value is returned by process(..), then finalize() function will be called to get the aggregation result. Proton will emit the results immediately. finalize() function should also reset its state for next aggregation and emit. Caveats: 1. One streaming SQL supports up to 1 UDAF with has_customized_emit=true 2. If there are 1 million unique key, there will be 1 million UDAF invocations and each of them handles its own partitioned data. 3. If one key has aggregation results to emit, but other keys don't have, then Proton only emit results for that key. This is an advanced feature. Please contact us or discuss your use case in Community Slack with us. ## Debug Tips​ ### console.log​ Staring from Timeplus Proton 1.6.5 or Timeplus Enterprise 2.5, you can use console.log(..) to add logging messages. The logs will be available in the server logs, such as /var/log/proton-server/proton-server.log for the Linux-based docker container console.log, e.g. ### Test the JS UDF without running in Timeplus​ To improve the debug efficiency, you can test the JS UDF without running them in Timeplus SQL. Taking the is_work_email UDF as an example, you can create a JS file and run it with node directly, e.g. Then you can run the JavaScript file with nodejs, e.g. You may use any IDE and set breakpoint or watch the variables. - Register a JS UDF via SQL - Register a JS UDF via Web Console - Arguments - Returned value - Develop a scalar functionScalar function with 1 argumentScalar function with 2 or more argumentsScalar function with no argument - Scalar function with 1 argument - Scalar function with 2 or more arguments - Scalar function with no argument - Develop an aggregate function3 required and 3 optional functionsExample: get second largest numberAdvanced Example for Complex Event Processing - 3 required and 3 optional functions - Example: get second largest number - Advanced Example for Complex Event Processing - Debug Tipsconsole.logTest the JS UDF without running in Timeplus - console.log - Test the JS UDF without running in Timeplus - Scalar function with 1 argument - Scalar function with 2 or more arguments - Scalar function with no argument - 3 required and 3 optional functions - Example: get second largest number - Advanced Example for Complex Event Processing - console.log - Test the JS UDF without running in Timeplus ## K8s-helm - [Deploy on Kubernetes with Helm | Timeplus](https://docs.timeplus.com/k8s-helm): You can deploy Timeplus Enterprise on a Kubernetes cluster with Helm. - Deployment & Operations - Timeplus Enterprise Self-hosted - Deploy on Kubernetes with Helm # Deploy on Kubernetes with Helm You can deploy Timeplus Enterprise on a Kubernetes cluster with Helm. For visual learning, you can watch the following video: ## Prerequisites​ - Ensure you have Helm 3.12 + installed in your environment. For details about how to install Helm, see the Helm documentation - Ensure you have Kubernetes 1.25 or higher installed in your environment. We tested our software and installation process on Amazon EKS, Google GKE, and self-hosted Kubernetes. Other Kubernetes distributions should work in the similar way. - Ensure you have allocated enough resources for the deployment. For a 3-nodes cluster deployment, by default each timeplusd requires 2 cores and 4GB memory. Please refer to Planning capacity section for production deployment. - Network access to Internet. If your environment is air-gapped, please refer to Offline installation. - Port 8464 needs to be open on each k8s node. The pods behind timeplusd Statefulset uses this port to talk to each other. ## Quickstart with self-hosted Kubernetes​ This is the quickstart guide to install a 3 nodes Timeplus Enterprise cluster with default configurations on a self-hosted Kubernetes using Helm package manager. You need to update configurations accordingly to fit your Kubernetes environment. Please refer to Configuration Guide for available values of the chart. ### Add Timeplus Helm chart repository​ Simply run the following commands to add the repo and list charts from the repo. A sample output would be: Staring from v3.0.0 chart version, the APP VERSION is the same version as Timeplus Enterprise. Prior to v3.0.0 chart version, the APP VERSION is the same version as the timeplusd component. ### Create Namespace​ User can choose to install the Timeplus Enterprise into different namespace. In this guide, we use namespace name timeplus. ### Prepare the values.yaml​ Copy and paste the following yaml snippet into values.yaml. Then make changes to better fit your need. 1. Update the storage class name, size and selector accordingly. Please check the Planning capacity section for storage recommendations. You can check available storage class on your cluster by running kubectl get storageclass. 2. Update defaultAdminPassword. This is the password for the default admin user proton, which is used internally in the system. 3. Review and update the replicas. Set it to 3 to setup a cluster with 3 timeplusd nodes. Set it to 1 to setup a single node for testing or small workload. Please note that you cannot change the number of replicas after the deployment. 4. Update the resources and make sure your cluster has enough CPU and memory to run the stack. By default each timeplusd pod requires 2 cores and 4GB memory. However, you'd better to have at least 8 cores and 20Gi memory for each node to make sure Timeplus Enterprise works well under small to medium workload. 5. Optionally refer to Configuration Guide and add other configurations. ### Install Helm chart​ In this step, we install the helm chart using release name timeplus. You should see following output within a couple of minutes. To make sure everything is up and running, run kubectl get pods -n $NS to check the stack status If any of the pod is in error status, you can try to use kubectl describe pod -n $NS to debug. For more details, please refer to Troubleshooting section. Starting from Timeplus Enterprise 2.9, the timeplus-web pod is no longer available. ### Expose the Timeplus Console​ There are different ways to expose the services of Timeplus stack. In this step, we use port forward of kubectl to get a quick access. Run kubectl port-forward svc/timeplus-appserver 8000:8000 -n $NS --address 0.0.0.0 and then open the address http://localhost:8000 in your browser to visit Timeplus Console web UI. As long as you are able to access the UI, you are now ready to explore the Timeplus Enterprise. ### Offline installation​ The above quick start guide assume there is network access to the Docker Hub to pull all required images. In the case there is no access to the Docker Hub, users need import required images into Kubernetes cluster first. You can run: 1. Run helm template -f ./values.yaml timeplus/timeplus-enterprise | grep image: | cut -d ":" -f2,3 | sort | uniq | sed 's/"//g' to list all required images. 2. Use docker save to save the images locally. Please refer to https://docs.docker.com/reference/cli/docker/image/save/. 3. Upload the images to your Kubernetes image registry. Note, you may need update the imageRegistry in values.yaml to point to your own Kubernetes registry. ### Uninstall and cleanup​ To uninstall the helm release, just run helm -n $NS uninstall $RELEASE to uninstall it. Please note, by default, all the PVCs will not be deleted. You can use kubectl get pvc -n $NS and kubectl delete pvc -n $NS to manually delete them. You can run kubectl delete namespace $NS to delete all PVCs and the namespace. ## Go Production​ ### Planning capacity​ This section provides recommendations for sizing your Timeplus Enterprise deployment. The actual resource requirements may vary based on your specific use case and workload. The timeplusd component is the core of the Timeplus Enterprise stack. It requires significant CPU and memory resources to handle data processing and queries. It is highly recommended to run timeplusd dedicatedly on the node. For small to medium-sized deployment, you may consider the following cluster configuration as the start point: - 3 nodes with: 16 cores each 32 Gi memory each 500Gi storage with iops > 3000 each - 16 cores each - 32 Gi memory each - 500Gi storage with iops > 3000 each - 16 cores each - 32 Gi memory each - 500Gi storage with iops > 3000 each You cannot change the number of nodes after the deployment. If your development environment is a single-node Timeplus Enterprise, and you want to setup a multi-node cluster, you need to install a new cluster and migrate the data via timeplus migrate CLI. A sample values.yaml configuration: ## Operations​ ### Update Configuration​ After the installation, you can further customize the configuration by updating the values.yaml. Please refer to Configuration Guide. Once the values.yaml is ready, apply this via: ### Update PV size for timeplusd​ Due to the limitation of Kubernetes Statefulset, you will need to manually update the PV size for timeplusd. Notice that this will cause downtime of Timeplus Enterprise. 1. Make sure the global.pvcDeleteOnStsDelete is not set or is set to be false. You can double check this by running command kubectl -n get sts timeplusd -ojsonpath='{.spec.persistentVolumeClaimRetentionPolicy}' and make sure both whenDeleted and whenScaled are retain. This is extremely important otherwise your PV may be deleted and all the data will be lost. 2. Run kubectl -n delete sts/timeplusd to temporarily delete the statefulset. Wait until all timeplusd pods are terminated. This step is necessary to workaround the Kubernetes limitation. 3. Run kubectl -n get pvc to list all the PVCs and their corresponding PVs. For each PV you want to resize, run command kubectl -n edit pvc to update the spec.resources.requests.storage. Notice that all timeplusd replicas need to have the same storage size so please make sure all updated PVCs have the same storage size. 4. Run kubectl get pv -o=jsonpath='{.spec.capacity.storage}' to make sure all corresponding PVs have been updated. It takes a while before Kubernetes update the capacity field of the PVC so as long as you can see the underlying storage size gets updated, you can process to the next step. 5. Update the timeplusd.storage.stream.size and/or timeplusd.storage.stream.history.size in values.yaml that you used to deploy Timeplus Enterprise. 6. Run helm upgrade command to upgrade the deployment. New statefulset will be created to pick up the PV size changes. ### Upgrade Timeplus Enterprise​ #### Do not attempt to upgrade across multiple major chart versions at a time​ This helm chart follows Semantic Versioning. It is always the best practice to upgrade one major chart version at a time even if the breaking change doesn't impact your deployment. #### Check if there is an incompatible breaking change needing manual actions​ Each major chart version contains a new major Timeplus Enterprise version. If you are not going to upgrade the major version, you can just go ahead to run the helm upgrade command. Otherwise, please check: 1. The release notes of Timeplus Enterprise to confirm the target version can be upgraded in-place, by reusing the current data and configuration. For example 2.3 and 2.4 are incompatible and you have to use migration tools. 2. The upgrade guide of helm chart. You may need to modify your values.yaml according to the guide before upgrade. #### Run helm upgrade​ If you confirm you can upgrade to the new version, you can run the following commands to upgrade: ### Prometheus metrics​ Timeplus Enterprise exposes its metrics in Prometheus format to allow monitoring the cluster status. It is recommended to configure your existing Prometheus metrics collector such as Grafana Agent or Vector to scrape metrics from timeplusd pods directly. The metrics of timeplusd are exposed at :9363/metrics. You will need to collect the metrics from all timeplusd pods. For example, if your Timeplus Enterprise is installed in my_ns namespace, you can configure the collector to collect metrics from - timeplusd-0.timeplusd-svc.my_ns.svc.cluster.local:9363/metrics - timeplusd-1.timeplusd-svc.my_ns.svc.cluster.local:9363/metrics - timeplusd-2.timeplusd-svc.my_ns.svc.cluster.local:9363/metrics ... (content truncated) ## Kafka-connect - [Push data to Timeplus via Kafka Connect | Timeplus](https://docs.timeplus.com/kafka-connect): Kafka Connect is a framework for connecting Kafka with external systems such as databases, key-value stores, search indexes, and file systems, using so-called Connectors. - Integrations - Third-party Tools - Push data to Timeplus via Kafka Connect # Push data to Timeplus via Kafka Connect Kafka Connect is a framework for connecting Kafka with external systems such as databases, key-value stores, search indexes, and file systems, using so-called Connectors. Kafka Connectors are ready-to-use components, which can help data teams to import data from external systems into Kafka topics and export data from Kafka topics into external systems. A Kafka Connect plugin for Timeplus is provided to continuously send data from Kafka topics to self-managed Timeplus deployment, no matter if you run Kafka in the cloud or on-prem. Depending on how you run Kafka (with open-source Kafka, Confluent Platform, Confluent Cloud, or Redpanda), you can check the corresponding documentation to set up the Kafka Connect. ## Setup with Apache Kafka​ For example, if you are using open-source Kafka, please check https://kafka.apache.org/documentation.html#connect. As a reference, the step-by-step instructions to setup Kafka, Kafka Connect and the Kafka Connect plugin for Timeplus are: 1. Make sure you have Java installed, say openjdk 17.0.5. 2. Download the latest Kafka binary from https://kafka.apache.org/downloads, say kafka_2.13-3.3.1.tgz. 3. Unarchive the file and open a terminal window and change directory to this folder. 4. Start the ZooKeeper service via bin/zookeeper-server-start.sh config/zookeeper.properties 5. Open another terminal session and start the Kafka broker service via bin/kafka-server-start.sh config/server.properties. 6. Open another terminal session and create a topic bin/kafka-topics.sh --create --topic my_topic --bootstrap-server localhost:9092 7. Download the latest kafka-connect-timeplus JAR file and put it in a new folder, say kakfa-connect-jars. 8. Edit the config/connect-standalone.properties file and uncomment the last line and point it to your folder, e.g. plugin.path=/Users/name/Dev/kakfa-connect-jars 9. Create an API key in your Timeplus workspace and create a timeplus-sink.properties like this: 1. Start the Kafka Connect service with the Timeplus Kafka Connect plugin via bin/connect-standalone.sh config/connect-standalone.properties config/timeplus-sink.properties It will move data in my_topic to the data_from_kakfa stream in the remote Timeplus. (You can produce sample event via kcat -P -b localhost:9092 -t my_topic enter some lines and produce the data via Ctrl+D) ## Setup with Confluent Platform​ 1. Make sure you have Java installed, only 1.8 or 1.11 is supported. 2. Install Confluent Platform per official documentation. Download the zip file from https://github.com/timeplus-io/kafka-connect-timeplus/releases. Then install it with confluent-hub install /path/to/timeplus-kafka-timeplus-connector-sink-version.zip 3. Start the Confluent Platform via confluent local services start 4. Access the Control Center via http://localhost:9021/ and create a topic say my_topic 5. Choose Connect menu item and click the Add connector button. Choose the TimeplusSinkConnector tile, and input the settings as: Topics: my_topic Name: any name is okay Tasks max: 1 Key and Value converter class: org.apache.kafka.connect.storage.StringConverter Scroll down to the Timeplus section and set the Timeplus server address, workspace, API Key, etc. Click Next, you can preview the JSON configuration as these: { "name": "TimeplusSink1", "config": { "name": "TimeplusSink1", "connector.class": "com.timeplus.kafkaconnect.TimeplusSinkConnector", "tasks.max": "1", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "value.converter": "org.apache.kafka.connect.storage.StringConverter", "topics": "my_topic", "timeplus.sink.address": "https://host", "timeplus.sink.workspace": "abc123", "timeplus.sink.apikey": "60_char_API_Key", "timeplus.sink.createStream": "false", "timeplus.sink.stream": "data_from_kafka", "timeplus.sink.dataFormat": "raw" }} 6. Topics: my_topic 7. Name: any name is okay 8. Tasks max: 1 9. Key and Value converter class: org.apache.kafka.connect.storage.StringConverter 10. Scroll down to the Timeplus section and set the Timeplus server address, workspace, API Key, etc. 11. Click Next, you can preview the JSON configuration as these: { "name": "TimeplusSink1", "config": { "name": "TimeplusSink1", "connector.class": "com.timeplus.kafkaconnect.TimeplusSinkConnector", "tasks.max": "1", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "value.converter": "org.apache.kafka.connect.storage.StringConverter", "topics": "my_topic", "timeplus.sink.address": "https://host", "timeplus.sink.workspace": "abc123", "timeplus.sink.apikey": "60_char_API_Key", "timeplus.sink.createStream": "false", "timeplus.sink.stream": "data_from_kafka", "timeplus.sink.dataFormat": "raw" }} 12. Click Launch button, in a few seconds, you should see the connector is running. 13. You can open the Query Console in Timeplus and run a streaming query like SELECT * FROM data_from_kafka. Then create a message in Confluent Control Center (choose Topics, select the my_topic, choose Messages tab, and produce a sample message). Almost in the same time, the message will appear in the Timeplus Console. Make sure you have Java installed, only 1.8 or 1.11 is supported. Install Confluent Platform per official documentation. Download the zip file from https://github.com/timeplus-io/kafka-connect-timeplus/releases. Then install it with confluent-hub install /path/to/timeplus-kafka-timeplus-connector-sink-version.zip Start the Confluent Platform via confluent local services start Access the Control Center via http://localhost:9021/ and create a topic say my_topic Choose Connect menu item and click the Add connector button. Choose the TimeplusSinkConnector tile, and input the settings as: 1. Topics: my_topic 2. Name: any name is okay 3. Tasks max: 1 4. Key and Value converter class: org.apache.kafka.connect.storage.StringConverter 5. Scroll down to the Timeplus section and set the Timeplus server address, workspace, API Key, etc. 6. Click Next, you can preview the JSON configuration as these: { "name": "TimeplusSink1", "config": { "name": "TimeplusSink1", "connector.class": "com.timeplus.kafkaconnect.TimeplusSinkConnector", "tasks.max": "1", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "value.converter": "org.apache.kafka.connect.storage.StringConverter", "topics": "my_topic", "timeplus.sink.address": "https://host", "timeplus.sink.workspace": "abc123", "timeplus.sink.apikey": "60_char_API_Key", "timeplus.sink.createStream": "false", "timeplus.sink.stream": "data_from_kafka", "timeplus.sink.dataFormat": "raw" }} Topics: my_topic Name: any name is okay Tasks max: 1 Key and Value converter class: org.apache.kafka.connect.storage.StringConverter Scroll down to the Timeplus section and set the Timeplus server address, workspace, API Key, etc. Click Next, you can preview the JSON configuration as these: Click Launch button, in a few seconds, you should see the connector is running. You can open the Query Console in Timeplus and run a streaming query like SELECT * FROM data_from_kafka. Then create a message in Confluent Control Center (choose Topics, select the my_topic, choose Messages tab, and produce a sample message). Almost in the same time, the message will appear in the Timeplus Console. - Setup with Apache Kafka - Setup with Confluent Platform ## Log-stream - [Log Files | Timeplus](https://docs.timeplus.com/log-stream): You can use Timeplus as a lightweight and high-performance tool for log analysis. Please check the blog for more details. - Core Features - External Streams & Tables - External Streams - Log Files # Log Files You can use Timeplus as a lightweight and high-performance tool for log analysis. Please check the blog for more details. ## Syntax​ Create an external stream with the log type to monitor log files, e.g. The required settings: - log_files - log_dir - timestamp_regex - row_delimiter. Only 1 capturing group is expected in the regex. - Syntax ## Marimo - [Query Kafka with SQL using marimo Python Notebook | Timeplus](https://docs.timeplus.com/marimo): This tutorial demonstrates how to query Apache Kafka with SQL using Timeplus Proton and marimo notebooks. You'll learn how to set up a lightweight environment to process and visualize streaming data from Kafka without complex infrastructure requirements. - Guides & Tutorials - Query Kafka with SQL using marimo Python Notebook # Query Kafka with SQL using marimo Python Notebook This tutorial demonstrates how to query Apache Kafka with SQL using Timeplus Proton and marimo notebooks. You'll learn how to set up a lightweight environment to process and visualize streaming data from Kafka without complex infrastructure requirements. ## Key Highlights of This Approach​ - Lightweight Setup: No Docker, JVM, or complex Python dependencies - Real Data: Processing real-time GitHub events from Kafka - Efficient Processing: Processing millions of Kafka messages without local storage - Interactive Visualization: Charts update automatically and support interactive filtering ## Quick Start​ Run the following commands to set up the environment: These commands will: 1. Download uv (a Rust-based Python manager) 2. Download Timeplus Proton (an OSS streaming database in C++) 3. Run a marimo notebook with all dependencies auto-installed ## Detailed Setup and Tutorial Steps​ ### Step 1: Introduction to marimo​ marimo is a modern Python notebook that offers several advantages over traditional Jupyter notebooks: - Pure Python code (no .ipynb JSON) - Git-friendly format - Integrated dependency management with uv The first part of the github.py notebook declares its dependencies: When you run uvx marimo run --sandbox github.py, uv will check and download Python 3.13, create a new Python virtual environment, and install all required dependencies. ### Step 2: Start the Timeplus Proton Server​ To connect to Kafka, you'll first need to run Timeplus Proton: This downloads the single binary of Timeplus Proton to your current folder and starts the server. To connect to Timeplus Proton in the marimo notebook, you can use the web UI or write Python code: ### Step 3: Connect to Kafka​ To query a Kafka topic using SQL, create an external stream in Timeplus Proton: Notes: - The Kafka topic contains live GitHub events data in JSON format - If you have too many columns or variable schema, you can create a stream with a single string column and parse the JSON at query time ### Step 4: Run Your First SQL Query Against Kafka​ Let's count all messages in the Kafka topic: This query is optimized by Timeplus to check the offset difference between the first and last Kafka message, providing a fast count without scanning all data. ### Step 5: Visualize Data with marimo​ To visualize the count result in marimo, update the output variable of the SQL cell to cntdf and create a stat widget: ### Step 6: Implement Auto-Refresh​ To automatically refresh the count, add a refresh widget: Then modify your SQL to reference this refresh widget: By adding the comment that references cnt_refresh.value, the SQL query will re-run whenever the refresh state changes. To show the delta between counts: ### Step 7: Create Interactive Charts​ Let's create interactive charts to visualize GitHub event data: 1. Query for top event types: 1. Query for top repositories by event type: 1. Handle selections to filter the bar chart based on pie chart clicks: 1. Create the charts using Altair: 1. Arrange elements in a layout: ## Advanced Features Applied​ - Time-Based Filtering: Use seek_to='-{range.value}m' to analyze data from a specific time window - Optimized Aggregations: Use top_k for efficient ranking operations - Interactive Filtering: Link charts for dynamic data exploration ## Conclusion​ This tutorial demonstrates how to query and visualize Kafka data using SQL with Timeplus Proton and marimo notebooks. The approach provides a lightweight, SQL-native way to work with streaming data without the overhead of traditional big data systems. For more information and examples, visit the Timeplus GitHub repository. - Key Highlights of This Approach - Quick Start - Detailed Setup and Tutorial StepsStep 1: Introduction to marimoStep 2: Start the Timeplus Proton ServerStep 3: Connect to KafkaStep 4: Run Your First SQL Query Against KafkaStep 5: Visualize Data with marimoStep 6: Implement Auto-RefreshStep 7: Create Interactive Charts - Step 1: Introduction to marimo - Step 2: Start the Timeplus Proton Server - Step 3: Connect to Kafka - Step 4: Run Your First SQL Query Against Kafka - Step 5: Visualize Data with marimo - Step 6: Implement Auto-Refresh - Step 7: Create Interactive Charts - Advanced Features Applied - Conclusion - Step 1: Introduction to marimo - Step 2: Start the Timeplus Proton Server - Step 3: Connect to Kafka - Step 4: Run Your First SQL Query Against Kafka - Step 5: Visualize Data with marimo - Step 6: Implement Auto-Refresh - Step 7: Create Interactive Charts ## Mongo-external - [MongoDB External Table | Timeplus](https://docs.timeplus.com/mongo-external): Since Timeplus Enterprise v2.9 and v2.8.2, you can send data to and read data from MongoDB collections via the MongoDB External Table. - Core Features - External Streams & Tables - External Tables - MongoDB # MongoDB External Table Since Timeplus Enterprise v2.9 and v2.8.2, you can send data to and read data from MongoDB collections via the MongoDB External Table. ## CREATE EXTERNAL TABLE​ To create an external table for MongoDB, you can run the following DDL SQL: For the full list of settings, see the DDL Settings section. ### Examples​ #### Write to Self-Hosting MongoDB​ Assuming you have created an index students in a deployment of MongoDB, you can create the following external stream to write data to the index. Then you can insert data via a materialized view or just #### Write to MongoDB Atlas​ The MongoDB Atlas by default shows the connection URI in the format mongodb+srv://:@/. The mongodb+srv protocol is not supported yet. Please use the mongodb protocol instead. You can find the connection URI in the MongoDB Atlas UI, and choose C++ driver and 3.1.x as the version. The connection URI will look like this: #### Read From MongoDB​ You can also read data from MongoDB collections via the external table. The following example reads data from the students collection in MongoDB. You can also filter the data using a WHERE clause or apply aggregations on the data. Timeplus will try best to push down the filter and aggregation operations to the MongoDB, so that only the necessary data is transferred from MongoDB to Timeplus. ### DDL Settings​ #### type​ The type of the external stream. The value must be mongodb to send data to MongoDB. #### uri​ The endpoint of the MongoDB service. #### collection​ The name of the MongoDB collection to read/write data from/to. #### connection_options​ MongoDB connection string options as a URL formatted string. e.g. 'authSource=admin&ssl=true'. #### oid_columns​ A comma-separated list of columns that should be treated as oid in the WHERE clause. Default to _id. ### Query Settings​ #### mongodb_throw_on_unsupported_query​ By default this setting is true. While querying the MongoDB external table with SQL, if the query contains GROUP BY, HAVING or other aggregations, Timeplus will throw exceptions. Set this to false or 0 to disable this behavior, and Timeplus will read full table data from MongoDB and execute the query in Timeplus. For example: ## DROP EXTERNAL TABLE​ - CREATE EXTERNAL TABLEExamplesWrite to Self-Hosting MongoDBWrite to MongoDB AtlasRead From MongoDBDDL Settingstypeuricollectionconnection_optionsoid_columnsQuery Settingsmongodb_throw_on_unsupported_query - ExamplesWrite to Self-Hosting MongoDBWrite to MongoDB AtlasRead From MongoDB - Write to Self-Hosting MongoDB - Write to MongoDB Atlas - Read From MongoDB - DDL Settingstypeuricollectionconnection_optionsoid_columns - type - uri - collection - connection_options - oid_columns - Query Settingsmongodb_throw_on_unsupported_query - mongodb_throw_on_unsupported_query - DROP EXTERNAL TABLE - ExamplesWrite to Self-Hosting MongoDBWrite to MongoDB AtlasRead From MongoDB - Write to Self-Hosting MongoDB - Write to MongoDB Atlas - Read From MongoDB - DDL Settingstypeuricollectionconnection_optionsoid_columns - type - uri - collection - connection_options - oid_columns - Query Settingsmongodb_throw_on_unsupported_query - mongodb_throw_on_unsupported_query - Write to Self-Hosting MongoDB - Write to MongoDB Atlas - Read From MongoDB - type - uri - collection - connection_options - oid_columns - mongodb_throw_on_unsupported_query ## Mutable-stream - [Mutable Stream | Timeplus](https://docs.timeplus.com/mutable-stream): This type of stream is only available in Timeplus Enterprise, with high performance query and UPSERT (UPDATE or INSERT). Starting from Timeplus Enterprise 2.7, mutable streams are enhanced to support DELETE operation. - Core Features - Streams - Mutable Stream # Mutable Stream This type of stream is only available in Timeplus Enterprise, with high performance query and UPSERT (UPDATE or INSERT). Starting from Timeplus Enterprise 2.7, mutable streams are enhanced to support DELETE operation. As the name implies, the data in the stream is mutable. Value with the same primary key(s) will be overwritten. The primary use case of mutable streams is serving as the lookup/dimensional data in Streaming JOIN, supporting millions or even billions of unique keys. You can also use mutable streams as the "fact table" to efficiently do range queries or filtering for denormalized data model, a.k.a. OBT (One Big Table). Learn more about why we introduced Mutable Streams by checking this blog. ## CREATE​ Since Timeplus Enterprise 2.7, if you create a mutable stream with low_cardinality columns, the system will ignore the low_cardinality modifier to improve performance. Learn more. PARTITION BY, ORDER BY or SAMPLE BY clauses are not allowed while creating the mutable stream. Since Timeplus Enterprise 2.8.2, the following features are added: - you can set coalesced (default to false). If it's true and the insert data only contains partial columns in the WAL, the partial columns will merge with the existing rows. Learn more. - you can set ttl_seconds (default to -1). If it's set to a positive value, then data with primary key older than the ttl_seconds will be scheduled to be pruned in the next key compaction cycle. Learn more. - you can set version_column to make sure only rows with higher value of the version_column will override the rows with same primary key. This setting can work with or without coalesced. ## INSERT​ You can insert data to the mutable stream with the following SQL: ## UPDATE​ To update the data in the mutable stream, you can insert the data with the same primary key(s). The new data will overwrite the existing data. ## DELETE​ Starting from Timeplus Enterprise 2.7, you can delete data from the mutable stream. It's recommended to use the primary key(s) in the condition to delete the data efficiently. You can also use the secondary index or other columns in the condition. ## SELECT​ You can query the mutable stream with the following SQL: Mutable streams can be used in JOINs or as the source or cache for Dictionaries. ## Example​ ### Create a mutable stream​ Create the stream with the following SQL: Note: - The compound primary key is a combination of device_id, timestamp and the batch_id. Data with exactly the same value for those 3 columns will be overridden. - Searching data with any column in the primary key is very fast. - By default there is only 1 shard and no extra index or optimization. ### Load millions of rows​ You can use CREATE RANDOM STREAM and a Materialized View to generate data and send to the mutable stream. But since we are testing massive historical data with duplicated keys, we can also use INSERT INTO .. SELECT to load data. Depending on your hardware and server configuration, it may take a few seconds to add all data. ### Query the mutable stream​ When you query the mutable stream, Timeplus will read all historical data without any duplicated primary key. Sample output: You can filter data efficiently with any part of the primary key: Sample output: Another example: Sample output: You can also query the mutable stream in the streaming SQL. This will query all existing data and accept new incoming data. Mutable stream can also be used in JOINs. ## Advanced Settings​ ### Retention Policy for Historical Storage​ Like normal streams in Timeplus, mutable streams use both streaming storage and historical storage. New data are added to the streaming storage first, then continuously write to the historical data with deduplication/merging process. Starting from Timeplus Enterprise 2.9 (also backported to 2.8.2), you can set ttl_seconds on mutable streams. If the data's age (based on when the data is inserted, not _tp_time or particular columns) is older than this value, it is scheduled to be pruned in the next key compaction cycle. Default value is -1. Any value less than 0 means this feature is disabled. ### Retention Policy for Streaming Storage​ When you create the mutable stream, you can configure the maximum size of the streaming storage or Time-To-Live (TTL). For example, if you want to keep up to 8GB or half an hour data in the streaming storage, you can add the following settings in the DDL: ### Secondary Index​ Regardless of whether you choose a single column or multiple columns as the primary key(s), Timeplus will build an index for those columns. Queries with filtering on these columns will leverage the index to boost performance and minimize data scanning. For other columns, if they are frequently filtered, you can also define secondary indexes for them. For example: When you query data with filters on those columns, Timeplus will automatically leverage the indexed data to improve query performance. ### Column Family​ For One-Big-Table(OBT) or extra wide table with dozens or even hundreds of columns, it's not recommended to run SELECT * FROM .., unless you need to export data. More commonly, you need to query a subset of the columns in different use cases. For those columns which are commonly queried together, you can define column families to group them, so that data for those columns will be saved together in the same file. Properly defining column families can optimize the disk i/o and avoid reading unnecessary data files. Please note, one column can appear in up to one column family. The columns as primary keys are in a special column family. There should be no overlap for the column families or primary keys. Taking the previous device_metrics as an example, the lat and lon are commonly queried together. You can define a column family for them. ### Multi-shard​ Another optimization is to create multiple shards to partition the data when it scales. For example, to create 3 shards for device_metrics: ### Coalesced and Versioned Mutable Stream​ For a mutable stream with many columns, there are some cases that only some columns are updated over time. Create a mutable stream with Column Family and coalesced=true setting to enable the partial merge. For example, given a mutable stream: If we insert one row with m1=1: Query the mutable stream. You will get one row. Then insert the other row with the same primary key and m2=2. Query it again with You will see one row with m1 and m2 updated and other columns in the default value. Compared to the Versioned Stream, coalesced mutable streams don't require you to set all column values when you update a primary key. You can also set version_column to the column name to indicate which column with the version number. Say there are updates for the same primary key, v as the version_column, the first update is "v=1,p=1,m=1" and the second update is "v=2,p=1,m=2". For some reasons, if Timeplus receives the second update first, then when it gets the "v=1,p=1,m=1", since the version is 1, lower than the current version, so this update will be reject and we keep the latest update as "v=2,p=1,m=2". This is beneficial specially in distributed environment with potential out of order events. ## Performance Tuning​ If you are facing performance challenges with massive data in mutable streams, please consider adding secondary indexes, column families and use multiple shards. ### key_space_full_scan_threads​ Additionally, you can configure the number of threads for full-scan of the key space at the query time using the key_space_full_scan_threads setting, e.g.: - CREATE - INSERT - UPDATE - DELETE - SELECT - ExampleCreate a mutable streamLoad millions of rowsQuery the mutable stream - Create a mutable stream - Load millions of rows - Query the mutable stream - Advanced SettingsRetention Policy for Historical StorageRetention Policy for Streaming StorageSecondary IndexColumn FamilyMulti-shardCoalesced and Versioned Mutable Stream - Retention Policy for Historical Storage - Retention Policy for Streaming Storage - Secondary Index - Column Family - Multi-shard - Coalesced and Versioned Mutable Stream - Performance Tuningkey_space_full_scan_threads - key_space_full_scan_threads - Create a mutable stream - Load millions of rows - Query the mutable stream - Retention Policy for Historical Storage - Retention Policy for Streaming Storage - Secondary Index - Column Family - Multi-shard - Coalesced and Versioned Mutable Stream - key_space_full_scan_threads ## Mysql-external-table - [MySQL External Table | Timeplus](https://docs.timeplus.com/mysql-external-table): Timeplus can read or write MySQL tables directly. This unlocks a set of new use cases, such as - Core Features - External Streams & Tables - External Tables - MySQL # MySQL External Table Timeplus can read or write MySQL tables directly. This unlocks a set of new use cases, such as - Use Timeplus to efficiently process real-time data in Kafka/Redpanda, apply flat transformation or stateful aggregation, then write the data to the local or remote MySQL for further analysis or visualization. - Enrich the live data with the static or slow-changing data in MySQL. Apply streaming JOIN. - Use Timeplus to query historical or recent data in MySQL. This integration is done by introducing "External Table" in Timeplus. Similar to External Stream, there is no data persisted in Timeplus. However, since the data in MySQL is in the form of table, not data stream, so we call this as External Table. Currently, we support MySQL and ClickHouse. In the roadmap, we will support more integration by introducing other types of External Table. ## CREATE EXTERNAL TABLE​ ### Syntax​ The required settings are type and address. For other settings, the default values are - 'default' for user - '' (empty string) for password - 'default' for database - If you omit the table name, it will use the name of the external table The config_file setting is available since Timeplus Enterprise 2.7. You can specify the path to a file that contains the configuration settings. The file should be in the format of key=value pairs, one pair per line. You can set the MySQL user and password in the file. Please follow the example in Kafka External Stream. You don't need to specify the columns, since the table schema will be fetched from the MySQL server. Once the external table is created successfully, you can run the following SQL to list the columns: The data types in the output will be Timeplus data types, such as uint8, instead of MySQL data type. Timeplus maintains a mapping for those types. Learn more. You can define the external table and use it to read data from the MySQL table, or write to it. ### Connect to a local MySQL​ Example SQL to connect to a local MySQL server without password: ## Read data from MySQL​ Once the external table is created successfully, it means Timeplus can connect to the MySQL server and fetch the table schema. You can query it via the regular select .. from table_name. Please note, in the current implementation, all rows will be fetched from MySQL to Timeplus, with the selected columns. Then Timeplus applies the SQL functions and LIMIT n locally. It's not recommended to run SELECT * for a large MySQL table. Also note, use the Timeplus function names when you query the external table, such as to_int, instead of MySQL's naming convention, e.g. CONVERT. In current implementation, the SQL functions are applied in Timeplus engine. We plan to support some function push-down to MySQL in future versions. Limitations: 1. tumble/hop/session/table functions are not supported for External Table (coming soon) 2. scalar or aggregation functions are performed by Timeplus, not the remote MySQL 3. LIMIT n is performed by Timeplus, not the remote MySQL ## Write data to MySQL​ You can run regular INSERT INTO to add data to MySQL table. However it's more common to use a Materialized View to send the streaming SQL results to MySQL. Say you have created an external table mysql_table. You can create a materialized view to read Kafka data(via an external stream) and transform/aggregate the data and send to the external table: ### Batching Settings​ In Timeplus Enterprise, additional performance tuning settings are available, such as - max_insert_block_size - The maximum block size for insertion, i.e. maximum number of rows in a batch. Default value: 65409 - max_insert_block_bytes - The maximum size in bytes of block for insertion. Default value: 1 MiB. - insert_block_timeout_ms - The maximum time in milliseconds for constructing a block(a block) for insertion. Increasing the value gives greater possibility to create bigger blocks (limited by max_insert_block_bytes and max_insert_block_size), but also increases latency. Negative numbers means no timeout. Default value: 500. ## Supported data types​ All MySQL data types are supported in the external table. While reading or writing data, Timeplus applies a data type mapping, such as converting Timeplus' uint8 to MySQL's SMALLINT. If you find anything wrong with the data type, please let us know. - CREATE EXTERNAL TABLESyntaxConnect to a local MySQL - Syntax - Connect to a local MySQL - Read data from MySQL - Write data to MySQLBatching Settings - Batching Settings - Supported data types - Syntax - Connect to a local MySQL - Batching Settings ## Pg-external-table - [PostgreSQL External Table | Timeplus](https://docs.timeplus.com/pg-external-table): Timeplus can read or write PostgreSQL tables directly. This unlocks a set of new use cases, such as - Core Features - External Streams & Tables - External Tables - PostgreSQL # PostgreSQL External Table Timeplus can read or write PostgreSQL tables directly. This unlocks a set of new use cases, such as - Use Timeplus to efficiently process real-time data in Kafka/Redpanda, apply flat transformation or stateful aggregation, then write the data to the local or remote PostgreSQL for further analysis or visualization. - Enrich the live data with the static or slow-changing data in PostgreSQL. Apply streaming JOIN. - Use Timeplus to query historical or recent data in PostgreSQL. This integration is done by introducing "External Table" in Timeplus. Similar to External Stream, there is no data persisted in Timeplus. However, since the data in PostgreSQL is in the form of table, not data stream, so we call this as External Table. Currently, we support S3, MySQL, PostgreSQL and ClickHouse. In the roadmap, we will support more integration by introducing other types of External Table. ## CREATE EXTERNAL TABLE​ ### Syntax​ The required settings are type and address. For other settings, the default values are - 'default' for user - '' (empty string) for password - 'default' for database - 'false' for secure - If you omit the table name, it will use the name of the external table The config_file setting is available since Timeplus Enterprise 2.7. You can specify the path to a file that contains the configuration settings. The file should be in the format of key=value pairs, one pair per line. You can set the PostgreSQL user and password in the file. Please follow the example in Kafka External Stream. You don't need to specify the columns, since the table schema will be fetched from the PostgreSQL server. Once the external table is created successfully, you can run the following SQL to list the columns: The data types in the output will be Timeplus data types, such as uint8, instead of PostgreSQL data type. Timeplus maintains a mapping for those types. Learn more. You can define the external table and use it to read data from the PostgreSQL table, or write to it. ### Connect to a local PostgreSQL​ You can use the following command to start a local PostgreSQL via Docker: Then open a new terminal and run the following command to connect to the PostgreSQL server: Create a table and add some rows: In Timeplus, you can create an external table to read data from the PostgreSQL table: Then query the table: ### Connect to Aiven for PostgreSQL​ Example SQL to connect to Aiven for PostgreSQL: ## Read data from PostgreSQL​ Once the external table is created successfully, it means Timeplus can connect to the PostgreSQL server and fetch the table schema. You can query it via the regular select .. from table_name. Please note, in the current implementation, all rows will be fetched from PostgreSQL to Timeplus, with the selected columns. Then Timeplus applies the SQL functions and LIMIT n locally. It's not recommended to run SELECT * for a large PostgreSQL table. Also note, use the Timeplus function names when you query the external table, such as to_int, instead of PostgreSQL's naming convention, e.g. CONVERT. In current implementation, the SQL functions are applied in Timeplus engine. We plan to support some function push-down to PostgreSQL in future versions. Limitations: 1. tumble/hop/session/table functions are not supported for External Table (coming soon) 2. scalar or aggregation functions are performed by Timeplus, not the remote PostgreSQL 3. LIMIT n is performed by Timeplus, not the remote PostgreSQL ## Write data to PostgreSQL​ You can run regular INSERT INTO to add data to PostgreSQL table, such as: Please note, since the price column is in float8 type, in Timeplus, you need to insert via 90.99, instead of a string "90.99" as in PostgreSQL INSERT command. However it's more common to use a Materialized View in Timeplus to send the streaming SQL results to PostgreSQL. Say you have created an external table pg_table. You can create a materialized view to read Kafka data(via an external stream) and transform/aggregate the data and send to the external table: ### Batching Settings​ In Timeplus Enterprise, additional performance tuning settings are available, such as - max_insert_block_size - The maximum block size for insertion, i.e. maximum number of rows in a batch. Default value: 65409 - max_insert_block_bytes - The maximum size in bytes of block for insertion. Default value: 1 MiB. - insert_block_timeout_ms - The maximum time in milliseconds for constructing a block(a block) for insertion. Increasing the value gives greater possibility to create bigger blocks (limited by max_insert_block_bytes and max_insert_block_size), but also increases latency. Negative numbers means no timeout. Default value: 500. ## Supported data types​ All PostgreSQL data types are supported in the external table. While reading or writing data, Timeplus applies a data type mapping, such as converting Timeplus' uint8 to PostgreSQL's SMALLINT. If you find anything wrong with the data type, please let us know. - CREATE EXTERNAL TABLESyntaxConnect to a local PostgreSQLConnect to Aiven for PostgreSQL - Syntax - Connect to a local PostgreSQL - Connect to Aiven for PostgreSQL - Read data from PostgreSQL - Write data to PostgreSQLBatching Settings - Batching Settings - Supported data types - Syntax - Connect to a local PostgreSQL - Connect to Aiven for PostgreSQL - Batching Settings ## Private-beta-1 - [Timeplus Cloud Private Beta 1 | Timeplus](https://docs.timeplus.com/private-beta-1): We are thrilled to launch the first private beta of Timeplus cloud release. A lot of cool features and unlimited possibilities. We will update the beta version from time to time and list key enhancements in this page. - Release Notes - Older Releases - Timeplus Cloud Private Beta 1 # Timeplus Cloud Private Beta 1 We are thrilled to launch the first private beta of Timeplus cloud release. A lot of cool features and unlimited possibilities. We will update the beta version from time to time and list key enhancements in this page. (in year 2022) ### Week of 8/1​ Last weekly release in Private Beta 1. Starting from August 8, we are transiting to Private Beta 2. Customers will be migrated to the new environments batch by batch. URLs to access beta tenants are changed from https://TENANT.beta.timeplus.com to https://beta.timeplus.cloud/TENANT - Streaming engine Added 2 geo related functions: point_in_polygon and geo_distance. - Added 2 geo related functions: point_in_polygon and geo_distance. - Source, sink, API and SDK Updated the layout of the "Sources" page to leave more space for the source definitions. - Updated the layout of the "Sources" page to leave more space for the source definitions. - UI improvements Added a visual indicator on the query tab if the query is running. Updated the error or page-not-found screens. - Added a visual indicator on the query tab if the query is running. - Updated the error or page-not-found screens. Streaming engine - Added 2 geo related functions: point_in_polygon and geo_distance. Source, sink, API and SDK - Updated the layout of the "Sources" page to leave more space for the source definitions. UI improvements - Added a visual indicator on the query tab if the query is running. - Updated the error or page-not-found screens. ### Week of 7/25​ - Streaming engine Enhanced json_extract_array function to return clean string values. select '{"a":1,"tags":["x","y"]}' as raw, json_extract_array(raw:tags)now returns [ "x", "y" ] , instead of [ "\"x\"", "\"y\"" ] in the previous releases. Added a new shortcut to access json arrays without having to use the json_extract_array function. The above query can be simplified as select '{"a":1,"tags":["x","y"]}' as raw, raw:tags[*] Refined typing system and logical comparisons return bool instead of uint8 - Enhanced json_extract_array function to return clean string values. select '{"a":1,"tags":["x","y"]}' as raw, json_extract_array(raw:tags)now returns [ "x", "y" ] , instead of [ "\"x\"", "\"y\"" ] in the previous releases. - Added a new shortcut to access json arrays without having to use the json_extract_array function. The above query can be simplified as select '{"a":1,"tags":["x","y"]}' as raw, raw:tags[*] - Refined typing system and logical comparisons return bool instead of uint8 - Source, sink, API and SDK All Timeplus sinks now use the {{.columnName}} syntax to access the column value in the sink title or body. Numbers and other primitive types are supported (previously only string columns are supported). Fixed an issue that canceled queries could be marked as finished. Fixed an issue that EPS(event per second) won't be shown if the query finishes too fast. - All Timeplus sinks now use the {{.columnName}} syntax to access the column value in the sink title or body. Numbers and other primitive types are supported (previously only string columns are supported). - Fixed an issue that canceled queries could be marked as finished. - Fixed an issue that EPS(event per second) won't be shown if the query finishes too fast. - UI improvements Added a new option in the 'Send data to..' dialog to send the results to a stream in the Timeplus tenant. Show the number of running queries when you create a new query tab. Enhanced the font colors. Enhanced the chart colors. - Added a new option in the 'Send data to..' dialog to send the results to a stream in the Timeplus tenant. - Show the number of running queries when you create a new query tab. - Enhanced the font colors. - Enhanced the chart colors. - Enhanced json_extract_array function to return clean string values. select '{"a":1,"tags":["x","y"]}' as raw, json_extract_array(raw:tags)now returns [ "x", "y" ] , instead of [ "\"x\"", "\"y\"" ] in the previous releases. - Added a new shortcut to access json arrays without having to use the json_extract_array function. The above query can be simplified as select '{"a":1,"tags":["x","y"]}' as raw, raw:tags[*] - Refined typing system and logical comparisons return bool instead of uint8 - All Timeplus sinks now use the {{.columnName}} syntax to access the column value in the sink title or body. Numbers and other primitive types are supported (previously only string columns are supported). - Fixed an issue that canceled queries could be marked as finished. - Fixed an issue that EPS(event per second) won't be shown if the query finishes too fast. - Added a new option in the 'Send data to..' dialog to send the results to a stream in the Timeplus tenant. - Show the number of running queries when you create a new query tab. - Enhanced the font colors. - Enhanced the chart colors. ### Week of 7/18​ - Streaming engine Refined the behavior of materialized views, to keep it consistent with the other Timeplus queries. SELECT * FROM table(a_materialized_view) will get all past results, instead of the recent one. Added the count_if function and unique_exact_if function to count the number of rows or unique value matching certain conditions. Added json_extract_keys function to get the keys for the JSON map object. Added the to_bool function to convert other types to bool Added is_nan, is_infinite, is_finite functions to detect the edge cases when a number column contains infinite numbers, etc. Added to_type_name function to show the data type name, mainly for troubleshooting purposes. - Refined the behavior of materialized views, to keep it consistent with the other Timeplus queries. SELECT * FROM table(a_materialized_view) will get all past results, instead of the recent one. - Added the count_if function and unique_exact_if function to count the number of rows or unique value matching certain conditions. - Added json_extract_keys function to get the keys for the JSON map object. - Added the to_bool function to convert other types to bool - Added is_nan, is_infinite, is_finite functions to detect the edge cases when a number column contains infinite numbers, etc. - Added to_type_name function to show the data type name, mainly for troubleshooting purposes. - Source, sink, API and SDK Updated the Python SDK to show the metrics - Updated the Python SDK to show the metrics - UI improvements Added new visualization types: bar chart and streaming table Enhanced the management page of sinks, to show the sink status, number of messages sent, failure count, etc. Enhanced the SQL editor to highlight the beginning/ending (),,[]. This could be very helpful for complex SQLs with nested function calls. - Added new visualization types: bar chart and streaming table - Enhanced the management page of sinks, to show the sink status, number of messages sent, failure count, etc. - Enhanced the SQL editor to highlight the beginning/ending (),,[]. This could be very helpful for complex SQLs with nested function calls. Streaming engine - Refined the behavior of materialized views, to keep it consistent with the other Timeplus queries. SELECT * FROM table(a_materialized_view) will get all past results, instead of the recent one. - Added the count_if function and unique_exact_if function to count the number of rows or unique value matching certain conditions. - Added json_extract_keys function to get the keys for the JSON map object. - Added the to_bool function to convert other types to bool - Added is_nan, is_infinite, is_finite functions to detect the edge cases when a number column contains infinite numbers, etc. - Added to_type_name function to show the data type name, mainly for troubleshooting purposes. Source, sink, API and SDK - Updated the Python SDK to show the metrics UI improvements - Added new visualization types: bar chart and streaming table - Enhanced the management page of sinks, to show the sink status, number of messages sent, failure count, etc. - Enhanced the SQL editor to highlight the beginning/ending (),,[]. This could be very helpful for complex SQLs with nested function calls. ### Week of 7/11​ - Streaming engine Fixed an issue of lags function to get a range of past results for the specific column Exposed the coalesce function to skip null value and return the first non-NULL value. Many functions expect the arguments cannot be NULL - Fixed an issue of lags function to get a range of past results for the specific column - Exposed the coalesce function to skip null value and return the first non-NULL value. Many functions expect the arguments cannot be NULL - Source, sink, API and SDK Updated the Python SDK to support the new source API and add authentication to websockets Added the optional description field for sinks - Updated the Python SDK to support the new source API and add authentication to websockets - Added the optional description field for sinks - UI improvements Show the newly-opened query tab such as 'Tab 1', or 'Tab 2', instead of 'Untitled' Able to delete the first query tab Consolidated various streamlit demos to a single URL Replaced alert API with sink API. ACTION-REQUIRED: please recreate the alert/sink after the upgrade - Show the newly-opened query tab such as 'Tab 1', or 'Tab 2', instead of 'Untitled' - Able to delete the first query tab - Consolidated various streamlit demos to a single URL - Replaced alert API with sink API. ACTION-REQUIRED: please recreate the alert/sink after the upgrade Streaming engine - Fixed an issue of lags function to get a range of past results for the specific column - Exposed the coalesce function to skip null value and return the first non-NULL value. Many functions expect the arguments cannot be NULL Source, sink, API and SDK ... (content truncated) ## Private-beta-2 - [Timeplus Cloud Private Beta 2 | Timeplus](https://docs.timeplus.com/private-beta-2): We are thrilled to launch the second private beta of Timeplus cloud release. Comparing to the Private Beta 1, most of the backend and frontend changes are incremental enhancements, except the entry point is changed from https//beta.timeplus.cloud/TENANT - Release Notes - Older Releases - Timeplus Cloud Private Beta 2 # Timeplus Cloud Private Beta 2 We are thrilled to launch the second private beta of Timeplus cloud release. Comparing to the Private Beta 1, most of the backend and frontend changes are incremental enhancements, except the entry point is changed from https://TENANT.beta.timeplus.com to https://beta.timeplus.cloud/TENANT We will update the beta version from time to time and list key enhancements in this page. (in year 2022) ### Biweekly Update 9/19-9/30​ - Streaming engine Enhanced dedup function to only cache the unique keys for a given time period. This is useful to suppress the same alerts in the short time period. Support sub-stream, e.g. select cid,speed_kmh, lag(speed_kmh) OVER (PARTITION BY cid) as last_spd from car_live_data - Enhanced dedup function to only cache the unique keys for a given time period. This is useful to suppress the same alerts in the short time period. - Support sub-stream, e.g. select cid,speed_kmh, lag(speed_kmh) OVER (PARTITION BY cid) as last_spd from car_live_data - Source, sink, API and SDK Updated Python SDK https://pypi.org/project/timeplus/ to auto-delete the query history, refine error handling, Please note there is a breaking change, Env().tenant(id) is changed to Env().workspace(id) to be align with our terminology. Updated the REST API to show the optional description for source/sink, and replace "tenant" with "workspace-id" in the documentation. The Kafka sink no longer auto-create the topics - Updated Python SDK https://pypi.org/project/timeplus/ to auto-delete the query history, refine error handling, Please note there is a breaking change, Env().tenant(id) is changed to Env().workspace(id) to be align with our terminology. - Updated the REST API to show the optional description for source/sink, and replace "tenant" with "workspace-id" in the documentation. - The Kafka sink no longer auto-create the topics - UI improvements Show the total data size on the home page, as well as the average data in and data out throughput. Added a new 'SQL Templates' button in the query page to help you quickly add common snippets, such as settings seek_to='-1h' Added a closable page description, as well as context aware help side panel. Refined "Data Lineage" page to show stream schema, view SQL, and sink types. Able to set units when you choose to view the latest data. Mobile friendly login/signup page. - Show the total data size on the home page, as well as the average data in and data out throughput. - Added a new 'SQL Templates' button in the query page to help you quickly add common snippets, such as settings seek_to='-1h' - Added a closable page description, as well as context aware help side panel. - Refined "Data Lineage" page to show stream schema, view SQL, and sink types. - Able to set units when you choose to view the latest data. - Mobile friendly login/signup page. Streaming engine - Enhanced dedup function to only cache the unique keys for a given time period. This is useful to suppress the same alerts in the short time period. - Support sub-stream, e.g. select cid,speed_kmh, lag(speed_kmh) OVER (PARTITION BY cid) as last_spd from car_live_data Source, sink, API and SDK - Updated Python SDK https://pypi.org/project/timeplus/ to auto-delete the query history, refine error handling, Please note there is a breaking change, Env().tenant(id) is changed to Env().workspace(id) to be align with our terminology. - Updated the REST API to show the optional description for source/sink, and replace "tenant" with "workspace-id" in the documentation. - The Kafka sink no longer auto-create the topics UI improvements - Show the total data size on the home page, as well as the average data in and data out throughput. - Added a new 'SQL Templates' button in the query page to help you quickly add common snippets, such as settings seek_to='-1h' - Added a closable page description, as well as context aware help side panel. - Refined "Data Lineage" page to show stream schema, view SQL, and sink types. - Able to set units when you choose to view the latest data. - Mobile friendly login/signup page. ### Biweekly Update 9/5-9/16​ - Streaming engine Added a round function to round a value to a specified number of decimal places. Improved cluster support. - Added a round function to round a value to a specified number of decimal places. - Improved cluster support. - Source, sink, API and SDK Added new CRUD API for dashboards. The previous charts on the homepage will be removed automatically during upgrade. Simplified the hostname for the streamlit demo to https://timeplus.streamlitapp.com - Added new CRUD API for dashboards. The previous charts on the homepage will be removed automatically during upgrade. - Simplified the hostname for the streamlit demo to https://timeplus.streamlitapp.com - UI improvements Introduced full dashboard management. You can create multiple dashboards with name/description/charts. Redesigned home page to show high level information for the workspace. Introduced a new 'Data Lineage' page to visualize the relationship between sources/streams/views/sinks. Enhanced the workflow for visualizing the query result. You need to choose whether to view the latest data, check data trend, or view detailed data. Now you can visualize the query results from a historical query, e.g. select .. from table(..).., and add the charts to dashboards. Removed the browser side data aggregation to improve performance. If the rate of the data is greater than the render interval (200ms), only the last data point within the interval will be rendered. If you run a streaming tail or filter, you no longer can visualize the data with a bar chart. Please use GROUP BY in the SQL for such analysis. Enhanced the SQL editor to show column names without using stream names. Mobile friendly login/signup page. - Introduced full dashboard management. You can create multiple dashboards with name/description/charts. - Redesigned home page to show high level information for the workspace. - Introduced a new 'Data Lineage' page to visualize the relationship between sources/streams/views/sinks. - Enhanced the workflow for visualizing the query result. You need to choose whether to view the latest data, check data trend, or view detailed data. - Now you can visualize the query results from a historical query, e.g. select .. from table(..).., and add the charts to dashboards. - Removed the browser side data aggregation to improve performance. If the rate of the data is greater than the render interval (200ms), only the last data point within the interval will be rendered. If you run a streaming tail or filter, you no longer can visualize the data with a bar chart. Please use GROUP BY in the SQL for such analysis. - Enhanced the SQL editor to show column names without using stream names. - Mobile friendly login/signup page. Streaming engine - Added a round function to round a value to a specified number of decimal places. - Improved cluster support. Source, sink, API and SDK - Added new CRUD API for dashboards. The previous charts on the homepage will be removed automatically during upgrade. - Simplified the hostname for the streamlit demo to https://timeplus.streamlitapp.com UI improvements - Introduced full dashboard management. You can create multiple dashboards with name/description/charts. - Redesigned home page to show high level information for the workspace. - Introduced a new 'Data Lineage' page to visualize the relationship between sources/streams/views/sinks. - Enhanced the workflow for visualizing the query result. You need to choose whether to view the latest data, check data trend, or view detailed data. - Now you can visualize the query results from a historical query, e.g. select .. from table(..).., and add the charts to dashboards. - Removed the browser side data aggregation to improve performance. If the rate of the data is greater than the render interval (200ms), only the last data point within the interval will be rendered. If you run a streaming tail or filter, you no longer can visualize the data with a bar chart. Please use GROUP BY in the SQL for such analysis. - Enhanced the SQL editor to show column names without using stream names. - Mobile friendly login/signup page. ### Biweekly Update 8/22-9/2​ We have migrated beta1 customers to the beta2. https://demo.timeplus.com is no longer accessible. Please visit https://beta.timeplus.cloud/demo if you have an account for the beta testing. - Streaming engine Revised logical functions to return bool instead of uint8 Added experimental support for ARM chips - Revised logical functions to return bool instead of uint8 - Added experimental support for ARM chips - Source, sink, API and SDK Updated datapm Timeplus sink to support beta2 multi-tenant API Enhanced the snowflake sink to specify the data warehouse Published the sample Java code as a public Github repo. You can easily get Timeplus query results and do other operations without directly handling the low level REST API - Updated datapm Timeplus sink to support beta2 multi-tenant API - Enhanced the snowflake sink to specify the data warehouse - Published the sample Java code as a public Github repo. You can easily get Timeplus query results and do other operations without directly handling the low level REST API - UI improvements ... (content truncated) ## Prometheus - [Prometheus Integration | Timeplus](https://docs.timeplus.com/prometheus): Timeplus Enterprise exposes a number of metrics that can be scraped by Prometheus. These metrics can be integrated with dashboards and alerting systems like Grafana or Datadog. - Monitoring & Troubleshooting - Prometheus Integration # Prometheus Integration Timeplus Enterprise exposes a number of metrics that can be scraped by Prometheus. These metrics can be integrated with dashboards and alerting systems like Grafana or Datadog. ## Prometheus Endpoints​ By default, Timeplus Enterprise exposes timeplusd metrics on port 9363, and timeplus-connector metrics on port 4196. - Kubernetes: http://timeplusd-0.timeplusd-svc.my_ns.svc.cluster.local:9363/metrics - Bare metal: http://localhost:9363/metrics - Kubernetes: http://timeplus-connector.my_ns:4196/metrics - Bare metal: http://localhost:4196/metrics ## Explore Metrics with Grafana​ Grafana is a popular tool for visualizing metrics. Here is a simplified guide to visualize Timeplus Enterprise metrics with Grafana. ### Step 1: Install Grafana, Prometheus and Alloy​ ### Step 2: Configure Prometheus​ Make sure the Prometheus accept remote write. Edit /opt/homebrew/etc/prometheus.args and add --web.enable-remote-write-receiver. Start Prometheus via brew services start prometheus. ### Step 3: Configure Grafana Alloy​ Grafana Alloy is the successor of Grafana Agent. It is a vendor-neutral distribution of the OpenTelemetry (OTel) Collector. Edit /opt/homebrew/etc/alloy/config.alloy and add the following: Alloy will read the metrics from timeplusd and forward them to Prometheus. Start Alloy via brew services start alloy. ### Step 4: Configure Grafana​ Start Grafana via brew services start grafana. Open http://localhost:3000 in your browser. The default username and password are admin and admin. Add a new Data source with the type of Prometheus and the URL of http://localhost:9090. Explore the metrics via http://localhost:3000/explore/metrics. Contact us if you need help to build Grafana dashboards or alerts with Timeplus Enterprise metrics. - Prometheus Endpoints - Explore Metrics with GrafanaStep 1: Install Grafana, Prometheus and AlloyStep 2: Configure PrometheusStep 3: Configure Grafana AlloyStep 4: Configure Grafana - Step 1: Install Grafana, Prometheus and Alloy - Step 2: Configure Prometheus - Step 3: Configure Grafana Alloy - Step 4: Configure Grafana - Step 1: Install Grafana, Prometheus and Alloy - Step 2: Configure Prometheus - Step 3: Configure Grafana Alloy - Step 4: Configure Grafana ## Proton - [Timeplus Proton | Timeplus](https://docs.timeplus.com/proton): Timeplus Proton is a stream processing engine and database. It is fast and lightweight alternative to ksqlDB or Apache Flink, powered by the libraries and engines in ClickHouse. It enables developers to solve streaming data processing, multi-stream JOINs, sophisticated incremental materialized views, routing and analytics challenges from Apache Kafka, Redpanda and more sources, and send aggregated data to the downstream systems. Timeplus Proton is the core engine of Timeplus Enterprise. - Timeplus Proton (OSS) # Timeplus Proton Timeplus Proton is a stream processing engine and database. It is fast and lightweight alternative to ksqlDB or Apache Flink, powered by the libraries and engines in ClickHouse. It enables developers to solve streaming data processing, multi-stream JOINs, sophisticated incremental materialized views, routing and analytics challenges from Apache Kafka, Redpanda and more sources, and send aggregated data to the downstream systems. Timeplus Proton is the core engine of Timeplus Enterprise. ## 💪 Why use Timeplus Proton?​ 1. Apache Flink or ksqlDB alternative. Timeplus Proton provides powerful streaming SQL functionalities, such as streaming ETL, tumble/hop/session windows, watermarks, materialized views, CDC and data revision processing, etc. In contrast to pure stream processors, it also stores queryable analytical/row based materialized views within Proton itself for use in analytics dashboards and applications. 2. Fast. Timeplus Proton is written in C++, with optimized performance through SIMD. For example, on an Apple MacBookPro with M2 Max, Timeplus Proton can deliver 90 million EPS, 4 millisecond end-to-end latency, and high cardinality aggregation with 1 million unique keys. 3. Lightweight. Timeplus Proton is a single binary (<500MB). No JVM or any other dependencies. You can also run it with Docker, or on an AWS t2.nano instance (1 vCPU and 0.5 GiB memory). 4. Powered by the fast, resource efficient and mature ClickHouse. Timeplus Proton extends the historical data, storage, and computing functionality of ClickHouse with stream processing. Thousands of SQL functions are available in Timeplus Proton. Billions of rows are queried in milliseconds. 5. Best streaming SQL engine for Kafka, Redpanda, or Pulsar. Query the live data in Kafka or other compatible streaming data platforms, with external streams. See our architecture doc for technical details and our FAQ for more information. ## How is it different from ClickHouse?​ ClickHouse is an extremely performant Data Warehouse built for fast analytical queries on large amounts of data. While it does support ingesting data from streaming sources such as Apache Kafka, it is itself not a stream processing engine which can transform and join streaming event data based on time-based semantics to detect patterns that need to be acted upon as soon as it happens. ClickHouse also has incremental materialized view capability but is limited to creating materialized view off of ingestion of blocks to a single table. Timeplus Proton uses ClickHouse as a table store engine inside of each stream (alongside a Write Ahead Log and other data structures) and uses to unify real-time and historical data together to detect signals in the data. In addition, Timeplus Proton can act as an advanced data pre-processor for ClickHouse (and similar systems) where the bulk of the data preparation and batching is done ahead of ingestion. See Timeplus and ClickHouse for more details on this. ## 🎬 Demo Video​ ## ⚡ Deployment​ Proton can be installed as a single binary on Linux or Mac, via: Once the proton binary is available, you can run Timeplus Proton in different modes: - Local Mode. You run proton local to start it for fast processing on local and remote files using SQL without having to install a full server - Config-less Mode. You run proton server to start the server and put the config/logs/data in the current folder proton-data. Then use proton client in the other terminal to start the SQL client. - Server Mode. You run sudo proton install to install the server in predefined path and a default configuration file. Then you can run sudo proton server -C /etc/proton-server/config.yaml to start the server and use proton client in the other terminal to start the SQL client. For Mac users, you can also use Homebrew to manage the install/upgrade/uninstall: You can also install Proton in Docker, Docker Compose or Kubernetes. Please check Server Ports to determine which ports to expose, so that other tools can connect to Timeplus, such as DBeaver. The Docker Compose stack demonstrates how to read/write data in Kafka/Redpanda with external streams. ### Timeplus Cloud Demo​ Don't want to setup by yourself? Try Timeplus in Cloud. ### 🔎 Usage​ SQL is the main interface. You can start a new terminal window with proton client to start the SQL shell. You can also integrate Timeplus Proton with Python/Java/Go SDK, REST API, or BI plugins. Please check Integration. In the proton client, you can write SQL to create External Stream for Kafka or External Table for ClickHouse. You can also run the following SQL to create a stream of random data: You should see data like the following: ### ⏩ What's next?​ To see more examples of using Timeplus Proton, check out the examples folder. To access more features, such as sources, sinks, dashboards, alerts, and data lineage, try our live demo with pre-built live data and dashboards. ## 🧩 Integration​ The following drivers are available: - https://github.com/timeplus-io/proton-java-driver JDBC and other Java clients - https://github.com/timeplus-io/proton-go-driver - https://github.com/timeplus-io/proton-python-driver Integration with other systems: - ClickHouse https://docs.timeplus.com/proton-clickhouse-external-table - Docker and Testcontainers - Sling - Grafana https://github.com/timeplus-io/proton-grafana-source - Metabase https://github.com/timeplus-io/metabase-proton-driver - Pulse UI https://github.com/timeplus-io/pulseui/tree/proton - Homebrew https://github.com/timeplus-io/homebrew-timeplus - dbt https://github.com/timeplus-io/dbt-proton ## Contributing​ We welcome your contributions! If you are looking for issues to work on, try looking at the issue list. Please see the wiki for more details, and BUILD.md to compile Proton in different platforms. We also encourage you to join our Timeplus Community Slack to ask questions and meet other active contributors from Timeplus and beyond. ## Need help?​ Join our Timeplus Community Slack to connect with Timeplus engineers and other Timeplus Proton users. For filing bugs, suggesting improvements, or requesting new features, see the open issues here on GitHub. ## Licensing​ Proton uses Apache License 2.0. See details in the LICENSE. - 💪 Why use Timeplus Proton? - How is it different from ClickHouse? - 🎬 Demo Video - ⚡ DeploymentTimeplus Cloud Demo🔎 Usage⏩ What's next? - Timeplus Cloud Demo - 🔎 Usage - ⏩ What's next? - 🧩 Integration - Contributing - Need help? - Licensing - Timeplus Cloud Demo - 🔎 Usage - ⏩ What's next? ## Proton-clickhouse-external-table - [ClickHouse External Table | Timeplus](https://docs.timeplus.com/proton-clickhouse-external-table): Timeplus can read or write ClickHouse tables directly. This unlocks a set of new use cases, such as - Core Features - External Streams & Tables - External Tables - ClickHouse # ClickHouse External Table Timeplus can read or write ClickHouse tables directly. This unlocks a set of new use cases, such as - Use Timeplus to efficiently process real-time data in Kafka/Redpanda, apply flat transformation or stateful aggregation, then write the data to the local or remote ClickHouse for further analysis or visualization. - Enrich the live data with the static or slow-changing data in ClickHouse. Apply streaming JOIN. - Use Timeplus to query historical or recent data in ClickHouse This integration is done by introducing "External Table" in Timeplus. Similar to External Stream, there is no data persisted in Timeplus. However, since the data in ClickHouse is in the form of table, not data stream, so we call this as External Table. In the roadmap, we will support more integration by introducing other types of External Table. ## Demo Video​ This video demonstrates how to read live data from Redpanda, apply stream processing and send results to ClickHouse. ## CREATE EXTERNAL TABLE​ ### Syntax​ The required settings are type and address. For other settings, the default values are - 'default' for user - '' (empty string) for password - 'default' for database - 'false' for secure - If you omit the table name, it will use the name of the external table The config_file setting is available since Timeplus Enterprise 2.7. You can specify the path to a file that contains the configuration settings. The file should be in the format of key=value pairs, one pair per line. You can set the ClickHouse user and password in the file. Please follow the example in Kafka External Stream. You don't need to specify the columns, since the table schema will be fetched from the ClickHouse server. Once the external table is created successfully, you can run the following SQL to list the columns: The data types in the output will be Timeplus data types, such as uint8, instead of ClickHouse type UInt8. Timeplus maintains a mapping for those types. Learn more. You can define the external table and use it to read data from the ClickHouse table, or write to it. ### Connect to a local ClickHouse​ Example SQL to connect to a local ClickHouse server without password: ### Connect to ClickHouse Cloud​ Example SQL to connect to ClickHouse Cloud: ### Connect to Aiven for ClickHouse​ Example SQL to connect to Aiven for ClickHouse: ## Read data from ClickHouse​ Once the external table is created successfully, it means Timeplus can connect to the ClickHouse server and fetch the table schema. You can query it via the regular select .. from table_name. Please note, in the current implementation, all rows will be fetched from ClickHouse to Timeplus, with the selected columns. Then Timeplus applies the SQL functions and LIMIT n locally. It's not recommended to run SELECT * for a large ClickHouse table. Also note, use the Timeplus function names when you query the external table, such as to_int, instead of ClickHouse's naming convention, e.g. toInt. In current implementation, the SQL functions are applied in Timeplus engine. We plan to support some function push-down to ClickHouse in future versions. Limitations: 1. tumble/hop/session/table functions are not supported for External Table (coming soon) 2. scalar or aggregation functions are performed by Timeplus, not the remote ClickHouse 3. LIMIT n is performed by Timeplus, not the remote ClickHouse ## Write data to ClickHouse​ You can run regular INSERT INTO to add data to ClickHouse table. However it's more common to use a Materialized View to send the streaming SQL results to ClickHouse. Say you have created an external table ch_table. You can create a materialized view to read Kafka data(via an external stream) and transform/aggregate the data and send to the external table: ### Batching Settings​ In Timeplus Enterprise, additional performance tuning settings are available, such as - max_insert_block_size - The maximum block size for insertion, i.e. maximum number of rows in a batch. Default value: 65409 - max_insert_block_bytes - The maximum size in bytes of block for insertion. Default value: 1 MiB. - insert_block_timeout_ms - The maximum time in milliseconds for constructing a block(a block) for insertion. Increasing the value gives greater possibility to create bigger blocks (limited by max_insert_block_bytes and max_insert_block_size), but also increases latency. Negative numbers means no timeout. Default value: 500. ## Supported data types​ All ClickHouse data types are supported in the external table, except Point. While reading or writing data, Timeplus applies a data type mapping, such as converting Timeplus' uint8 to ClickHouse's UInt8. If you find anything wrong with the data type, please let us know. - Demo Video - CREATE EXTERNAL TABLESyntaxConnect to a local ClickHouseConnect to ClickHouse CloudConnect to Aiven for ClickHouse - Syntax - Connect to a local ClickHouse - Connect to ClickHouse Cloud - Connect to Aiven for ClickHouse - Read data from ClickHouse - Write data to ClickHouseBatching Settings - Batching Settings - Supported data types - Syntax - Connect to a local ClickHouse - Connect to ClickHouse Cloud - Connect to Aiven for ClickHouse - Batching Settings ## Proton-client - [proton-client | Timeplus](https://docs.timeplus.com/proton-client): Timeplus Proton provides a native command-line client: proton-client to run SQL commands. You can also launch the client via proton client. - Integrations - CLI, APIs & SDKs - proton-client # proton-client Timeplus Proton provides a native command-line client: proton-client to run SQL commands. You can also launch the client via proton client. For detailed usage of the SQL client, please check timeplusd-client, since the SQL clients in Timeplus Proton and Timeplus Enterprise share the same flags and configurations. However, please use proton-client to connect to Timeplus Proton and use timeplusd-client to connect to the timeplusd in Timeplus Enterprise. Otherwise certain features won't be available and may not work. For example, when you run proton-client to connect to a Timeplus Enterprise deployment and try to create a mutable stream, the proton-client won't accept the CREATE MUTABLE STREAM SQL at the client side. ## Proton-faq - [Timeplus Proton FAQ | Timeplus](https://docs.timeplus.com/proton-faq): On September 21, 2023, Timeplus announced the open source project: Timeplus Proton. We're using this FAQ as the primary reference for learning about what Timeplus Proton is, how we licensed the code open source, how you can use Timeplus Proton today, and more. - Timeplus Proton (OSS) - Timeplus Proton FAQ # Timeplus Proton FAQ On September 21, 2023, Timeplus announced the open source project: Timeplus Proton. We're using this FAQ as the primary reference for learning about what Timeplus Proton is, how we licensed the code open source, how you can use Timeplus Proton today, and more. ## What is Timeplus Proton?​ Timeplus Proton is the core engine powering Timeplus Enterprise, a unified streaming and historical data processing engine designed for efficiency and strong performance. Timeplus Proton has no external service dependencies, which allows you to deploy it on bare metal, edge devices, within containers, or as part of an orchestrated cloud environment. Timeplus Proton builds on top of the popular open source ClickHouse project for its historical data, storage, computing functionality, and a portion of its query engine. By leveraging the proven technologies of ClickHouse, Timeplus Proton brings more mature online analytical processing (OLAP) capabilities to the open source community with lots of new development to unify the streaming and processing engines. You can use Timeplus Proton alongside your existing ClickHouse deployment to enable additional functionality. ## How is Timeplus Proton licensed?​ Timeplus Proton follows the ClickHouse licensing model with Apache License 2.0, which is also used by popular open source projects like Kubernetes and Apache Flink. Apache License 2.0 is a permissive license that allows for most uses without restriction. We'll talk more about how you can use Timeplus Proton in the following section. We chose this license for a few important reasons: - We're following the path ClickHouse first paved. Because Timeplus Proton leverages many excellent ClickHouse technologies, we hope to see our communities grow together and the two open source projects become more deeply integrated over time. - We want to see Timeplus Proton push the limits of streaming and data processing in unique and exotic environments. While Timeplus Proton already powers the enterprise-ready Timeplus Cloud, developers or other end users can download and deploy Timeplus Proton or modify the code locally for use in a private cloud infrastructure. Using Timeplus Proton does not require you adopt Timeplus Enterprise or meet with our sales team. - We're eager to see what value a free streaming and historical data processing engine delivers. By releasing the single-node edition of Timeplus Proton to the open source community, we're giving developers, hobbyists, enthusiasts, and anyone who wants to try new technologies a new path that's entirely free. - We're building a new community around unified streaming and data processing. ClickHouse paved the way for processing, but we have much to experiment and discover together around streaming. We can't wait to get feedback from developers and users within organizations of all sizes and degrees of streaming maturity. We're following the path ClickHouse first paved. Because Timeplus Proton leverages many excellent ClickHouse technologies, we hope to see our communities grow together and the two open source projects become more deeply integrated over time. We want to see Timeplus Proton push the limits of streaming and data processing in unique and exotic environments. While Timeplus Proton already powers the enterprise-ready Timeplus Cloud, developers or other end users can download and deploy Timeplus Proton or modify the code locally for use in a private cloud infrastructure. Using Timeplus Proton does not require you adopt Timeplus Enterprise or meet with our sales team. We're eager to see what value a free streaming and historical data processing engine delivers. By releasing the single-node edition of Timeplus Proton to the open source community, we're giving developers, hobbyists, enthusiasts, and anyone who wants to try new technologies a new path that's entirely free. We're building a new community around unified streaming and data processing. ClickHouse paved the way for processing, but we have much to experiment and discover together around streaming. We can't wait to get feedback from developers and users within organizations of all sizes and degrees of streaming maturity. ## What uses, commercial and beyond, are allowed with the Timeplus Proton project?​ Under Apache License 2.0, you are allowed to modify, distribute, and sublicense Timeplus Proton as it's available on GitHub. You can also include Timeplus Proton in proprietary software that you then sell to customers, but that does not grant you any rights to use Proton or Timeplus trademarks or imply a commercial relationship or partnership between your organization and Timeplus. Apache License 2.0 also prevents any contributor to Timeplus Proton—a member of the Timeplus team or an outside contributor—from being held liable by end users, whether they are individual developers or commercial organizations. ## What features are available with Timeplus Proton versus Timeplus Enterprise?​ Please refer to our comparison page for a detailed comparison of Timeplus Proton and Timeplus Enterprise. ## My organization already uses ClickHouse—are there plans to integrate Timeplus Proton with the open source ClickHouse project?​ You can create an External Table to read or write ClickHouse tables from Timeplus Proton. Check the tutorials for how to build streaming ETL from Kafka to ClickHouse, or from MySQL to ClickHouse, via Timeplus. We are also in conversation with the folks at ClickHouse, Inc., and the ClickHouse open source project at large, to scope the possibility of deep integration between the projects. ## If I'm familiar with ClickHouse, how easy is it for me to use Timeplus Proton?​ Short answer: very easy. We designed Timeplus Proton's usage to be similar to ClickHouse, with a few key differences: - Timeplus' default SQL query mode is streaming, which means it is long-running and continuously tracks and evaluates changed data and pushes results to users or target systems. To create a historical data query, wrap your SQL in table(stream). - The SQL keyword AS is required to create a temporary name for a table, stream, or a column. - We renamed data types and functions to remove camelcase. For example, ClickHouse's toInt8() is renamed to_int8() in Timeplus Proton. Our functions docs have additional details. - Not all ClickHouse functions are currently enabled in Timeplus Proton or work in a streaming query. If we should add or enhance the functions available in Timeplus Proton, let us know in the GitHub issues. - Materialized Views in ClickHouse works for one source table, and data is processed at the index time. In Timeplus Proton, you can define a Materialized View with a streaming SQL, for any number of streams, with JOIN, CTE, or subqueries. Timeplus Proton continuously runs the query and sends the results to the internal stream or the target stream. - In Timeplus Proton, JOINs are a powerful and flexible means of combining data from multiple sources into a single stream. See the documentation for full usage details. ## Is Timeplus Proton usage tracked?​ Yes. We have enabled telemetry in the following areas to understand how the community is using Timeplus Proton and help us improve the project: - Via Docker image download statistics, which are provided by GitHub without any personally identifying information (PII), such as IP addresses. - On start, Timeplus Proton reports the following data to a public endpoint: Current Timeplus Proton version CPU and memory availability - Current Timeplus Proton version - CPU and memory availability Via Docker image download statistics, which are provided by GitHub without any personally identifying information (PII), such as IP addresses. On start, Timeplus Proton reports the following data to a public endpoint: - Current Timeplus Proton version - CPU and memory availability No user data, schemas, SQL statements, or personally identifiable information (PII) is ever sent by Timeplus Proton to this public endpoint. You can disable telemetry in Timeplus Proton via the environment variable TELEMETRY_ENABLED, such as docker run --env TELEMETRY_ENABLED=false --name proton ghcr.io/timeplus-io/proton:latest or update the configuration with the following steps: 1. Start the Timeplus Proton Docker image 2. Connect to the running container with docker exec -it proton bin/sh 3. Run the following command to edit the container's configuration: 1. Stop and start the container again to run Timeplus Proton with all telemetry disabled. If you use the single binary, the environment variable TELEMETRY_ENABLED also works. Alternatively, you can manually update config.yaml file to set telemetry_enabled: false. See our privacy policy for complete details about the telemetry we collect and use. ## Does Timeplus Proton provide a JDBC/ODBC driver?​ JDBC driver is available at https://github.com/timeplus-io/proton-java-driver, and the ODBC driver is available at https://github.com/timeplus-io/proton-odbc. In the meanwhile, you can send the processed data to Kafka topics via External Stream, use the proton-go-driver, or Redpanda Connect to send the data to other systems. If you are on Timeplus Enterprise, you can use the REST API or SDK to run queries or manage resources in Timeplus, via the API server, which is not part of Timeplus Proton. ## Can I contribute to Timeplus Proton?​ Yes! The best way to get started is to check out the existing issues in the Timeplus Proton repository on GitHub. We're also actively discussing future versions of Timeplus Proton in the #proton and #contributing channels in our Timeplus Community Slack. ## Where can I learn more about Timeplus Proton?​ We're currently building out resources where you can learn about Timeplus Proton's architecture, features, and future: - GitHub - Documentation - High-level architecture - Videos - Wiki ... (content truncated) ## Proton-format-schema - [Protobuf/Avro Schema | Timeplus](https://docs.timeplus.com/proton-format-schema): Timeplus supports reading or writing messages in Protobuf or Avro format for Kafka External Stream or Pulsar External Stream. This document covers how to process data without a Schema Registry. Check this page if your Kafka topics are associated with a Schema Registry. - Core Features - External Streams & Tables - External Streams - Apache Kafka - Protobuf/Avro Schema # Protobuf/Avro Schema Timeplus supports reading or writing messages in Protobuf or Avro format for Kafka External Stream or Pulsar External Stream. This document covers how to process data without a Schema Registry. Check this page if your Kafka topics are associated with a Schema Registry. ## Create A Schema​ Without a Schema Registry, you need to define the Protobuf or Avro schema using SQL. ### Protobuf​ Then refer to this schema while creating an external stream for Kafka or Pulsar: Then you can run INSERT INTO or use a materialized view to write data to the topic. Please note: 1. If you want to ensure there is only a single Protobuf message per Kafka message, please set data_format to ProtobufSingle. If you set it to Protobuf, then there could be multiple Protobuf messages in a single Kafka message. 2. The format_schema setting contains two parts: the registered schema name (in this example: schema_name), and the message type (in this example: SearchRequest). Combining them together with a semicolon. 3. You can use this external stream to read or write Protobuf messages in the target Kafka/Confluent topics. 4. For more advanced use cases, please check the examples for complex schema. ### Avro​ Available since Timeplus Proton 1.5.10. Then refer to this schema while creating an external stream for Kafka or Pulsar: Then you can run INSERT INTO or use a materialized view to write data to the topic. ## List Schemas​ List schemas in the current Timeplus deployment: ## Show Details For A Schema​ ## Drop A Schema​ ## Examples For Complex Protobuf Schema​ ### Nested Schema​ Please note: 1. Person is the top level message type. It refers to the Name message type. 2. Use name as the prefix as the column names. Use either _ or . to connect the prefix with the nested field names. 3. When you create an external stream to read the Protobuf messages, you don't have to define all possible columns. Only the columns you defined will be read. Other columns/fields are skipped. ### Enum​ Say in your Protobuf definition, there is an enum type: You can use the enum type in Proton, e.g. ### Repeat​ Say in your Protobuf definition, there is a repeated type: You can use the array type in Proton, e.g. ### Repeated and Nested​ Say in your Protobuf definition, there is a field in a custom type and also repeated: You can use the tuple type in Proton, e.g. The streaming data will be shown as: ### Package​ Say in your Protobuf definition, there is a package: If there is only 1 package in the Protobuf definition type, you don't have to include the package name. For example: If there are multiple packages, you can use the fully qualified name with package, e.g. ### Import Schemas​ If you have used CREATE FORMAT SCHEMA to register a format schema, say schema_name, you can create the other schema and import this: Please make sure to add .proto as the suffix. ## Avro Data Types Mapping​ ### Avro Primitive Types​ The table below shows supported Avro primitive data types and how they match Timeplus data types in INSERT and SELECT queries. ### Avro Logical Types​ If you use logicalType in your Avro schema, Timeplus will automatically map it to the corresponding Timeplus data type: - UUID: maps to uuid. - Date: maps to date. - Timestamp (millisecond precision): maps to datetime64(3). - Timestamp (microsecond precision): maps to datetime64(6). Other logical types are not implemented yet. For example, given the following Avro schema: The external stream uses tuple will be like this: ### record​ There are two ways to map a record. The simple one is using tuple. Here is an example: First given a Avro schema like this: The external stream uses tuple will be like this: The other way is flatting the fields, i.e. we will create a column for each field. The external stream can be defined as: The column name for each field will be the name of the record field itself (in this case a_record_field) followed by a dot (.), and followed by the field name. This is how "flatten" works. ### array of record​ To map an array of record, you can use either array(tuple(...)) or nested(), they are the same. By default, Timeplus will flatten the columns. For example: Give an Avro schema: We would create a stream like this: will become: The Avro output format can handle this properly. You can use SET flatten_nested = 0 to disable the flatten behavior. The Avro output format can handle it well too. ### union​ Since Timeplus does not support native union, there is no "perfect" way to handle Avro unions. One stream can only handle one of the union elements (except for null, more details later). If you need to generate values for different element types, you will need to create multiple streams. For example. Given an Avro schema: When we create the stream, we can only map the int_or_string field to either int or string, for example: This stream can only write int values. If you want to write string values, you will need to create another stream like this: We can also use the flatten naming convention to map the union field. For this example, the streams will be: For named types (record, fixed, and enum), use the name property instead of the type name. For example, given an Avro schema: In order to map to the record element of the union, we need to use the name details, so the stream definition will be: The Avro input format only supports the flatten naming convention, so, if you create the stream using the first method: then, SELECT * FROM avro won't work. ### nullable​ There is a special case for union, which is, when the union has two elements, and one of it is null, then this union field will be mapped to a nullable column. Example: Avro schema: Stream: However, in Timeplus, nullable cannot be applied on all the types. For instance, nullable(tuple(...)) is invalid. If a field in the Avro schema is "type": ["null", {"type": "record"}], you can only map it to a tuple, and it can't be null. - Create A SchemaProtobufAvro - Protobuf - Avro - List Schemas - Show Details For A Schema - Drop A Schema - Examples For Complex Protobuf SchemaNested SchemaEnumRepeatRepeated and NestedPackageImport Schemas - Nested Schema - Enum - Repeat - Repeated and Nested - Package - Import Schemas - Avro Data Types MappingAvro Primitive TypesAvro Logical Typesrecordarray of recordunionnullable - Avro Primitive Types - Avro Logical Types - record - array of record - union - nullable - Protobuf - Avro - Nested Schema - Enum - Repeat - Repeated and Nested - Package - Import Schemas - Avro Primitive Types - Avro Logical Types - record - array of record - union - nullable ## Proton-ingest-api - [Proton Ingest REST API | Timeplus](https://docs.timeplus.com/proton-ingest-api): You can run INSERT INTO stream VALUES (..) SQL to insert data to Timeplus Proton or timeplusd in Timeplus Enterprise. You can also call the ingestion REST API to push data to Timeplus Proton or timeplusd, with any preferred languages. Timeplus Enterprise also provides REST API in the application sever. Comparing to that REST API, the Proton/Timeplusd Ingest REST API performs better, but doesn't support API Key, or flexible payload schema. - Integrations - CLI, APIs & SDKs - Proton Ingest REST API # Proton Ingest REST API You can run INSERT INTO [stream](/column1, column2) VALUES (..) SQL to insert data to Timeplus Proton or timeplusd in Timeplus Enterprise. You can also call the ingestion REST API to push data to Timeplus Proton or timeplusd, with any preferred languages. Timeplus Enterprise also provides REST API in the application sever. Comparing to that REST API, the Proton/Timeplusd Ingest REST API performs better, but doesn't support API Key, or flexible payload schema. ## Prerequisites​ ### Expose port 3218 from Proton container​ The Proton ingest REST API is on port 3218 by default. Please start the Proton container with the 3218 port exposed. For example: ### Create a stream in Proton​ You need to create a stream in Timeplus via CREATE STREAM. Columns with proper names and types should be set. First run the SQL client Then run the following SQL to create the stream. ## Push data to Timeplus​ The endpoint for real-time data ingestion is http://localhost:3218/proton/v1/ingest/streams/{stream_name}. HTTP method is POST. Request samples: - Node.js - curl - Python - Java Note: - the columns is an array of string, with the column names - the data is an array of arrays. Each nested array represents a row of data. The value order must match the exact same order in the columns. For example: You can also use one of our SDKs to ingest data without handling the low level details of REST API. - PrerequisitesExpose port 3218 from Proton containerCreate a stream in Proton - Expose port 3218 from Proton container - Create a stream in Proton - Push data to Timeplus - Expose port 3218 from Proton container - Create a stream in Proton ## Proton-kafka - [Kafka External Stream | Timeplus](https://docs.timeplus.com/proton-kafka): You can read data from Apache Kafka (as well as Confluent Cloud, or Redpanda) in Timeplus with External Stream. Combining with Materialized View and Target Stream, you can also write data to Apache Kafka with External Stream. - Core Features - External Streams & Tables - External Streams - Apache Kafka # Kafka External Stream You can read data from Apache Kafka (as well as Confluent Cloud, or Redpanda) in Timeplus with External Stream. Combining with Materialized View and Target Stream, you can also write data to Apache Kafka with External Stream. ## Tutorial with Docker Compose​ Please check the tutorials: - Query Kafka with SQL - Streaming JOIN - Streaming ETL ## CREATE EXTERNAL STREAM​ In Timeplus Proton, the external stream supports Kafka API as the only type. In Timeplus Enterprise, it also supports External Stream for Apache Pulsar, and External Stream for other Timeplus deployment. To create an external stream for Apache Kafka or Kafka-compatible messaging platforms, you can run the following DDL SQL: For examples to connect to various Kafka API compatitable message platforms, please check this doc. ### DDL Settings​ #### type​ Need to be kafka. This works with Apache Kafka, Redpanda, Confluent Platform or Cloud,and many other Kafka compatible message platforms. #### brokers​ One or more brokers with ports. #### topic​ One external stream will connect to one topic. #### security_protocol​ The supported values for security_protocol are: - PLAINTEXT: when this option is omitted, this is the default value. - SASL_SSL: when this value is set, username and password should be specified. If you need to specify own SSL certification file, add another setting ssl_ca_cert_file='/ssl/ca.pem' New in Proton 1.5.5, you can also put the full content of the pem file as a string in the ssl_ca_pem setting if you don't want to, or cannot use a file path, such as on Timeplus Enterprise or in Docker/Kubernetes environments. Skipping the SSL certification verification can be done via SETTINGS skip_ssl_cert_check=true. - If you need to specify own SSL certification file, add another setting ssl_ca_cert_file='/ssl/ca.pem' New in Proton 1.5.5, you can also put the full content of the pem file as a string in the ssl_ca_pem setting if you don't want to, or cannot use a file path, such as on Timeplus Enterprise or in Docker/Kubernetes environments. - Skipping the SSL certification verification can be done via SETTINGS skip_ssl_cert_check=true. - If you need to specify own SSL certification file, add another setting ssl_ca_cert_file='/ssl/ca.pem' New in Proton 1.5.5, you can also put the full content of the pem file as a string in the ssl_ca_pem setting if you don't want to, or cannot use a file path, such as on Timeplus Enterprise or in Docker/Kubernetes environments. - Skipping the SSL certification verification can be done via SETTINGS skip_ssl_cert_check=true. #### sasl_mechanism​ The supported values for sasl_mechanism are: - PLAIN: when you set security_protocol to SASL_SSL, this is the default value for sasl_mechanism. - SCRAM-SHA-256 - SCRAM-SHA-512 - AWS_MSK_IAM: this is available since Timeplus Enterprise 2.7.0 and Proton 1.6.12. Set to this value if you are using AWS MSK with IAM authentication and the EC2 instance or Kubernetes pod has the proper IAM role to access the Kafka topic. #### username​ Required when sasl_mechanism is set to value other than PLAIN or AWS_MSK_IAM. #### password​ Required when sasl_mechanism is set to value other than PLAIN or AWS_MSK_IAM. Since Timeplus Enterprise v2.7, you can also use the config_file setting to specify the username and password in a separate file. #### config_file​ The config_file setting is available since Timeplus Enterprise 2.7. You can specify the path to a file that contains the Kafka configuration settings. The file should be in the format of key=value pairs, one pair per line. For example: Not just for username and password, you can also put other Kafka settings in the file. Avoid defining the value both in the config_file and in the DDL. If you manage Kubernetes secrets using HashiCorp Vault, you can use the Vault Agent Injector to mount the secrets to the pod and use the config_file setting to specify the path to the file. For example, you create the following annotation to inject the secrets as a local file: #### data_format​ The supported values for data_format are: - JSONEachRow: parse each row of the message as a single JSON document. The top level JSON key/value pairs will be parsed as the columns. Learn More. - CSV: less commonly used. Learn More. - TSV: similar to CSV but tab as the separator - ProtobufSingle: for single Protobuf message per message - Protobuf: there could be multiple Protobuf messages in a single message. - Avro: added in Proton 1.5.2 - RawBLOB: the default value. Read/write message as plain text. #### format_schema​ Required if data_format is set to ProtobufSingle, Protobuf or Avro. #### one_message_per_row​ If the external stream is used to write data to a Kafka topic and the data_format is set to JSONEachRow, setting this to true to make sure each Kafka message only contains one JSON document. #### kafka_schema_registry_url​ Set to the address of Kafka Schema Registry server. http or https need to be included. Learn more for this setting and other settings with kafka_schema_ as the prefix. #### kafka_schema_registry_credentials​ Set in the 'username:password' format. Learn more. #### ssl_ca_cert_file​ Set to the path of the CA certificate file. #### ssl_ca_pem​ Set to the content of the CA certificate file. Usually starts with -----BEGIN CERTIFICATE-----. #### skip_ssl_cert_check​ Default to false. Set to true to skip the SSL certificate check and don't specify the CA certificate file or content. #### properties​ For more advanced use cases, you can specify customized properties while creating the external streams. Those properties will be passed to the underlying Kafka client, which is librdkafka. Please refer to the section in the bottom of this page for more details. ## Read Data in Kafka​ ### Read messages in a single column​ If the message in Kafka topic is in plain text format or JSON, you can create an external stream with only a raw column in string type. Example: Then use query time JSON extraction functions or shortcut to access the values, e.g. raw:id. ### Read messages as multiple columns​ If the keys in the JSON message never change, or you don't care about the new columns, you can also create the external stream with multiple columns. You can pick up some top level keys in the JSON as columns, or all possible keys as columns. Please note the behaviors are changed in recent versions, based on user feedback: Example: If there are nested complex JSON in the message, you can define the column as a string type. Actually any JSON value can be saved in a string column. Protobuf messages can be read with all or partial columns. Please check this page. ### Query Settings​ #### shards​ Starting from Proton 1.3.18, you can also read in specified Kafka partitions. By default, all partitions will be read. But you can also read from a single partition via the shards setting, e.g. Or you can specify a set of partition ID, separated by comma, e.g. ### Read existing messages​ When you run SELECT raw FROM ext_stream , Timeplus will read the new messages in the topics, not the existing ones. #### seek_to​ If you need to read all existing messages, you can use the following settings: Or the following SQL if you are running Proton 1.5.9 or above: Please avoid scanning all data via select * from table(ext_stream). However select count(*) from table(ext_stream) is optimized to get the number of current message count from the Kafka topic. ### Virtual Columns​ Besides the message body, Timeplus provides several virtual columns for each message in the Kafka topic. #### _tp_time​ You can read the timestamp of the message via _tp_time, e.g. Starting from Timeplus Enterprise 2.8.1, you can also specify the message timestamp by set a value to the _tp_time column. #### _tp_message_key​ Starting from Timeplus Enterprise 2.4, you can define the _tp_message_key column to read or write the message key in the preferred format. For example: When doing a SELECT query, the message key will be populated to the _tp_message_key column. SELECT * FROM foo will return 'some-key' for the _tp_message_key message. _tp_message_key support the following types: uint8, uint16, uint32, uint64, int8, int16, int32, int64, bool, float32, float64, string, and fixed_string. _tp_message_key also support nullable. Thus we can create an external stream with optional message key. For example: For how to use _tp_message_key to write the message key when you insert data into the Kafka topic, please refer to the section. #### _tp_message_headers​ Starting from Timeplus Proton 1.6.11 and Timeplus Enterprise 2.7, you can read the Kafka message headers as map(string,string), for example: To get the value for a certain key in the header, you can access it via _tp_message_headers['key'], for example: #### _tp_sn​ You can read the message offset of the message via _tp_sn, e.g. #### _tp_shard​ You can read the partition ID of the message via _tp_shard, e.g. ## Write Data to Kafka​ ### Write to Kafka in Plain Text​ You can write plain text messages to Kafka topics with an external stream with a single column. Then use either INSERT INTO VALUES (v), or Ingest REST API, or set it as the target stream for a materialized view to write message to the Kafka topic. The actual data_format value is RawBLOB but this can be omitted. By default one_message_per_row is true. Since Timeplus Proton 1.5.11, a new setting kafka_max_message_size is available. When multiple rows can be written to the same Kafka message, this setting will control how many data will be put in a Kafka message, ensuring it won't exceed the kafka_max_message_size limit. ### Multiple columns to write to Kafka​ ... (content truncated) ## Proton-ports - [Server Ports | Timeplus](https://docs.timeplus.com/proton-ports): Timeplus serves on many ports for different purposes. Only few of them are exposed in sample Docker Compose files. - Deployment & Operations - Server Ports # Server Ports Timeplus serves on many ports for different purposes. Only few of them are exposed in sample Docker Compose files. Here is a summary of common ports you may need. You can check the default server config.yaml for details. ## Proton-schema-registry - [Kafka Schema Registry | Timeplus](https://docs.timeplus.com/proton-schema-registry): Read Messages in Protobuf or Avro Schema - Core Features - External Streams & Tables - External Streams - Apache Kafka - Kafka Schema Registry # Kafka Schema Registry ## Read Messages in Protobuf or Avro Schema​ To read Kafka data in Protobuf or Avro schema with a schema registry, you can create an external stream with kafka_schema_registry_url settings, e.g. Please note: 1. kafka_schema_registry_credentials is optional. Skip this if the schema registry server doesn't require authentication. 2. Make sure to add http:// or https:// in the kafka_schema_registry_url. In Proton 1.5.3 or above, self-signed HTTPS certification is supported. One solution is to set kafka_schema_registry_skip_cert_check to true. This will completely skip the TLS certification verification. In this case, you don't need to specify the certification files. A more secure solution is to keep the default value of kafka_schema_registry_skip_cert_check, which is false. Omit this setting and specify the following 3 settings: kafka_schema_registry_private_key_file: the file path to the private key file used for encryption. Please use absolute file path and make sure Proton can access this file. If you are using Kubernetes or Docker, please mount the file systems properly. kafka_schema_registry_cert_file: the file path to the certificate file (in PEM format). If the private key and the certificate are stored in the same file, this can be empty if kakfa_schema_registry_private_key_file is specified. kafka_schema_registry_ca_location: the path to the file or directory containing the CA/root certificates. 3. One solution is to set kafka_schema_registry_skip_cert_check to true. This will completely skip the TLS certification verification. In this case, you don't need to specify the certification files. 4. A more secure solution is to keep the default value of kafka_schema_registry_skip_cert_check, which is false. Omit this setting and specify the following 3 settings: kafka_schema_registry_private_key_file: the file path to the private key file used for encryption. Please use absolute file path and make sure Proton can access this file. If you are using Kubernetes or Docker, please mount the file systems properly. kafka_schema_registry_cert_file: the file path to the certificate file (in PEM format). If the private key and the certificate are stored in the same file, this can be empty if kakfa_schema_registry_private_key_file is specified. kafka_schema_registry_ca_location: the path to the file or directory containing the CA/root certificates. 5. kafka_schema_registry_private_key_file: the file path to the private key file used for encryption. Please use absolute file path and make sure Proton can access this file. If you are using Kubernetes or Docker, please mount the file systems properly. 6. kafka_schema_registry_cert_file: the file path to the certificate file (in PEM format). If the private key and the certificate are stored in the same file, this can be empty if kakfa_schema_registry_private_key_file is specified. 7. kafka_schema_registry_ca_location: the path to the file or directory containing the CA/root certificates. 8. Make sure you define the columns matching the fields in the Avro schema. You don't have to define all top level fields in Avro schema as columns in the stream. For example, if there are 4 fields in Avro schema, you can choose only 2 of them as columns in the external stream. But make sure the data types match. 9. data_format can be Avro, or ProtobufSingle. 10. Schema reference is not supported yet. kafka_schema_registry_credentials is optional. Skip this if the schema registry server doesn't require authentication. Make sure to add http:// or https:// in the kafka_schema_registry_url. In Proton 1.5.3 or above, self-signed HTTPS certification is supported. 1. One solution is to set kafka_schema_registry_skip_cert_check to true. This will completely skip the TLS certification verification. In this case, you don't need to specify the certification files. 2. A more secure solution is to keep the default value of kafka_schema_registry_skip_cert_check, which is false. Omit this setting and specify the following 3 settings: kafka_schema_registry_private_key_file: the file path to the private key file used for encryption. Please use absolute file path and make sure Proton can access this file. If you are using Kubernetes or Docker, please mount the file systems properly. kafka_schema_registry_cert_file: the file path to the certificate file (in PEM format). If the private key and the certificate are stored in the same file, this can be empty if kakfa_schema_registry_private_key_file is specified. kafka_schema_registry_ca_location: the path to the file or directory containing the CA/root certificates. 3. kafka_schema_registry_private_key_file: the file path to the private key file used for encryption. Please use absolute file path and make sure Proton can access this file. If you are using Kubernetes or Docker, please mount the file systems properly. 4. kafka_schema_registry_cert_file: the file path to the certificate file (in PEM format). If the private key and the certificate are stored in the same file, this can be empty if kakfa_schema_registry_private_key_file is specified. 5. kafka_schema_registry_ca_location: the path to the file or directory containing the CA/root certificates. 1. kafka_schema_registry_private_key_file: the file path to the private key file used for encryption. Please use absolute file path and make sure Proton can access this file. If you are using Kubernetes or Docker, please mount the file systems properly. 2. kafka_schema_registry_cert_file: the file path to the certificate file (in PEM format). If the private key and the certificate are stored in the same file, this can be empty if kakfa_schema_registry_private_key_file is specified. 3. kafka_schema_registry_ca_location: the path to the file or directory containing the CA/root certificates. Make sure you define the columns matching the fields in the Avro schema. You don't have to define all top level fields in Avro schema as columns in the stream. For example, if there are 4 fields in Avro schema, you can choose only 2 of them as columns in the external stream. But make sure the data types match. data_format can be Avro, or ProtobufSingle. Schema reference is not supported yet. For examples to read Avro message in various Kafka API compatitable message platforms, please check this doc. ## Write Messages in Avro Schema​ Writing Avro/Protobuf data with schema registry is not supported in Timeplus Proton. Since Timeplus Enterprise 2.4.7 (with timeplusd 2.3.10), it can produce Kafka messages using the Avro schema registry output format. You need to set data_format='Avro', and also specify the schema registry related settings while creating the external stream. For example: When you run a INSERT query like INSERT INTO my_ex_stream ..., it will call the schema registry API to fetch the latest schema set on the topic ( currently, it does not support specifying the schema ID directly). And then, it will cache the schema in memory for that topic. Since the schema fetched from schema registry is cached in the memory, next time when another INSERT query is executed, Timeplus won't fetch the schema from the registry again. Please note, the cache is for the topic, not just for the external stream. So if you create another external stream using the same topic, it will use the cached schema as well. To force the query to refresh the schema (for example, the schema gets evolved ), you can use the force_refresh_schema setting: For the data type mappings between Avro and Timeplus data type, please check this doc. - Read Messages in Protobuf or Avro Schema - Write Messages in Avro Schema ## Public-beta-1 - [Timeplus Cloud Public Beta 1 | Timeplus](https://docs.timeplus.com/public-beta-1): We are thrilled to launch the public beta of Timeplus cloud release. - Release Notes - Older Releases - Timeplus Cloud Public Beta 1 # Timeplus Cloud Public Beta 1 We are thrilled to launch the public beta of Timeplus cloud release. We will update the beta version from time to time and list key enhancements in this page. (in year 2022) ## Biweekly Update 12/26-1/6​ - When the local disk is 90% full, Timeplus will stop writing new data. This threshold is configurable. - In the Data Lineages page, sources are shown by default. - Applied unified UI look & feel in many pages. - Released Timeplus Python SDK 1.1.1 with a more friendly API for create/ingest/query streams. - (Experimental) you can now push data to Timeplus, from your local Kafka cluster with kafka-connect-timeplus, or from your local Pulsar cluster with pulsar-io-sink, or from an AirByte cluster with destination-timeplus connector. We also documented how Timeplus Cloud can pull data from your local data source via ngrok. ## Biweekly Update 12/12-12/23​ - With the recent enhancements of the Ingest API, in many cases, you can configure other systems to push data directly to Timeplus via webhook, without writing code. - Now you can set descriptions while creating/updating streams or views. - You can edit a view to change its SQL query, without having to delete then recreate it. - For materialized views, you can see their row count and disk size. The retention policy can be updated too. - In the query page, you can filter results with keyword, without changing the SQL. - In the query page, we removed the pagination. The latest results are shown at the bottom. - For columns with long text or JSON value, in the previous version, the content can be trimmed in the query result. Now you can click on the row to show them in a side panel. With the recent enhancements of the Ingest API, in many cases, you can configure other systems to push data directly to Timeplus via webhook, without writing code. Now you can set descriptions while creating/updating streams or views. You can edit a view to change its SQL query, without having to delete then recreate it. For materialized views, you can see their row count and disk size. The retention policy can be updated too. In the query page, you can filter results with keyword, without changing the SQL. In the query page, we removed the pagination. The latest results are shown at the bottom. For columns with long text or JSON value, in the previous version, the content can be trimmed in the query result. Now you can click on the row to show them in a side panel. ## Biweekly Update 11/28-12/9​ - We added an experimental feature to create User-Defined-Aggregation-Function (UDAF) via JavaScript. You can create highly customized logic with JavaScript, even for Complex Event Processing (CEP) scenarios. Contact us if you want to try this feature. - Refined the documentation for the Python SDK https://pypi.org/project/timeplus/ - Source In the source management page, we added a sparkline to show the throughput for each source. This sparkline auto-refreshes every 15 seconds. When you create a new source and choose to send data to an existing stream, only the streams with matching schema will be shown. If no existing streams match, you have to create a new stream. In the preview step, the first 3 rows will be fetched from the source. If Timeplus cannot detect the column data type automatically, the column type will be set as unknown. This could happen if the value in those 3 events contain null. Please check with your data source provider. If you are sure the future events will be in a certain data type, such as string, you can change the column type and choose to create a new stream to receive data from the source. - In the source management page, we added a sparkline to show the throughput for each source. This sparkline auto-refreshes every 15 seconds. - When you create a new source and choose to send data to an existing stream, only the streams with matching schema will be shown. If no existing streams match, you have to create a new stream. - In the preview step, the first 3 rows will be fetched from the source. If Timeplus cannot detect the column data type automatically, the column type will be set as unknown. This could happen if the value in those 3 events contain null. Please check with your data source provider. If you are sure the future events will be in a certain data type, such as string, you can change the column type and choose to create a new stream to receive data from the source. - When you create a new materialized view, you can set a retention policy, specifying the max size or max age for the data in the materialized view. - Clicking on a recent query on the homepage will now open the query page, instead of showing the query history. - We removed the purple page description banners formerly at the top of each page. If there is no object defined in a certain page, a customized help message is shown. - You can click-and-drag to resize column width in the streaming table (query page). - An experimental alert manager UI is added. Please check our user guide. - In the source management page, we added a sparkline to show the throughput for each source. This sparkline auto-refreshes every 15 seconds. - When you create a new source and choose to send data to an existing stream, only the streams with matching schema will be shown. If no existing streams match, you have to create a new stream. - In the preview step, the first 3 rows will be fetched from the source. If Timeplus cannot detect the column data type automatically, the column type will be set as unknown. This could happen if the value in those 3 events contain null. Please check with your data source provider. If you are sure the future events will be in a certain data type, such as string, you can change the column type and choose to create a new stream to receive data from the source. ## Biweekly Update 11/14-11/25​ - Source, sink, API and SDK We built and open-sourced the Pulsar Sink Connector for Timeplus. You can install this connector in your Pulsar cluster and push real-time data to Timeplus. We released a major upgrade of the Python SDK https://pypi.org/project/timeplus/1.0.0/ The code and documentations are automatically generated by Swagger Codegen , so that it will be always aligned with our latest REST API. Please note this is not compatible with previous 0.2.x or 0.3.x SDK. If you are using those SDKs, please plan your migration. New API is only available in the 1.x SDK. We further enhanced our ingest REST API to support more systems, such as vector and auth0. If you would like to leverage such a 3rd party system/tool to push data to Timeplus but it doesn't allow custom content type, then you can use the standard application/json content type, and send POST request to /api/v1beta1/streams/$STREAM_NAME/ingest?format=streaming. This will ensure the Timeplus API server treats the POST data as NDJSON. For the API authentication, besides the custom HTTP Header X-Api-Key:THE_KEY, we now also support Authorization: ApiKey THE_KEY Learn more Ingest API - We built and open-sourced the Pulsar Sink Connector for Timeplus. You can install this connector in your Pulsar cluster and push real-time data to Timeplus. - We released a major upgrade of the Python SDK https://pypi.org/project/timeplus/1.0.0/ The code and documentations are automatically generated by Swagger Codegen , so that it will be always aligned with our latest REST API. Please note this is not compatible with previous 0.2.x or 0.3.x SDK. If you are using those SDKs, please plan your migration. New API is only available in the 1.x SDK. - We further enhanced our ingest REST API to support more systems, such as vector and auth0. If you would like to leverage such a 3rd party system/tool to push data to Timeplus but it doesn't allow custom content type, then you can use the standard application/json content type, and send POST request to /api/v1beta1/streams/$STREAM_NAME/ingest?format=streaming. This will ensure the Timeplus API server treats the POST data as NDJSON. For the API authentication, besides the custom HTTP Header X-Api-Key:THE_KEY, we now also support Authorization: ApiKey THE_KEY Learn more Ingest API - UI improvements In the signup/login page, we added the WeChat integration. You can scan the QR code with your phone and sign up or log in. When a query is finished, canceled, or paused, you can download the current results as a CSV. This is helpful when there are multiple pages of results. When you click an entity on a Data Lineages page, such as a stream or a view, a summary is now shown in the side panel, instead of a pop-up, allowing you to see more detailed information. We added an experimental UI for the alert manager. Want to be the first to try this feature? Get in touch with us! - In the signup/login page, we added the WeChat integration. You can scan the QR code with your phone and sign up or log in. - When a query is finished, canceled, or paused, you can download the current results as a CSV. This is helpful when there are multiple pages of results. - When you click an entity on a Data Lineages page, such as a stream or a view, a summary is now shown in the side panel, instead of a pop-up, allowing you to see more detailed information. - We added an experimental UI for the alert manager. Want to be the first to try this feature? Get in touch with us! ... (content truncated) ## Public-beta-2 - [Timeplus Cloud Public Beta 2 | Timeplus](https://docs.timeplus.com/public-beta-2): We are thrilled to launch the next phase of the public beta of Timeplus cloud release. Compared to our Public Beta 1, most of the backend and frontend changes are incremental enhancements, except the entry point is changed from https//us.timeplus.cloud - Release Notes - Older Releases - Timeplus Cloud Public Beta 2 # Timeplus Cloud Public Beta 2 We are thrilled to launch the next phase of the public beta of Timeplus cloud release. Compared to our Public Beta 1, most of the backend and frontend changes are incremental enhancements, except the entry point is changed from https://beta.timeplus.cloud to https://us.timeplus.cloud We will update the beta version from time to time and list key enhancements in this page. ## Jul 24, 2023​ Dashboards and Charts - When you make changes to your SQL, we will try to preserve the chart's formatting settings, such as when modifying the WHERE condition or ORDER BY. - Table chart: click and drag to resize column widths in edit mode. - Single value chart: more formatting options are added – change the colors of the number, delta increases/decreases, and sparkline. - Enhanced the color picker for charts to show shades of a color. Other UI Enhancements - In the results table of the Query page, for date/time columns, we now show the original timestamp (no longer showing the browser's local timezone). In Timeplus, by default the _tp_time column is created in the UTC timezone. Please check the _tp_time (Event time) for details. - Improved how expiry dates are shown for API keys in Personal Settings. Streaming Database and SQLs - Performance for Sources and Sinks are greatly improved. - Improvements for JOIN: Previously, if you run JOIN for a normal stream (append-only) with a Versioned Stream, all primary key columns of the Versioned Stream need to be in the ON clause. Now, as long as all primary key columns are selected, you can use one or more primary key columns in the ON clause. (Experimental) We also added support for Changelog Stream JOIN Changelog Stream, or Changelog Stream JOIN Versioned Stream. - Previously, if you run JOIN for a normal stream (append-only) with a Versioned Stream, all primary key columns of the Versioned Stream need to be in the ON clause. Now, as long as all primary key columns are selected, you can use one or more primary key columns in the ON clause. - (Experimental) We also added support for Changelog Stream JOIN Changelog Stream, or Changelog Stream JOIN Versioned Stream. - Previously, if you run JOIN for a normal stream (append-only) with a Versioned Stream, all primary key columns of the Versioned Stream need to be in the ON clause. Now, as long as all primary key columns are selected, you can use one or more primary key columns in the ON clause. - (Experimental) We also added support for Changelog Stream JOIN Changelog Stream, or Changelog Stream JOIN Versioned Stream. ## Jul 10, 2023​ Resource Tables UI - Resource tables (such as Sources, Streams, Sinks. etc.) now show a Created By/On column. - These tables no longer auto-refresh. Click the arrow icon at the top-right corner to manually refresh the table. - For Versioned and Changelog streams, we now show a key icon to indicate that it's not an Append-Only stream. The row count and earliest/latest event columns are also now hidden. Query Page - Added keyboard shortcuts to add or remove comments in the SQL editor: On PC: Ctrl + / to toggle line comment, and Shift + Alt + A to toggle block comment On Mac: Cmd + / to toggle line comment, and Shift + Option + A to toggle block comment - More improved error messages for SQL queries, to show more details. Sources - For Apache Kafka, Confluent Cloud, and Redpanda sources, we removed "Advanced Settings" during the Pull Data step, so the "Consumer Group" field is now shown directly. ## Jun 27, 2023​ A beta version of the new Timeplus Python SDK is available. It supports SQL Academy so we can integrate Timeplus with rich ecosystems in Python, such as Superset, QueryBook, and LangChain. Contact us if you want to try this new feature. The Meltano plugin for Timeplus is updated to use the latest Python SDK and support more flexible data schema. Console UI - For changelog or versioned streams, you can create complex primary keys which include multiple columns. - Side panels in the Query page have a new look: the SQL Helper panel no longer overlaps the SQL editor and results table, and the row details panel is now shown inside the results table. - For Sources and Sinks, click on the "i" next to the status, to see the last error message. - Improved error messages for SQL queries, to show more details. - For API keys, we now show the first four and last four characters, and the previous name is now the description. REST API - We marked the required parameters in our REST API documentation and completed validation to avoid creating resources without required information. - Updated our REST API docs to include more descriptions. ## Jun 12, 2023​ New Feature - Introducing collaboration in Timeplus: you can invite team members to your Timeplus workspace (during our Public Beta, you may invite unlimited team members). Members in the same workspace can access and edit all streams, sources, sinks, dashboards, etc. Only the workspace owner can invite or remove members. Data Ingestion and Sinks - You can now ingest data to Timeplus via Kafka Connect plugin for Timeplus. It is published on Confluent Hub, and you can install it on Confluent Cloud or on-prem Confluent Platform and Apache Kafka. - We've added Protobuf encoding for Apache Kafka and Redpanda sinks, currently available via REST API. Contact us to try it out! Other Enhancements - For streams, you can now set separate retention policies for streaming and historical data. For example, for a stream with IoT data, you can set 1 day for the streaming data and 90 days for the historical data. - Please note: REST API v1beta1 is now deprecated, and it will be removed in a couple of months. Please check the v1beta2 API Docs. Python SDK, Java Sample Code and Streamlit Demo App are updated. ## May 29, 2023​ Data Ingestion - We now have a consolidated page for adding your data to Timeplus. The new "Add Data" page allows you to add a source (such as Apache Kafka, Confluent Cloud, Redpanda, etc.), push data via Ingest REST API, import a CSV file, and connect an external stream (with Kafka data). Your added sources and external streams are also listed on this page, in separate tabs. - In the Apache Kafka and Redpanda sources, you can now encode/decode Protobuf. During the "Pull Data" step, enter your Protobuf message name and definition. - We also now support Debezium JSON as a read data format, with Upsert and CRUD (with CRUD currently being in preview). Streaming Database and SQL - We've made a significant performance enhancement for the hop window, from 10x to 100x. This is especially helpful for large window sizes, such as 24 hours as a total window size, but showing updates every second. - User-defined functions (UDFs) and user-defined aggregates (UDAs) are now able to return an array as a data type. Other Enhancements - (Experimental) We've added a collaboration feature, allowing you to invite team members to your workspace. Contact us to try it out! - We've reorganized our navigation menu to provide a clearer workflow: "Add Data" is now at the top, followed by "Run SQL". We've also added "Workspace Settings" to the menu, for quicker access to your UDFs and team members. ## May 16, 2023​ **Enhancements: Data Ingestion ** - We've made it easier for you to push data via Ingest REST API. We've added a wizard page in our console to guide you through the steps of selecting a target stream, a request template, an API key, and then generating a sample code for the push request. Learn more. - Our CSV file upload wizard has a new look, in the same UI style as our other wizards. Enhancements: Dashboards and Charts - When you make formatting changes to a chart, your changes will be instant, instead of needing to reload the chart. - We added an icon on mouseover to Dashboard charts, while in View-Only mode, that will open the chart's SQL in the Query editor page. ## May 1, 2023​ Streaming Database and SQL - (Experimental) In addition to the default append-only data streams, you can now create streams with mutations and versions. You can leverage tools, such as Debezium, to load CDC (Change Data Capture) data from different sources and track the INSERT, UPDATE, DELETE operations in Timeplus. You can always get the latest event for any primary key. Learn more. Dashboards - (Experimental) A new map visualization type is added, showing the latest locations in a live map view. - For table visualization, if the update mode is per-key, you can enable a sparkline to track value changes for certain columns. Other Enhancements - If you run SELECT * FROM my_stream and there is no data coming in for the first while, Timeplus will show you a hint message with SQL to query historical data. - A better color picker is added in visualization configuration forms. - The UI for our sample dataset is simplified. Instead of the regular configuration flow for other sources, you can now quickly generate sample data with a couple of clicks. ## April 17, 2023​ Streaming Database and SQL - For stateful time window aggregations (tumble/hop/session), Timeplus now supports sub-second intervals: ms for milliseconds, us for microseconds, and ns for nanoseconds. For example, you can run a streaming SQL to show results every 10 milliseconds for the past 1 second sliding window: select window_start, stock, avg(price) from hop(stocks,10ms,1s) group by window_start, stock. Two months ago, we also added the capability to run global aggregations with fixed interval at sub-second level, such as select sum(price) from stocks emit periodic 50ms - Added new functions md5, md4, and hex, which can help you generate hash keys. Dashboards ... (content truncated) ## Pulsar-external-stream - [Pulsar External Stream | Timeplus](https://docs.timeplus.com/pulsar-external-stream): Apache® Pulsar™ is a multi-tenant, high-performance solution for server-to-server messaging. - Core Features - External Streams & Tables - External Streams - Apache Pulsar # Pulsar External Stream Apache® Pulsar™ is a multi-tenant, high-performance solution for server-to-server messaging. In Timeplus Enterprise v2.5, we added the first-class integration for Apache Pulsar, as a new type of External Stream. You can read or write data in Apache Pulsar or StreamNative Cloud. This is also available in Timeplus Proton, since v1.6.8. ## CREATE EXTERNAL STREAM​ To create an external stream for Apache Pulsar, you can run the following DDL SQL: ### Connect to a local Apache Pulsar​ If you have a local Apache Pulsar server running, you can run the following SQL DDL to create an external stream to connect to it. ### Connect to StreamNative Cloud​ If you have the access to StreamNative Cloud, you can run the following SQL DDL to create an external stream to connect to it, with a proper API Key for a service account. Make sure you choose "Create API Key", instead of the "Get JWT Token (Depreciated)". ### DDL Settings​ #### skip_server_cert_check​ Default false. If set to true, it will accept untrusted TLS certificates from brokers. #### validate_hostname​ Default false. Configure whether it allows validating hostname verification when a client connects to a broker over TLS. #### ca_cert​ The CA certificate (PEM format), which will be used to verify the server's certificate. #### client_cert​ The certificate (PEM format) for the client to use mTLS authentication. Learn more. #### client_key​ The private key (PEM format) for the client to use mTLS authentication. #### jwt​ The JSON Web Tokens for the client to use JWT authentication. Learn more. #### config_file​ The config_file setting is available since Timeplus Enterprise 2.7. You can specify the path to a file that contains the configuration settings. The file should be in the format of key=value pairs, one pair per line. You can set the Pulsar credentials in the file. Please follow the example in Kafka External Stream. #### connections_per_broker​ Default 1. Sets the max number of connection that this external stream will open to a single broker. By default, the connection pool will use a single connection for all the producers and consumers. #### memory_limit​ Default 0 (unlimited). Configure a limit on the amount of memory that will be allocated by this external stream. #### io_threads​ Default 1. Set the number of I/O threads to be used by the Pulsar client. Like Kafka External Stream, Pulsar External Stream also supports all format related settings: data_format, format_schema, one_message_per_row, etc. #### data_format​ The supported values for data_format are: - JSONEachRow: parse each row of the message as a single JSON document. The top level JSON key/value pairs will be parsed as the columns. Learn More. - CSV: less commonly used. Learn More. - TSV: similar to CSV but tab as the separator - ProtobufSingle: for single Protobuf message per message - Protobuf: there could be multiple Protobuf messages in a single message. - Avro - RawBLOB: the default value. Read/write message as plain text. For data formats which write multiple rows into one single message (such as JSONEachRow or CSV), two more advanced settings are available: #### max_insert_block_size​ max_insert_block_size to control the maximum number of rows can be written into one message. #### max_insert_block_bytes​ max_insert_block_bytes to control the maximum size (in bytes) that one message can be. ## Read Data in Pulsar​ ### Read messages in a single column​ If the message in Pulsar topic is in plain text format or JSON, you can create an external stream with only a raw column in string type. Example: Then use query time JSON extraction functions or shortcut to access the values, e.g. raw:id. ### Read messages as multiple columns​ If the keys in the JSON message never change, or you don't care about the new columns, you can also create the external stream with multiple columns. You can pick up some top level keys in the JSON as columns, or all possible keys as columns. Example: If there are nested complex JSON in the message, you can define the column as a string type. Actually any JSON value can be saved in a string column. ### Virtual Columns​ Pulsar external stream has the follow virtual columns: #### _tp_time​ the event time of the Pulsar message if it's available, or it's the publish time otherwise. #### _tp_append_time​ the publish time of the pulsar message. #### _tp_process_time​ the timestamp when the message was read by Pulsar External Stream. #### _tp_shard​ the partition ID, starting from 0. #### _pulsar_message_id​ an array which contains 4 elements: ledger_id, entry_id, partition, and batch_index. #### _tp_sn​ the sequence number in Timeplus, in int64 type. #### _tp_message_key​ the message key (a.k.a partition key). Can be empty. #### _tp_message_headers​ Starting from Timeplus Enterprise 2.8.2, you can read and write custom headers via this column. Define the column in the DDL: Then insert data to the external stream via INSERT INTO or materialized views, with a map of string pairs as custom headers for each message. ### Query Settings​ #### shards​ You can read in specified Pulsar partitions. By default, all partitions will be read. But you can also read from a single partition via the shards setting, e.g. Or you can specify a set of partition ID, separated by comma, e.g. #### record_consume_timeout_ms​ Use setting record_consume_timeout_ms to determine how much time the external can wait for new messages before returning the query result. The smaller the value is, the smaller the latency will be, but also will be less performant. ### Read existing messages​ When you run SELECT raw FROM ext_stream , Timeplus will read the new messages in the topics, not the existing ones. #### seek_to​ If you need to read all existing messages, you can use the following settings: Or the following SQL: Note: both earliest and latest are supported. You can also use seek_to='2024-10-14' for date or datetime based rewind. But number-based seek_to is not supported. Please avoid scanning all existing data via select * from table(ext_stream). ### Read/Write Pulsar Message Key​ For each message in the topic, the value is critical for sure. The key is optional but could carry important meta data. You can define the _tp_message_key column when you create the external stream. For example: You can insert any data to the Pulsar topic. When insert a row to the stream like: 'some-key' will be used for the message key for the Pulsar message (and it will be excluded from the message body, so the message will be {"id": 1, "name": "John"} for the above SQL). When doing a SELECT query, the message key will be populated to the _tp_message_key column as well. SELECT * FROM test_msg_key will return 'some-key' for the _tp_message_key message. _tp_message_key support the following types: uint8, uint16, uint32, uint64, int8, int16, int32, int64, bool, float32, float64, string, and fixed_string. _tp_message_key also support nullable. Thus we can create an external stream with optional message key. For example: ## Write Data to Pulsar​ ### Write to Pulsar in Plain Text​ You can write plain text messages to Pulsar topics with an external stream with a single column. Then use either INSERT INTO VALUES (v), or Ingest REST API, or set it as the target stream for a materialized view to write message to the Pulsar topic. The actual data_format value is RawBLOB but this can be omitted. By default one_message_per_row is true. #### Advanced Settings for writing data​ Settings for controlling the producer behavior: - output_batch_max_messages - Set the max number of messages permitted in a batch. If you set this option to a value greater than 1, messages are queued until this threshold is reached or batch interval has elapsed. - output_batch_max_size_bytes - Set the max size of messages permitted in a batch. If you set this option to a value greater than 1, messages are queued until this threshold is reached or batch interval has elapsed. - output_batch_max_delay_ms - Set the max time for message publish delay permitted in a batch. - pulsar_max_pending_messages - Set the max size of the producer's queue holding the messages pending to receive an acknowledgment from the broker. When the queue is full, the producer will be blocked. ### Multiple columns to write to Pulsar​ To write structured data to Pulsar topics, you can choose different data formats: ##### RawBLOB​ Write the content as pain text. #### JSONEachRow​ You can use data_format='JSONEachRow',one_message_per_row=true to inform Timeplus to write each event as a JSON document. The columns of the external stream will be converted to keys in the JSON documents. For example: The messages will be generated in the specific topic as Please note, by default multiple JSON documents will be inserted to the same Pulsar message. One JSON document each row/line. Such default behavior aims to get the maximum writing performance to Pulsar. But you need to make sure the downstream applications are able to properly split the JSON documents per message. If you need a valid JSON per each message, instead of a JSONL, please set one_message_per_row=true e.g. The default value of one_message_per_row, if not specified, is false for data_format='JSONEachRow' and true for data_format='RawBLOB'. #### CSV​ You can use data_format='CSV' to inform Timeplus to write each event as a JSON document. The columns of the external stream will be converted to keys in the JSON documents. For example: The messages will be generated in the specific topic as ##### TSV​ Similar to CSV but tab as the separator. #### ProtobufSingle​ You can write Protobuf-encoded messages in Pulsar topics. ... (content truncated) ## Py-udf - [Python UDF | Timeplus](https://docs.timeplus.com/py-udf): In addition to Remote UDF and JavaScript UDF, starting from v2.7, Timeplus Enterprise also supports Python-based UDF, as a feature in technical preview. You can develop User-defined scalar functions (UDFs) or User-defined aggregate functions (UDAFs) with the embedded Python 3.10 runtime in Timeplus core engine. No need to deploy extra server/service for the UDF. - SQL Reference - Functions - User Defined Functions - Python UDF # Python UDF In addition to Remote UDF and JavaScript UDF, starting from v2.7, Timeplus Enterprise also supports Python-based UDF, as a feature in technical preview. You can develop User-defined scalar functions (UDFs) or User-defined aggregate functions (UDAFs) with the embedded Python 3.10 runtime in Timeplus core engine. No need to deploy extra server/service for the UDF. For visual learners, please watch the following video: ## Why Python UDF​ Python is recognized as one of the most popular languages in the field of data science. Its flexibility as a scripting language, ease of use, and extensive range of statistical libraries make it an indispensable tool for data scientists and analysts. Python excels in writing complex parsing and data transformation logic, especially in scenarios where SQL capabilities are insufficient. Python User-Defined Functions (UDFs) offer the flexibility to implement intricate data processing mechanisms. These include: - Custom Tokenization: Breaking down data into meaningful elements based on specific criteria. - Data Masking: Concealing sensitive data elements to protect privacy. - Data Editing: Modifying data values according to specific rules or requirements. - Encryption Mechanisms: Applying encryption to data for security purposes. ## Data type mapping​ This is the mapping for Timeplus data type and Python data type: If your use cases require more data type support, please contact us at support@timeplus.com. ## Register a Python UDF​ You can create or replace a Python UDF with SQL. Web UI will be added. ### Scalar UDF​ Scalar UDF is stateless UDF to convert columns in one row to other values. Syntax: You need to make sure the SQL function name is identical to the function name in the Python code. ### UDAF​ UDAF or User Defined Aggregation Function is stateful. It takes one or more columns from a set of rows and return the aggregated result. Syntax: The function list: - process the core logic of the aggregation function, required. - finalize return the aggregation result, required. - serialize save the state as a string or pickle binary and put in checkpoint, optional. - deserialize load the state from checkpoint to the internal state, optional. - merge for multi-shard processing, merge the states from each shard, optional. ## Examples​ ### A simple UDF without dependency​ Timeplus Python UDF supports the standard Python library and the built-in functions. This example takes the number as input, add 5. Please note: - To improve the performance, Timeplus calls the UDF with a batch of inputs. The input of the Python function add_five is list(int). - The function name add_five in the SQL statement should match the function name in the Python code block. - Python code block should be enclosed in $$. Alternatively, you can use ' to enclose the code block, but this may cause issues with the Python code block if it contains '. - Python code is indented with spaces or tabs. It's recommended to put def at the beginning of the line without indentation. ### A simple UDF with numpy​ Numpy is a general-purpose array-processing package. It provides a high-performance multidimensional array object, and tools for working with these arrays. It is the fundamental package for scientific computing with Python. This library is not installed by default. You need to install it manually by following the guide. This example takes the number as input, add 5 via numpy. Please note, to improve the performance, Timeplus calls the UDF with a batch of inputs. The input of the Python function add_five is list(int). We use numpy.array(list) to convert it to a numpy array. ### A simple UDAF with pickle​ Pickle implements binary protocols for serializing and de-serializing a Python object structure. This example gets the maximum number and use pickle to save/load the state. ## Manage Python Libraries​ By default, Timeplus Enterprise ships a clean Python 3.10 environment, plus the following essential libraries: ### Verified Libraries​ Follow the guide below to install extra Python libraries. The following libraries are verified by Timeplus team. - Numpy - Pandas - Arrow - Scipy - Sklearn - River - Statsmodels Some Python libraries may require additional dependencies or OS specific packages. Contact us if you need help. ### Install Python Libraries​ To install new Python libraries, you can either call the REST API of timeplusd in Timeplus Enterprise v2.7, or use the new timeplusd python pip install command-line tool introduced in Timeplus Enterprise v2.8. #### Install via timeplusd python pip​ Starting from Timeplus Enterprise v2.8, you can use the timeplusd python pip install command-line tool to install Python libraries. For example, to install the numpy library, you can use the following command: For example, with the timeplusd docker image, you can use the following command: #### Install via REST API​ You can also call the REST API of timeplusd in Timeplus Enterprise v2.7 or above. To access the REST API, you need to create an administrator account and set the HTTP headers x-timeplus-user and x-timeplus-key with the user and password, such as curl -H "x-timeplus-user: theUser" -H "x-timeplus-key:thePwd" ... For example, if you want to install the numpy library, you can use the following command: If you need to install a specific version of a library, you can specify it in the version field. For example, to install numpy version 2.2.3, you can use the following command: ### List Python Libraries​ To list the extra Python libraries installed in Timeplus Enterprise, you can use the following command: ### Delete Python Libraries​ To delete Python libraries, you can call the REST API of timeplusd in Timeplus Enterprise. For example, if you want to delete the numpy library, you can use the following command: ### Update Python Libraries​ Currently we don't support updating Python libraries. You can delete the library and reinstall it with the desired version. ## Limitations​ - For Linux bare metal deployments, Glibc version 2.35 or higher is required. - Only Python 3.10 is supported. Contact us if you need to install a specific version. - Not all Python libraries can be installed in Timeplus Enterprise. Contact us if you need to install a specific library. - Why Python UDF - Data type mapping - Register a Python UDFScalar UDFUDAF - Scalar UDF - UDAF - ExamplesA simple UDF without dependencyA simple UDF with numpyA simple UDAF with pickle - A simple UDF without dependency - A simple UDF with numpy - A simple UDAF with pickle - Manage Python LibrariesVerified LibrariesInstall Python LibrariesInstall via timeplusd python pipInstall via REST APIList Python LibrariesDelete Python LibrariesUpdate Python Libraries - Verified Libraries - Install Python LibrariesInstall via timeplusd python pipInstall via REST API - Install via timeplusd python pip - Install via REST API - List Python Libraries - Delete Python Libraries - Update Python Libraries - Limitations - Scalar UDF - UDAF - A simple UDF without dependency - A simple UDF with numpy - A simple UDAF with pickle - Verified Libraries - Install Python LibrariesInstall via timeplusd python pipInstall via REST API - Install via timeplusd python pip - Install via REST API - List Python Libraries - Delete Python Libraries - Update Python Libraries - Install via timeplusd python pip - Install via REST API ## Query-api - [Query API with Server-sent Events (SSE) | Timeplus](https://docs.timeplus.com/query-api): All Timeplus functions are available through REST API. To support real-time query result pushed from server to client side, there are two popular solutions, websocket and server-sent events. Timeplus is now leveraging server-sent events to push real-time query results to the client. - Integrations - CLI, APIs & SDKs - Query API with Server-sent Events (SSE) # Query API with Server-sent Events (SSE) All Timeplus functions are available through REST API. To support real-time query result pushed from server to client side, there are two popular solutions, websocket and server-sent events. Timeplus is now leveraging server-sent events to push real-time query results to the client. ## Event Stream Protocol​ An SSE event stream is delivered as a streaming HTTP response: the client initiates a regular HTTP request, the server responds with a custom "text/event-stream" content-type, and then streams the UTF-8 encoded event data. Here is a sample: There are three different types of events: ### Query Metadata (event: query)​ Query metadata is always the first SSE message returned through the query API, it has event type query, which is a json object, some often used fields are id, sql and result.header. The content of the metadata is the same as the response of get query. In case of historical (table) query, the last event will query metadata as well. The reason we send it again at the end of the SSE session is to notify the client about the final status of the query. A typical use case here is that the client can make use of the final duration and end_time of the query. ### Query Results (no event type)​ We don't assign any event type in order to save bandwidth here. Each query result message is an array of arrays, representing multiple rows of query results. The data in each row follows the same order defined in the result.header. ### Query Metric (event: metric)​ The last event type is metrics, which returns the current query statistics in a json object, which can be used to observe the health status of the current query. Take the example above: ### Query Metric (event: metrics)​ Query metric message represents the current query statistics, which can be used to observe the status of current query such as number of scanned rows. ## SSE Client​ There are different SSE libraries that provide SSE client functions, Timeplus also has a python SDK project which already included SSE event parsing to avoid handling those low-level details. - Event Stream ProtocolQuery Metadata (event: query)Query Results (no event type)Query Metric (event: metric)Query Metric (event: metrics) - Query Metadata (event: query) - Query Results (no event type) - Query Metric (event: metric) - Query Metric (event: metrics) - SSE Client - Query Metadata (event: query) - Query Results (no event type) - Query Metric (event: metric) - Query Metric (event: metrics) ## Query-settings - [Query Settings | Timeplus](https://docs.timeplus.com/query-settings): Timeplus supports some advanced SETTINGS to fine tune the streaming query processing behaviors, listed below: - SQL Reference - Query Settings # Query Settings Timeplus supports some advanced SETTINGS to fine tune the streaming query processing behaviors, listed below: ## query_mode​ query_mode=. By default, if it's omitted, it's streaming. A general setting which decides if the overall query is streaming data processing or historical data processing. This can be overwritten in the port. If you use 3128, default is streaming. If you use 8123, default is historical. ## seek_to​ seek_to=. By default, if it's omitted, it's latest. A setting which tells Timeplus to seek old data in the streaming storage by timestamp. It can be a relative timestamp or an absolute timestamp. By default, it is latest, which tells Timeplus to not seek old data. Example:seek_to='2022-01-12 06:00:00.000', seek_to='-2h', or seek_to='earliest' Please note, as of Jan 2023, we no longer recommend you use SETTINGS seek_to=..(except for External Stream). Please use WHERE _tp_time>='2023-01-01' or similar. _tp_time is the special timestamp column in each raw stream to represent the event time. You can use >, <, BETWEEN .. AND operations to filter the data in Timeplus storage. The only exception is External Stream. If you need to scan all existing data in the Kafka topic, you need to run the SQL with seek_to, e.g. select raw from my_ext_stream settings seek_to='earliest' ## enable_backfill_from_historical_store​ enable_backfill_from_historical_store=0|1. By default, if it's omitted, it's 1. - When it's 0, the query engine either loads data from streaming storage, or from historical storage. - When it's 1, the query engine evaluates whether it's necessary to load data from historical storage(such as the time range is outside of the streaming storage), or it'll be more efficient to get data from historical storage(for example, count/min/max is pre-computed in historical storage, faster than scanning data in streaming storage). ## force_backfill_in_order​ force_backfill_in_order=0|1. By default, if it's omitted, it's 0. 1. When it's 0, the data from the historical storage are turned without extra sorting. This would improve the performance. 2. When it's 1, the data from the historical storage are turned with extra sorting. This would decrease the performance. So turn on this flag carefully. ## emit_during_backfill​ emit_during_backfill=0|1. By default, if it's omitted, it's 0. 1. When it's 0, the query engine won't emit intermediate aggregation results during the historical data backfill. 2. When it's 1, the query engine will emit intermediate aggregation results during the historical data backfill. This will ignore the force_backfill_in_order setting. As long as there are aggregation functions and time window functions(e.g. tumble/hop/session) in the streaming SQL, when the emit_during_backfill is on, force_backfill_in_order will be applied to 1 automatically. ## recovery_policy​ recovery_policy=. By default, if it's omitted, it's strict. The main use case for materialized views, if new events fail to process, such as converting a string to an int32, the default behavior will make the materialized view unusable. You may monitor the Timeplus logs to act on the dirty data. However, if you set SETTINGS recovery_policy=best_effort, then Timeplus will attempt to recover from checkpoint and try up to 3 times, then skip dirty data and continue processing the rest of the data. ## replay_speed​ You can replay historical data with replay_speed setting via a streaming SQL query. The syntax is: When replay_speed set to 1, it will use the replay_time_column (_tp_append_time as the default) to replay the historical data. When replay_speed is not set or set to 0, historical data will be replayed as fast as possible. When replay_speed set to 0 to 1, it will replay slower. If it's greater than 1, it will replay faster. Starting from Timeplus Enterprise 2.7 and Timeplus Proton 1.6.10, you can replay both normal streams and Kafka external streams. ## replay_time_column​ replay_time_column=columnName Specify the replay time column, default column is _tp_append_time. ## replay_start_time​ replay_start_time is the start time for replaying historical data. It's a timestamp string. For example, replay_start_time='2025-01-01 00:00:00.000'. This should be equal or greater than WHERE _tp_time > .. time rewind condition. ## replay_end_time​ replay_end_time is the end time for replaying historical data. It's a timestamp string. For example, replay_end_time='2025-01-02 00:00:00.000'. When all data is replayed, the query will end. ## default_hash_table​ Default value is memory. To optimize memory usage for large data streams, you can set default_hash_table='hybrid'. This will use both memory and disk to store the hash table. Available since Timeplus Enterprise 2.6.0. ## default_hash_join​ Default value is memory. To optimize memory usage for joining large data streams, you can set default_hash_join='hybrid'. This will use both memory and disk to store the hash table. Available since Timeplus Enterprise 2.6.0. ## max_threads​ Type: uint64 Default: 0, which means the number of threads for request execution is equal to the number of CPU cores. ## enable_idempotent_processing​ Type: bool Default: false Whether to enable idempotent processing for the query. If set to true, the query will be processed idempotently. If set to 0, the query will be processed non-idempotently. Learn more. ## join_max_buffered_bytes​ Type: uint64 Default: 524288000 (500MB) Maximum size of the buffer for streaming join operations. Affects memory usage and performance. Example: ## join_algorithm​ Type: string Default: default Specifies which JOIN algorithm is used. Possible values: - default: Same as direct,parallel_hash,hash, i.e. try to use direct join, parallel hash join, and hash join (in this order). - hash: Use hash join. The most generic implementation that supports all combinations of kind and strictness and multiple join keys that are combined with OR in the JOIN ON section. When using the hash algorithm, the right part of JOIN is uploaded into RAM. - direct: This algorithm can be applied when the storage for the right table supports key-value requests, such as a dictionary. The direct algorithm performs a lookup in the right table using rows from the left table as keys. It only supports INNER JOIN and LEFT JOIN. - auto: When set to auto, hash join is tried first, and the algorithm is switched on the fly to another algorithm if the memory limit is violated. ## precise_float_parsing​ By default this setting is disabled. Setting this to 1 or true to handle precise parsing for float values. For example: Please note, enabling this can slow down the query performance to some extent. One of the common use cases is the upstream, such as CDC, shares the precise float numbers as string values in Kafka or Pulsar messages. You can add this setting to the materialized view to parse the float value from string precisely. - query_mode - seek_to - enable_backfill_from_historical_store - force_backfill_in_order - emit_during_backfill - recovery_policy - replay_speed - replay_time_column - replay_start_time - replay_end_time - default_hash_table - default_hash_join - max_threads - enable_idempotent_processing - join_max_buffered_bytes - join_algorithm - precise_float_parsing ## Query-syntax - [Syntax | Timeplus](https://docs.timeplus.com/query-syntax): Timeplus introduces several SQL extensions to support streaming processing. The overall syntax looks like this: - SQL Reference - Syntax # Syntax Timeplus introduces several SQL extensions to support streaming processing. The overall syntax looks like this: Only SELECT and FROM clauses are required (you can even omit FORM, such as SELECT now(), but it's less practical). Other clauses in [..] are optional. We will talk about them one by one in the reverse order, i.e. SETTINGS, then EMIT, LIMIT, etc. SQL keywords and function names are case-insensitive, while the column names and stream names are case-sensitive. ## Streaming First Query Behavior​ Before we look into the details of the query syntax, we'd like to highlight the default query behavior in Timeplus Proton is in the streaming mode, i.e. - SELECT .. FROM stream will query the future events. Once you run the query, it will process new events. For example, if there are 1,000 events in the stream already, running SELECT count() FROM stream could return 0, if there is more new events. - SELECT .. FROM table(stream) will query the historical data, just like many of other databases. In the above sample stream, if you run SELECT count() FROM table(stream), you will get 1000 as the result and the query completed. ## SETTINGS​ Timeplus supports some advanced SETTINGS to fine tune the streaming query processing behaviors. Check Query Settings. ## PARTITION BY​ PARTITION BY in Streaming SQL is to create substreams. ## GROUP BY and HAVING​ GROUP BY applies aggregations for 1 or more columns. When GROUP BY is applied, HAVING is optional to filter the aggregation results. The difference between WHERE andHAVING is data will be filtered by WHERE clause first, then apply GROUP BY, and finally apply HAVING. ## LIMIT​ LIMIT n When the nth result is emitted, the query will stop, even it's a streaming SQL. ### OFFSET​ You can combine LIMIT and OFFSET, such as: This will fetch the 3 rows from the 2nd smallest value of a. ## JOINs​ Please check Joins. ## WITH cte​ CTE, or Common Table Expression, is a handy way to define subqueries one by one, before the main SELECT clause. ## Subquery​ ### Vanilla Subquery​ A vanilla subquery doesn't have any aggregation (this is a recursive definition), but can have arbitrary number of filter predicates, transformation functions. Some systems call this flat map. Examples Vanilla subquery can be arbitrarily nested until Timeplus' system limit is hit. The outer parent query can be any normal vanilla query or windows aggregation or global aggregation. Users can also write the query by using Common Table Expression (CTE) style. Multiple CTE can be defined in one query, such as CTE with column alias is not supported. ### Streaming Window Aggregated Subquery​ A window aggregate subquery contains windowed aggregation. There are some limitations users can do with this type of subquery. 1. Timeplus supports window aggregation parent query over windowed aggregation subquery (hop over hop, tumble over tumble etc), but it only supports 2 levels. When laying window aggregation over window aggregation, please pay attention to the window size: the window 2. Timeplus supports multiple outer global aggregations over a windowed subquery. (Not working for now). 3. Timeplus allows arbitrary flat transformation (vanilla query) over a windows subquery until a system limit is hit. Examples ### Global Aggregated Subquery​ A global aggregated subquery contains global aggregation. There are some limitations users can do with global aggregated subquery: 1. Timeplus supports global over global aggregation and there can be multiple levels until a system limit is hit. 2. Flat transformation over global aggregation can be multiple levels until a system limit is hit. 3. Window aggregation over global aggregation is not supported. Examples ## Common Types of Queries​ ### Streaming Tailing​ Examples The above example continuously evaluates the filter expression on the new events in the stream device_utils to filter out events which have cpu_usage less than 99. The final events will be streamed to clients. - Streaming First Query Behavior - SETTINGS - PARTITION BY - GROUP BY and HAVING - LIMITOFFSET - OFFSET - JOINs - WITH cte - SubqueryVanilla SubqueryStreaming Window Aggregated SubqueryGlobal Aggregated Subquery - Vanilla Subquery - Streaming Window Aggregated Subquery - Global Aggregated Subquery - Common Types of QueriesStreaming Tailing - Streaming Tailing - OFFSET - Vanilla Subquery - Streaming Window Aggregated Subquery - Global Aggregated Subquery - Streaming Tailing ## Quickstart - [Timeplus Enterprise Quickstart | Timeplus](https://docs.timeplus.com/quickstart): The following section contains step-by-step instructions on how to easily get started with a self-hosted Timeplus Enterprise. - Quickstart # Timeplus Enterprise Quickstart The following section contains step-by-step instructions on how to easily get started with a self-hosted Timeplus Enterprise. ### Step 1: install Timeplus Enterprise via a single command​ If your server or computer is running Linux or MacOS, you can run the following command to download the package and start Timeplus Enterprise without any other dependencies. For Windows users or Docker users, please follow our guide for running Timeplus Enterprise with Docker. This script will download the latest package (based on your operating system and CPU architecture) to the current folder, uncompress the package and start the Timeplus Enterprise server. You can also download the package manually from here, uncompress it and start the server via timeplus/bin/timeplus start. ### Step 2: Setup the user account​ Access the Timeplus Enterprise web console via http://localhost:8000. On your first login, please create an account with a password to start the 30-day free trial. After creating the account, login with that username and password. ### Step 3: Load your streaming data​ If your streaming data resides in or a publicly accessible Kafka or Pulsar instance, follow one of following docs to setup data access in Timeplus, then return here to complete the quickstart: - Apache Kafka - Confluent Cloud - Apache Pulsar - REST API, SDK, and others If you don't yet have a streaming data source and would like test out how Timeplus works, Timeplus provides a built-in data source to generate streaming data for some common use cases. #### Load sample data​ 1. In Timeplus Enterprise, open the Data Collection Wizard and choose Demo Stream. 2. Choose one of the three sample datasets and click Next button to create a random stream with specified name. 3. After the stream is created. Click on the first button to run an ad-hoc query to review the data from the newly created data stream. SELECT * FROM .. You should see the live data with multiple columns. #### Explore streaming data​ The streaming results table will now appear below the editor. To create charts, click on the Visualization tab. Learn more about Dashboards and Charts ### Step 4: Send data out​ To send data to Kafka, ClickHouse or other systems, you can submit a streaming SQL in SQL Console, and click the "Save As" button and choose "Sink". Timeplus supports various systems as the downstreams: - Send data to Kafka topics - Send data to Pulsar topics - Send data to ClickHouse tables - Send data to another Timeplus deployment - Send data to Webhook endpoints - Notify others via Slack - Send data to other systems via Redpanda Connect - Step 1: install Timeplus Enterprise via a single command - Step 2: Setup the user account - Step 3: Load your streaming dataLoad sample dataExplore streaming data - Load sample data - Explore streaming data - Step 4: Send data out - Load sample data - Explore streaming data ## Rbac - [Role-Based Access Control | Timeplus](https://docs.timeplus.com/rbac): Timeplus Enterprise supports access control management based on RBAC approach. - Deployment & Operations - Role-Based Access Control # Role-Based Access Control Timeplus Enterprise supports access control management based on RBAC approach. You can configure the access entities via SQL. This works in both single node and cluster deployments. ## Roles​ You can create roles and associate them with Privileges. ### Create an admin role​ The following SQL creates the admin role and grant all permissions to it. ### Create a power user role​ The following SQL creates the power_user role and grant all permissions of default database to it, also revoke the permission for this role to manage other roles or users. ### Create a read only user role​ The following SQL creates the read_only role and grant read permissions of default database to it. ## Users​ ### Create a user​ The following SQL creates a user and assign the specific role to it. ### Drop a user​ The following SQL drop the user. This won't delete the SQL resources created by the user. ## Privileges​ ### ALL​ A special privilege to include all permissions. ### SELECT​ Able to run SELECT on the specified streams or views, e.g. ### INSERT​ Able to run INSERT on the specified streams, e.g. ### CREATE​ Able to create new streams, views or materialized views, e.g. You can also specify the resource type, such as GRANT CREATE USER. ### DROP​ Able to drop new streams, views or materialized views, e.g. You can also specify the resource type, such as GRANT DROP USER. ### KILL QUERY​ Able to kill a running query, e.g. ### ACCESS MANAGEMENT​ Able to manage users, roles and row policies, e.g. ## Row Policies​ Currently row-based access control is available for historical query, not for streaming query yet. For example: ## Quota​ Quota limits resource usage. Quota contains a set of limits for some duration, as well as a list of roles and/or users which should use this quota. Management queries: - CREATE QUOTA - ALTER QUOTA - DROP QUOTA - SHOW CREATE QUOTA - SHOW QUOTA - SHOW QUOTAS For example, the following SQL set the max query numbers per hour to 1, then change it to 10, and apply this to user1. - RolesCreate an admin roleCreate a power user roleCreate a read only user role - Create an admin role - Create a power user role - Create a read only user role - UsersCreate a userDrop a user - Create a user - Drop a user - PrivilegesALLSELECTINSERTCREATEDROPKILL QUERYACCESS MANAGEMENT - ALL - SELECT - INSERT - CREATE - DROP - KILL QUERY - ACCESS MANAGEMENT - Row Policies - Quota - Create an admin role - Create a power user role - Create a read only user role - Create a user - Drop a user - ALL - SELECT - INSERT - CREATE - DROP - KILL QUERY - ACCESS MANAGEMENT ## Redpanda-connect - [Redpanda Connect | Timeplus](https://docs.timeplus.com/redpanda-connect): Since December 2022, Timeplus has utilized the Benthos framework to connect with external systems beyond Kafka and ClickHouse. For example, WebSocket sources are powered by Redpanda Connect. We offer a user-friendly wizard that allows developers to effortlessly consume data from WebSockets. - Integrations - Redpanda Connect # Redpanda Connect Since December 2022, Timeplus has utilized the Benthos framework to connect with external systems beyond Kafka and ClickHouse. For example, WebSocket sources are powered by Redpanda Connect. We offer a user-friendly wizard that allows developers to effortlessly consume data from WebSockets. With the release of Timeplus Enterprise v2.5, we've elevated this integration. All Redpanda Connect input and output components are now available within Timeplus Enterprise. Since Timeplus Enterprise v2.7, we've added support for postgres_cdc, mysql_cdc, and snowflake_streaming connectors. Please note those connectors require an enterprise license from Redpanda, which you need to set the REDPANDA_CONNECT_LICENSE environment variable. ## Key Concepts in Redpanda Connect​ Redpanda Connect is a declarative data streaming service that solves a wide range of data engineering problems with simple, chained, stateless processing steps. ### Inputs​ An input is a source of data piped through an array of optional processors, such as: In Timeplus Enterprise, you can open the "Data Collection" page and choose a type of input connector provided by Redpanda Connect, then set the configuration yaml. You can add any number of inputs or processors, but cannot set the output in the yaml file, since Timeplus will send the input data to a stream. You can run queries on the stream or send data from the stream to other systems. ### Outputs​ An output is a sink where we wish to send our consumed data after applying an optional array of processors. In Timeplus Enterprise, you can submit a streaming SQL in SQL Console, and click the "Save As" button and choose "Sink". Expand the "Send data via Redpanda Connect" section or type a keyword to search for suitable output components. You should not add input or processor in the yaml configuration file, since Timeplus will send the streaming SQL results to the pipeline as input. You can transform the input data via SQL. ### Processors​ Redpanda Connect processors are functions applied to messages passing through a pipeline. The function signature allows a processor to mutate or drop messages depending on the content of the message. There are many types on offer but the most powerful are the mapping and mutation processors, for example: Processors can be defined in inputs section or at the top level. ## Available Connectors​ All input/output/processor components from the latest Redpanda Connect are available in Timeplus Enterprise, except the following ones: - kafka and kafka_franz: please create external streams for Kafka to read or write Kafka data. - pulsar: please create external streams for Pulsar to read or write Pulsar data. - snowflake_put and splunk: those 2 components require Enterprise Edition license from Redpanda. Please contact Redpanda Sales to request a trial license, or to purchase an Enterprise Edition license. - csv, file, stdin: those are designed for local development and tests. Please use the "CSV Upload" feature in "Data Collection" page. ## Support Level​ For those data sources and sinks listed in "Timeplus Connect" section, those are supported by Timeplus Inc, such as external streams for Kafka or Pulsar, or external tables for ClickHouse, as well as WebSocket, HTTP Stream, NATS. UI wizards are also available for those integrations. For items in the "Redpanda Connect" section, no matter in "Data Collection" page or "Sink" page, you need to ensure the proper yaml configuration file. Some of them are supported by Redpanda Data Inc, and some are supported by Redpanda Community. Please check the support level tag in the web UI or refer to the Connect Catalog documentation of Redpanda. ## How to build and debug the yaml​ Please read the Get Started documentation of Redpanda Connect to learn how to use the framework. It's recommended you setup a local Redpanda Connect environment and use proper logging to debug the pipeline. You can set while debugging an input pipeline. After you confirm it works as expected, you can put the yaml configuration in Timeplus Enterprise, just need to remove the output section. ## Example: Read data from S3 bucket and send outliers to Postgres​ In this demo, we'll show you how to load a large file from Amazon S3, store it in Timeplus, and apply streaming ETL to downsample the data and write it into an external Postgres database. For visual learners, please check the following video: PLACEHOLDER ### Step 1: Create the S3 Source​ Begin by typing s3 in the search bar and select the AWS S3 input. Here is the sample configuration file: There is a web access log file (3.3GB) in the bucket. Each line is a plain text, like this: You can save each line as a string column in a Timeplus stream. ### Step 2: Create a Stream​ Create a stream, such as web_access_logs. As data is ingested, immediate querying is possible via: On the streams list page, you can monitor the number of rows imported. Next, let's parse this web access log data using the built-in grok function, which is more readable than regular expressions. ### Step 3: Create a Materialized View for transformation​ We’ll now create a materialized view in Timeplus to apply streaming transformations. ### Step 4: Query the Materialized View and send to Postgres​ Finally, let's use a tumbling window aggregation to identify the top five IP addresses with the largest HTTP request sizes and send this data to a Postgres database. Execute the query to read from the materialized view and save it as a sink. By typing 'postgres' as the keyword, you can choose sql_insert. Set the configuration file as following: This will create the table if it doesn’t already exist. Once the data sink is created, let's review the end-to-end data pipeline. This is visualized on our data lineage page, where a diagram is automatically generated, illustrating the relationships between the data source, data stream, materialized views, and sink. You can review configurations and metrics for each component. Lastly, we’ll verify the data in Postgres by a SQL client and checking the newly inserted rows. ## Summary​ The new version of Timeplus Enterprise unlocks the full potential of Redpanda Connect. Developers now have access to over 200 connectors within Timeplus Enterprise. By combining the powerful stream processing and unified analytics capabilities of Timeplus, we anticipate exciting new applications from our developer community. ## Limitations​ - Many input components from Redpanda Connect are designed for one-time batch load. After all data is loaded, please delete the source in Timeplus. Otherwise, when the Timeplus server is restarted, those input pipelines will be executed again. - For data sources and downstream supported by Timeplus, such as Kafka, Pulsar, ClickHouse, S3 (coming), please use the native integrations. Those will perform better and work well in clusters. - Key Concepts in Redpanda ConnectInputsOutputsProcessors - Inputs - Outputs - Processors - Available Connectors - Support Level - How to build and debug the yaml - Example: Read data from S3 bucket and send outliers to PostgresStep 1: Create the S3 SourceStep 2: Create a StreamStep 3: Create a Materialized View for transformationStep 4: Query the Materialized View and send to Postgres - Step 1: Create the S3 Source - Step 2: Create a Stream - Step 3: Create a Materialized View for transformation - Step 4: Query the Materialized View and send to Postgres - Summary - Limitations - Inputs - Outputs - Processors - Step 1: Create the S3 Source - Step 2: Create a Stream - Step 3: Create a Materialized View for transformation - Step 4: Query the Materialized View and send to Postgres ## Release-downloads - [Package Downloads | Timeplus](https://docs.timeplus.com/release-downloads): 2.9 - Release Notes - Package Downloads # Package Downloads ## 2.9​ ### v2.9.0-preview.2​ Released on 07-15-2025 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.9.0-preview.2 ### v2.9.0-preview.1​ Released on 06-03-2025 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.9.0-preview.1 ## 2.8​ ### v2.8.2​ Released on 07-22-2025 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.8.2 ### v2.8.1​ Released on 05-27-2025 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.8.1 ### v2.8.0​ Released on 03-24-2025 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.8.0 ## 2.7​ ### v2.7.9​ Released on 06-05-2025 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.7.9 ### v2.7.8​ Released on 05-27-2025 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.7.8 ### v2.7.7​ Released on 05-06-2025 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.7.7 ### v2.7.6​ Released on 04-29-2025 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.7.6 ### v2.7.5​ Released on 04-14-2025 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.7.5 ### v2.7.4​ Released on 04-03-2025 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.7.4 ### v2.7.3​ Released on 03-24-2025 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.7.3 ### v2.7.2​ Released on 03-18-2025 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.7.2 ### v2.7.1​ Released on 03-09-2025 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.7.1 ### v2.7.0​ Released on 02-28-2025 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.7.0 ## 2.6​ ### v2.6.8​ Released on 04-13-2025 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.6.8 ### v2.6.7​ Released on 03-26-2025 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.6.7 ### v2.6.6​ Released on 03-23-2025 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.6.6 ### v2.6.5​ Released on 03-18-2025 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.6.5 ### v2.6.4​ Released on 03-06-2025 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.6.4 ### v2.6.3​ Released on 02-19-2025 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.6.3 ### v2.6.2​ Released on 02-14-2025 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.6.2 ### v2.6.0​ Released on 01-14-2025 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.6.0 ## 2.5​ ### v2.5.13​ Released on 07-08-2025 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.5.13 ### v2.5.12​ Released on 01-09-2025 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.5.12 ### v2.5.11​ Released on 12-01-2024 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.5.11 ### v2.5.10​ Released on 11-21-2024 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.5.10 ### v2.5.9​ Released on 11-15-2024 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.5.9 ## 2.4​ ### v2.4.26​ Released on 02-18-2025 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.4.26 ### v2.4.25​ Released on 01-31-2025 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.4.25 ### v2.4.23​ Released on 08-22-2024 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.4.23 ### v2.4.19​ Released on 08-15-2024 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.4.19 ### v2.4.17​ Released on 08-14-2024 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.4.17 ### v2.4.16​ Released on 08-04-2024 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.4.16 ### v2.4.15​ Released on 07-31-2024 (Change logs). - Bare metal installation: Linux x86_64 | Linux ARM64 | macOS x86_64 | macOS ARM64 - All-in-one Docker image (not recommended for production): docker run -p 8000:8000 docker.timeplus.com/timeplus/timeplus-enterprise:2.4.15 ## 2.3​ ### v2.3.5​ Released on 07-01-2024 (Change logs). ... (content truncated) ## Release-notes - [Release Notes | Timeplus](https://docs.timeplus.com/release-notes): This page summarizes changes for Timeplus Enterprise and Timeplus Proton, including new features and important bug fixes. - Release Notes # Release Notes This page summarizes changes for Timeplus Enterprise and Timeplus Proton, including new features and important bug fixes. Release notes by version: - Timeplus Enterprise 2.9 - Timeplus Enterprise 2.8 - Timeplus Enterprise 2.7 - Timeplus Enterprise 2.6 - Timeplus Enterprise 2.5 - Timeplus Enterprise 2.4 - Timeplus Enterprise 2.3 Release notes by date: - Biweekly Summary for Timeplus Enterprise v2 and Proton (Jun 18, 2024 til now) - Timeplus Cloud v1 and Proton (Aug 8, 2023 to Jun 11, 2024) - Timeplus Cloud Public Beta 2 (Jan 19, 2023 to Aug 3, 2023) - Timeplus Cloud Public Beta 1 (Oct 3, 2022 to Jan 18, 2023) - Timeplus Cloud Private Beta 2 (Aug 9 to Oct 2, 2022) - Timeplus Cloud Private Beta 1 (Mar 8 to Aug 8, 2022) ## Remote-udf - [Remote UDF | Timeplus](https://docs.timeplus.com/remote-udf): Register a webhook as the UDF. You may use any programming language/framework to develop/deploy the webhook. A good starting point is using AWS Lambda. - SQL Reference - Functions - User Defined Functions - Remote UDF # Remote UDF Register a webhook as the UDF. You may use any programming language/framework to develop/deploy the webhook. A good starting point is using AWS Lambda. ## IP Lookup Example​ Let’s start with an example. It’s a common use case for IT admin or business analysts to turn a client IP address into a city or country, then get the total number of visitors per city or country. This might be roughly doable with pure SQL, with a lot of regular expressions or case/when branches. Even so, the city/country won’t be very accurate, since there could be some edge cases that won’t be well-covered in such a static analysis. Luckily, there are many online services (e.g. ipinfo.io) to turn IP addresses into cities/countries, even with enriched details such as addresses or internet providers. Here is an example of an UDF(ip_lookup) in Timeplus: ## Build the UDF with AWS Lambda​ In this example, the ip_lookup function is built as a “Remote UDF”, actually powered by a AWS Lambda function. I chose Node.js but you can also build it with other supported languages such as Python, Ruby, Go, Java, etc. Here is the full source code for the Lambda function: The code is straightforward. A few notes: 1. You can call the UDF for more than 1 row, such as select my_udf(col) from my_stream. To improve the efficiency, Timeplus will send batch requests to the remote UDF, e.g. my_udf([input1, input2, input3])and the return value is an array too [return1, return2, return3] 2. The input data is wrapped in a JSON document {"ip":["ip1","ip2","ip3"]} 3. We simply call the REST API of ipinfo.io with the API token from the Lambda environment variable 4. The response from ipinfo.io REST API will be put into a JSON document {“result”:[..]} and sent out as the Lambda output 5. Since the Lambda function is running outside Timeplus servers, there are no restrictions for 3rd party libraries. In this example, we are using the built-in node.js “https” library. For more complex data processing, feel free to include more sophisticated libraries, such as machine learning. Once you have deployed the Lambda function, you can generate a publicly accessible URL, then register the function in Timeplus Web Console. ## Register the UDF in Timeplus Console​ In Timeplus web console, open "UDFs" from the navigation menu on the left, and click the 'Register New Function' button. Choose "Remote" as the UDF type. Set a name for the function and specify the arguments and return data type. Set the webhook URL(e.g. Lambda URL) in the form. You can choose to enable extra authentication key/value in the HTTP header, securing the endpoint to avoid unauthorized access. ### Arguments​ The data transferring between Timeplus and Remote UDF endpoint is JSONColumns format. For example, if a remote UDF has two arguments, one feature argument is of type array(float32) and the other model argument is of type string, below is the data transferring to UDF endpoint in JSONColumns format: The following data types in Timeplus are supported as Remote UDF arguments: ### Returned value​ The remote UDF endpoint should return the text representation of a JSON document: Timeplus will take each element of the result array and convert back to Timeplus data type. The supported return type are similar to argument types. The only difference is that if you return a complex data structure as a JSON, it will be converted to a tuple in Timeplus. ## Register the UDF via SQL​ You can also create the UDF in Timeplus Proton or Timeplus Enterprise via SQL. The detailed SQL reference. If you need to protect the end point and only accept requests with a certain HTTP header, you can use the AUTH_HEADER and AUTH_KEY setting, e,g. Run DROP FUNCTION udf_name to delete or recreate the UDF. ## Other ways to build UDF​ You can also build the remote UDF with your own microservices or long-running application services to gain better control of the hardware resources, or gain even better performance or low latency. “Remote UDF” is the recommended solution for our Timeplus customers to extend the capabilities of built-in functionality, without introducing potential security risks for our cloud services. For our large customers with strong on-prem deployment needs, we also built a “Local UDF” mode which allows Timeplus to call local programs to process data. ## Best Practices for UDF​ User-defined functions open the door for new possibilities to process and analyze the data with full programming capabilities within Timeplus. There are some additional factors to consider when building and using User-Defined Functions: 1. For Timeplus Enterprise customers, it’s highly recommended to enable Authentication for the UDF. For example, when you register the function, you can set the key as ‘passcode’ and the value as a random word. Timeplus will set this in the HTTP header while making requests to the remote UDF endpoints. In your endpoint code, be sure to check whether the key/value pairs in the HTTP header matches the setting in Timeplus. If not, return an error code to deny the UDF request. 2. Calling a single UDF may only take 100ms or less, however, if you call a UDF for millions of rows, this could slow down the entire query. It’s recommended to aggregate the data first, then call the UDF with a lesser number of requests. E.g. SELECT ip_lookup(ip):city as city, sum(cnt) FROM (SELECT ip, count(*) as cnt FROM access_log GROUP BY ip) GROUP BY city instead of SELECT ip_lookup(ip):city, count(*) as cnt FROM access_log GROUP BY city 3. The Remote UDF in Timeplus is not designed for aggregation. Please turn to JavaScript based local UDF for User-Defined Aggregate Functions (UDAF). 4. To improve performance, Timeplus automatically sends batched requests to the UDF endpoints. For example, if there are 1000 requests to the UDF in a single SQL execution, the framework may send 10 requests with 100 each for the input. That’s why in the sample code, I will process the ip as an array and also return the value in the other array. Please make sure the returned value matches the inputs. 5. Properly adding logs to your UDF code can greatly help troubleshoot/tune the function code. 6. Only the Timeplus workspace administrators can register new User-Defined Functions, while all members in the workspace can use the UDFs. 7. Make sure the UDF name doesn’t conflict with the built-in functions or other UDFs in the same workspace. - IP Lookup Example - Build the UDF with AWS Lambda - Register the UDF in Timeplus ConsoleArgumentsReturned value - Arguments - Returned value - Register the UDF via SQL - Other ways to build UDF - Best Practices for UDF - Arguments - Returned value ## S3-external - [S3 External Table | Timeplus](https://docs.timeplus.com/s3-external): Amazon S3 is cloud object storage with industry-leading scalability, data availability, security, and performance. - Core Features - External Streams & Tables - External Tables - Amazon S3 # S3 External Table Amazon S3 is cloud object storage with industry-leading scalability, data availability, security, and performance. In Timeplus Enterprise v2.7, we added the first-class integration for S3-compatible object storage systems, as a new type of External Table. You can read or write data in Amazon S3 or S3-compatible cloud or local storage. ## CREATE EXTERNAL TABLE​ To create an external table for S3, you can run the following DDL SQL: For the full list of settings, see the DDL Settings section. ### Examples​ #### Read from a public S3 bucket​ The following SQL creates an external table to read data in parquet format, from a public S3 bucket. No AWS credentials are required to read from the public S3 bucket. You can get the number of rows in the external table by running SELECT count(*) FROM amazon_reviews_2015. #### Read from a private S3 bucket with credentials from environment variables​ It's recommended to avoid hardcoding the AWS credentials in the DDL. You can attach a proper IAM role if Timeplus is running inside AWS, or define environment variables to store the AWS credentials. The following SQL creates an external table to read the CloudTrail logs in a compressed JSON file, from a private S3 bucket. Since all CloudTrail events are put in the Records array, you can use array_join to flatten the array and query the data, e.g.: #### Read from a private S3 bucket with static credentials​ The following SQL creates an external table to read the S3 access logs in raw format, from a private S3 bucket, with static credentials. #### Read from a GCS bucket​ To read data from Google Cloud Storage, please follow the guide to generate HMAC keys and use them as the access key and secret key. You don't need to set the region or bucket. Put the bucket name in the endpoint setting. ### DDL Settings​ #### type​ The type of the external table. The value must be s3. #### use_environment_credentials​ Whether to use the AWS credentials from the environment variables, thus allowing access through IAM roles. Specifically, the following order of retrieval is performed: - A lookup for the environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN - Check the config or credentials files in $HOME/.aws - Temporary credentials obtained via the AWS Security Token Service - i.e. via AssumeRole API - Checks for credentials in the ECS environment variables AWS_CONTAINER_CREDENTIALS_RELATIVE_URI or AWS_CONTAINER_CREDENTIALS_FULL_URI and AWS_ECS_CONTAINER_AUTHORIZATION_TOKEN. - Obtains the credentials via Amazon EC2 instance metadata provided AWS_EC2_METADATA_DISABLED is not set to true. The default value is false. #### access_key_id​ The AWS access key ID. It's optional when use_environment_credentials is true. #### secret_access_key​ The AWS secret access key. It's optional when use_environment_credentials is true. #### config_file​ The config_file setting is available since Timeplus Enterprise 2.7. You can specify the path to a file that contains the configuration settings. The file should be in the format of key=value pairs, one pair per line. You can set the AWS access key ID and secret access key in the file. Please follow the example in Kafka External Stream. #### region​ The region where the S3 bucket is located, such as us-west-1. Optional for GCS. #### bucket​ The name of the S3 bucket. Optional for GCS. #### endpoint​ The endpoint of the S3-compatible object storage system. It's optional. If it's missing, Timeplus will use the default endpoint for the region. For example, if you have a minio running locally using the default ports. Then you should use endpoint = 'http://localhost:9000' to connect to the minio service. If you are connecting to Google Cloud Storage, set the endpoint to 'https://storage.googleapis.com/'. #### data_format​ The data_format is optional. When it's missing, Timeplus will try to infer the data format from the file extension in read_from or write_to. The supported values for data_format are: - JSONEachRow: parse each row of the message as a single JSON document. The top level JSON key/value pairs will be parsed as the columns. - CSV: less commonly used. - TSV: similar to CSV but tab as the separator - ProtobufSingle: for single Protobuf message per message - Protobuf: there could be multiple Protobuf messages in a single message. - Avro - RawBLOB: the default value. Read/write message as plain text. For data formats which write multiple rows into one single message (such as JSONEachRow or CSV), two more advanced settings are available: #### read_from​ The path(a.k.a. S3 key) to the file in the bucket to read from. It can be a single file or a path with a wildcard, such as read_from='CostUsageReportsParquet/TpDevBilling/TpDevBilling/year=2024/month={6..12}/TpDevBilling-0000{1..9}.snappy.parquet' is to read the parquet files from June to December in 2024. Bash-like wildcards are supported. The list of files is determined during SELECT (not at CREATE moment). - * — Substitutes any number of any characters except /, including empty string. - ** — Substitutes any number of any character include /, including empty string. - ? — Substitutes any single character. - {some_string,another_string,yet_another_one} — Substitutes any of strings 'some_string', 'another_string', 'yet_another_one'. - {N..M} — Substitutes any number in range from N to M including both borders. N and M can have leading zeroes e.g. 000..078. If you only set read_from, not write_to, the S3 external table becomes a read-only table, i.e. you can't run INSERT queries on it. #### write_to​ As Timeplus is a streaming engine, when you write data into a S3 external table, data will keep flowing into your S3 bucket. Thus, instead of creating one single S3 object, a S3 external table will keep creating new S3 object continuously. So the object key specified in write_to actually is a template. S3 external table will add an index ( a timestamp ) to that template as the actual object keys. For example, with write_to = 'example/data.json', the actual object keys will be something like example/data.202410291101101530247.json. 202410291101101530247 is the index added by the external table ( it's a timestamp consist of the year, month, day, hour, minute, second, and millisecond ). The index is added before the extension name (if any), so that the object key will still have the correct extension name as expected. If you only set write_to, not read_from, Timeplus will try to infer read_from from write_to, so that you can read the data that you write to the same S3 external table. If this does not work for you, you can always specify read_from manually to get the correct results. #### s3_min_upload_file_size​ The minimum size (in bytes) of the file to write to S3. If the file size is less than this value, Timeplus will buffer the data in memory and upload it when the buffer is full. The default value is 500x1024x1024 (500MB). #### s3_max_upload_idle_seconds​ The maximum idle time in seconds to wait for the buffer to be full. If the buffer is not full after this time, Timeplus will upload the data to S3. Both s3_min_upload_file_size and s3_max_upload_idle_seconds can be set in the DDL, but also can be set in the INSERT statement. The value in the INSERT statement will override the value in the DDL, e.g.: #### s3_upload_part_size_multiply_factor​ Default 2. Multiply s3_min_upload_part_size by this factor each time s3_multiply_parts_count_threshold parts were uploaded from a single write to S3. #### s3_upload_part_size_multiply_parts_count_threshold​ Default 1000. Each time this number of parts was uploaded to S3 s3_min_upload_part_size multiplied by s3_upload_part_size_multiply_factor. #### s3_max_single_part_upload_size​ Default 32MB. The maximum size of a single part upload to S3. If the part size is larger than this value, Timeplus will split the part into multiple parts. #### s3_check_objects_after_upload​ Default false. After uploading a part to S3, Timeplus will check if the object exists in the bucket. If the object does not exist, Timeplus will retry the upload. #### compression_method​ The compression algorithm to use when writing data to S3. The supported values are auto, none, gzip, deflate, br, xz, zstd, lz4, bz2, and snappy. By default, it will also be automatically inferred from the object key extension name (if any). #### s3_max_connections​ The maximum number of connections to use when uploading data to S3. The default value is 1024. #### s3_max_get_rps​ The maximum number of requests per second to use when reading data from S3. The default value is 0, which is unlimited. #### s3_max_get_burst​ The maximum burst requests that can be issued simultaneously before hitting request per second limit. By default (0) equals to s3_max_get_rps. #### s3_max_put_rps​ The maximum number of requests per second to use when writing data to S3. The default value is 0, which is unlimited. #### s3_max_put_burst​ The maximum burst requests that can be issued simultaneously before hitting request per second limit. By default (0) equals to s3_max_put_rps. #### s3_max_redirects​ The maximum number of redirects to follow when reading data from S3. The default value is 10. #### s3_max_single_read_retries​ The maximum number of retries when reading a single object from S3. The default value is 4. #### s3_max_unexpected_write_error_retries​ The maximum number of retries when an unexpected error occurs during writing to S3. The default value is 4. ### PARTITION BY​ When you write data to S3, you can partition the data by one or more columns. You can define the partition logic in the PARTITION BY clause, and use {_partition_id} in the write_to setting. ... (content truncated) ## Search - [Search the documentation | Timeplus](https://docs.timeplus.com/search): No description available # Search the documentation - Docs - Product - Pricing - Open Source - Demos - Blog - About Us - Privacy Policy - Terms of Service - EUSA ## Server_config - [Server Configuration | Timeplus](https://docs.timeplus.com/server_config): When you run Timeplus Enterprise in a self-hosted environment, the default settings are designed to accommodate common use cases with ease and optimal performance. The server can be configured via the web console, or via Kubernetes Helm chart, or via environment variables and local configuration files if you are running Timeplus Enterprise on bare metal. - Deployment & Operations - Server Configuration # Server Configuration When you run Timeplus Enterprise in a self-hosted environment, the default settings are designed to accommodate common use cases with ease and optimal performance. The server can be configured via the web console, or via Kubernetes Helm chart, or via environment variables and local configuration files if you are running Timeplus Enterprise on bare metal. Notes: - For Kubernetes deployment, after the installation, you can further customize the configuration by updating the values.yaml. Please refer to Configuration section in Kubernetes guide. Once the values.yaml is ready, apply this via: - For bare metal deployment, you can edit the configuration files in the conf folder. The configuration files are in YAML format. After editing the configuration files, you need to restart the Timeplus Enterprise to apply the changes. If it's a multi-node cluster, you need to update the configuration files on all nodes and restart them. ## Configuration Items​ ### Timeplusd​ Timeplusd is the core engine of Timeplus Enterprise. It reads configurations from the configuration file (by default conf/timeplusd.yaml) and environment variables. Some of the key configurations are listed below. You can review the full list of configurations in the configuration file. #### max_threads​ Type: uint64 Default: 0, which means the number of threads for request execution is equal to the number of CPU cores. #### http_port​ Type: uint16 Default: 3218 The port for HTTP API, which is used by JDBC/ODBC drivers and low level timeplusd REST API. #### tcp_port​ Type: uint16 Default: 8463 The port for native TCP API, which is used for communication between Timeplusd nodes. #### data.datastore.log​ There are several configurations for the logstore. The default values listed below are the default values that are hardcoded inside the binary file. The actual default value may be different depends on your deployment (bare-metal or Kubernetes) ##### preallocate​ Type: bool Default: true Whether to preallocate disk space for log files. This can improve the performance of log writing but may consume more disk space, especially for large number of small streams. ##### max_entry_size​ Type: uint64 Default: 10485760 (10MB) Maximum size of a single log entry. Increase this value if you have large log entries and encounter DB::Exception: TOO_LARGE_RECORD error. ##### segment_size​ Type: uint64 Default: 2147483648 (2GB) Size of a log segment file. ##### retention_ms​ Type: uint64 Default: 86400000 (1 day) Maximum time to keep the logstore in milliseconds. ##### index_interval_bytes​ Type: uint64 Default: 1048576 (1 MB) Bytes between two index entries in the logstore. Affect the search performance and index size. ##### index_interval_entries​ Type: uint64 Default: 1000 Number of entries between two index entries in the logstore. Affect the search performance and index size. ##### flush_interval_ms​ Type: int64 Default: 120000 (2 minutes) Maximum time to wait before flushing the logstore to disk. ##### hard_state_ckpt_log_size​ Type: uint64 Default: 1048576 (1 MB) Size of the hard state checkpoint log in Raft consensus. ##### leader_epoch_index_log_size​ Type: uint64 Default: 8388608 (8 MB) Size of the leader epoch index log in replication. ##### cache_max_cached_entries_per_shard​ Type: uint64 Default: 100 Maximum number of cached entries per shard in the logstore cache. Affects memory usage and read performance. ##### cache_max_cached_bytes_per_shard​ Type: uint64 Default: 4194304 Maximum number of cached bytes per shard in the logstore cache. Affects memory usage and read performance. #### metadata.metastore​ ##### storage_path​ Type: string Default: ./metadata Path to store the metadata. ##### log_storage_path​ Type: string Default: ./metastore_log Path to store the log of the metadata operation log. ##### snapshot_storage_path​ Type: string Default: ./metastore_snapshot Path to store the snapshots of the metadata. #### max_concurrent_queries​ Type: uint64 Default: 100 Maximum number of concurrent queries that Timeplusd can handle. Increase this value if you encounter TOO_MANY_SIMULTANEOUS_QUERIES error. #### max_concurrent_insert_queries​ Type: uint64 Default: 100 Maximum number of concurrent INSERT queries that Timeplusd can handle. Increase this value if you encounter TOO_MANY_SIMULTANEOUS_QUERIES error. #### max_concurrent_select_queries​ Type: uint64 Default: 100 Maximum number of concurrent SELECT queries that Timeplusd can handle. Increase this value if you encounter TOO_MANY_SIMULTANEOUS_QUERIES error. #### settings.global​ ##### max_block_size​ Type: uint64 Default: 65409 Maximum block size for query processing. Affects memory usage and performance. Configurable via query setting or YAML configuration. ##### max_insert_block_size​ Type: uint64 Default: 65409 Maximum block size for INSERT operations. Adjust this value if you encounter TOO_LARGE_RECORD error. Can be configured through query settings or YAML configuration. Example: ##### javascript_max_memory_bytes​ Type: uint64 Default: 209715200 (200MB) Maximum memory usage for JavaScript UDF execution. Adjust this value if you encounter UDF_MEMORY_THRESHOLD_EXCEEDED error. Affects memory usage and performance. Can be configured through query settings or YAML configuration. Example: Or apply settings directly in the query: #### force_drop_big_stream​ Type: bool Default: false Whether to force drop a stream with large data. Use this setting when you encounter STREAM_SIZE_THRESHOLD_EXCEEDED error. Can only be configured through query settings. Example: #### join_max_buffered_bytes​ Type: uint64 Default: 524288000 (500MB) Maximum size of the buffer for streaming join operations. Affects memory usage and performance. Can only be configured through query settings. Example: Or apply settings directly in the join query: #### logger​ ##### level​ Type: string Default: information Log level for Timeplusd. Possible values are none, fatal, critical, error, warning, notice, information, debug, trace. ### Timeplus Appserver​ Timeplus Appserver reads configurations from the configuration file (.yaml) and environment variables. The default values listed below are the default values that are hardcoded inside the binary file. The actual default value may be different depends on your deployment (bare-metal or Kubernetes) ### Timeplus Web​ Timeplus Web reads configurations from the environment variables. The default values listed below are the default values that are hardcoded inside the binary file. The actual default value may be different depends on your deployment (bare-metal or Kubernetes) ### Timeplus Connector​ Timeplus Connector reads configurations from the environment variables. The default values listed below are the default values that are hardcoded inside the binary file. The actual default value may be different depends on your deployment (bare-metal or Kubernetes) ## User Management​ For single node deployments, when you launch the web console of Timeplus Enterprise for the first time, you will be prompted to create a new account with password. For multi-node clusters deployed via Helm Chart, please set the system account and user accounts in the values.yaml. The system account is created automatically for internal components to communicate to each other. The username is proton, with the password defaulting to timeplusd@t+. To edit or add new users, you can use the timeplus user CLI or container, which supports bare metal and Kubernetes, both single node and multi-node. ## Secret Management​ Starting from Timeplus Enterprise 2.7, you can integrate with HashiCorp Vault for secret management. You can store the secrets of Kafka External Stream in Vault and specify them in the config_file setting. For bare metal deployments, you can also use this setting when all nodes have access to the same file in the same path. ## License Management​ Your 30-day free trial starts when you start Timeplus Enterprise and access the web console for the first time. When your free trial ends, Timeplus Enterprise will stop working. You can contact us to purchase a license, then upload it in the web console. Click Workspace Settings in the left navigation menu and choose the License tab. Copy and paste the license file or upload the file. You can also import a license by running the following command when Timeplus Enterprise is running. Starting from Timeplus Enterprise 2.7, you can also configure license_key_path and license_file_path in the server/config.yaml file. This is useful for headless deployment without web console interaction. ## Enable HTTPS​ By default, Timeplus Enterprise web console is running on 8000, on a plain HTTP port. If you need to turn on self-signed or CA-signed HTTPS, you can edit conf/timeplus_appserver.yaml as follows: To create a self-signed certificate, follow this doc and put the certificate files in the timeplus/cert folder. Stop and restart Timeplus after the configuration change. ## Run as a system service​ To run Timeplus Enterprise as a service, you need a OS that supports systemd. To install it as a systemd service, run sudo ./bin/timeplus service enable -u user -g user_group. Note: 1. Root privilege is required to enable the service 2. Use the same user/user_group for uncompressing the Timeplus installation package 3. This command will add a service into /etc/systemd/system/timeplus.service. When it is successfully installed, it will enable and start the service. Later on you can use systemctl command to manage the service. ... (content truncated) ## Showcases - [Showcases | Timeplus](https://docs.timeplus.com/showcases): At Timeplus, we drink our own champagne and apply our technologies in many different use cases. Many our customers also contribute creative ways to build real-time solutions with Timeplus. This document lists known use cases in different categories. Hopefully this can inspire you to gain more insights from real-time data with low cost and effort. - Overview - Showcases # Showcases At Timeplus, we drink our own champagne and apply our technologies in many different use cases. Many our customers also contribute creative ways to build real-time solutions with Timeplus. This document lists known use cases in different categories. Hopefully this can inspire you to gain more insights from real-time data with low cost and effort. If you have an interesting use case you'd like to share, please join the Timeplus Community Slack. ## Real-Time Analytics​ ### Optimized MySQL-to-ClickHouse CDC Pipeline​ Salla, Saudi Arabia's leading e-commerce platform provider, partnered with Timeplus to improve the efficiency of its CDC and ETL data pipeline, from MySQL to ClickHouse. By implementing Timeplus Enterprise, they significantly reduced the CPU and memory footprint of ClickHouse and expanded the scope of analytics to handle e-commerce data from thousands of MySQL tables. Salla chose Timeplus Enterprise to modernize their data infrastructure, by adding Timeplus between the Kafka and ClickHouse. Read case study ### ksqlDB Alternative​ ksqlDB, a stream processing engine by Confluent, is primarily designed for consuming data from Apache Kafka topics, performing stateless or stateful transformations, and writing results back to Kafka. For comprehensive analytical querying, data typically needs to be offloaded to specialized downstream systems. While ksqlDB offers some capabilities for querying derived state (e.g., materialized views), these ad-hoc queries are generally restricted to primary key lookups or simple range scans. 1️⃣ Limited Ad-Hoc Querying and Join Capabilities: ksqlDB's query capabilities fall short of those offered by traditional databases or data warehouses, primarily supporting key-based lookups and range scans. Its JOIN operations are also restricted, often limited to primary key relationships. 2️⃣ Performance Considerations: ksqlDB's performance can be affected by serialization/deserialization overhead between Apache Kafka and its underlying state store (RocksDB). Its tight coupling with Kafka, involving frequent data transmission and reception, can contribute to increased latency and operational costs. 3️⃣ State Management Complexity: Managing ksqlDB state stores can be challenging due to limitations in Time-To-Live (TTL) configurations and other storage settings. State persistence, often reliant on Kafka topics, can lead to increased storage and network bandwidth requirements for achieving high availability and resilience. Timeplus is designed from the ground up in C++ based on database technology but extended for Stream Processing. his allows for a much simpler and more performant system for data ingestion, processing and analytics all in one single binary. Data products created within Timeplus can be pushed out to external systems via Streaming or consumed via Ad-hoc querying. As such it can easily integrate into the wider ecosystem of systems that integrate with Apache Kafka or Database/BI Tools. Learn more. ## Observability​ ### OpenTelemetry + SQL​ OpenTelemetry (OTel) provides a standardized framework of APIs, SDKs, and tools for instrumenting, generating, collecting, and exporting telemetry data (metrics, logs, and traces). This data is crucial for analyzing software performance and behavior. OTel collectors, deployable on systems such as Linux, can export this data, often as JSON documents, to messaging systems like Apache Kafka. Timeplus can then consume this data for streaming ETL, data routing, alert generation, and real-time visualization. Learn more. ## AI​ ### Real-Time AI Hallucination Detection​ While AI agents get smarter and more independent, what happens when they make mistakes? It’s important that we have a way to watch them closely and catch errors as they happen. We built an application where two AI agents play chess against each other (based on the autogen core samples). Each agent uses a language model to think about the game and make moves. It looks simple, but it's actually a great way to test and learn how AI agents behave. Blog ### Real-Time AI and Machine Learning with SQL and Python UDF​ SQL and Python are two of the most widely used tools in the data domain, both have their strengths and weaknesses, and understanding when to use each can help maximize efficiency and performance in data analysis, machine learning, and data transformation tasks. Blog ## FinTech​ ### Real-time post-trade analytics​ Based on real-world customer scenarios for post-trade capital markets, we turn real-time market and transaction data into real-time insights. > "Timeplus fills a major gap in today’s rapidly changing markets, where businesses must go real-time or become obsolete. It makes extracting insights from streaming data even easier, saving us from writing thousands of lines of code and hundreds of hours of development. The ability to monitor and analyze massive amounts of real-time investment data leads to greater risk control and cost analysis.” -Ling Wang, Head of IT, Huatai Securities "Timeplus fills a major gap in today’s rapidly changing markets, where businesses must go real-time or become obsolete. It makes extracting insights from streaming data even easier, saving us from writing thousands of lines of code and hundreds of hours of development. The ability to monitor and analyze massive amounts of real-time investment data leads to greater risk control and cost analysis.” -Ling Wang, Head of IT, Huatai Securities Read case study | See live demo ### Real-time pricing​ As one of our first case studies, check out how leading fintech player Alpha Stream deployed Timeplus to quickly upgrade its real-time analytics capabilities. > "We are able to simply plug the sources into Timeplus and start writing queries over the streaming data to get the results. No need to compile and deploy a code. This makes prototyping to deploy applications really fast." -Hamilton Araujo, Managing Director, Alpha Stream "We are able to simply plug the sources into Timeplus and start writing queries over the streaming data to get the results. No need to compile and deploy a code. This makes prototyping to deploy applications really fast." -Hamilton Araujo, Managing Director, Alpha Stream Read case study ## DevOps​ ### Real-time observability​ At Timeplus, we collect various logs, metrics and usage data and send them to our own Timeplus workspace for continuously infrastructure monitoring and alerts. Read case study ### Metering for usage-based pricing​ By leveraging streaming SQL, Versioned Stream, HTTP ingestion, WebHook sink and many other features, we collect real-time infrastructure usage per tenants, apply lookup and aggregation, and send data to our usage-based pricing vendor, (Paigo). Read case study. ### Real-time GitHub insights​ We all love GitHub. But do you know what’s trending on Github right now? We built a real-time app with Timeplus API and GitHub API. Read case study | Demo: Timeplus Cloud, Streamlit | Github repo ## Security Compliance​ ### Container vulnerability monitoring​ Build real-time monitoring systems for container vulnerabilities with Timeplus. Eric Guo, DevOps Director of Aurora Health Science & Technology Co., shares how his team set up a system to provide actionable insights to keep their system secure at all times. > "We are delighted to have integrated Timeplus into our data infrastructure at Aurora, replacing our previous Flink clusters while utilizing just a fraction of the hardware resources, a reduction of nearly 80%. With Timeplus, we have significantly improved the analytical capabilities of AuroraPrime, reducing the turnaround time for user-facing reports and dashboards." – Eric Guo, DevOps Director, Aurora "We are delighted to have integrated Timeplus into our data infrastructure at Aurora, replacing our previous Flink clusters while utilizing just a fraction of the hardware resources, a reduction of nearly 80%. With Timeplus, we have significantly improved the analytical capabilities of AuroraPrime, reducing the turnaround time for user-facing reports and dashboards." – Eric Guo, DevOps Director, Aurora Read case study ### Monitor Superblocks user activities​ At Timeplus, we built a few internal tools with Superblocks. To track how our internal tools are being used, we configured Superblocks to send audit logs and user activities to Confluent Cloud, then load them into Timeplus. Next, we built dashboards and alerts in our own platform to understand the usage or capture any potential issues. Read case study ### Protect sensitive information in Slack​ Many organizations rely on Slack to connect people, tools, customers, and partners in a digital HQ. We built a showcase app to demonstrate how to process messages in real-time and trigger actions via the Timeplus platform, for example, removing messages that contain sensitive keywords. Read case study ## IoT​ ### Real-time fleet monitoring​ Gain real-time visibility into fleet operations using only SQL. Based on real-world customer scenarios, here's how to monitor the entire truck fleet’s status in real-time to detect speeding and fatigued drivers, and to conduct geofencing related checks. > "We are thrilled to partner with Timeplus and gain real-time visibility to our 2000+ trucks, with much lower latency, smarter alerts and shorter Time-To-Value, compared to our previous solutions." - Minfeng Xie, Chief Technology Officer, Duckbill "We are thrilled to partner with Timeplus and gain real-time visibility to our 2000+ trucks, with much lower latency, smarter alerts and shorter Time-To-Value, compared to our previous solutions." - Minfeng Xie, Chief Technology Officer, Duckbill Read case study | See live demo ### Real-time sensor data from your phone​ ... (content truncated) ## Sling - [Load MySQL/Postgres/CSV/S3 to Timeplus via Sling CLI | Timeplus](https://docs.timeplus.com/sling): While Kafka and Debezium provide robust Change Data Capture (CDC) capabilities for streaming data from MySQL or PostgreSQL to other systems, they may introduce unnecessary complexity for historical data migration scenarios. The infrastructure requirements for Kafka clusters and JVM-based deployments can present significant operational overhead. Sling offers a lightweight alternative for data transfer operations to and from Timeplus, utilizing YAML-based configuration files and scheduled execution through cron jobs. - Integrations - Third-party Tools - Load MySQL/Postgres/CSV/S3 to Timeplus via Sling CLI # Load MySQL/Postgres/CSV/S3 to Timeplus via Sling CLI While Kafka and Debezium provide robust Change Data Capture (CDC) capabilities for streaming data from MySQL or PostgreSQL to other systems, they may introduce unnecessary complexity for historical data migration scenarios. The infrastructure requirements for Kafka clusters and JVM-based deployments can present significant operational overhead. Sling offers a lightweight alternative for data transfer operations to and from Timeplus, utilizing YAML-based configuration files and scheduled execution through cron jobs. Sling is a powerful data integration CLI tool. Whether ingesting CSV or JSON files, transferring data between databases, or exporting a custom SQL query to a Parquet file — Sling is the solution that empowers you to achieve it effortlessly. Since from v1.2.14, Sling adds built-in support for Timeplus. You just need a single binary for your OS to load any data to Timeplus, with a simple command such as: ## Supported Sources​ Sling supports many databases and storage systems. Data in those systems can be imported to Timeplus. ### Databases​ ClickHouse , DuckDB , Google BigQuery , Google BigTable , MariaDB , MongoDB , MotherDuck , MySQL , Oracle , PostgreSQL , Prometheus , Redshift , Snowflake , SQL Server , SQLite , StarRocks , Trino ### Storage Systems​ Amazon S3 , Azure Storage , Cloudflare R2 , DigitalOcean Spaces , FTP , Google Storage , Local File System , Mini IO , SFTP , Wasabi ## Install Sling​ Similar to Timeplus, Sling is a single binary, running natively on the OS without Java/Docker. For best compatibility with Timeplus Enterprise, please download the latest release from https://github.com/timeplus-io/sling-cli/releases The latest official release of Sling CLI may or may not work with Timeplus. ## Configure Sling to connect to Timeplus​ Sling looks for connections and credentials in several places: - Environment variables - Sling env file (located at ~/.sling/env.yaml) - dbt profiles files (located at ~/.dbt/profiles.yml) For example, to setup a connection to your local Timeplus, as well as a connection to a local postgres, you can set ~/.sling/env.yaml: ## Sync tables from Postgres to Timeplus​ Sling allows to load all tables or specific tables from a database to Timeplus. Please create a yaml file, say pg2tp.yml: Then run sling run -r pg2tp.yml. This will load all tables from the inventory schema, create the streams in Timeplus if they don't exist and load all records. You can also specify the table names, or set a SQL with filter condition, e.g. Please check Sling docs for more details. - Supported SourcesDatabasesStorage Systems - Databases - Storage Systems - Install Sling - Configure Sling to connect to Timeplus - Sync tables from Postgres to Timeplus - Databases - Storage Systems ## Sql-alter-stream - [ALTER STREAM | Timeplus](https://docs.timeplus.com/sql-alter-stream): You can modify the retention policy for historical store via MODIFY TTL and modify the retention policy for streaming storage via MODIFY SETTING. For mutable streams, you can also run MODIFY SETTING to change the RocksDB settings. - SQL Reference - Statements - ALTER STREAM # ALTER STREAM You can modify the retention policy for historical store via MODIFY TTL and modify the retention policy for streaming storage via MODIFY SETTING. For mutable streams, you can also run MODIFY SETTING to change the RocksDB settings. You can also use ALTER VIEW to modify the settings of materialized views (only available in Timeplus Enterprise). ## MODIFY TTL​ You can add or modify the retention policy. e.g. ## MODIFY SETTING​ You can add or modify the retention policy for streams or mutable streams, e.g. Starting from Timeplus Enterprise 2.7, you can also modify the RocksDB settings for mutable streams. e.g. You can also change the codec for mutable streams. e.g. Starting from Timeplus Enterprise 2.8.2, you can also modify the TTL for mutable stream. ## MODIFY QUERY SETTING​ This feature is available in Timeplus Enterprise v2.2.8 or above. Not available in Timeplus Proton. Please use ALTER VIEW for this use cases. Altering views or materialized views will be deprecated and removed from the ALTER STREAM SQL command. By default, the checkpoint will be updated every 15 minutes for materialized views. You can change the checkpoint interval without recreating the materialized views. ## RESET QUERY SETTING​ This feature is available in Timeplus Enterprise v2.2.8 or above. Not available in Timeplus Proton. Please use ALTER VIEW for this use cases. Altering views or materialized views will be deprecated and removed from the ALTER STREAM SQL command. By default, the checkpoint will be updated every 15 minutes for materialized views. After you change the interval you can reset it. ## ADD COLUMN​ You can add a column to an existing stream. The value of the new column in the existing rows will be set to the default value of the data type, such as 0 for integer. Syntax: Since Timeplus Enterprise 2.8.2, you can also add multiple columns at once: DELETE COLUMN is not supported yet. Contact us if you have strong use cases. ## RENAME COLUMN​ Since Timeplus Enterprise 2.9, you can rename columns in append streams. ## ADD INDEX​ Since Timeplus Enterprise v2.6.0, you can add an index to a mutable stream. ## DROP INDEX​ Since Timeplus Enterprise v2.9.0, you can drop an index from a mutable stream. ## MATERIALIZE INDEX​ Since Timeplus Enterprise 2.8.2, you can rebuild the secondary index name for the specified partition_name. For example: ## CLEAR INDEX​ Since Timeplus Enterprise 2.8.2, you can delete the secondary index name from disk. For example: ## DROP PARTITION​ You can delete some data in the stream without dropping the entire stream via ALTER STREAM .. DROP PARTITION ... By default the streams in Timeplus are partitioned by _tp_time: PARTITION BY to_YYYYMMDD(_tp_time). You can query the system.parts table to check the partitions for the given streams: This may show results like this: You can delete data in certain partitions via SQL: Such as ## See also​ - CREATE STREAM - Create streams - CREATE MATERIALIZED VIEW - Create materialized views - ALTER VIEW - Alter views or materialized views - DROP STREAM - Drop streams - DROP VIEW - Drop materialized views - MODIFY TTL - MODIFY SETTING - MODIFY QUERY SETTING - RESET QUERY SETTING - ADD COLUMN - RENAME COLUMN - ADD INDEX - DROP INDEX - MATERIALIZE INDEX - CLEAR INDEX - DROP PARTITION - See also ## Sql-alter-view - [ALTER VIEW | Timeplus](https://docs.timeplus.com/sql-alter-view): You can use this SQL to change a view or a materialized view. Today only the settings can be changed. To change the SQL query behinds the view, you have to drop and re-create it. - SQL Reference - Statements - ALTER VIEW # ALTER VIEW You can use this SQL to change a view or a materialized view. Today only the settings can be changed. To change the SQL query behinds the view, you have to drop and re-create it. ## MODIFY QUERY SETTING​ This feature will be available in Timeplus Enterprise v2.5. Not available in Timeplus Proton. By default, the checkpoint will be updated every 15 minutes for materialized views. You can change the checkpoint interval without recreating the materialized views. ## RESET QUERY SETTING​ This feature will be available in Timeplus Enterprise v2.5. Not available in Timeplus Proton. By default, the checkpoint will be updated every 15 minutes for materialized views. After you change the interval you can reset it. ## See also​ - ALTER STREAM - Alter streams - CREATE STREAM - Create streams - CREATE MATERIALIZED VIEW - Create materialized views - DROP STREAM - Drop streams - DROP VIEW - Drop materialized views - MODIFY QUERY SETTING - RESET QUERY SETTING - See also ## Sql-create-alert - [CREATE ALERT | Timeplus](https://docs.timeplus.com/sql-create-alert): Starting from Timeplus Enterprise 2.9, you can create alerts to monitor your streaming data and automatically trigger actions when specific conditions are met. - SQL Reference - Statements - CREATE ALERT # CREATE ALERT Starting from Timeplus Enterprise 2.9, you can create alerts to monitor your streaming data and automatically trigger actions when specific conditions are met. ## Syntax​ For example: ## Limitations​ - The alerts only run on the metadata leader node. - The return value of the Python UDF is ignored. - The select query cannot include any aggregation or JOIN. - Check system.stream_state_log for the alert states or logs. - The checkpoints of the alerts are available in system.alert_ckpt_log stream with the _tp_sn column. ## Python UDF​ You can import Python libraries and build the custom alert action via Python UDF. The return value doesn't matter. Here is an example to send events to a specific Slack channel via Slack webhook: Please note, similar to regular Python UDF, the input parameter of the Python UDF is an array, instead of a single event. The return value can be anything but you can return the input value, so that you can test the Python UDF via SELECT udf_name(input). ## See also​ - SHOW ALERTS - Show alerts - DROP ALERT - Drop alerts - Syntax - Limitations - Python UDF - See also ## Sql-create-database - [CREATE DATABASE | Timeplus](https://docs.timeplus.com/sql-create-database): Starting from Timeplus Enterprise 2.6, you can create a database to organize your streams, materialized views and other resources. The database is a logical container to help you to manage and query the data more efficiently. - SQL Reference - Statements - CREATE DATABASE # CREATE DATABASE Starting from Timeplus Enterprise 2.6, you can create a database to organize your streams, materialized views and other resources. The database is a logical container to help you to manage and query the data more efficiently. When you create a database in a cluster, the database is automatically replicated to all the nodes in the cluster. You can create a database with the following SQL: Once the database is created, you can create streams, materialized views, and other resources in the database by running USE my_database before creating the resources. For example: Please note User-Defined Functions are global and not bound to a specific database. ## External Database​ Starting from Timeplus Enterprise 2.8, you can create an external database to connect to external MySQL and Iceberg databases. ### MySQL Database​ Syntax: Then you can list all tables from the remote MySQL database: Or query them: ### Iceberg Database​ Syntax: Learn more ## See also​ - USE - Use a database - SHOW DATABASES - Show databases - DROP DATABASE - Drop databases - External DatabaseMySQL DatabaseIceberg Database - MySQL Database - Iceberg Database - See also - MySQL Database - Iceberg Database ## Sql-create-dictionary - [CREATE DICTIONARY | Timeplus](https://docs.timeplus.com/sql-create-dictionary): Starting from Timeplus Enterprise 2.7, you can create dictionaries to store key-value pairs in memory, with data from various sources, such as files, MySQL/ClickHouse databases, or streams in Timeplus. - SQL Reference - Statements - CREATE DICTIONARY # CREATE DICTIONARY Starting from Timeplus Enterprise 2.7, you can create dictionaries to store key-value pairs in memory, with data from various sources, such as files, MySQL/ClickHouse databases, or streams in Timeplus. When you create a dictionary in a cluster, the dictionary is automatically replicated to all the nodes in the cluster. ## Syntax​ You can create a dictionary with the following SQL: ### PRIMARY KEY​ The PRIMARY KEY clause specifies the key columns of the dictionary. You can specify multiple columns as the primary key. ### SOURCE​ The source for a dictionary can be a: - stream in the current Timeplus service - stream in a remote Timeplus service - file available by HTTP(S) - another database, such as MySQL or PostgreSQL #### Local Timeplus Stream​ You can create a dictionary from a stream in the current Timeplus service. Mutable streams are recommended for dictionary sources, as they can be updated in real-time. Syntax: Note: - You need specify either STREAM or QUERY. A sample QUERY is: SELECT key_column, second_column, third_column FROM table(stream1) - The USER and PASSWORD are optional. If you don't specify them, Timeplus will use the default user and empty password, which only works if you have set up the default user in the Timeplus configuration file. You can also use the current_user() function to get the current user. Please check the example in the Create an ip_trie dictionary from a Timeplus stream section. #### Remote Timeplus Stream​ Similar to the local stream, you can create a dictionary from a stream in a remote Timeplus service. Syntax: #### ClickHouse Table​ You can create a dictionary from a table in a remote ClickHouse service. Syntax: Either one of the TABLE or QUERY fields must be declared. #### MySQL Table​ You can create a dictionary from a table in a remote MySQL database. Syntax: Note: - Either one of the TABLE or QUERY fields must be declared. - You can optionally specify BG_RECONNECT true to enable background reconnection to MySQL. Please check the example in the Create a dictionary from a MySQL table section. You can optionally setup a mutable stream as the cache for the MySQL table, please check the Create a dictionary from a MySQL table with a mutable stream as the cache section. #### PostgreSQL Table​ You can create a dictionary from a table in a remote PostgreSQL database. Syntax: Note: - Either one of the TABLE or QUERY fields must be declared. - You can optionally specify BG_RECONNECT true to enable background reconnection to MySQL. #### Remote File​ You can create a dictionary from a file available by HTTP(S). Syntax: Please check the example in the Create a dictionary from a CSV file section. #### Local File​ You can create a dictionary from a local file. Syntax: Please note the file path needs to be located in the user_files directory to prevent unauthorized access to the file system. Make sure each node in the cluster has the same file in the same location. #### Local Executable​ You can create a dictionary from a local executable. Syntax: The executable should output data in the format of the specified format, in this case, TabSeparated. The executable should be located in the user_files directory. #### Executable Pool​ Executable pool allows loading data from pool of processes. This source does not work with dictionary layouts that need to load all data from source. Executable pool works if the dictionary is stored using cache, complex_key_cache, ssd_cache, complex_key_ssd_cache, direct, or complex_key_direct layouts. ### LAYOUT​ The LAYOUT clause specifies how the dictionary data is stored in memory. The available layout options are listed below. Please note the layout options and their settings are case-insensitive. #### FLAT​ The FLAT layout stores the dictionary data in a flat array. This layout provides the best performance, with the drawback of higher memory usage. The default INITIAL_ARRAY_SIZE is 1,024 and default MAX_ARRAY_SIZE is 500,000. Please note that the FLAT layout expects the primary key to be a number which can be converted to uint64. If the primary key is not a number, such as a string, you should switch to the HASHED or COMPLEX_KEY_HASHED layout. #### HASHED​ The HASHED layout stores the dictionary data in a hash table. This layout provides better memory usage than the FLAT layout, with slightly lower performance. The dictionary can contain any number of elements with any identifiers. Please note that the HASHED layout expects the primary key to be a number which can be converted to uint64. If the primary key is not a number, such as a string, Timeplus will automatically switch to the COMPLEX_KEY_HASHED layout. Such optimization is available for other CACHE/HASHED layouts, but not FLAT layout. #### SPARSE_HASHED​ The SPARSE_HASHED layout is similar to the HASHED layout, but it is optimized for sparse data. It uses less memory than the HASHED layout. #### COMPLEX_KEY_HASHED​ The COMPLEX_KEY_HASHED layout is similar to the HASHED layout, but it supports composite keys. #### COMPLEX_KEY_SPARSE_HASHED​ The COMPLEX_KEY_SPARSE_HASHED layout is similar to the SPARSE_HASHED layout, but it supports composite keys. #### HASHED_ARRAY​ The HASHED_ARRAY layout stores the dictionary data in memory. Each attribute is stored in an array. The key attribute is stored in the form of a hashed table where value is an index in the attributes array. The dictionary can contain any number of elements with any identifiers. #### COMPLEX_KEY_HASHED_ARRAY​ The COMPLEX_KEY_HASHED_ARRAY layout is similar to the HASHED_ARRAY layout, but it supports composite keys. #### RANGE_HASHED​ The dictionary is stored in memory in the form of a hash table with an ordered array of ranges and their corresponding values. #### COMPLEX_KEY_RANGE_HASHED​ The COMPLEX_KEY_RANGE_HASHED layout is similar to the RANGE_HASHED layout, but it supports composite keys. #### CACHE​ The CACHE layout is stored in a cache that has a fixed number of cells. These cells contain frequently used elements. #### COMPLEX_KEY_CACHE​ The COMPLEX_KEY_CACHE layout is similar to the CACHE layout, but it supports composite keys. #### SSD_CACHE​ The SSD_CACHE layout is similar to the CACHE layout, but it is optimized for SSD storage. #### COMPLEX_KEY_SSD_CACHE​ The COMPLEX_KEY_SSD_CACHE layout is similar to the SSD_CACHE layout, but it is optimized for SSD storage. #### MUTABLE_CACHE​ The MUTABLE_CACHE layout is used to cache frequently accessed keys in a mutable stream. The dictionary will first look up the keys in the mutable stream, and if not found, it will fetch the data from the source. Syntax: The default value for UPDATE_FROM_SOURCE is false. If set to true, when there is a lookup miss, the dictionary will update the mutable stream with the data from the source. If set to false, the dictionary will only fetch the data from the source without updating the mutable stream. #### COMPLEX_KEY_MUTABLE_CACHE​ The COMPLEX_KEY_MUTABLE_CACHE layout is similar to the MUTABLE_CACHE layout, but it supports composite keys. #### HYBRID_HASH_CACHE​ The HYBRID_HASH_CACHE layout leverages Timeplus hybrid hash table to store the dictionary in both memory and disk. You need to specify both the TTL and PATH parameters. #### DIRECT​ The dictionary with DIRECT layout is not stored in memory and directly goes to the source during the processing of a request. #### COMPLEX_KEY_DIRECT​ The COMPLEX_KEY_DIRECT layout is similar to the DIRECT layout, but it supports composite keys. #### IP_TRIE​ The IP_TRIE layout is for mapping network prefixes (IP addresses) to metadata such as ASN. Please check the example in the Create an ip_trie dictionary from a Timeplus stream section. ### LIFETIME​ Timeplus can update the dictionary data automatically. You can specify the update interval with the LIFETIME clause. The MIN and MAX values are in seconds. The MIN value is the minimum interval between updates, and the MAX value is the maximum interval between updates. For example: specifies the dictionary to update after some random time between 1 and 10 seconds. This can be simplified to: You can disable automatic updates by setting both MIN and MAX to 0, i.e. Or simply: ## Examples​ ### Create a dictionary from a CSV file​ For example, the following SQL creates a dictionary named taxi_zone_dictionary with data from a CSV file in a public S3 bucket: You can list the content of the dictionary with the following SQL: Or check the number of rows in the dictionary with the following SQL: Although you can query the dictionary with the regular SELECT .. FROM dict WHERE .. syntax, it is recommended to use the dict_get* functions for better performance. A common use case for dictionaries is to join them with other data streams. For example, you can join the taxi_zone_dictionary with a stream taxi_trips to get the borough of the pickup location: Under the hood, Timeplus will use the dict_get function to look up the value from the dictionary. Depending on the LIFETIME settings, the dictionary will be updated automatically during the JOIN query execution. ### Create an ip_trie dictionary from a Timeplus stream​ For example, you can create a mutable stream in Timeplus, and add some sample records to it: Then you can create a dictionary, referencing the stream: This will create an IP_TRIE dictionary, which is optimized for mapping network prefixes (IP addresses) to metadata such as ASN. The dictionary will be updated every 0 to 10 seconds. Then you can query the dictionary with the dict_get function: To demonstrate the update capability of the dictionary, you can insert a new record into the mutable stream: Then run the query again: It will return BB instead of ZZ. ### Create a dictionary from a MySQL table​ ... (content truncated) ## Sql-create-disk - [CREATE DISK | Timeplus](https://docs.timeplus.com/sql-create-disk): By default, Timeplus only created a default disk for local storage. - SQL Reference - Statements - CREATE DISK # CREATE DISK By default, Timeplus only created a default disk for local storage. Starting from Timeplus Enterprise 2.8, you can create S3 disks for tiered storage or autoscaling materialized views. ## Syntax​ You can create a S3 disk with the following SQL: Please refer to S3 External Table for how to connect to the S3 storage. It's not recommended to hardcode the access key and secret access key in the DDL. Instead, users should use environment variables or IAM role to secure these credentials. You can use the following SQL to list the disks: ## See also​ - SHOW DISKS - Show disks - Syntax - See also ## Sql-create-external-stream - [CREATE EXTERNAL STREAM | Timeplus](https://docs.timeplus.com/sql-create-external-stream): External stream for Kafka is official supported. The external stream for local log files is at technical preview. In Timeplus Enterprise, it also supports another type of External Stream to read/write data for a remote Timeplus Enterprise. - SQL Reference - Statements - CREATE EXTERNAL STREAM # CREATE EXTERNAL STREAM External stream for Kafka is official supported. The external stream for local log files is at technical preview. In Timeplus Enterprise, it also supports another type of External Stream to read/write data for a remote Timeplus Enterprise. ## Kafka External Stream​ Please check the Kafka External Stream for more details about the settings, and this doc for examples to connect to various Kafka API compatible message platforms. ## Pulsar External Stream​ Please check the Pulsar External Stream for more details. ## Timeplus External Stream​ Settings: - hosts: the IP or host for the remote timeplusd. When you set a set of hosts with ports, e.g. 'host1:port1,host2:port2', this will treat each host as a shard. hosts is required and there is no default value. - db: the database name in the remote Timeplusd. The default value is 'default'. - user: the user name for the remote Timeplusd. The default value is 'default'. - password: the password for the remote Timeplusd. The default value is an empty string. - secure: a bool for whether to use secure connection to the remote Timeplusd. The default value is false. Use port 9440 when secure is set to true, otherwise use port 8463. - stream: the stream name in the remote Timeplusd. It's required and there is no default value. ### Examples​ #### Migrate data from Timeplus Proton to Timeplus Enterprise​ If you have deployed Timeplus Proton and want to load those data to a Timeplus Enterprise deployment, you cannot upgrade in place. You can use the Timeplus External Stream to migrate data. The Timeplus Proton need to be 1.5.15 or above. For example, there is a stream streamA in Timeplus Proton, running on host1. In your Timeplus Enterprise, you can create the stream with the same name and same schema. Then use a materialized view to load all data from Timeplus Proton to Timeplus Enterprise. When all data in Proton has been imported into Timeplus Enterprise, you can drop the materialized view. #### Upload data from edge server to the cloud​ If you deploy Timeplus Proton or Timeplus Enterprise at edge servers, it can collect and process live data with high performance and low footprint. The important data can be uploaded to the other Timeplus Enterprise in the cloud when the internet is available. For example, on the edge server, you collect the real-time web access log and only want to upload error logs to the server. When the network is not available, you can pause the materialized view by: When the network is restored, you can resume it: ### Limitations​ This is a relatively new feature. There are some known limitations which we plan to improve later on. - table function is not supported. In order to query all or part of the historical data, you can start a streaming query with WHERE _tp_time>earliest_ts() or WHERE _tp_time>now()-2d. - window functions like tumble/hop are not working yet. - can't read virtual columns on remote streams. - Kafka External Stream - Pulsar External Stream - Timeplus External StreamExamplesMigrate data from Timeplus Proton to Timeplus EnterpriseUpload data from edge server to the cloudLimitations - ExamplesMigrate data from Timeplus Proton to Timeplus EnterpriseUpload data from edge server to the cloud - Migrate data from Timeplus Proton to Timeplus Enterprise - Upload data from edge server to the cloud - Limitations - ExamplesMigrate data from Timeplus Proton to Timeplus EnterpriseUpload data from edge server to the cloud - Migrate data from Timeplus Proton to Timeplus Enterprise - Upload data from edge server to the cloud - Limitations - Migrate data from Timeplus Proton to Timeplus Enterprise - Upload data from edge server to the cloud ## Sql-create-external-table - [CREATE EXTERNAL TABLE | Timeplus](https://docs.timeplus.com/sql-create-external-table): Timeplus supports 3 types of external tables: ClickHouse, MySQL, and S3. Reading data from external tables are bounded, which means the query will end when all the data is read. Writing data to external tables are unbounded, which means the query will keep running until you cancel it. - SQL Reference - Statements - CREATE EXTERNAL TABLE # CREATE EXTERNAL TABLE Timeplus supports 3 types of external tables: ClickHouse, MySQL, and S3. Reading data from external tables are bounded, which means the query will end when all the data is read. Writing data to external tables are unbounded, which means the query will keep running until you cancel it. ## ClickHouse/MySQL External Table​ The required settings are type and address. For other settings, the default values are - 'default' for user - '' (empty string) for password - 'default' for database - 'false' for secure - If you omit the table name, it will use the name of the external table You don't need to specify the columns, since the table schema will be fetched from the ClickHouse or MySQL server. ## S3 External Table​ ## DESCRIBE​ Once the external table is created successfully, you can run the following SQL to list the columns: The data types in the output will be Timeplus data types, such as uint8. You can define the external table and use it to read data from the ClickHouse or MySQL table, or write to it. - ClickHouse/MySQL External Table - S3 External Table - DESCRIBE ## Sql-create-format-schema - [CREATE FORMAT | Timeplus](https://docs.timeplus.com/sql-create-format-schema): Timeplus supports reading or writing messages in Protobuf or Avro format. This document covers how to process data without a Schema Registry. Check this page if your Kafka topics are associated with a Schema Registry. - SQL Reference - Statements - CREATE FORMAT # CREATE FORMAT Timeplus supports reading or writing messages in Protobuf or Avro format. This document covers how to process data without a Schema Registry. Check this page if your Kafka topics are associated with a Schema Registry. Without a Schema Registry, you need to define the Protobuf or Avro schema using SQL. ## Protobuf​ Then refer to this schema while creating an external stream for Kafka: Please note: 1. If you want to ensure there is only a single Protobuf message per Kafka message, please set data_format to ProtobufSingle. If you set it to Protobuf, then there could be multiple Protobuf messages in a single Kafka message. 2. The format_schema setting contains two parts: the registered schema name (in this example: schema_name), and the message type (in this example: SearchRequest). Combining them together with a semicolon. 3. You can use this external stream to read or write Protobuf messages in the target Kafka/Confluent topics. 4. For more advanced use cases, please check the examples for complex schema. ## Avro​ Available since Proton 1.5.10. Then refer to this schema while creating an external stream for Kafka: ## See also​ SHOW FORMAT SCHEMA - Show format schema DROP FORMAT SCHEMA - Drop format schema - Protobuf - Avro - See also ## Sql-create-function - [CREATE FUNCTION | Timeplus](https://docs.timeplus.com/sql-create-function): Timeplus supports four ways to develop/register UDF. Please check UDF page for the overview. - SQL Reference - Statements - CREATE FUNCTION # CREATE FUNCTION Timeplus supports four ways to develop/register UDF. Please check UDF page for the overview. ## SQL UDF​ You can create or replace a SQL UDF, by specifying the function name, parameters and the expression. Syntax: For example: Learn More ## Remote UDF​ Register a webhook as the UDF. You may use any programming language/framework to develop/deploy the webhook. A good starting point is using AWS Lambda. Syntax: If you need to protect the end point and only accept requests with a certain HTTP header, you can use the AUTH_HEADER and AUTH_KEY setting, e,g. Learn More ## JavaScript UDF​ ### UDF​ You can create or replace a JavaScript UDF, by specifying the function name, input and output data types. Please check the mapping of data types for input and output. The following example defines a new function test_add_five_5: Note: - Line 1: the function is to be created with name test_add_five_5, taking a single float32 parameter. - Line 2: the function is to return a float32 - Line 4: the same function name is defined in the code block. To improve performance, multiple UDF calls will be batched. The values is an array of float - Line 5 and 6: iterating the input array and add 5 to each value - Line 8: return an array of new values - Line 10: close the code block. You can also add EXECUTION_TIMEOUT to the end of the CREATE FUNCTION to customize the timeout for calling remote endpoints. By default the timeout is 10000 milliseconds (i.e. 10 seconds). You can add debug information via console.log(..) in the JavaScript UDF. The logs will be available in the server log files. Check more examples for scalar function with 2 or more arguments or 0 argument. ### UDAF​ Creating a user-defined-aggregation function (UDAF) requires a bit more effort. Please check this documentation for the 3 required and 3 optional functions. Learn More ## Python UDF​ starting from v2.7, Timeplus Enterprise also supports Python-based UDF. You can develop User-defined scalar functions (UDFs) or User-defined aggregate functions (UDAFs) with the embedded Python 3.10 runtime in Timeplus core engine. No need to deploy extra server/service for the UDF. Learn more why Python UDF, and how to map the data types in Timeplus and Python, as well as how to manage dependencies. ### UDF​ Syntax: ### UDAF​ UDAF or User Defined Aggregation Function is stateful. It takes one or more columns from a set of rows and return the aggregated result. Syntax: Learn More - SQL UDF - Remote UDF - JavaScript UDFUDFUDAF - UDF - UDAF - Python UDFUDFUDAF - UDF - UDAF - UDF - UDAF - UDF - UDAF ## Sql-create-materialized-view - [CREATE MATERIALIZED VIEW | Timeplus](https://docs.timeplus.com/sql-create-materialized-view): The difference between a materialized view and a regular view is that the materialized view is running in background after creation and the resulting stream is physically written to internal storage (hence it's called materialized). - SQL Reference - Statements - CREATE MATERIALIZED VIEW # CREATE MATERIALIZED VIEW The difference between a materialized view and a regular view is that the materialized view is running in background after creation and the resulting stream is physically written to internal storage (hence it's called materialized). To create a materialized view: Once the materialized view is created, Timeplus will run the query in the background continuously and incrementally emit the calculated results according to the semantics of its underlying streaming select. Different ways to use the materialized views: 1. Streaming mode: SELECT * FROM materialized_view Get the result for future data. This works in the same way as views. 2. Historical mode: SELECT * FROM table(materialized_view) Get all past results for the materialized view. 3. Historical + streaming mode: SELECT * FROM materialized_view WHERE _tp_time>='1970-01-01' Get all past results and as well as the future data. 4. Pre-aggregation mode: SELECT * FROM table(materialized_view) where _tp_time in (SELECT max(_tp_time) as m from table(materialized_view)) This immediately returns the most recent query result. If _tp_time is not available in the materialized view, or the latest aggregation can produce events with different _tp_time, you can add the emit_version() to the materialized view to assign a unique ID for each emit and pick up the events with largest emit_version(). For example: create materialized view mv asselect emit_version() as version, window_start as time, count() as n, max(speed_kmh) as h from tumble(car_live_data,10s)group by window_start, window_end;select * from table(mv) where version in (select max(version) from table(mv)); We are considering providing new syntax to simplify this. Streaming mode: SELECT * FROM materialized_view Get the result for future data. This works in the same way as views. Historical mode: SELECT * FROM table(materialized_view) Get all past results for the materialized view. Historical + streaming mode: SELECT * FROM materialized_view WHERE _tp_time>='1970-01-01' Get all past results and as well as the future data. Pre-aggregation mode: SELECT * FROM table(materialized_view) where _tp_time in (SELECT max(_tp_time) as m from table(materialized_view)) This immediately returns the most recent query result. If _tp_time is not available in the materialized view, or the latest aggregation can produce events with different _tp_time, you can add the emit_version() to the materialized view to assign a unique ID for each emit and pick up the events with largest emit_version(). For example: We are considering providing new syntax to simplify this. ## Target Stream​ By default, when you create a materialized view, an internal stream will be created automatically as the data storage. Querying on the materialized view will result in querying the underlying internal stream. Use cases for specifying a target stream: 1. In some cases, you may want to build multiple materialized views to write data to the same stream. In this case, each materialized view serves as a real-time data pipeline. 2. Or you may need to use Changelog Stream or Versioned Stream to build lookups. 3. Or you may want to set the retention policy for the materialized view. 4. You can also use materialized views to write data to Apache Kafka with an external stream. To create a materialized view with the target stream: ## Settings​ The following settings are supported when you create a materialized view, with syntax: ### pause_on_start​ By default, once the materialized view is created, the streaming query will start automatically. If you don't want to start the query immediately, you can set pause_on_start=true. The default value is false. ### memory_weight​ Starting from Timeplus Enterprise v2.3, when you create a materialized view with DDL SQL, you can add an optional memory_weight setting for those memory-consuming materialized views, e.g. When memory_weight is not set, by default the value is 0. When Timeplus Enterprise server starts, the system will list all materialized views, ordered by the memory weight and view names, and schedule them in the proper node. For example, in a 3-node cluster, you define 10 materialized views with names: mv1, mv2, .., mv9, mv10. If you create the first 6 materialized views with SETTINGS memory_weight = 10, then node1 will run mv1 and mv4; node2 will run mv2 and mv5; node3 will run mv3 and mv6; Other materialized views(mv7 to mv10) will be randomly scheduled on any nodes. It's recommended that each node in the Timeplus Enterprise cluster shares the same hardware specifications. For those resource-consuming materialized views, it's recommended to set the same memory_weight, such as 10, to get the expected behaviors to be dispatched to the proper nodes for load-balancing. ### mv_preferred_exec_node​ Starting from Timeplus Enterprise v2.7.6, when you create a materialized view with DDL SQL, you can add an optional mv_preferred_exec_node setting to explicitly assign a node to run the materialized view. In most cases, you don't need to specify this setting. Timeplus will automatically select an available node to run the materialized view. It's also recommended to set memory_weight to have the system to automatically choose the appropriate node for load balancing. If you need to fine-tune the load balancing or resource utilization, you can specify this setting. As a result, load balancing or failover won't be available when this is set. You cannot change the value after the materialized view is created, even the node is having issue. In this case, please drop and re-create the materialized view with new node ID. - Target Stream - Settingspause_on_startmemory_weightmv_preferred_exec_node - pause_on_start - memory_weight - mv_preferred_exec_node - pause_on_start - memory_weight - mv_preferred_exec_node ## Sql-create-mutable-stream - [CREATE MUTABLE STREAM | Timeplus](https://docs.timeplus.com/sql-create-mutable-stream): Regular streams in Timeplus are immutable, and stored in columnar format. Mutable streams are stored in row format (implemented via RocksDB), and can be updated or deleted. It is only available in Timeplus Enterprise. Please check this page for details. - SQL Reference - Statements - CREATE MUTABLE STREAM # CREATE MUTABLE STREAM Regular streams in Timeplus are immutable, and stored in columnar format. Mutable streams are stored in row format (implemented via RocksDB), and can be updated or deleted. It is only available in Timeplus Enterprise. Please check this page for details. Since Timeplus Enterprise 2.7, if you create a mutable stream with low_cardinality columns, the system will ignore the low_cardinality modifier to improve performance. Only the PRIMARY KEY is required. INDEX, FAMILY, or the SETTINGS are optional. PARTITION BY, ORDER BY or SAMPLE BY clauses are not allowed while creating the mutable stream. ## PRIMARY KEY​ Primary key is required for a mutable stream. It can be a single column or a composite key. The primary key is used for efficient data lookups, range scans and updates. Supported column types for primary key: - integers (int8/16/32/64, uint8/16/32/64) - floating point (float32, float64) - date and datetime types (date, date32, datetime, datetime64) - string and fixed_string - enum8, enum16 - decimal32, decimal64 - bool - ipv4, ipv6 - interval ## INDEX​ You can add secondary indexes to a mutable stream. The index can be unique or non-unique. The index can be used for efficient lookups and range scans, for those columns not as the primary keys. Syntax: - index_name: Name of the secondary index - column_list: Comma-separated list of columns to index - UNIQUE: Optional, specifies if the index enforces uniqueness - STORING: Optional, additional columns to store with the index For example: ## FAMILY​ Column families in mutable streams provide a way to organize columns into groups for optimized storage and retrieval. Each column family is stored separately, allowing for better performance and more efficient data management. Syntax - family_name: Name of the column family - column_list: Comma-separated list of columns to include in this family For example: Recommended practices: - Put frequently accessed columns together in the same family. - Use automatic column family grouping for optimal performance, by setting auto_cf=true. In this mode, the system automatically organizes columns into appropriate groups based on their data types and access patterns. This provides optimal performance without requiring manual column family configuration. - Primary key columns are automatically managed by the system and do not need to be explicitly assigned to a family. - Use column families to improve query performance when only certain columns need to be queried. ## SETTINGS​ ### Storage Settings​ #### kvstore_codec​ Type: string Default: 'snappy' Compression mode: "none", "lz4", "zstd", "snappy" #### enable_hash_index​ Type: bool Default: false Use RocksDB HashIndex for better point lookup performance #### enable_statistics​ Type: bool Default: false Enable RocksDB statistics collection #### auto_cf​ Type: bool Default: false Automatically group column families ### RocksDB Performance Settings​ #### kvstore_options​ Type: string Default: '' (empty) RocksDB options (key=value pairs separated by semicolons). Check more details at https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide #### flush_rows​ Type: uint64 Default: 100,000 Number of rows to trigger a flush #### flush_ms​ Type: int64 Default: 30,000 Time in milliseconds to trigger a flush ### Logstore Settings​ #### logstore_retention_bytes​ Type: int64 Default: -1 Maximum size of the logstore in bytes. -1 means no limit. #### logstore_retention_ms​ Type: int64 Default: -1 Maximum time to keep the logstore in milliseconds. -1 means no limit. #### logstore_flush_messages​ Type: int64 Default: 1,000 Messages to trigger a fsync #### logstore_flush_ms​ Type: int64 Default: 120,000 Time in milliseconds to trigger a fsync #### fetch_max_bytes​ Type: uint64 Default: 8,388,608 Maximum bytes to fetch in a single read #### fetch_max_wait_ms​ Type: int64 Default: 500 Maximum time to wait for a fetch ## More Examples​ - PRIMARY KEY - INDEX - FAMILY - SETTINGSStorage Settingskvstore_codecenable_hash_indexenable_statisticsauto_cfRocksDB Performance Settingskvstore_optionsflush_rowsflush_msLogstore Settingslogstore_retention_byteslogstore_retention_mslogstore_flush_messageslogstore_flush_msfetch_max_bytesfetch_max_wait_ms - Storage Settingskvstore_codecenable_hash_indexenable_statisticsauto_cf - kvstore_codec - enable_hash_index - enable_statistics - auto_cf - RocksDB Performance Settingskvstore_optionsflush_rowsflush_ms - kvstore_options - flush_rows - flush_ms - Logstore Settingslogstore_retention_byteslogstore_retention_mslogstore_flush_messageslogstore_flush_msfetch_max_bytesfetch_max_wait_ms - logstore_retention_bytes - logstore_retention_ms - logstore_flush_messages - logstore_flush_ms - fetch_max_bytes - fetch_max_wait_ms - More Examples - Storage Settingskvstore_codecenable_hash_indexenable_statisticsauto_cf - kvstore_codec - enable_hash_index - enable_statistics - auto_cf - RocksDB Performance Settingskvstore_optionsflush_rowsflush_ms - kvstore_options - flush_rows - flush_ms - Logstore Settingslogstore_retention_byteslogstore_retention_mslogstore_flush_messageslogstore_flush_msfetch_max_bytesfetch_max_wait_ms - logstore_retention_bytes - logstore_retention_ms - logstore_flush_messages - logstore_flush_ms - fetch_max_bytes - fetch_max_wait_ms - kvstore_codec - enable_hash_index - enable_statistics - auto_cf - kvstore_options - flush_rows - flush_ms - logstore_retention_bytes - logstore_retention_ms - logstore_flush_messages - logstore_flush_ms - fetch_max_bytes - fetch_max_wait_ms ## Sql-create-random-stream - [CREATE RANDOM STREAM | Timeplus](https://docs.timeplus.com/sql-create-random-stream): You may use this special stream to generate random data for tests. For example: - SQL Reference - Statements - CREATE RANDOM STREAM # CREATE RANDOM STREAM You may use this special stream to generate random data for tests. For example: The following functions are available to use: 1. rand to generate a number in uint32 2. rand64 to generate a number in uint64 3. random_printable_ascii to generate printable characters 4. random_string to generate a string 5. random_fixed_string to generate string in fixed length 6. random_in_type to generate value with max value and custom logic The data of random stream is kept in memory during the query time. If you are not querying the random stream, there is no data generated or kept in memory. By default, Timeplus tries to generate as many data as possible. If you want to (roughly) control how frequent the data is generated, you can use the eps setting. For example, the following SQL generates 10 events every second: You can further customize the rate of data generation via the interval_time setting. For example, you want to generate 1000 events each second, but don't want all 1000 events are generated at once, you can use the following sample SQL to generate events every 200 ms. The default interval is 5ms (in Proton 1.3.27 or the earlier versions, the default value is 100ms) Please note, the data generation rate is not accurate, to balance the performance and flow control. New in Proton v1.4.2, you can set eps less than 1. Such as eps=0.5 will generate 1 event every 2 seconds. eps less than 0.00001 will be treated as 0. For testing or demonstration purpose, you can create a random stream with multiple columns and use the table function to generate random data at once. The number of rows generated by this way is predefined and subject to change. The current value is 65409. ## Sql-create-remote-function - [CREATE REMOTE FUNCTION | Timeplus](https://docs.timeplus.com/sql-create-remote-function): At Timeplus, we leverage SQL to make powerful streaming analytics more accessible to a broad range of users. Without SQL, you have to learn and call low-level programming API, then compile/package/deploy them to get analytics results. This is a repetitive and tedious process, even for small changes. - SQL Reference - Statements - CREATE REMOTE FUNCTION # CREATE REMOTE FUNCTION At Timeplus, we leverage SQL to make powerful streaming analytics more accessible to a broad range of users. Without SQL, you have to learn and call low-level programming API, then compile/package/deploy them to get analytics results. This is a repetitive and tedious process, even for small changes. But some developers have concerns that complex logic or systems integration are hard to express using SQL. That's why we add User-Defined Functions (UDF) support in Timeplus. This enables users to leverage existing programming libraries, integrate with external systems, or just make SQL easier to maintain. Timeplus Proton supports Local UDF in JavaScript. You can develop User-defined scalar functions (UDFs) or User-defined aggregate functions (UDAFs) with modern JavaScript (powered by V8). No need to deploy extra server/service for the UDF. More languages will be supported. In Timeplus Enterprise, the Python UDF will be ready soon. ## CREATE REMOTE FUNCTION​ Please refer to the example for how to build a IP lookup service via AWS Lambda and register it as a remote UDF in Timeplus. ## Authentication with HTTP Header​ If you need to protect the end point and only accept requests with a certain HTTP header, you can use the AUTH_HEADER and AUTH_KEY setting, e,g. ## Customize the timeout​ By default, Timeplus sets 10 seconds as the timeout for the remote UDF call. You can customize this via EXECUTION_TIMEOUT. The unit is millisecond. This feature is added in Timeplus Proton 1.5.18. - CREATE REMOTE FUNCTION - Authentication with HTTP Header - Customize the timeout ## Sql-create-stream - [CREATE STREAM | Timeplus](https://docs.timeplus.com/sql-create-stream): Stream is a key concept in Timeplus. All data lives in streams, no matter static data or data in motion. We don't recommend you to create or manage TABLE in Timeplus. - SQL Reference - Statements - CREATE STREAM # CREATE STREAM Stream is a key concept in Timeplus. All data lives in streams, no matter static data or data in motion. We don't recommend you to create or manage TABLE in Timeplus. ## Syntax​ ### Append Stream​ Stream creation is an async process. If you omit the database name, default will be used. Stream name can be any utf-8 characters and needs backtick quoted if there are spaces in between. Column name can be any utf-8 characters and needs backtick quoted if there are spaces in between. ### Mutable Stream​ Mutable Streams are available in Timeplus Enterprise. Guide: CREATE MUTABLE STREAM ### Versioned Stream​ Versioned Stream allows you to specify the primary key(s) and focus on the latest value. For example: The default version_column is _tp_time. For the data with same primary key(s), Proton will use the ones with maximum value of version_column. So by default, it tracks the most recent data for same primary key(s). If there are late events, you can use specify other column to determine the end state for your live data. ### Changelog Stream​ Changelog Stream allows you to specify the primary key(s) and track the add/delete/update of the data. For example: The default version_column is _tp_time. For the data with same primary key(s), Proton will use the ones with maximum value of version_column. So by default, it tracks the most recent data for same primary key(s). If there are late events, you can use specify other column to determine the end state for your live data. ## SETTINGS​ #### mode​ Type: string Default: append Stream mode. It can be append, versioned_kv, changelog_kv, changelog. #### shards​ Type: int64 Default: 1 Number of shards for the stream. #### replication_factor​ Type: int64 Default: 1 for single node and 3 for cluster. #### version_column​ Type: string Default: _tp_time Column name to determine the version of the data. It's only used in versioned_kv and changelog_kv mode. #### keep_versions​ Type: int64 Default: 1 Number of versions to keep for each key. It's only used in versioned_kv mode. #### event_time_column​ Type: string Default: now64(3, 'UTC') Column name or expression to determine the event time of the data. #### logstore_codec​ Type: string Default: none Codec for the logstore. It can be none, zstd, lz4. #### logstore_retention_bytes​ Type: int64 Default: -1 Maximum size of the logstore in bytes. -1 means no limit. #### logstore_retention_ms​ Type: int64 Default: -1 Maximum time to keep the logstore in milliseconds. -1 means no limit. #### logstore_flush_messages​ Type: int64 Default: 1,000 Messages to trigger a fsync #### logstore_flush_ms​ Type: int64 Default: 120,000 Time in milliseconds to trigger a fsync #### replication_type This is an advanced setting. Default value is hybrid to use both a streaming storage and a historical storage for the stream. It can be: - streaming: Use only streaming storage, together with settings logstore_codec, logstore_retention_bytes, logstore_retention_ms. - memory: put data in memory only, mainly for testing. ## TTL (Time-To-Live)​ The logstore_retention_bytes and logstore_retention_ms settings control the maximum size and time to keep the streaming storage. The historical storage for the stream is controlled by the TTL expression. Syntax: For example, TTL _tp_time + INTERVAL 1 DAY will delete rows older than one day, TTL _tp_time + INTERVAL 7 DAY will delete rows older than one week. For S3 Tried Storage, you can also specify when the cold data will be moved to S3. For example: - SyntaxAppend StreamMutable StreamVersioned StreamChangelog Stream - Append Stream - Mutable Stream - Versioned Stream - Changelog Stream - SETTINGSmodeshardsreplication_factorversion_columnkeep_versionsevent_time_columnlogstore_codeclogstore_retention_byteslogstore_retention_mslogstore_flush_messageslogstore_flush_msreplication_type - mode - shards - replication_factor - version_column - keep_versions - event_time_column - logstore_codec - logstore_retention_bytes - logstore_retention_ms - logstore_flush_messages - logstore_flush_ms - replication_type - TTL (Time-To-Live) - Append Stream - Mutable Stream - Versioned Stream - Changelog Stream - mode - shards - replication_factor - version_column - keep_versions - event_time_column - logstore_codec - logstore_retention_bytes - logstore_retention_ms - logstore_flush_messages - logstore_flush_ms - replication_type ## Sql-create-view - [CREATE VIEW | Timeplus](https://docs.timeplus.com/sql-create-view): There are two types of views in Timeplus: logical view (or just view ) and materialized view. - SQL Reference - Statements - CREATE VIEW # CREATE VIEW There are two types of views in Timeplus: logical view (or just view ) and materialized view. You can create views for all kinds of queries, and refer to the views in other queries. - If the view is created based on a streaming query, then you can consider the view as a virtual stream. For example, create view view1 as select * from my_stream where c1 ='a' will create a virtual stream to filter all events with c1 = 'a'. You can use this view as if it's another stream, e.g. select count(*) from tumble(view1,1m) group by window_start Creating a view won't trigger any query execution. Views are evaluated only when other queries refer to it. - a view could be a bounded stream if the view is created with a bounded query using table() function, e.g. create view view2 as select * from table(my_stream) then each time you run select count(*) from view2 will return the current row number of the my_stream immediately without waiting for the future events. Please note, once the view is created based on a streaming query, you cannot turn it to a bounded stream via table(streaming_view) To create a vanilla view: ## Sql-delete - [DELETE | Timeplus](https://docs.timeplus.com/sql-delete): Starting from Timeplus Enterprise 2.7, you can delete data from the streams. - SQL Reference - Statements - DELETE # DELETE Starting from Timeplus Enterprise 2.7, you can delete data from the streams. Note: - If you delete from a mutable stream, it's recommended to use the primary key(s) in the condition to delete the data efficiently. You can also use the secondary index or other columns in the condition. The operation is synchronous and will block until the data is deleted. - If you delete from a normal stream, the operation is asynchronous and will return immediately. The data will be deleted in the background. ## Sql-drop-alert - [DROP ALERT | Timeplus](https://docs.timeplus.com/sql-drop-alert): Starting from Timeplus Enterprise 2.9, you can create alerts to monitor your streaming data and automatically trigger actions when specific conditions are met. - SQL Reference - Statements - DROP ALERT # DROP ALERT Starting from Timeplus Enterprise 2.9, you can create alerts to monitor your streaming data and automatically trigger actions when specific conditions are met. ## DROP ALERT​ When you no longer need an alert, you can drop it with the following SQL: ## See also​ - CREATE ALERT - Create alerts - SHOW ALERTS - Show alerts - DROP ALERT - See also ## Sql-drop-database - [DROP DATABASE | Timeplus](https://docs.timeplus.com/sql-drop-database): Starting from Timeplus Enterprise 2.6, you can create a database to organize your streams, materialized views and other resources. The database is a logical container to help you to manage and query the data more efficiently. - SQL Reference - Statements - DROP DATABASE # DROP DATABASE Starting from Timeplus Enterprise 2.6, you can create a database to organize your streams, materialized views and other resources. The database is a logical container to help you to manage and query the data more efficiently. ## DROP DATABASE​ When you no longer need a database, you can drop it with the following SQL: If there are streams, materialized views, or other resources in the database, this operation will fail. You need to drop them first, or use the CASCADE option to drop all the resources in the database. ## DROP DATABASE CASCADE​ This feature is available in Timeplus Enterprise 2.8 and later versions. Adding CASCADE to the DROP DATABASE command is a permanent operation and cannot be undone. All the streams, materialized views, and other resources in the database will be deleted. Please be cautious when you run this command. The only exception is the external database to Iceberg or MySQL/PostgreSQL. Running DROP DATABASE my_database CASCADE won't remove tables in the remote database. ## See also​ - CREATE DATABASE - Create databases - SHOW DATABASES - Show databases - DROP DATABASE - DROP DATABASE CASCADE - See also ## Sql-drop-dictionary - [DROP DICTIONARY | Timeplus](https://docs.timeplus.com/sql-drop-dictionary): Starting from Timeplus Enterprise 2.7, you can create dictionaries to store key-value pairs in memory, with data from various sources, such as files, MySQL/ClickHouse databases, or streams in Timeplus. - SQL Reference - Statements - DROP DICTIONARY # DROP DICTIONARY Starting from Timeplus Enterprise 2.7, you can create dictionaries to store key-value pairs in memory, with data from various sources, such as files, MySQL/ClickHouse databases, or streams in Timeplus. When you no longer need a dictionary, you can drop it with the following SQL: ## See also​ - CREATE DICTIONARY - Create dictionaries - SHOW DICTIONARIES - Show dictionaries - See also ## Sql-drop-external-table - [DROP EXTERNAL TABLE | Timeplus](https://docs.timeplus.com/sql-drop-external-table): Run the following SQL to drop an external table from Timeplus, while keeping the data in external database intact. - SQL Reference - Statements - DROP EXTERNAL TABLE # DROP EXTERNAL TABLE Run the following SQL to drop an external table from Timeplus, while keeping the data in external database intact. ## Sql-drop-format-schema - [DROP FORMAT SCHEMA | Timeplus](https://docs.timeplus.com/sql-drop-format-schema): No description available - SQL Reference - Statements - DROP FORMAT SCHEMA # DROP FORMAT SCHEMA ## Sql-drop-function - [DROP FUNCTION | Timeplus](https://docs.timeplus.com/sql-drop-function): No matter UDF or UDAF, you can remove the function via DROP FUNCTION - SQL Reference - Statements - DROP FUNCTION # DROP FUNCTION No matter UDF or UDAF, you can remove the function via DROP FUNCTION Example: ## Drop a function forcefully​ If the UDF or UDAF is used in other queries, you can force to drop it. This new setting is available since Timeplus Enterprise v2.5.12. Please recreate the UDF or UDAF before running the queries that depend on it. - Drop a function forcefully ## Sql-drop-stream - [DROP STREAM | Timeplus](https://docs.timeplus.com/sql-drop-stream): Run the following SQL to drop a stream or an external stream, with all data in streaming storage and historical storage. - SQL Reference - Statements - DROP STREAM # DROP STREAM Run the following SQL to drop a stream or an external stream, with all data in streaming storage and historical storage. Like CREATE STREAM, stream deletion is an async process. ## force_drop_big_stream​ By default, if the total size of the stream is more than 50 GB, the SQL will fail to run. You can edit the config.yml to set a different threshold. You can also add force_drop_big_stream=true to the DROP command, e.g. force_drop_big_stream is available since Timeplus Enterprise 2.4.17 and Timeplus Proton 1.5.17. - force_drop_big_stream ## Sql-drop-view - [DROP VIEW | Timeplus](https://docs.timeplus.com/sql-drop-view): Run the following SQL to drop a view or a materialized view. - SQL Reference - Statements - DROP VIEW # DROP VIEW Run the following SQL to drop a view or a materialized view. Like CREATE STREAM, stream deletion is an async process. ## force_drop_big_stream​ By default, if the total size of the materialized view is more than 50 GB, the SQL will fail to run. You can edit the config.yml to set a different threshold. You can also add force_drop_big_stream=true to the DROP command, e.g. force_drop_big_stream is available since Timeplus Enterprise 2.4.17 and Timeplus Proton 1.5.17. - force_drop_big_stream ## Sql-explain - [EXPLAIN | Timeplus](https://docs.timeplus.com/sql-explain): The execution of a query in Timeplus is decomposed into many steps. Each step of the query execution can be analyzed and troubleshooted using the corresponding EXPLAIN query. These steps are summarized in the chart below: - SQL Reference - Statements - EXPLAIN # EXPLAIN The execution of a query in Timeplus is decomposed into many steps. Each step of the query execution can be analyzed and troubleshooted using the corresponding EXPLAIN query. These steps are summarized in the chart below: ## Syntax​ ## EXPLAIN Types​ - AST — Abstract syntax tree. - SYNTAX — Query text after AST-level optimizations. - PLAN — Query execution plan. This is the default explain type. - PIPELINE — Query execution pipeline. - ESTIMATE — Query estimated number of rows, marks and parts. ### EXPLAIN AST​ Dump query AST. Supports all types of queries, not only SELECT. Examples: ### EXPLAIN SYNTAX​ Shows the Abstract Syntax Tree (AST) of a query after syntax analysis. It's done by parsing the query, constructing query AST and query tree, optionally running query analyzer and optimization passes, and then converting the query tree back to the query AST. Settings are not supported for EXPLAIN SYNTAX query. Example: Output: ### EXPLAIN PLAN​ Dump query plan steps. This is the default explain type. Running EXPLAIN is same as EXPLAIN PLAN. Settings: - json — Prints query plan steps as a row in JSON format. Default: 0. It is recommended to use TSVRaw format to avoid unnecessary escaping. - description — Prints step description. Default: 1. - header — Prints output header for step. Default: 0. - indexes — Shows used indexes, the number of filtered parts and the number of filtered granules for every index applied. Default: 0. Supported for streams with columnar storages, i.e. Append Stream, Versioned KV Stream, and Changelog KV Stream. - actions — Prints detailed information about step actions. Default: 0. When json=1 step names will contain an additional suffix with unique step identifier. Example: Step and query cost estimation is not supported. #### json​ When json = 1, the query plan is represented in JSON format. Every node is a dictionary that always has the keys Node Type and Plans. Node Type is a string with a step name. Plans is an array with child step descriptions. Other optional keys may be added depending on node type and settings. Example: #### description​ With description = 1, the Description key is added to the step: #### header​ With header = 1, the Header key is added to the step as an array of columns. Example: #### indexes​ With indexes = 1, the Indexes key is added. It contains an array of used indexes. Each index is described as JSON with Type key (a string MinMax, Partition, PrimaryKey or Skip) and optional keys: - Name — The index name (currently only used for Skip indexes). - Keys — The array of columns used by the index. - Condition — The used condition. - Description — The index description (currently only used for Skip indexes). - Parts — The number of parts before/after the index is applied. - Granules — The number of granules before/after the index is applied. Example: #### actions​ With actions = 1, added keys depend on step type. Example: ### EXPLAIN PIPELINE​ Settings: - header — Prints header for each output port. Default: 0. - graph — Prints a graph described in the DOT graph description language. Default: 0. - compact — Prints graph in compact mode if graph setting is enabled. Default: 1. When compact=0 and graph=1 processor names will contain an additional suffix with unique processor identifier. Example: #### graph​ With graph = 1, the result is shown in graph description language. Example: #### compact​ Example: ### EXPLAIN ESTIMATE​ Shows the estimated number of rows, marks and parts to be read from the streams while processing the query. Example Creating a stream: Query: Result: - Syntax - EXPLAIN TypesEXPLAIN ASTEXPLAIN SYNTAXEXPLAIN PLANjsondescriptionheaderindexesactionsEXPLAIN PIPELINEgraphcompactEXPLAIN ESTIMATE - EXPLAIN AST - EXPLAIN SYNTAX - EXPLAIN PLANjsondescriptionheaderindexesactions - json - description - header - indexes - actions - EXPLAIN PIPELINEgraphcompact - graph - compact - EXPLAIN ESTIMATE - EXPLAIN AST - EXPLAIN SYNTAX - EXPLAIN PLANjsondescriptionheaderindexesactions - json - description - header - indexes - actions - EXPLAIN PIPELINEgraphcompact - graph - compact - EXPLAIN ESTIMATE - json - description - header - indexes - actions - graph - compact ## Sql-optimize - [OPTIMIZE | Timeplus](https://docs.timeplus.com/sql-optimize): Run the following SQL to initiate an unscheduled merge of a mutable stream or a table with MergeTree engine. This will reorganize the data in the stream and optimize the performance of the stream. Note that we generally recommend against using OPTIMIZE STREAM ... FINAL as its use case is meant for administration, not for daily operations. - SQL Reference - Statements - OPTIMIZE # OPTIMIZE Run the following SQL to initiate an unscheduled merge of a mutable stream or a table with MergeTree engine. This will reorganize the data in the stream and optimize the performance of the stream. Note that we generally recommend against using OPTIMIZE STREAM ... FINAL as its use case is meant for administration, not for daily operations. Syntax: Note: - If you specify FINAL or FORCE, optimization is performed even when all the data is already optimized. The operation is resource intensive, consuming significant CPU and disk I/O. - If you specify DEDUPLICATE, then completely identical rows (unless by-clause is specified) will be deduplicated (all columns are compared), it makes sense only for the MergeTree engine. - If you specify DEDUPLICATE BY .., then only rows in the specified columns will be deduplicated. Examples: OPTIMIZE is only available in Timeplus Enterprise v2.7.x or above, and ttl_seconds is available in Timeplus Enterprise v2.9.x or above. ## Sql-pattern-topn - [Top-N | Timeplus](https://docs.timeplus.com/sql-pattern-topn): You can get the top-most or bottom-most events in a Timeplus stream or a view, based on a condition. - Guides & Tutorials - Top-N # Top-N You can get the top-most or bottom-most events in a Timeplus stream or a view, based on a condition. There are 3 types of Top-N query patterns: 1. Top-K: to get the most common value for a field. For example, each event with an action field, can be add/remove/service. You want to check which actions are most common in the live data. To avoid confusion with the general Top-N patterns, we call this pattern as Top-K. 2. Max-K: to get the top-most value for a field. For example, each event with a speed_kmh field. You want to get the cars with top speed. In some systems, this is called TopN. But we think Max-N or Max-K will be more accurate. 3. Min-K: to get the bottom-most value for a field. For example, each event with a speed_kmh field. You want to get the cars with slowest speed. Please read on for more detailed explanations and examples. ## Top-K​ Sample query: Result: This query lists the most common action in the bookings stream since 1d (one day) ago. If you need the number of appearance for each value, you can pass true as the 3rd parameter for the top_k function, e.g. Sample query: Result: Please note, this sample query is a Global Aggregation, which calculate and emit results every 2 seconds. You can also use various time windows to run the aggregation, e.g. ## Max-K​ In some systems, this is called TopN. But we think Max-N or Max-K will be more accurate. Sample query: Result: In many cases, you need to know the value of other columns with such max value. You can add any number of column name as the optional parameters in the max_k function. Result: ## Min-K​ The opposite of Max-K. Get the bottom-most value with optional context columns. Sample queries: Or: ## Top-N By Group​ No matter top_k, max_k, or min_k, you can also combine them with partition by to get the top-N value in a specify group. Say if there is a model attribute for each car data. This query can get the fastest 3 cars in each model - Top-K - Max-K - Min-K - Top-N By Group ## Sql-rename-stream - [RENAME STREAM | Timeplus](https://docs.timeplus.com/sql-rename-stream): You can rename a stream using the RENAME STREAM statement. - SQL Reference - Statements - RENAME STREAM # RENAME STREAM You can rename a stream using the RENAME STREAM statement. Syntax: Make sure the new stream name is unique within the database. You cannot rename a stream when the stream is being used by any active queries or when the stream is part of a view or materialized view. Renaming a stream cannot change the database it belongs to. ## Sql-show-alerts - [SHOW ALERTS | Timeplus](https://docs.timeplus.com/sql-show-alerts): Starting from Timeplus Enterprise 2.9, you can create alerts to monitor your streaming data and automatically trigger actions when specific conditions are met. - SQL Reference - Statements - SHOW ALERTS # SHOW ALERTS Starting from Timeplus Enterprise 2.9, you can create alerts to monitor your streaming data and automatically trigger actions when specific conditions are met. You can list all alerts in the current database via: Or list alerts from a specified database namespace: If you add SETTINGS verbose=true to the SQL command, more details will be printed out. ## See also​ - CREATE ALERT - Create alerts - DROP ALERT - Drop alerts - See also ## Sql-show-create - [SHOW CREATE | Timeplus](https://docs.timeplus.com/sql-show-create): Show Details For A Stream - SQL Reference - Statements - SHOW CREATE # SHOW CREATE ## Show Details For A Stream​ or ## Show Details For A Schema​ ## Show Details For A UDF​ ## Show Details For An Alert​ If you add SETTINGS show_multi_versions=true, it will list earlier versions. ## show_uuid​ Starting from Timeplus Enterprise v2.7, you can show the uuid of the SQL object to map the name with folder name in file systems via settings show_uuid=true. ## show_multi_versions​ Starting from Timeplus Enterprise v2.8.2, you can track the history of the SQL definition via settings show_multi_versions=true. For example: The output includes the following columns: - statement - version - last_modified - last_modified_by - created - created_by - uuid ## verbose​ Starting from Timeplus Enterprise v2.9, you can use settings verbose=true to show extra information for materialized views. For example: Besides the statement column, there is also a placements column with content like - Show Details For A Stream - Show Details For A Schema - Show Details For A UDF - Show Details For An Alert - show_uuid - show_multi_versions - verbose ## Sql-show-databases - [SHOW DATABASES | Timeplus](https://docs.timeplus.com/sql-show-databases): Starting from Timeplus Enterprise 2.6, you can create a database to organize your streams, materialized views and other resources. The database is a logical container to help you to manage and query the data more efficiently. - SQL Reference - Statements - SHOW DATABASES # SHOW DATABASES Starting from Timeplus Enterprise 2.6, you can create a database to organize your streams, materialized views and other resources. The database is a logical container to help you to manage and query the data more efficiently. You can list all the databases in the cluster with the following SQL: ## See also​ - CREATE DATABASE - Create databases - DROP DATABASE - Drop databases - See also ## Sql-show-dictionaries - [SHOW DICTIONARIES | Timeplus](https://docs.timeplus.com/sql-show-dictionaries): Starting from Timeplus Enterprise 2.7, you can create dictionaries to store key-value pairs in memory, with data from various sources, such as files, MySQL/ClickHouse databases, or streams in Timeplus. - SQL Reference - Statements - SHOW DICTIONARIES # SHOW DICTIONARIES Starting from Timeplus Enterprise 2.7, you can create dictionaries to store key-value pairs in memory, with data from various sources, such as files, MySQL/ClickHouse databases, or streams in Timeplus. You can list all the dictionaries in the cluster with the following SQL: The output will include a single column name with the names of the dictionaries. You can also add a WHERE clause to filter the dictionaries by name: ## See also​ - CREATE DICTIONARY - Create dictionaries - DROP DICTIONARY - Drop dictionaries - See also ## Sql-show-disks - [SHOW DISKS | Timeplus](https://docs.timeplus.com/sql-show-disks): Starting from Timeplus Enterprise 2.8, you can create S3 disks for tiered storage or autoscaling materialized views. - SQL Reference - Statements - SHOW DISKS # SHOW DISKS Starting from Timeplus Enterprise 2.8, you can create S3 disks for tiered storage or autoscaling materialized views. ## Syntax​ You can list all the disks in the cluster with the following SQL: ## See also​ - CREATE DISKS - Create disks - Syntax - See also ## Sql-show-format-schemas - [SHOW FORMAT SCHEMAS | Timeplus](https://docs.timeplus.com/sql-show-format-schemas): List schemas in the current Timeplus deployment: - SQL Reference - Statements - SHOW FORMAT SCHEMAS # SHOW FORMAT SCHEMAS List schemas in the current Timeplus deployment: ## Sql-show-functions - [SHOW FUNCTIONS | Timeplus](https://docs.timeplus.com/sql-show-functions): List available User-Defined Functions. - SQL Reference - Statements - SHOW FUNCTIONS # SHOW FUNCTIONS List available User-Defined Functions. This feature is only available in Timeplus Enterprise v2.x. ## Show All UDF​ ## Show Matching UDF​ This command will list all UDFs with names starting with 'get'. - Show All UDF - Show Matching UDF ## Sql-show-streams - [SHOW STREAMS | Timeplus](https://docs.timeplus.com/sql-show-streams): List Streams From Current Database - SQL Reference - Statements - SHOW STREAMS # SHOW STREAMS ## List Streams From Current Database​ ## List Streams From Other Database​ - List Streams From Current Database - List Streams From Other Database ## Sql-system-pause - [SYSTEM PAUSE | Timeplus](https://docs.timeplus.com/sql-system-pause): SYSTEM PAUSE MATERIALIZED VIEW - SQL Reference - Statements - SYSTEM PAUSE # SYSTEM PAUSE ## SYSTEM PAUSE MATERIALIZED VIEW​ Materialized views are long-running queries. In some cases, you may want to pause the query so that no more data is sent to the downstream. You can unpause the materialized view later. Please avoid restarting the server when the materialized view is paused. - SYSTEM PAUSE MATERIALIZED VIEW ## Sql-system-recover - [SYSTEM RECOVER | Timeplus](https://docs.timeplus.com/sql-system-recover): This is an advanced feature, for debugging purpose. - SQL Reference - Statements - SYSTEM RECOVER # SYSTEM RECOVER This is an advanced feature, for debugging purpose. ## SYSTEM RECOVER MATERIALIZED VIEW​ - SYSTEM RECOVER MATERIALIZED VIEW ## Sql-system-resume - [SYSTEM RESUME | Timeplus](https://docs.timeplus.com/sql-system-resume): SYSTEM RESUME MATERIALIZED VIEW - SQL Reference - Statements - SYSTEM RESUME # SYSTEM RESUME ## SYSTEM RESUME MATERIALIZED VIEW​ Materialized views are long-running queries. In some cases, you may want to pause the query so that no more data is sent to the downstream. Then resume the materialized view later. Please avoid restarting the server when the materialized view is paused. This command is only available in Timeplus Enterprise v2.7.0 or above. In earlier versions, you can use SYSTEM UNPAUSE MATERIALIZED VIEW to resume the materialized view. - SYSTEM RESUME MATERIALIZED VIEW ## Sql-system-set-log-level - [SYSTEM SET LOG LEVEL | Timeplus](https://docs.timeplus.com/sql-system-set-log-level): This feature is available in Timeplus Enterprise v2.8.2 or above. Not available in Timeplus Proton. - SQL Reference - Statements - SYSTEM SET LOG LEVEL # SYSTEM SET LOG LEVEL This feature is available in Timeplus Enterprise v2.8.2 or above. Not available in Timeplus Proton. Example: You don't need to restart the server to apply the changes. The log level will be applied immediately across all the nodes in the cluster. ## Log Levels​ The following log levels are supported: - critical - debug - error - fatal - information - none - notice - test - trace - warning - Log Levels ## Sql-system-transfer-leader - [SYSTEM TRANSFER LEADER | Timeplus](https://docs.timeplus.com/sql-system-transfer-leader): Transfer the leader of a materialized view to another node in the cluster. - SQL Reference - Statements - SYSTEM TRANSFER LEADER # SYSTEM TRANSFER LEADER Transfer the leader of a materialized view to another node in the cluster. For example: This feature is only available in Timeplus Enterprise v2.5.x or above. This SQL need to be run on the lead node of the cluster. ## Sql-truncate-stream - [TRUNCATE STREAM | Timeplus](https://docs.timeplus.com/sql-truncate-stream): Run the following SQL to remove the historical data from a stream, keeping the schema and the stream itself. - SQL Reference - Statements - TRUNCATE STREAM # TRUNCATE STREAM Run the following SQL to remove the historical data from a stream, keeping the schema and the stream itself. This feature is only available in Timeplus Enterprise v2.6.x or above. ## Sql-udf - [SQL UDF | Timeplus](https://docs.timeplus.com/sql-udf): SQL UDFs are simple yet powerful extensions to SQL on Timeplus. You can create common SQL snippets as SQL UDFs and reuse them in your queries. They provide a layer of abstraction to make your queries more readable and modularized. Unlike UDFs that are written in other programming languages, SQL UDFs are stateless and have the best performance, powered by Timeplus C++ engine. - SQL Reference - Functions - User Defined Functions - SQL UDF # SQL UDF SQL UDFs are simple yet powerful extensions to SQL on Timeplus. You can create common SQL snippets as SQL UDFs and reuse them in your queries. They provide a layer of abstraction to make your queries more readable and modularized. Unlike UDFs that are written in other programming languages, SQL UDFs are stateless and have the best performance, powered by Timeplus C++ engine. Create a SQL UDF from a lambda SQL expression. The expression must consist of function parameters, constants, operators, or other function calls. A function can have an arbitrary number of parameters. There are a few restrictions: - The name of a function must be unique among user defined and system functions. - Recursive functions are not allowed. - All variables used by a function must be specified in its parameter list. - If any restriction is violated then an exception is raised. ## Examples​ ### SQL UDFs as constants​ Let's start with a simple example. You can define a SQL UDF that returns a constant value, so that you can avoid using literals in your queries. Let's use the function: ### SQL UDF encapsulating a complex expression​ You can also define a SQL UDF that encapsulates a complex expression, so that you don't have to copy-paste some lengthy expressions over and over again in multiple SQL queries. Notes: - You can overwrite the definition of a SQL UDF by using CREATE OR REPLACE FUNCTION. - The parameters of a SQL UDF can be of any type, including complex types like arrays and maps. - Running SELECT color_hex(12,120,200) is equivalent to running SELECT '#'||hex(12)||hex(120)||hex(200). - In the function body, you can call any built-in functions, other SQL UDFs, or remote/JavaScript UDFs. - ExamplesSQL UDFs as constantsSQL UDF encapsulating a complex expression - SQL UDFs as constants - SQL UDF encapsulating a complex expression - SQL UDFs as constants - SQL UDF encapsulating a complex expression ## Sql-use - [USE | Timeplus](https://docs.timeplus.com/sql-use): Starting from Timeplus Enterprise 2.6, you can create a database to organize Timeplus streams, materialized views, or connect to other systems such as MySQL and Iceberg without creating multiple external tables repeating the connection information. - SQL Reference - Statements - USE # USE Starting from Timeplus Enterprise 2.6, you can create a database to organize Timeplus streams, materialized views, or connect to other systems such as MySQL and Iceberg without creating multiple external tables repeating the connection information. Once the database is created, run USE my_database to switch to the database. ## See also​ - CREATE DATABASE - Create databases - SHOW DATABASES - Show databases - DROP DATABASE - Drop databases - See also ## Stream-query - [Streaming Query | Timeplus](https://docs.timeplus.com/stream-query): Query is unbounded by default - Core Features - Stream Processing - Streaming Query # Streaming Query ## Query is unbounded by default​ By default, Timeplus query behavior is different from traditional SQL which answers the question of what had happened. Instead, Timeplus query tries to answer the question of what is happening now in real-time and continuously updates the answer when new events enter the system. Timeplus query is running on an unbounded stream. In most of the cases, the query won't stop unless the user cancels it. For example, the following query will return all the events in the stream that enter the Timeplus system in real-time after the query is executed. Each new event will trigger a new query result. And this query won't stop unless the user cancels the query. The unbounded query can be converted to a bounded query by applying the function table(), when the user wants to ask the question about what has happened like the traditional SQL. The table() function could be used to decorate the stream. For example, the following query will return all the events that have already existed in the stream at the moment of the query execution. The query will terminate once all results have been returned to the user and it won't wait for a new event coming. ## How streaming queries are triggered​ There are three categories of Timeplus streaming queries, based on how the data is aggregated. If you see some new terms, no worries. Let's explore them more. ### Non-aggregation​ Aggregation is the process to combine data from across events into one or more new data. Some queries don't involve any aggregations, such as: #### Tail​ List all incoming data, such as #### Filter​ Only show certain columns or data matches certain pattern, such as #### Transform​ For each event, transform the data to remove sensitive information, or convert type, such as Non-aggregation is triggered per event arrival, which means every time there is a new event entering Timeplus, the query will use the new event to execute the related analysis and the analysis result will be then triggered and sent to the client. ### Window aggregation​ Window based aggregation is a typical analytic method in stream analysis. Each window has a fixed range with a specific start time and an end time. The window might move during analysis by a fixed step. The analysis result will be based on an aggregation function of all the events lived in that window range. When using the window function for aggregation, the event time is used to decide whether the event is in that window. In the case the user does not specify the timestamp, the default one will be used. The user can also use any field in that event which is a datetime type as timestamp or dynamically generate a datetime field as timestamp. Learn More. ### Global aggregation​ Global aggregation will start the aggregation for all incoming events since the query is submitted, and never ends. Learn More. - Query is unbounded by default - How streaming queries are triggeredNon-aggregationTailFilterTransformWindow aggregationGlobal aggregation - Non-aggregationTailFilterTransform - Tail - Filter - Transform - Window aggregation - Global aggregation - Non-aggregationTailFilterTransform - Tail - Filter - Transform - Window aggregation - Global aggregation - Tail - Filter - Transform ## Streaming-aggregations - [Aggregations & Emit Polices | Timeplus](https://docs.timeplus.com/streaming-aggregations): Global Streaming Aggregation - Core Features - Stream Processing - Aggregations & Emit Polices # Aggregations & Emit Polices ### Global Streaming Aggregation​ In Timeplus, we define global aggregation as an aggregation query without using streaming windows like tumble, hop. Unlike streaming window aggregation, global streaming aggregation doesn't slice the unbound streaming data into windows according to timestamp, instead it processes the unbounded streaming data as one huge big global window. Due to this property, Timeplus for now can't recycle in-memory aggregation states / results according to timestamp for global aggregation. PERIODIC tells Timeplus to emit the aggregation periodically. UNIT can be ms(millisecond), s(second), m(minute),h(hour),d(day). shall be an integer greater than 0. Examples Like in Streaming Tail, Timeplus continuously monitors new events in the stream device_utils, does the filtering and then continuously does incremental count aggregation. Whenever the specified delay interval is up, project the current aggregation result to clients. ## Window Aggregation​ Timeplus supports various window aggregations, such as Tumble Window, Hop Window, and Session Window. ## EMIT​ As an advanced feature, Timeplus supports various policies to emit results during streaming query. Please note, we updated the EMIT syntax in Timeplus Enterprise 2.7.6. Please upgrade to the latest version to use those refined emit polices. For global aggregations, the syntax is: By default EMIT STREAM and PERIODIC 2s are applied. Advanced settings: - EMIT CHANGELOG works for global aggregations and non-aggregation tail/filter. It will output +1 or -1 for _tp_delta column. For time-window aggregations, the syntax is: You can only choose one of the emit policies: AFTER WINDOW CLOSE, PERIODIC, or ON UPDATE. If you omit any of them, the default policy is AFTER WINDOW CLOSE. Examples: ### EMIT .. WITH DELAY​ WITH DELAY and AND TIMEOUT only can be applied to time-window based aggregations. By default, the query engine will emit the results immediately when the window is closed or other conditions are met. This behavior can be customized using the WITH DELAY clause. It allows you to specify extra time to progress the watermark, which can be useful for handling late data. For example, if you want to wait for 1 second before emitting the results, you can use the following syntax: Please check the interactive demo on Understanding Watermark. ### EMIT .. WITH DELAY AND TIMEOUT​ For time window based aggregations, when the window is closed is decided by the watermark. A new event outside the window will progress the watermark and inform the query engine to close the previous window and to emit aggregation results. Say you only get one event for the time window. Since there is no more event, the watermark cannot be moved so the window won't be closed. EMIT .. TIMEOUT is to force the window close, with a timeout after seeing last event. Please note, if there no single event in the data stream, or in the time window, Proton won't emit result. For example, in the following SQL, you won't get 0 as the count: Even you add EMIT .. TIMEOUT in the SQL, it won't trigger timeout, because the query engine doesn't see any event in the window. If you need to detect such missing event for certain time window, one workaround is to create a heartbeat stream and use UNION to create a subquery to combine both heartbeat stream and target stream, for a time window, if all observed events are from heartbeat stream, this means there is no event in the target stream. Please discuss more with us in community slack. ### EMIT AFTER WINDOW CLOSE​ You can omit EMIT AFTER WINDOW CLOSE, since this is the default behavior for time window aggregations. For example: The above example SQL continuously aggregates max cpu usage per device per tumble window for the stream devices_utils. Every time a window is closed, Timeplus Proton emits the aggregation results. How to determine the window should be closed? This is done by Watermark, which is an internal timestamp. It is guaranteed to be increased monotonically per stream query. ### EMIT AFTER WINDOW CLOSE WITH DELAY​ Example: The above example SQL continuously aggregates max cpu usage per device per tumble window for the stream device_utils. Every time a window is closed, Timeplus Proton waits for another 2 seconds and then emits the aggregation results. ### EMIT PERIODIC​ PERIODIC tells Timeplus to emit the aggregation periodically. UNIT can be ms(millisecond), s(second), m(minute),h(hour),d(day). shall be an integer greater than 0. Example: For Global Streaming Aggregation the default periodic emit interval is 2s, i.e. 2 seconds. You can also apply EMIT PERIODIC in time windows, such as tumble/hop/session. When you run a tumble window aggregation, by default Timeplus will emit results when the window is closed. So tumble(stream,5s) will emit results every 5 seconds, unless there is no event in the window to progress the watermark. In some cases, you may want to get aggregation results even the window is not closed, so that you can get timely alerts. For example, the following SQL will run a 5-second tumble window and every 1 second, if the number of event is over 300, a row will be emitted. ### EMIT PERIODIC REPEAT​ Starting from Timeplus Proton 1.6.2, you can optionally add REPEAT to the end of EMIT PERIODIC . For global aggregations, by default every 2 seconds, the aggregation result will be emitted. But if there is no new event since last emit, no result will be emitted. With the REPEAT at the end of the emit policy, Timeplus will emit results at the fixed interval, even there is no new events since last emit. For example: ### EMIT ON UPDATE​ You can apply EMIT ON UPDATE in time windows, such as tumble/hop/session, with GROUP BY keys. For example: During the 5 second tumble window, even the window is not closed, as long as the aggregation value(cnt) for the same cid is different , the results will be emitted. ### EMIT ON UPDATE WITH DELAY​ Adding the WITH DELAY to EMIT ON UPDATE will allow late event for the window aggregation. ### EMIT ON UPDATE WITH BATCH​ You can combine EMIT PERIODIC and EMIT ON UPDATE together. In this case, even the window is not closed, Timeplus will check the intermediate aggregation result at the specified interval and emit rows if the result is changed. ### EMIT AFTER KEY EXPIRE IDENTIFIED BY .. WITH MAXSPAN .. AND TIMEOUT ..​ This emit policy is introduced in Timeplus Enterprise 2.9. Please watch the presentation from 2.9 launch webinar: The syntax is: Note: - EMIT AFTER KEY EXPIRE will emit results when the keys are expired. This EMIT policy ought to be applied to a global aggregation with a primary key as GROUP BY, usually using an ID for multiple tracing events. - IDENTIFIED BY col will calculate the span of the trace, usually you can set IDENTIFIED BY _tp_time. - MAXSPAN interval to identify whether the span of the related events over a certain interval, for example MAXSPAN 500ms to flag those events with same tracing ID but over 0.5 second span. - ONLY: if you add this keyword, then only those events over the MAXSPAN will be emitted, other events less than the MAXSPAN will be omitted, so that you can focus on those events over the SLA. - AND TIMEOUT interval to avoid waiting for late events for too long. If there is no more events with the same key (e.g. tracing ID) after this interval, Timeplus will close the session for the key and emit results. It's required to use SETTINGS default_hash_table='hybrid' with this emit policy to avoid putting too many data in memory. Here is an example to get the log streams and only show the events with over 0.5 second as the end-to-end latency. ### EMIT TIMEOUT​ You can apply EMIT TIMEOUT on global aggregation, e.g. It also can be applied to window aggregations and EMIT AFTER WINDOW CLOSE is automatically appended, e.g. - Global Streaming Aggregation - Window Aggregation - EMITEMIT .. WITH DELAYEMIT .. WITH DELAY AND TIMEOUTEMIT AFTER WINDOW CLOSEEMIT AFTER WINDOW CLOSE WITH DELAYEMIT PERIODICEMIT PERIODIC REPEATEMIT ON UPDATEEMIT ON UPDATE WITH DELAYEMIT ON UPDATE WITH BATCHEMIT AFTER KEY EXPIRE IDENTIFIED BY .. WITH MAXSPAN .. AND TIMEOUT ..EMIT TIMEOUT - EMIT .. WITH DELAY - EMIT .. WITH DELAY AND TIMEOUT - EMIT AFTER WINDOW CLOSE - EMIT AFTER WINDOW CLOSE WITH DELAY - EMIT PERIODIC - EMIT PERIODIC REPEAT - EMIT ON UPDATE - EMIT ON UPDATE WITH DELAY - EMIT ON UPDATE WITH BATCH - EMIT AFTER KEY EXPIRE IDENTIFIED BY .. WITH MAXSPAN .. AND TIMEOUT .. - EMIT TIMEOUT - EMIT .. WITH DELAY - EMIT .. WITH DELAY AND TIMEOUT - EMIT AFTER WINDOW CLOSE - EMIT AFTER WINDOW CLOSE WITH DELAY - EMIT PERIODIC - EMIT PERIODIC REPEAT - EMIT ON UPDATE - EMIT ON UPDATE WITH DELAY - EMIT ON UPDATE WITH BATCH - EMIT AFTER KEY EXPIRE IDENTIFIED BY .. WITH MAXSPAN .. AND TIMEOUT .. - EMIT TIMEOUT ## Streaming-windows - [Tumble/Hop/Session Windows | Timeplus](https://docs.timeplus.com/streaming-windows): Tumble - Core Features - Stream Processing - Tumble/Hop/Session Windows # Tumble/Hop/Session Windows ### Tumble​ Tumble slices the unbounded data into different windows according to its parameters. Internally, Timeplus observes the data streaming and automatically decides when to close a sliced window and emit the final results for that window. Tumble window means a fixed non-overlapped time window. Here is one example for a 5 seconds tumble window: tumble window in Timeplus is left closed and right open [) meaning it includes all events which have timestamps greater or equal to the lower bound of the window, but less than the upper bound of the window. tumble in the above SQL spec is a table function whose core responsibility is assigning tumble window to each event in a streaming way. The tumble table function will generate 2 new columns: window_start, window_end which correspond to the low and high bounds of a tumble window. tumble table function accepts 4 parameters: and are optional, the others are mandatory. When the parameter is omitted from the query, the stream's default event timestamp column which is _tp_time will be used. When the parameter is omitted the system's default timezone will be used. is a string type parameter, for example UTC. is an interval parameter: where supports s, m, h, d, w. It doesn't yet support M, q, y. For example, tumble(my_stream, 5s). More concrete examples: The above example SQL continuously aggregates max cpu usage per device per tumble window for the stream devices_utils. Every time a window is closed, Timeplus Proton emits the aggregation results. Let's change tumble(stream, 5s) to tumble(stream, timestmap, 5s) : Same as the above delayed tumble window aggregation, except in this query, user specifies a specific time column timestamp for tumble windowing. The example below is so called processing time processing which uses wall clock time to assign windows. Timeplus internally processes now/now64 in a streaming way. ### Hop​ Like Tumble, Hop also slices the unbounded streaming data into smaller windows, and it has an additional sliding step. Hop window is a more generalized window compared to tumble window. Hop window has an additional parameter called which means window progresses this slide size every time. There are 3 cases: 1. is less than . Hop windows have overlaps meaning an event can fall into several hop windows. 2. is equal to . Degenerated to a tumble window. 3. is greater than . Windows has a gap in between. Usually not useful, hence not supported so far. Please note, at this point, you need to use the same time unit in and , for example hop(device_utils, 1s, 60s) instead of hop(device_utils, 1s, 1m). Here is one hop window example which has 2 seconds slide and 5 seconds hop window. Except that the hop window can have overlaps, other semantics are identical to the tumble window. The above example SQL continuously aggregates max cpu usage per device per hop window for stream device_utils. Every time a window is closed, Timeplus emits the aggregation results. ### Session​ This is similar to tumble and hop window. Please check the session function. - Tumble - Hop - Session ## System-stream-metric-log - [system.stream_metric_log | Timeplus](https://docs.timeplus.com/system-stream-metric-log): You can query the system.streammetriclog stream to check the performance and usage statistics. - Monitoring & Troubleshooting - system.stream_metric_log # system.stream_metric_log You can query the system.stream_metric_log stream to check the performance and usage statistics. ## Schema​ This system stream is provisioned by Timeplus and cannot be modified. We first introduced this stream in Timeplus Enterprise 2.6. Based on user feedback and performance optimization, we have updated the schema in Timeplus Enterprise 2.7. If you upgrade from 2.6 to 2.7, the system will automatically recreate the stream with the new schema. The previous metric log data will be dropped. Here is the schema definition with comments: Notes: - By default, every 5 seconds Timeplus collects the metrics and adds the data points to this stream. - Each collection represents the delta value since the last collection. - For Materialized View, the metrics are collected both for the view and the target stream. - System databases (such as system, information_schema) are excluded from the metrics. - The database and type columns use the low_cardinality data type optimization, which improves query performance for columns with a small number of distinct values. ## Examples​ ### Average Throughput in the Last 5 Minutes​ You can get the throughput of the read/write operations. For example, the following query will get the average read/write bytes per second and read/write rows per second for each stream in the last 5 minutes: ### Average Throughput for every 5 Minutes​ The following query will get the average read/write bytes per second and read/write rows per second for each stream in every 5 minutes: ### Daily Ingestion Volume​ The following query will get the daily ingestion volume for all streams: ### Metrics for a Specific Node​ The following query will get the total read/write bytes and rows for each node in the last minute: ### Performance by Storage Type​ This query shows metrics aggregated by storage type: - Schema - ExamplesAverage Throughput in the Last 5 MinutesAverage Throughput for every 5 MinutesDaily Ingestion VolumeMetrics for a Specific NodePerformance by Storage Type - Average Throughput in the Last 5 Minutes - Average Throughput for every 5 Minutes - Daily Ingestion Volume - Metrics for a Specific Node - Performance by Storage Type - Average Throughput in the Last 5 Minutes - Average Throughput for every 5 Minutes - Daily Ingestion Volume - Metrics for a Specific Node - Performance by Storage Type ## System-stream-state-log - [system.stream_state_log | Timeplus](https://docs.timeplus.com/system-stream-state-log): You can query the system.streamstatelog stream to check the state changes of database resources in Timeplus. This stream gives you full visibility of the state of the streams, materialized views, and other resources in Timeplus. - Monitoring & Troubleshooting - system.stream_state_log # system.stream_state_log You can query the system.stream_state_log stream to check the state changes of database resources in Timeplus. This stream gives you full visibility of the state of the streams, materialized views, and other resources in Timeplus. ## Schema​ This system stream is provisioned by Timeplus and cannot be modified. We first introduced this stream in Timeplus Enterprise 2.6. Based on user feedback and performance optimization, we have updated the schema in Timeplus Enterprise 2.7. If you upgrade from 2.6 to 2.7, the system will automatically recreate the stream with the new schema. The previous state log data will be dropped. Here is the schema definition with comments: By default, every 5 seconds Timeplus collects the states and add the data points to this stream. ## Categories of State​ Different types of resources in Timeplus have different states. Here are the categories of the state: ### Stream Storage​ You can get the current size of the stream on disk for both streaming storage and historical storage. For example, the following query will get all stream storage states: #### disk_size​ The disk size metric with dimensions: - log_store: The size of the logstore on disk. - historical_store: The size of the historical store on disk. ### Stream Sequence Number​ The sequence number of the stream. It's used to track the progress of the stream. For example, the following query will compare the sequence number of the streams: #### applied_sn​ The sequence number of the stream that has been applied. The shard number is stored in the dimension field. #### committed_sn​ The sequence number of the stream that has been committed. The shard number is stored in the dimension field. #### start_sn​ The starting sequence number of the stream or materialized view source. The source information is stored in the dimension field. #### end_sn​ The ending sequence number of the stream or materialized view source. The source information is stored in the dimension field. #### processed_sn​ Last processed sequence number for the stream or materialized view source. The source information is stored in the dimension field. ### Materialized View​ #### checkpoint_storage_size​ The size of the checkpoint storage on disk. The dimension is set to materialized_view. #### status​ The status of the materialized view. Read the state_string_value column for the status. If the materialized view is running healthy, the status will be ExecutingPipeline. The dimension is set to materialized_view. #### last_error_message​ The last error message of the materialized view. Read the state_string_value column for the error message. The timestamp of the error is stored in the state_value field. The dimension is set to materialized_view. #### recover_times​ The number of times the materialized view has been recovered. The dimension is set to materialized_view. #### memory_usage​ The memory usage of the materialized view. The dimension is set to materialized_view. #### processed_record_ts​ The timestamp of the last processed record for the materialized view source. The source information is stored in the dimension field. #### ckpt_sn​ Last checkpoint sequence number for the materialized view source. The source information is stored in the dimension field. ### Ingestion Performance​ #### ingest​ The number of dropped messages during ingestion. The dimension is set to dropped. #### ingest_latency​ The number of messages that are ingested with different latency ranges, indicated by the following dimensions: - lt_500ms: Within 500ms - 500ms_1s: Between 500ms and 1s - 1_3s: Between 1s and 3s - 3_6s: Between 3s and 6s - gt_6s: Over 6s Example query: ### External Stream​ #### read_failed​ The number of failed read operations from the external stream. The dimension is set to external_stream. #### written_failed​ The number of failed write operations to the external stream. The dimension is set to external_stream. ### Dictionary​ Dictionary metrics all use the dimension value dict: #### bytes_allocated​ The number of bytes allocated in the memory for the dictionary. #### hierarchical_index_bytes_allocated​ The number of bytes allocated in the memory for the hierarchical index. #### query_count​ The number of queries executed with the dictionary. #### hit_rate_pct​ The hit rate percentage of the dictionary. #### found_rate_pct​ The found rate percentage of the dictionary. #### element_count​ The number of elements in the dictionary. #### load_factor_pct​ The load factor percentage of the dictionary. #### loading_start_time​ The start time of the loading process as milliseconds since epoch. #### last_successful_update_time​ The last successful update time of the dictionary as milliseconds since epoch. #### loading_duration_ms​ The duration of the loading process in milliseconds. #### last_exception​ The last exception message of the dictionary. Read the state_string_value column for the exception message. Example query for dictionary stats: ### Replication​ #### quorum_replication_status​ The status of the quorum replication. Read the state_string_value column for the status. The shard ID is now stored in the dimension field. For example, the following query will get the quorum replication status: A sample message of the quorum replication status is: You can further extract the fields from the JSON message. For example: - Schema - Categories of StateStream Storagedisk_sizeStream Sequence Numberapplied_sncommitted_snstart_snend_snprocessed_snMaterialized Viewcheckpoint_storage_sizestatuslast_error_messagerecover_timesmemory_usageprocessed_record_tsckpt_snIngestion Performanceingestingest_latencyExternal Streamread_failedwritten_failedDictionarybytes_allocatedhierarchical_index_bytes_allocatedquery_counthit_rate_pctfound_rate_pctelement_countload_factor_pctloading_start_timelast_successful_update_timeloading_duration_mslast_exceptionReplicationquorum_replication_status - Stream Storagedisk_size - disk_size - Stream Sequence Numberapplied_sncommitted_snstart_snend_snprocessed_sn - applied_sn - committed_sn - start_sn - end_sn - processed_sn - Materialized Viewcheckpoint_storage_sizestatuslast_error_messagerecover_timesmemory_usageprocessed_record_tsckpt_sn - checkpoint_storage_size - status - last_error_message - recover_times - memory_usage - processed_record_ts - ckpt_sn - Ingestion Performanceingestingest_latency - ingest - ingest_latency - External Streamread_failedwritten_failed - read_failed - written_failed - Dictionarybytes_allocatedhierarchical_index_bytes_allocatedquery_counthit_rate_pctfound_rate_pctelement_countload_factor_pctloading_start_timelast_successful_update_timeloading_duration_mslast_exception - bytes_allocated - hierarchical_index_bytes_allocated - query_count - hit_rate_pct - found_rate_pct - element_count - load_factor_pct - loading_start_time - last_successful_update_time - loading_duration_ms - last_exception - Replicationquorum_replication_status - quorum_replication_status - Stream Storagedisk_size - disk_size - Stream Sequence Numberapplied_sncommitted_snstart_snend_snprocessed_sn - applied_sn - committed_sn - start_sn - end_sn - processed_sn - Materialized Viewcheckpoint_storage_sizestatuslast_error_messagerecover_timesmemory_usageprocessed_record_tsckpt_sn - checkpoint_storage_size - status - last_error_message - recover_times - memory_usage - processed_record_ts - ckpt_sn - Ingestion Performanceingestingest_latency - ingest - ingest_latency - External Streamread_failedwritten_failed - read_failed - written_failed - Dictionarybytes_allocatedhierarchical_index_bytes_allocatedquery_counthit_rate_pctfound_rate_pctelement_countload_factor_pctloading_start_timelast_successful_update_timeloading_duration_mslast_exception - bytes_allocated - hierarchical_index_bytes_allocated - query_count - hit_rate_pct - found_rate_pct - element_count - load_factor_pct - loading_start_time - last_successful_update_time - loading_duration_ms - last_exception - Replicationquorum_replication_status - quorum_replication_status - disk_size - applied_sn - committed_sn - start_sn - end_sn - processed_sn - checkpoint_storage_size - status - last_error_message - recover_times - memory_usage - processed_record_ts - ckpt_sn - ingest - ingest_latency - read_failed - written_failed - bytes_allocated - hierarchical_index_bytes_allocated - query_count - hit_rate_pct - found_rate_pct - element_count - load_factor_pct - loading_start_time - last_successful_update_time - loading_duration_ms - last_exception - quorum_replication_status ## System-views - [Views in system namespace | Timeplus](https://docs.timeplus.com/system-views): Timeplus provides system views that enable effective troubleshooting and monitoring of your streaming data operations. - Monitoring & Troubleshooting - Views in system namespace # Views in system namespace Timeplus provides system views that enable effective troubleshooting and monitoring of your streaming data operations. ## v_failed_mat_views​ ## v_mat_view_lags​ ## v_no_leader_shards​ ## v_replication_lags​ ## v_shard_leaders​ ## v_under_replication_replicas​ - v_failed_mat_views - v_mat_view_lags - v_no_leader_shards - v_replication_lags - v_shard_leaders - v_under_replication_replicas ## Terraform - [Terraform Provider | Timeplus](https://docs.timeplus.com/terraform): HashiCorp Terraform is an Infrastructure as Code tool that lets you define both cloud and on-prem resources in human-readable configuration files. # Terraform Provider HashiCorp Terraform is an Infrastructure as Code tool that lets you define both cloud and on-prem resources in human-readable configuration files. The Timeplus Terraform Provider allows businesses to: - Reduce complexity and risk with infrastructure managed as code and deployed through automated GitOps integration. - Increase developer autonomy and productivity with consistent, version-controlled access to data streams, Apache Kafka sources, materialized views, dashboards, UDF, and more. With this Provider, Timeplus customers at any scale can safely accelerate data streaming initiatives in the cloud (public, private, or hybrid) with infrastructure management that is fully automated through code and integrated within continuous delivery workflows. Starting from v0.1.6, the Terraform Provider works for Timeplus Enterprise self-hosting. ## Why not just SQL script?​ At its core, Timeplus is a SQL database that works well for both streaming and batch/OLAP. So you may wonder why we developed this Terraform Provider for our users, instead of encouraging them to use a set of SQL statements, or SQL script, to create, update, or delete resources. The short answer is that SQL script works well for Timeplus, while a Terraform-based workflow can work well for certain teams. They are complementary to each other. A longer answer for why we built this Terraform Provider: 1. Terraform allows users to manage resources in a declarative way. As a data engineer or administrator, you describe the end state in the Terraform file, without handling the complexities of how to reach that end state. If using SQL or REST API instead, users have to figure out whether to create new resources, or change existing resources, or delete then recreate resources, etc. 2. Timeplus provides more than just streams and views. For example, it's hard to imagine defining and updating dashboard definitions (which is in JSON format) with SQL/dbt scripts. 3. Regular SQL scripts don't manage the dependencies for you. You have to write a set of CREATE.. IF NOT EXISTS, or DROP .. CREATE ... When you just want to make a small change to your SQL, you may end up recreating every resource and that can lead to data loss or inconsistency. 4. Data engineers and platform engineers are the main target users for Timeplus. In the modern software development process, it’s a common requirement to track changes in version control systems, with branching, code review, and CICD. With the Terraform-based change management process, engineers can create a branch, make code changes, and ask for code review. At the same time, Terraform can generate the deployment plan for the code maintainer and reviewer to better understand which resources will be created, updated, or deleted, in addition to reviewing the code change itself. If the reviewer is okay with the code change and deployment plan, they can approve the code and deploy the exact change to the target environment, without any guess work. ## Which resources are available?​ Resources are the most important element in the Terraform language. A Terraform resource describes one or more infrastructure objects, such as streams, sources, sinks and long-running materialized views. You can manage the following Timeplus resources in Terraform: - Streams: Timeplus streams are similar to tables in the traditional SQL databases. Timeplus streams can be append-only, changelog or versioned streams. - Sources: Timeplus sources are processes run in background to collect data from specific data sources (such as Apache Kafka) and ingest them into streams. - Views: Timeplus views are named queries. It is useful for wrapping a commonly used complex query as a view. - Materialized Views: Timeplus materialized views are special views that persist its data. Once created, a materialized view will keep running in the background and continuously writes the query results to the underlying storage system. - Sinks: Timeplus sinks run queries in background and send query results to the target system continuously. - Alerts: Alerts are like sinks, they are used to send data to external systems. How alerts are different is that alerts have two statuses: 'triggered' and 'resolved'. - Dashboards: A dashboard is a set of one or more panels organized and arranged in one web page. A variety of panels are supported to make it easy to construct the visualization components so that you can create the dashboards for specific monitoring and analytics needs. - Remote UDF: Timeplus remote functions are one of the supported user defined function types. Remote functions allow users to register a HTTP webhook as a function which can be called in queries. - JavaScript UDF: Timeplus JavaScript functions are one of the supported user defined function types. JavaScript functions allow users to implement functions with the javascript programming language, and be called in queries. An example of Terraform script to define a stream and a Kafka source is ## Getting started​ Now that you’ve learned a bit about the Timeplus Terraform provider, let’s get started and put it to work. ### Prerequisites​ You’ll need these prerequisites to follow along: 1. A Timeplus Enterprise self-hosting deployment. 2. Terraform (1.0+) installed. 3. Golang (1.20.0+) installed. ### Create an account​ For self-hosting deployments, create a user with password and required permissions. ### Set up Terraform configuration​ To use the provider, simply add it to your terraform file, for example: Then you can start provisioning Timeplus resources, and below is an example of stream: Follow the Terraform documentations to build, change and destroy resources, e.g. - terraform init to download the plugin - terraform apply to review and approve the changes - terraform destroy to delete the resources - Why not just SQL script? - Which resources are available? - Getting startedPrerequisitesCreate an accountSet up Terraform configuration - Prerequisites - Create an account - Set up Terraform configuration - Prerequisites - Create an account - Set up Terraform configuration ## Tiered-storage - [Tiered Storage | Timeplus](https://docs.timeplus.com/tiered-storage): Since Timeplus Enterprise 2.8, we have introduced a new feature called Tiered Storage. This feature allows users to store data in a mix of local and remote storage. For example, users can store hot data in a local high-performance storage(e.g. NVMe SSD) for quick access and move the data to object storage(e.g. S3) for long-term retention. - Core Features - Tiered Storage # Tiered Storage Since Timeplus Enterprise 2.8, we have introduced a new feature called Tiered Storage. This feature allows users to store data in a mix of local and remote storage. For example, users can store hot data in a local high-performance storage(e.g. NVMe SSD) for quick access and move the data to object storage(e.g. S3) for long-term retention. To configure Tiered Storage, users need to define a storage policy that specifies the storage tiers and their priorities. The policy can be applied to one or more specific streams. ## Create a S3 Disk Storage​ By default, Timeplus only created a default disk for local storage. To create a S3 disk storage, users need to create a new disk storage with the following command: The type needs to be s3 to create a S3 disk storage. Timeplus also supports disk with s3_plain but that is for S3-based materialized view checkpointing. Please refer to S3 External Table for how to connect to the S3 storage. It's not recommended to hardcode the access key and secret access key in the DDL. Instead, users should use environment variables or IAM role to secure these credentials. You can use the following SQL to list the disks: ## Create a Storage Policy​ You can use SQL to define a storage policy. For example, the following policy will create a storage policy hcs(hot-cold-storage) to use both the local disk and S3 disk storage. You can use the following SQL to list the storage policies: The content of the policy is a YAML document that is put into the $$ block. The schema of the policy is defined by the following YAML schema: ### max_data_part_size​ The value is in uint64. The maximum size of a part that can be stored on any of the volume's disks. If the a size of a merged part estimated to be bigger than max_data_part_size_bytes then this part will be written to a next volume. Basically this feature allows to keep new/small parts on a hot (SSD) volume and move them to a cold (HDD) volume when they reach large size. Do not use this setting if your policy has only one volume. ### move_factor​ The value is in float64. When the amount of available space gets lower than this factor, data automatically starts to move on the next volume if any (by default, 0.1). Timeplus sorts existing parts by size from largest to smallest (in descending order) and selects parts with the total size that is sufficient to meet the move_factor condition. If the total size of all parts is insufficient, all parts will be moved. ### perform_ttl_move_on_insert​ The value is in uint8. Default is 1 (enabled). Set to 0 to disable TTL move on data part INSERT. By default if we insert a data part that already expired by the TTL move rule it immediately goes to a volume/disk declared in move rule. Set this to 0 can improve the performance of INSERT operations, but newly inserted data won't be available in S3 immediately. ## Create a stream with the policy​ After configuring the S3 disk and storage policy, you can create a stream with the policy. This stream will keep the recent 7 days' historical data locally and move older data to the cold volume, which is in a remote S3 bucket defined in the storage policy hcs. The TTL expression only applies to the historical storage. For the data in the streaming storage, the retention policy is controlled by logstore_retention_bytes and logstore_retention_ms settings. - Create a S3 Disk Storage - Create a Storage Policymax_data_part_sizemove_factorperform_ttl_move_on_insert - max_data_part_size - move_factor - perform_ttl_move_on_insert - Create a stream with the policy - max_data_part_size - move_factor - perform_ttl_move_on_insert ## Timeplus-connect - [timeplus-connect (Python) | Timeplus](https://docs.timeplus.com/timeplus-connect): timeplus-connect is the recommended Python library to connect to Timeplus Proton and Timeplus Enterprise. It's a high performance database driver for connecting Timeplus to Python, Pandas, and Superset - Integrations - CLI, APIs & SDKs - timeplus-connect (Python) # timeplus-connect (Python) timeplus-connect is the recommended Python library to connect to Timeplus Proton and Timeplus Enterprise. It's a high performance database driver for connecting Timeplus to Python, Pandas, and Superset - Pandas DataFrames - Numpy Arrays - PyArrow Tables - Superset Connector - SQLAlchemy 1.3 and 1.4 (limited feature set) Timeplus Connect currently uses the Timeplus HTTP interface for maximum compatibility, defaulting to 8123 (batch query). You can also use 3218 port for streaming queries. The source code and latest releases are available on GitHub. ## Installation​ Timeplus Connect requires Python 3.9 or higher. ## Streaming Query​ The SQLAlchemy or DBAPI don't support streaming queries. You can use the low level HTTP client interface to perform streaming queries. ## SQLAlchemy Implementation​ Timeplus Connect incorporates a minimal SQLAlchemy implementation (without any ORM features) for compatibility with Superset. It has only been tested against SQLAlchemy versions 1.3.x and 1.4.x, and is unlikely to work with more complex SQLAlchemy applications. When creating a SQLAlchemy Data Source, use a SqlAlchemy DSN in the form timeplus://{username}:{password}@{host}:{port}, such as timeplus://default:password@localhost:8123. ## Superset Connectivity​ Timeplus Connect is fully integrated with Apache Superset. When creating a Superset Data Source, use a SqlAlchemy DSN in the form timeplus://{username}:{password}@{host}:{port}, such as timeplus://default:password@localhost:8123. - Installation - Streaming Query - SQLAlchemy Implementation - Superset Connectivity ## Timeplus-enterprise - [Page Moved | Timeplus](https://docs.timeplus.com/timeplus-enterprise): We've recently reorganized our documentation to provide you with better, more focused content. If you arrived here from a search engine or bookmark, the information you're looking for has likely moved to one of these updated pages: # Page Moved We've recently reorganized our documentation to provide you with better, more focused content. If you arrived here from a search engine or bookmark, the information you're looking for has likely moved to one of these updated pages: - Introduction to Timeplus Enterprise - Why Timeplus - Use cases - Comparing Timeplus Enterprise and Timeplus Proton - Deploying in Kubernetes or Bare Metal ## Need Help?​ If you can't find what you're looking for, please use the search function (press / or use the search box) or click the logo or any link in the top banner to access the updated navigation menu. - Need Help? ## Timeplus-external-stream - [Timeplus External Stream | Timeplus](https://docs.timeplus.com/timeplus-external-stream): In addition to Kafka External Stream and Pulsar External Stream, Timeplus also supports another type of external stream to read/write data from/to another Timeplus Enterprise or Timeplus Proton deployment. - Core Features - External Streams & Tables - External Streams - Remote Timeplus # Timeplus External Stream In addition to Kafka External Stream and Pulsar External Stream, Timeplus also supports another type of external stream to read/write data from/to another Timeplus Enterprise or Timeplus Proton deployment. ## Use Cases​ By introducing the external stream for Timeplus, you can implement many new use cases, such as: - Hybrid Deployment: you can deploy Timeplus Enterprise in both public cloud and private cloud, even on the edge servers. Using the Timeplus External Stream, you can run federation search from one Timeplus deployment to query data from the other deployment, without replicating the data. Alternatively, you can continuously send data from one deployment to the other deployment; or accumulate data at the edge servers and forward high value data to the cloud deployment when the edge server can connect to the servers in the cloud. - Data Migration or Upgrade: when you are ready to go production, you may use the Timeplus External Stream to transfer data from the staging cluster to the production cluster. Or if you need to upgrade Timeplus Enterprise across major releases, this type of external stream can help you to transfer data. ## Syntax​ Settings: - hosts: the IP or host for the remote timeplusd. You can set 10.1.2.3 or 10.1.2.3:8463. When you set a set of hosts with ports, e.g. host1:port1,host2:port2, this will treat each host as a shard. hosts is required and there is no default value. - db: the database name in the remote Timeplusd. The default value is 'default'. - user: the user name for the remote Timeplusd. The default value is 'default'. - password: the password for the remote Timeplusd. The default value is an empty string. - secure: a bool for whether to use secure connection to the remote Timeplusd. The default value is false. Use port 9440 when secure is set to true, otherwise use port 8463. - stream: the stream name in the remote Timeplusd. It's required and there is no default value. - config_file: since Timeplus Enterprise 2.7, you can specify a config file to load the settings from. Please follow the example in Kafka External Stream. ## Examples​ ### Migrate data from Timeplus Proton to Timeplus Enterprise​ If you have deployed Timeplus Proton and want to load those data to a Timeplus Enterprise deployment, you cannot upgrade in place. You can use the Timeplus External Stream to migrate data. The Timeplus Proton need to be 1.5.15 or above. For example, there is a stream streamA in Timeplus Proton, running on host1. In your Timeplus Enterprise, you can create the stream with the same name and same schema. Then use INSERT INTO .. SELECT to load all data from Timeplus Proton to Timeplus Enterprise. ### Upload data from edge server to the cloud​ If you deploy Timeplus Proton or Timeplus Enterprise at edge servers, it can collect and process live data with high performance and low footprint. The important data can be uploaded to the other Timeplus Enterprise in the cloud when the internet is available. For example, on the edge server, you collect the real-time web access log and only want to upload error logs to the server. ## Limitations​ - window functions like tumble/hop are not working yet. - can't read virtual columns on remote streams. - table function is not supported in timeplusd 2.3.21 or earlier version. This has been enhanced since timeplusd 2.3.22. - Timeplus Proton earlier than 1.6.9 doesn't support the Timeplus external stream. - In Timeplus Proton, if your materialized view queries a Timeplus external stream, the checkpoint of the external stream may not be properly persisted. No such issue for Timeplus Enterprise and we are working on the fix. - Use Cases - Syntax - ExamplesMigrate data from Timeplus Proton to Timeplus EnterpriseUpload data from edge server to the cloud - Migrate data from Timeplus Proton to Timeplus Enterprise - Upload data from edge server to the cloud - Limitations - Migrate data from Timeplus Proton to Timeplus Enterprise - Upload data from edge server to the cloud ## Timeplusd-client - [timeplusd-client | Timeplus](https://docs.timeplus.com/timeplusd-client): Timeplus Enterprise provides a command-line interface (CLI) to run SQL commands via timeplusd client. The timeplusd executable is available in the timeplus/bin folder of the bare metal packages and in the PATH of the timeplus/timeplusd Docker image. - Integrations - CLI, APIs & SDKs - timeplusd-client # timeplusd-client Timeplus Enterprise provides a command-line interface (CLI) to run SQL commands via timeplusd client. The timeplusd executable is available in the timeplus/bin folder of the bare metal packages and in the PATH of the timeplus/timeplusd Docker image. In the Docker/Kubernetes environments, you can also run timeplusd-client command, which will call timeplusd client command. Different client and server versions are compatible with one another, but some features may not be available in older clients. We recommend using the same version of the client as the server app. When you try to use a client of the older version, then the server, timeplusd client displays the message: ## Usage​ The client can be used in interactive and non-interactive (batch) mode. To use batch mode, specify the ‘query’ parameter, or send data to ‘stdin’ (it verifies that ‘stdin’ is not a terminal), or both. Similar to the HTTP interface, when using the ‘query’ parameter and sending data to ‘stdin’, the request is a concatenation of the ‘query’ parameter, a line feed, and the data in ‘stdin’. This is convenient for large INSERT queries. Example of using the client to insert data: In batch mode, the default data format is TabSeparated. You can set the format in the FORMAT clause of the query. By default, you can only process a single query in batch mode. To make multiple queries from a script, use the --multiquery parameter. This works for all queries except INSERT. Query results are output consecutively without additional separators. Similarly, to process a large number of queries, you can run ‘timeplusd client’ for each query. Note that it may take tens of milliseconds to launch the ‘timeplusd client’ program. In interactive mode, you get a command line where you can enter queries. If ‘multiline’ is not specified (the default): To run the query, press Enter. The semicolon is not necessary at the end of the query. To enter a multiline query, enter a backslash \ before the line feed. After you press Enter, you will be asked to enter the next line of the query. If multiline is specified: To run a query, end it with a semicolon and press Enter. If the semicolon was omitted at the end of the entered line, you will be asked to enter the next line of the query. Only a single query is run, so everything after the semicolon is ignored. You can specify \G instead of or after the semicolon. This indicates Vertical format. In this format, each value is printed on a separate line, which is convenient for wide tables. This unusual feature was added for compatibility with the MySQL CLI. The command line is based on ‘replxx’ (similar to ‘readline’). In other words, it uses the familiar keyboard shortcuts and keeps a history. The history is written to ~/.timeplusd-client-history. By default, the format used is PrettyCompact. You can change the format in the FORMAT clause of the query, or by specifying \G at the end of the query, using the --format or --vertical argument in the command line, or using the client configuration file. To exit the client, press Ctrl+D, or enter one of the following instead of a query: “exit”, “quit”, “logout”, “exit;”, “quit;”, “logout;”, “q”, “Q”, “:q”. When processing a query, the client shows: 1. Progress, which is updated no more than 10 times per second (by default). For quick queries, the progress might not have time to be displayed. 2. The formatted query after parsing, for debugging. 3. The result in the specified format. 4. The number of lines in the result, the time passed, and the average speed of query processing. You can cancel a long query by pressing Ctrl+C. However, you will still need to wait for a little for the server to abort the request. It is not possible to cancel a query at certain stages. If you do not wait and press Ctrl+C a second time, the client will exit. The command-line client allows passing external data (external temporary tables) for querying. For more information, see the section “External data for query processing”. ### Queries with Parameters​ You can create a query with parameters and pass values to them from client application. This allows to avoid formatting query with specific dynamic values on client side. For example: It is also possible to set parameters from within an interactive session: #### Query Syntax​ Format a query as usual, then place the values that you want to pass from the app parameters to the query in braces in the following format: - name — Placeholder identifier. In the console client it should be used in app parameters as --param_ = value. - data type — Data type of the app parameter value. For example, a data structure like (integer, ('string', integer)) can have the tuple(uint8, tuple(string, uint8)) data type (you can also use another integer types). It's also possible to pass table, database, column names as a parameter, in that case you would need to use identifier as a data type. #### Example​ ## Configuration​ You can pass parameters to timeplusd client (all parameters have a default value) using: - From the Command Line Command-line options override the default values and settings in configuration files. - Configuration files. Settings in the configuration files override the default values. From the Command Line Command-line options override the default values and settings in configuration files. Configuration files. Settings in the configuration files override the default values. ### Command Line Options​ - --host, -h – The server name, ‘localhost’ by default. You can use either the name or the IPv4 or IPv6 address. - --port – The port to connect to. Default value: 8463. Note that the HTTP interface and the native interface use different ports. - --user, -u – The username. Default value: default. - --password – The password. Default value: empty string. - --ask-password - Prompt the user to enter a password. - --query, -q – The query to process when using non-interactive mode. You must specify either query or queries-file option. - --queries-file – file path with queries to execute. You must specify either query or queries-file option. - --database, -d – Select the current default database. Default value: the current database from the server settings (‘default’ by default). - --multiline, -m – If specified, allow multiline queries (do not send the query on Enter). - --multiquery, -n – If specified, allow processing multiple queries separated by semicolons. - --format, -f – Use the specified default format to output the result. - --vertical, -E – If specified, use the Vertical format by default to output the result. This is the same as –format=Vertical. In this format, each value is printed on a separate line, which is helpful when displaying wide tables. - --time, -t – If specified, print the query execution time to ‘stderr’ in non-interactive mode. - --stacktrace – If specified, also print the stack trace if an exception occurs. - --config-file – The name of the configuration file. - --secure – If specified, will connect to server over secure connection (TLS). You might need to configure your CA certificates in the configuration file. (../operations/server-configuration-parameters/settings.md#server_configuration_parameters-openssl). - --history_file — Path to a file containing command history. - --param_ — Value for a query with parameters. - --hardware-utilization — Print hardware utilization information in progress bar. - --print-profile-events – Print ProfileEvents packets. - --profile-events-delay-ms – Delay between printing ProfileEvents packets (-1 - print only totals, 0 - print every single packet). ### Configuration Files​ timeplusd client uses the first existing file of the following: - Defined in the --config-file parameter. - ./timeplusd-client.xml - ~/.timeplusd-client/config.xml - /etc/timeplusd-client/config.xml Example of a config file: ### Query ID Format​ In interactive mode timeplusd client shows query ID for every query. By default, the ID is formatted like this: - UsageQueries with ParametersQuery SyntaxExample - Queries with ParametersQuery SyntaxExample - Query Syntax - Example - ConfigurationCommand Line OptionsConfiguration FilesQuery ID Format - Command Line Options - Configuration Files - Query ID Format - Queries with ParametersQuery SyntaxExample - Query Syntax - Example - Query Syntax - Example - Command Line Options - Configuration Files - Query ID Format ## Troubleshooting - [Troubleshooting Guide | Timeplus](https://docs.timeplus.com/troubleshooting): This guide provides solutions to common issues you may encounter when using Timeplus. - Monitoring & Troubleshooting - Troubleshooting Guide # Troubleshooting Guide This guide provides solutions to common issues you may encounter when using Timeplus. Please also query system.stream_state_log and system.stream_metric_log to check the state changes, metrics and errors of database resources in Timeplus. ## Memory Issues​ ### JOIN queries run out of memory​ Symptoms: Solution: Increase the JOIN buffer size for the query or materialized view. ### JavaScript UDF runs out of memory​ Symptoms: Solution: Increase the memory limit for the JavaScript UDF. ## Data Consistency​ ### Materialized view is not updating​ Symptoms: Solution: - Check the definition of the materialized view via SHOW CREATE . - Drop the materialized view and recreate it. ## Data Size and Ingestion​ ### TOO_LARGE_RECORD​ Symptoms: Solution: For ad-hoc stream ingestion, you can increase the max_insert_block_size setting to allow larger records. If you encounter this issue with a materialized view, you can increase the max_entry_size setting in timeplusd.yml. 1. Stop the Timeplus service. 2. Edit the data.datastore.log.max_entry_size setting in the timeplusd.yml configuration file. The default value is 10485760(10MB). You can set to 100MB, which is 104857600. 3. Start the Timeplus service. ### Fail to drop streams with large data​ Symptoms: Solution: You can drop with the force_drop_big_stream setting. ### DISK_USAGE_RATIO_THRESHOLD_EXCEEDED​ Symptoms: Solution: Please free up some disk space to allow ingestion to continue. ## Concurrency​ ### Too many concurrent queries​ Symptoms: Solution: Increase the max_concurrent_queries setting in timeplusd.yml. 1. Stop the Timeplus service. 2. Edit the max_concurrent_queries setting in the timeplusd.yml configuration file. The default value is 100. You can set to 200 or higher. 3. Start the Timeplus service. You can also set different values for max_concurrent_insert_queries and max_concurrent_select_queries. Increasing these limits may impact the performance and require more system resources. Please monitor the system performance after changing these settings. ## Kafka External Stream​ ### NO_AVAILABLE_KAFKA_CONSUMER​ Symptoms: Solution: Increase the max_consumers_per_stream setting in conf/timeplusd.yml. 1. Stop the Timeplus service. 2. Edit the external_stream.kafka.max_consumers_per_stream setting in the timeplusd.yml configuration file. The default value is 50. You can set to 100 or higher. 3. Restart the Timeplus service. - Memory IssuesJOIN queries run out of memoryJavaScript UDF runs out of memory - JOIN queries run out of memory - JavaScript UDF runs out of memory - Data ConsistencyMaterialized view is not updating - Materialized view is not updating - Data Size and IngestionTOO_LARGE_RECORDFail to drop streams with large dataDISK_USAGE_RATIO_THRESHOLD_EXCEEDED - TOO_LARGE_RECORD - Fail to drop streams with large data - DISK_USAGE_RATIO_THRESHOLD_EXCEEDED - ConcurrencyToo many concurrent queries - Too many concurrent queries - Kafka External StreamNO_AVAILABLE_KAFKA_CONSUMER - NO_AVAILABLE_KAFKA_CONSUMER - JOIN queries run out of memory - JavaScript UDF runs out of memory - Materialized view is not updating - TOO_LARGE_RECORD - Fail to drop streams with large data - DISK_USAGE_RATIO_THRESHOLD_EXCEEDED - Too many concurrent queries - NO_AVAILABLE_KAFKA_CONSUMER ## Tutorial-cdc-rpcn-pg-to-ch - [Streaming CDC: PostgreSQL to ClickHouse via Redpanda Connect | Timeplus](https://docs.timeplus.com/tutorial-cdc-rpcn-pg-to-ch): It's a common practice to utilize Debezium as the engine for implementing CDC, Change Data Capture. Then push the changes into Apache Kafka, and subsequently direct them to various destinations via Kafka Connect or Apache Flink. - Guides & Tutorials - Streaming CDC: PostgreSQL to ClickHouse via Redpanda Connect # Streaming CDC: PostgreSQL to ClickHouse via Redpanda Connect It's a common practice to utilize Debezium as the engine for implementing CDC, Change Data Capture. Then push the changes into Apache Kafka, and subsequently direct them to various destinations via Kafka Connect or Apache Flink. However, managing these JVM-based components can be challenging and costly, in terms of both hardware resources and human effort. With the new version of Redpanda Connect, as well as the built-in integration with Timeplus, now you can setup streaming CDC without the need for any Java components—delivering simplicity, power, and cost-efficiency. Redpanda Connect is a declarative data streaming service that solves a wide range of data engineering problems with simple, chained, stateless processing steps. Implemented in Golang, it's simple to deploy, and comes with a wide range of connectors. Since version 4.40, it supports streaming data changes from a PostgreSQL database using logical replication, with plans to extend support to MySQL and other databases. In this tutorial, we will configure Redpanda Connect to load existing data and real-time changes from a PostgreSQL database, send the CDC events to a Timeplus stream, then subsequently write to a ClickHouse table via a materialized view. ## Install Redpanda Connect​ There are various methods to install and setup Redpanda Connect. Follow the guide to setup rpk command line tool. When you first run rpk connect , Redpanda Connect will be automatically installed. ## Setup PostgreSQL and ClickHouse​ You can set up a PostgreSQL server locally or utilize cloud services. For this tutorial, we'll use the PostgreSQL service on Aiven cloud, with a table orders structured as follows: Similarly, you will create a table in a ClickHouse database: ## Redpanda CDC Pipeline​ Now create a connect.yml with the following content: Note: - The database connection is specified in the dsn You can store the database password as an environment variable and reference it using the ${VAR} syntax in the pipeline. - Use the schema and tables parameters to specify which tables to read. - Set stream_snapshot to true to read both existing and newly inserted rows. Insert some rows in your PostgreSQL orders table. Then run the pipeline using: You will receive all the existing rows in JSON format: If you add new rows in PostgreSQL now, they are shown immediately in the console. ## Setup Timeplus​ Now, modify the output from stdout to timeplus . You can install Timeplus Enterprise locally. Follow the guide to create an API key. Save it as an environment variable then update the pipeline accordingly: You will need to create several resources in Timeplus: ### Append-only stream​ The stream will receive data pushed from Redpanda Connect. ### ClickHouse External Table​ Create an external table to connect to the ClickHouse database: ### Materialized View​ Create a materialized view to synchronize data from the append-only stream to the ClickHouse external table: With everything in place, run rpk connect run connect.yml again and you will observe all data in PostgreSQL is now available in ClickHouse! ## What's Next​ This tutorial illustrates how to setup streaming CDC and optional ETL from PostgreSQL to ClickHouse. Additionally, you can retain the PostgreSQL data in Timeplus, serving as a query layer. Utilize Mutable Streams and Changelog Streams in Timeplus to manage UPSERT and DELETE operations: - Mutable Streams take 1 or more columns as the primary key. If you insert data with the same primary key(s), the original data will be overwritten. You can use the above pipeline to write to a mutable stream. - Changelog Streams take the _tp_delta column , where a value of 1 indicates data insertion and -1 indicates data deletion. You can create a mapping processor in the Redpanda Connect pipeline to generate the proper _tp_delta, e.g. This demonstration provides a robust foundation for implementing efficient and scalable data streaming solutions without the complexities associated with traditional Java-based components. - Install Redpanda Connect - Setup PostgreSQL and ClickHouse - Redpanda CDC Pipeline - Setup TimeplusAppend-only streamClickHouse External TableMaterialized View - Append-only stream - ClickHouse External Table - Materialized View - What's Next - Append-only stream - ClickHouse External Table - Materialized View ## Tutorial-github - [Query Kafka with SQL (GitHub live data) | Timeplus](https://docs.timeplus.com/tutorial-github): In this tutorial, you will process real-time data from GitHub. We have setup a public accessible Kafka cluster for you to consume data from Kafka topic. If you are on Timeplus Enterprise, you can also build real-time dashboards and alerts. - Guides & Tutorials - Query Kafka with SQL (GitHub live data) # Query Kafka with SQL (GitHub live data) In this tutorial, you will process real-time data from GitHub. We have setup a public accessible Kafka cluster for you to consume data from Kafka topic. If you are on Timeplus Enterprise, you can also build real-time dashboards and alerts. ## Read Live GitHub Events​ We all love GitHub. But do you know what’s trending on Github right now? Do you know which repos have received the most pushes or PR reviews over the past 10 minutes? There are daily/weekly leaderboards at https://github.com/trending, but no real-time feeds. You can write a script to call GitHub Events API with a dedicated Personal Access Token Please note, the public events from the GitHub API has a 5-minute delay (source). Here is a sample Python script for your reference. But we have made the live data accessible via Kafka API. In Timeplus, you read data from Kafka via an External Stream. Here is the SQL to create such an external stream to read from our Kafka clusters on Aiven: Just run this SQL via proton client or the SQL Console in Timeplus web UI. This Kafka user is configured with read-only access to the topic/cluster. We may change the password. Please come back if the password doesn't work. ## Sample Streaming SQL​ ### Streaming Tail​ You can explore the live data via This is a streaming SQL and keeps reading new events in the Kafka topic. You need to manually cancel the query to terminate it. ### Streaming Filter​ Add some condition in the WHERE clause to apply streaming filters, e.g. ### Aggregation​ #### Global Aggregation​ This will show how many new events received, since the query is started. So you may see a number like 158, then a couple seconds later, 334. This is so-called Global Aggregation. #### Tumble Aggregation​ This query counts events by repo every 30 seconds. Tumble windows are fixed windows, without overlaps. 30s is the shortcut for SQL expression INTERVAL 30 SECOND. You can also use 2m for 2 minutes and 3h for 3 hours. Please note, this query will wait for up to 30s to show the first results. Because by default, streaming SQL in Timeplus will look for future events, not existing events. We will talk about how to get past data shortly. #### Hopping Aggregation​ This query counts events by repo every 30 seconds and update results every second. Hop window is also called as sliding windows. ## Time Rewind​ By default, streaming SQL in Timeplus will look for future events, not existing events. For externals streams, you can use SETTINGS seek_to='..' to go back to a past timestamp or offset in the Kafka topic. For example, if you want to get total number of events since April 1, you can run: If you want to get data 6 hours ago: ## Save Kafka data in Timeplus​ Using external streams to query data in Kafka won't consume any storage in Timeplus. In some cases, you may want to save the data in Timeplus, so that you can apply more sophisticated data processing, or avoid too many query load on Kafka, or want to set a small retention policy on Kafka but would keep more data in Timeplus. You can create a materialized view to save data in Timeplus, e.g. The materialized view is a long-running query to turn the streaming SQL results in its internal storage. You can also have the materialized view to write data to other streams, external streams, or external tables. This can build streaming pipelines. You can query data in the materialized view just like other streams, e.g. ## Learn More​ You can check this blog for more details. - Read Live GitHub Events - Sample Streaming SQLStreaming TailStreaming FilterAggregationGlobal AggregationTumble AggregationHopping Aggregation - Streaming Tail - Streaming Filter - AggregationGlobal AggregationTumble AggregationHopping Aggregation - Global Aggregation - Tumble Aggregation - Hopping Aggregation - Time Rewind - Save Kafka data in Timeplus - Learn More - Streaming Tail - Streaming Filter - AggregationGlobal AggregationTumble AggregationHopping Aggregation - Global Aggregation - Tumble Aggregation - Hopping Aggregation - Global Aggregation - Tumble Aggregation - Hopping Aggregation ## Tutorial-kv - [UPSERT with Versioned Stream | Timeplus](https://docs.timeplus.com/tutorial-kv): By default, streams in Timeplus are in append-only mode. When you create a stream with the mode versioned_kv, you can update the data with same primary key(s). When you query the stream with table function, only the latest version for the same primary key(s) will be shown. When you use this stream as "right-table" in a streaming JOIN with other streams, Timeplus will automatically choose the closest version. - Guides & Tutorials - UPSERT with Versioned Stream # UPSERT with Versioned Stream By default, streams in Timeplus are in append-only mode. When you create a stream with the mode versioned_kv, you can update the data with same primary key(s). When you query the stream with table function, only the latest version for the same primary key(s) will be shown. When you use this stream as "right-table" in a streaming JOIN with other streams, Timeplus will automatically choose the closest version. Versioned Stream and Mutable Stream support INSERT and UPDATE. To delete data with certain primary key(s), please create a Changelog Stream. A HOWTO video: ## Query Single Stream​ In this example, you create a stream dim_products in versioned_kv mode with the following columns: This stream can be created using UI wizard on Timeplus Enterprise. You can also create it with SQL in Timeplus Proton: If you don't add any data, query SELECT * FROM dim_products will return no results and keep waiting for the new results. Now cancel this query, and add a few more rows to the stream. Running SELECT * FROM dim_products again will get those 2 rows. Now if you add one more row: Then query SELECT * FROM dim_products again will get 2 rows (not 3, because the initial price of "iPhone15" is overwritten). As you can imagine, you can keep adding new rows. If the primary key is new, then you will get a new row in the query result. If the primary key exists already, then the previous row is overwritten with the values in the newly-added row. In fact, you can assign an expression as the primary key. For example you can use first_name||' '||last_name to use the combined full name as the primary key, instead of using a single column. Or you can create a tuple as compound keys PRIMARY KEY (first_name,last_name) You can also query the stream in the table mode, i.e. select * from table(dim_products) ## Use Versioned Stream in INNER JOIN​ In the above example, you always get the latest version of the event with the same primary key. This works in the similar way as Changelog Stream. The reason why this stream mode is called Versioned Stream is that multiple versions will be tracked by Timeplus. This is mainly used when the Versioned Stream acts as the "right-table" for the JOIN. Imagine you have the other versioned stream for the orders: Now start a streaming SQL: Then add 2 rows: In the query console, you will see 2 rows one by one: Then you can change the price of iPhone15 back to 799, by adding a new row in dim_products Also add a new row in orders You will get the 3rd row in the previous streaming SQL: It shows that the latest price of iPhone15 is applied to the JOIN of new events. You can also run a streaming SQL select sum(price) from dim_products, it should show the number 1698, because the latest prices are 799 and 899. If you add a new row to set iPhone15 to 800, cancel the previous query and run again, you will get 1699. ## Use Versioned Stream in LEFT JOIN​ Since Proton 1.5.7, LEFT JOIN 2 versioned streams are also supported. For example, you run a streaming SQL: Then add a new product: Because there is no order for this product, you will get revenue 0 with the LEFT JOIN (if you are using INNER JOIN or just JOIN, this new product won't be counted). Adding a new order: The LEFT JOIN SQL will updated the result. ## Use Versioned Stream in ASOF JOIN​ The best part of Versioned Stream is that in ASOF JOIN Timeplus is able to automatically choose the closest version. Continue on our previous scenario. If the current iPhone15 price is 800, and you add a new order for 1 iPhone15, then you will get a transaction amount of 800. Then you change the iPhone15 price to 799, and add a new order for 1 iPhone15, you will get a transaction amount of 799. But if you add an order with _tp_time before the price change, you will get the transaction amount as 800 again, because Timeplus keeps multiple versions for the price and chooses the older version that best matches the order time. If you are not familiar with ASOF JOIN, this special JOIN provides non-exact matching capabilities. This can work well if two streams have the same id, but not exactly the same timestamps. - Query Single Stream - Use Versioned Stream in INNER JOIN - Use Versioned Stream in LEFT JOIN - Use Versioned Stream in ASOF JOIN ## Tutorial-python-udf - [Python UDF for Real-Time Machine Learning | Timeplus](https://docs.timeplus.com/tutorial-python-udf): SQL and Python are two of the most widely used tools in the data domain, both have their strengths and weaknesses, and understanding when to use each can help maximize efficiency and performance in data analysis, machine learning, and data transformation tasks. - Guides & Tutorials - Python UDF for Real-Time Machine Learning # Python UDF for Real-Time Machine Learning SQL and Python are two of the most widely used tools in the data domain, both have their strengths and weaknesses, and understanding when to use each can help maximize efficiency and performance in data analysis, machine learning, and data transformation tasks. Timeplus is a unified streaming SQL platform designed for high performance and efficiency, all within a single binary. Its primary interface is SQL, which comes with inherent limitations, as mentioned earlier. To address these constraints, Timeplus has recently introduced a new feature: Python user-defined functions (UDFs). With Python UDFs, users can seamlessly integrate Python functions into SQL queries, bridging the gap between SQL and Python. This enhancement enables developers and data engineers to leverage the strengths of both languages within a single platform, unlocking greater flexibility and capability. Please read the blog post for more details, or watch the demo video on YouTube. ## Tutorial-sql-connect-ch - [Connect to ClickHouse | Timeplus](https://docs.timeplus.com/tutorial-sql-connect-ch): You can define the external table and use it to read data from the ClickHouse table, or write to it. - Guides & Tutorials - Connect to ClickHouse # Connect to ClickHouse You can define the external table and use it to read data from the ClickHouse table, or write to it. ### Connect to a local ClickHouse​ Example SQL to connect to a local ClickHouse server without password: ### Connect to ClickHouse Cloud​ Example SQL to connect to ClickHouse Cloud: ### Connect to Aiven for ClickHouse​ Example SQL to connect to Aiven for ClickHouse: - Connect to a local ClickHouse - Connect to ClickHouse Cloud - Connect to Aiven for ClickHouse ## Tutorial-sql-connect-kafka - [Connect to Kafka, Confluent, Redpanda, Aiven, etc. | Timeplus](https://docs.timeplus.com/tutorial-sql-connect-kafka): Connect to local Kafka or Redpanda - Guides & Tutorials - Connect to Kafka, Confluent, Redpanda, Aiven, etc. # Connect to Kafka, Confluent, Redpanda, Aiven, etc. ### Connect to local Kafka or Redpanda​ Example: ### Connect to Confluent Cloud​ Example: ### Connect to Redpanda Cloud​ Example: ### Connect to Aiven for Apache Kafka​ You can connect Proton with an Aiven for Apache Kafka service. Example: Make sure the ssl_ca_cert_file can be accessed via Proton. You can do so via: Alternatively, you can put the full content of the CA pem file in the DDL SQL. This could help you to setup secure connections with Kafka brokers which use a certificate that is signed by a CA certificate that Proton does not know. You may want to put the CA content inline, when you cannot set a local file path or don't want to mount or modify files, such as using Docker or Kubernetes, or in Timeplus Enterprise. If you want to skip verifying the CA (not recommended), you can create the external stream in the following way: ### Connect to WarpStream​ You can connect Proton with local deployment of WarpStream or WarpStream Serverless Cloud. Example: sasl_mechanism can be either PLAIN or SCRAM-SHA-512, but cannot be SCRAM-SHA-256. - Connect to local Kafka or Redpanda - Connect to Confluent Cloud - Connect to Redpanda Cloud - Connect to Aiven for Apache Kafka - Connect to WarpStream ## Tutorial-sql-etl - [Streaming ETL: Kafka to Kafka | Timeplus](https://docs.timeplus.com/tutorial-sql-etl): You can quickly build streaming ETL pipelines with Timeplus. For example, the original web access logs in Kafka topics contain the raw IP address. To further protect user privacy, you can build a data pipeline to read new data from Kafka, masking the IP address and send to a different Kafka topic. - Guides & Tutorials - Streaming ETL - Streaming ETL: Kafka to Kafka # Streaming ETL: Kafka to Kafka You can quickly build streaming ETL pipelines with Timeplus. For example, the original web access logs in Kafka topics contain the raw IP address. To further protect user privacy, you can build a data pipeline to read new data from Kafka, masking the IP address and send to a different Kafka topic. Follow the guide for Timeplus Proton or Timeplus Enterprise. ## Timeplus Proton​ You can follow the previous tutorial to setup the sample data and run the following SQL to build the pipeline. ## Timeplus Enterprise​ A blog is published with the detailed steps to read data from Kafka/Redpanda, apply the transformation and send to Kafka/Redpanda. A few key steps: 1. Connect to Redpanda: 1. Specify the Redpanda broker address and the authentication method. 1. Choose a topic and preview data. 1. Set the name for the external stream, say frontend_events. 1. Explore the live data in the stream/topic. 1. Write a streaming SQL to transform data. 1. Send the result to another topic. Timeplus will create a new external stream as the destination and a materialized view as pipeline. The data lineage visualizes the relationships. 1. New data is available in Kafka/Redpanda topic. - Timeplus Proton - Timeplus Enterprise ## Tutorial-sql-etl-kafka-to-ch - [Streaming ETL: Kafka to ClickHouse | Timeplus](https://docs.timeplus.com/tutorial-sql-etl-kafka-to-ch): This video demonstrates how to read live data from Redpanda, apply stream processing and send results to ClickHouse. Related blog. - Guides & Tutorials - Streaming ETL - Streaming ETL: Kafka to ClickHouse # Streaming ETL: Kafka to ClickHouse This video demonstrates how to read live data from Redpanda, apply stream processing and send results to ClickHouse. Related blog. ## Demo Video​ ## Examples​ A Docker Compose stack is provided at https://github.com/timeplus-io/proton/tree/develop/examples/clickhouse together with the sample SQL statements. When you start the stack, the latest version of Proton and ClickHouse, as well as Redpanda and data generator will be automatically started. ### Example: ETL with masked data​ First, create a table with ClickHouse MergeTree table engine by running clickhouse client in the ClickHouse container. This will serve as the destination of Proton External Table for ClickHouse. Later on, you can also read the data in Timeplus. In the demo docker compose stack, a Redpanda container is started, together with a data generator and Redpanda Console for you to easily explore live data. For example, go to http://localhost:8080, you will see the live data in the owlshop-frontend-events topic. The goal of this tutorial is to read these access logs and turn the sensitive IP addresses into md5 and ingest them to ClickHouse for more business analysis. To read data from Kafka or Redpanda, you just need to create an Kafka External Stream with the following DDL SQL: Then run the following DDL SQL to setup the connection between Timeplus and ClickHouse. For local Clickhouse without security settings, it can be as simple as: Then create a materialized view to read data from Redpanda, extract the values and turn the IP to masked md5, and send data to the external table. By doing so, the transformed data will be written to ClickHouse continuously. Once the materialized view is created, it will work as a background ETL job in Proton, to continuously read data from Kafka/Redpanda, apply transformations or aggregations, then send results to ClickHouse. To learn more about Materialized View in Proton, please refer to this documentation. Now if you go back to ClickHouse and run select * from events, you will see new data coming at sub-second latency. You can do more with streaming SQL in Proton, such as late event processing, complex event processing, or leverage thousands of ClickHouse functions to customize the transformation/enrichment logics. Many of Proton’s functions are powered by ClickHouse. So if you are a ClickHouse user already, you can use Proton in a similar way. As mentioned above, the External Table in Proton can be used to read data from ClickHouse, or even apply data lookup in streaming JOIN. Simply run SELECT .. FROM external_table_name in Proton. It will read data from ClickHouse for the selected columns and apply the transformation or JOIN in Proton. ### Example: tumble + join​ A typical use case, if you have static or slowly changing dimensions (SCD) in ClickHouse, you don’t need to duplicate them in Proton. Just create an external table in Proton, and you can enrich your live data by JOIN the stream with such an external table, then send the high quality data to ClickHouse. For example: - Demo Video - ExamplesExample: ETL with masked dataExample: tumble + join - Example: ETL with masked data - Example: tumble + join - Example: ETL with masked data - Example: tumble + join ## Tutorial-sql-etl-mysql-to-ch - [Streaming ETL: MySQL to ClickHouse | Timeplus](https://docs.timeplus.com/tutorial-sql-etl-mysql-to-ch): A Docker Compose stack is provided at https://github.com/timeplus-io/proton/tree/develop/examples/cdc, to demonstrate how to mirror the data from MySQL to ClickHouse. The common use case is to keep MySQL as the transactional database, while using ClickHouse for the analytics workload. - Guides & Tutorials - Streaming ETL - Streaming ETL: MySQL to ClickHouse # Streaming ETL: MySQL to ClickHouse A Docker Compose stack is provided at https://github.com/timeplus-io/proton/tree/develop/examples/cdc, to demonstrate how to mirror the data from MySQL to ClickHouse. The common use case is to keep MySQL as the transactional database, while using ClickHouse for the analytics workload. ## Start the example​ Clone the proton repo or just download the docker-compose.yml in a folder. Run docker compose up in the folder. Six docker containers in the stack: 1. ghcr.io/timeplus-io/proton:latest, as the streaming database. 2. docker.redpanda.com/redpandadata/redpanda, as the Kafka compatible streaming message bus 3. docker.redpanda.com/redpandadata/console, as the web UI to explore data in Kafka/Redpanda 4. debezium/connect, as the CDC engine to read changes from OLTP and send data to Kafka/Redpanda 5. debezium/example-mysql, a pre-configured MySQL, as pipeline source 6. clickhouse/clickhouse-server, the real-time OLAP as the pipeline destination ## Prepare the table in ClickHouse​ Open the clickhouse client in the clickhouse container. Run the following SQL to create a regular MergeTree table. ## Create the CDC job​ Perform the following command in your host server, since port 8083 is exposed from Debezium Connect. ## Run SQL​ You can use docker exec -it proton-client -h 127.0.0.1 -m -n to run the SQL client in Proton container. Or use the Docker Desktop UI to choose the container, choose "Exec" tab and type proton-client -h 127.0.0.1 -m -n to start the SQL client. Copy the content of mysql-to-clickhouse.sqland paste in the Proton Client and run them together. What will happen: 1. One Timeplus External Stream will be created to read the MySQL CDC data from the Kafka/Redpanda topic. 2. One External Table will be created to write data from Timeplus to ClickHouse. 3. One Materialized View will be created to continuously read data from Kafka and write to ClickHouse. The content of the mysql-to-clickhouse.sql is: ## Verify the data is copied to ClickHouse​ Go back to the ClickHouse client and run select * from customers to see the 4 rows from MySQL. Use a MySQL client(e.g. DBeaver) to add some records to see the update from select * from customers. You can also run select * from table(customers) to avoid waiting for new updates. Since we create a regular MergeTree-based table in ClickHouse. It doesn't support update or delete. You may create the table with ReplacingMergeTree engine, if you need to change the data in MySQL. In this case, Timeplus needs to handle CDC data with op='u'. - Start the example - Prepare the table in ClickHouse - Create the CDC job - Run SQL - Verify the data is copied to ClickHouse ## Tutorial-sql-join - [Streaming JOIN | Timeplus](https://docs.timeplus.com/tutorial-sql-join): 1. This tutorial is mainly for Timeplus Proton users. For Timeplus Enterprise users, please check the guide for connecting Timeplus with Confluent Cloud with web UI. SQL in this guide can be ran both in Timeplus Proton and Timeplus Enterprise. - Guides & Tutorials - Streaming JOIN # Streaming JOIN 1. This tutorial is mainly for Timeplus Proton users. For Timeplus Enterprise users, please check the guide for connecting Timeplus with Confluent Cloud with web UI. SQL in this guide can be ran both in Timeplus Proton and Timeplus Enterprise. 2. Check the previous tutorial to setup the sample data. In the owlshop-customers topic, there are a list of customers with the following metadata - id - firstName - lastName - gender - email In the owlshop-addresses topic, it contains the detailed address for each customer - customer.id - street, state, city, zip - firstName, lastName You can create a streaming JOIN to validate the data in these 2 topics matches to each other. Note: - Two CTE are defined to parse the JSON attribute as columns - SETTINGS seek_to='earliest' is the special settings to fetch earliest data from the Kafka topic - USING(id) is same as ON left.id=right.id - Check JOIN for more options to join dynamic and static data By default, proton-client is started in single line and single query mode. To run multiple query statements together, start with the -n parameters, i.e. docker exec -it proton-container-name proton-client -n ## Tutorial-sql-kafka - [Query Kafka with SQL (mock data) | Timeplus](https://docs.timeplus.com/tutorial-sql-kafka): This tutorial is mainly for SQL users. For Timeplus Enterprise users, you can also use the UI wizard. SQL in this guide can be ran both in Timeplus Proton and Timeplus Enterprise. - Guides & Tutorials - Query Kafka with SQL (mock data) # Query Kafka with SQL (mock data) This tutorial is mainly for SQL users. For Timeplus Enterprise users, you can also use the UI wizard. SQL in this guide can be ran both in Timeplus Proton and Timeplus Enterprise. A docker-compose file is created to bundle proton image with Redpanda (as lightweight server with Kafka API), Redpanda Console, and owl-shop as sample live data. 1. Download the docker-compose.yml and put into a new folder. 2. Open a terminal and run docker compose up in this folder. 3. Wait for few minutes to pull all required images and start the containers. Visit http://localhost:8080 to use Redpanda Console to explore the topics and live data. 4. Use proton-client to run SQL to query such Kafka data: docker exec -it -proton-1 proton-client You can get the container name via docker ps 5. Create an external stream to connect to a topic in the Kafka/Redpanda server and run SQL to filter or aggregate data. ### Create an external stream​ Since Proton 1.3.24, you can also define multiple columns. Then select the columns directly, without JSON parsing, e.g. select method from frontend_events_json For nested data, you can select headers:referrer from frontend_events_json ### Explore the data in Kafka​ Then you can scan incoming events via There are about 10 rows in each second. Only one column raw with sample data as following: Cancel the query by pressing Ctrl+C. ### Get live count​ This query will show latest count every 2 seconds, without rescanning older data. This is a good example of incremental computation in Proton. ### Filter events by JSON attributes​ Once you start the query, any new event with method value as POST will be selected. raw:key is a shortcut to extract string value from the JSON document. It also supports nested structure, such as raw:headers.accept ### Aggregate data every second​ Every second, it will show the aggregation result for the number of events per HTTP method. ### Show a live ASCII bar chart​ Combining the interesting bar function from ClickHouse, you can use the following streaming SQL to visualize the top 5 HTTP methods per your clickstream. Note: - This is a global aggregation, emitting results every 2 seconds (configurable). - emit_version() function to show an auto-increasing number for each emit of streaming query result - limit 5 by emit_version() to get the first 5 rows with the same emit_version(). This is a special syntax in Proton. The regular limit 5 will cancel the entire SQL once 5 results are returned. But in this streaming SQL, we'd like to show 5 rows for each emit interval. ### Create a materialized view to save notable events in Proton​ With External Stream, you can query data in Kafka without saving the data in Proton. You can create a materialized view to selectively save some events, so that even the data in Kafka is removed, they remain available in Timeplus. For example, the following SQL will create a materialized view to save those broken links with parsed attributes from JSON, such as URL, method, referrer. Later on you can directly query on the materialized view: - Create an external stream - Explore the data in Kafka - Get live count - Filter events by JSON attributes - Aggregate data every second - Show a live ASCII bar chart - Create a materialized view to save notable events in Proton ## Tutorial-sql-read-avro - [Encode/Decode Avro Messages | Timeplus](https://docs.timeplus.com/tutorial-sql-read-avro): Example: Read Avro Encoded Data in Confluent Cloud - Guides & Tutorials - Encode/Decode Avro Messages # Encode/Decode Avro Messages ### Example: Read Avro Encoded Data in Confluent Cloud​ Say you create such a Avro schema definition: Create a topic in Confluent Cloud and you can push data to the topic in Avro format with the following command: You can add messages line by line, for example Now let's create an external stream in Proton to read such messages: After running this SQL successfully, you can fetch existing data via Or only fetch the incoming new messages via ### Example: Read Avro Encoded Data in Confluent Platform​ You can follow Confluent Docs to start Confluent Platform with Schema Registry via Docker Compose. The Avro schema definition: Follow the Schema Registry tutorial to create a new topic transactions. Create a $HOME/.confluent/java.config with content: Then use Maven to compile the sample code and produce Avro-encoded message to the local Kafka server with schema registry: Then create an external steam in Proton: After running this SQL successfully, you can fetch existing data via Or only fetch the incoming new messages via ### Example: Read Avro Encoded Data in Kafka service on Aiven​ The schema registry endpoint on Aiven is signed with CA, but you need to provide ssl_ca_cert_file for the broker. - Example: Read Avro Encoded Data in Confluent Cloud - Example: Read Avro Encoded Data in Confluent Platform - Example: Read Avro Encoded Data in Kafka service on Aiven ## Tutorial-testcontainers-java - [Automation with Testcontainers | Timeplus](https://docs.timeplus.com/tutorial-testcontainers-java): Testcontainers is a Java library that supports JUnit tests, providing lightweight, throwaway instances of common databases (including Timeplus), message brokers, or anything else that can run in a Docker container. You don't need to write docker-compose files, or call the Docker API, or mock the infrastructure by yourself. Simply run your test code, containers will be created and deleted automatically. - Guides & Tutorials - Automation with Testcontainers # Automation with Testcontainers Testcontainers is a Java library that supports JUnit tests, providing lightweight, throwaway instances of common databases (including Timeplus), message brokers, or anything else that can run in a Docker container. You don't need to write docker-compose files, or call the Docker API, or mock the infrastructure by yourself. Simply run your test code, containers will be created and deleted automatically. You can run any Docker images in Testcontainers. But starting from Testcontainers v1.20.2, Timeplus module is supported out of the box. You can start an instance of Timeplus via: Compared to a GenericContainer, you can call various methods on TimeplusContainer, such as setting the username, password, or run SQL queries with the default JDBC driver. In this tutorial, we will walk through how to setup Apache Kafka and Timeplus via Testcontainers for Java, create a few Kafka topics, generate data, and apply streaming ETL and routing, and finally tear down the services. With the GraalVM based native Kafka image and C++ based Timeplus, all of these operations can be done within 4 to 5 seconds. The source code of the tutorial is available at GitHub. The data generator and demo scenario are inspired by Confluent's blog: How to integration test a Kafka application with a native (non-JVM) Kafka binary with Testcontainers ## Set up the dependencies​ We will be using Gradle, a popular open source build system for Java ecosystems. In the build.gradle file, we load the required dependencies, namely: Please load 1.20.2 or newer version of Testcontainers Kafka and Timeplus modules. If you need to run SQL to query Timeplus in your test code, please load the Timeplus JDBC driver via com.timeplus:timeplus-native-jdbc:2.0.4 . ## Write test code​ You don't need to write any code in the src/main/java folder since this tutorial only focuses on test automation. Create a Java source code under src/test/java folder, and use org.junit.jupiter.api.Test annotation to mark one function as a test case. ### Start the Kafka container​ As one of the best practices, you can start the Kafka test container in a try block, so that the container will be teared down automatically. Please note, we create a network object and set a listener for the Kafka broker, so that our Timeplus instance can access to the Kafka broker without hardcoding the IP address. ### Create Kafka topics​ We will be using multiple Kafka topics. One topic for the input, literally input-topic . If the number is a prime, put it in the primes topic. If the number is not a prime, put it in the composites topic. If the data is not a number, which is not supposed to happen, put the data in the dlq topic (Dead Letter Queue), which is a common practice to handle dirty data or retries. There could be many ways to create Kafka topics, such as using Shell command or using tools like kcat. Since we are writing Java code, the easiest solution will be using Kafka AdminClient to create the topics programmatically. Since we are running the Kafka broker without authentication, you can retrieve the bootstrap URLs with Testcontainers kafka.getBootstrapServers() method, then use AdminClient to create 4 topics with 1 partition and 1 replica. ### Put 1..100 to the input topic​ For the happy path testing, we will generate numbers from 1 to 100 to the input topic. The number will be saved as a string as both key and value for the Kafka message. ### Put a dirty message in the input topic​ We will also put our favorite hello world string in the input topic to test our error handling. ### Start Timeplus container​ Now let's start the Timeplus container to process the data and validate the data pipeline. Please note 1. timeplus/timeplusd:2.3.31 is the latest build as the time of writing. Feel free to change it to a newer version, if available. 2. We attach the same network to the Timeplus container, so that they can access to each other. 3. withInitScript method is called to setup Timeplus timeplus/timeplusd:2.3.31 is the latest build as the time of writing. Feel free to change it to a newer version, if available. We attach the same network to the Timeplus container, so that they can access to each other. withInitScript method is called to setup Timeplus ### Define stream processing logic in Timeplus​ We will create a init.sql file under src/test/resources, so that the file is available at runtime classpath. #### Create Kafka external streams​ First we will create 4 Kafka External Streams so that we can read data and write data. Please note: 1. brokers are set to 'kafka:19092', since Kafka and Timeplus containers are running in the same docker network. 2. Kafka External Stream in Timeplus is bi-directional. You can read Kafka data by SELECT.. FROM or write data to Kafka via INSERT INTO or Materialized View target stream. 3. For primes and composites external streams, we also defined the _tp_message_key virtual column to write the message key. The default key value is set to the message value. brokers are set to 'kafka:19092', since Kafka and Timeplus containers are running in the same docker network. Kafka External Stream in Timeplus is bi-directional. You can read Kafka data by SELECT.. FROM or write data to Kafka via INSERT INTO or Materialized View target stream. For primes and composites external streams, we also defined the _tp_message_key virtual column to write the message key. The default key value is set to the message value. #### Create a JavaScript UDF to check prime numbers​ There is no built-in SQL functions to determine whether a number is a prime or not. This can be done via defining a JavaScript UDF in Timeplus, as following: Please note: 1. the input type is a int8 or a small int, since our test data will be from 1 to 100. 2. the return type is a bool, either true or false 3. To improve performance, Timeplus will batch the input values. That's why in the is_prime method, the values is an array of int8. You can check whether they are prime and put the result as an array of bool. 4. The algorithm of how to check the prime is based on the discussion on stackoverflow the input type is a int8 or a small int, since our test data will be from 1 to 100. the return type is a bool, either true or false To improve performance, Timeplus will batch the input values. That's why in the is_prime method, the values is an array of int8. You can check whether they are prime and put the result as an array of bool. The algorithm of how to check the prime is based on the discussion on stackoverflow #### Create materialized views to process data​ Materialized Views in Timeplus are long-running queries. It leverage the full power of streaming SQL reading from any number of sources, versus just acting on the block of data inserted into a source ClickHouse table. Fist let's handle the case if the input value is not a number, this can be done via the following SQL: Please note: 1. This materialized view reads data from input-topic via the input Kafka External Stream. We set _tp_time>earliest_ts() in the WHERE clause so that all existing messages in the Kafka topic will be read. 2. to_int8_or_zero(string) is a built-in SQL function to parse the string as a int8. If it fails, the function will return 0. Then we put the data into the dlq topic. 3. INTO dlq informs the materialized view to send results to the specific target, instead of using the materialized view internal storage. This materialized view reads data from input-topic via the input Kafka External Stream. We set _tp_time>earliest_ts() in the WHERE clause so that all existing messages in the Kafka topic will be read. to_int8_or_zero(string) is a built-in SQL function to parse the string as a int8. If it fails, the function will return 0. Then we put the data into the dlq topic. INTO dlq informs the materialized view to send results to the specific target, instead of using the materialized view internal storage. Similarly, we can define other Materialized Views to check whether the numbers are prime or not, then send them to correspondingly topics. The is_prime UDF is called after making sure the input value is a number. ### Validate the processing logic​ Now back to the test code. We will create a Kafka Java Consumer to read from both primes and composites topics, to make sure 1. Each message in primes topic is a prime, etc. 2. Totally 100 messages in both primesandcomposites topics. No more, no less. 3. In the Dead Letter Queue topic, there is a hello world message. Each message in primes topic is a prime, etc. Totally 100 messages in both primesandcomposites topics. No more, no less. In the Dead Letter Queue topic, there is a hello world message. We will call the buildConsumer to set up a Kafka Java Consumer with StringDeserializer, and use JUnit assert.. methods to validate the results. ### Tear down Kafka and Timeplus containers​ At the end of the test code, tear down Timeplus first then Kafka. ## Run the test​ Just run gradle test in the testcontainers folder, if you have the Gradle CLI configured. It's amazing that only 4 seconds are spent to start Kafka and Timeplus containers, create topics, send test data, setup & verify the data pipeline, and tear down everything. ... (content truncated) ## Udf - [User-Defined Functions | Timeplus](https://docs.timeplus.com/udf): At Timeplus, we leverage SQL to make powerful streaming analytics more accessible to a broad range of users. Without SQL, you have to learn and call low-level programming API, then compile/package/deploy them to get analytics results. This is a repetitive and tedious process, even for small changes. - SQL Reference - Functions - User Defined Functions # User-Defined Functions At Timeplus, we leverage SQL to make powerful streaming analytics more accessible to a broad range of users. Without SQL, you have to learn and call low-level programming API, then compile/package/deploy them to get analytics results. This is a repetitive and tedious process, even for small changes. But some developers have concerns that complex logic or systems integration are hard to express using SQL. That's why we add User-Defined Functions (UDF) support in Timeplus. This enables users to leverage existing programming libraries, integrate with external systems, or just make SQL easier to maintain. As of today, Timeplus supports four ways to develop/register UDF. - SQL UDF. Register a UDF from a lambda SQL expression. You can create common SQL snippets as SQL UDFs and reuse them in your queries. - Remote UDF. Register a webhook as the UDF. You may use any programming language/framework to develop/deploy the webhook. A good starting point is using AWS Lambda. - Local UDF in JavaScript. We also added the support of JavaScript-based local UDF. You can develop User-defined scalar functions (UDFs) or User-defined aggregate functions (UDAFs) with modern JavaScript (powered by V8). No need to deploy extra server/service for the UDF. - Local UDF in Python. Since Timeplus Enterprise 2.7, the Python UDF is supported. You can build high performance Python UDFs with the embedded CPython interpreter in Timeplus core engine. Additional libraries can be installed. Please choose the right UDF type based on your requirements. Please note, there are many factors to determine the number of function calls. For example, when you run a query, Timeplus query analyzer will dry-run the query first. During the query execution, a batch of data will be sent to the UDF, depending on how the data is organized. Long story short, developers should not make assumption for the number of function calls. For User-defined scalar functions (UDFs) it should be stateless, and for User-defined aggregate functions (UDAFs), data might be aggregated more than once, but the final result is correct. Check CREATE FUNCTION for how to create functions via SQL. ## Understanding-watermark - [Understanding Watermark | Timeplus](https://docs.timeplus.com/understanding-watermark): In stream processing, a watermark is a mechanism used to track the progress of data processing in one or more streams. It helps determine when all events up to a specific timestamp have likely arrived, enabling the system to process time-based operations like window aggregations. - Guides & Tutorials - Understanding Watermark # Understanding Watermark In stream processing, a watermark is a mechanism used to track the progress of data processing in one or more streams. It helps determine when all events up to a specific timestamp have likely arrived, enabling the system to process time-based operations like window aggregations. Since stream processing deals with unbounded, out-of-order data, watermarks are essential for defining bounded windows and for discarding late or outdated data in a consistent way. Watermarks allow the system to advance event-time, trigger computations (e.g., closing a tumbling window and emitting results), and maintain accuracy even when data arrives with delays. ## Try It Out​ To better understand how watermarks work, use the following tool. It generates random data and simulates watermark progression for a tumbling window aggregation with a 5-second window size and a 2-second watermark delay. You can click the Play button to start the simulation, or click the Next or Prev to view the next or previous step. ### Watermark & Window Processing A video is available to demonstrate this tool. - Try It Out ## Usecases - [More Query Examples | Timeplus](https://docs.timeplus.com/usecases): This document demonstrates some typical query patterns that you can achieve in Timeplus to solve various use cases. - Guides & Tutorials - More Query Examples # More Query Examples This document demonstrates some typical query patterns that you can achieve in Timeplus to solve various use cases. A docker-compose file is created to bundle proton image with a data generator. 1. Download the docker-compose.yml and put into a new folder. 2. Open a terminal and run docker compose up in this folder. 3. Wait for few minutes to pull all required images and start the containers. 4. Use proton-client to run SQL to query the data: docker exec -it -proton-1 proton-client You can get the container name via docker ps. ## Customer Scenario and Data Model​ You are the lead business analyst in a carsharing company. Sensors are equipped in each car to report car locations. The customers use the mobile app to find available cars nearby, book them, unlock them and hit the road. At the end of the trip, the customer parks the car, locks it, and ends the trip. The payment will proceed automatically with the registered credit card. Some of the typical use cases for time-sensitive insights are: - How many cars are being driven by users in certain locations? Do we need to move some cars from less busy locations to those hot zones? - Which cars are being driven too fast or running low on fuel? The service team may need to take action. - Which users keep booking cars then canceling them? Shall we send real-time notification to those users to avoid abuse. There are multiple data streams in the systems: ### dim_user_info​ A relative static stream with all registered user information. ### dim_car_info​ A relative static stream with all registered cars ### car_live_data​ A data stream with the latest data from car sensors. When the car engine is started, report data every second. Otherwise, report data every half an hour. ### bookings​ A data stream with trip details and payment info. Each row is generated during the booking lifecycle - when the user books the car, a new event with action=add, with the booking_time=now, expire=now+30m - when the user unlock the car, a new event with action=service - when the user completes the trip and lock the car, a new event with action=end - when the user cancels the booking, a new event with action=cancel - when the user extends the booking for another 30 min, a new event with action=extend, and update the expire field - if the user doesn't unlock the car before the expire time, then new event is added with action=expire ### trips​ A data stream with trip details and payment info. Each row is generated at the end of the trip The following sections show how to query Timeplus to understand the business. ## Streaming Analysis​ ### S-TAIL: Showing raw data with or without filter conditions​ Use Case: to start the data exploration, the analyst wants to show all recently reported car iot data or focusing on which cars are almost running out of gas (so that they can send service team to fill gas or suspend the car) Result: ### S-DOWNSAMPLING: Converting detailed data points to high level data​ Use Case: The sensors on each car may report data from half a second to every 10 seconds. The analyst may reduce the granularity and only need to save per-minute data to downstream Result: More practically, the user can create a materialized view to automatically put downsampled data into a new stream/view. Then the user can search the data via Result: ### S-AGG-RECENT: Showing aggregation for the recent data​ Use Case: the analyst wants to monitor the total revenue for the past 1 hour. Timeplus provides a special syntax to get such result easily Once the query is submitted, it will show quite a few rows based on the past day, then show new results in a streaming fashion. Result: There are other ways to get similar results, with more verbose queries 1. We can apply a global aggregation for data in a recent 1 hour window. select sum(amount) from trips where end_time > date_sub(now(), 1h) 2. The other solution is to use hop window aggregation. Similar to the tumble window in S-DOWNSAMPLING ,the data are grouped per a fixed size time window, such an hour. Tumble windows are not overlapped to each other, so it's ideal for downsampling without data duplication (for example, for count aggregation, no data will be counted twice) For hop window, it will be shifted to the left or right(past or future in the timeline) with a sliding step. For example, the following query will use the hop window to get total revenue for the past 1 hour, the result will be sent out every second. select window_start,window_end, sum(amount) from hop(trips,end_time,1s,1h) group by window_start,window_end We can apply a global aggregation for data in a recent 1 hour window. select sum(amount) from trips where end_time > date_sub(now(), 1h) The other solution is to use hop window aggregation. Similar to the tumble window in S-DOWNSAMPLING ,the data are grouped per a fixed size time window, such an hour. Tumble windows are not overlapped to each other, so it's ideal for downsampling without data duplication (for example, for count aggregation, no data will be counted twice) For hop window, it will be shifted to the left or right(past or future in the timeline) with a sliding step. For example, the following query will use the hop window to get total revenue for the past 1 hour, the result will be sent out every second. select window_start,window_end, sum(amount) from hop(trips,end_time,1s,1h) group by window_start,window_end ### S-SESSION: analyzing activities with active sessions​ Use Case: The analyst wants to track the daily movement of the cars. The sensors on the cars report data every second while the engine is started, and report data every half an hour when the engine is off. If the server doesn't receive the data for a running car for 5 seconds, the car is considered disconnected. We can run the following query to show the trip distances for each running cars Result: More complex queries can be created to aggregate the data by car id and trip ending time. Result: This query is a continuously streaming query. Every hour (or every day, depending on tumble window size), the analysis results can be sent to email/slack or a Kafka topic for further processing. ### S-TIME-TRAVEL: Going back to a past time and run analysis since then​ Use Case: the analysts don't need to keep watching the streaming charts or dashboards. They can rewind to a past time to run the streaming analysis since that moment. This could help them to better understand what happened a few hours ago (such as midnight). For example, the analyst wants to understand how the users book the car 2 hours ago Or they can specify an exactly timestamp, e.g. Result: Not only the past data will be analyzed, but also the latest incoming data will be processed continuously. ### S-MVIEW: Creating materialized view to keep latest analysis result and cache for other systems to query​ Use Case: Unlike the traditional SQL queries, streaming queries never end until the user cancels it. The analysis results are kept pushing to the web UI or slack/kafka destinations. The analysts want to run advanced streaming queries in Timeplus and cache the results as a materialized view. So that they can use regular SQL tools/systems to get the streaming insights as regular tables. Materialized views are also useful to downsample the data to reduce the data volume for future analysis and storage ### S-DROP-LATE: Dropping late events to get real-time aggregation insights​ Use Case: The streaming data may arrive late for many reasons, such as network latency, iot sensor malfunction, etc. When we run streaming analysis (such as payment per minute), we aggregate the data based on their event time (when the payment actually happened, instead of when Timeplus receives the data), and we don't want to wait for events which are significantly too late. Watermark is a common mechanism in the streaming processing world to set the bar on how late the events can be. Unlike other systems, Timeplus makes it very easy to identify late events without explicitly setting a watermark policy. For a query like this It will show the total payment every minute, for example Considering two cars are returned at the same time at 10:00:10. For tripA and trip, both of them are supposed to be calculated into the first time window. However, for some reason, the data point tripA arrives in Timeplus on 10:01:15, and tripB data point arrives on 10:01:16. Timeplus will accept tripA data and add it into the 1st window aggregation, and also close the first window. The watermark will be sent to 10:01:00. So when tripB data point arrives, it's considered to be too late and won't be calculated in the streaming result. But it'll still be available when we run a historical query. ### S-WAIT-LATE: Waiting for extra time for late events​ Use Case: for more advanced use for the above case, it might be desirable for the analysts to set an extra time to wait for late events. This will make the streaming analysis not so real-time, however can include more data points, if the network latency is unpredictable. Given the similar scenario, the query with the advanced setting is ### S-TOP-K: Getting the most common value for each streaming window​ Use Case: the analysts want to understand which cars are booked most often every day or every hour This will generate a daily report like this ### S-MAX-K: Getting the maximum value for each streaming window​ Use Case: the analysts want to understand which trips are longest every day This will generate a daily report like this To get the booking id for the 2nd longest trip, you can select ..,longest_trips[2].2 AS bookingId ### S-MIN-K: Getting the minimal value for each streaming window​ Use Case: the analysts want to understand which trips are shortest every day This will generate a daily report like this ... (content truncated) ## V1-release-notes - [Timeplus Cloud v1 and Proton | Timeplus](https://docs.timeplus.com/v1-release-notes): This page summarizes changes for each major update in Proton and Timeplus Cloud, including new features and important bug fixes. - Release Notes - Older Releases - Timeplus Cloud v1 and Proton # Timeplus Cloud v1 and Proton This page summarizes changes for each major update in Proton and Timeplus Cloud, including new features and important bug fixes. ## Jun 24, 2024​ Timeplus Core Engine (Proton v1.5.10): - Avro-Encoded Messages: Previously, Schema Registry must be enabled to read Avro-encoded messages. Now, SQL can be used to define the Avro schema and read these messages. Learn more - Improved Proton Client: -h 127.0.0.1 is no longer used when launching the proton-client. Proton v1.5.10 listens on both IPv4 and IPv6 ports. ## May 28, 2024​ Timeplus Core Engine (Proton v1.5.9): - You can now run table function on Kafka External Streams. This will read all existing data in the Kafka topic, e.g. select * from table(topic) where condition=true. - To get the number of messages in a Kafka topic, you can run select count() from table(topic). This will return the message count efficiently in less than 1 second, without scanning each message body. - A new advanced setting is available to control the speed of historical data backfill with a certain column. E.g. select * from test_stream where _tp_time > earliest_timestamp() settings replay_speed=1, replay_time_column='time_col'. - New SQL functions: parse_datetime and parse_datetime_in_joda_syntax. For example parse_datetime('2021-01-04+23:00:00', '%Y-%m-%d+%H:%i:%s') returns a value in datetime type, for '2021-01-04 23:00:00'. Timeplus Console: - The number of External Tables is now shown in the "Workspace at a Glance" section on the Homepage. - Refined the SQL editor's height adjustment and lock height behavior. - Workspace owners can now see who created API keys. ## May 13, 2024​ Timeplus Core Engine (Proton v1.5.8): - A new setting in config.yaml: max_consumers_per_stream defines how many consumers can be created for each Kafka external stream. When a SELECT query on a Kafka external stream is executed, a consumer will be created for that query. This means no more than max_consumers_per_stream SELECT queries can be executed in parallel on the same Kafka external stream. - New example in the Timeplus Proton repo /examples folder: Analyzing Nginx Access Logs, contributed by Timeplus Community member Saïd Abiola Timeplus Console: - Send us questions or comments via the new “Talk to Us” button in the top header. - "External Tables" resource list now available from the left navigation menu. - To see resource schema in the SQL editor, mouseover on the underlined resource and press Ctrl + click to open the details side panel. - For new charts, the default color will be pink (color scheme will be Dawn). - In charts, the legend will be hidden when the series count exceeds 30. ## Apr 29, 2024​ Timeplus Core Engine (Proton v1.5.7): - You can now join multiple versioned streams using LEFT JOIN and by assigning primary key(s). Results will be emitted whenever there are updates to either side of the JOIN. Learn more - New examples in the Timeplus Proton repo /examples folder: One Billion Rows Challenge (1BRC), contributed by Timeplus Community member Saïd Abiola Real-time retrieval-augmented generation (RAG) - One Billion Rows Challenge (1BRC), contributed by Timeplus Community member Saïd Abiola - Real-time retrieval-augmented generation (RAG) - One Billion Rows Challenge (1BRC), contributed by Timeplus Community member Saïd Abiola - Real-time retrieval-augmented generation (RAG) Timeplus Console: - External tables configuration: A new wizard is available to integrate with ClickHouse. - New "Save As" button on the Query Page: Save SQL as a view, materialized view, or bookmark. - In the SQL editor, click the name of the stream, view, or materialized view to show the resource's schema. - After creating a new source or external stream, you will now be redirected to the SQL Console page. - View and edit resource details: Removed "Edit" from high-level actions in resource lists – it's now available in the side panel after clicking the "View" icon. ## Apr 15, 2024​ Timeplus Core Engine (Proton v1.5.5 and v1.5.6): - Added two new Apache Kafka external stream settings for Kafka broker certifications: skip_ssl_cert_check allows you to skip verifying the server's certification ssl_ca_pem allows you to specify the Certificate Authority (CA) content, without needing to upload the CA file - skip_ssl_cert_check allows you to skip verifying the server's certification - ssl_ca_pem allows you to specify the Certificate Authority (CA) content, without needing to upload the CA file - Added functions for datetime: to_unix_timestamp64_milli returns the UNIX timestamp with millisecond of the datetime64 to_unix_timestamp64_micro returns with microsecond to_unix_timestamp64_nano returns with nanosecond - to_unix_timestamp64_milli returns the UNIX timestamp with millisecond of the datetime64 - to_unix_timestamp64_micro returns with microsecond - to_unix_timestamp64_nano returns with nanosecond - Newly created streams will have streaming data retention set to 1 day storage by default. - On MacOS, sudo proton install will install proton to /usr/local/bin instead of /usr/bin. - skip_ssl_cert_check allows you to skip verifying the server's certification - ssl_ca_pem allows you to specify the Certificate Authority (CA) content, without needing to upload the CA file - to_unix_timestamp64_milli returns the UNIX timestamp with millisecond of the datetime64 - to_unix_timestamp64_micro returns with microsecond - to_unix_timestamp64_nano returns with nanosecond Timeplus Cloud and Timeplus Enterprise: - Added workspace templates: you can create a new workspace from scratch, or with a demo template (Market Data or Stream Processing). - New layout for Data Ingestion page, with labels to show Source or External Stream. - In Data Lineage, we've optimized the layout for when there are large amounts of resources in the workspace. Additionally, zooming out will show the icon and color, and added a button to zoom out to show all. - Added TLS settings and Certificate Authority (CA) input field for Apache Kafka external streams and sinks. ## Apr 1, 2024​ Timeplus Proton: - Streaming processing now supports nullable data type. - External Table: ClickHouse external tables with names containing special characters (such as dashes) are now supported. Simply set table='test-a-b' in the CREATE EXTERNAL TABLE DDL. - External Stream: Error handling and connection pooling/retry for Kafka external streams have been greatly improved. - Materialized View: Added option to skip dirty/unexpected data. If you set SETTINGS recovery_policy='best_effort', Timeplus will try up to 3 times, then skip dirty data and continue processing the rest of the data. Timeplus Cloud and Timeplus Enterprise: - Added built-in support for Redpanda Serverless. See our how-to guide - Ingesting Apache Kafka data: We changed the implementation of Kafka integration from Benthos-based implementations to external stream-based. Our configuration wizard in the web console remains the same. - New left menu item for “Sources”. The “Data Ingestion” page will now only show options to ingest data, and not list added sources. - Based on user feedback, we enhanced the built-in OHLC chart to show live updates even when the current window of OHLC is not closed. - Updated layout for Home page, with new stats on existing resources, and number of currently running queries in the top right corner. If you don't have a certain resource in your workspace, we will now hide the stat box, instead of showing 0. - Enhancements to the 3 demos in our Timeplus Demo workspace. In the Stream Processing demo, we've added a dashboard to show how to calculate standard deviation with live data and streaming SQL, and use query variables. ## Mar 18, 2024​ Introducing three new demos with live data in our Timeplus Demo workspace: Stream Processing, Market Data, and ksqlDB Alternative. Choose a demo via the dropdown in the top header. You can explore the demos in read-only mode, or take a quick guided tour. Timeplus Cloud: - Data Definition Language (DDL) is supported in the SQL Console (previously called the Query page). You can modify resources using commands such as CREATE and DROP. - In Data Lineage, after clicking a resource tile, you can make edits to it directly from the details side panel. Please note that only certain fields may be edited for a particular resource type. - Also in Data Lineage, ClickHouse external tables are now included. Click the tile to see more details, such as the address and username. - In addition to streams, you can now set external streams and external tables as the target for materialized views. ## Mar 4, 2024​ Proton: - Proton can now natively integrate with ClickHouse, available for both ClickHouse Cloud or local/self-managed versions of ClickHouse. Learn more - Bulk CSV import is enhanced, in Proton 1.5.2. You can load billions of rows in multiple CSV files via a single SQL. - Kafka Schema Registry is supported with Protobuf and Avro format (Proton 1.5.2). Learn more - Self-signed HTTPS certification for Schema Registry is supported (Proton 1.5.3). - Proton now can be compiled on SUSE Linux. Timeplus Cloud: - In Data Lineage side panels, more details about the resources are added. Click on a tile to view. - Data types such as float or integer are depreciated. The UI will show precise data types such as int8, uint16, etc. - In dashboard charts, the legend is maintained when rendering. Click to show or hide series. ## Feb 20, 2024​ Proton: ... (content truncated) ## V2-release-notes - [Biweekly Updates | Timeplus](https://docs.timeplus.com/v2-release-notes): This page summarizes changes for Timeplus Enterprise and Timeplus Proton, on a biweekly basis, including new features and important bug fixes. - Release Notes - Biweekly Updates # Biweekly Updates This page summarizes changes for Timeplus Enterprise and Timeplus Proton, on a biweekly basis, including new features and important bug fixes. ## Jul 21, 2025​ ### Timeplus Enterprise v2.5.13​ - Timeplus Enterprise v2.5.13: bug fixes without new features ### Timeplus Enterprise v2.9 Preview 2​ Another preview edition of Timeplus Enterprise v2.9 with many new features. This is scheduled to be Generally Available (GA) by the end of August. ## Jun 9, 2025​ ### Timeplus Proton 1.6.16​ - fixed the issue for map_cast(array1,array2) ### Timeplus Enterprise v2.8 GA​ - 2.8.1 is the first GA version of Timeplus Enterprise v2.8, with the key features: New Compute Node server role to run materialized views elastically with checkpoints on S3 storage. Timeplus can read or write data in Apache Iceberg tables. Learn more Timeplus can read or write PostgreSQL tables directly via PostgreSQL External Table or look up data via dictionaries. Use S3 as the tiered storage for streams. New SQL command to rename streams or columns. JavaScript UDFs benefit from multiple V8 instances, improving concurrency and isolation. A new page to visualize nodes in a cluster. New page to view the details of streams or materialized views. - New Compute Node server role to run materialized views elastically with checkpoints on S3 storage. - Timeplus can read or write data in Apache Iceberg tables. Learn more - Timeplus can read or write PostgreSQL tables directly via PostgreSQL External Table or look up data via dictionaries. - Use S3 as the tiered storage for streams. - New SQL command to rename streams or columns. - JavaScript UDFs benefit from multiple V8 instances, improving concurrency and isolation. - A new page to visualize nodes in a cluster. - New page to view the details of streams or materialized views. - New Compute Node server role to run materialized views elastically with checkpoints on S3 storage. - Timeplus can read or write data in Apache Iceberg tables. Learn more - Timeplus can read or write PostgreSQL tables directly via PostgreSQL External Table or look up data via dictionaries. - Use S3 as the tiered storage for streams. - New SQL command to rename streams or columns. - JavaScript UDFs benefit from multiple V8 instances, improving concurrency and isolation. - A new page to visualize nodes in a cluster. - New page to view the details of streams or materialized views. ### Timeplus Enterprise v2.7.8, v2.7.9​ Patch releases with bug fixes. ### Timeplus Enterprise v2.9 Preview 1​ Another preview edition of Timeplus Enterprise v2.9 with many new features. This is scheduled to be Generally Available (GA) by the end of August. ### Timeplus Go Driver v2.1.2​ The proton-go-driver provides Go connector to interact with Timeplus Enterprise or Timeplus Proton. In this release: - simplify json marshaling and support selecting nested fields ## May 26, 2025​ ### Timeplus Native JDBC v2.0.10​ - Upgraded aircompressor to 0.27 to fix a security risk - Upgraded proton-jdbc from 0.6.0 to 0.7.1 ## May 12, 2025​ ### Timeplus Enterprise v2.7.6, 2.7.7​ - The EMIT Policies have been updated, for example EMIT AFTER WATERMARK has been changed to EMIT AFTER WINDOW CLOSE. - Patch releases with bug fixes. ### Timeplus Go Driver v2.1.1​ The proton-go-driver provides Go connector to interact with Timeplus Enterprise or Timeplus Proton. In this release: - fixed the issue of nil elements in map ## Apr 28, 2025​ ### Timeplus Proton v1.6.15​ - fixed the issue that meta store raft service is hardcoded to listen on ipv6 ### Timeplus Connect v0.8.17​ timeplus-connect provides Python connector to interact with Timeplus Enterprise or Timeplus Proton. In this release: - added support for new json data type - other bug fixes and enhancements ### Timeplus Go Driver v2.1.0​ The proton-go-driver provides Go connector to interact with Timeplus Enterprise or Timeplus Proton. In this release: - added support for new json data type - updated protocol to support receiving query id from the server - other refinements and sample code updates ## Apr 14, 2025​ ### Timeplus Enterprise v2.6.8, v2.7.4, v2.7.5​ Patch releases with bug fixes. ### Timeplus Connect v0.8.16​ timeplus-connect provides Python connector to interact with Timeplus Enterprise or Timeplus Proton. In this release: - fixed warning of dbapi() - supported vectorstore of llamaindex - improved sqlalchemy support ### marimo Python Notebook added support for Timeplus​ You can add connection to Timeplus Proton or Timeplus Enterprise in marimo Python notebook. ## Mar 31, 2025​ ### Timeplus Enterprise v2.8 (Preview)​ Timeplus Enterprise v2.8.0 is now available as a technical preview for the 2.8 release. Not ready for production use but feel free to try the new features and provide feedback. - New Compute Node server role to run materialized views elastically with checkpoints on S3 storage. - Timeplus can read or write data in Apache Iceberg tables. Learn more - Timeplus can read or write PostgreSQL tables directly via PostgreSQL External Table or look up data via dictionaries. - Use S3 as the tiered storage for streams. - New SQL command to rename streams. - A new page to visualize nodes in a cluster. - New page to view the details of streams or materialized views. ### Timeplus Enterprise v2.6.6, v2.6.7, v2.7.3​ Patch releases with bug fixes. ### Iceberg Integration Coming to Timeplus Proton​ - https://github.com/timeplus-io/proton/pull/928 a PR to add Apache Iceberg read/write support to Timeplus Proton. ## Mar 17, 2025​ ### Timeplus Enterprise v2.7.1 and v2.6.4​ Patch releases with bug fixes. ### Timeplus Proton v1.6.12 and v1.6.14​ - Support AWS_MSK_IAM authentication for accessing Amazon MSK - Allowing tuple datatype for the 1st param of array_map - New function group_array_last - Support format schema for Avro ## Mar 3, 2025​ ### Timeplus Enterprise v2.7.0​ Timeplus Enterprise v2.7 is now Generally Available! Key enhancements: - Built-in support for MySQL and S3 read and write - External data lookup with dictionaries - Python UDF - Monitoring and troubleshooting UI for materialized views and streams, optimized for multi-node clusters - Support IAM authentication for accessing Amazon MSK - Read the header key/value pairs for Kafka messages - PostgreSQL and MySQL CDC via Redpanda Connect - Mutable stream delete - In the web console, you can choose a non-default database and manage the streams and materialized views in that namespace. ### Timeplus Proton v1.6.11​ - Read the header key/value pairs for Kafka messages ### Timeplus Native JDBC v2.0.9​ Repackaged with JDK 8 support. ### Timeplus Connect v0.8.15​ timeplus-connect provides Python connector to interact with Timeplus Enterprise or Timeplus Proton. The code is based on clickhouse-connect. It uses HTTP port 8123 and supports SQLAlchemy and Superset. ## Feb 17, 2025​ ### Timeplus Enterprise v2.6.2​ We released a patch release for Timeplus Enterprise v2.6. The key changes are: - fix potential corruption for a stream when it's altered multiple times - better data recovery for file corruption due to power loss - set mutable streams' default logstore retention policy from keeping forever to automatic - fix issue where failed Materialized View creation attempts could block subsequent DDL operations under specific conditions in cluster If you are running Timeplus Enterprise v2.6, we recommend upgrading to this version. ### Timeplus Enterprise v2.4.26​ We released a patch release for Timeplus Enterprise v2.4. The key changes are: - gracefully handle unsupported metadata commands - improve garbage collection for NativeLog on clusters - fix a bug during versioned schema fetch for inner storage of materialized views If you are running Timeplus Enterprise v2.4, we recommend upgrading to this version. ### Timeplus Proton v1.6.10​ - enhanced the replay capability for streams and Kafka external streams. You can now replay data from a specific timestamp and end timestamp. Learn more. - added a new function group_concat to aggregate multiple values into a single string. ### Timeplus Cloud​ We're sunsetting Timeplus Cloud. Please migrate to self-hosted Timeplus Enterprise or Timeplus Proton. No new cloud workspace creation is allowed. You can still access the live demo at https://demo.timeplus.cloud. ### Timeplus Native JDBC v2.0.8​ Support named tuples, such as tuple(a int8, b string). This is useful when you want to return multiple columns in a function, such as dict_get. ### Timeplus Grafana plugin v2.1.2​ Support more transformations in Grafana query panel, such as Config from query results. ## Feb 3, 2025​ ### Timeplus Enterprise v2.4.25​ We released a patch release for Timeplus Enterprise v2.4. The key changes are: - fix potential corruption for a stream when it's altered multiple times - better data recovery for file corruption due to power loss - set mutable streams' default logstore retention policy from keeping forever to automatic If you are running Timeplus Enterprise v2.4, we recommend upgrading to this version. ### Timeplus Grafana plugin v2.1.1​ The new version of Grafana plugin improved the batching strategy to render results from streaming queries. If there is any error in the SQL syntax, the error message will be shown in the Grafana query panel. ## Jan 20, 2025​ ### Timeplus Enterprise v2.6​ Timeplus Enterprise v2.6 is now Generally Available! Key breakthroughs: ... (content truncated) ## Versioned-stream - [Versioned Key Value Stream | Timeplus](https://docs.timeplus.com/versioned-stream): When you create a stream with the mode versioned_kv, the data in the stream is no longer append-only. When you query the stream with table function, only the latest version for the same primary key(s) will be shown. When you use this stream as "right-table" in a streaming JOIN with other streams, Timeplus will automatically choose the closest version. - Core Features - Streams - Versioned Key Value Stream # Versioned Key Value Stream When you create a stream with the mode versioned_kv, the data in the stream is no longer append-only. When you query the stream with table function, only the latest version for the same primary key(s) will be shown. When you use this stream as "right-table" in a streaming JOIN with other streams, Timeplus will automatically choose the closest version. For Timeplus Enterprise customers, we recommend to use Mutable Streams with the enhanced performance for UPSERT and queries. A HOWTO video: ## Query Single Stream​ In this example, you create a stream dim_products in versioned_kv mode with the following columns: This stream can be created using UI wizard on Timeplus Enterprise. You can also create it with SQL in Timeplus Proton: If you don't add any data, query SELECT * FROM dim_products will return no results and keep waiting for the new results. Now cancel this query, and add a few more rows to the stream. Running SELECT * FROM dim_products again will get those 2 rows. Now if you add one more row: Then query SELECT * FROM dim_products again will get 2 rows (not 3, because the initial price of "iPhone15" is overwritten). As you can imagine, you can keep adding new rows. If the primary key is new, then you will get a new row in the query result. If the primary key exists already, then the previous row is overwritten with the values in the newly-added row. In fact, you can assign an expression as the primary key. For example you can use first_name||' '||last_name to use the combined full name as the primary key, instead of using a single column. Or you can create a tuple as compound keys PRIMARY KEY (first_name,last_name) You can also query the stream in the table mode, i.e. select * from table(dim_products) ## Use Versioned Stream in INNER JOIN​ In the above example, you always get the latest version of the event with the same primary key. This works in the similar way as Changelog Stream. The reason why this stream mode is called Versioned Stream is that multiple versions will be tracked by Timeplus. This is mainly used when the Versioned Stream acts as the "right-table" for the JOIN. Imagine you have the other versioned stream for the orders: Now start a streaming SQL: Then add 2 rows: In the query console, you will see 2 rows one by one: Then you can change the price of iPhone15 back to 799, by adding a new row in dim_products Also add a new row in orders You will get the 3rd row in the previous streaming SQL: It shows that the latest price of iPhone15 is applied to the JOIN of new events. You can also run a streaming SQL select sum(price) from dim_products, it should show the number 1698, because the latest prices are 799 and 899. If you add a new row to set iPhone15 to 800, cancel the previous query and run again, you will get 1699. ## Use Versioned Stream in LEFT JOIN​ Since Proton 1.5.7, LEFT JOIN 2 versioned streams are also supported. For example, you run a streaming SQL: Then add a new product: Because there is no order for this product, you will get revenue 0 with the LEFT JOIN (if you are using INNER JOIN or just JOIN, this new product won't be counted). Adding a new order: The LEFT JOIN SQL will updated the result. ## Use Versioned Stream in ASOF JOIN​ The best part of Versioned Stream is that in ASOF JOIN Timeplus is able to automatically choose the closest version. Continue on our previous scenario. If the current iPhone15 price is 800, and you add a new order for 1 iPhone15, then you will get a transaction amount of 800. Then you change the iPhone15 price to 799, and add a new order for 1 iPhone15, you will get a transaction amount of 799. But if you add an order with _tp_time before the price change, you will get the transaction amount as 800 again, because Timeplus keeps multiple versions for the price and chooses the older version that best matches the order time. If you are not familiar with ASOF JOIN, this special JOIN provides non-exact matching capabilities. This can work well if two streams have the same id, but not exactly the same timestamps. ## The advanced keep_versions setting:​ In the above example, you can add settings keep_versions=3 at the end of the query. This will inform the query engine to read the base version from the historical storage for the versioned_kv stream, then read the new events in the streaming storage and based on the ASOF time condition to pick up 3 versions in the memory, and finally join events from left append_stream with the right 3 versions, and find the best candidate to join together. ## Retention Policy​ You should not set the TTL(Time-To-Live) for the historical storage of versioned_kv stream. Because only the last version of the same primary key is kept in the historical storage (via an auto-compact background process). Manually set a TTL may remove those events who haven't been updated recently. You may set a time-based or size-based retention policy for the streaming storage for the versioned_kv stream. But this is optional. By default a 4GB segment file is used for the streaming storage. - Query Single Stream - Use Versioned Stream in INNER JOIN - Use Versioned Stream in LEFT JOIN - Use Versioned Stream in ASOF JOIN - The advanced keep_versions setting: - Retention Policy ## View - [View and Materialized View | Timeplus](https://docs.timeplus.com/view): Real-time data pipelines are built via Materialized Views in Timeplus. - Core Features - Materialized Views # View and Materialized View Real-time data pipelines are built via Materialized Views in Timeplus. There are two types of views in Timeplus: logical view (or just view) and materialized view. ## View​ If you have a common query you'll run multiple times a week or month, you can save it as a "Bookmark" in web console. While you can load the bookmarked query in the query editor without typing it again, you can't refer to bookmarked queries in a SQL query. This is where views come in. ### Create or Drop Views​ You can create views for all kinds of queries, and refer to the views in other queries. Creating a view won't trigger any query execution. Views are evaluated only when other queries refer to it. To create a vanilla view: To drop a vanilla view: If the view is created based on a streaming query, then you can consider the view as a virtual stream. For example: This will create a virtual stream to filter all events with c1 = 'a'. You can use this view as if it's another stream, e.g. A view could be a bounded stream if the view is created with a bounded query using table function, e.g. Then each time you run SELECT count(*) FROM view2 will return the current row number of the my_stream immediately without waiting for the future events. ### Parameterized Views​ Starting from Timeplus Enterprise 2.9, you can create views with parameters. For example: ## Materialized view​ The difference between a materialized view and a regular view is that the materialized view is running in background after creation and the resulting stream is physically written to internal storage (hence it's called materialized). Once the materialized view is created, Timeplus will run the query in the background continuously and incrementally emit the calculated results according to the semantics of its underlying streaming select. ### Create a Materialized View​ It's required to create a materialized view using a streaming select query. #### Without Target Stream​ By default, when you create a materialized view without INTO .., an internal stream will be created automatically as the data storage. Querying on the materialized view will result in querying the underlying internal stream. While this approach is easy to use, it's not recommended for production data processing. The internal stream will be created with default settings, lack of optimization for sharding, retention, etc. #### With Target Stream​ It's recommended to specify a target stream when creating a materialized view, no matter a stream in Timeplus, an external stream to Apache Kafka, Apache Pulsar, or external tables to ClickHouse, S3, Iceberg, etc. Use cases for specifying a target stream: 1. In some cases, you may want to build multiple materialized views to write data to the same stream. In this case, each materialized view serves as a real-time data pipeline. 2. Or you may need to use Changelog Stream, Versioned Stream or Mutable Stream to build lookups. 3. Or you may want to set the retention policy for the materialized view. 4. You can also use materialized views to write data to downstream systems, such as ClickHouse, Kafka, or Iceberg. To create a materialized view with the target stream: ### Use Materialized Views​ There are different ways to use the materialized views in Timeplus: 1. Streaming mode: SELECT * FROM materialized_view Get the result for future data. This works in the same way as views. 2. Historical mode: SELECT * FROM table(materialized_view) Get all past results for the materialized view. 3. Historical + streaming mode: SELECT * FROM materialized_view WHERE _tp_time>='1970-01-01' Get all past results and as well as the future data. 4. Pre-aggregation mode: SELECT * FROM table(materialized_view) where _tp_time in (SELECT max(_tp_time) as m from table(materialized_view)) This immediately returns the most recent query result. If _tp_time is not available in the materialized view, or the latest aggregation can produce events with different _tp_time, you can add the emit_version() to the materialized view to assign a unique ID for each emit and pick up the events with largest emit_version(). For example: You build data pipelines in Timeplus using materialized views. ### Load Balancing​ It's common to define many materialized views in Timeplus for various computation and analysis. Some materialized views can be memory-consuming or cpu-consuming. In Timeplus Enterprise cluster mode, you can schedule the materialized views in a proper way to ensure each node gets similar workload. #### Manual Load Balancing​ Starting from Timeplus Enterprise v2.3, when you create a materialized view with DDL SQL, you can add an optional memory_weight setting for those memory-consuming materialized views, e.g. When memory_weight is not set, by default the value is 0. When Timeplus Enterprise server starts, the system will list all materialized views, ordered by the memory weight and view names, and schedule them in the proper node. For example, in a 3-node cluster, you define 10 materialized views with names: mv1, mv2, .., mv9, mv10. If you create the first 6 materialized views with SETTINGS memory_weight = 10, then node1 will run mv1 and mv4; node2 will run mv2 and mv5; node3 will run mv3 and mv6; Other materialized views(mv7 to mv10) will be randomly scheduled on any nodes. It's recommended that each node in the Timeplus Enterprise cluster shares the same hardware specifications. For those resource-consuming materialized views, it's recommended to set the same memory_weight, such as 10, to get the expected behaviors to be dispatched to the proper nodes for load-balancing. #### Auto Load Balancing​ Starting from Timeplus Enterprise v2.5, you can also apply auto-load-balancing for memory-consuming materialized views in Timeplus Enterprise cluster. By default, this feature is enabled and there are 3 settings at the cluster level: As the administrator, you no longer need to determine which materialized views need to set a memory_weight setting. In a cluster, Timeplus will monitor the memory consumption for each materialized view. Every 30 seconds, configurable via workload_rebalance_check_interval, the system will check whether there are any node with memory over 50% full. If so, check whether there is any materialized view in such node consuming 10% or more memory. When those conditions are all met, rescheduling those materialized views to less busy nodes. During the rescheduling, the materialized view on the previous node will be paused and its checkpoint will be transferred to the target node, then the materialized view on target node will resume the streaming SQL based on the checkpoint. ### Auto-Scaling Materialized Views​ Starting from Timeplus Enterprise v2.8, materialized views can be configured to run on elastic compute nodes. This can reduce TCO (Total Cost of Ownership), by enabling high concurrent materialized views scheduling, auto scale-out and scale-in according to workload. To enable this feature, you need to 1. create a S3 disk in the s3_plain type. 2. create a materialized view by setting the checkpoint storage to s3 and use the s3 disk. 3. enable compute nodes in the cluster, with optional autoscaling based on your cloud or on-prem infrastructure. For example: ### Drop Materialized Views​ Run the following SQL to drop a view or a materialized view. Like CREATE STREAM, stream deletion is an async process. ### Best Practices​ - It's recommended to specify a target stream when creating a materialized view, no matter a stream in Timeplus, an external stream to Apache Kafka, Apache Pulsar, or external tables to ClickHouse, S3, Iceberg, etc. - ViewCreate or Drop ViewsParameterized Views - Create or Drop Views - Parameterized Views - Materialized viewCreate a Materialized ViewWithout Target StreamWith Target StreamUse Materialized ViewsLoad BalancingManual Load BalancingAuto Load BalancingAuto-Scaling Materialized ViewsDrop Materialized ViewsBest Practices - Create a Materialized ViewWithout Target StreamWith Target Stream - Without Target Stream - With Target Stream - Use Materialized Views - Load BalancingManual Load BalancingAuto Load Balancing - Manual Load Balancing - Auto Load Balancing - Auto-Scaling Materialized Views - Drop Materialized Views - Best Practices - Create or Drop Views - Parameterized Views - Create a Materialized ViewWithout Target StreamWith Target Stream - Without Target Stream - With Target Stream - Use Materialized Views - Load BalancingManual Load BalancingAuto Load Balancing - Manual Load Balancing - Auto Load Balancing - Auto-Scaling Materialized Views - Drop Materialized Views - Best Practices - Without Target Stream - With Target Stream - Manual Load Balancing - Auto Load Balancing ## Viz - [Data Visualization | Timeplus](https://docs.timeplus.com/viz): Timeplus provides out-of-box streaming charts and dashboards to visualize the real-time data and understand the pattern/trend. You can also integrate Timeplus with your external BI systems, such as Redash, Metabase, Grafana, etc. - Core Features - Data Visualization # Data Visualization Timeplus provides out-of-box streaming charts and dashboards to visualize the real-time data and understand the pattern/trend. You can also integrate Timeplus with your external BI systems, such as Redash, Metabase, Grafana, etc. ## Chart​ After you run a query, you can switch to the Visualization tab to turn the results into a chart. We offer different chart types for your use cases: ### Line chart​ Line charts work best with time series data. - Data settings: X-axis: Event Time (_tp_time), or Arrival Time(when the browser gets the data point), or a custom column. Y-axis: a custom column in numeric data types (int/float/etc) Color: by default None is selected. You can choose a categorical column and draw a line for each unique value of the column value in different colors. - X-axis: Event Time (_tp_time), or Arrival Time(when the browser gets the data point), or a custom column. - Y-axis: a custom column in numeric data types (int/float/etc) - Color: by default None is selected. You can choose a categorical column and draw a line for each unique value of the column value in different colors. - Format settings: X-axis title and the data range(last 1 minute, last 1 hour, all time, etc.). By default, 'Last 1 minute' is selected. If you are running a historical query, please change it to 'All time'. Y-axis title, min/max value, line color, style Show/hide the data label for the last value of the line chart, number of decimal and whether to show prefix/suffix as unit. Show/hide grid lines Show/hide legend Show/hide data points - X-axis title and the data range(last 1 minute, last 1 hour, all time, etc.). By default, 'Last 1 minute' is selected. If you are running a historical query, please change it to 'All time'. - Y-axis title, min/max value, line color, style - Show/hide the data label for the last value of the line chart, number of decimal and whether to show prefix/suffix as unit. - Show/hide grid lines - Show/hide legend - Show/hide data points Data settings: - X-axis: Event Time (_tp_time), or Arrival Time(when the browser gets the data point), or a custom column. - Y-axis: a custom column in numeric data types (int/float/etc) - Color: by default None is selected. You can choose a categorical column and draw a line for each unique value of the column value in different colors. Format settings: - X-axis title and the data range(last 1 minute, last 1 hour, all time, etc.). By default, 'Last 1 minute' is selected. If you are running a historical query, please change it to 'All time'. - Y-axis title, min/max value, line color, style - Show/hide the data label for the last value of the line chart, number of decimal and whether to show prefix/suffix as unit. - Show/hide grid lines - Show/hide legend - Show/hide data points ### Area chart​ Shows a stacked area chart, with the same format settings as Line Chart. ### Column chart​ - Data settings: X-axis: a categorical column Y-axis: a numeric column Color: whether to show grouped data in either stack mode or dodge mode. Update mode: append only, or show data points from the last timestamp, or choose a key column to show the latest data value for each key value. - X-axis: a categorical column - Y-axis: a numeric column - Color: whether to show grouped data in either stack mode or dodge mode. - Update mode: append only, or show data points from the last timestamp, or choose a key column to show the latest data value for each key value. - Format settings: X-axis title, Y-axis title, column color, show/hide data label (with decimal and prefix/suffix unit), show/hide grid lines, show/hide legend - X-axis title, Y-axis title, column color, show/hide data label (with decimal and prefix/suffix unit), show/hide grid lines, show/hide legend Data settings: - X-axis: a categorical column - Y-axis: a numeric column - Color: whether to show grouped data in either stack mode or dodge mode. - Update mode: append only, or show data points from the last timestamp, or choose a key column to show the latest data value for each key value. Format settings: - X-axis title, Y-axis title, column color, show/hide data label (with decimal and prefix/suffix unit), show/hide grid lines, show/hide legend ### Bar chart​ Similar to the Column chart, with data points shown as horizontal bars instead of vertical columns. Best fit for show top-N values. ### Single value chart​ - Data settings: choose a numeric column to show its value - choose a numeric column to show its value - Format settings: Number of decimal Font size Unit as suffix or prefix Show/hide sparkline Show/hide delta for last value vs. the current value - Number of decimal - Font size - Unit as suffix or prefix - Show/hide sparkline - Show/hide delta for last value vs. the current value Data settings: - choose a numeric column to show its value Format settings: - Number of decimal - Font size - Unit as suffix or prefix - Show/hide sparkline - Show/hide delta for last value vs. the current value ### Map chart​ - Data settings: Set columns for longitude and latitude. Color: show grouped data in stack mode or dodge mode. Update mode: append only, or show data points from the last timestamp, or choose a key column to show the latest data value for each key value. - Set columns for longitude and latitude. - Color: show grouped data in stack mode or dodge mode. - Update mode: append only, or show data points from the last timestamp, or choose a key column to show the latest data value for each key value. - Format settings: Map dots: color scheme, opacity, and size (fixed size, or set a min. and max size) - Map dots: color scheme, opacity, and size (fixed size, or set a min. and max size) Data settings: - Set columns for longitude and latitude. - Color: show grouped data in stack mode or dodge mode. - Update mode: append only, or show data points from the last timestamp, or choose a key column to show the latest data value for each key value. Format settings: - Map dots: color scheme, opacity, and size (fixed size, or set a min. and max size) ### OHLC chart​ This is a new chart type, currently in technical preview. The open-high-low-close (OHLC) chart is common in the finance industry, to visualize the movement of prices over time. Please contact us if you'd like to try this preview feature. Please make sure there are 5 columns with names: time, open, close, high, low, e.g. ### Table​ Shows the data as a list table. - Data settings Maximum row count. Update mode: Show All: show all query results (limited by the maximum row count) By Time: show data points from the last timestamp By Key: choose a key column to show the latest row for each key value. - Maximum row count. - Update mode: Show All: show all query results (limited by the maximum row count) By Time: show data points from the last timestamp By Key: choose a key column to show the latest row for each key value. - Show All: show all query results (limited by the maximum row count) - By Time: show data points from the last timestamp - By Key: choose a key column to show the latest row for each key value. - Format settings For each column, you can choose to show or hide it, or set column width and decimal for numeric columns. Conditional formatting: highlight a cell or an entire row if the value meets a certain condition you've set - For each column, you can choose to show or hide it, or set column width and decimal for numeric columns. - Conditional formatting: highlight a cell or an entire row if the value meets a certain condition you've set Data settings - Maximum row count. - Update mode: Show All: show all query results (limited by the maximum row count) By Time: show data points from the last timestamp By Key: choose a key column to show the latest row for each key value. - Show All: show all query results (limited by the maximum row count) - By Time: show data points from the last timestamp - By Key: choose a key column to show the latest row for each key value. - Show All: show all query results (limited by the maximum row count) - By Time: show data points from the last timestamp - By Key: choose a key column to show the latest row for each key value. Format settings - For each column, you can choose to show or hide it, or set column width and decimal for numeric columns. - Conditional formatting: highlight a cell or an entire row if the value meets a certain condition you've set Charts can be added to the dashboard by clicking the Add to dashboard button on the right. ## Dashboard​ One or more dashboards can be created in each workspace to tell stories about your data. You can set a name and optional description for each dashboard. One or more charts can be added to the dashboard, with 3 size options: - Small: takes up 1/4 of the page width - Medium: takes up 1/2 of the page width - Large: takes up entire page width ## Dashboard filters and query variables​ As a new feature, you can add filters in dashboards. Here is an example if you want to list speeding vehicles. First, in the query page, to run a SQL with a fixed condition, e.g. Run the query and make sure it meets your needs. Then you can change the fixed condition with a query variable, e.g. The query editor will show a text input for the value of speed_limit. You can put a value and run the parameterized query and turn it to a visualization and add it to a new dashboard or an existing dashboard. Please make sure the SQL syntax is correct with query variables. For example if you are filtering with a string value, you should add quote around the variable placeholder, e.g. ... (content truncated) ## Why-timeplus - [Why Timeplus? | Timeplus](https://docs.timeplus.com/why-timeplus): Timeplus simplifies stateful stream processing and analytics with a fast, single-binary engine. Using SQL as a domain-specific language and both row and column-based state stores, it enables developers to build real-time applications, data pipelines, and analytical dashboards at the edge or in the cloud, reducing the cost, time, and complexity of multi-component stacks. - Overview - Why Timeplus? # Why Timeplus? Timeplus simplifies stateful stream processing and analytics with a fast, single-binary engine. Using SQL as a domain-specific language and both row and column-based state stores, it enables developers to build real-time applications, data pipelines, and analytical dashboards at the edge or in the cloud, reducing the cost, time, and complexity of multi-component stacks. ## Architecture: The Best of Both Worlds​ ## Unified streaming and historical data processing​ Timeplus streams offer high performance, resiliency, and seamless querying by using an internal Write Ahead Log (WAL) and Historical Store. The WAL ensures ultra-fast inserts and updates, while the Historical Store, optimized for various query types, handles efficient historical queries. This architecture transparently serves data to users based on query type from both, often eliminating the need for Apache Kafka as a commit log or a separate downstream database, streamlining your data infrastructure. ## Append and Mutable Streams​ Configure types of streams to optimize performance. - Append streams: Excel at complex aggregations, storing data in a columnar format for faster access and processing. - Mutable streams: Support UPSERTs and DELETEs, ideal for applications like Materialized Caches or GDPR compliance, using a row-based store optimized for fast data retrieval and query consistency. ## Single Binary​ Timeplus is a fast, powerful, and efficient SQL stream processing platform with no dependencies, JVM, or ZK. It runs in bare-metal or Kubernetes environments, from edge to cloud, using a single binary (~150MB). Timeplus scales easily from edge devices to multi-node clusters, and with its Append-Only data structures and historical stores, some use cases may not need Kafka or a separate database at all. ## Multi-JOINs and ASOF JOINs​ Stream processing involves combining multiple data sources, and MULTI-JOINs are essential for enriching and correlating events in streaming queries. Timeplus allows you to run ad-hoc historical queries on the same data, reducing the need for denormalization in downstream data warehouses. In many cases, Business Intelligence and analytical queries can be executed directly in Timeplus, eliminating the need for a separate data warehouse. ASOF JOINs enable approximate time-based lookups for comparing recent versus historical data. ## Python and JavaScript UDF​ We understand that SQL may not be able to express all business logic for streaming or querying. JavaScript and Python User Defined Functions (UDFs) and User Defined Aggregate Functions (UDAFs) can be used to extend Timeplus to encapsulate custom logic for both stateless and stateful queries. With Python UDFs, this opens up the possibility to bring in pre-existing and popular libraries, including data science and machine learning libraries! ## External Stream, External Table​ We want to simplify the experience of joining data from Apache Kafka and writing results out to data warehouses such as Clickhouse, or another Timeplus instance. Timeplus implements native integration to these systems in timeplusd via EXTERNAL STREAM (with Kafka and Timeplus) and EXTERNAL TABLE (with ClickHouse). No need for deploying yet another Connector component. We understand that we cannot do this for all systems and for that, we have Timeplus Connector which can be configured to integrate with hundreds of other systems if needed. ## Collection​ With built-in External Streams and External Tables, Timeplus can natively collect real-time data from, or send data to, Kafka, Redpanda, ClickHouse, or another Timeplus instance, without any data duplication. Timeplus also supports a wide range of data sources through sink/source connectors. Users can push data from files (CSV/TSV), via native SDKs in Java, Go, or Python, JDBC/ODBC, Websockets, or REST APIs. ## Transformation​ With a powerful streaming SQL console, users can leverage their preferred query language to create Streams, Views, and incremental Materialized Views. This enables them to transform, roll up, join, correlate, enrich, aggregate, and downsample real-time data, generating meaningful outputs for real-time alerting, analytics, or any downstream systems. ## Routing​ Timeplus allows data to be routed to different sinks based on SQL-based criteria and provides a data lineage view of all derived streams in its console. A single data result can generate multiple outputs for various scenarios and systems, such as analytics, alerting, compliance, etc., without any vendor lock-in. ## Analytics and Alerting​ Powered by SSE (Server-Sent Events), Timeplus supports push-based, low-latency dashboards to visualize real-time insights through data pipelines or ad-hoc queries. Additionally, users can easily build observability dashboards using Grafana plugins. SQL-based rules can be used to trigger or resolve alerts in systems such as PagerDuty, Slack, and other downstream platforms. - Architecture: The Best of Both Worlds - Unified streaming and historical data processing - Append and Mutable Streams - Single Binary - Multi-JOINs and ASOF JOINs - Python and JavaScript UDF - External Stream, External Table - Collection - Transformation - Routing - Analytics and Alerting ## Working-with-streams - [Streams | Timeplus](https://docs.timeplus.com/working-with-streams): All data live in streams - Core Features - Streams # Streams ## All data live in streams​ Timeplus is a streaming analytics platform and data lives in streams. Timeplus streams are similar to tables in the traditional SQL databases. Both of them are essentially datasets. The key difference is that Timeplus stream is an append-only (by default), unbounded, constantly changing events group. Timeplus supports multiple types of streams: 1. By default, the streams are append-only and immutable (older data can be purged automatically by setting a retention policy). 2. If you want to create a stream to track the latest value for a primary key or a set of keys, you can create Mutable Streams. This is only available in Timeplus Enterprise. 3. In Timeplus Proton, you can also create Versioned Streams and Changelog Stream. But those 2 stream modes will be deprecated and replaced by mutable streams. 4. You can also define External Streams to run SQL against remote Kafka/Redpanda brokers, or the other Timeplus/Proton server. ## Create a stream​ You can create a stream via the Timeplus Console UI, or via SQL. When you ingest data into Timeplus from Kafka or file sources, streams can be created automatically to match the schema of the data. ## Query a stream​ By default, querying the stream will continuously scan new events and output new results. It never ends unless the user cancels the query. For example, you can get the latest web logs with HTTP 500 error or get the min/max/avg of a metric for every minute from an IoT device. Please read Streaming Queries for more details. If you only want to analyze the existing data and need an immediate response, you can run Non-streaming Queries via the table function. This will turn the query in the bounded mode and only scan the existing data. For example, you can run select count(*) from table(stream1) to get the total number of rows in the data stream. ## Delete a stream​ From the web console, you can delete the stream. This will permanently delete all data in the stream and delete the stream itself. Data cannot be recovered after deletion. - All data live in streams - Create a stream - Query a stream - Delete a stream