Troubleshooting Guide
This guide provides solutions to common issues you may encounter when using Timeplus.
Please also query system.stream_state_log and system.stream_metric_log to check the state changes, metrics and errors of database resources in Timeplus.
Memory Issues
JOIN queries run out of memory
Symptoms:
default.hop_mv: Background runtime error: Code: 191. DB::Exception:
Streaming join's memory reaches max size: 524288000,
current total: 524297968, left total: 177712480, right total: 346585488:
While executing StreamingJoinTransform.
Solution: Increase the JOIN buffer size for the query or materialized view.
ALTER VIEW hop_mv MODIFY QUERY SETTING join_max_buffered_bytes=8589934592
JavaScript UDF runs out of memory
Symptoms:
default.omv: Background runtime error: Code: 2531. DB::Exception:
Current V8 heap size used=571000 bytes, total=40307261440 bytes,
javascript_max_memory_bytes=2, exceed the limit=2 bytes, v8 heap stat={...}:
While executing GlobalAggregatingTransform. (UDF_MEMORY_THRESHOLD_EXCEEDED)
Solution: Increase the memory limit for the JavaScript UDF.
ALTER VIEW <mv_name> MODIFY QUERY SETTING javascript_max_memory_bytes = 10485760000
Data Consistency
Materialized view is not updating
Symptoms:
default.HOP_mv: Background runtime error: Code: 2547. DB::Exception:
Failed to fetch for ns=default name=HOP_mv id=65496080-a11f-4db1-838c-072b5afcf4c0
shard=0, SEQUENCE_COMPACTED_AWAY: While executing StreamingStoreSource.
(SEQUENCE_COMPACTED_AWAY)
Solution:
- Check the definition of the materialized view via
SHOW CREATE <mv_name>
. - Drop the materialized view and recreate it.
Data Size and Ingestion
TOO_LARGE_RECORD
Symptoms:
Code: 2514. DB::Exception: Failed to ingest, error=TOO_LARGE_RECORD. (TOO_LARGE_RECORD)
Solution:
For ad-hoc stream ingestion, you can increase the max_insert_block_size
setting to allow larger records.
INSERT INTO appendOnlyStream
SETTINGS max_insert_block_size=10, max_insert_threads=8
SELECT ..
If you encounter this issue with a materialized view, you can increase the max_entry_size
setting in timeplusd.yml.
- Stop the Timeplus service.
- Edit the
data.datastore.log.max_entry_size
setting in thetimeplusd.yml
configuration file. The default value is 10485760(10MB). You can set to 100MB, which is 104857600. - Start the Timeplus service.
Fail to drop streams with large data
Symptoms:
Stream or Partition in default.hop_mv was not dropped.
Reason: Size (81.49 GB) is greater than max_[table/partition]_size_to_drop (50.00 GB)
Solution:
You can drop with the force_drop_big_stream
setting.
DROP STREAM hop_mv SETTINGS force_drop_big_stream=1
DISK_USAGE_RATIO_THRESHOLD_EXCEEDED
Symptoms:
Code: 2529. DB::Exception: Disk default utilization is 0.9007095458237306, exceeds the max_disk_util 0.9. (DISK_USAGE_RATIO_THRESHOLD_EXCEEDED)
Solution:
Please free up some disk space to allow ingestion to continue.
Concurrency
Too many concurrent queries
Symptoms:
DB::Exception: Too many simultaneous queries. Maximum: 100.
(TOO_MANY_SIMULTANEOUS_QUERIES)
Solution:
Increase the max_concurrent_queries
setting in timeplusd.yml.
- Stop the Timeplus service.
- Edit the
max_concurrent_queries
setting in thetimeplusd.yml
configuration file. The default value is 100. You can set to 200 or higher. - Start the Timeplus service.
You can also set different values for max_concurrent_insert_queries
and max_concurrent_select_queries
.
Increasing these limits may impact the performance and require more system resources. Please monitor the system performance after changing these settings.
Kafka External Stream
NO_AVAILABLE_KAFKA_CONSUMER
Symptoms:
DB::Exception: Reached consumers limit 50.
Existing queries need to be stopped before running other queries.
Or update external_stream.kafka.max_consumers_per_stream to a bigger number in the config file.
(NO_AVAILABLE_KAFKA_CONSUMER)
Solution:
Increase the max_consumers_per_stream
setting in conf/timeplusd.yml.
- Stop the Timeplus service.
- Edit the
external_stream.kafka.max_consumers_per_stream
setting in thetimeplusd.yml
configuration file. The default value is 50. You can set to 100 or higher. - Restart the Timeplus service.
external_stream:
kafka:
max_consumers_per_stream: 200