Replies: 4 comments 15 replies
-
|
Thanks for the kind words @brabeji. Could you share your pipeline configuration and SQL? Do you have storage enabled? |
Beta Was this translation helpful? Give feedback.
-
|
@brabeji, yes this does indicate that the bottleneck is the pipeline itself, not the connector. 10K is a very low throughput for this type of workload, so something must be off. As @lalithsuresh said, please make sure that you have storage enabled and are not running out of memory. Prometheus + Grafana may not help much in this case. We are building a self-serve profiler for Feldera. In the meantime, you can extract a profile from the circuit in the graphviz format using the |
Beta Was this translation helpful? Give feedback.
-
|
I think |
Beta Was this translation helpful? Give feedback.
-
|
these are complementary. |
Beta Was this translation helpful? Give feedback.
-
|
The updated query using explicit max_ts is running without buffering 👍 |
Beta Was this translation helpful? Give feedback.
-
|
If your dataset is append-only, some tables logically will grow unbounded. You may need to rely on some of our tools for garbage-collecting state if you want to run this for a long time: https://docs.feldera.com/tutorials/time-series. Note that we still have quite a few optimizations left to implement for these use cases, including the one I mentioned above. |
Beta Was this translation helpful? Give feedback.
-
|
@brabeji good to hear the buffering issues is fixed. @mihaibudiu I wonder if we need to add a warning if we end up generating code with cross products underneath. |
Beta Was this translation helpful? Give feedback.
-
Can you please clarify how exactly you do this? |
Beta Was this translation helpful? Give feedback.
-
|
The query decorrelator essentially generates this CROSS JOIN. I am not sure we can easily come up with a better plan automatically. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hello and thank you, I'm really enjoying Feldera!
I'm ingesting 10,000 records/s using parquet format from ClickHouse through redpanda into one non-materialized, append-only table. Additionally ~5 records/s the same way into another table. I then do simple group by in the latter table and stream last 10 records from each table using HTTP.
After initial period of sustained throughput of ~10,000 records/s it starts to fluctuate and records start to buffer:
Some observations:
max_batch_sizeis set to 10,000 which I've suspected, so it now sits atmax_batch_size: 20000,. That improved the situation quite a bit (longer period of smooth ingress, slower buffering afterwards)I presume that if redpanda was too slow, the records wouldn't end up buffered. Rather the opposite but that is just my assumption.
I will certainly set up prometheus+grafana to dig in a bit later. Just trying to see if someone with more experience can provide some obvious quickfixes.
Beta Was this translation helpful? Give feedback.
All reactions