What’s the best way to build analytics for up to one million event records?

0
5
Asked By MellowCedar42 On

I'm building a driver-monitoring system for roughly 150–200 buses. It produces alert events, and a Svelte supervision dashboard may need to analyze around 360,000 records in typical cases, with a possible upper bound near one million.

The current design is Firestore → Redis → Python, where I load the alert records into Redis and calculate dashboard metrics in Python. Each cached record is about 798 bytes, which puts the Redis usage around 287 MB for 360,000 records or 798 MB for one million.

I'm concerned that storing all raw events in Redis and repeatedly recalculating metrics may not scale well. Firestore provides server-side count, sum, and average operations, but my dashboard also needs grouped results, rankings, time windows, multiple alert types, driver statistics, and other custom calculations.

How would you normally design this? Should I use precomputed rollups, a SQL or time-series database, Firestore queries, Redis caching, or some combination of these?

4 Answers

Answered By VelvetOrbit56 On

Make the rollup process idempotent so events can be replayed or corrected safely. You’ll also want a strategy for late-arriving data, such as reopening recent time buckets and recomputing them. Redis can still be useful as a short-lived cache for especially expensive dashboard responses, but it probably shouldn’t hold the whole event history.

Answered By QuietMarble7 On

A million rows is not especially large, so you probably don’t need an exotic architecture. A conventional SQL database with good indexes and a few materialized views for common dashboard queries would likely be simpler and more flexible than using Firestore as the analytics layer. Keep the raw events available for auditing, and let the database handle grouping and aggregation.

Answered By BlueFern88 On

I wouldn’t use Redis as the primary analytical store for hundreds of megabytes of raw events that you recalculate repeatedly. Keep the events in your durable database, then create incremental rollups for the metrics users actually need. For example, upsert hourly buckets grouped by time period, bus, driver, and alert type. The dashboard can read those much smaller tables and only query raw events when someone drills into details.

Answered By CopperLynx31 On

If the time-based queries become the main challenge, a time-series option such as TimescaleDB is worth considering. It can retain the raw events for audits while supporting time buckets, retention policies, and continuous or incremental aggregates. That gives you analytical queries without loading the entire dataset into application memory.

Related Questions

LEAVE A REPLY

Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.