Arcadia Solutions Inc.
Medallion Clickstream Cost Optimization
Cut storage and compute ~20% by tuning the Delta Lake storage layer.
Jan 2020 – Aug 2020
Cutting lakehouse cost ~20% by fixing file layout, not buying compute
A Medallion clickstream platform was accumulating cost faster than value, the classic small-files and poor-partitioning tax on a Delta Lake. Rising lakehouse cost usually looks like a compute problem and is actually a file-layout one, so I tuned partitioning to real query predicates, compacted small files, and added indexing, cutting storage and compute roughly 20% while the predictive-sales and user-behavior workloads on top kept running untouched.
- ~20%
- storage + compute cost cut
- 0
- consumer-facing changes
- Delta Lake
- storage layer tuned
- Delta Lake
- Medallion Architecture
- Spark
- Partitioning
- Compaction
The problem
A Medallion-architecture clickstream platform was accumulating storage and compute cost faster than it was accumulating value: the classic small-files and poor-partitioning tax on a Delta Lake.
What I built
- 01Tuned partitioning strategy across the Delta Lake storage layer to match real query predicates.
- 02Introduced file compaction to collapse small-file overhead.
- 03Added indexing to reduce scanned data per query.
- 04Kept the platform serving predictive sales models and user-behavior analytics throughout.
Outcome
Roughly 20% reduction in storage and compute cost, with analytics workloads unaffected.