Arcadia Solutions Inc.
Medallion Clickstream Cost Optimization
Cut storage and compute ~20% by tuning the Delta Lake storage layer.
Jan 2020 – Aug 2020
Framed for
~20% storage and compute savings from file-layout work
Partitioning, file compaction, and indexing across a Delta Lake storage layer on a Medallion clickstream platform: cost down, workloads untouched.
- ~20%
- storage + compute cost cut
- Medallion
- clickstream architecture
- Delta Lake
- storage layer tuned
- Delta Lake
- Medallion Architecture
- Partitioning
- Compaction
- Spark
Reducing platform cost without touching the consumers
Diagnosed rising lakehouse cost as a storage-layout problem rather than a capacity problem, and cut roughly 20% while predictive sales models and user-behavior analytics kept running.
- ~20%
- cost reduction
- 0
- consumer-facing changes
- Lakehouse
- cost governance
- Lakehouse Architecture
- Cost Optimization
- Delta Lake
- Data Modeling
The problem
A Medallion-architecture clickstream platform was accumulating storage and compute cost faster than it was accumulating value: the classic small-files and poor-partitioning tax on a Delta Lake.
What I built
- 01Tuned partitioning strategy across the Delta Lake storage layer to match real query predicates.
- 02Introduced file compaction to collapse small-file overhead.
- 03Added indexing to reduce scanned data per query.
- 04Kept the platform serving predictive sales models and user-behavior analytics throughout.
Outcome
Roughly 20% reduction in storage and compute cost, with analytics workloads unaffected.