Lemonpeak (client: a major US satellite/streaming TV provider)
Subscriber Streaming Analytics
Lambda-architecture pipelines for subscriber analytics at 5M daily users.
Dec 2020 – Jun 2021
Lambda-architecture analytics for a 5M-user streaming platform
Subscriber analytics for a satellite and streaming TV provider that had to serve two incompatible needs at once: real-time signal for operations, and correct, reprocessable history for ad-ROI and executive reporting. I built Lambda-architecture pipelines on Kafka and Spark Streaming, a speed layer for latency and a batch layer for correctness, with data-quality assertions in the pipeline because at 5M daily users broken data corrupts a dashboard more quietly than broken code.
- 5M
- daily users on the platform
- Lambda
- speed + batch architecture
- 2
- engineers mentored
- Kafka
- Spark Streaming
- AWS EC2/S3
- Great Expectations
- pytest
The problem
A satellite and streaming TV provider needed subscriber analytics that served two incompatible needs at once: real-time signal for operations, and correct, reprocessable history for ad-ROI and executive reporting.
What I built
- 01Built Lambda-architecture pipelines on Kafka, Spark Streaming, and AWS EC2/S3: a speed layer for low-latency views and a batch layer for correctness.
- 02Supported ad-ROI tracking, A/B testing, and executive dashboards off the same platform.
- 03Mentored two junior engineers on testing and warehousing practice using pytest and Great Expectations.
Outcome
Delivered subscriber analytics on a 5M-daily-user platform supporting ad-ROI tracking, A/B testing, and executive dashboards, with two junior engineers brought up on testing and warehousing practice.