Lemonpeak (client: a major US satellite/streaming TV provider)
Subscriber Streaming Analytics
Lambda-architecture pipelines for subscriber analytics at 5M daily users.
Dec 2020 – Jun 2021
Framed for
Kafka and Spark Streaming on a 5M-daily-user platform
Lambda-architecture pipelines (Kafka and Spark Streaming for the speed layer, batch for correctness) on AWS EC2/S3, with data-quality assertions via Great Expectations.
- 5M
- daily users on the platform
- Lambda
- speed + batch architecture
- 2
- junior engineers mentored
- Kafka
- Spark Streaming
- AWS EC2/S3
- pytest
- Great Expectations
One platform serving operations, ad-ROI, and the executive team
Designed a two-layer architecture that reconciled real-time operational signal with the reprocessable history that ad-ROI tracking, A/B testing, and executive dashboards required.
- 3
- consumer groups served
- 5M
- daily users
- 2
- engineers mentored
- Lambda Architecture
- Distributed Systems
- AWS
- Data Quality
The problem
A satellite and streaming TV provider needed subscriber analytics that served two incompatible needs at once: real-time signal for operations, and correct, reprocessable history for ad-ROI and executive reporting.
What I built
- 01Built Lambda-architecture pipelines on Kafka, Spark Streaming, and AWS EC2/S3: a speed layer for low-latency views and a batch layer for correctness.
- 02Supported ad-ROI tracking, A/B testing, and executive dashboards off the same platform.
- 03Mentored two junior engineers on testing and warehousing practice using pytest and Great Expectations.
Outcome
Delivered subscriber analytics on a 5M-daily-user platform supporting ad-ROI tracking, A/B testing, and executive dashboards, with two junior engineers brought up on testing and warehousing practice.