Clickstream Stream
TypeUser Behavior Logs
FormatJSON Events
Volume10M events/day
Project not found
This case study does not exist yet.
Return to the project library and choose another project.
Back to Projects →Monzo user sentiment insights dashboard
A robust event-streaming and lakehouse infrastructure processing over 10M actions daily to power live dashboard telemetry.
monzo
Project hero visual
Project Overview
This project implements a scalable real-time data ingestion and analytics pipeline. Designed to capture clickstream events, user transaction data, and inventory updates, this pipeline channels unstructured data directly into a unified lakehouse pattern. The goal is to provide instantaneous sales telemetry, real-time stock levels, and session-by-session behavioral analytics to product and marketing teams.
Using a combination of Apache Kafka for stream brokering, Spark Structured Streaming for schema reinforcement and transformations, and Delta Lake on AWS S3 for the storage tier, we achieved sub-second latency for complex analytical queries.
The Problem
Prior to the new pipeline, e-commerce transactions and catalog change events were processed via batch jobs running every 6 hours. This resulted in significant data latency, preventing live alerts for inventory depletion and causing marketing campaigns to target users with recently sold-out products. Furthermore, concurrent read/write locks on the legacy relational database during analytics queries degraded consumer-facing portal performance.
The team needed a solution capable of handling bursty clickstream volume during peak hours without impacting database performance, while enforcing ACID transaction properties on the destination analytical store.
Key Requirements
Real-time event processing with end-to-end latency below 2 seconds.
Scalable ingestion to comfortably handle peaks of 5,000 requests per second.
Strict data quality validation preventing schema drift from corrupting historical raw tables.
ACID transactions on storage tier for safe concurrent reading of analytics dashboards and writing of pipelines.
Minimal maintenance cost leveraging stateless containerization and serverless orchestration rules.
Ingested Data Sources
Transactions Store
TypeOLTP Database
FormatCDC (Debezium/Avro)
Volume100K records/day
Inventory Database
TypeCatalog Store
FormatPostgreSQL Replica
Volume50K rows updated/day
Ad Campaign Telemetry
TypeThird-party webhook
FormatJSON webhook
Volume1M events/day
Pipeline Architecture Diagram
Architecture flow
project-specific system flow
100%
Ctrl/⌘ + scroll to zoom · + − 0
Interactive project architecture diagram.
End-to-End Pipeline Phases
01
Event Capture
Stateless API Gateways forward stream payloads directly into individual Kafka topic partition pools.
02
Stream Ingestion
Structured PySpark streams continuously poll Kafka topics, enforcing baseline parsing rules.
03
Schema Guard
Validation step checking structures against the centralized AVRO schema catalog before commit.
04
Medallion Lakehouse
Bronze (raw JSON), Silver (enriched schema), and Gold (aggregates) Delta tables write sequentially.
Data Warehouse Schema Design
Schema design
fact + dimension model
100%
Ctrl/⌘ + scroll to zoom
Interactive schema design.
Data Quality & Observability Metrics
Schema Compliance
99.98%
Target: 100% strictly enforced
Validation Pass Rate
100.0%
Zero silent failures allowed
SLA Adherence
99.95%
Pipeline uptime metric
Critical Engineering Tradeoffs
Spark Streaming over Flink
Context: Figma requirements dictated micro-batch latency was fully acceptable.
Choice Made: Spark Structured Streaming
Rationale: Dramatically lowered infrastructure overhead while allowing full codebase reuse with historical Python transformations.
Delta Lake over Parquet
Context: Concurrent write blocks occurred constantly with standard partition overrides.
Choice Made: Delta Lake Storage Format
Rationale: Enabled optimistic concurrency control with ACID transaction logs, guaranteeing zero dirty reads on analytics tools.
Pipeline Implementation & Syntax Snippets
pyspark_streaming_kafka.pyPYTHON
from pyspark.sql import SparkSession
from pyspark.sql.functions import from_json, col
# Initialize structured stream reading from broker...
df = spark.readStream.format("kafka") \
.option("kafka.bootstrap.servers", "broker:9092") \
.option("subscribe", "events") \
.load()Conclusion
Impact of the implementation
The case study demonstrates how a stream-first architecture can reduce analytical latency, isolate transactional workloads from reporting activity, and create a governed foundation for live operational analytics.
Take Aways
The strongest outcomes came from separating ingestion, validation, transformation and serving concerns, then making observability and schema enforcement first-class parts of the pipeline rather than afterthoughts.
“Design is a solution to a problem. Art is a question to a problem.” — John Maeda