Real-Time Ad Data Streaming

Factored built a real-time data streaming platform that scaled to millions of requests, cut latency, and boosted ad performance.

Key Takeaways:

Improving Performance in Real-Time.

The advertisement platform needed to transition from batch processing to a real-time, multi-tenant data streaming platform to support real-time machine learning (ML) and analytics. This new system had to handle massive data volumes, scale efficiently, reduce latency, and ensure high availability while providing self-service capabilities for different teams.

A Multi-Stage Solution.

We implemented a Kafka-based data streaming platform, scaling to 1M requests/sec and 100TB daily.

• Trino, Flink, Vertica, and ClickHouse enabled real-time analytics, while self-managed Kubernetes Airflow improved orchestration across 40 teams.
• A self-service platform allowed teams to deploy pipelines with 24/7 monitoring. <10ms latency for Kafka-to-Aerospike transfers ensured real-time bidding.
• Custom monitoring, metadata tracking, and open-source enhancements improved governance, visibility, and platform stability.

Improved Data Streaming Capacity 10x.

• Reducing operational overhead.
• Achieved ultra-low latency (sub-10ms) for real-time bidding and streaming analytics.
• Enhanced metadata and lineage tracking.

Continue Reading

Governed Metrics for AI

Single logic across every interface

Compliance Reporting

Reporting automated

Audience Segmentation

Targeting efficiency improved