# RTB Model Serving Platform

Factored built an ML serving platform for RTB that scaled throughput 20X, cut latency to 10ms, and improved reliability and cost.

## Key Takeaways:

#### Handling the Volume of Demand.

The ML ad serving platform had limited throughput (50,000 requests per second) and struggled with high latency. Additionally, there was a need for multi-framework support, robust monitoring, and data quality assurance in both the serving and feature store pipeline.

#### Using ML for Operational Efficiency & Insights.

- We enabled ML inference for multiple model development libraries like PyTorch and TensorFlow using NVIDIA Triton for flexible deployment.
- To enhance reliability, we integrated Prometheus and Grafana for real-time monitoring, improving error handling, automated recovery, and rate limiting to prevent overload.
- Performance was optimized through Grafana load testing, Golang MLServing enhancements, and Aerospike Cache integration.

#### Scaling Throughput 20X - 1M Per Second.

- Latency as low as 10ms.
- Reduced cost and better performance.
- More reliable and transparent ML operations.

### Continue Reading