• At Meta’s scale, even a few milliseconds of latency degradation can have a significant negative impact on ad performance.
• When a Linux kernel upgrade threatened to reduce latency across Meta’s ad serving fleet, we turned to sched_ext – the upstream, BPF-based extensible scheduling framework – to create a scheduling policy tailored to the ad serving workload.
• The result: ある 28% reduction in the ad retrieval phase (99th percentile) of latency, ある 3.28 megawatt (MW) energy savings, そして 1.1% increase in the number of ads displayed, proving that workload-specific scheduling optimization can directly drive business value.

Meta’s ad serving fleet processes an average of more than 5 million requests per second at the delivery platform entry point, which equates to over 400 billion per day across all monetized surfaces¹. Every millisecond shaved from p99 latency makes ads more relevant to people on our platforms, and better matches mean greater ROI for advertisers.

This presents a real opportunity to reduce latency through workload-specific scheduling. That’s why our Ads and Linux Kernel teams worked together to create a scheduling policy tailored to the ad serving workload using sched_ext, the upstream, BPF-based extensible scheduling framework. So far we have used the schedulers commonly built into the Linux kernel (CFS and EEVDF), which distribute threads across CPUs without knowing the workload. しかし, here we know the purpose and importance of each thread. With sched_ext we can encode this knowledge directly into the scheduler. Work to improve p99 request latency is planned first, everything else takes a back seat.

sched_ext at Meta

Sched_ext is an open source BPF-based scheduler framework officially introduced in kernel v6.12. We developed it in collaboration with the authors at Google ghOSt to design a scheduler suitable for upstream Linux integration. It has already been used in several services at Meta and provides a significant reduction in scheduling latency.

As we upgrade our fleet to the latest stable version of Linux (Kernel v6.9) we observed that the new Earliest Eligible Virtual Appointment First (EEVDF) Scheduler introduced in Linux kernel v6.6 caused a latency regression that reduced the number of ads classified as responsive. 結果として, a portion of ad hosts were forced to remain on the older v6.4 kernel, resulting in technical debt and operational fragmentation.

Given its already strong performance, sched_ext was a great candidate to handle these scheduling regressions.

Custom scheduling with sched_ext

Sched_ext allows scheduler developers to implement their preferences Planning policy as a BPF program. When a host begins running the ad workload, an ad-optimized policy is applied. From this point on, the kernel calls the BPF scheduler through a series of event-driven callbacks to handle common scheduling events such as:

• Thread wakeup: Select a CPU when a thread becomes executable.
• Queue: Place a thread in an execution queue.
• 送信: Select the next thread when a CPU becomes idle.
• Idle transitions: Respond to CPUs entering and exiting idle states.

Politics at a high level Softly distributes CPUs into two poolsone for threads on the latency-critical request path and one for less latency-sensitive work. Which thread goes into which pool is part of the domain-specific knowledge encoded in the policy. The size of each pool is dynamically adjusted using load-based heuristics. This approach tends to keep related work on the same CPUs over time, improving it Locality of the last level cache (L3). and reducing costly DRAM access.

The policy is packaged as a user space binary that loads the BPF program. This design significantly speeds up experimentation and performance optimization. To roll out a change, we can simply restart the scheduler process to unload the old policy and load the new one without rebuilding or installing the kernel.

The first start There was a transition from kernel 6.4 with the CFS scheduler to kernel 6.9 with sched_ext on the server type with the largest ad serving. Based on the backtest experiment, the introduction provided:

• +1.1% in weighted ad ranking (Metric for the number of ads viewed and ranked).
• 3.28 megawatts of energy savings across the fleet.
• 28% reduction in service P99 latency on the ad retrieval path².

Compound Improvements. Two subsequent scheduler policy updates, delivered as user-only changes, extended the victory:

• Additional 60% reduction in service p99 latency.
• 18% reduction in critical path timeout errors.

This is a non-trivial gain provided without dependency on kernel releases. Each of the subsequent iterations above shipped in days instead of months because the scheduler policy runs as a BPF program in user space. This rhythm has transformed sched_ext from a “kernel upgrade unblocker” to a continuous optimization platform for ad delivery.

What began as a targeted response to a very specific operational problem has proven to be much more strategic and broadly applicable than we originally expected. sched_ext offers Meta some important advantages:

A parallel and decoupled scheduler optimization path. Upstream Linux planning naturally evolves over time, sometimes in larger increments (例えば. transitioning from CFS to EEVDF), which can be disruptive to downstream consumers. sched_ext gives Meta the flexibility to continually improve these custom schedulers in parallel with this development. We maintain and refine our own BPF-based scheduling logic tailored to the unique needs of our production workloads, so our critical services remain optimized regardless of what happens upstream.

Independent deployment and reduced overhead costs. Planner improvements are delivered as BPF program updates, in days rather than months. The resulting reduction in experimentation costs is transformative. Ideas that previously required a kernel patch and months of validation – local cache-aware placement, ROI-based executor routing, NUMA-aware control – become manageable iterations rather than large-scale projects.

A shared industry value. sched_ext was upstreamed in Linux v6.12, so the Meta mechanism used here is now available to the entire Linux ecosystem. Any operator with a workload that doesn’t fit the general model – hyperscaler, cloud provider, embedded systems team – can provide workload-specific scheduling policies without forking the kernel.

sched_ext already allows us to identify opportunities for further improvements in ad performance by giving the application more granular control over the scheduler’s behavior. 例えば, ad services have important context about the relative importance of service requests and may be able to signal the scheduler when a thread begins work on an important request. When the planner receives this notice, he or she can take appropriate steps, のような: B. increasing the size of this thread’s scheduling section or ensuring that it is always at the top of the queue.

Special thanks goes to Samuel Nair, Usama Arif, GP Musumeci, Praveen Alevoor, Ye Wang, and the broader Ads Capacity Efficiency and Kernel team for their contributions and collaboration.

Ads Infra Leadership Team: ウラジミール・パシュケヴィチ, ヴァルナ プヴァダ, Prabhakar Goyal, Neeraj Agrawal, Tak Yan, Liz Shepherd, Drew Lackman