Introducing EAGLE-4: One Flexible Drafter for All Batch Sizes and Hardware

Today we're launching EAGLE-4: One Flexible Drafter for All Batch Sizes and Hardware. One trained drafter per target model adapts from single requests to busy servers, across GPUs and local CPUs.

EAGLE-4 builds on EAGLE-1, EAGLE-2 and EAGLE-3 with hardware-aware control over drafting and a more efficient LM head. One trained drafter per target model adapts its speculation budget to the hardware and workload, from a single request to a server handling hundreds at once.

On NVIDIA Nemotron 3.5 Super VL, EAGLE-4 reaches 647 tokens/s for one request and 33,480 aggregate tokens/s at 512 concurrent requests: 2.74× and 1.69× vanilla throughput, respectively. Both measurements use four NVIDIA B300 GPUs, TP4, at temperature 0.

EAGLE-4 Is Here. GitHub (opens in a new tab) · Paper (opens in a new tab) · eagle.ai (opens in a new tab)

Addressing AI's infrastructure bottleneck

One of AI's biggest bottlenecks is the gap between demand for inference and the compute and electrical power available to deliver it. Reasoning models and multi-step agents raise the work required per task, while operators face long lead times for new hardware, power equipment and grid connections.

The scale is measured in trillions: at Dell Technologies World 2026, Michael Dell projected that worldwide AI infrastructure spending could reach $3–4 trillion by 2030 (opens in a new tab). The IEA's 2026 analysis (opens in a new tab) describes tightening bottlenecks in energy supply chains and advanced chip manufacturing, alongside slow grid connections that constrain new data centre capacity.

Meanwhile, the installed base faces a moving performance target. Each new model generation and more demanding workload raises expectations for hardware already in service. Operators need to keep that investment productive as newer accelerators arrive.

EAGLE-4 addresses the software side of this capacity problem. It reduces repeated target-model execution and adapts drafting to the hardware and workload, helping supported deployments produce more useful output from available compute. Our goal is to extend the productive life of existing infrastructure and improve the return on new systems, with gains measured in throughput, response time and, in the phone evaluation, energy per token.

Qwen: one drafter from 1 to 512 requests

The paper evaluates the same trained Qwen3.8-27B drafter at concurrency 1 and 512. EAGLE-4 reaches 863.8 tokens/s for a single request and 28,340 aggregate tokens/s at 512 concurrent requests. The four-way comparison below uses the same inference engine with MegaKernel disabled for every method.

At concurrency 1, EAGLE-4 delivers 3.59× vanilla, 1.28× DFlash 2 and 1.40× DSpark throughput. At concurrency 512, those gains are 1.34×, 1.70× and 1.99×, respectively. The high-concurrency rates measure total output across all requests.

Both settings use the same trained drafter. EAGLE-Harness changes the runtime drafting configuration as the cost of speculation changes with load.

Accelerating NVIDIA Nemotron

On NVIDIA Nemotron 3.5 Super VL, the 120B-A12B target model, EAGLE-4 reaches 2.74× vanilla throughput for a single request. At 512 concurrent requests, it sustains 33,480 aggregate tokens/s, a 1.69× gain over vanilla decoding.

The throughput gains extend beyond Qwen to a large mixture-of-experts target. EAGLE-4 adjusts the amount of speculative work while keeping the target-model weights unchanged.

Day-0 acceleration on AMD Instinct MI350X

The AMD team provided day-0 support and evaluated EAGLE-4 on a single AMD Instinct MI350X GPU with tensor parallelism 1. On a 117-turn SPEED-Bench workload, single-request throughput rises from 75.27 to 262.89 tokens/s, a 3.49× speedup over vanilla decoding.

The gains persist across every tested concurrency from 1 to 32. At 16 concurrent requests, EAGLE-4 delivers 2.30× vanilla throughput. At 32, it reaches 2,360.83 aggregate tokens/s versus 1,390.52 for vanilla, a 1.70× improvement. The evaluation caps each request at 4,096 newly generated tokens.

EAGLE-4 on TokenSpeed

SPEED-Bench performance versus the SGLang reference

The benchmark reported in PR #3 (opens in a new tab) reaches 961.6 tokens/s for a single request at temperature 0, versus 837.6 tokens/s for the bundled SGLang reference: a 1.15× speedup. Both implementations run EAGLE-4 with the same EAGLE-Harness dynamic tree. TokenSpeed is faster in all three reported configurations, with similar mean accepted lengths.

The comparison uses the same machine, day, client and data: four NVIDIA B300 GPUs, TP4, Qwen3.8-27B in BF16, and the same EAGLE-4 drafter. Both runs use a 12/4/16 dynamic tree, execution mode 0, the high_intensity setting and shared profiled cost data.

The multi-turn SPEED-Bench test uses 88 conversations and 117 turns, thinking enabled with xhigh reasoning effort, up to 4,096 tokens per turn and seed 721. Throughput is generated tokens divided by wall-clock time. The PR reports bitwise regression checks against TokenSpeed before the changes on the tested prompts, plus matching host/device EAGLE-Harness policy outputs at temperatures 0 and 1.

Long-context results

The TokenSpeed team completed day-0 integration and measured Qwen3.8-27B in BF16 with an EAGLE-4 drafter on four NVIDIA B300 GPUs, TP4, using one request at a time.

Single-request decode throughput stays between 759 and 822 tokens/s across contexts from 32K to 250K tokens: 759 at 32K, 822 at 128K and 761 at 250K. Time to first token is 0.55, 2.1 and 5.7 seconds, respectively.

Useful speedups on a 4-vCPU server

For local inference, we evaluated 4-bit Gemma-4-E2B-it on an AWS c5.xlarge instance with 4 vCPUs and 8 GB RAM, using llama.cpp. EAGLE-4 improves throughput across coding, mathematical reasoning and open-ended conversation.

The mean of the four benchmark speedups is 2.71×. In the same evaluation, EAGLE-3 delivers 1.25×–1.65× vanilla throughput; EAGLE-4 raises that range to 2.28×–2.92×.

Fast inference on a phone CPU

We then ran the same Gemma target and drafter on an unmodified, non-rooted Motorola Razr 70 Ultra. The device has a Snapdragon 8 Elite, eight CPU cores and 16 GB RAM. The full inference stack runs on the CPU through llama.cpp using six threads.

On HumanEval, throughput rises from 22.8 to 82.6 tokens/s, a 3.62× speedup. Long-form generation rises from 23.3 to 74.7 tokens/s, or 3.21× vanilla. Time to first token on HumanEval falls from 1.2 seconds to 0.8 seconds.

In the long-form test, energy per generated token drops from 356 to 166 mJ, a 53.4% reduction measured at the USB input. The generation completes in 14 seconds instead of 43, finishing before thermal throttling begins. These throughput measurements start with a cool device; the paper discusses throttling during longer continuous runs.

A small memory footprint

The Gemma EAGLE-4 drafter occupies 134 MB. Total additional resident memory during generation, including its KV cache and computation buffers, is 0.27 GB on the CPU server and 0.26 GB on the phone.

The Qwen drafter also uses a single Transformer layer. The paper's Table 1 reports 0.44B Transformer-layer parameters for EAGLE-4, compared with 1.66B for DFlash 2 and 1.65B for DSpark, each with five layers. These counts exclude feature fusion, token embeddings and output-vocabulary layers.

The phone implementation also optimizes the ARM CPU path with a persistent thread pool, vector arithmetic and a small-batch verification attention kernel. Together with the output-head changes, the reported optimizations reduce draft-step latency from 8.7 ms to 3.9 ms.

What changed in EAGLE-4

EAGLE-Harness adapts speculation to its cost

A draft tree that is useful at low concurrency can become expensive under load. Wider or deeper trees may accept more tokens, but they also add drafting and verification work. EAGLE-Harness accounts for both costs when deciding how far to speculate.

A warm-up profiles the draft and target models across batch sizes, context lengths and speculative token counts. Those measurements are cached in lookup tables. During generation, EAGLE-Harness combines the cost estimates with draft confidence to choose tree depth, width and verification budget.

Autoregressive drafting makes these decisions possible before the next draft step is executed. Each branch can use its own drafted prefix to decide whether further work is worthwhile. The objective is the expected number of useful tokens advanced per unit of drafting and verification cost.

EAGLE Efficient LM Head reduces drafting overhead

In the paper's profiled configuration, computing full-vocabulary logits accounts for 65.4% of a 574 μs draft forward pass. EAGLE Efficient LM Head reduces that cost in two stages: a low-rank head retrieves candidates across the full vocabulary, then the original full-rank head computes exact logits for the shortlisted tokens.

The candidate set changes with the hidden state, allowing retrieval across the full vocabulary at each step. The full LM head is retained, and its selected rows supply the final candidate scores.

Attention and cache management for long contexts

The design also addresses long-context drafting costs. It considers sparse and sliding-window attention to limit how much history each draft step processes, and uses append-only cache updates within each drafting phase. The drafter's feed-forward architecture is aligned with the target model, balancing generalization, memory and training cost.

The evolution from EAGLE-1 to EAGLE-4

EAGLE-4 is the next step in a research programme that began with EAGLE-1 in January 2024. As documented on eagle.ai (opens in a new tab) and in the linked papers, each generation has addressed a different constraint on inference acceleration.

EAGLE-1 (January 2024): drafting from model features

The original EAGLE introduced feature-level autoregression: a lightweight drafter predicted the target model's internal features, then used its LM head to propose tokens. A token sequence shifted one step ahead helped resolve uncertainty in feature prediction. This established the foundation for efficient drafting with target-model verification.

EAGLE-2 (June 2024): context-aware draft trees

EAGLE-2 made the draft tree adaptive. It used draft confidence to estimate which tokens were likely to be accepted, then adjusted the tree structure to the current context. This increased accepted work per verification pass without training a separate model to predict the tree.

EAGLE-3 (March 2025): scaling draft quality with data

EAGLE-3 moved from feature prediction to direct token prediction and combined features from multiple target-model layers. Its training-time test procedure exposed the drafter to its own multi-step predictions during training. These changes let the drafter benefit more from larger training datasets, with evaluation across both chat and reasoning models.

EAGLE-4: adapting to execution cost

EAGLE-4 carries this progression into serving conditions. EAGLE-Harness chooses drafting depth, width and verification budget using profiled costs and draft confidence, while EAGLE Efficient LM Head reduces vocabulary-scoring overhead. The focus is useful acceleration across request loads and hardware with one trained drafter per target model.

The EAGLE family already appears in inference work from NVIDIA, Meta and Perplexity. With EAGLE-4, we're extending that research toward a wider range of serving loads and devices, while keeping a lightweight, single-layer Transformer drafter.

Benchmark setup

The paper's NVIDIA GPU comparisons use the SPEED-Bench qualitative split, four NVIDIA B300 GPUs, tensor parallelism 4 and a maximum output length of 4,096 tokens. The Qwen comparison uses the same inference engine with MegaKernel disabled. Its concurrency-512 evaluation repeats the dataset four times, producing 3,520 conversations to keep the batch full. EAGLE-4 uses a 12-4-16 tree at concurrency 1 and a 15-1-16 chain at concurrency 512.

The paper's headline NVIDIA GPU comparisons, CPU tests and phone tests use temperature 0. At temperature 1, its Qwen comparisons reach 2.59× vanilla at concurrency 1 and 1.03× at concurrency 512; Nemotron reaches 2.40× and 1.51×, respectively. The newer TokenSpeed integration table includes both T=0 and T=1; its SGLang reference also uses EAGLE-4. Speedup depends on the model, workload and sampling configuration.

The CPU and phone evaluations use 4-bit Gemma-4-E2B-it, llama.cpp and a single request at temperature 0. Phone throughput is measured on a cool device. Energy figures refer specifically to the long-form test and are measured at the USB input.

The target-model weights stay unchanged. With exact greedy verification at temperature 0, the accepted draft prefix matches the target model's greedy predictions, preserving its output.

Thank you to our collaborators

Thank you to our partners NVIDIA, AMD, GMI Cloud, LightSeek, fal, Fireworks, vLLM, SGLang, and others, and to the researchers, engineers and contributors helping bring EAGLE-4 to developers.

Explore the release

EAGLE-4 Is Here. Read the paper, explore the implementation and evaluate it on your own models, hardware and request loads.

• GitHub: github.com/eagle-hq/EAGLE-4 (opens in a new tab)
• Paper: EAGLE4_Paper.pdf (opens in a new tab)
• Website: eagle.ai (opens in a new tab)
• vLLM: vLLM (opens in a new tab)
• SGLang: SGLang (opens in a new tab)
• TokenSpeed: TokenSpeed (opens in a new tab)
• Qwen3.8-27B: HF EAGLE-Corp/Qwen3.8-27B-EAGLE4 (opens in a new tab)
• Nemotron-3.5-Super-VL-120B-A12B: HF EAGLE-Corp/Nemotron-3.5-Super-VL-120B-A12B-EAGLE4 (opens in a new tab)

If you're building an inference platform or an on-device application, we'd like to hear what you're running and what you build with EAGLE-4.