EAGLE CORP
All posts

4 min readEagle Team

EAGLE-4 Delivers 2.7× Faster Decoding for NVIDIA Nemotron 3.5 Super VL in Preliminary Benchmarks

In preliminary single-request benchmarks on 4× NVIDIA B300 GPUs, EAGLE-4 reached 619 tokens/s on Nemotron 3.5 Super VL vs 229 with vanilla decoding.

  • EAGLE-4
  • NVIDIA
  • Nemotron
  • Speculative Decoding
  • Benchmarks
EAGLE-4 brings 2.7× faster decoding to NVIDIA Nemotron 3.5 Super VL

In preliminary text-only, single-request benchmarks on 4× NVIDIA B300 GPUs using BF16 precision, EAGLE-4 achieved 619 tokens/s for Nemotron 3.5 Super VL (opens in a new tab), compared with 229 tokens/s using vanilla decoding. That is 2.70× the decoding speed, or a 170% increase, in our preliminary single-request benchmark.

Bar charts comparing vanilla decoding at 229 tokens/s with EAGLE-4 at 619 tokens/s, and the time to produce 4,096 output tokens: 17.9 seconds versus 6.6 seconds.
Nemotron 3.5 Super VL, 2.7× faster with EAGLE-4 (preliminary, Sept 29).

Decoding speed, one request at a time (BF16)

MethodTokens/sSpeedup
EAGLE-46192.70×
Vanilla decoding2291.00×

Setup: 4× B300 (tensor parallel 4), batch size 1, SGLang 0.5.20, SpeedBench at temperature 0, up to 4,096 output tokens, prefill time excluded.

Where the speed comes from

Diagram of speculative decoding in three steps: a small draft model proposes 7 tokens, the full model verifies them in one forward pass, and the matching prefix is accepted while the first mismatch is corrected. Vanilla decoding yields 1 token per pass; EAGLE-4 averages 4.36 per step.
How speculative decoding works: draft, verify, accept. Illustration of the average from a fixed draft-length-7 measurement.

Vanilla decoding generates one token per forward pass of the full model. EAGLE-4 uses a small draft model to propose several tokens ahead. Nemotron verifies those proposals in a single pass, accepting the matching prefix and correcting the first mismatch. Accepting multiple tokens per verification step reduces the number of full-model passes needed to generate an answer.

In a fixed draft-length-7 measurement across 1,070 SpeedBench turns, EAGLE-4 keeps an average of 4.36 tokens per verification step. The 619 tokens/s result comes from a separate configuration that adapts the draft length at each step.

Horizontal bar chart of average accepted tokens per verification step by SpeedBench category at draft length 7: Coding 5.39, RAG 5.35, Multilingual 5.26, Summarization 5.18, Math 4.87, Reasoning 4.43, Humanities 3.85, STEM 4.14, Roleplay 3.74, QA 4.14, Writing 3.80, overall 4.36.
EAGLE-4 keeps 4.36 tokens per verification step, by SpeedBench category.

The benchmark covers 11 SpeedBench categories. In the fixed-length draft measurements, EAGLE-4's acceptance is highest in coding, RAG, multilingual tasks, and summarization:

  • Coding: 5.39 tokens per verification step
  • RAG: 5.35
  • Multilingual: 5.26
  • Summarization: 5.18
  • Math: 4.87

Creative writing and roleplay average approximately 3.8 tokens per verification step. These acceptance measurements describe drafting behavior; they are not category-level speedup ratios against vanilla decoding.

What 2.7× faster decoding means for applications

At the reported decoding rates, generating 4,096 output tokens would take approximately:

  • 17.9 seconds with vanilla decoding
  • 6.6 seconds with EAGLE-4

That is about 11.3 seconds less decoding time per response. For ten sequential responses of that length, the estimated decoding time falls from 179 seconds to 66 seconds. These estimates exclude prefill, network latency, tool execution, and other application overhead.

Same answers

At temperature 0, EAGLE-4 accepts only draft tokens that match the full model's greedy decoding decisions, preserving the vanilla decoding output. Nemotron's model weights remain unchanged; the acceleration comes from the serving process.

What's preliminary

  • Measured on September 29 using early access provided by NVIDIA.
  • These results use chain drafting. Tree drafting is still being tuned; updated results will follow.
  • Single request, batch size 1. Higher-batch performance has not yet been reported.
  • SpeedBench is text-only. These results do not measure image or multimodal workloads.

Ending notes

NVIDIA Nemotron 3.5 Super VL, accelerated with EAGLE-4: 2.70× the vanilla decoding speed in this preliminary benchmark.

Thanks to the NVIDIA Nemotron team for early access, which enabled this work.