Skip to content

Avoid redundant MTP prefill work - #196

Merged
jakmro merged 2 commits into
cactus-compute:mainfrom
VyasGuru:mtp-prefill-redundant-work
Nov 15, 2026
Merged

jakmro merged 2 commits into
cactus-compute:mainfrom
VyasGuru:mtp-prefill-redundant-work

Conversation

@VyasGuru

Copy link
Copy Markdown
Contributor

Summary

  • Embed each prompt chunk once and reuse those embeddings for the MTP draft cache as well
  • Skip vocabulary projection for intermediate prefill chunks
  • Apply lm_head once to the final prompt position

Testing

  • Added equivalence coverage for prompts shorter than, equal to, and not divisible by the chunk size, let me know if any more needed
  • identical seeds, final logits, hidden state, target cache, and draft cache as well
  • tests passed

Performance
on Apple M3, using four-token chunks and a 16K-vocabulary projection:

Prompt Before After Speedup
256 tokens 28.53 ms 11.48 ms 2.48×
1,024 tokens 117.90 ms 42.66 ms 2.76×

gains on full model behavour will of course be smaller, this benchmark only isolates the removed work.

VyasGuru and others added 2 commits November 14, 2026 18:42
Signed-off-by: VyasGuru <71374747+VyasGuru@users.noreply.github.com>
forward_hidden keeps its signature. Prefill never evaluates the per-chunk
logits, so they are never computed, and runs lm_head once on the last
prompt position.
@jakmro
jakmro merged commit 74a3e44 into cactus-compute:main Nov 15, 2026
1 check failed
@jakmro

jakmro commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

@VyasGuru thank you for this one

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants