Skip to content

Add Edda ASR with DFlash2 speculative decoding (edda-v0.2-large-hurtig) - #1223

Open
lkarlslund wants to merge 7 commits into
0xShug0:mainfrom
lkarlslund:feat/edda-asr
Open

lkarlslund wants to merge 7 commits into
0xShug0:mainfrom
lkarlslund:feat/edda-asr

Conversation

@lkarlslund

Copy link
Copy Markdown
Contributor

Adds an experimental edda_asr community family for Edda v0.2 large, a Danish Whisper-large-v3 fine-tune, and for edda-v0.2-large-hurtig, which adds syv.ai's 170M-parameter DFlash2 drafter for lossless speculative decoding.

What it does

  • Speculative decoding. The drafter proposes up to 15 tokens per step from edda decoder layers 1/8/15/22/29, its own cross-attention and DFlash2's grouped dynamic causal convolutions and candidate selector; edda verifies the block in one pass. 5.24 tokens per edda pass on FLEURS da_dk (the model card reports 5.56).
  • Greedy decoding for the plain edda release, or with speculative=false.
  • Beam search (num_beams, length_penalty) following Transformers' _beam_search, using a token-tree KV cache so pruned beams need no cache copies.
  • Silence-based chunking for long audio, pull-streaming output per chunk, server support, GGUF conversion with the drafter under a drafter. prefix.
  • Docs in docs/models/edda_asr.md, model spec, model-manager packages, CLI path cases, WebUI catalog entry.

The encoder is model-local rather than the shared Whisper embedding: it packs QKV and uses flash attention over F16 K/V, which took it from ~430 ms to ~285 ms on gfx1151.

Results (AMD Radeon 8060S, HIP, 11.3 s mean FLEURS clip)

Mode Decoder Total
Greedy, F16 597 ms 862 ms
Speculative, F16 193 ms 519 ms
Speculative, Q8_0 decoder + F16 encoder 152 ms 428 ms
Beam 5, F16 929 ms 1234 ms

Against PyTorch FP32 references: FLEURS 60 clips 59/60 identical for both speculative and greedy (the misses are near-ties on a name spelling, as the model card reports for fp16); beam search 10/10 on the clips where beam differs from greedy; NST 5/5 for every GGUF variant, beam search and the CPU backend; 180 s long-form at 0.0% WER. Full table in the docs.

Notes for review

  • It is a separate family rather than an extension of crisperwhisper, because the drafter needs its own decoder loop and edda needs its own prompt tokens. Happy to restructure if you prefer otherwise.
  • No audio.cpp GGUF is published yet, so the default package is marked "convert locally".
  • webui/native/dist is not rebuilt: the committed bundle is already behind its sources, and a rebuild would pull unrelated changes into this PR.
  • Not tested on CUDA (the test card was fully occupied).
  • model_spec_system_test and tools/check_loader_catalog_sync.py pass.

Edda v0.2 large is a Danish Whisper-large-v3 fine-tune. The
edda-v0.2-large-hurtig checkpoint adds a DFlash2 block-diffusion
drafter; when present, each target pass verifies the anchor token
plus up to 15 drafted tokens, keeping the longest agreeing prefix and
the target's next token, so output matches greedy decoding.

The target decoder runs block verification with KV slots indexed by
position, taps hidden states from five decoder layers into the draft
context, and the drafter runs as one GGML graph with the grouped
dynamic causal convolutions and the candidate selector walked on the
host.
The shared Whisper encoder materializes F32 attention scores for all
1500 frames. Packing QKV and running flash attention over F16 keys and
values cuts the edda encoder from ~430 ms to ~285 ms per 30-second
window on gfx1151 with identical transcripts. Also log drafter time.
List the hurtig and plain edda safetensors releases at pinned
revisions plus the local GGUF conversion layout, and add the family
to the README, ASR and license tables.
Beam search follows Transformers generate(num_beams): log-softmax
before the suppress processors, top 2*num_beams continuations,
length-normalized finished scores and the early_stopping=False
heuristic. Beams share one KV cache as a token tree; each beam's
newest token gets a fresh slot and its mask row admits only its
ancestors, so pruning never copies or reorders the cache.

Apply clean_up_tokenization_spaces like Transformers' decode, which
fixes ' .' before punctuation (FLEURS parity 57/60 -> 59/60).

The drafter now packs Q/K/V into one projection, reuses its K/V rows
for the target context and uses flash cross-attention on CUDA/HIP.
Path cases cover speculative, greedy, argmax-draft, short-block and
beam decoding plus Danish longform chunk modes. The WebUI entry uses
the hurtig safetensors package; webui/native/dist is not rebuilt here
because the committed bundle already lags its sources.
@0xShug0

0xShug0 commented Nov 6, 2026 •

Copy link
Copy Markdown
Owner

@lkarlslund Thanks for contributing a new model! One issue: with edda_asr.drafter=false, the code still calls std::clamp with an invalid range: minimum 2, maximum 1. The normal release build works, but the code can crash when C++ standard-library assertions are enabled. Please skip the clamp calculation when the drafter is disabled and use block size 1 directly.

Update: found another issue --- corrupted drafter weights are silently ignored. A damaged drafter.safetensors still succeeds with drafter=true, reporting drafter=absent and switching to greedy. It should throw an exception to make the error explicit.

…p without a drafter

A drafter.safetensors that fails to parse or lacks fc.weight now throws
instead of falling back to greedy decoding. A missing drafter file, or a
GGUF converted without the drafter namespace, still decodes greedily.
ResourceBundle gains has_tensor_source() to tell the two apart.

With the drafter disabled the draft block is 1, so the clamp to [2, 1]
no longer runs.
@lkarlslund

Copy link
Copy Markdown
Contributor Author

Thanks for catching these, both fixed in 4efd331:

  • With the drafter disabled the draft block is now 1 directly, so the clamp to [2, 1] never runs.
  • A drafter.safetensors that can't be parsed or lacks fc.weight now throws ("Edda could not read the drafter weights: …") instead of reporting drafter=absent. A missing drafter file, or a GGUF converted without the drafter/ namespace, still decodes greedily, since that's the plain edda release. To tell the two apart I added a small ResourceBundle::has_tensor_source().

Tested with a valid, garbage, truncated and missing drafter, a GGUF with and without the drafter, and drafter=false.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants