Repository navigation
Add Edda ASR with DFlash2 speculative decoding (edda-v0.2-large-hurtig) - #1223
lkarlslund wants to merge 7 commits into
Conversation
Edda v0.2 large is a Danish Whisper-large-v3 fine-tune. The edda-v0.2-large-hurtig checkpoint adds a DFlash2 block-diffusion drafter; when present, each target pass verifies the anchor token plus up to 15 drafted tokens, keeping the longest agreeing prefix and the target's next token, so output matches greedy decoding. The target decoder runs block verification with KV slots indexed by position, taps hidden states from five decoder layers into the draft context, and the drafter runs as one GGML graph with the grouped dynamic causal convolutions and the candidate selector walked on the host.
The shared Whisper encoder materializes F32 attention scores for all 1500 frames. Packing QKV and running flash attention over F16 keys and values cuts the edda encoder from ~430 ms to ~285 ms per 30-second window on gfx1151 with identical transcripts. Also log drafter time.
List the hurtig and plain edda safetensors releases at pinned revisions plus the local GGUF conversion layout, and add the family to the README, ASR and license tables.
Beam search follows Transformers generate(num_beams): log-softmax before the suppress processors, top 2*num_beams continuations, length-normalized finished scores and the early_stopping=False heuristic. Beams share one KV cache as a token tree; each beam's newest token gets a fresh slot and its mask row admits only its ancestors, so pruning never copies or reorders the cache. Apply clean_up_tokenization_spaces like Transformers' decode, which fixes ' .' before punctuation (FLEURS parity 57/60 -> 59/60). The drafter now packs Q/K/V into one projection, reuses its K/V rows for the target context and uses flash cross-attention on CUDA/HIP.
Path cases cover speculative, greedy, argmax-draft, short-block and beam decoding plus Danish longform chunk modes. The WebUI entry uses the hurtig safetensors package; webui/native/dist is not rebuilt here because the committed bundle already lags its sources.
|
@lkarlslund Thanks for contributing a new model! One issue: with Update: found another issue --- corrupted drafter weights are silently ignored. A damaged drafter.safetensors still succeeds with |
…p without a drafter A drafter.safetensors that fails to parse or lacks fc.weight now throws instead of falling back to greedy decoding. A missing drafter file, or a GGUF converted without the drafter namespace, still decodes greedily. ResourceBundle gains has_tensor_source() to tell the two apart. With the drafter disabled the draft block is 1, so the clamp to [2, 1] no longer runs.
|
Thanks for catching these, both fixed in 4efd331:
Tested with a valid, garbage, truncated and missing drafter, a GGUF with and without the drafter, and |
Adds an experimental
edda_asrcommunity family for Edda v0.2 large, a Danish Whisper-large-v3 fine-tune, and for edda-v0.2-large-hurtig, which adds syv.ai's 170M-parameter DFlash2 drafter for lossless speculative decoding.What it does
da_dk(the model card reports 5.56).speculative=false.num_beams,length_penalty) following Transformers'_beam_search, using a token-tree KV cache so pruned beams need no cache copies.drafter.prefix.docs/models/edda_asr.md, model spec, model-manager packages, CLI path cases, WebUI catalog entry.The encoder is model-local rather than the shared Whisper embedding: it packs QKV and uses flash attention over F16 K/V, which took it from ~430 ms to ~285 ms on gfx1151.
Results (AMD Radeon 8060S, HIP, 11.3 s mean FLEURS clip)
Against PyTorch FP32 references: FLEURS 60 clips 59/60 identical for both speculative and greedy (the misses are near-ties on a name spelling, as the model card reports for fp16); beam search 10/10 on the clips where beam differs from greedy; NST 5/5 for every GGUF variant, beam search and the CPU backend; 180 s long-form at 0.0% WER. Full table in the docs.
Notes for review
crisperwhisper, because the drafter needs its own decoder loop and edda needs its own prompt tokens. Happy to restructure if you prefer otherwise.webui/native/distis not rebuilt: the committed bundle is already behind its sources, and a rebuild would pull unrelated changes into this PR.model_spec_system_testandtools/check_loader_catalog_sync.pypass.