Eval bug: E slot operator (): id 0 | task 793 | failed to restore speculative checkpoint transaction

[kirstenjg]

Name and Version

version: 11207 (ba27eda)
built with MSVC 19.51.36256.0 for x64

Operating systems

Windows

GGML backends

CUDA

Hardware

Ryzen 9900x & 5070 ti

Models

& 'C:\Users\xxx\source\beellama.cpp\build\bin\Release\llama-server.exe' --threads 9
--ctx-size 103424 --batch-size 1024
--ubatch-size 512 --flash-attn on
--cache-type-k q5_1 --cache-type-v q4_0
--load-mode none --gpu-layers all
--fit-target 512 --model 'C:\AI Models\unsloth\Qwen3.8-27B-GGUF\Qwen3.8-27B-UD-IQ3_XXS.gguf'
--temp 1.0 --top-k 20
--top-p 0.95 --min-p 0.0
--repeat-penalty 1.0 --presence-penalty 0.0
--spec-draft-n-max 3 --spec-type 'ngram-mod,draft-mtp'
--spec-ngram-mod-n-min 4 --spec-ngram-mod-n-max 8
--spec-ngram-mod-n-match 32 --ctx-checkpoints 96
--parallel 1 --mmproj 'C:\AI Models\unsloth\Qwen3.8-27B-GGUF\mmproj-BF16.gguf'
--no-mmproj-offload --host 0.0.0.0
--chat-template-kwargs '{"reasoning_effort":"medium"}' --jinja
--reasoning on

Problem description & steps to reproduce

0.36.752.450 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = 28331309
0.36.778.231 I slot launch_slot_: id 0 | task 793 | processing task, is_child = 0
0.37.556.879 W slot restore_chec: id 0 | task 793 | checkpoint restore preparation failed; destination is unchanged
0.37.556.886 E slot operator (): id 0 | task 793 | failed to restore speculative checkpoint transaction

I run an MTP-Bench which tests various coding languages. There is a python script for this somewhere. I did port it to my own application. But when I run it the first time it works fine. When I run it the 2nd time the server fails with the above logs.

First Bad Commit

No response

Relevant log output

Logs

0.36.752.450 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = 28331309 0.36.778.231 I slot launch_slot_: id 0 | task 793 | processing task, is_child = 0 0.37.556.879 W slot restore_chec: id 0 | task 793 | checkpoint restore preparation failed; destination is unchanged 0.37.556.886 E slot operator (): id 0 | task 793 | failed to restore speculative checkpoint transaction

[kirstenjg]

Note running the exact same llama-server command on llama.cpp does not have this issue

[kirstenjg]

same issue with:
version: 0.4.4-dev (build 11429, commit a749684)
built with MSVC 19.51.36256.0 for x64
ggml op count: 105

[kirstenjg]

Works when I remove ngram and keep MTP:
--spec-type 'ngram-mod,draft-mtp' --spec-ngram-mod-n-min 4
--spec-ngram-mod-n-max 8 --spec-ngram-mod-n-match 32

Something not working with ngram in beellama

[Anbeeld]

Will investigate.

[anthonyalayo]

I'm running into the exact same issue. The very first inference request fails with ngram:

0.01.821.691 I srv  llama_server: listening on http://0.0.0.0:8080
0.01.821.691 W srv  llama_server: NOTICE: server default port will be changed to :9931 in a future release
0.01.821.691 W srv  llama_server:         ref: https://github.com/ggml-org/llama.cpp/pull/26508
0.05.127.632 I slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
0.05.127.751 I slot launch_slot_: id  0 | task 0 | processing task, is_child = 0
0.05.710.299 W slot restore_chec: id  0 | task 0 | checkpoint restore preparation failed; destination is unchanged
0.05.710.303 E slot   operator(): id  0 | task 0 | failed to restore speculative checkpoint transaction
0.05.710.305 I slot      release: id  0 | task 0 | stop processing: n_tokens = 204, truncated = 0

[rafpigna]

Same issue here
But happens after a while, during generation, so the prefill+generation starts but get stopped due this error.
0.21.502.545 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = -1
0.21.502.654 I slot launch_slot_: id 0 | task 0 | processing task, is_child = 0
0.27.023.831 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 2048, progress = 0.07, t = 5.52 s / 370.97 tokens per second
...........
..........
1.21.497.921 I slot print_timing: id 0 | task 0 | n_decoded = 765, tg = 36.18 t/s, tg_3s = 35.53 t/s
1.24.500.554 I slot print_timing: id 0 | task 0 | n_decoded = 884, tg = 36.61 t/s, tg_3s = 39.64 t/s
1.25.407.949 W slot restore_chec: id 0 | task 0 | checkpoint restore preparation failed; destination is unchanged
1.25.407.961 E slot operator(): id 0 | task 0 | failed to restore speculative checkpoint transaction
1.25.407.962 I slot release: id 0 | task 0 | stop processing: n_tokens = 30945, truncated = 0

Here too I'm using mtp + ngram-mod + ngram-map-k4v
Confirmed that using mtp only, solves the issue.

Beellama 0.4.3
CUDA 12.4 - Windows 11
RTX3080 + Ryzen 9 5900x + 64GB Ram

[Anbeeld]

This should be fixed in current v0.4.4 by c7a6b18.

This repro was especially useful because it showed the checkpoint failure with ordinary q5/q4 KV and --parallel 1, which helped isolate it from KVarN. The speculative checkpoint restore path has since been reworked to prepare and validate target, draft, and speculative state before committing the restore.

Brief testing with ngram + MTP looks clean now. Please retest v0.4.4 release with the original command.

[rafpigna]

This should be fixed in current v0.4.4 by c7a6b18.

This repro was especially useful because it showed the checkpoint failure with ordinary q5/q4 KV and --parallel 1, which helped isolate it from KVarN. The speculative checkpoint restore path has since been reworked to prepare and validate target, draft, and speculative state before committing the restore.

Brief testing with ngram + MTP looks clean now. Please retest v0.4.4 release with the original command.

I can confirm that with the v0.4.4 the error is gone, but the new release has massive performance drop on generation speed, even on the baseline model without ngram activated.

v0.4.3
0.27.390.086 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 2048, progress = 0.06, t = 5.97 s / 343.32 tokens per second
0.30.117.352 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 4096, progress = 0.12, t = 8.69 s / 471.21 tokens per second
.....
1.03.244.460 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 34944, progress = 1.00, t = 41.82 s / 835.60 tokens per second
1.07.391.660 I slot print_timing: id 0 | task 0 | n_decoded = 108, tg = 35.76 t/s, tg_3s = 35.75 t/s
.....
1.34.571.006 I slot print_timing: id 0 | task 0 | n_decoded = 947, tg = 31.36 t/s, tg_3s = 33.18 t/s
1.35.280.684 I slot print_timing: id 0 | task 0 | prompt eval time = 42944.90 ms / 35024 tokens ( 1.23 ms per token, 815.56 tokens per second)
1.35.280.690 I slot print_timing: id 0 | task 0 | eval time = 30909.15 ms / 966 tokens ( 32.00 ms per token, 31.25 tokens per second)
1.35.280.691 I slot print_timing: id 0 | task 0 | total time = 73854.04 ms / 35990 tokens

v0.4.4 - same model, same flags, same enviroment (launched just 5 seconds after the v0.4.3 completed) and the same byte-identical prompt:

0.27.484.780 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 2048, progress = 0.06, t = 5.90 s / 347.39 tokens per second
0.30.259.928 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 4096, progress = 0.12, t = 8.85 s / 462.90 tokens per second
.....
1.03.673.146 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 34944, progress = 1.00, t = 42.24 s / 827.19 tokens per second
1.08.748.497 I slot print_timing: id 0 | task 0 | n_gen = 102, tg = 25.69 t/s, tg_3s = 25.90 t/s
.....
1.23.973.568 I slot print_timing: id 0 | task 0 | n_gen = 447, tg = 23.28 t/s, tg_3s = 21.22 t/s
1.24.598.011 I slot print_timing: id 0 | task 0 | prompt eval time = 43425.70 ms / 35024 tokens ( 1.24 ms per token, 806.53 tokens per second)
1.24.598.017 I slot print_timing: id 0 | task 0 | eval time = 19781.80 ms / 463 tokens ( 42.82 ms per token, 23.35 tokens per second)
1.24.598.018 I slot print_timing: id 0 | task 0 | total time = 63207.49 ms / 35487 tokens

v0.4.4 is ~ -10tok/s on generation speed compared to v0.4.3

[Anbeeld]

@rafpigna send your full launch config please.