Feature Request: KVarN support for RPC backend

[deorth-kku]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Please consider adding support for KVarN KV cache quantization to the RPC backend.

Motivation

I'm using commit a749684, v0.4.4 branch. Both for RPC server and client.
I would like to offload some layers to another PC via RPC.
on llama-server side, I do:

version = 1

[*]
flash-attn = on
cache-type-k = q8_0
cache-type-v = q8_0
no-warmup = true
parallel = 4
cont-batching = true
reasoning-budget-message = ... thinking budget exceeded, we will answer now.
jinja = true
kvu = true
backend-sampling = false
load-mode = mlock

[Qwen3.8-27B]
model = F:\models\Qwen3.8-27B-APEX-I-Mini.gguf
mmproj = F:\models\mmproj-Qwen3.8-27B-F16.gguf
rpc = qwen.lan:50052
mmdev = Vulkan1
ctx-size = 128000
ngl = all
t = 8
load-mode = none
ot = blk\.([0-7]|10|13|16|19|22|25|28|31|34|37)\.ffn_.*=RPC0[qwen.lan:50052]
threads = 16
temp = 0.6 
dynatemp-range = 0.2 
dynatemp-exp = 1.0 
top-p = 0.95 
top-k = 20 
min-p = 0.0 
image-min-tokens = 1024 
ubatch-size = 1024 
chat-template-file = F:\models\qwen3.8-chat_template.jinja
reasoning-preserve = true
cache-type-k = kvarn6
cache-type-v = kvarn5
spec-type = ngram-mod
spec-ngram-mod-n-match = 32 
spec-ngram-mod-n-min = 16 
spec-ngram-mod-n-max = 64

and on the rpc-server:
./build/bin/ggml-rpc-server -H 0.0.0.0 -p 50052

however, it does not work:

[63128] 0.31.190.039 E llama_init_from_model: cannot enable kvarn_k6v5_g128: KVarN requires a backend with KVarN store and materialization support
[63128] 0.31.190.043 E cmn  common_init_: failed to create context with model 'F:\models\Qwen3.8-27B-APEX-I-Mini.gguf'

It would be great to add RPC backend support for KVarN.

Possible Implementation

No response

[Jakdaw]

deorth-kku: does beellama.cpp work for you with a vulkan backend in the first place?

For me (AMD GPUs on Linux) it seems to half both the pp and tg rates, going from k=q8_0, v=bf16 -> k=v=kvarn6.

I've a patch that should add RPC support (it's just a case of plumbing in the existing logic -- tho it does mean adding several RPC commands, so I expect getting it upstreamed would be a faff), but if I can't get around the performance thing then I'm not going to get very far with testing it.

[deorth-kku]

deorth-kku: does beellama.cpp work for you with a vulkan backend in the first place?

For me (AMD GPUs on Linux) it seems to half both the pp and tg rates, going from k=q8_0, v=bf16 -> k=v=kvarn6.

I've a patch that should add RPC support (it's just a case of plumbing in the existing logic -- tho it does mean adding several RPC commands, so I expect getting it upstreamed would be a faff), but if I can't get around the performance thing then I'm not going to get very far with testing it.

I'm only using Vulkan for offloading mmproj to an iGPU to save some VRAM. The main model was run on 2 CUDA cards. The performance is mostly on par with mainline llama.cpp for me on CUDA.
I can help with testing.

[Jakdaw]

https://github.com/Jakdaw/beellama.cpp/tree/rpc-support

[deorth-kku]

https://github.com/Jakdaw/beellama.cpp/tree/rpc-support

@Jakdaw

Prefill seems to regress hard. My local agent haven't seem to found out why.

commit	kv cache type	pp/s	tg/s
ba27eda	q5_1/q4_1	645.43	14.80
5e3eae1	q5_1/q4_1	432.39	15.02
5e3eae1	kvarn5/4	408.49	20.56

My config changed a little since first post. But I don't think these changes matters.

[*]
flash-attn = on
cache-type-k = q8_0
cache-type-v = q8_0
no-warmup = true
parallel = 4
cont-batching = true
reasoning-budget-message = ... thinking budget exceeded, we will answer now.
jinja = true
kvu = true
t = 8
backend-sampling = false
ot = per_layer_token_embd\.weight=CPU
load-mode = mlock
#log-verbosity = 5

[Qwen3.8-27B]
model = F:\models\Qwen3.8-27B-UD-Q3_K_XL.gguf
mmproj = F:\models\mmproj-Qwen3.8-27B-F16.gguf
rpc = qwen.lan:50052
#mmdev flag was only merged on v0.4.4, so I have to disable it to tests on your branch.
#mmdev = RPC0
ctx-size = 131072
ngl = all
parallel = 1
cont-batching = false
kvu = false
t = 8
load-mode = mlock
#ot = blk\.([0-7]|10|13|16|19|22|25|28|31|34|37|40|43|46|49)\.ffn_.*=CPU
temp = 0.6 
dynatemp-range = 0.2 
dynatemp-exp = 1.0 
top-p = 0.95 
top-k = 20 
min-p = 0.0 
image-min-tokens = 1024 
ubatch-size = 512 
chat-template-file = F:\models\qwen_sharp_chat_template.jinja
reasoning-preserve = true
reasoning-budget = -1
cache-type-k = q5_1
cache-type-v = q4_1

Hardware:
Master: 3080ti 12GB CUDA13.3 Windows
RPC Worker: 2070Ti 8GB CUDA13.3 Debian

[deorth-kku]

It seems to be the overhead of real remote capability probing. My local agent made a patch for me and it does resolve the prefill regression.
However I do not have enough expertise to review the code, so consider this only a POC.
861a87b