Feature Request: Vulkan Flash Attention for q6

[crusaderky]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

On Vulkan, -ctk q6_0 -ctv q6_0 runs on CPU, with abysmal performance. Please move it to GPU.

Motivation

KLD sweeps on most models show negligible quality degradation from q8_0/q8_0 to q6_0/q6_0. It would be nice to be able to use it on Vulkan. However, at the moment q6_0/q6_0 runs on CPU, which causes speed to implode.

Ling-3.0-tiny does not support kvarn; I understand it's due to its geometry; so kvarn6 is not an option on the table.

hf = bartowski/Ling-3.0-tiny-GGUF:Q6_K
ngl = 99
jinja = true
flash-attn = on
ctx-size = 262144
cache-type-k = XXX
cache-type-v = XXX
kv-unified = true

All builds from https://github.com/Anbeeld/beellama.cpp/releases/tag/preview-v0.4.4 @ a749684
On RTX 3090:

Backend	KV quant	decode tok/s
CUDA	q8_0/q8_0	216
CUDA	q6_0/q6_0	209
Vulkan	q8_0/q8_0	186
Vulkan	q6_0/q6_0	37

On Strix Point laptop:

Backend	KV quant	decode tok/s
ROCm	q8_0/q8_0	Fails coherence test[1]
ROCm	q6_0/q6_0	Fails coherence test[1]
Vulkan	q8_0/q8_0	53
Vulkan	q6_0/q6_0	14

[1] The model repeats a single word over and over again. This seems to be specific to Ling-3.0-tiny; Qwen3.8-27B is unaffected.

Possible Implementation

No response

[Anbeeld]

Thanks for the report, will do.