QAT Models supported?

[FabianSchuetze]

Thanks for the wonderful repo. It's a pleasure to work with it.

Are the new 4-bit quantized models for gemma 3 supported by gemma.cpp? Judging from the press-release https://developers.googleblog.com/en/gemma-3-quantized-aware-trained-state-of-the-art-ai-to-consumer-gpus/ and looking through the gemma.cpp repo, it doesn't seem like it, but I am bit surprised about it.

[jan-wassenberg]

Thanks :) Glad it's useful. I'm curious what you are using it for?

Gemma3 was actually also QAT-ed for our SFP format. But that is 8-bit and I understand 4 bit would be useful. We have several areas of investigation including non-uniform 4.5 bit (NUQ), which is not what the most recent Gemma3 QAT uses, and partial support for int4 which was IMO not very appealing until int4 QAT weights were available.

FYI there are major changes to our weight handling/importer in progress which would help. After that lands, we can discuss potentially supporting the int4.

[FabianSchuetze]

Thanks for your kind and detailed reply, Jan!

I'm personally using the small models from gemma.cpp for education and fun. We've been looking into running small models on embedded targets one or two years ago at work, but the relationship between the hardware capabilities and the model's utility wasn't in our favor.

Although the switched fp8 format is impressive, I find the int format slightly more interesting, because it's widely supported by accelerators. Int4 seems appealing as it would allow reducing the memory traffic for the weights, while still working on accelerators that support int8 computations.

Btw: Are the optimization sections in #164 still relevant? Maybe I could try my luck with an item.

[jan-wassenberg]

Got it, thanks for sharing.
Personally, I find accelerator HW is not a great fit for what we'd ideally want to do. Particularly 2:4 and scalar uniform quant are a bit too far towards "HW convenience at the expense of SW/utility". But I acknowledge they are useful in the short term.

I've updated #164. Maybe biggest low-hanging fruit would be converting the bf16 embedding into sfp. But that is best done after the weight change lands. Anything else that might interest you?

[FabianSchuetze]

Thanks for the detailed and informative reply.

I am most interested in vectorization and optimization and would be delighted to contribute something in this regard. Given your expertise in that subject, I thought that would be fun :-)

The roadmap mentions RoPE vectorization. What is precisely meant by this? The RopeAndMulBy is vectorized and contributes only slightly to the total inference costs according to the profiler.

I didn't understand when the normal Rope function gets called. According to the docs, it gets called from backprop, but I wasn't aware of the ability to train a Gemma model with gemma.cpp. I also set the kUseHalfRope option manually, but that produced funky results.

[jan-wassenberg]

Nice! Happy to collaborate :)
kUseHalfRope is set by the Griffin SSM; the model has to be trained for that, so it does not make sense to just enable it.
backprop/ could be useful for fine-tuning in future.

One tiny and low-hanging fruit is changing some activations to bf16.
This is particularly useful if you have a CPU such as M3 or Sapphire Rapids or Zen4 that support bf16 natively.
Even if not, we might still see a small win due to less memory usage.
Most are MatMul outputs which currently must be f32. And att_out/att should probably stay float.
But pre_att_rms_out could be changed to bf16 and I'm curious what difference that makes.

For vectorization, Rope() is called for Griffin. Like you say, not a huge cost.
LayerNormBatched in ViT/Gemma3 and AddAbsolutePositionalEmbeddings might be more useful to vectorize.
Would be happy to advise on that. One super important tip that helps me is to first keep the original scalar function, call it from the new vectorized function with a separately allocated output, then ensure that matches what the vectorized version computes. Then we can remove the comparison.