How to convert safetensors into sbs?

[zeerd]

I know it's an old question, but I still have no idea how to finish it.

I got some files after fine-tune and merge_and_unload, like below:

added_tokens.json  config.json  generation_config.json  model.safetensors  special_tokens_map.json  
tokenizer.json  tokenizer.model  tokenizer_config.json

I only find the script python/convert_from_safetensors.py.
It seems to be created for PaliGemma only, but I still tried it with the error below:

python3 python/convert_from_safetensors.py --load_path /model/gemma-3-1b-finetune/merged_gemma/
I0401 15:06:34.645604 140119943138176 convert_from_safetensors.py:423] Restoring pure python logging
I0401 15:06:34.645855 140119943138176 convert_from_safetensors.py:429]
====
Reading from /model/gemma-3-1b-finetune/merged_gemma/ and writing to /tmp/gemmacpp.sbs
====
Traceback (most recent call last):
  File "/model/gemma-3-1b-finetune/gemma.cpp/python/convert_from_safetensors.py", line 436, in <module>
    app.run(main)
  File "/usr/local/lib/python3.11/site-packages/absl/app.py", line 316, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.11/site-packages/absl/app.py", line 261, in _run_main
    sys.exit(main(argv))
             ^^^^^^^^^^
  File "/model/gemma-3-1b-finetune/gemma.cpp/python/convert_from_safetensors.py", line 432, in main
    export_paligemma_sbs(load_path, metadata_file, sbs_file)
  File "/model/gemma-3-1b-finetune/gemma.cpp/python/convert_from_safetensors.py", line 212, in export_paligemma_sbs
    with safetensors.safe_open(file, framework="pt") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: No such device (os error 19)

Any suggest for the convert scripts?
Or maybe there is another fine-tune guide?

[jan-wassenberg]

I think load_path is meant to point directly to a .json that contains weight_map, which is a list of the other files. Do any of your .json contain weight_map?

Otherwise, it should be a single file, not directory - presumably /path/to/model.safetensors. Does that help?

This script is indeed for PaliGemma. We've done some work to improve the importing via a single script but this is unfortunately not yet open sourced.

[zeerd]

Thanks for the reply.

grep -rn weight_map /model/gemma-3-1b-finetune/merged_gemma/ returns nothing .

And by the python python/convert_from_safetensors.py --load_path /model/gemma-3-1b-finetune/merged_gemma/model.safetensors, it runs but failed.

# python python/convert_from_safetensors.py --load_path /model/gemma-3-1b-finetune/merged_gemma/model.safetensors
I0401 23:55:42.470013 140308256914304 convert_from_safetensors.py:423] Restoring pure python logging
I0401 23:55:42.470223 140308256914304 convert_from_safetensors.py:429]
====
Reading from /model/gemma-3-1b-finetune/merged_gemma/model.safetensors and writing to /tmp/gemmacpp.sbs
====
model.embed_tokens.weight torch.Size([262144, 1152]) -0.00372314453125
model.layers.0.input_layernorm.weight torch.Size([1152]) 4.09375
model.layers.0.mlp.down_proj.weight torch.Size([3981312, 1]) 167
model.layers.0.mlp.down_proj.weight.absmax torch.Size([124416]) 0.0269775390625
model.layers.0.mlp.down_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.0.mlp.down_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.0.mlp.gate_proj.weight torch.Size([3981312, 1]) 71
model.layers.0.mlp.gate_proj.weight.absmax torch.Size([124416]) 0.037353515625
model.layers.0.mlp.gate_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.0.mlp.gate_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.0.mlp.up_proj.weight torch.Size([3981312, 1]) 57
model.layers.0.mlp.up_proj.weight.absmax torch.Size([124416]) 0.09521484375
model.layers.0.mlp.up_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.0.mlp.up_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.0.post_attention_layernorm.weight torch.Size([1152]) -0.59765625
model.layers.0.post_feedforward_layernorm.weight torch.Size([1152]) 0.015869140625
model.layers.0.pre_feedforward_layernorm.weight torch.Size([1152]) 7.5
model.layers.0.self_attn.k_norm.weight torch.Size([256]) 0.69921875
model.layers.0.self_attn.k_proj.weight torch.Size([147456, 1]) 140
model.layers.0.self_attn.k_proj.weight.absmax torch.Size([4608]) 0.0693359375
model.layers.0.self_attn.k_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.0.self_attn.k_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.0.self_attn.o_proj.weight torch.Size([589824, 1]) 91
model.layers.0.self_attn.o_proj.weight.absmax torch.Size([18432]) 0.06396484375
model.layers.0.self_attn.o_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.0.self_attn.o_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.0.self_attn.q_norm.weight torch.Size([256]) 1.1953125
model.layers.0.self_attn.q_proj.weight torch.Size([589824, 1]) 59
model.layers.0.self_attn.q_proj.weight.absmax torch.Size([18432]) 0.05083116516470909
model.layers.0.self_attn.q_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.0.self_attn.q_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.0.self_attn.v_proj.weight torch.Size([147456, 1]) 5
model.layers.0.self_attn.v_proj.weight.absmax torch.Size([4608]) 0.1031426265835762
model.layers.0.self_attn.v_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.0.self_attn.v_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.1.input_layernorm.weight torch.Size([1152]) 13.1875
model.layers.1.mlp.down_proj.weight torch.Size([3981312, 1]) 189
model.layers.1.mlp.down_proj.weight.absmax torch.Size([124416]) 0.0255126953125
model.layers.1.mlp.down_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.1.mlp.down_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.1.mlp.gate_proj.weight torch.Size([3981312, 1]) 184
model.layers.1.mlp.gate_proj.weight.absmax torch.Size([124416]) 0.10546875
model.layers.1.mlp.gate_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.1.mlp.gate_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.1.mlp.up_proj.weight torch.Size([3981312, 1]) 83
model.layers.1.mlp.up_proj.weight.absmax torch.Size([124416]) 0.095703125
model.layers.1.mlp.up_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.1.mlp.up_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.1.post_attention_layernorm.weight torch.Size([1152]) 0.045654296875
model.layers.1.post_feedforward_layernorm.weight torch.Size([1152]) -0.1533203125
model.layers.1.pre_feedforward_layernorm.weight torch.Size([1152]) 13.375
model.layers.1.self_attn.k_norm.weight torch.Size([256]) 1.1640625
model.layers.1.self_attn.k_proj.weight torch.Size([147456, 1]) 58
model.layers.1.self_attn.k_proj.weight.absmax torch.Size([4608]) 0.09765625
model.layers.1.self_attn.k_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.1.self_attn.k_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.1.self_attn.o_proj.weight torch.Size([589824, 1]) 85
model.layers.1.self_attn.o_proj.weight.absmax torch.Size([18432]) 0.07666015625
model.layers.1.self_attn.o_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.1.self_attn.o_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.1.self_attn.q_norm.weight torch.Size([256]) 1.6484375
model.layers.1.self_attn.q_proj.weight torch.Size([589824, 1]) 184
model.layers.1.self_attn.q_proj.weight.absmax torch.Size([18432]) 0.009934683330357075
model.layers.1.self_attn.q_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.1.self_attn.q_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.1.self_attn.v_proj.weight torch.Size([147456, 1]) 99
model.layers.1.self_attn.v_proj.weight.absmax torch.Size([4608]) 0.11938636004924774
model.layers.1.self_attn.v_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.1.self_attn.v_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.10.input_layernorm.weight torch.Size([1152]) 7.28125
model.layers.10.mlp.down_proj.weight torch.Size([3981312, 1]) 198
model.layers.10.mlp.down_proj.weight.absmax torch.Size([124416]) 0.03125
model.layers.10.mlp.down_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.10.mlp.down_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.10.mlp.gate_proj.weight torch.Size([3981312, 1]) 205
model.layers.10.mlp.gate_proj.weight.absmax torch.Size([124416]) 0.056884765625
model.layers.10.mlp.gate_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.10.mlp.gate_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.10.mlp.up_proj.weight torch.Size([3981312, 1]) 27
model.layers.10.mlp.up_proj.weight.absmax torch.Size([124416]) 0.07080078125
model.layers.10.mlp.up_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.10.mlp.up_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.10.post_attention_layernorm.weight torch.Size([1152]) 13.0
model.layers.10.post_feedforward_layernorm.weight torch.Size([1152]) 9.6875
model.layers.10.pre_feedforward_layernorm.weight torch.Size([1152]) 2.953125
model.layers.10.self_attn.k_norm.weight torch.Size([256]) 0.63671875
model.layers.10.self_attn.k_proj.weight torch.Size([147456, 1]) 177
model.layers.10.self_attn.k_proj.weight.absmax torch.Size([4608]) 0.03662109375
model.layers.10.self_attn.k_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.10.self_attn.k_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.10.self_attn.o_proj.weight torch.Size([589824, 1]) 184
model.layers.10.self_attn.o_proj.weight.absmax torch.Size([18432]) 0.054443359375
model.layers.10.self_attn.o_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.10.self_attn.o_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.10.self_attn.q_norm.weight torch.Size([256]) 1.734375
model.layers.10.self_attn.q_proj.weight torch.Size([589824, 1]) 65
model.layers.10.self_attn.q_proj.weight.absmax torch.Size([18432]) 0.02154083363711834
model.layers.10.self_attn.q_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.10.self_attn.q_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.10.self_attn.v_proj.weight torch.Size([147456, 1]) 74
model.layers.10.self_attn.v_proj.weight.absmax torch.Size([4608]) 0.08676782250404358
model.layers.10.self_attn.v_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.10.self_attn.v_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.11.input_layernorm.weight torch.Size([1152]) 2.859375
model.layers.11.mlp.down_proj.weight torch.Size([3981312, 1]) 126
model.layers.11.mlp.down_proj.weight.absmax torch.Size([124416]) 0.0223388671875
model.layers.11.mlp.down_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.11.mlp.down_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.11.mlp.gate_proj.weight torch.Size([3981312, 1]) 212
model.layers.11.mlp.gate_proj.weight.absmax torch.Size([124416]) 0.046875
model.layers.11.mlp.gate_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.11.mlp.gate_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.11.mlp.up_proj.weight torch.Size([3981312, 1]) 163
model.layers.11.mlp.up_proj.weight.absmax torch.Size([124416]) 0.07080078125
model.layers.11.mlp.up_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.11.mlp.up_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.11.post_attention_layernorm.weight torch.Size([1152]) 14.25
model.layers.11.post_feedforward_layernorm.weight torch.Size([1152]) 13.8125
model.layers.11.pre_feedforward_layernorm.weight torch.Size([1152]) 3.90625
model.layers.11.self_attn.k_norm.weight torch.Size([256]) -0.91796875
model.layers.11.self_attn.k_proj.weight torch.Size([147456, 1]) 37
model.layers.11.self_attn.k_proj.weight.absmax torch.Size([4608]) 0.068359375
model.layers.11.self_attn.k_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.11.self_attn.k_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.11.self_attn.o_proj.weight torch.Size([589824, 1]) 232
model.layers.11.self_attn.o_proj.weight.absmax torch.Size([18432]) 0.04150390625
model.layers.11.self_attn.o_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.11.self_attn.o_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.11.self_attn.q_norm.weight torch.Size([256]) -0.9453125
model.layers.11.self_attn.q_proj.weight torch.Size([589824, 1]) 136
model.layers.11.self_attn.q_proj.weight.absmax torch.Size([18432]) 0.176894873380661
model.layers.11.self_attn.q_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.11.self_attn.q_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.11.self_attn.v_proj.weight torch.Size([147456, 1]) 224
model.layers.11.self_attn.v_proj.weight.absmax torch.Size([4608]) 0.06778275221586227
model.layers.11.self_attn.v_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.11.self_attn.v_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.12.input_layernorm.weight torch.Size([1152]) 6.65625
model.layers.12.mlp.down_proj.weight torch.Size([3981312, 1]) 100
model.layers.12.mlp.down_proj.weight.absmax torch.Size([124416]) 0.021728515625
model.layers.12.mlp.down_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.12.mlp.down_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.12.mlp.gate_proj.weight torch.Size([3981312, 1]) 49
model.layers.12.mlp.gate_proj.weight.absmax torch.Size([124416]) 0.05419921875
model.layers.12.mlp.gate_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.12.mlp.gate_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.12.mlp.up_proj.weight torch.Size([3981312, 1]) 74
model.layers.12.mlp.up_proj.weight.absmax torch.Size([124416]) 0.0751953125
model.layers.12.mlp.up_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.12.mlp.up_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.12.post_attention_layernorm.weight torch.Size([1152]) 14.75
model.layers.12.post_feedforward_layernorm.weight torch.Size([1152]) 18.125
model.layers.12.pre_feedforward_layernorm.weight torch.Size([1152]) 2.234375
model.layers.12.self_attn.k_norm.weight torch.Size([256]) 0.498046875
model.layers.12.self_attn.k_proj.weight torch.Size([147456, 1]) 131
model.layers.12.self_attn.k_proj.weight.absmax torch.Size([4608]) 0.0419921875
model.layers.12.self_attn.k_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.12.self_attn.k_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.12.self_attn.o_proj.weight torch.Size([589824, 1]) 235
model.layers.12.self_attn.o_proj.weight.absmax torch.Size([18432]) 0.043701171875
model.layers.12.self_attn.o_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.12.self_attn.o_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.12.self_attn.q_norm.weight torch.Size([256]) 0.71484375
model.layers.12.self_attn.q_proj.weight torch.Size([589824, 1]) 200
model.layers.12.self_attn.q_proj.weight.absmax torch.Size([18432]) 0.04585137963294983
model.layers.12.self_attn.q_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.12.self_attn.q_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.12.self_attn.v_proj.weight torch.Size([147456, 1]) 174
model.layers.12.self_attn.v_proj.weight.absmax torch.Size([4608]) 0.15498410165309906
model.layers.12.self_attn.v_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.12.self_attn.v_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.13.input_layernorm.weight torch.Size([1152]) 5.40625
model.layers.13.mlp.down_proj.weight torch.Size([3981312, 1]) 150
model.layers.13.mlp.down_proj.weight.absmax torch.Size([124416]) 0.0152587890625
model.layers.13.mlp.down_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.13.mlp.down_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.13.mlp.gate_proj.weight torch.Size([3981312, 1]) 90
model.layers.13.mlp.gate_proj.weight.absmax torch.Size([124416]) 0.051513671875
model.layers.13.mlp.gate_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.13.mlp.gate_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.13.mlp.up_proj.weight torch.Size([3981312, 1]) 55
model.layers.13.mlp.up_proj.weight.absmax torch.Size([124416]) 0.0732421875
model.layers.13.mlp.up_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.13.mlp.up_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.13.post_attention_layernorm.weight torch.Size([1152]) 18.375
model.layers.13.post_feedforward_layernorm.weight torch.Size([1152]) 24.25
model.layers.13.pre_feedforward_layernorm.weight torch.Size([1152]) 1.9375
model.layers.13.self_attn.k_norm.weight torch.Size([256]) -0.60546875
model.layers.13.self_attn.k_proj.weight torch.Size([147456, 1]) 215
model.layers.13.self_attn.k_proj.weight.absmax torch.Size([4608]) 0.126953125
model.layers.13.self_attn.k_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.13.self_attn.k_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.13.self_attn.o_proj.weight torch.Size([589824, 1]) 174
model.layers.13.self_attn.o_proj.weight.absmax torch.Size([18432]) 0.03662109375
model.layers.13.self_attn.o_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.13.self_attn.o_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.13.self_attn.q_norm.weight torch.Size([256]) 0.66015625
model.layers.13.self_attn.q_proj.weight torch.Size([589824, 1]) 170
model.layers.13.self_attn.q_proj.weight.absmax torch.Size([18432]) 0.048201195895671844
model.layers.13.self_attn.q_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.13.self_attn.q_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.13.self_attn.v_proj.weight torch.Size([147456, 1]) 85
model.layers.13.self_attn.v_proj.weight.absmax torch.Size([4608]) 0.17029201984405518
model.layers.13.self_attn.v_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.13.self_attn.v_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.14.input_layernorm.weight torch.Size([1152]) 4.5
model.layers.14.mlp.down_proj.weight torch.Size([3981312, 1]) 242
model.layers.14.mlp.down_proj.weight.absmax torch.Size([124416]) 0.023193359375
model.layers.14.mlp.down_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.14.mlp.down_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.14.mlp.gate_proj.weight torch.Size([3981312, 1]) 196
model.layers.14.mlp.gate_proj.weight.absmax torch.Size([124416]) 0.07080078125
model.layers.14.mlp.gate_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.14.mlp.gate_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.14.mlp.up_proj.weight torch.Size([3981312, 1]) 234
model.layers.14.mlp.up_proj.weight.absmax torch.Size([124416]) 0.0869140625
model.layers.14.mlp.up_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.14.mlp.up_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.14.post_attention_layernorm.weight torch.Size([1152]) 17.875
model.layers.14.post_feedforward_layernorm.weight torch.Size([1152]) 31.25
model.layers.14.pre_feedforward_layernorm.weight torch.Size([1152]) 1.28125
model.layers.14.self_attn.k_norm.weight torch.Size([256]) -1.03125
model.layers.14.self_attn.k_proj.weight torch.Size([147456, 1]) 89
model.layers.14.self_attn.k_proj.weight.absmax torch.Size([4608]) 0.1943359375
model.layers.14.self_attn.k_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.14.self_attn.k_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.14.self_attn.o_proj.weight torch.Size([589824, 1]) 223
model.layers.14.self_attn.o_proj.weight.absmax torch.Size([18432]) 0.10009765625
model.layers.14.self_attn.o_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.14.self_attn.o_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.14.self_attn.q_norm.weight torch.Size([256]) -0.9375
model.layers.14.self_attn.q_proj.weight torch.Size([589824, 1]) 73
model.layers.14.self_attn.q_proj.weight.absmax torch.Size([18432]) 0.15727438032627106
model.layers.14.self_attn.q_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.14.self_attn.q_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.14.self_attn.v_proj.weight torch.Size([147456, 1]) 113
model.layers.14.self_attn.v_proj.weight.absmax torch.Size([4608]) 0.17046324908733368
model.layers.14.self_attn.v_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.14.self_attn.v_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.15.input_layernorm.weight torch.Size([1152]) 3.234375
model.layers.15.mlp.down_proj.weight torch.Size([3981312, 1]) 105
model.layers.15.mlp.down_proj.weight.absmax torch.Size([124416]) 0.029541015625
model.layers.15.mlp.down_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.15.mlp.down_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.15.mlp.gate_proj.weight torch.Size([3981312, 1]) 135
model.layers.15.mlp.gate_proj.weight.absmax torch.Size([124416]) 0.06103515625
model.layers.15.mlp.gate_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.15.mlp.gate_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.15.mlp.up_proj.weight torch.Size([3981312, 1]) 83
model.layers.15.mlp.up_proj.weight.absmax torch.Size([124416]) 0.0712890625
model.layers.15.mlp.up_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.15.mlp.up_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.15.post_attention_layernorm.weight torch.Size([1152]) 21.5
model.layers.15.post_feedforward_layernorm.weight torch.Size([1152]) 39.0
model.layers.15.pre_feedforward_layernorm.weight torch.Size([1152]) 1.234375
model.layers.15.self_attn.k_norm.weight torch.Size([256]) 0.59375
model.layers.15.self_attn.k_proj.weight torch.Size([147456, 1]) 94
model.layers.15.self_attn.k_proj.weight.absmax torch.Size([4608]) 0.04736328125
model.layers.15.self_attn.k_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.15.self_attn.k_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.15.self_attn.o_proj.weight torch.Size([589824, 1]) 135
model.layers.15.self_attn.o_proj.weight.absmax torch.Size([18432]) 0.06298828125
model.layers.15.self_attn.o_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.15.self_attn.o_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.15.self_attn.q_norm.weight torch.Size([256]) 0.84375
model.layers.15.self_attn.q_proj.weight torch.Size([589824, 1]) 74
model.layers.15.self_attn.q_proj.weight.absmax torch.Size([18432]) 0.039224788546562195
model.layers.15.self_attn.q_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.15.self_attn.q_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.15.self_attn.v_proj.weight torch.Size([147456, 1]) 23
model.layers.15.self_attn.v_proj.weight.absmax torch.Size([4608]) 0.14202597737312317
model.layers.15.self_attn.v_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.15.self_attn.v_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.16.input_layernorm.weight torch.Size([1152]) 3.46875
model.layers.16.mlp.down_proj.weight torch.Size([3981312, 1]) 119
model.layers.16.mlp.down_proj.weight.absmax torch.Size([124416]) 0.030517578125
model.layers.16.mlp.down_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.16.mlp.down_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.16.mlp.gate_proj.weight torch.Size([3981312, 1]) 154
model.layers.16.mlp.gate_proj.weight.absmax torch.Size([124416]) 0.051513671875
model.layers.16.mlp.gate_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.16.mlp.gate_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.16.mlp.up_proj.weight torch.Size([3981312, 1]) 82
model.layers.16.mlp.up_proj.weight.absmax torch.Size([124416]) 0.111328125
model.layers.16.mlp.up_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.16.mlp.up_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.16.post_attention_layernorm.weight torch.Size([1152]) 37.25
model.layers.16.post_feedforward_layernorm.weight torch.Size([1152]) 55.5
model.layers.16.pre_feedforward_layernorm.weight torch.Size([1152]) 0.984375
model.layers.16.self_attn.k_norm.weight torch.Size([256]) 0.80078125
model.layers.16.self_attn.k_proj.weight torch.Size([147456, 1]) 128
model.layers.16.self_attn.k_proj.weight.absmax torch.Size([4608]) 0.044921875
model.layers.16.self_attn.k_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.16.self_attn.k_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.16.self_attn.o_proj.weight torch.Size([589824, 1]) 39
model.layers.16.self_attn.o_proj.weight.absmax torch.Size([18432]) 0.040771484375
model.layers.16.self_attn.o_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.16.self_attn.o_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.16.self_attn.q_norm.weight torch.Size([256]) 2.046875
model.layers.16.self_attn.q_proj.weight torch.Size([589824, 1]) 115
model.layers.16.self_attn.q_proj.weight.absmax torch.Size([18432]) 0.009442701004445553
model.layers.16.self_attn.q_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.16.self_attn.q_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.16.self_attn.v_proj.weight torch.Size([147456, 1]) 165
model.layers.16.self_attn.v_proj.weight.absmax torch.Size([4608]) 0.1190003827214241
model.layers.16.self_attn.v_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.16.self_attn.v_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.17.input_layernorm.weight torch.Size([1152]) 0.462890625
model.layers.17.mlp.down_proj.weight torch.Size([3981312, 1]) 35
model.layers.17.mlp.down_proj.weight.absmax torch.Size([124416]) 0.022705078125
model.layers.17.mlp.down_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.17.mlp.down_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.17.mlp.gate_proj.weight torch.Size([3981312, 1]) 212
model.layers.17.mlp.gate_proj.weight.absmax torch.Size([124416]) 0.061279296875
model.layers.17.mlp.gate_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.17.mlp.gate_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.17.mlp.up_proj.weight torch.Size([3981312, 1]) 9
model.layers.17.mlp.up_proj.weight.absmax torch.Size([124416]) 0.068359375
model.layers.17.mlp.up_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.17.mlp.up_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.17.post_attention_layernorm.weight torch.Size([1152]) 52.75
model.layers.17.post_feedforward_layernorm.weight torch.Size([1152]) 60.5
model.layers.17.pre_feedforward_layernorm.weight torch.Size([1152]) 1.03125
model.layers.17.self_attn.k_norm.weight torch.Size([256]) 1.21875
model.layers.17.self_attn.k_proj.weight torch.Size([147456, 1]) 154
model.layers.17.self_attn.k_proj.weight.absmax torch.Size([4608]) 0.025634765625
model.layers.17.self_attn.k_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.17.self_attn.k_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.17.self_attn.o_proj.weight torch.Size([589824, 1]) 181
model.layers.17.self_attn.o_proj.weight.absmax torch.Size([18432]) 0.049072265625
model.layers.17.self_attn.o_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.17.self_attn.o_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.17.self_attn.q_norm.weight torch.Size([256]) 0.423828125
model.layers.17.self_attn.q_proj.weight torch.Size([589824, 1]) 85
model.layers.17.self_attn.q_proj.weight.absmax torch.Size([18432]) 0.006169961299747229
model.layers.17.self_attn.q_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.17.self_attn.q_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.17.self_attn.v_proj.weight torch.Size([147456, 1]) 133
model.layers.17.self_attn.v_proj.weight.absmax torch.Size([4608]) 0.19383904337882996
model.layers.17.self_attn.v_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.17.self_attn.v_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.18.input_layernorm.weight torch.Size([1152]) 1.8828125
model.layers.18.mlp.down_proj.weight torch.Size([3981312, 1]) 182
model.layers.18.mlp.down_proj.weight.absmax torch.Size([124416]) 0.0172119140625
model.layers.18.mlp.down_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.18.mlp.down_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.18.mlp.gate_proj.weight torch.Size([3981312, 1]) 150
model.layers.18.mlp.gate_proj.weight.absmax torch.Size([124416]) 0.09765625
model.layers.18.mlp.gate_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.18.mlp.gate_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.18.mlp.up_proj.weight torch.Size([3981312, 1]) 124
model.layers.18.mlp.up_proj.weight.absmax torch.Size([124416]) 0.0947265625
model.layers.18.mlp.up_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.18.mlp.up_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.18.post_attention_layernorm.weight torch.Size([1152]) 35.25
model.layers.18.post_feedforward_layernorm.weight torch.Size([1152]) 65.0
model.layers.18.pre_feedforward_layernorm.weight torch.Size([1152]) 0.6015625
model.layers.18.self_attn.k_norm.weight torch.Size([256]) -1.0703125
model.layers.18.self_attn.k_proj.weight torch.Size([147456, 1]) 180
model.layers.18.self_attn.k_proj.weight.absmax torch.Size([4608]) 0.12158203125
model.layers.18.self_attn.k_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.18.self_attn.k_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.18.self_attn.o_proj.weight torch.Size([589824, 1]) 209
model.layers.18.self_attn.o_proj.weight.absmax torch.Size([18432]) 0.034423828125
model.layers.18.self_attn.o_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.18.self_attn.o_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.18.self_attn.q_norm.weight torch.Size([256]) 0.024658203125
model.layers.18.self_attn.q_proj.weight torch.Size([589824, 1]) 152
model.layers.18.self_attn.q_proj.weight.absmax torch.Size([18432]) 0.03622017800807953
model.layers.18.self_attn.q_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.18.self_attn.q_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.18.self_attn.v_proj.weight torch.Size([147456, 1]) 103
model.layers.18.self_attn.v_proj.weight.absmax torch.Size([4608]) 0.1647098809480667
model.layers.18.self_attn.v_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.18.self_attn.v_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.19.input_layernorm.weight torch.Size([1152]) 4.15625
model.layers.19.mlp.down_proj.weight torch.Size([3981312, 1]) 222
model.layers.19.mlp.down_proj.weight.absmax torch.Size([124416]) 0.01806640625
model.layers.19.mlp.down_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.19.mlp.down_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.19.mlp.gate_proj.weight torch.Size([3981312, 1]) 93
model.layers.19.mlp.gate_proj.weight.absmax torch.Size([124416]) 0.0986328125
model.layers.19.mlp.gate_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.19.mlp.gate_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.19.mlp.up_proj.weight torch.Size([3981312, 1]) 186
model.layers.19.mlp.up_proj.weight.absmax torch.Size([124416]) 0.0810546875
model.layers.19.mlp.up_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.19.mlp.up_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.19.post_attention_layernorm.weight torch.Size([1152]) 40.25
model.layers.19.post_feedforward_layernorm.weight torch.Size([1152]) 67.5
model.layers.19.pre_feedforward_layernorm.weight torch.Size([1152]) 0.59375
model.layers.19.self_attn.k_norm.weight torch.Size([256]) 1.3984375
model.layers.19.self_attn.k_proj.weight torch.Size([147456, 1]) 194
model.layers.19.self_attn.k_proj.weight.absmax torch.Size([4608]) 0.04931640625
model.layers.19.self_attn.k_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.19.self_attn.k_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.19.self_attn.o_proj.weight torch.Size([589824, 1]) 2
model.layers.19.self_attn.o_proj.weight.absmax torch.Size([18432]) 0.068359375
model.layers.19.self_attn.o_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.19.self_attn.o_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.19.self_attn.q_norm.weight torch.Size([256]) 1.078125
model.layers.19.self_attn.q_proj.weight torch.Size([589824, 1]) 193
model.layers.19.self_attn.q_proj.weight.absmax torch.Size([18432]) 0.02902359701693058
model.layers.19.self_attn.q_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.19.self_attn.q_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.19.self_attn.v_proj.weight torch.Size([147456, 1]) 218
model.layers.19.self_attn.v_proj.weight.absmax torch.Size([4608]) 0.08185100555419922
model.layers.19.self_attn.v_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.19.self_attn.v_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.2.input_layernorm.weight torch.Size([1152]) 17.0
model.layers.2.mlp.down_proj.weight torch.Size([3981312, 1]) 103
model.layers.2.mlp.down_proj.weight.absmax torch.Size([124416]) 0.027099609375
model.layers.2.mlp.down_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.2.mlp.down_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.2.mlp.gate_proj.weight torch.Size([3981312, 1]) 233
model.layers.2.mlp.gate_proj.weight.absmax torch.Size([124416]) 0.08056640625
model.layers.2.mlp.gate_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.2.mlp.gate_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.2.mlp.up_proj.weight torch.Size([3981312, 1]) 91
model.layers.2.mlp.up_proj.weight.absmax torch.Size([124416]) 0.09326171875
model.layers.2.mlp.up_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.2.mlp.up_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.2.post_attention_layernorm.weight torch.Size([1152]) -0.150390625
model.layers.2.post_feedforward_layernorm.weight torch.Size([1152]) 0.5078125
model.layers.2.pre_feedforward_layernorm.weight torch.Size([1152]) 17.125
model.layers.2.self_attn.k_norm.weight torch.Size([256]) 0.482421875
model.layers.2.self_attn.k_proj.weight torch.Size([147456, 1]) 188
model.layers.2.self_attn.k_proj.weight.absmax torch.Size([4608]) 0.11669921875
model.layers.2.self_attn.k_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.2.self_attn.k_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.2.self_attn.o_proj.weight torch.Size([589824, 1]) 24
model.layers.2.self_attn.o_proj.weight.absmax torch.Size([18432]) 0.05322265625
model.layers.2.self_attn.o_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.2.self_attn.o_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.2.self_attn.q_norm.weight torch.Size([256]) -0.296875
model.layers.2.self_attn.q_proj.weight torch.Size([589824, 1]) 164
model.layers.2.self_attn.q_proj.weight.absmax torch.Size([18432]) 0.040793806314468384
model.layers.2.self_attn.q_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.2.self_attn.q_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.2.self_attn.v_proj.weight torch.Size([147456, 1]) 136
model.layers.2.self_attn.v_proj.weight.absmax torch.Size([4608]) 0.14678765833377838
model.layers.2.self_attn.v_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.2.self_attn.v_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.20.input_layernorm.weight torch.Size([1152]) 3.171875
model.layers.20.mlp.down_proj.weight torch.Size([3981312, 1]) 106
model.layers.20.mlp.down_proj.weight.absmax torch.Size([124416]) 0.029541015625
model.layers.20.mlp.down_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.20.mlp.down_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.20.mlp.gate_proj.weight torch.Size([3981312, 1]) 198
model.layers.20.mlp.gate_proj.weight.absmax torch.Size([124416]) 0.072265625
model.layers.20.mlp.gate_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.20.mlp.gate_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.20.mlp.up_proj.weight torch.Size([3981312, 1]) 216
model.layers.20.mlp.up_proj.weight.absmax torch.Size([124416]) 0.072265625
model.layers.20.mlp.up_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.20.mlp.up_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.20.post_attention_layernorm.weight torch.Size([1152]) 36.75
model.layers.20.post_feedforward_layernorm.weight torch.Size([1152]) 72.5
model.layers.20.pre_feedforward_layernorm.weight torch.Size([1152]) 0.46875
model.layers.20.self_attn.k_norm.weight torch.Size([256]) 0.734375
model.layers.20.self_attn.k_proj.weight torch.Size([147456, 1]) 166
model.layers.20.self_attn.k_proj.weight.absmax torch.Size([4608]) 0.050537109375
model.layers.20.self_attn.k_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.20.self_attn.k_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.20.self_attn.o_proj.weight torch.Size([589824, 1]) 104
model.layers.20.self_attn.o_proj.weight.absmax torch.Size([18432]) 0.10009765625
model.layers.20.self_attn.o_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.20.self_attn.o_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.20.self_attn.q_norm.weight torch.Size([256]) 0.419921875
model.layers.20.self_attn.q_proj.weight torch.Size([589824, 1]) 90
model.layers.20.self_attn.q_proj.weight.absmax torch.Size([18432]) 0.04355091601610184
model.layers.20.self_attn.q_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.20.self_attn.q_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.20.self_attn.v_proj.weight torch.Size([147456, 1]) 114
model.layers.20.self_attn.v_proj.weight.absmax torch.Size([4608]) 0.08155611902475357
model.layers.20.self_attn.v_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.20.self_attn.v_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.21.input_layernorm.weight torch.Size([1152]) 2.8125
model.layers.21.mlp.down_proj.weight torch.Size([3981312, 1]) 150
model.layers.21.mlp.down_proj.weight.absmax torch.Size([124416]) 0.0185546875
model.layers.21.mlp.down_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.21.mlp.down_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.21.mlp.gate_proj.weight torch.Size([3981312, 1]) 222
model.layers.21.mlp.gate_proj.weight.absmax torch.Size([124416]) 0.06103515625
model.layers.21.mlp.gate_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.21.mlp.gate_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.21.mlp.up_proj.weight torch.Size([3981312, 1]) 86
model.layers.21.mlp.up_proj.weight.absmax torch.Size([124416]) 0.0693359375
model.layers.21.mlp.up_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.21.mlp.up_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.21.post_attention_layernorm.weight torch.Size([1152]) 41.25
model.layers.21.post_feedforward_layernorm.weight torch.Size([1152]) 74.5
model.layers.21.pre_feedforward_layernorm.weight torch.Size([1152]) 0.3828125
model.layers.21.self_attn.k_norm.weight torch.Size([256]) 0.703125
model.layers.21.self_attn.k_proj.weight torch.Size([147456, 1]) 231
model.layers.21.self_attn.k_proj.weight.absmax torch.Size([4608]) 0.0546875
model.layers.21.self_attn.k_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.21.self_attn.k_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.21.self_attn.o_proj.weight torch.Size([589824, 1]) 81
model.layers.21.self_attn.o_proj.weight.absmax torch.Size([18432]) 0.0830078125
model.layers.21.self_attn.o_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.21.self_attn.o_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.21.self_attn.q_norm.weight torch.Size([256]) 0.71875
model.layers.21.self_attn.q_proj.weight torch.Size([589824, 1]) 130
model.layers.21.self_attn.q_proj.weight.absmax torch.Size([18432]) 0.027513999491930008
model.layers.21.self_attn.q_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.21.self_attn.q_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.21.self_attn.v_proj.weight torch.Size([147456, 1]) 147
model.layers.21.self_attn.v_proj.weight.absmax torch.Size([4608]) 0.18787690997123718
model.layers.21.self_attn.v_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.21.self_attn.v_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.22.input_layernorm.weight torch.Size([1152]) 4.40625
model.layers.22.mlp.down_proj.weight torch.Size([3981312, 1]) 156
model.layers.22.mlp.down_proj.weight.absmax torch.Size([124416]) 0.02490234375
model.layers.22.mlp.down_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.22.mlp.down_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.22.mlp.gate_proj.weight torch.Size([3981312, 1]) 135
model.layers.22.mlp.gate_proj.weight.absmax torch.Size([124416]) 0.1123046875
model.layers.22.mlp.gate_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.22.mlp.gate_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.22.mlp.up_proj.weight torch.Size([3981312, 1]) 134
model.layers.22.mlp.up_proj.weight.absmax torch.Size([124416]) 0.07763671875
model.layers.22.mlp.up_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.22.mlp.up_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.22.post_attention_layernorm.weight torch.Size([1152]) 51.75
model.layers.22.post_feedforward_layernorm.weight torch.Size([1152]) 84.5
model.layers.22.pre_feedforward_layernorm.weight torch.Size([1152]) 0.21875
model.layers.22.self_attn.k_norm.weight torch.Size([256]) 0.369140625
model.layers.22.self_attn.k_proj.weight torch.Size([147456, 1]) 47
model.layers.22.self_attn.k_proj.weight.absmax torch.Size([4608]) 0.07177734375
model.layers.22.self_attn.k_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.22.self_attn.k_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.22.self_attn.o_proj.weight torch.Size([589824, 1]) 134
model.layers.22.self_attn.o_proj.weight.absmax torch.Size([18432]) 0.03125
model.layers.22.self_attn.o_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.22.self_attn.o_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.22.self_attn.q_norm.weight torch.Size([256]) 0.703125
model.layers.22.self_attn.q_proj.weight torch.Size([589824, 1]) 70
model.layers.22.self_attn.q_proj.weight.absmax torch.Size([18432]) 0.016618086025118828
model.layers.22.self_attn.q_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.22.self_attn.q_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.22.self_attn.v_proj.weight torch.Size([147456, 1]) 69
model.layers.22.self_attn.v_proj.weight.absmax torch.Size([4608]) 0.10513052344322205
model.layers.22.self_attn.v_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.22.self_attn.v_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.23.input_layernorm.weight torch.Size([1152]) 0.55078125
model.layers.23.mlp.down_proj.weight torch.Size([3981312, 1]) 21
model.layers.23.mlp.down_proj.weight.absmax torch.Size([124416]) 0.034423828125
model.layers.23.mlp.down_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.23.mlp.down_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.23.mlp.gate_proj.weight torch.Size([3981312, 1]) 199
model.layers.23.mlp.gate_proj.weight.absmax torch.Size([124416]) 0.099609375
model.layers.23.mlp.gate_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.23.mlp.gate_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.23.mlp.up_proj.weight torch.Size([3981312, 1]) 165
model.layers.23.mlp.up_proj.weight.absmax torch.Size([124416]) 0.060546875
model.layers.23.mlp.up_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.23.mlp.up_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.23.post_attention_layernorm.weight torch.Size([1152]) 39.5
model.layers.23.post_feedforward_layernorm.weight torch.Size([1152]) 94.0
model.layers.23.pre_feedforward_layernorm.weight torch.Size([1152]) -0.0703125
model.layers.23.self_attn.k_norm.weight torch.Size([256]) -1.0234375
model.layers.23.self_attn.k_proj.weight torch.Size([147456, 1]) 76
model.layers.23.self_attn.k_proj.weight.absmax torch.Size([4608]) 0.1904296875
model.layers.23.self_attn.k_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.23.self_attn.k_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.23.self_attn.o_proj.weight torch.Size([589824, 1]) 66
model.layers.23.self_attn.o_proj.weight.absmax torch.Size([18432]) 0.0927734375
model.layers.23.self_attn.o_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.23.self_attn.o_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.23.self_attn.q_norm.weight torch.Size([256]) -0.96875
model.layers.23.self_attn.q_proj.weight torch.Size([589824, 1]) 117
model.layers.23.self_attn.q_proj.weight.absmax torch.Size([18432]) 0.08114121109247208
model.layers.23.self_attn.q_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.23.self_attn.q_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.23.self_attn.v_proj.weight torch.Size([147456, 1]) 109
model.layers.23.self_attn.v_proj.weight.absmax torch.Size([4608]) 0.16252905130386353
model.layers.23.self_attn.v_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.23.self_attn.v_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.24.input_layernorm.weight torch.Size([1152]) 2.484375
model.layers.24.mlp.down_proj.weight torch.Size([3981312, 1]) 232
model.layers.24.mlp.down_proj.weight.absmax torch.Size([124416]) 0.033203125
model.layers.24.mlp.down_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.24.mlp.down_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.24.mlp.gate_proj.weight torch.Size([3981312, 1]) 199
model.layers.24.mlp.gate_proj.weight.absmax torch.Size([124416]) 0.08349609375
model.layers.24.mlp.gate_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.24.mlp.gate_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.24.mlp.up_proj.weight torch.Size([3981312, 1]) 24
model.layers.24.mlp.up_proj.weight.absmax torch.Size([124416]) 0.06689453125
model.layers.24.mlp.up_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.24.mlp.up_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.24.post_attention_layernorm.weight torch.Size([1152]) 49.25
model.layers.24.post_feedforward_layernorm.weight torch.Size([1152]) 115.0
model.layers.24.pre_feedforward_layernorm.weight torch.Size([1152]) -0.20703125
model.layers.24.self_attn.k_norm.weight torch.Size([256]) 0.8984375
model.layers.24.self_attn.k_proj.weight torch.Size([147456, 1]) 114
model.layers.24.self_attn.k_proj.weight.absmax torch.Size([4608]) 0.07177734375
model.layers.24.self_attn.k_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.24.self_attn.k_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.24.self_attn.o_proj.weight torch.Size([589824, 1]) 95
model.layers.24.self_attn.o_proj.weight.absmax torch.Size([18432]) 0.041015625
model.layers.24.self_attn.o_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.24.self_attn.o_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.24.self_attn.q_norm.weight torch.Size([256]) 0.84375
model.layers.24.self_attn.q_proj.weight torch.Size([589824, 1]) 148
model.layers.24.self_attn.q_proj.weight.absmax torch.Size([18432]) 0.044018905609846115
model.layers.24.self_attn.q_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.24.self_attn.q_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.24.self_attn.v_proj.weight torch.Size([147456, 1]) 101
model.layers.24.self_attn.v_proj.weight.absmax torch.Size([4608]) 0.1010797917842865
model.layers.24.self_attn.v_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.24.self_attn.v_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.25.input_layernorm.weight torch.Size([1152]) 1.734375
model.layers.25.mlp.down_proj.weight torch.Size([3981312, 1]) 3
model.layers.25.mlp.down_proj.weight.absmax torch.Size([124416]) 0.015380859375
model.layers.25.mlp.down_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.25.mlp.down_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.25.mlp.gate_proj.weight torch.Size([3981312, 1]) 237
model.layers.25.mlp.gate_proj.weight.absmax torch.Size([124416]) 0.054931640625
model.layers.25.mlp.gate_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.25.mlp.gate_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.25.mlp.up_proj.weight torch.Size([3981312, 1]) 241
model.layers.25.mlp.up_proj.weight.absmax torch.Size([124416]) 0.08154296875
model.layers.25.mlp.up_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.25.mlp.up_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.25.post_attention_layernorm.weight torch.Size([1152]) 38.25
model.layers.25.post_feedforward_layernorm.weight torch.Size([1152]) 88.0
model.layers.25.pre_feedforward_layernorm.weight torch.Size([1152]) 0.265625
model.layers.25.self_attn.k_norm.weight torch.Size([256]) 0.63671875
model.layers.25.self_attn.k_proj.weight torch.Size([147456, 1]) 45
model.layers.25.self_attn.k_proj.weight.absmax torch.Size([4608]) 0.0576171875
model.layers.25.self_attn.k_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.25.self_attn.k_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.25.self_attn.o_proj.weight torch.Size([589824, 1]) 110
model.layers.25.self_attn.o_proj.weight.absmax torch.Size([18432]) 0.064453125
model.layers.25.self_attn.o_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.25.self_attn.o_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.25.self_attn.q_norm.weight torch.Size([256]) 0.83984375
model.layers.25.self_attn.q_proj.weight torch.Size([589824, 1]) 63
model.layers.25.self_attn.q_proj.weight.absmax torch.Size([18432]) 0.051811277866363525
model.layers.25.self_attn.q_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.25.self_attn.q_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.25.self_attn.v_proj.weight torch.Size([147456, 1]) 106
model.layers.25.self_attn.v_proj.weight.absmax torch.Size([4608]) 0.06666383892297745
model.layers.25.self_attn.v_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.25.self_attn.v_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.3.input_layernorm.weight torch.Size([1152]) 5.0625
model.layers.3.mlp.down_proj.weight torch.Size([3981312, 1]) 19
model.layers.3.mlp.down_proj.weight.absmax torch.Size([124416]) 0.0224609375
model.layers.3.mlp.down_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.3.mlp.down_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.3.mlp.gate_proj.weight torch.Size([3981312, 1]) 99
model.layers.3.mlp.gate_proj.weight.absmax torch.Size([124416]) 0.083984375
model.layers.3.mlp.gate_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.3.mlp.gate_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.3.mlp.up_proj.weight torch.Size([3981312, 1]) 101
model.layers.3.mlp.up_proj.weight.absmax torch.Size([124416]) 0.0751953125
model.layers.3.mlp.up_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.3.mlp.up_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.3.post_attention_layernorm.weight torch.Size([1152]) 8.6875
model.layers.3.post_feedforward_layernorm.weight torch.Size([1152]) 1.015625
model.layers.3.pre_feedforward_layernorm.weight torch.Size([1152]) -1.0
model.layers.3.self_attn.k_norm.weight torch.Size([256]) 0.07763671875
model.layers.3.self_attn.k_proj.weight torch.Size([147456, 1]) 157
model.layers.3.self_attn.k_proj.weight.absmax torch.Size([4608]) 0.06298828125
model.layers.3.self_attn.k_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.3.self_attn.k_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.3.self_attn.o_proj.weight torch.Size([589824, 1]) 166
model.layers.3.self_attn.o_proj.weight.absmax torch.Size([18432]) 0.0703125
model.layers.3.self_attn.o_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.3.self_attn.o_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.3.self_attn.q_norm.weight torch.Size([256]) 0.31640625
model.layers.3.self_attn.q_proj.weight torch.Size([589824, 1]) 166
model.layers.3.self_attn.q_proj.weight.absmax torch.Size([18432]) 0.047232143580913544
model.layers.3.self_attn.q_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.3.self_attn.q_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.3.self_attn.v_proj.weight torch.Size([147456, 1]) 116
model.layers.3.self_attn.v_proj.weight.absmax torch.Size([4608]) 0.10998813062906265
model.layers.3.self_attn.v_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.3.self_attn.v_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.4.input_layernorm.weight torch.Size([1152]) 0.828125
model.layers.4.mlp.down_proj.weight torch.Size([3981312, 1]) 88
model.layers.4.mlp.down_proj.weight.absmax torch.Size([124416]) 0.01422119140625
model.layers.4.mlp.down_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.4.mlp.down_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.4.mlp.gate_proj.weight torch.Size([3981312, 1]) 179
model.layers.4.mlp.gate_proj.weight.absmax torch.Size([124416]) 0.06640625
model.layers.4.mlp.gate_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.4.mlp.gate_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.4.mlp.up_proj.weight torch.Size([3981312, 1]) 121
model.layers.4.mlp.up_proj.weight.absmax torch.Size([124416]) 0.09375
model.layers.4.mlp.up_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.4.mlp.up_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.4.post_attention_layernorm.weight torch.Size([1152]) 0.412109375
model.layers.4.post_feedforward_layernorm.weight torch.Size([1152]) 7.03125
model.layers.4.pre_feedforward_layernorm.weight torch.Size([1152]) 6.34375
model.layers.4.self_attn.k_norm.weight torch.Size([256]) -0.0400390625
model.layers.4.self_attn.k_proj.weight torch.Size([147456, 1]) 145
model.layers.4.self_attn.k_proj.weight.absmax torch.Size([4608]) 0.060302734375
model.layers.4.self_attn.k_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.4.self_attn.k_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.4.self_attn.o_proj.weight torch.Size([589824, 1]) 170
model.layers.4.self_attn.o_proj.weight.absmax torch.Size([18432]) 0.0169677734375
model.layers.4.self_attn.o_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.4.self_attn.o_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.4.self_attn.q_norm.weight torch.Size([256]) 0.365234375
model.layers.4.self_attn.q_proj.weight torch.Size([589824, 1]) 136
model.layers.4.self_attn.q_proj.weight.absmax torch.Size([18432]) 0.07248757034540176
model.layers.4.self_attn.q_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.4.self_attn.q_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.4.self_attn.v_proj.weight torch.Size([147456, 1]) 121
model.layers.4.self_attn.v_proj.weight.absmax torch.Size([4608]) 0.08169206976890564
model.layers.4.self_attn.v_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.4.self_attn.v_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.5.input_layernorm.weight torch.Size([1152]) 7.90625
model.layers.5.mlp.down_proj.weight torch.Size([3981312, 1]) 185
model.layers.5.mlp.down_proj.weight.absmax torch.Size([124416]) 0.0257568359375
model.layers.5.mlp.down_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.5.mlp.down_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.5.mlp.gate_proj.weight torch.Size([3981312, 1]) 200
model.layers.5.mlp.gate_proj.weight.absmax torch.Size([124416]) 0.09375
model.layers.5.mlp.gate_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.5.mlp.gate_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.5.mlp.up_proj.weight torch.Size([3981312, 1]) 154
model.layers.5.mlp.up_proj.weight.absmax torch.Size([124416]) 0.09326171875
model.layers.5.mlp.up_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.5.mlp.up_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.5.post_attention_layernorm.weight torch.Size([1152]) 2.640625
model.layers.5.post_feedforward_layernorm.weight torch.Size([1152]) 3.078125
model.layers.5.pre_feedforward_layernorm.weight torch.Size([1152]) 22.75
model.layers.5.self_attn.k_norm.weight torch.Size([256]) -1.828125
model.layers.5.self_attn.k_proj.weight torch.Size([147456, 1]) 203
model.layers.5.self_attn.k_proj.weight.absmax torch.Size([4608]) 0.04150390625
model.layers.5.self_attn.k_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.5.self_attn.k_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.5.self_attn.o_proj.weight torch.Size([589824, 1]) 72
model.layers.5.self_attn.o_proj.weight.absmax torch.Size([18432]) 0.04736328125
model.layers.5.self_attn.o_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.5.self_attn.o_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.5.self_attn.q_norm.weight torch.Size([256]) -0.96875
model.layers.5.self_attn.q_proj.weight torch.Size([589824, 1]) 252
model.layers.5.self_attn.q_proj.weight.absmax torch.Size([18432]) 0.09575549513101578
model.layers.5.self_attn.q_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.5.self_attn.q_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.5.self_attn.v_proj.weight torch.Size([147456, 1]) 136
model.layers.5.self_attn.v_proj.weight.absmax torch.Size([4608]) 0.09307155013084412
model.layers.5.self_attn.v_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.5.self_attn.v_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.6.input_layernorm.weight torch.Size([1152]) 18.875
model.layers.6.mlp.down_proj.weight torch.Size([3981312, 1]) 250
model.layers.6.mlp.down_proj.weight.absmax torch.Size([124416]) 0.0289306640625
model.layers.6.mlp.down_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.6.mlp.down_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.6.mlp.gate_proj.weight torch.Size([3981312, 1]) 247
model.layers.6.mlp.gate_proj.weight.absmax torch.Size([124416]) 0.07568359375
model.layers.6.mlp.gate_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.6.mlp.gate_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.6.mlp.up_proj.weight torch.Size([3981312, 1]) 68
model.layers.6.mlp.up_proj.weight.absmax torch.Size([124416]) 0.08056640625
model.layers.6.mlp.up_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.6.mlp.up_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.6.post_attention_layernorm.weight torch.Size([1152]) 1.796875
model.layers.6.post_feedforward_layernorm.weight torch.Size([1152]) 3.1875
model.layers.6.pre_feedforward_layernorm.weight torch.Size([1152]) 11.75
model.layers.6.self_attn.k_norm.weight torch.Size([256]) 0.087890625
model.layers.6.self_attn.k_proj.weight torch.Size([147456, 1]) 173
model.layers.6.self_attn.k_proj.weight.absmax torch.Size([4608]) 0.12158203125
model.layers.6.self_attn.k_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.6.self_attn.k_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.6.self_attn.o_proj.weight torch.Size([589824, 1]) 133
model.layers.6.self_attn.o_proj.weight.absmax torch.Size([18432]) 0.04833984375
model.layers.6.self_attn.o_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.6.self_attn.o_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.6.self_attn.q_norm.weight torch.Size([256]) 0.7265625
model.layers.6.self_attn.q_proj.weight torch.Size([589824, 1]) 101
model.layers.6.self_attn.q_proj.weight.absmax torch.Size([18432]) 0.04954914003610611
model.layers.6.self_attn.q_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.6.self_attn.q_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.6.self_attn.v_proj.weight torch.Size([147456, 1]) 156
model.layers.6.self_attn.v_proj.weight.absmax torch.Size([4608]) 0.1571962535381317
model.layers.6.self_attn.v_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.6.self_attn.v_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.7.input_layernorm.weight torch.Size([1152]) 10.1875
model.layers.7.mlp.down_proj.weight torch.Size([3981312, 1]) 132
model.layers.7.mlp.down_proj.weight.absmax torch.Size([124416]) 0.025634765625
model.layers.7.mlp.down_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.7.mlp.down_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.7.mlp.gate_proj.weight torch.Size([3981312, 1]) 51
model.layers.7.mlp.gate_proj.weight.absmax torch.Size([124416]) 0.052001953125
model.layers.7.mlp.gate_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.7.mlp.gate_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.7.mlp.up_proj.weight torch.Size([3981312, 1]) 184
model.layers.7.mlp.up_proj.weight.absmax torch.Size([124416]) 0.076171875
model.layers.7.mlp.up_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.7.mlp.up_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.7.post_attention_layernorm.weight torch.Size([1152]) 3.09375
model.layers.7.post_feedforward_layernorm.weight torch.Size([1152]) 4.46875
model.layers.7.pre_feedforward_layernorm.weight torch.Size([1152]) 6.40625
model.layers.7.self_attn.k_norm.weight torch.Size([256]) 0.306640625
model.layers.7.self_attn.k_proj.weight torch.Size([147456, 1]) 58
model.layers.7.self_attn.k_proj.weight.absmax torch.Size([4608]) 0.06298828125
model.layers.7.self_attn.k_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.7.self_attn.k_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.7.self_attn.o_proj.weight torch.Size([589824, 1]) 152
model.layers.7.self_attn.o_proj.weight.absmax torch.Size([18432]) 0.061279296875
model.layers.7.self_attn.o_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.7.self_attn.o_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.7.self_attn.q_norm.weight torch.Size([256]) 1.3515625
model.layers.7.self_attn.q_proj.weight torch.Size([589824, 1]) 155
model.layers.7.self_attn.q_proj.weight.absmax torch.Size([18432]) 0.0702059268951416
model.layers.7.self_attn.q_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.7.self_attn.q_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.7.self_attn.v_proj.weight torch.Size([147456, 1]) 153
model.layers.7.self_attn.v_proj.weight.absmax torch.Size([4608]) 0.09226664155721664
model.layers.7.self_attn.v_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.7.self_attn.v_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.8.input_layernorm.weight torch.Size([1152]) 8.125
model.layers.8.mlp.down_proj.weight torch.Size([3981312, 1]) 174
model.layers.8.mlp.down_proj.weight.absmax torch.Size([124416]) 0.0142822265625
model.layers.8.mlp.down_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.8.mlp.down_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.8.mlp.gate_proj.weight torch.Size([3981312, 1]) 166
model.layers.8.mlp.gate_proj.weight.absmax torch.Size([124416]) 0.058837890625
model.layers.8.mlp.gate_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.8.mlp.gate_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.8.mlp.up_proj.weight torch.Size([3981312, 1]) 47
model.layers.8.mlp.up_proj.weight.absmax torch.Size([124416]) 0.06396484375
model.layers.8.mlp.up_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.8.mlp.up_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.8.post_attention_layernorm.weight torch.Size([1152]) 4.125
model.layers.8.post_feedforward_layernorm.weight torch.Size([1152]) 6.40625
model.layers.8.pre_feedforward_layernorm.weight torch.Size([1152]) 4.875
model.layers.8.self_attn.k_norm.weight torch.Size([256]) 0.58203125
model.layers.8.self_attn.k_proj.weight torch.Size([147456, 1]) 169
model.layers.8.self_attn.k_proj.weight.absmax torch.Size([4608]) 0.041748046875
model.layers.8.self_attn.k_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.8.self_attn.k_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.8.self_attn.o_proj.weight torch.Size([589824, 1]) 83
model.layers.8.self_attn.o_proj.weight.absmax torch.Size([18432]) 0.049072265625
model.layers.8.self_attn.o_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.8.self_attn.o_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.8.self_attn.q_norm.weight torch.Size([256]) 0.5
model.layers.8.self_attn.q_proj.weight torch.Size([589824, 1]) 40
model.layers.8.self_attn.q_proj.weight.absmax torch.Size([18432]) 0.04762272909283638
model.layers.8.self_attn.q_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.8.self_attn.q_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.8.self_attn.v_proj.weight torch.Size([147456, 1]) 135
model.layers.8.self_attn.v_proj.weight.absmax torch.Size([4608]) 0.16637928783893585
model.layers.8.self_attn.v_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.8.self_attn.v_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.9.input_layernorm.weight torch.Size([1152]) 7.5625
model.layers.9.mlp.down_proj.weight torch.Size([3981312, 1]) 88
model.layers.9.mlp.down_proj.weight.absmax torch.Size([124416]) 0.0277099609375
model.layers.9.mlp.down_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.9.mlp.down_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.9.mlp.gate_proj.weight torch.Size([3981312, 1]) 115
model.layers.9.mlp.gate_proj.weight.absmax torch.Size([124416]) 0.05322265625
model.layers.9.mlp.gate_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.9.mlp.gate_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.9.mlp.up_proj.weight torch.Size([3981312, 1]) 140
model.layers.9.mlp.up_proj.weight.absmax torch.Size([124416]) 0.0693359375
model.layers.9.mlp.up_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.9.mlp.up_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.9.post_attention_layernorm.weight torch.Size([1152]) 4.90625
model.layers.9.post_feedforward_layernorm.weight torch.Size([1152]) 8.6875
model.layers.9.pre_feedforward_layernorm.weight torch.Size([1152]) 5.03125
model.layers.9.self_attn.k_norm.weight torch.Size([256]) 1.1015625
model.layers.9.self_attn.k_proj.weight torch.Size([147456, 1]) 212
model.layers.9.self_attn.k_proj.weight.absmax torch.Size([4608]) 0.06396484375
model.layers.9.self_attn.k_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.9.self_attn.k_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.layers.9.self_attn.o_proj.weight torch.Size([589824, 1]) 104
model.layers.9.self_attn.o_proj.weight.absmax torch.Size([18432]) 0.056396484375
model.layers.9.self_attn.o_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.9.self_attn.o_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.9.self_attn.q_norm.weight torch.Size([256]) 0.140625
model.layers.9.self_attn.q_proj.weight torch.Size([589824, 1]) 104
model.layers.9.self_attn.q_proj.weight.absmax torch.Size([18432]) 0.03868338465690613
model.layers.9.self_attn.q_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.9.self_attn.q_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([81]) 123
model.layers.9.self_attn.v_proj.weight torch.Size([147456, 1]) 166
model.layers.9.self_attn.v_proj.weight.absmax torch.Size([4608]) 0.1953616440296173
model.layers.9.self_attn.v_proj.weight.quant_map torch.Size([16]) -1.0
model.layers.9.self_attn.v_proj.weight.quant_state.bitsandbytes__nf4 torch.Size([80]) 123
model.norm.weight torch.Size([1152]) 5.15625
Traceback (most recent call last):
  File "/model/gemma-3-1b-finetune/gemma.cpp/python/convert_from_safetensors.py", line 436, in <module>
    app.run(main)
  File "/usr/local/lib/python3.11/site-packages/absl/app.py", line 316, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.11/site-packages/absl/app.py", line 261, in _run_main
    sys.exit(main(argv))
             ^^^^^^^^^^
  File "/model/gemma-3-1b-finetune/gemma.cpp/python/convert_from_safetensors.py", line 432, in main
    export_paligemma_sbs(load_path, metadata_file, sbs_file)
  File "/model/gemma-3-1b-finetune/gemma.cpp/python/convert_from_safetensors.py", line 219, in export_paligemma_sbs
    params["language_model.model.embed_tokens.weight"] = params[
                                                         ^^^^^^^
KeyError: 'language_model.model.embed_tokens.weight'

And then, I tried the google/gemma-3-1b-it/model.safetensors download from Hugging-Face, which return the same Traceback.

[jan-wassenberg]

Thanks for confirming. Looks like pointing to the .safetensors file is good.

So we're crashing because the line params["language_model.model.embed_tokens.weight"] = params[... is specific to PaliGemma. That could be commented out, but a bunch of other keys in our script also reference language_model, whereas this file calls it "model". Looks like the script would require major updates for non-PaliGemma models, or preferably we'd be able to open source our internal one.

FYI I am currently in the middle of a huge change that simplifies the tensor loading and metadata, which would help with that. May take a week yet until that's landed.

[jan-wassenberg]

Quick update, this change has finally landed - took quite a bit longer. We are working toward updating and open sourcing the exporter.