# MACaQ quantizer ships 2-4 bit GGUF and W4A4 from Huawei under Apache 2.0

> Huawei's CSL group open-sourced MACaQ, a calibrated quantizer for text, image, video and audio models. It outputs llama.cpp-compatible GGUF, plus W4A4 checkpoints for vLLM, Diffusers and PyTorch.

- Published: 2026-11-12
- By: Samir Sengupta (https://www.samcodeman.com/about)
- Canonical: https://www.samcodeman.com/writing/macaq-quantizer-2-4-bit-gguf-w4a4-apache-2
- How this was made: Drafted with Claude, Anthropic's model from 1 cited source, then checked against it in a second editing pass. Editorial policy: https://www.samcodeman.com/editorial-policy

## The short version

- MACaQ produces 2-, 3- and 4-bit GGUF for llama.cpp and W4A4 checkpoints for vLLM, Diffusers and PyTorch, all under Apache 2.0.
- MACaQ-R refines an existing Q4_K_XL GGUF of Qwen3.6-35B-A3B to a KL divergence of 0.013367, against 0.015731 for Unsloth's UD-Q4_K_XL, at the same 22.36 GB. The file runs on standard llama.cpp.
- The README compares against MR-GPTQ and DuQuant++ but gives no AWQ comparison and no Llama-3-8B perplexity values.
- Ascend support is W4A4 dense and grouped matmul kernels in two CANN pull requests. Full-model execution still needs integration with an Ascend inference runtime.

Huawei's huawei-csl GitHub organisation released the MACaQ quantizer. The repository snapshot is dated 2026-11-11. MACaQ produces 2-, 3- and 4-bit GGUF files for llama.cpp and W4A4 checkpoints for vLLM, Diffusers and PyTorch. It covers text, image, video and audio models and is licensed under Apache 2.0. The README reports a 122B-parameter model quantized in 64 minutes on 32.5 GB of device memory, and claims lower perplexity than MR-GPTQ and DuQuant++. It gives no AWQ comparison. Against existing llama.cpp quants, the companion tool MACaQ-R produces a Qwen3.6-35B-A3B Q4_K_XL GGUF with a KL divergence to FP16 of 0.013367, against 0.015731 for Unsloth's UD-Q4_K_XL at the same 22.36 GB. Hardware targets are CPU through llama.cpp, vLLM serving, and Ascend kernels that are still awaiting runtime integration.

MACaQ is a calibrated quantizer. It quantizes both weights and activations and uses calibration data, including conditioning images for image-to-video. The paper is on arXiv as 2611.10649, titled "MACaQ: Fast, Model Agnostic Calibrated Quantization for Low-Precision Weights and Activations". Its authors are Lorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, Jiawei Zhuang and Lukas Cavigelli. The release also includes MACaQ-R, a separate tool that refines GGUF files that are already quantized. MACaQ-R output runs on a standard llama.cpp installation, so it is the relevant piece for teams that already deploy on llama.cpp and do not want to change runtimes.

## Bit-widths and models the MACaQ quantizer supports

Each backend supports a different precision. The README says to use Python 3.12 and a separate environment for each backend, so using all four means maintaining four environments.

- llama.cpp: text generation, 2-, 3- and 4-bit GGUF, with CPU execution supported.
- vLLM: text generation and serving. W4A4 is the default inference mode. Weight-only is also supported, but the README does not state its bit-width.
- Diffusers: image, video and audio generation, W4A4, saved as safetensors checkpoints.
- PyTorch: speech generation, W4A4, with saved checkpoints that load without repeating calibration.

The published llama.cpp recipes are mostly Qwen:
- dense Qwen3-4B
- hybrid Qwen3.8-27B
- the MoE models Qwen3-30B-A3B, Qwen3-Coder-30B-A3B-Instruct, Qwen3.6-35B-A3B and Qwen3-Next-80B-A3B-Instruct
- Qwen3.8-Flash-Next, which has a model-specific recipe

The only non-Qwen text recipe is allenai/OLMoE-1B-7B-0125-Instruct. The Diffusers recipes cover FLUX.2-klein-4B, Wan2.2-TI2V-5B-Diffusers and stable-audio-1.0. The PyTorch recipes cover Qwen3-TTS and Chatterbox Turbo. The project calls itself model-agnostic, but the README publishes no Llama, Mistral, Gemma or DeepSeek GGUF workflow, so quality on those architectures is not shown in the repository.

## How MACaQ compares with GPTQ and AWQ

The README makes two claims against MR-GPTQ and DuQuant++. First, MACaQ reaches lower perplexity than both. Second, it can match their best Llama-3-8B perplexity with 11 times less quantization time. The README gives neither the Llama-3-8B perplexity values nor the bit configuration used, and points to the paper for details. It does not compare against plain GPTQ and does not mention AWQ. Teams using AWQ as their baseline get no direct number from this release and will need to run their own evaluation.

## MACaQ-R against Unsloth's llama.cpp quants

The README's numeric comparison with existing llama.cpp quants is for MACaQ-R. MACaQ-R refines the weight codes inside an already-quantized GGUF and preserves its native formats and block scales, so the output runs on a standard llama.cpp installation. On Qwen3.6-35B-A3B the Q4_K_XL file stays at 22.36 GB. KL divergence to FP16 drops from 0.015731 for Unsloth's UD-Q4_K_XL to 0.013367, about 15 percent lower. Separately, the README offers a comparison of MACaQ itself with published Qwen3-4B GGUFs by file size and KL divergence to FP16, but its text gives no numbers and points to the MACaQ blog for details of the models compared. No downstream task accuracy is reported in the README for either model.

MACaQ-R takes four inputs:
- the quantized GGUF
- the original Hugging Face weights
- an FP16 GGUF teacher
- a calibration text file

The README describes the input as a quantized MoE GGUF and does not say whether dense models are supported.

```bash
.venv-llamacpp/bin/python llamacpp/macaq.py refine \
 --model models/model-q4.gguf --hf models/original \
 --teacher models/original-f16.gguf --calibration data/calibration.txt \
 --output models/model-macaq-r.gguf --work-dir work/refinement
```

There is a distinction to watch. MACaQ-R files run on stock llama.cpp, but the full MACaQ workflow with activation transforms uses the runtime bundled in the repo. Commenters on Hacker News pointed to people on r/LocalLLaMA reproducing the method and publishing GGUFs. One commenter called the results much better than those of the previously widely used quantizers. That is community reaction, not a controlled benchmark.

## Quantization memory and speed: 32.5 GB for 122B

On the quantization side, the README reports a 122B-parameter model quantized in 64 minutes using 32.5 GB of device memory. It does not name the 122B model or the accelerator used. That figure is memory used during quantization, not the inference footprint. The README gives no memory comparison with GPTQ, AWQ or llama.cpp quants, and its only cost comparison is the 11x reduction in quantization time against MR-GPTQ and DuQuant++.

On the inference side, MACaQ W4A4 on Qwen3.8-27B in vLLM reaches a perplexity of 2.8994, against 2.8463 for BF16. Relative to BF16, prefill is 2.33 times faster and decode is 2.58 times faster. The README does not give:
- the GPU that produced those speedups
- the batch size or sequence length
- the dataset the perplexity was measured on

Those details are needed before assuming the speedup transfers to other serving hardware.

## Which hardware MACaQ targets: CPU and Ascend

For Huawei's own accelerators, MACaQ W4A4 dense and grouped matmul kernels are in two CANN pull requests:
- ops-nn #12568 adds PTO-ISA kernels via ascend_ops_nn.
- ops-tilelang #44 adds TileLang kernels with the CCE backend.

The README does not say whether these have been merged. It says full-model execution requires integration with an Ascend inference runtime, so Ascend currently has kernels, not an end-to-end serving path. The README does not name the GPUs the vLLM backend was tested on. The llama.cpp backend supports CPU execution, and MACaQ-R files run on a standard llama.cpp installation, which makes CPU-only llama.cpp the most portable target in the release.

That note covered large MoE GGUFs running CPU-only in llama.cpp, the deployment where a lower-KL 4-bit file at the same size matters most. See: [llama-modchip runs 284B DeepSeek-V4-Flash in 64GB RAM at 9.1-14.7 tok/s, no GPU](https://www.samcodeman.com/writing/llama-modchip-284b-deepseek-v4-flash-64gb-ram-cpu)

## Apache 2.0, and how it differs from SINQ and KVarN

MACaQ is licensed under the Apache License 2.0, and bundled third-party components retain their own licenses. The repository includes a NOTICE file, which is worth checking before vendoring the code. At the time of the snapshot the repo had 182 stars, 7 forks and 22 commits. One Hacker News commenter worried that Huawei-hosted pages outside China could disappear because of sanctions. Another replied that many of the organisation's GitHub repos have been around for a while. Under Apache 2.0, the code can be forked and kept.

Commenters also asked why Huawei's three most-starred GitHub repositories are all quantizers. One replied that the three do different jobs:
- SINQ, created last year, is weight-only and calibration-free.
- KVarN quantizes the KV cache to get longer context.
- MACaQ is multimodal, calibrated quantization of both weights and activations.

That split comes from a commenter, not from the README. It is consistent with what the repository shows, where W4A4 activation quantization and calibration data are central.

## Questions this raises

### Does MACaQ output run on a standard llama.cpp install?

MACaQ-R files do, because MACaQ-R refines weight codes inside an existing GGUF and preserves its native formats and block scales. The full MACaQ workflow with activation transforms uses the runtime bundled in the repo.

### How does MACaQ compare with GPTQ and AWQ?

The README claims lower perplexity than MR-GPTQ and DuQuant++ and says it matches their best Llama-3-8B perplexity with 11 times less quantization time. It gives no plain GPTQ comparison and does not mention AWQ, so AWQ users need to run their own evaluation.

### Which models have published MACaQ recipes?

The llama.cpp recipes are mostly Qwen, including Qwen3-4B, Qwen3.8-27B and several Qwen3 MoE models, plus OLMoE-1B-7B. Diffusers recipes cover FLUX.2-klein-4B, Wan2.2-TI2V-5B and stable-audio-1.0, and PyTorch recipes cover Qwen3-TTS and Chatterbox Turbo. No Llama, Mistral, Gemma or DeepSeek GGUF workflow is published.

## Key facts

- MACaQ outputs 2-, 3- and 4-bit GGUF for llama.cpp and W4A4 checkpoints for vLLM, Diffusers and PyTorch, under Apache 2.0.
- MACaQ-R's Qwen3.6-35B-A3B Q4_K_XL GGUF has KL divergence to FP16 of 0.013367 vs 0.015731 for Unsloth's UD-Q4_K_XL, both at 22.36 GB.
- The README reports a 122B-parameter model quantized in 64 minutes on 32.5 GB of device memory, without naming the model or accelerator.
- W4A4 Qwen3.8-27B in vLLM scores 2.8994 perplexity vs 2.8463 for BF16, with 2.33x faster prefill and 2.58x faster decode on unnamed hardware.
- The paper is arXiv 2611.10649, and the repository snapshot is dated 2026-11-11 with 182 stars and 22 commits.

## Sources

1. [Huawei releases MACaQ, its internal quantizer for local inference](https://github.com/huawei-csl/MACaQ) - github.com

Discussion read (practitioners' reactions, not a source): https://news.ycombinator.com/item?id=50458166