LittleBit compression stores LLM weights at 0.1 bits via binarized latent factors

The SamsungLabs/LittleBit repository is the official implementation of two papers. The first is LittleBit: Ultra Low-Bit Quantization via Latent Factorization (NeurIPS 2025), by Banseok Lee, Dongkyu Kim, Youngcheon You and Youngmin Kim. The second is LittleBit-2: Maximizing the Spectral Energy Gain in Sub-1-Bit LLMs via Latent Geometry Alignment (ICML 2026), by Banseok Lee and Youngmin Kim. At the time of the source snapshot the repository had 219 stars, 28 forks and 5 commits.

LittleBit does not keep a quantized copy of the full weight matrix. It keeps low-rank latent factors whose entries are binarized, plus lightweight learned scales that restore magnitude information. The README does not spell out the arithmetic. Because low-rank factors have fewer entries than the dense matrix they replace, the average storage per original weight can fall below one bit, which is how the README can name a 0.1 bits-per-weight setting. The project says the method preserves the original model architecture at inference time.

The README does not give the formula that maps a bit budget to factor rank. It also does not say what precision the scales are stored in or how residual factorization adds to the budget. An engineer can see the knobs but not the exact accounting. The training flags in the README examples are:

LittleBit-2 targets what the authors call latent geometry misalignment in the initialization stage. The original method takes SVD-derived latent factors and binarizes them. LittleBit-2 first applies Internal Latent Rotation with Joint Iterative Quantization (Joint-ITQ), which aligns those factors with the binary hypercube before QAT begins. Joint-ITQ is opt-in through --use_itq True. By default, LittleBitLinear uses the original SVD-only initialization.

The README states that LittleBit-2 produces no additional inference overhead and that the deployed factorized layer remains the same. For anyone planning a serving path, this means a LittleBit checkpoint and a LittleBit-2 checkpoint should load through the same layer. The choice between them is a training-time decision.

The release does not answer this. The README lists the evaluation suite: perplexity on wikitext2 and c4 at sequence length 2048, and zero-shot boolq, piqa, hellaswag, winogrande, arc_easy, arc_challenge and openbookqa. It reports no results on any of them. It also contains no comparison against 1-bit, 1.58-bit or 2-bit baselines, so anyone who needs those numbers has to go to the papers or rerun the evaluation.

Commenters on Hacker News who read the paper describe the loss as large. One commenter summarized it as performance going from 80% to 47% on Wikitext-2, without saying which metric or configuration that refers to. The same commenter objected that there was no comparison to FP4 approaches, which they said keep or exceed that 80% at a 4.25 to 4.5 bit budget. Another commenter said 1-bit quants have so far been less useful than smaller models that use the same memory. The README does not address that comparison, and it is the one that matters to anyone choosing what to deploy.

A reply in the thread offered a narrower reading. On this view, the contribution is better relative loss at lower precisions, not a model anyone would want to run today. That reading fits what the repository ships: a method and a training recipe, with no ready-to-use checkpoints of stated quality.

LittleBit is not a post-training quantizer you point at a checkpoint. The README's example trains Llama-2-7b-hf on the c4_wiki dataset for 5 epochs, with a learning rate of 4e-05, per-device batch size 4, warmup ratio 0.02 and an l2l_loss_scale of 10.0. Multi-GPU runs go through DeepSpeed with a ZeRO-3 config. The README's single-GPU command, with --report wandb dropped and LittleBit-2 enabled, is:

One commenter argued that QAT is the fix for sub-4-bit quality, because post-training quantization never computes the training loss on the quantized state. The same commenter noted that QAT is far more expensive than a one-time offline conversion of existing weights. The README gives no GPU-hours, memory requirement or wall-clock time for a 5-epoch run on a 7B model. Teams cannot budget the compression step from the release alone.

The README says LittleBit preserves the original model architecture at inference time. In the code, the quantized linear layers are the repository's own module, selected with --quant_mod LittleBitLinear. Evaluation loads checkpoints with eval.py, which reads quantization settings in this order: explicit CLI arguments, then littlebit_config.json in the model directory, then config.json for older checkpoints. Legacy checkpoints without littlebit_config.json need --quant_func, --quant_mod and --split_dim passed by hand. The Hugging Face example, username/littlebit-llama-7b-0.1bpw, uses a placeholder namespace, not a published model.

The README does not mention llama.cpp, GGUF, vLLM, a bit-packed storage format or custom CUDA kernels. It also reports no measurements of runtime memory, tokens per second or latency. Based on the release, there is no documented way to serve these models on existing kernels today, and no evidence that the storage savings carry over to inference memory. The README recommends Python 3.12, installs CUDA toolkit 12.4.1 (with a note to adjust the version if needed) and torch 2.8.0+cu124. It says reproducing the paper results requires transformers 4.51.x, because newer releases may change model internals or evaluation behavior.

The codebase supports OPT, Llama and Llama 2/3, Phi-4, Qwen2.5 and QwQ, Gemma 2 and Gemma 3, and Qwen3. It is licensed CC BY-NC 4.0, a non-commercial licence. The README does not mention any commercial licensing option. Teams considering LittleBit compression for a paid product should check the licence before investing in a QAT run.

Two commenters suggested directions the repository does not attempt. One proposed a hybrid scheme that drops to sub-bit representations only where the information does not need more, such as later layers, while keeping task performance. Another proposed raising total parameter count while quantizing individual weights, so that active parameters shrink, streaming from RAM or disk becomes more efficient, or cache behavior improves. The README examples use a single eff_bit target and document no per-layer setting, so neither idea can be tested with the released flags as documented.