Quantization, as it ships.

• LittleBit compression stores LLM weights at 0.1 bits via binarized latent factors

  October 9, 2026QuantizationTraining & Fine-TuningLLM Serving

  LittleBit compression from Samsung Labs targets 1.0 to 0.1 bits per weight via binarized low-rank factors. Code ships QAT and eval, no serving path.

• The ds4 LLM engine generates 39.4 t/s at q2 on a 128 GB M5 Max

  October 2, 2026Local InferenceLLM ServingQuantization

  ds4 LLM engine (DwarfStar 4) generates 39.4 t/s and prefills 790.2 t/s at q2 on a 128 GB M5 Max. MIT-licensed C for Metal, CUDA and ROCm, short model list.

• STEPQuant keeps delta-rule recurrent states near FP32 accuracy at a 6-bit budget

  September 30, 2026LLM ServingQuantizationLinear Attention

  STEPQuant quantizes delta-rule recurrent states by error magnitude, memory lifetime and key-row impact, staying near FP32 accuracy at a nominal 6-bit budget.

• GLM-5.3 ships open weights with no published parameter count

  August 28, 2026LLM ServingOpen WeightsModel Licensing

  Z.ai released GLM-5.3 on Hugging Face on 28 August 2026 with no parameter count, context length, or license name stated, plus a reported move off MIT.

• The local serving defaults that make an open-weight model feel dumber than it is

  August 23, 2026LLM ServingQuantizationvLLM

  A Level1Techs post argues sampler settings, chat templates, attention backends and undocumented KLD claims explain why local open-weight models underperform.

• Unsloth Dynamic 3.0 puts its accuracy gains in the small GGUFs

  August 20, 2026QuantizationLLM ServingGGUF

  Unsloth Dynamic v3.0 GGUFs for Qwen3.8-27B claim up to +10% top-1 accuracy at equal size, a new imatrix calibration set, and MTP stripped below UD-Q2_K_XL.

• Qwen3.8-27B-FP8 lands with XML tool calls and xhigh reasoning on by default

  August 14, 2026LLM ServingOpen WeightsAgent Infrastructure

  Qwen3.8-27B-FP8 implies roughly 27GB of weights. Its chat template mandates XML tool calls, xhigh reasoning by default, and raises on tool-only turns.

• Meta's 30B Muse Glimmer fits a local coding agent under 20GB, Apache 2.0

  August 10, 2026Local InferenceOpen WeightsAgentic Coding

  Meta's Muse Glimmer is a 30B open-weights agentic model under Apache 2.0, quantized to ~4-bit to fit under 20GB with room for KV cache and a drafter.

Other topics

• LLM Serving 42
• Local Inference 19
• Open-Weight Models 24
• AI Agents 32
• Coding Agents 23
• AI Agent Security 19
• Benchmarks & Evaluation 28
• Decision Models 10
• LLM Pricing & Cost 13
• Training & Fine-Tuning 8
• GPUs & Systems 17
• Web & Crawlers 7

Every note, newest first