model-quantization-recipes collects reproducible workflows for FP8, NVFP4, AWQ, and TensorRT-based inference. Each recipe is designed as a self-contained engineering reference with its own ...

We propose a novel post-training quantization method for large language models with learnable parameters, novel loss function and Test-time adaptation scheme. Post-training quantization (PTQ) for ...

Companies running large language models face a persistent bottleneck: the memory consumed by key-value caches during inference grows with every token generated, forcing operators to choose between ...

Large language models have moved out of the research lab and into engineers’ daily workflow. LLMs serve as reasoning engines ...

OpenAI has found a way to reduce its inference costs by roughly 50%, a development that could reshape the economics of running large language models at scale. Inference is the process of actually ...

Abstract: This paper presents a comprehensive evaluation of microscaling (MX) quantization in the pre-training of large language models (LLMs), investigating its potential to enhance the computation ...

By Pietro Antonio Ciclese, Senior Technical Marketing Engineer, Ambarella The workloads that generate the most commercial ...

Tether successfully integrated Google’s TurboQuant into the inference engine of its local AI framework, QVAC. It is the ...