Quantization of LLM: Auto-ADPQ

As Large Language Models (LLMs) continue to grow in size and capability, deploying them efficiently has become increasingly challenging. Enter Auto-ADPQ (Adaptive Post-Training Quantization) - a Python package I developed that implements a zero-shot, calibration-free method for quantizing LLMs to 4-bit precision while maintaining model quality.

This project is part of my master’s thesis and aims to replicate the ideas from the paper “AdpQ: A Zero-shot Calibration Free Adaptive Post Training Quantization Method for LLMs”. The implementation is available as an open-source package on PyPI and GitHub.

The Problem: Memory Bottleneck

Modern LLMs like Llama 3.1 with 8 billion parameters require substantial memory:

16-bit (bfloat16): ~16GB VRAM
8-bit: ~8GB VRAM
4-bit: ~4GB VRAM

Traditional quantization methods either require calibration datasets or suffer from significant quality degradation. Auto-ADPQ solves this by using an adaptive outlier detection mechanism inspired by LASSO optimization.

How Auto-ADPQ Works

The core innovation of AdpQ lies in its data-free calibration approach. Instead of requiring a dataset to calibrate the quantization parameters, it uses a mathematical optimization technique to identify and handle outlier values:

1. Outlier Detection via LASSO

The algorithm splits weight matrices into groups and applies a LASSO-inspired optimization to detect outlier values:

\[\text{minimize} \quad \|\mathbf{w}\|_0 + \lambda' \|\mathbf{w}\|_1\]

This is solved using a bisection method (similar to Brent’s method) to find the optimal regularization parameter \(\lambda'\) that achieves a target outlier ratio \(\alpha\).

2. Dual Quantization Scheme

Non-outlier values: Quantized to 4 bits using symmetric or asymmetric quantization
Outlier values: Stored separately with higher precision to preserve critical information

3. Efficient Storage

The quantized weights are stored in a custom format that includes:

Quantized non-outlier weights (optionally bit-packed)
Outlier indices and values
Scale and zero-point parameters per group

Performance Results

I’ve quantized several Llama models and compared them against other quantization methods like BitsAndBytes (BNB), AWQ, and GPTQ. Here are the perplexity scores (lower is better):

Llama 3.1 8B

Method	Perplexity
Baseline (bf16)	4.8693
BNB 4-bit	5.0733
AdpQ 4-bit	5.3671

Llama 3.1 8B Instruct

Method	Perplexity
Baseline	4.9080
BNB	4.9993
AdpQ	5.0069
AWQ	5.0440

Llama 3.2 3B Instruct

Method	Perplexity
Baseline	5.7864
AWQ	5.8339
AdpQ	5.9040

The results show that Auto-ADPQ achieves competitive performance, particularly excelling with instruction-tuned models.

Quick Start

Installation is straightforward via pip:

pip install auto_adpq

Here’s a minimal example to quantize a model:

from auto_adpq import Auto_AdpQ, AutoAdpQConfig
from transformers import AutoModelForCausalLM

# Configure quantization
config = AutoAdpQConfig(
    group_size=128,
    n_iters=250,
    alpha=0.05,  # 5% outlier ratio
    q_bit=4,
    symmetrical_quantization=True,
)

# Load and quantize model
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    torch_dtype="bfloat16"
)

# Apply quantization (virtual/simulated)
Auto_AdpQ.apply_quantization(model, config, multi_threaded=16)

# Push to HuggingFace Hub
model.push_to_hub("your-username/llama-3.1-8b-adpq-4bit")

Key Features

Zero-shot calibration: No dataset required
Adaptive outlier detection: Automatic threshold selection
Multi-threaded quantization: Fast processing with configurable worker threads
Flexible configuration: Support for symmetric/asymmetric quantization, adjustable bit-width
High test coverage: 91% code coverage
Well-documented: Full documentation on ReadTheDocs

Pre-quantized Models

I’ve published several quantized models on HuggingFace as part of the AdpQ collection:

Llama-3.1-8B-adpq-4bit-sim
Llama-3.1-8B-Instruct-adpq-4bit-sim
Llama-3.2-1B-adpq-4bit-sim
Llama-3.2-3B-Instruct-adpq-4bit-sim

Note: These are “simulated” models stored in bfloat16 format that represent the quality and rounding errors of true 4-bit quantization. Implementing custom CUDA kernels for efficient inference remains future work.

Technical Deep Dive

The Optimization Algorithm

The heart of Auto-ADPQ is the lasso_outlier_detection method, which uses a bisection algorithm to find the optimal regularization parameter:

def lasso_outlier_detection(self, matrix):
    """Detect outliers using LASSO-inspired optimization."""
    x0, x1 = 0.0, np.max(np.abs(matrix))
    target_outlier = self.alpha * matrix.size
    
    # Bisection loop
    for iteration in range(self.n_iters):
        n_outlier = self._optimization_function_fast(matrix, x1)
        fx1 = n_outlier - target_outlier
        
        if abs(fx1) < tolerance:
            break
            
        # Brent-like interpolation
        x_new = self._brent_function(x1, x0, fx1, fx0)
        x0, x1 = x1, x_new
    
    return outlier_mask, lambda_prime

Group-based Quantization

Weights are divided into groups (typically 128 elements) to maintain fine-grained quantization parameters:

def quantize(self, sub_vector):
    """Quantize a group of weights."""
    if self.symmetrical_quantization:
        max_abs = np.max(np.abs(sub_vector))
        scale = (2**(self.q_bit - 1) - 1) / max_abs
        quantized = np.round(scale * sub_vector).astype(np.int8)
        return quantized, scale, np.nan

Development Insights

Building Auto-ADPQ came with several challenges:

1. Data Packing Implementation

Efficiently packing 4-bit values into 8-bit integers required careful bit manipulation. I initially struggled with this (#1 in the issues) but eventually solved it, achieving proper serialization and deserialization.

2. Performance Optimization

The outlier detection algorithm needs to run on every weight group. I implemented:

Multi-threading: Process multiple layers in parallel (16 workers by default)
Vectorized NumPy operations: Avoid Python loops wherever possible
Fast outlier counting: _optimization_function_fast counts outliers without materializing the full mask

3. Validation and Testing

With 91% test coverage, I’ve ensured the implementation is robust:

Unit tests for quantization/dequantization
Integration tests with actual model weights
Perplexity evaluation on WikiText-2

Future Work

The project roadmap includes:

Custom CUDA kernels: Enable efficient inference with true 4-bit weights
Integration with .safetensors: Better model serialization
SpQR integration: Combine with sparse quantization for even better compression
Pydantic optimization: Reduce overhead in AdpQQuantizedWeights class

Conclusion

Auto-ADPQ demonstrates that calibration-free quantization is not only possible but can achieve competitive results with existing methods. The zero-shot nature makes it particularly useful when:

You don’t have access to representative calibration data
You want to quickly experiment with different quantization settings
You need a simple, dependency-light quantization pipeline

The complete source code, documentation, and examples are available on GitHub. Contributions are welcome!

Data-free calibration for efficient quantization