Optimizing Vision-Language Models for Production: A Deep Dive into Quantization and Pruning

TL;DR

I benchmarked popular Vision-Language Models (VLMs) like LLaVA, Qwen-VL, and PaliGemma to see how they handle quantization and pruning. The takeaway: 4-bit quantization is a no-brainer for most use cases, offering massive memory savings with minimal quality loss. For those needing speed improvements, GLU pruning (removing MLP neurons) achieved the best latency reductions, delivering significantly faster inference than head pruning or L1 methods, though at some cost to quality preservation.

Introduction

Vision-Language Models (VLMs) like LLaVA, Qwen-VL, and PaliGemma are transforming how machines understand the world. But deploying these behemoths in production—especially on edge devices or latency-sensitive applications—is a massive challenge.

In this project, I set out to answer a critical question: How much can I compress these models without breaking them?

I conducted an extensive benchmark of popular VLMs, experimenting with quantization (4-bit, 8-bit) and structural pruning to find the sweet spot between performance and efficiency. Here’s what I found.

Part 1: The Baseline Benchmark

I started by establishing a baseline. I took top contenders:

• BLIP-2: The reliable veteran.
• Qwen-VL: A powerhouse from Alibaba.
• PaliGemma: Google’s efficient, open VLM.
• LLaVA: The robust open-source standard (specifically LLaVA-1.5-7b).

The Metrics That Matter

I didn’t just look at accuracy. For production, you need to balance three things:

1. Quality: I defined a composite “Quality Score” as the average of three metrics: (METEOR + ROUGE-1 + BERTScore-F1) / 3. This gives a balanced view of n-gram overlap and semantic similarity.
2. Latency: How long does it take to get an answer?
3. VRAM: Can it fit on a consumer GPU?

Quantization: The “Free” Lunch?

I ran everything at FP16 (half-precision), 8-bit, and 4-bit quantization.

The Result? Surprisingly, 4-bit quantization is almost always worth it.

• Size: Massive reduction (often 3-4x smaller than FP16).
• Speed: Significant latency improvements on memory-bound systems.
• Quality: The drop in semantic understanding (BERTScore) was negligible for most tasks, though some nuance is lost in complex reasoning.

Part 2: Surgical Precision with Pruning

Quantization is great, but what if we want to physically remove parts of the model? I focused my pruning efforts on LLaVA, applying three structural pruning techniques:

1. GLU Pruning: Trimming the Feed-Forward Networks (MLPs). I followed the approach described by Pere Martra in his article Exploring GLU expansion ratios: Structured pruning in Llama-3.2 models. This method calculates neuron pair importance based on the Maximum Absolute Weight of the gate_proj and up_proj layers. I pruned the least important neuron pairs and resized the intermediate layers accordingly.
2. Head Pruning: Removing attention heads that contribute least to the output. To select which heads to prune, I calculated the L2 norm of the o_proj (output projection) weights for each head. I then averaged these norms across all layers and removed the specific head indices that had the lowest average importance globally. This ensures we remove the “weakest” heads consistently across the model structure.
3. L1 Pruning: Magnitude-based pruning of weights. Note: Standard PyTorch L1 pruning only applies a mask (setting weights to zero) and does not physically reduce the model size or architecture. It serves as a baseline for comparison.

The Findings

I tested pruning intensities of 30% and 70%.

• 30% Pruning: This is the safe zone. Most pruning methods retained >90% of their original quality. Head pruning performed best for quality preservation (99% retention at fp16), while GLU pruning delivered the best latency improvements (1.27s vs. 2.29s for heads).
• 70% Pruning: Quality degradation becomes significant. Head pruning retained quality better (89% vs. 77% for GLU), but GLU pruning achieved dramatically lower latency (0.51s vs. 2.08s), making it the clear winner for speed-critical applications willing to trade some quality.

The Efficiency Frontier

When I plot Quality vs. Latency, a clear “Pareto Frontier” emerges:

• Best for Quality Preservation: LLaVA (8-bit) or Qwen-VL achieve the highest quality scores while offering reasonable memory footprints.
• Best for Speed: PaliGemma’s architecture delivers sub-0.3s latency out-of-the-box, while heavily pruned LLaVA variants (70% GLU) can reach 0.51s.
• The Sweet Spot: LLaVA with 30% GLU pruning strikes an excellent balance—maintaining 92% of baseline quality while achieving 47% latency reduction.

Conclusion

After extensive benchmarking across quantization levels and pruning strategies, here are the optimal configurations for different use cases:

Recommended Configurations

For balanced performance (most use cases):

• LLaVA with 30% GLU pruning + fp16: Quality score 0.363 (~8% drop from baseline), 11GB VRAM, 1.27s latency
• This delivers significant speed improvements (47% faster) and moderate size reduction (18% smaller) while maintaining good quality

For deployment under strict memory constraints:

• LLaVA with combined 30% GLU+Heads pruning + 4-bit: Quality score 0.332, 2.8GB VRAM, 2.34s latency
• Achieves 79% size reduction from baseline with acceptable quality and comparable latency

For maximum speed (real-time applications):

• LLaVA with 70% GLU pruning + fp16: Quality score 0.305, 7.7GB VRAM, 0.51s latency
• Fastest inference (78% faster than baseline), though with noticeable quality degradation

When quality cannot be compromised:

• Standard LLaVA with 8-bit quantization: Quality score 0.396 (highest), 7GB VRAM, 8.35s latency
• Nearly preserves full quality while halving memory footprint

For naturally efficient architectures:

• PaliGemma (4-bit): Quality score 0.793 (BERTScore-F1), 2.1GB VRAM, 0.30s latency
• Excellent out-of-the-box efficiency without requiring pruning

Key Takeaways

1. 4-bit quantization is essential for memory-constrained deployments, offering 3-4x size reduction with minimal quality loss across all models tested.

2. GLU pruning outperforms head pruning for speed gains but sacrifices more quality. Choose based on your speed-vs-quality tolerance.

3. Combined pruning strategies (GLU + heads) can achieve extreme compression (>75%) but require careful validation for your specific use case.

4. Quantization and pruning compound effectively: combining 30% GLU pruning with 4-bit quantization yields both speed and size benefits.

The code for this benchmark and analysis tools are open-source. I encourage you to run these tests on your own hardware and datasets to find the optimal configuration for your specific requirements. code: —

Supplementary materials - All experimental data