Open-Weight Model Families & Model Selection

A Decision Framework for On-Prem Inference

April 2026

By Robert Barcik
LearningDoe s.r.o.

About This Booklet

You have GPUs. You have a model running in production. It works. Tickets get resolved, code gets reviewed, documents get summarised. Then someone on your team opens HuggingFace, sees a new model with better benchmark scores, and asks: should we switch?

You do not know. Not because you are uninformed, but because the question is unanswerable without context. Switch for which task? On which GPU? At what quantisation? Serving how many users? Under what license? The model is one variable in a system of constraints, and the leaderboard answers none of them.

This booklet does not tell you which model to pick today. By the time you read this, the rankings will have shifted. What it gives you is the machinery to answer the question yourself — this time and every time after. Part 1 maps the five major families: who builds them, why, and what that means for you. Part 2 connects the landscape to your actual hardware. Part 3 gives you a decision framework and four scenarios to stress-test it.

The numbers are current as of April 2026. The framework should outlast them.

Who This Is For

• Operations and services professionals deploying AI on company hardware
• IT teams managing GPU infrastructure for internal or client workloads
• Technical decision-makers evaluating which models to run on-prem
• Anyone deploying LLMs on datacenter or workstation GPUs who needs to make model choices without a dedicated ML research team

How to Read

Read in order for the full picture. If you already know the model families and just need the practical selection guidance, skip to Part 2. If you want the decision framework and exercise, jump to Part 3 — but you will get more from it having read Parts 1 and 2 first.

Table of Contents

Part 1: Model Families

1. The Landscape in 2026
2. Family by Family
3. Beyond LLMs

Part 2: Practical Selection

4. The Hardware Reality
5. What Fits Where
6. Quantization
7. Inference Frameworks
8. Memory & Throughput

Part 3: Decision Framework

9. The Decision Framework
10. Scenario Exercise
11. What to Watch

Why Not Just a Comparison Table?

In December 2024, the safe recommendation was straightforward: Llama 3.3 70B, run it on an H100, done. Six months later, Qwen 3 had overtaken it on most benchmarks. Six months after that, Gemma 4 arrived with Apache 2.0 licensing and Mistral Small 4 rewrote what a 6B-active-parameter model could do. Anyone who built their strategy around a December 2024 comparison table spent 2025 on the wrong model.

The problem is not that comparison tables are wrong when published. The problem is that they are right for about eight weeks. The mean engagement duration for any model on HuggingFace is approximately six weeks before it is superseded. Building your model strategy around a snapshot is like choosing a server vendor based on last quarter’s benchmark — the number was accurate, the decision was not.

What does not change every eight weeks: your hardware constraints, your use case requirements, the licensing implications, and the inference cost structure. A team that understands these variables can read a model card on release day and reach a defensible decision by end of day. A team that waits for someone else’s recommendation table is always two months behind.

The goal: When the next model drops, you evaluate it yourself — against your hardware, your constraints, your use cases — without waiting for someone else’s table.

The Landscape in 2026

Eighteen months ago, if someone asked you to run an LLM on your own hardware, the answer was some variant of Llama. Today, five organisations compete to be the default choice, and their models have converged on a licensing model that would have seemed unlikely in 2024: genuine permissive open licensing.

The Families

Five families dominate the open-weight landscape: Llama (Meta), Gemma (Google DeepMind), Qwen (Alibaba Cloud), Mistral (Mistral AI, Paris), and Phi (Microsoft). Between them, they cover parameter counts from 600 million to 680 billion, every major modality, and nearly every viable on-prem deployment scenario.

These families are more stable reference points than specific model versions. A version (Llama 3.3, Qwen 3.5) might be superseded in months. The family behind it — its philosophy, its licensing approach, its community ecosystem, its preferred architecture — evolves more slowly and tells you more about where your investment will end up.

Two Architecture Eras

Understanding one distinction will help you make sense of everything that follows: dense versus Mixture of Experts (MoE).

A dense model activates all its parameters for every token. A 70B dense model does 70 billion computations per token and needs enough memory to hold all 70 billion parameters. Straightforward.

A MoE model has a large total parameter count but activates only a fraction per token. Mistral Small 4 has 119B total parameters but activates only ~6B per token. You get frontier-quality outputs at small-model inference cost — but you still need enough memory to store all 119B parameters. MoE trades memory for compute efficiency.

Every frontier open-weight release since mid-2025 uses MoE: DeepSeek R1, Qwen 3.5, Mistral Large 3, Llama 4, Gemma 4’s 26B variant. Dense models (27–32B) remain the sweet spot for single-GPU deployment, but the trend is clear.

The Licensing Convergence

The biggest practical shift of the past year is not a model — it is a license. Apache 2.0 has become the default for new open-weight releases. Gemma 4, Qwen 3/3.5, Mistral’s current lineup, and DeepSeek V3 all use it. Microsoft’s Phi uses MIT (even more permissive). The only major holdout is Meta’s Llama, which uses a custom community license with branding requirements, a 700M monthly-active-user cap, and — critically for EU-based teams — restrictions on multimodal models in the EU.

For an IT services company, this means that four of the five families impose zero commercial restrictions on your use. You can deploy them for clients, embed them in products, fine-tune them, and redistribute derivatives without legal friction. That was not the case 18 months ago.

Family by Family

Do not read this section as a catalog. Read it as context for decisions you will make in Part 3. For each family, the question that matters is: under what circumstances would I choose this one? The corporate strategy behind each is a single sentence — enough to understand their commitment to open weights. The rest is what you need for model selection.

Specific model versions (Llama 3.3, Gemma 4, Qwen 3.5) will be superseded. The families, their licensing philosophies, and the “when to choose” criteria are more stable — those are what to retain.

Licensing at a Glance

Beyond LLMs: When Smaller Models Win

The default instinct — “just use the LLM for everything” — is one of the most expensive mistakes an operations team can make. An LLM is a generalist. For defined, repeatable tasks, a specialised small model will be faster, cheaper, and often more accurate.

Consider a ticket routing task. A fine-tuned DistilBERT (66M parameters) achieves 95%+ accuracy at 2–5ms latency. A 70B LLM achieves 85–92% accuracy at 500ms–5s latency. At a million classifications per month, the cost difference is roughly $50 versus $5,000–50,000 in GPU infrastructure.

Here is what the specialised model landscape looks like:

The Right-Sizing Principle

The optimal architecture uses each model for what it does best. A practical IT operations pipeline:

Steps 1–4 use approximately 1.2GB VRAM total and complete in under 100ms. The LLM in step 5 is called only when needed, with pre-filtered context, reducing LLM costs by 80%+ while improving answer quality.

The principle: Do not use a 70B model for a task a 300M model handles better. Reserve the LLM for reasoning and generation. Use specialised models for classification, extraction, search, and ranking.

The Hardware Reality

Before you pick a model, you need to know what your hardware can actually do. Not what the marketing says — what happens when you load a model and start generating tokens.

Two numbers determine your experience: memory capacity (can the model fit?) and memory bandwidth (how fast can it generate?). The second one is less intuitive but more important for user experience.

Why Bandwidth Matters More Than You Think

During token generation, the GPU must stream the entire model from memory for every single token. The formula is simple:

tokens/second ≈ memory bandwidth ÷ model size in bytes

This is why the DGX Spark, despite having 128GB of memory (more than an H100), generates tokens 12× slower than the H100 for models that fit on both. The Spark’s 273 GB/s bandwidth is just 8% of the H100’s 3.35 TB/s. It can load enormous models. It cannot run them fast.

The practical threshold for interactive use is roughly >10 tokens/second. Below 5 tok/s, users will find the experience frustrating. This directly constrains which models are viable for which hardware — not just whether they fit, but whether they are usable.

The VRAM Budget

A model’s memory footprint is not just its weights. You also need room for the KV cache (which grows with context length and concurrent users) and framework overhead.

The base formula: weight memory = parameters × bytes per weight. At FP16 (2 bytes per parameter), a 70B model needs ~140GB just for weights. At Q4 (0.5 bytes), it drops to ~35GB. The 1.3–1.5× multiplier accounts for KV cache and overhead at moderate concurrency.

The KV cache deserves special attention. For Llama 3.1 70B at BF16, it consumes roughly 0.31 MB per token per request. At 8K context with 4 concurrent users, that is ~10GB — manageable. At 128K context with 4 users, it is ~160GB — exceeding the model weights. Long context and high concurrency are the hidden VRAM killers.

What Fits Where

Here is the table that matters. For each model size, it shows the VRAM required at different precision levels and whether it fits on your three hardware targets. “Fits” means loads with room for KV cache at moderate usage. “Tight” means it loads but leaves minimal headroom.

The VRAM column values are durable — they follow from the formula (params × bytes). The specific model names will rotate; the size tiers and memory math will not.

The rule of thumb: multiply billion parameters by 2 to get GB at FP16. Divide by 2 for FP8, by 4 for Q4. Then multiply by 1.3–1.5× for practical deployment with KV cache headroom.

Key insight: MoE models are the DGX Spark’s best friend. A 120B MoE that activates only 20B parameters per token runs at 40–55 tok/s on the Spark — because the bandwidth bottleneck scales with active parameters, not total. A 70B dense model on the same hardware crawls at 4–8 tok/s. If you are deploying on the Spark, think MoE first.

Quantization: The Compression Trade-off

Quantization reduces the precision of model weights to fit larger models in less memory and run them faster. The question operations teams care about: how much quality do you actually lose?

The short answer: at 4-bit, almost nothing. Below 4-bit, a lot.

The Format Zoo, Simplified

Five quantization formats matter in practice. Do not worry about the rest.

Where to Get Pre-Quantised Models

You almost never need to quantise yourself. On HuggingFace, look for bartowski and Unsloth for GGUF models (every quant level, importance-matrix calibrated). Check the nvidia/ namespace for NVFP4 models. Model publishers increasingly provide official AWQ/GPTQ variants.

When evaluating pre-quantised models: prefer K-quants (Q4_K_M) over legacy quants (Q4_0), check download counts for community validation, and verify the upload date — quantisation of a brand-new model may have bugs in the first week.

Inference Frameworks

You have picked a model, chosen a quantisation level, and confirmed it fits your hardware. Now: what software do you use to actually run it?

The right answer depends on one question: how many concurrent users do you need to serve?

The Five That Matter

The Decision in Practice

One note on HuggingFace TGI: it entered maintenance mode in December 2025. HuggingFace now officially recommends vLLM or SGLang for new deployments. If you see TGI mentioned in tutorials, know that it is a legacy recommendation.

Memory & Throughput Realities

A model that “fits” is not the same as a model that “performs.” Here is what your hardware actually delivers at the token level.

Single-User Decode Speed

Snapshot data — April 2026. These numbers change with firmware updates, framework versions, and new model architectures. The formula (bandwidth ÷ model size) is durable; the specific tok/s values are not. Re-benchmark when your stack changes.

This is the number that determines whether a model feels interactive. Measured in tokens per second during generation:

The pattern is clear. On the DGX Spark:

• 8B–14B dense models at 20–40 tok/s — comfortable for interactive use
• 27–32B dense models at 9–11 tok/s — borderline interactive
• MoE models up to 120B at 40–55 tok/s — comfortable, because bandwidth scales with active parameters
• 70B dense at 4–8 tok/s — not viable for interactive use

The H200 Advantage

Moving from H100 to H200 gives you a consistent ~1.4× speedup from the bandwidth increase (4.8 versus 3.35 TB/s). But the bigger win is capacity: Llama 70B at FP8 (~84GB with overhead) fits on one H200 but needs two H100s with tensor parallelism. That eliminates inter-GPU communication overhead and effectively doubles your serving capacity per model.

Batching Changes Everything

The single-user numbers above tell only half the story. Batched inference shifts the bottleneck from bandwidth to compute. Llama 3.1 8B on the DGX Spark at batch size 32 achieved 368 tok/s total throughput via SGLang — an 18× increase over single-user. If your use case is a queue of requests rather than an interactive chat, smaller models batched efficiently can handle surprisingly high volumes.

Practical guidance: An H100 will run circles around the DGX Spark for any model that fits in 80GB. The Spark’s value is running models that do not fit on an H100 (120B MoE, prototyping 200B+) and serving as a quiet, desk-side development environment. When a model graduates from prototyping to production, move it to datacenter GPUs behind vLLM.

The Decision Framework

When a new model drops, do not start with the benchmarks. Start with your constraints. The following sequence will get you to a defensible choice in under an hour.

Six Questions to Ask Every Time

When to Re-Evaluate

You do not need to re-evaluate every time a new model drops. Re-evaluate when:

• Your hardware changes (new GPUs, new memory capacity)
• Your use case requirements change (new language, higher concurrency, new modality)
• A new model family appears (not a version update — a genuinely new player)
• Your current model’s family changes licensing (check release notes, not just headlines)
• You see a >20% improvement on your specific task in independent benchmarks (not leaderboard games — your actual evaluation set)

Version updates within a family (Qwen 3 → Qwen 3.5) are worth a quick test on your evaluation set. They are not worth a full re-architecture.

Scenario Exercise

Four scenarios. For each, use the decision framework to choose a model, quantisation level, and inference framework. Justify your choice. Then reveal the suggested approach and compare.

There is no single correct answer — the point is whether your reasoning follows the framework and whether you can defend your choice against the constraints.

What to Watch

The landscape changes fast. Here is what to monitor so you know when to re-evaluate.

Signals That Matter

• New model releases from the five families. Not every release requires action. Check: is it a new architecture or just a version bump? Does it target a size range you use? Does it change the licensing? A new Qwen 4 with a different license matters. A Qwen 3.5.1 patch does not
• Hardware changes. When new hardware arrives — H200s, next-gen Blackwell, a second DGX Spark — re-run the “what fits” analysis. Models that were too large or too slow may suddenly become viable. Two H200s (282GB combined) open up the 235B MoE tier
• Quantisation breakthroughs. Recent advances in 2-bit quantisation (like QuIP# and AQLM) are narrowing the quality gap below Q4. If 2-bit reaches the Q4 quality level, it doubles the effective models you can fit on existing hardware
• Inference framework updates. SGLang and vLLM release roughly monthly. Major features (better MoE support, new quantisation backends, speculative decoding improvements) can change throughput by 30–50%
• Licensing changes. Track the specific model version’s license, not the family’s reputation. Mistral went from Apache 2.0 to restrictive and back. Gemma went from restrictive to Apache 2.0. The next shift could go either direction for any family
• EU regulatory developments. The EU AI Act is now enforceable. Watch for guidance on training data transparency requirements (affects Qwen), high-risk AI system classifications, and any new restrictions on model deployment in regulated sectors

Where to Look

The open-weight landscape in April 2026 offers more options, better licensing, and more capable models than at any point in the history of accessible AI. Five families compete fiercely. Apache 2.0 is the norm. A single H100 runs models that would have required a data centre three years ago.

But the models are not the hard part. The hard part is matching the right model to the right task on the right hardware with the right trade-offs — and having the confidence to make that call yourself when the next release drops.

That is what the framework is for. Use it.

Robert Barcik · barcik.training · LearningDoe s.r.o. · April 2026