LLM Model Selection

A repeatable evaluation framework for on-premise production

Author: Diego Bardella · Lab: Dielabs · Reference date: November 2026

How do you choose an open-weight model to run on your own GPUs, and make the decision repeatable whenever a new candidate arrives? This framework starts with the workload’s failure modes and ends with a justified shortlist on the actual hardware and serving stack.

The starting point is Fabio Carta’s talk for Agile Lab at MLOpsDay 2026, Bologna, 29 October 2026. Its running example is CWYD, Chat With Your Data: an agent that answers business questions by querying SQL databases. Product names remain as concrete examples. The pipeline, timings and model scores attributed to the talk are Agile Lab’s reported work, not measurements made by Dielabs. Dielabs adds the agent acceptance track, the connection to capacity planning and customer golden sets, and the discussion of sampling uncertainty and limitations.

1. What this framework does—and what it does not

It does: select among shortlisted open-weight models on the real stack using repeatable criteria. The output is a table with one row per candidate and a reasoned shortlist.

It does not, by itself, establish:

• The finished agent’s quality on customer data. That is track C, using a golden set of real questions and expected outcomes, as defined in the AI Technical Presales Framework’s evaluation and acceptance process.
• Platform acceptance against the performance promised in a proposal. That belongs to the agreed acceptance criteria and benchmark protocol.
• Hardware sizing. The measured throughput becomes an input to sizing; it is not the sizing calculation itself.

Where it fits: after discovery identifies failure modes and the workload, before hardware sizing. The selected model changes the sizing inputs, so the capability decision comes first. From Idea to Production, available on request, describes the broader technical path. The AI Technical Presales Framework is available on request.

2. Why leaderboards are not enough

Public rankings answer someone else’s question. Four reasons matter:

The framework therefore asks: which model is best on this hardware, behind this stack, for this work?

3. From the use case to the benchmarks

3.1 Failure modes, capabilities, benchmarks

Build the chain by identifying what the agent must do, how it fails when it cannot, and which measurement catches that failure.

Rule: no requirement, no benchmark. If a test covers no relevant failure mode, leave it out.

The table changes with the use case; the method does not. A document RAG agent replaces “write SQL” with “find the right passage and cite it”. Its direct evaluation must assess retrieval and grounded answers with citations.

3.2 How strong is the proxy?

Not every benchmark measures the actual job. State how close each one is to it.

Weak proxies are the first place to improve when adapting the pipeline to a customer. Even a direct task benchmark on public databases does not replace evaluation on that customer’s schema.

3.3 CWYD example: one question, eight capabilities

Question: “Which region increased its margin the most in Q3 compared with Q2? Answer with a table.”

Two capabilities cut across the steps: operating at the intended load, assessed through serving tests, and any Python post-processing, assessed through executable coding tests such as HumanEval.

4. Six criteria for choosing a benchmark

Covering a failure mode is necessary but not sufficient. In this framework, a benchmark enters the suite only if it meets all six criteria.

Harness integration is a practical criterion, not a scientific one, and is stated as such: ease of repetition takes priority over an ideal measurement that is never repeated. Deterministic scoring does not guarantee deterministic model generation.

5. Three evaluation tracks

A and B select the model; C accepts the agent. The talk covers A and B. Track C is a Dielabs addition.

Track B and platform capacity. The talk measures requests per second with ShareGPT prompts. For sizing, measure both hardware-anchored capacity in closed loop, Cr_closed, and capacity under service-level objectives in open loop, Cr_open, using the method in One Capacity Is Not Enough. The result can then inform replica and GPU counts. Prompt and response lengths, traffic and concurrency must come from discovery.

Calibrate proxies against track C. Once a golden set exists, compare the A/B ranking with C. Downgrade or remove benchmarks that do not predict customer-task performance. Until this comparison is made, it is unknown how well the selection pipeline approximates the final agent’s evaluation.

6. Pipeline architecture

6.1 One endpoint, one container per harness

The talk uses an on-premise VM with one A100 80 GB and no external calls during evaluation:

YAML configuration → launcher → harness containers → adapter → inference engine

Target: one OpenAI-compatible endpoint
Execution: local
Harnesses: BFCL v3 (AST), bigcode-evaluation-harness (HumanEval+),
           lm-evaluation-harness (IFEval, GSM8K, BBH, MMLU-Pro), IFBench
Serving: vLLM on the same VM, /v1/chat/completions
Per-model settings: context, chat template, tool parser; thinking disabled
Outputs: collected results in a common format

A harness combines a dataset, prompt templates and a scoring rule. The important choices are:

• The evaluation framework does not load the model. The engine serves the model once, at a pinned version, through one endpoint. All benchmarks target it and may run concurrently within controlled limits.
• One version-pinned container per harness. Each test tool runs in its own reproducible environment.
• One command and configuration for the suite. Results use a common format, avoiding reconciliation scripts.
• One serving configuration per model. Record context limits, chat template and parsers rather than assuming identical settings work across families.

The tool used in the talk is NVIDIA NeMo Evaluator, an open-source SDK for running evaluations through different harnesses. The launcher manages configuration, execution and result export. Track B uses a custom BIRD implementation on 11 SQLite databases and vllm_bench with ShareGPT traffic.

Results: one row per model, including scores, settings and responses, versioned in Git. In the talk, a CSV is packaged into a shareable MLflow bundle that colleagues can import without a running server.

6.2 The adapter

A small configurable layer between the test and model can:

• Add a format instruction, such as “answer with the letter only”.
• Cache responses.
• Log requests and responses.
• Separate reasoning from the final answer when a model mixes them.

Write shared transformations once in configuration and apply them consistently. Record model-specific parsing explicitly so it remains auditable. A fair comparison does not silently repair one candidate more than another.

6.3 Concurrency and timing

On one GPU, too many simultaneous tests fill the serving queue and cause timeouts. The talk reports:

The total is approximately 75 minutes for a fast non-thinking model, and several hours for slower or thinking models. Duration depends strongly on how much the model generates, not just how many questions it receives. Reasoning tokens therefore affect the evaluation budget as well as production latency and capacity.

7. A zero score often points to the setup

Treat a near-zero or unexpectedly low score as a possible integration failure until investigated. Two families of causes matter.

Test-side failures

Server-side failures

Rule: check the setup before blaming the model. Compatibility with the actual server is itself a requirement: parsing problems on the deployment stack matter in production too.

A unified evaluation framework may expose multiple implementations of a benchmark, allowing a harness issue to be isolated without rewriting the entire pipeline.

8. Sampling and uncertainty

Full benchmarks on a single GPU can take days. Use a fixed sample, balanced across benchmark topics where appropriate. The talk reports 84 questions from roughly 12,000 in MMLU-Pro and 138 from roughly 6,500 in BBH. Choose sample size from the time budget, allowing margin for slow models.

A sample score is an estimate. A simple approximate 95% margin for a proportion, with p as a fraction and n as the number of questions, is:

margin ≈ 1.96 × sqrt(p × (1 − p) / n)

Conversely, around p = 0.5, n ≈ 1 / margin² is a useful rounded planning rule: about 400 questions for ±5 points, or 1,100 for ±3 points.

Decision rule: do not rank candidates confidently on small score gaps. Overlapping individual intervals are a conservative prompt to investigate further or prefer operational criteria; they do not prove equivalence. When models face the same questions, inspect paired wins and losses. For small samples or proportions near zero or one, use a more appropriate interval, such as Wilson, rather than relying on the normal approximation. Correlated tasks and stratified sampling also require care.

The talk’s leading BIRD candidates cluster between 54.0 and 57.8. Those point estimates alone do not establish a reliable ordering. This qualification makes explicit the limits of the source note’s shorthand, “overlapping bars = tie”.

9. Reading the results

9.1 Gates, capability profile and two axes

Read results in this order:

1. Minimum gates. A required capability below threshold disqualifies a candidate regardless of its average.
2. Verify before rejecting. Investigate a failing cell using section 7: is it the model or the integration?
3. Profile before average. A balanced candidate can be preferable to a higher average with a critical weakness.
4. Quality and serving efficiency together. Quality alone can select a model too slow to deploy; speed alone can select one unable to do the job. Shortlist candidates with strong quality and speed after the gates.
5. Use operational criteria for close candidates. Consider speed, failure types, license and engine support maturity, alongside statistical uncertainty.

A composite score, such as the mean of min-max-normalized benchmark scores, can help draw a chart. It must not replace the gates or the individual capability profile.

9.2 CWYD example output

The following is the Agile Lab pipeline snapshot described in the source notes as spring 2026, reported in the October talk: one A100 80 GB, with a separate serving configuration for each model. BFCL is the average of their selected categories, not the official overall score. This is an illustration of the pipeline’s output, not a current recommendation or a Dielabs benchmark. Values are reproduced from the supplied notes; raw runs were not independently reproduced here. Columns use the source’s scales: BIRD and BFCL are percentage-style scores, while the other quality columns are fractions.

Bold values reproduce the source table’s highlighted values. Read the table as follows:

• No candidate wins every column.
• Reported throughput ranges from 0.6 to 7.9 requests/s on the same A100, a roughly 13-fold spread using the displayed rounded values.
• In the talk’s quality/speed trade-off, Qwen3.5-35B-A3B emerges as the main balanced candidate. Qwen3.5-27B offers stronger quality but lower speed; gpt-oss-20b and Nemotron-Cascade are fast but miss required capabilities. These are interpretations of that snapshot, not universal model rankings.
• Very low cells in otherwise capable models, such as IQuest’s BFCL result, require the setup checks in section 7 first.

10. Pipeline reproducibility

Agile Lab’s pipeline audit provides a checklist for other implementations:

These are inexpensive fixes. Without them, the example table cannot support a defensible customer decision.

11. The benchmark lifecycle

11.1 Saturation

A benchmark loses selection value when leading candidates cluster too closely to distinguish with the available evidence. It may remain useful as a minimum gate.

In the talk’s data:

• The leading seven IFEval scores span 0.80–0.89, compared with 0.29–0.61 on IFBench. The latter shows more separation in that snapshot; uncertainty must still be considered.
• The top three HumanEval scores are 0.98.
• BIRD, BFCL and IFBench retain more discriminating value in the reported campaign.

11.2 Proposed replacements

Run a new benchmark alongside the old one for several campaigns before replacing it. This preserves a bridge to historical results. Candidates must still satisfy the six criteria in section 4; the list below is a set of proposals to qualify, not a pre-approved suite.

Notes accompanying the November 2026 framework:

• Terminal-Bench 2.0 comprises 89 terminal tasks. Harbor provides the evaluation infrastructure. The talk proposes a fixed, stratified subset of about 30 tasks in Git, fewer simultaneous long-context sessions, and turn/time limits, reserving the full set for the shortlist. Thirty tasks give a wide uncertainty band; they support coarse screening more readily than precise ranking.
• BFCL v4 expands into agentic evaluation, including web search and memory. The source note flags ageing expected web answers as a reason to review or exclude that portion. Check the pinned dataset and scoring implementation before adoption.
• IFBench introduces 58 verifiable instruction constraints and investigates generalization beyond IFEval-style instructions.
• AMBROSIA, Saparina and Lapata, NeurIPS 2024, evaluates ambiguous text-to-SQL questions, such as whether “margin” means gross or net margin.
• llama-benchy measures prompt-processing and generation speed at different context depths through OpenAI-compatible endpoints, useful when agent context grows over time.

12. Declared limitations

• Thinking disabled. The talk reports non-thinking runs to manage memory and runtime on one A100. Shorter contexts allow more concurrency. If production uses reasoning, repeat track B with it enabled: context use, latency and throughput can change substantially.
• Temperature and repeated runs. Temperature zero and thinking disabled reduce variation but do not guarantee identical runs. With active sampling or reasoning, run each candidate repeatedly and report central tendency and dispersion.
• No external LLM judge in model selection. This is a cost, repeatability and air-gap choice. A judge may be used in track C if checked against human-reviewed examples.
• Quantization. BF16, FP8 and INT4 versions are separate deployment candidates and need separate rows, even when the model name is the same.
• Generality. The talk’s suite targets a use-case category, not a particular customer. Real business vocabulary and schemas enter through track C or customer-specific benchmarks.
• Reported evidence. The talk’s results and timings are retained as attributed examples. This English publication does not provide or claim an independent reproduction of its raw runs.

13. Operational checklist

1. From discovery, record the use case’s failure modes and workload profile.
2. Map failure modes to capabilities and benchmarks, including proxy strength.
3. Apply the six selection criteria. Record why each excluded test was rejected.
4. Shortlist candidates, including quantization variants.
5. Fix the serving configuration for each model: templates, tool and reasoning parsers, context limits and engine version.
6. Version the suite: harnesses, image digests, dataset revisions/checksums, sample sizes and load presets.
7. Run A with bounded concurrency, then B. Keep a manifest for every result row.
8. Investigate zero scores and failed gates before rejecting candidates.
9. Read gates, capability profile, quality/speed trade-offs and uncertainty, in that order.
10. Run C on the leading candidate and one or two alternatives using the customer’s golden set. Compare with A/B to calibrate the proxies.
11. Rerun the same suite for each new model, and review benchmark saturation at every campaign.

Sources and attribution

• Fabio Carta, Agile Lab, Benchmarking open-source LLMs: a data-driven approach to model selection for on-premise production, MLOpsDay 2026, Bologna, 29 October 2026. Slides and transcript supplied by Diego Bardella; source of the CWYD example, pipeline, timing examples and result table. No public slide or recording URL is provided here.
• NVIDIA NeMo Evaluator repository and documentation.
• Terminal-Bench paper and Harbor.
• BFCL project. The original note also cites an Epoch AI review for its caution about web-search tasks; that is a secondary assessment.
• Generalizing Verifiable Instruction Following and IFBench repository.
• AMBROSIA paper and repository.
• llama-benchy repository.

Translated and edited from the Dielabs model-selection note of 3 November 2026. The English edition clarifies statistical uncertainty, uses the displayed throughput values for the spread calculation, and replaces vault-only references with public links or named internal methods. The Italian source note remains unchanged.