Skip to the content
searchboar.

AI search and visibility

Model evaluation: meaning, examples and practical use

Model evaluation systematically checks how well an AI model or a system built around it performs a defined task. It requires test cases, predetermined criteria and interpretable results. A strong general leaderboard position cannot replace checking your own language, questions and operating conditions.

Also known as: AI evaluation · LLM evaluation

At a glance

  • Define the task and success before choosing a metric.
  • Review individual failures as well as aggregate scores.
  • Changing the model, prompt or retrieval can change the result.
On this page
Several test answers are assessed using separate accuracy and source-support checklists.
Conceptual illustration: several test answers are assessed using separate accuracy and source-support checklists.

What should the system do well?

Summarising a document, answering a customer and drafting an article are different tasks. Evaluation should reflect that distinction. Google’s evaluation guidance starts with relevant prompts, selected metrics and review of the results. Evaluation workflow

Replace “it generally sounds good” with explicit success criteria: preserving all conditions, writing clear sentences in the required language and avoiding invented details when information is missing.

What belongs in the test set?

Include typical questions, less common exceptions and cases with insufficient evidence. A support system, for example, should handle two similar products with different warranty terms.

Do not limit the test to easy demonstration questions. If you repeatedly refine the system against the same examples, check fresh questions to see whether improvements generalise. Training data and material intended to assess performance serve different roles.

The size and composition of the test should match the decision. A handful of polished examples can reveal a problem, but cannot establish reliability across a large, varied workload.

Example: which system answers better?

Imagine a shop comparing two configurations on 40 of its own questions. One produces smoother prose but omits some return-policy exceptions; the other writes shorter answers while preserving the conditions. A single “quality” score may conceal that trade-off.

Assess factual correctness, groundedness, clarity and handling of missing information separately. When selecting an operational service, latency and actual cost are additional decision criteria.

How can results be compared fairly?

Record the model version, prompt, sources, date and settings. Answer variability means some tasks also need repeated runs. Read incorrect and disputed answers alongside the aggregate results.

An automated judge can itself be mistaken, so important cases deserve human review. Evaluation provides evidence about a defined task. It does not establish that a model is reliable in every subject, nor does it demonstrate Google rankings or AI recommendations.

Sources

  1. Google Cloud: Generative AI evaluation overview