DIAL Eval

DIAL Eval is a single place to evaluate everything your teams build on the platform. GenAI applications, AI models and MCP tools all go through the same environment: your team builds ground truth datasets and test suites here, then runs them against any solution, with every result kept together.

It replaces scattered evaluation solutions and quality stops being measured in a different way in every corner of the organization.

There is no universal measure of a good answer. What counts as correct depends on your business. Your team decides: the cases the solution must get right, the answers it should give, and how its responses are judged. A library of pre-built metrics covers the common checks, and you can add your own.

Real solutions rarely answer in one shot. An agent holds a conversation across many turns, calls tools, and works through several steps before it reaches an answer. Evaluation follows all of it: a multi-turn conversation or a chain of calls counts as one case, and every step gets scored on its own. So when quality drops, you see which step caused it instead of only knowing the final answer was poor.

A single score on its own tells you little. The value comes with the second run. Any two runs of the same test suite can be compared case by case, showing exactly what improved and what got worse after a model swap, a reworded prompt or a new tool. And across a whole series of runs, the scores form a trend, which is how slow drift surfaces before your users start noticing it.

The same ground truth and the same metrics can be pointed at another candidate, so a choice between two models, or two versions of a solution, rests on numbers measured the same way.

Most model decisions are really cost decisions. A cheaper model, a different vendor, a reserve model configured for failover but never actually tried: each one promises savings or safety, and each one risks quality that nobody has measured. Every run reports what it cost to execute alongside what it scored. A switch can then be judged on both numbers, before any production traffic moves to it.

Evaluation only helps if it actually happens. You can start a run yourself whenever you need one. You can also make it automatic, so that every new version of a solution is evaluated as it is deployed. And a case can be repeated several times in one run, so an answer that only works sometimes shows up as unstable instead of passing on luck.

DIAL Eval is a part of DIAL platform and includes two open-source projects, one running the evaluations and one calculating the metrics, both under Apache 2.0. Eval lives in DIAL Admin, works with the resources already registered on the platform, and follows the same governance and access rules as everything else. Teams already running DIAL can start evaluating without adding anything new to their landscape.

Every run is stored in full in your own environment: the requests made, the responses received, the score from each metric and the explanation behind it. Nothing is discarded when a run finishes, so the history is there to compare against later. And the results are yours to work with elsewhere, exported for analysis or connected into the observability and reporting tools your teams already run.