Running evaluations

Evaluate runs a program over a dev set and scores each result with a metric. It is the batch counterpart to calling a program once: you get an aggregate score across many examples.

Building an evaluator

Evaluate is configured with the dev set, the metric, and a few display and concurrency options:

def evaluator(devset: Vector[Example], metric: Metric): Evaluate =
  new Evaluate(EvaluateConfig(
    devset = devset,
    metric = metric,
    numThreads = Some(ThreadCount(1)),
    displayProgress = true,
    displayTable = Right(5)
  ))

You launch it by applying it to a program inside a runtime context:

evaluator(devset, metric).apply()(program)

The program is a function Example => Either[DspyError, RawPrediction], and the result carries the aggregate score plus the per-example outcomes.

The shape of evaluation

1. Build a dev set of Examples.
2. Choose a metric.
3. Run Evaluate to get a score.
4. Feed the same dev set and metric into an optimizer to improve the program, then evaluate again to confirm the gain.

Next: Optimization.