Examples & data

An Example is a single labeled data point: a set of named fields, with some of them marked as inputs. Examples are the unit of data that metrics score and optimizers learn from.

Building an example

val qaPair: Example = Example("question" := "This is a question?", "answer" := "This is an answer.")

Inputs and labels

An example holds both the inputs a program receives and the labels it should produce. Mark which fields are inputs with withInputs; the rest are treated as labels. inputs and labels split a row accordingly:

val articleSummary: Example =
  Example("article" := "This is an article.", "summary" := "This is a summary.").withInputs(Set("article"))
def inputKeyOnly = articleSummary.inputs // -> { article: "..." }
def nonInputOnly = articleSummary.labels // -> { summary: "..." }

This split is what lets a metric compare a program's output against the expected labels.

Datasets

A dataset is just a Vector[Example]. Build one directly and slice it with ordinary Scala (xs.take(75), xs.drop(75)) to make train and dev splits:

val trainset: Vector[Example] = Vector(
  Example("report" := "LONG REPORT 1", "summary" := "short summary 1").withInputs(Set("report")),
  Example("report" := "LONG REPORT 2", "summary" := "short summary 2").withInputs(Set("report"))
)

Next: Metrics.