Few-shot demonstrations

Few-shot optimizers improve a program by choosing good in-context demonstrations to put in front of the model. They differ in how they pick those demonstrations. All share the compile shape and return an improved program.

LabeledFewShot

The simplest: select k demonstrations directly from a labeled training set. It makes no model calls.

def labeledFewShot(student: DynamicPredict, trainset: Vector[Example])(using
    RuntimeContext
): Either[DspyError, DynamicPredict] =
  new LabeledFewShot[DynamicPredict](LabeledFewShotConfig(k = DemoCount(8)))
    .compile(student, trainset)
    .map(_.bestProgram)

BootstrapFewShot

Runs the program over the training set, keeps the examples it answers well (scored by the metric), and uses those as demonstrations:

def bootstrapFewShot(metric: Metric, student: DynamicPredict, trainset: Vector[Example])(using
    RuntimeContext
): Either[DspyError, DynamicPredict] =
  new BootstrapFewShot[DynamicPredict](BootstrapFewShotConfig(
    metric = Some(metric),
    maxBootstrappedDemos = DemoCount(4),
    maxLabeledDemos = DemoCount(16),
    maxRounds = RoundCount(1),
    maxErrors = ErrorLimit(10)
  )).compile(student, trainset).map(_.bestProgram)

BootstrapFewShotWithRandomSearch (shown on the overview) goes further, generating several candidate demonstration sets and keeping the best.

KNNFewShot

Selects demonstrations nearest to each input, using embeddings. It needs an Embedder over the training set:

def knnFewShot(student: DynamicPredict, trainset: NonEmptyTrainset, embedder: Embedder)(using
    RuntimeContext
): Either[DspyError, DynamicModule] =
  new KNNFewShot[DynamicPredict](k = NeighborCount(3), trainset = trainset, embedder = embedder).compile(student)

Choosing one

Optimizer	How it picks demos	Cost
LabeledFewShot	Straight from the trainset	No model calls
BootstrapFewShot	Self-generated, metric-filtered	Some model calls
BootstrapFewShotWithRandomSearch	Searched candidate sets	More model calls
KNNFewShot	Nearest neighbors per input	Needs an embedder

Next: Instruction optimization.