Configuring a language model

A program needs a language model to run. For text generation, dspy4s ships one provider, OpenAiLanguageModel, which speaks the OpenAI /chat/completions shape. That covers OpenAI and every OpenAI-compatible server (Azure, Ollama, vLLM, SGLang, LM Studio, OpenRouter).

Constructing a model

Pass a model name and an API key:

def openAi(model: String, apiKey: String): LanguageModel =
  OpenAiLanguageModel(model, apiKey)

For a local server that does not check credentials, use local with the server's base URL:

def ollama(model: String = "llama3.2"): LanguageModel =
  OpenAiLanguageModel.local(model, baseUrl = "http://localhost:11434/v1")

OpenAiLanguageModel.fromEnv(model) reads OPENAI_API_KEY from the environment, which is what the bundled examples use.

Installing it

A model becomes active by putting it in the runtime context. The Quickstart shows the full wiring with RuntimeEnvironment.withInstalled.

Calling a model directly

Most of the time a module calls the model for you. When you need the raw call, use LanguageModel.call, which returns Either[DspyError, LmResponse]:

def callDirect(prompt: String)(using RuntimeContext): Either[DspyError, String] =
  SettingsProgramRuntime.default.resolveModel.flatMap { lm =>
    val request = LmRequest(
      model = lm.id,
      messages = Vector(Message(role = MessageRole.User, text = Some(prompt)))
    )
    lm.call(request).map(_.outputs.headOption.map(_.text).getOrElse(""))
  }

A reply with no text is not an error of a direct call: its output has an empty text. A module gives a parse error for such a reply.

Per-call generation settings

Generation parameters such as temperature go in a per-call config bag. rolloutId is a dedicated field used to bust the cache for an otherwise-identical call:

def askWithConfig(question: String, temperature: Double, rolloutId: Int)(using
    RuntimeContext
): Either[DspyError, String] =
  qa(ProgramCall(
    input = (question = question),
    config = DynamicValues.record("temperature" := temperature),
    rolloutId = Some(rolloutId)
  )).map(_.output.answer)

Errors

dspy4s never throws on a model failure. Every call returns an Either, and DspyError carries a stable code and message.

A language model failure is an LmError. Its class tells the kind of the failure: LmRateLimitError, LmAuthError, LmTimeoutError, ContextWindowExceededError, and others. Its details hold the model, the provider, the HTTP status, the request id, and the delay that the provider asked for:

def askHandlingErrors(question: String)(using RuntimeContext): String =
  ask(question) match
    case Right(answer)                       => answer
    case Left(_: ContextWindowExceededError) => "Reduce the prompt, the retrieved passages, or the demos."
    case Left(e: LmRateLimitError)           =>
      val provider = e.details.provider.getOrElse("the provider")
      val delay    = e.details.retryAfter.fold("an unknown delay")(_.toString)
      s"Rate limited by $provider; retry after $delay"
    case Left(e: LmError) =>
      val model     = e.details.model.getOrElse("unknown")
      val requestId = e.details.requestId.getOrElse("unknown")
      s"LM failed: code=${e.code}, model=$model, request_id=$requestId"
    case Left(err) => s"Failed: code=${err.code}, message=${err.message}"

Next: Adapters.