Stop Hand-Tuning Prompts: A Practical Guide to DSPy Optimizers

You have a classification prompt that works "well enough." Then you switch models, or the vendor ships a silent update, or a new edge case shows up in production. You reopen the prompt, add another bullet point, run it against five examples by hand, and ship. Three weeks later you do it again. Nobody can tell you what the prompt's accuracy actually is, and nobody wants to touch it.

This is the problem DSPy attacks. Instead of treating the prompt as a string you edit, DSPy treats it as a program you compile against a metric and a dataset. You declare what you want, provide examples and a scoring function, and an optimizer searches for the instructions and few-shot demonstrations that maximize your score.

This guide walks through the mental model, a runnable example, the optimizer lineup, and the failure modes that bite people in week two.

The core abstractions

DSPy (Declarative Self-improving Python, from the Stanford NLP group) has three pieces you need to understand.

Signatures declare input and output fields in natural language. question -> answer is a signature. So is document, question -> answer: bool, evidence: str. The signature is a contract, not a prompt — DSPy generates the actual prompt text from it.

Modules implement a reasoning strategy over a signature. dspy.Predict does a single completion. dspy.ChainOfThought adds a reasoning field before the answer. dspy.ReAct runs a tool-use loop. You compose modules into larger programs like you compose functions.

Optimizers (the DSPy docs still sometimes call them "teleprompters") take your program, a training set, and a metric, then produce a new version of the program with tuned instructions and/or few-shot examples baked in. This is the "compile" step.

The important shift: you spend your time on the metric and the data, not on prompt wording. If your metric is good, the optimizer's output is good. If your metric is sloppy, no amount of manual prompting saves you — and now you know that.

A runnable example: labeling support tickets

Say you want to route support tickets into billing, bug, feature_request, or other. Here's the whole loop.

import dspy

# 1. Configure a model. Any provider DSPy's LiteLLM layer supports works here.
lm = dspy.LM("openai/gpt-4o-mini", max_tokens=200)
dspy.configure(lm=lm)

# 2. Declare the task as a signature.
class RouteTicket(dspy.Signature):
    """Classify a customer support ticket into exactly one category."""
    ticket: str = dspy.InputField()
    category: str = dspy.OutputField(desc="one of: billing, bug, feature_request, other")

# 3. Wrap it in a module.
router = dspy.ChainOfThought(RouteTicket)

# 4. Build a dataset of dspy.Example objects.
train = [
    dspy.Example(ticket="You charged me twice this month", category="billing").with_inputs("ticket"),
    dspy.Example(ticket="App crashes when I export to PDF", category="bug").with_inputs("ticket"),
    dspy.Example(ticket="Please add dark mode", category="feature_request").with_inputs("ticket"),
    # ... aim for 50-200 examples
]
dev = train[40:]   # hold some out
train = train[:40]

# 5. Define a metric. Return value can be a bool or a float.
def exact_match(example, pred, trace=None):
    return example.category.strip().lower() == pred.category.strip().lower()

# 6. Evaluate the baseline.
from dspy.evaluate import Evaluate
evaluator = Evaluate(devset=dev, metric=exact_match, num_threads=8, display_progress=True)
baseline_score = evaluator(router)

# 7. Compile with an optimizer.
from dspy.teleprompt import MIPROv2
optimizer = MIPROv2(metric=exact_match, auto="medium")
optimized_router = optimizer.compile(router, trainset=train)

optimized_score = evaluator(optimized_router)
print(f"{baseline_score:.1%} -> {optimized_score:.1%}")

# 8. Persist the compiled program.
optimized_router.save("router_v1.json")

Two things to notice. First, you never wrote a prompt. The instruction string on the signature docstring is a starting hint; MIPROv2 proposes and tests better ones. Second, everything is measured — you have a number before and after, on held-out data.

API note: DSPy's surface has changed across the 2.x and 3.x releases. Import paths, optimizer names, and the auto parameter have moved. Treat the code above as the shape of the workflow and check the version-matched docs at dspy.ai for exact signatures before you run it.

Choosing an optimizer

DSPy ships several optimizers with very different cost and data profiles. Pick based on how many labeled examples you have and how much you can spend compiling.

Optimizer	What it tunes	Examples needed	Compile cost	Use when
`LabeledFewShot`	Picks few-shot demos from your trainset	10–50	Near zero	You just want to inject examples, no search
`BootstrapFewShot`	Generates demos by running the program and keeping successes	20–100	Low	Small dataset, want self-generated CoT traces
`BootstrapFewShotWithRandomSearch`	Above + searches over demo sets	50–200	Medium	You have a dev set and some patience
`MIPROv2`	Jointly optimizes instructions and demos via Bayesian search	100–500+	High (many LM calls)	Production task, metric is trustworthy, budget for compile
`BootstrapFinetune`	Distills the compiled prompt into fine-tuned weights	300+	High + training	Latency/cost matters, task is stable
`GEPA` (reflective)	Uses LM reflection on failures to rewrite instructions	50–200	Medium–high	Fewer rollouts available, want interpretable prompt edits

A reasonable path: start with BootstrapFewShot to sanity-check that your metric moves, then graduate to MIPROv2 once you have 100+ examples and trust the metric.

Why the metric is the whole game

The optimizer maximizes whatever you give it. This has direct consequences.

Boolean metrics lose information. exact_match can't tell "close but wrong format" from "confidently wrong category." A metric that awards partial credit for a parseable-but-wrong answer gives the optimizer a smoother gradient to climb.

Cheap metrics let you search more. If your metric requires an LM-as-judge call, every optimizer iteration costs 2x. For anything with a checkable structure — JSON schema, enum membership, numeric range, string match — write a code metric.

Metrics can encode constraints. Want the router to never output free text? Return 0.0 when pred.category not in VALID, even if the intent was right. The optimizer will learn to comply.

VALID = {"billing", "bug", "feature_request", "other"}

def scored(example, pred, trace=None):
    cat = pred.category.strip().lower()
    if cat not in VALID:
        return 0.0
    if cat == example.category:
        return 1.0
    # partial credit: correct that it's not "other"
    return 0.3 if (cat != "other" and example.category != "other") else 0.0

Common pitfalls

Optimizing on your test set. DSPy makes it easy to pass the same list to trainset and your evaluator. Split three ways — train, dev (for the optimizer's internal search), test (touched once, at the end). Report the test number.

Too few examples. MIPROv2 with 15 examples will overfit to those 15. If you can't get 100 labeled rows, use BootstrapFewShot and keep expectations modest.

Non-deterministic metric. If your metric calls an LM judge with temperature > 0, the optimizer chases noise. Pin the judge to temperature=0 and, ideally, a fixed seed.

Silent prompt bloat. Compiled programs can carry 8–10 few-shot demos, which inflates every inference call. Check the token cost of the optimized program (dspy.inspect_history(n=1) shows the rendered prompt) and cap max_bootstrapped_demos if latency matters.

Treating the compiled JSON as readable config. The saved artifact is a serialized program state, not a prompt you hand-edit. If you need to change behavior, change the signature or metric and recompile. Version the JSON in git and tie it to the model name it was compiled against.

Assuming it transfers across models. A program compiled for gpt-4o-mini is not optimal for claude-sonnet or a Llama checkpoint. Recompile per target model. The upside: recompiling is a command, not a week of manual work.

When DSPy is not worth it

One-off prompts or prototypes you'll throw away.
Tasks where you genuinely cannot write a metric and cannot label data. DSPy has nothing to optimize against.
Extremely latency-sensitive single calls where you can't afford any few-shot demos and the base instruction is already saturating quality.

For anything you'll maintain for more than a quarter, across more than one model version, with a definition of "correct" you can write down — the compile-and-measure loop pays for itself the first time a model update would otherwise have broken you silently.

Next steps

Pick one prompt you already maintain that has a checkable output.
Label 100 examples. This is the real cost; budget half a day.
Write a code metric. Make it return floats, not bools.
Run BootstrapFewShot, confirm the dev score moves, then try MIPROv2.
Commit the compiled JSON with the model id in the filename. Add a CI check that re-runs the evaluator on the test set and fails if the score regresses.

That last step is the real win: your prompt now has a test suite, and model upgrades become a number you watch instead of a fire you fight.

If your team is standing up eval loops and moving prompt engineering from vibes to measurement, that's most of what our engineering cohort covers hands-on. See the Joinloop cohort for the syllabus and dates.

Sources: DSPy documentation, Prompt Optimization with DSPy — Haystack, DSPy Framework — Programmatic Prompt Optimization (2026)