Why Are We Still Hand-Writing Prompts?

A common language-model application begins as one string.

That is not a mistake. A prompt is the fastest way to discover whether a model can help with a task. It lets us work directly with the behavior instead of building a framework before we know the shape of the problem.

But the prompt often survives too long.

In this chapter we will build a small editorial assistant around a handwritten prompt. The task is realistic: given one sentence from a chapter, improve the sentence without changing its meaning or voice. This is a simplified version of the kind of sentence-improvement work that appears in Writer, where candidates, evidence packets, provider identity, fallback state, and later review decisions are all tracked explicitly.

We will start with the simpler thing:

sentence + goal + context
        ↓
handwritten prompt
        ↓
model
        ↓
candidate rewrite

Then we will inspect what the prompt hides.

1. A reasonable first prompt

Here is a first version that many teams would ship during exploration:

from dataclasses import dataclass

@dataclass
class RewriteRequest:
    sentence: str
    goal: str
    local_context: str

def build_prompt(request: RewriteRequest) -> str:
    return f"""
You are an expert line editor.

Rewrite the sentence to satisfy the editorial goal.
Preserve meaning, entities, point of view, and the author's voice.
Avoid adding new facts.

Local context:
{request.local_context}

Editorial goal:
{request.goal}

Sentence:
{request.sentence}

Return JSON with:
- rewritten_text: the improved sentence
- rationale: a short explanation of what changed
- confidence: a number from 0.0 to 1.0
""".strip()

This is not a deliberately bad prompt. It says what role the model should play, supplies local context, names constraints, and asks for structured output. If we have a working model client, the call is ordinary Python:

def improve_sentence(model, request: RewriteRequest) -> dict:
    prompt = build_prompt(request)
    raw = model(prompt)
    return parse_json_object(raw)

The shape is easy to understand:

Python object
    ↓
prompt string
    ↓
LM response
    ↓
JSON parser

For an experiment, this is fine. For an engineered system, the prompt is already carrying too much weight.

2. The first failure is not the model

Suppose we submit:

We can patch the parser:

def normalize_response(raw: dict) -> dict:
    return {
        "rewritten_text": raw.get("rewritten_text") or raw.get("rewrite"),
        "rationale": raw.get("rationale") or raw.get("why"),
        "confidence": float(raw.get("confidence", 0.0)),
    }

That patch is useful, but notice what happened. An interface problem was discovered at runtime, then repaired outside the thing that supposedly defines the task.

The prompt says “Return JSON with…” but that is not the same as a program contract. It is a natural-language request embedded in a larger natural-language instruction. The model may comply. The parser may recover. Neither fact gives us a stable interface.

The prompt has combined several concerns:

If we edit one sentence in the prompt, what changed?

Maybe the task contract changed. Maybe only the wording changed. Maybe we accidentally changed how much explanation the model produces. Maybe the new prompt is better for a hosted model and worse for a local one. The string does not tell us.

Here are three plausible failures from the same handwritten prompt:

These are not the same failure. The first is a semantic constraint violation. The second is an interface violation. The third is a quality failure whose detection requires an explicit evaluation criterion. A longer prompt can mention all three, but length does not classify failures. Once failures have different causes, they need different engineering handles.

3. Prompt versions become opaque strings

In a small application, a prompt version might be a filename:

sentence_rewrite_prompt_v4.txt

After enough iterations, the version history often looks like this:

v1: basic rewrite
v2: stricter JSON
v3: add examples
v4: fix over-compression
v5: mention voice
v6: local model variant
v7: do not rename characters

This is better than no history, but it is still weak experimental evidence. Each version changes several things at once. The examples may change with the instructions. The output fields may change with the role description. The model-specific phrasing may change with the task definition.

The result is a system where improvement is hard to attribute:

prompt string changed
        ↓
behavior changed
        ↓
unknown cause

That is the point where prompt engineering starts to become behavior engineering. We no longer only want a better string. We want a way to ask:

• What behavior did we specify?
• Which implementation executed that behavior?
• What examples or demonstrations influenced it?
• What metric judged it?
• Which model and configuration produced the result?
• Can we compare this version to the previous one?

Those are software questions.

4. Evaluation cannot be an afterthought

Imagine we manually inspect ten rewrites and decide that version 6 is “better.” That may be true, but the evidence is fragile.

What did we evaluate?

prompt v6
model
ten hand-picked sentences
human memory of v5

Writer’s newer DSPy-related work is stricter about this. The contextual preference judge records a signature version, prompt fingerprint, model provider, model name, model version, temperature, input fingerprint, and raw output. Its editorial optimization path separates train, dev, and test rows and reports that optimization evidence is not the same as deployment approval.

CoCoder makes the same point in a repository-repair setting. Its EngineeringProgram optimization docs explicitly separate a runtime that executes a program from an optimizer that proposes a candidate. The optimizer does not get holdout cases. A candidate is not promoted merely because DSPy produced it.

We do not need all of that machinery in chapter 1. We do need the lesson:

unmeasured prompt change
        ↓
opinion

specified program version
        ↓
evaluation
        ↓
evidence

The first system can still be evaluated, but nothing in its shape makes evaluation a first-class part of the program lifecycle. The prompt is one artifact. The parser is another. The model configuration may live elsewhere. Examples may be pasted directly into the prompt, while the metric exists in a notebook cell. Unless we deliberately bind those pieces together, an evaluation result does not identify one coherent program version.

This is exactly the class of problem DSPy is designed to address.

5. DSPy enters after the problem is visible

DSPy is often introduced as a framework with Signatures, Predict, ChainOfThought, optimizers, and examples. That is accurate, but it can hide the deeper engineering move.

The official DSPy documentation describes the framework as a way to express tasks as structured signatures instead of prompts, producing maintainable, modular, optimizable programs. In current DSPy, a minimal extraction program can be written by declaring a dspy.Signature, then running it with dspy.Predict.

For our sentence task, the rough shape becomes:

import dspy

class ImproveSentence(dspy.Signature):
    """Improve one sentence while preserving meaning and voice."""

    sentence: str = dspy.InputField()
    editorial_goal: str = dspy.InputField()
    local_context: str = dspy.InputField()

    rewritten_text: str = dspy.OutputField()
    rationale: str = dspy.OutputField()
    confidence: float = dspy.OutputField(desc="Confidence from 0.0 to 1.0")

improve = dspy.Predict(ImproveSentence)

result = improve(
    sentence="She opened the door and then she laughed.",
    editorial_goal="Make the sentence sharper without changing the event.",
    local_context="The scene is tense but not comic.",
)

print(result.rewritten_text)

This example assumes DSPy is installed and an LM has been configured:

lm = dspy.LM("openai/gpt-5.4-nano")
dspy.configure(lm=lm)

For a local Ollama setup, the exact model string and provider configuration depend on the current DSPy/LiteLLM support in your environment. In Writer, older sentence generation used a DSPyOllamaSentenceRepairProvider; newer DSPy optimization work routes through Writer’s ModelRunner adapter so DSPy does not own provider routing.

The important change is not that the code is shorter. It is that the task has started to become inspectable:

Signature
  inputs:
    sentence
    editorial_goal
    local_context
  outputs:
    rewritten_text
    rationale
    confidence

Module:
  Predict

LM:
  configured dependency

The prompt has not disappeared. DSPy will still construct prompts internally. But the prompt string is no longer the primary abstraction we edit by hand.

What Usually Goes Wrong

Conclusion

The handwritten prompt was a useful discovery tool. It was not a sufficient engineering unit.

We gained a clearer diagnosis: the problem is not that prompt strings are bad, but that they mix the task contract, execution strategy, examples, output schema, provider assumptions, and informal evaluation into one opaque artifact.

The assumption we removed is that improving an LM application means editing a prompt until it feels better.

The remaining problem is now precise:

If the prompt string is no longer our primary abstraction, what replaces it?

That question takes us to the idea of a language-model program.