From the free course Start building with Claude on Lykke

Prompt Engineering and Evaluation

Key concepts: Prompt Engineering · Golden Datasets · Evaluation Metrics · Iterative Refinement

Best practices for crafting effective prompts and establishing a rigorous evaluation framework for model outputs.

Prompt Engineering and Evaluation

The transition from a "toy" Large Language Model (LLM) implementation to a production-grade AI system is rarely a matter of model scaling alone. Instead, it is a function of Prompt Engineering and Evaluation (Evals)—the dual disciplines of optimizing the model’s input stimuli and rigorously measuring the resulting output distributions. In the early stages of development, engineers often rely on "vibe-based" development, where a prompt is tweaked until it "looks right." However, for systems requiring high reliability, safety, and performance, we must treat prompts as code and evaluations as unit tests.

AI_IMAGEI_IMAGE## The Taxonomy of Prompt Engineering

Prompt Engineering is the systematic process of refining the input provided to an LLM to steer its latent representations toward a specific, desired output. It is not merely "talking to the AI"; it is an exercise in constraint satisfaction and probability shaping.

The Anatomy of a High-Performing Prompt

A production prompt is typically composed of several distinct functional components. By modularizing these components, developers can isolate variables during the evaluation phase.

Component Purpose Example
Role/Persona Sets the stylistic and technical boundaries of the response. "You are a senior site reliability engineer..."
Instruction The core task the model must perform. "Analyze the following stack trace for memory leaks."
Context External data needed to complete the task (RAG). "Here are the relevant documentation snippets: [DOCS]"
Constraints Hard limits on output format, length, or content. "Return only valid JSON. Do not include explanations."
Few-Shot Examples Input-output pairs that demonstrate the desired pattern. "Input: A, Output: B; Input: C, Output: D"

Advanced Prompting Techniques

Beyond simple instructions, several architectural patterns have emerged to handle complex reasoning:

  1. Chain of Thought (CoT): Forcing the model to generate intermediate reasoning steps. This leverages the model's autoregressive nature, allowing it to "think" by generating tokens that serve as working memory.
  2. Few-Shot Prompting: Providing $n$ examples of a task. This performs "in-context learning," where the model aligns its output distribution with the provided examples without weight updates.
  3. ReAct (Reason + Act): A framework where the model generates a reasoning trace followed by an action (e.g., a tool call), then observes the result and repeats.

Definition: The In-Context Learning Hypothesis In-context learning is often theorized as a form of "implicit fine-tuning" where the prompt's examples allow the model to locate a specific task-relevant manifold within its pre-trained latent space, effectively narrowing the search space for the next token.

# Low-level implementation: A structured Prompt Template System
# This ensures consistency and allows for programmatic iteration over prompt variables.

import json
from typing import List, Dict

class PromptTemplate:
    def __init__(self, template_str: str):
        self.template = template_str

    def format(self, **kwargs) -> str:
        return self.template.format(**kwargs)

def build_chain_of_thought_prompt(task: str, context: str, examples: List[Dict[str, str]]) -> str:
    """
    Constructs a structured CoT prompt with few-shot examples.
    """
    system_role = "You are a logical reasoning assistant. Solve the task step-by-step."
    
    example_block = ""
    for ex in examples:
        example_block += f"Q: {ex['input']}\nA: Let's think step by step. {ex['thought']} Therefore, the answer is {ex['output']}.\n\n"
    
    prompt = f"{system_role}\n\n{example_block}Q: {task}\nContext: {context}\nA: Let's think step by step."
    return prompt

# Example usage
examples = [
    {"input": "Is 17 a prime?", "thought": "17 is only divisible by 1 and 17.", "output": "Yes"}
]
final_prompt = build_chain_of_thought_prompt("Is 91 a prime?", "Standard arithmetic rules apply.", examples)
print(final_prompt)

Golden Datasets: The Ground Truth

An evaluation is only as good as the data it uses. A Golden Dataset (or "Ground Truth" set) is a curated collection of inputs and their corresponding "ideal" outputs. This dataset serves as the benchmark against which all prompt iterations are measured.

Construction and Curation

Creating a Golden Dataset is often the most labor-intensive part of the LLM lifecycle. It requires:

  • Diversity: Covering edge cases, "happy paths," and adversarial inputs.
  • Accuracy: Human-verified or expert-generated outputs.
  • Size: Usually 50–500 samples for prompt engineering, though more are needed for fine-tuning.

Synthetic Data Generation

When human labeling is too slow, we use "LLM-as-a-Generator." A stronger model (e.g., Claude 3.5 Sonnet) generates synthetic inputs and outputs, which are then filtered by a human expert. This is known as Distillation or Self-Instruct.

Strategy Pros Cons
Manual Curation Highest quality, catches subtle nuances. Extremely slow, expensive, unscalable.
Production Logs Real-world distribution, captures user intent. Requires PII scrubbing, may contain "bad" examples.
Synthetic (LLM) Rapid, covers vast edge cases. Risk of "model collapse" or echo-chamber biases.

AI_DEMOI_DEMO## Evaluation Metrics: Quantifying Success

Once a Golden Dataset is established, we need a way to score the model's performance. Evaluation metrics range from simple string matching to complex model-based judgments.

Deterministic vs. Probabilistic Metrics

For tasks with a single correct answer (e.g., data extraction), deterministic metrics work best. For creative or open-ended tasks, we use probabilistic or semantic metrics.

The Mathematical Intuition of BERTScore Unlike BLEU, which relies on $n$-gram overlap, BERTScore calculates the cosine similarity between the contextual embeddings of the candidate and reference sentences: $$R_{BERT} = \frac{1}{|x|} \sum_{x_i \in x} \max_{\hat{x}_j \in \hat{x}} \mathbf{v}_i^\top \mathbf{\hat{v}}_j$$ where $\mathbf{v}_i$ and $\mathbf{\hat{v}}_j$ are the pre-trained embeddings of the tokens.

LLM-as-a-Judge (G-Eval)

The current state-of-the-art for evaluating complex outputs (like summaries or code) is using an LLM to grade another LLM. This is often more correlated with human judgment than traditional metrics like ROUGE.

\text{Score} = \text{LLM}(\text{Prompt}, \text{Response}, \text{Rubric}) \rightarrow [1-5]

Metric Comparison Table

Metric Type Best For Limitation
Exact Match (EM) Deterministic Classification, IDs Too rigid for natural language.
ROUGE-L Statistical Summarization Penalizes synonyms; rewards length.
BERTScore Semantic Translation, Paraphrasing Computationally expensive; opaque.
Pass@k Functional Code Generation Requires an execution environment.
LLM-Judge Model-based Nuance, Tone, Logic Potential for "self-preference" bias.
# Third code block: CLI invocation for an evaluation pipeline
# Using a tool like 'promptfoo' to run a matrix evaluation

# 1. Define the test suite in a YAML file (promptfooconfig.yaml)
# 2. Run the evaluation across multiple models/prompts
promptfoo eval \
  --prompts "prompts/v1.txt" "prompts/v2_chain_of_thought.txt" \
  --providers "anthropic:messages:claude-3-5-sonnet-20240620" "openai:gpt-4o" \
  --tests "data/golden_dataset.csv" \
  --output "results/eval_report.html" \
  --cache

# This command generates a side-by-side comparison of how different 
# prompt/model combinations performed against the golden dataset.

Iterative Refinement: The Eval Loop

Prompt engineering is an iterative cycle, not a one-off task. The "Eval Loop" mirrors the traditional software development lifecycle (SDLC) but centers on the stochastic nature of LLMs.

The Refinement Pipeline

  1. Baseline: Run the current prompt against the Golden Dataset.
  2. Error Analysis: Identify where the model failed. Is it hallucinating? Is the formatting wrong? Is it missing context?
  3. Hypothesis: Formulate a change (e.g., "Adding a negative constraint will stop the model from apologizing").
  4. Experiment: Update the prompt and re-run the eval.
  5. Regression Check: Ensure the change didn't break previously passing cases.

Common Pitfalls in Iteration

  • Overfitting to the Eval Set: Tweaking a prompt so specifically for 50 examples that it fails on the 51st.
  • Sensitivity to Permutation: LLMs are sensitive to the order of few-shot examples. Moving the "best" example to the end of the list often improves performance (Recency Bias).
  • The "Verbosity Bias": LLM judges often give higher scores to longer responses, even if they are less accurate.
-- Fourth code block: Analyzing evaluation results in a database
-- This query identifies specific categories where the model's 
-- 'faithfulness' score dropped after a prompt update.

SELECT 
    test_category,
    AVG(score_v1) as avg_score_old,
    AVG(score_v2) as avg_score_new,
    (AVG(score_v2) - AVG(score_v1)) as delta
FROM 
    eval_results
WHERE 
    metric_name = 'faithfulness'
GROUP BY 
    test_category
HAVING 
    delta < -0.1
ORDER BY 
    delta ASC;

-- This allows engineers to pinpoint regressions in specific logic domains.

Safety and Guardrails

In production, prompt engineering also involves Defensive Prompting. This ensures the model does not leak sensitive data, generate harmful content, or succumb to "Prompt Injection" (where a user tries to override the system instructions).

Guardrail Architectures

  • Input Sanitization: Using a smaller, faster model to check if the user's query contains malicious instructions.
  • Output Verification: Using regex or Pydantic validators to ensure the output matches the required schema before it reaches the user.
  • Constitutional AI: Providing the model with a set of "principles" (e.g., "Do not be harmful") that it must follow during its reasoning process.

Summary of Best Practices

To build a robust system, follow these heuristics:

  1. Version Control Everything: Prompts should be stored in Git, not hardcoded in the application logic.
  2. Automate the Benchmarks: Integrate your Golden Dataset evals into your CI/CD pipeline.
  3. Use the Right Model for the Job: Use a "Teacher" model (Claude 3 Opus) for evaluation and a "Student" model (Claude 3 Haiku) for high-speed production inference.
  4. Mind the Context Window: Large prompts are expensive and can lead to "Lost in the Middle" phenomena, where the model ignores information in the center of the prompt.
Prompt Engineering and Evaluation - Start building with Claude - image 1
Prompt Engineering and Evaluation - Start building with Claude - image 1

Source Materials

Study Start building with Claude on Lykke

This course page is free to open: signed out, you can read its first two sections, and signing in, on any plan, opens the rest. The free Lykke Chrome extension syncs your own Canvas courses, so ChatGPT, Claude and Muse can answer from your classes and deadlines. It only reads Canvas.

Add the free extensionSign in to open every section

All sections of Start building with Claude · Browse more course pages