From the free course Start building with Claude on Lykke
Prompt Engineering and Evaluation
Key concepts: Prompt Engineering · Golden Datasets · Evaluation Metrics · Iterative Refinement
Best practices for crafting effective prompts and establishing a rigorous evaluation framework for model outputs.
Prompt Engineering and Evaluation
The transition from a "toy" Large Language Model (LLM) implementation to a production-grade AI system is rarely a matter of model scaling alone. Instead, it is a function of Prompt Engineering and Evaluation (Evals)—the dual disciplines of optimizing the model’s input stimuli and rigorously measuring the resulting output distributions. In the early stages of development, engineers often rely on "vibe-based" development, where a prompt is tweaked until it "looks right." However, for systems requiring high reliability, safety, and performance, we must treat prompts as code and evaluations as unit tests.
AI_IMAGEI_IMAGE## The Taxonomy of Prompt Engineering
Prompt Engineering is the systematic process of refining the input provided to an LLM to steer its latent representations toward a specific, desired output. It is not merely "talking to the AI"; it is an exercise in constraint satisfaction and probability shaping.
The Anatomy of a High-Performing Prompt
A production prompt is typically composed of several distinct functional components. By modularizing these components, developers can isolate variables during the evaluation phase.
| Component | Purpose | Example |
|---|---|---|
| Role/Persona | Sets the stylistic and technical boundaries of the response. | "You are a senior site reliability engineer..." |
| Instruction | The core task the model must perform. | "Analyze the following stack trace for memory leaks." |
| Context | External data needed to complete the task (RAG). | "Here are the relevant documentation snippets: [DOCS]" |
| Constraints | Hard limits on output format, length, or content. | "Return only valid JSON. Do not include explanations." |
| Few-Shot Examples | Input-output pairs that demonstrate the desired pattern. | "Input: A, Output: B; Input: C, Output: D" |
Advanced Prompting Techniques
Beyond simple instructions, several architectural patterns have emerged to handle complex reasoning:
- Chain of Thought (CoT): Forcing the model to generate intermediate reasoning steps. This leverages the model's autoregressive nature, allowing it to "think" by generating tokens that serve as working memory.
- Few-Shot Prompting: Providing $n$ examples of a task. This performs "in-context learning," where the model aligns its output distribution with the provided examples without weight updates.
- ReAct (Reason + Act): A framework where the model generates a reasoning trace followed by an action (e.g., a tool call), then observes the result and repeats.
Definition: The In-Context Learning Hypothesis In-context learning is often theorized as a form of "implicit fine-tuning" where the prompt's examples allow the model to locate a specific task-relevant manifold within its pre-trained latent space, effectively narrowing the search space for the next token.
# Low-level implementation: A structured Prompt Template System
# This ensures consistency and allows for programmatic iteration over prompt variables.
import json
from typing import List, Dict
class PromptTemplate:
def __init__(self, template_str: str):
self.template = template_str
def format(self, **kwargs) -> str:
return self.template.format(**kwargs)
def build_chain_of_thought_prompt(task: str, context: str, examples: List[Dict[str, str]]) -> str:
"""
Constructs a structured CoT prompt with few-shot examples.
"""
system_role = "You are a logical reasoning assistant. Solve the task step-by-step."
example_block = ""
for ex in examples:
example_block += f"Q: {ex['input']}\nA: Let's think step by step. {ex['thought']} Therefore, the answer is {ex['output']}.\n\n"
prompt = f"{system_role}\n\n{example_block}Q: {task}\nContext: {context}\nA: Let's think step by step."
return prompt
# Example usage
examples = [
{"input": "Is 17 a prime?", "thought": "17 is only divisible by 1 and 17.", "output": "Yes"}
]
final_prompt = build_chain_of_thought_prompt("Is 91 a prime?", "Standard arithmetic rules apply.", examples)
print(final_prompt)
Golden Datasets: The Ground Truth
An evaluation is only as good as the data it uses. A Golden Dataset (or "Ground Truth" set) is a curated collection of inputs and their corresponding "ideal" outputs. This dataset serves as the benchmark against which all prompt iterations are measured.
Construction and Curation
Creating a Golden Dataset is often the most labor-intensive part of the LLM lifecycle. It requires:
- Diversity: Covering edge cases, "happy paths," and adversarial inputs.
- Accuracy: Human-verified or expert-generated outputs.
- Size: Usually 50–500 samples for prompt engineering, though more are needed for fine-tuning.
Synthetic Data Generation
When human labeling is too slow, we use "LLM-as-a-Generator." A stronger model (e.g., Claude 3.5 Sonnet) generates synthetic inputs and outputs, which are then filtered by a human expert. This is known as Distillation or Self-Instruct.
| Strategy | Pros | Cons |
|---|---|---|
| Manual Curation | Highest quality, catches subtle nuances. | Extremely slow, expensive, unscalable. |
| Production Logs | Real-world distribution, captures user intent. | Requires PII scrubbing, may contain "bad" examples. |
| Synthetic (LLM) | Rapid, covers vast edge cases. | Risk of "model collapse" or echo-chamber biases. |
AI_DEMOI_DEMO## Evaluation Metrics: Quantifying Success
Once a Golden Dataset is established, we need a way to score the model's performance. Evaluation metrics range from simple string matching to complex model-based judgments.
Deterministic vs. Probabilistic Metrics
For tasks with a single correct answer (e.g., data extraction), deterministic metrics work best. For creative or open-ended tasks, we use probabilistic or semantic metrics.
The Mathematical Intuition of BERTScore Unlike BLEU, which relies on $n$-gram overlap, BERTScore calculates the cosine similarity between the contextual embeddings of the candidate and reference sentences: $$R_{BERT} = \frac{1}{|x|} \sum_{x_i \in x} \max_{\hat{x}_j \in \hat{x}} \mathbf{v}_i^\top \mathbf{\hat{v}}_j$$ where $\mathbf{v}_i$ and $\mathbf{\hat{v}}_j$ are the pre-trained embeddings of the tokens.
LLM-as-a-Judge (G-Eval)
The current state-of-the-art for evaluating complex outputs (like summaries or code) is using an LLM to grade another LLM. This is often more correlated with human judgment than traditional metrics like ROUGE.
\text{Score} = \text{LLM}(\text{Prompt}, \text{Response}, \text{Rubric}) \rightarrow [1-5]
Metric Comparison Table
| Metric | Type | Best For | Limitation |
|---|---|---|---|
| Exact Match (EM) | Deterministic | Classification, IDs | Too rigid for natural language. |
| ROUGE-L | Statistical | Summarization | Penalizes synonyms; rewards length. |
| BERTScore | Semantic | Translation, Paraphrasing | Computationally expensive; opaque. |
| Pass@k | Functional | Code Generation | Requires an execution environment. |
| LLM-Judge | Model-based | Nuance, Tone, Logic | Potential for "self-preference" bias. |
# Third code block: CLI invocation for an evaluation pipeline
# Using a tool like 'promptfoo' to run a matrix evaluation
# 1. Define the test suite in a YAML file (promptfooconfig.yaml)
# 2. Run the evaluation across multiple models/prompts
promptfoo eval \
--prompts "prompts/v1.txt" "prompts/v2_chain_of_thought.txt" \
--providers "anthropic:messages:claude-3-5-sonnet-20240620" "openai:gpt-4o" \
--tests "data/golden_dataset.csv" \
--output "results/eval_report.html" \
--cache
# This command generates a side-by-side comparison of how different
# prompt/model combinations performed against the golden dataset.
Iterative Refinement: The Eval Loop
Prompt engineering is an iterative cycle, not a one-off task. The "Eval Loop" mirrors the traditional software development lifecycle (SDLC) but centers on the stochastic nature of LLMs.
The Refinement Pipeline
- Baseline: Run the current prompt against the Golden Dataset.
- Error Analysis: Identify where the model failed. Is it hallucinating? Is the formatting wrong? Is it missing context?
- Hypothesis: Formulate a change (e.g., "Adding a negative constraint will stop the model from apologizing").
- Experiment: Update the prompt and re-run the eval.
- Regression Check: Ensure the change didn't break previously passing cases.
Common Pitfalls in Iteration
- Overfitting to the Eval Set: Tweaking a prompt so specifically for 50 examples that it fails on the 51st.
- Sensitivity to Permutation: LLMs are sensitive to the order of few-shot examples. Moving the "best" example to the end of the list often improves performance (Recency Bias).
- The "Verbosity Bias": LLM judges often give higher scores to longer responses, even if they are less accurate.
-- Fourth code block: Analyzing evaluation results in a database
-- This query identifies specific categories where the model's
-- 'faithfulness' score dropped after a prompt update.
SELECT
test_category,
AVG(score_v1) as avg_score_old,
AVG(score_v2) as avg_score_new,
(AVG(score_v2) - AVG(score_v1)) as delta
FROM
eval_results
WHERE
metric_name = 'faithfulness'
GROUP BY
test_category
HAVING
delta < -0.1
ORDER BY
delta ASC;
-- This allows engineers to pinpoint regressions in specific logic domains.
Safety and Guardrails
In production, prompt engineering also involves Defensive Prompting. This ensures the model does not leak sensitive data, generate harmful content, or succumb to "Prompt Injection" (where a user tries to override the system instructions).
Guardrail Architectures
- Input Sanitization: Using a smaller, faster model to check if the user's query contains malicious instructions.
- Output Verification: Using regex or Pydantic validators to ensure the output matches the required schema before it reaches the user.
- Constitutional AI: Providing the model with a set of "principles" (e.g., "Do not be harmful") that it must follow during its reasoning process.
Summary of Best Practices
To build a robust system, follow these heuristics:
- Version Control Everything: Prompts should be stored in Git, not hardcoded in the application logic.
- Automate the Benchmarks: Integrate your Golden Dataset evals into your CI/CD pipeline.
- Use the Right Model for the Job: Use a "Teacher" model (Claude 3 Opus) for evaluation and a "Student" model (Claude 3 Haiku) for high-speed production inference.
- Mind the Context Window: Large prompts are expensive and can lead to "Lost in the Middle" phenomena, where the model ignores information in the center of the prompt.
Source Materials
Study Start building with Claude on Lykke
This course page is free to open: signed out, you can read its first two sections, and signing in, on any plan, opens the rest. The free Lykke Chrome extension syncs your own Canvas courses, so ChatGPT, Claude and Muse can answer from your classes and deadlines. It only reads Canvas.
Add the free extensionSign in to open every sectionAll sections of Start building with Claude · Browse more course pages