agentic-eval

Agentic Evaluation

Use this skill when you are designing or implementing an evaluation loop that lets an agent assess and improve its own outputs through iteration — not when you are running a pre-existing test suite or doing a one-off review with no refinement cycle.

The core pattern is: Generate → Evaluate → Critique → Refine → Output, looping until a convergence condition is met or a max-iteration budget is exhausted.

Use this skill when

• Implementing a self-critique or reflection loop that feeds output quality back into generation.
• Building an evaluator-optimizer pipeline that separates generation from evaluation responsibilities.
• Designing LLM-as-judge scoring to compare or rank multiple candidate outputs.
• Adding rubric-based scoring with weighted dimensions to iterative generation.
• Setting iteration limits, convergence checks, or structured evaluation output contracts.
• The task requires measurable improvement across runs, not just a single-shot best effort.

Do not use this skill when

• You are running an existing test suite to verify code — use verification-before-completion.
• You are diagnosing a specific failure or bug, not evaluating output quality — use systematic-debugging.
• The goal is writing test coverage (unit tests, integration tests) — use test-driven-development.
• You are reviewing a completed artifact once without a refinement loop (a single code review, an editorial pass, a PR check).

Routing boundary

Situation	Use this skill?	Route instead
Designing a reflection loop with a score threshold and max iterations	Yes	—
Implementing LLM-as-judge comparison of two candidate outputs	Yes	—
Running npm test to confirm a fix works	No	verification-before-completion
Tracing why a specific assertion fails	No	systematic-debugging
Writing Jest or pytest test coverage for a module	No	test-driven-development
Reviewing a PR diff once, no iteration	No	implementation-review

Inputs to gather

Required before starting

• The skill or agent behavior to evaluate.
• The evaluation objective and target metric for the application's outputs.

Helpful if present

• Existing evaluation fixtures or application quality criteria to extend.

First move

1. Identify the application behavior to evaluate and the desired outcome.
2. Define the evaluation criteria and target metric.
3. Open the relevant reference file based on the evaluation type.

Navigation

The three evaluation strategy patterns (outcome-based, LLM-as-judge, rubric-based) and full Python examples are in references/patterns.md.

The implementation checklist — criteria, threshold, loop wiring, convergence, logging — is in assets/eval-checklist.md.

For a new implementation, start with the checklist to confirm your setup is complete, then use the patterns reference to choose and adapt an evaluation strategy.

Outputs

• Evaluation loop design with defined criteria, convergence check, and max iteration budget.
• Structured evaluation scores per iteration with input, output, and critique logged.
• Convergence or budget-exhaustion result confirming the loop terminated cleanly.

Workflow

1. Name the application behavior under evaluation and the success criteria.
2. Choose a strategy from references/patterns.md: outcome-based, LLM-as-judge, or rubric-based.
3. Complete the setup checklist in assets/eval-checklist.md (criteria, threshold, max iterations, logging).
4. Isolate generate → evaluate → optimize steps; keep the evaluator replaceable.
5. Run the loop with a hard max_iterations bound and a convergence check.
6. Log full trajectories (input, output, score, critique) for every iteration.
7. Stop on threshold met, no improvement, or budget exhaustion; report structured scores.
8. Feed failures back into evaluator criteria, prompts, or optimization steps, then re-run.

Guardrails

• Always set a max_iterations bound (3–5 is a safe default) before wiring up a refinement loop. Unbounded loops stall agents.
• Require structured output (JSON) from the evaluation step so the optimize step has a reliable signal to act on. Free-text critique is fragile.
• Add a convergence check: if the score does not improve between iterations, stop early. Oscillating loops that never converge waste budget.
• Log the full iteration trajectory. Evaluation loops are hard to debug post-hoc without a history of inputs, outputs, scores, and critiques.
• Define evaluation criteria before generating any output. Criteria added mid-loop drift and make scores incomparable across iterations.
• Keep the evaluate step isolated from the generate step. Blending them makes it hard to replace the evaluator or diagnose score instability.
• Handle evaluation parse failures gracefully — if the LLM judge returns malformed JSON, fall back to a safe default (treat as failing) rather than crashing the loop.

Validation

After implementing an evaluation loop, confirm:

• max_iterations is set and respected by the loop
• Evaluate step returns structured output (JSON or equivalent)
• Convergence check exits early when score does not improve
• All iterations are logged with input, output, score, and critique
• Parse-failure fallback is present on the evaluate step
• Criteria are defined before any generation begins

Examples

• "Add a self-critique loop to my report-generation agent that retries up to three times if the rubric score is below 0.8."
• "Implement an evaluator-optimizer where a separate LLM judge scores code clarity and the generator rewrites until it passes."
• "Build a rubric-based evaluator with accuracy, completeness, and style dimensions that returns a weighted score as JSON."

Reference files

• references/patterns.md — The three evaluation strategy patterns (outcome-based, LLM-as-judge, rubric-based) with annotated Python examples and a best-practices table.
• assets/eval-checklist.md — Implementation checklist: setup, loop wiring, convergence, logging, and safety items to confirm before shipping.