Editorial recipe — not benchmarked — reviewed September 3, 2026

Catch an AI workflow regression before release

For teams changing prompts, models, tools, or agent instructions against repeatable examples.

Stack

Promptfoo → Frozen test cases → Cloud or local model providers → GitHub or GitLab CI → Human reviewer

Run this recipe

Work through it here

Progress stays in this browser. The downloaded Markdown kit works in any notes app or repository.

0/5 complete

Freeze representative success, edge, and refusal cases with expected deterministic properties; keep restricted production data out of the fixture.
Pin the prompt, provider identifiers, model settings, tools, and budget in one Promptfoo configuration and prefer deterministic assertions before model grading.
Run the baseline and candidate on the same cases with cache behavior recorded; export results, latency, token use, cost, and failures as versioned artifacts.
Run the configuration in isolated CI with scoped provider credentials and block only on reviewed assertions that represent real product requirements.
Have a human inspect failures and a sample of passes, record accepted tradeoffs, and add real escaped failures to the regression set without storing customer secrets.

Is this recipe useful?

Procedure

Freeze representative success, edge, and refusal cases with expected deterministic properties; keep restricted production data out of the fixture.
Pin the prompt, provider identifiers, model settings, tools, and budget in one Promptfoo configuration and prefer deterministic assertions before model grading.
Run the baseline and candidate on the same cases with cache behavior recorded; export results, latency, token use, cost, and failures as versioned artifacts.
Run the configuration in isolated CI with scoped provider credentials and block only on reviewed assertions that represent real product requirements.
Have a human inspect failures and a sample of passes, record accepted tradeoffs, and add real escaped failures to the regression set without storing customer secrets.

Acceptance artifact

A pinned evaluation configuration, frozen cases, comparable run exports, CI result, and reviewed regression decision

Do not use it blindly

Promptfoo configurations can execute code and are not a sandbox; treat downloaded configurations and fixtures as untrusted and isolate them from production secrets.

Work through it here

Procedure

Acceptance artifact

Do not use it blindly

Evidence and setup