pipecat eval run tests scenarios against an agent you started yourself. A suite goes one step further: you list agents and scenarios in a manifest, and pipecat eval suite spawns each agent with its eval transport on its own port, runs its scenarios, tears it down, and aggregates the results, several runs at a time.
Suites are the right tool when you have more than one agent, more than a handful of scenarios, or want a single command for CI. They are also how a simulation runs more than once: each run gets a fresh agent. Pipecat’s own release evals are a manifest with 100+ example agents plus this command.
The manifest
manifest.yaml
bots_dir, scenarios_dir, runs_dir, the bot: entries) resolve relative to the manifest file, so a manifest is portable: check it into your repo and run it from anywhere. A scenario name can include a subfolder (simulated/book_table), and a name ending in .yaml is a path relative to the manifest.
A scenarios: entry names a scenario file, and the file contributes one run per scenario it holds, named <file>/<scenario> after the file’s name: and the scenario’s own. Scripted scenarios and simulations are listed the same way, and each scenario says which it is: turns: makes it scripted, persona: makes it a simulation. Pipecat’s release evals keep the two in scenarios/scripted/ and scenarios/simulated/ folders, which is a convention worth copying.
Scenarios are reusable across agents. One greeting scenario can cover every agent in the suite.
Entry options
An entry can carry three optional keys besidebot: and scenarios::
runner_body:supplies session data the agent would normally receive in a/startrequest body, passed to it as--runner-body. It holds eitherpath:, a YAML or JSON file resolved relative to the manifest, ordata:, the body itself as a mapping, which the suite writes to a file among the run’s logs. An agent given a file runs with the file’s directory as its working directory, so a relative path inside the body (a vision agent’s image, say) resolves next to the file. A body that holds such paths belongs in a file for that reason.name:is the entry’s label: what the dashboard, the-pfilter,results.jsonl, and the artifact file names use. It defaults to thebot:path, so it is only needed when several entries share a bot and differ in theirrunner_body:, as when sweeping a model. Two entries may not run the same scenario under one label.concurrency:is how many of the suite’s slots the entry may hold at once, one by default. See Concurrency.
A bare
runner_body: <file> is deprecated since 1.11.0 and will be removed in
2.0.0. Write runner_body: { path: <file> } instead.Running a suite
0 only if every run passes.
Useful flags:
-s takes a run’s full <file>/<scenario> name or either half of it, so a file’s name selects every scenario it holds, and a scenario’s name selects it in whichever file it appears.
Everything except the suite: list can live in the manifest or be passed on the command line (the command line wins), so a manifest can be as minimal as a suite: list.
Concurrency
concurrency: is how many runs execute at once. Each entry’s runs go one after another on a single slot, so its scenarios finish as a block and a slow or rate-limited provider holds no more than one slot, whatever the suite’s figure. An entry that can take more sets its own concurrency:, the number of slots it may hold at once, and runs that many of its scenarios side by side. Entries take free slots in manifest order, and the suite’s figure, or --concurrency, bounds the whole.
Run output
Each invocation writes to<runs_dir>/<name>/ (a timestamp when -n is omitted):
name: when it has one, then the scenario’s <file>/<scenario> name with its slash written as a double underscore, so bots/turn-agent.py under the label openai/gpt-4o-mini running turn_completion/short_answer writes bots_turn-agent.py__openai_gpt-4o-mini__turn_completion__short_answer.log.
results.jsonl carries one line per run, and every line names its bot, its entry’s name, its scenario, its attempt, and its kind, script or simulation. A scripted record carries the outcome, duration, failures (each with a kind), per-turn results, and paths to its artifacts. Each turn lists its expectations, one per expectation the turn resolved, with whether it passed and what it matched: the marker of an llm_marker, a function call’s signature, a reply’s text. What a passed run matched is only recorded here. A simulation record carries the outcome (passed, succeeded, error), how the run ended (ended_by), the persona’s turn count, every metric’s score, value, failure_kind, and per-turn verdicts (yes, no, or none when the judge gave none), the judge’s reason, the persona’s own end_call claim, and the whole conversation as messages. Lines are appended as each run finishes, so an interrupted sweep keeps everything already done. Runs that didn’t pass also carry events_seen.
Each run executes in its own process, so a harness that loads local audio models does so on its own, and a crash in one run doesn’t stop the others.
results.jsonl is written by pipecat eval suite. pipecat eval run doesn’t
produce one..eval.log decision trace: it’s a timestamped record of every event the harness saw, what it matched, what the judge said, and why an assertion failed. The agent’s own log sits next to it.
Testing one agent with many scenarios
If you just want to run a batch of scenarios against an agent you already have running, you don’t need a manifest.pipecat eval run accepts multiple scenario files and shares the suite’s dashboard and tally:
.yaml and .yml files in filename order, non-recursively, and files and directories can be mixed in one invocation. A directory holding no scenarios is an error rather than an empty run.
By default the agent is left running afterward so it can serve more evals; pass --stop-bot to shut it down when the batch finishes.
Running a simulation more than once
A persona doesn’t say the same thing twice, so one run of a simulation proves little. A simulation’s ownruns: field tells the suite how many times to run it, and every run must pass. The suite reports a pass rate per simulation with a ✓ or ✗ beside it, and exits 1 when any run failed:
Repeating a run
A behavior with a race in it passes sometimes.repeat: in the manifest, or --repeat / -r on the command line, runs every (bot, scenario) pair N times and reports a pass rate per pair instead of a single verdict:
__001) only when repeat is above 1, so a single pass keeps the filenames it always had.
A repeat set on the command line or in the manifest applies to every run, simulations included, even when it is 1. It overrides a simulation’s runs: and turns the requirement into a measurement: rates are reported and the exit code stays 0. Leave repeat unset to let each simulation run its own runs: and gate on them.
Suites in CI
The exit code makes suites CI-ready with no extra glue:runs: 3 and expect them to take longer. -k script runs the scripted half alone when you want a fast gate on every push and the simulations on a schedule. Audio-mode scenarios work in CI too, but need the harness’s TTS and STT services available (local models by default, which also need more CPU).