A/B Experiments | GitHub Agentic Workflows
Skip to main content A/B Experiments Use the experiments frontmatter section to compare workflow variants across repeated runs. Each experiment declares a name and a set of variants. On every run, the activation job picks one variant and exposes it to the prompt. Experiments work best when you test one workflow choice at a time, such as: prompt wording model selection whether to delegate to a sub-agent which subskill (inline skill) to invoke Declaring experiments Section titled “Declaring experiments” Add an experiments map to the workflow frontmatter.
Each key names an experiment. The value is either a simple array of variants (bare-array form) or a rich object with additional metadata fields. Bare-array form Section titled “Bare-array form” --- on : issues : types : [ opened ] engine : copilot experiments : style : [ concise , detailed ] --- Summarize this issue in a ** ${{ experiments.style }} ** way. Rich object form Section titled “Rich object form” Use the object form when you want built-in reporting and experiment metadata: --- on : schedule : daily on weekdays engine : copilot evals : - id : focused question : Is the change focused and actionable? experiments : prompt_style : variants : [ concise , detailed ] description : " Test whether a concise prompt reduces cost without quality loss " hypothesis : " H0: no change in aic.
H1: concise reduces AIC by >=15% " metric : eval:focused secondary_metrics : [ duration_ms , discussion_word_count ] guardrail_metrics : - name : success_rate threshold : " >=0.95 " - name : empty_output_rate direction : min threshold : 0.0 weight : [ 50 , 50 ] min_samples : 25 start_date : " 2026-05-05 " end_date : " 2026-07-25 " issue : 1234 --- Summarize the findings in a ** ${{ experiments.prompt_style }} ** way. When evals are configured, metric can reference an eval question ID using eval:<id> (for example eval:focused ) or evals.<id> .
When graders are configured, metric can reference a grader result using grader:<id> (for example grader:tool-success-rate ) or graders.<id>.value . Use adoption:accepted or adoption:zero_touch to measure live GitHub adopted outputs with gh aw experiments analyze <workflow> --adoption . These references also work in secondary_metrics and guardrail_metrics , without declaring a grader or eval. gh aw experiments analyze <workflow> resolves the referenced eval question and, when eval result data is available, shows YES/NO/UNKNOWN totals for that eval-backed metric.
It also joins per-run eval answers to persisted experiment assignments and analyzes the valid YES/NO observations by variant using the same statistical comparisons available for grader-backed metrics. For a grader-backed primary metric, it joins persisted experiment assignments to completed run artifacts and analyzes the valid grader values by variant. Using variants in the prompt Section titled “Using variants in the prompt” Reference a variant with ${{ experiments.<name> }} .
At runtime, gh-aw replaces the expression with the selected variant string, such as concise . Use the {{#if experiments.<name> }} block syntax for conditional prompt sections. A variant value of no is treated as falsy, which makes yes/no experiments easy to express: --- experiments : caveman : [ yes , no ] --- {{#if experiments.caveman }} Talk like a caveman in all your responses. Me test. You run. {{/if}} Address the issue described above.
Common experiment ideas Section titled “Common experiment ideas” Most experiments compare a single decision in the workflow. The examples below show common patterns. Try different prompt styles Section titled “Try different prompt styles” --- experiments : style : [ concise , detailed ] --- Summarize this issue in a ** ${{ experiments.style }} ** way. Try different models Section titled “Try different models” Model experiments are useful when you want to compare speed, cost, and output quality. gh-aw model aliases such as small and large are often a good place to start.
See Model Aliases . --- engine : id : copilot model : ${{ experiments.model }} experiments : model : [ small , large ] --- Review the issue and recommend the next action. Try using a sub-agent Section titled “Try using a sub-agent” This pattern compares a direct prompt with a delegated sub-agent flow. --- experiments : use_summarizer : [ yes , no ] --- {{#if experiments.use_summarizer }} Use the `file-summarizer` sub-agent to summarize `README.md` , then continue. {{/if}} Write a short project overview for maintainers. ## agent: `file-summarizer` --- model: small description: Summarizes a file in a few sentences --- Read the given file and return a concise summary.
See Inline Sub-Agents for the full syntax. With Copilot, you can also declare subagent_model: [small, large] in experiments and set the sub-agent’s model to ${{ experiments.subagent_model }} . The activation job replaces the reference with the selected variant before it writes the sub-agent file, and records the same variant in the run’s declared-model metadata. If no assignment is available, activation fails instead of writing the expression.
Audit patterns and model-routing request policy expand every declared variant through the workflow’s model aliases, while models.allowed and models.blocked still constrain admission. Router candidates remain unchanged. These patterns describe possible models; matching observed usage does not prove that the selected variant was honored. Use a direct ${{ experiments.subagent_model }} reference for sub-agent models. Compound expressions such as ${{ experiments.subagent_model || 'small' }} or format(...) are rejected because the sub-agent runtime does not evaluate them.
Put the complete model or alias in each experiment variant instead. Pi sub-agents require literal models or aliases, not expressions. Try different subskills Section titled “Try different subskills” This pattern compares two reusable instruction blocks, sometimes called subskills, without changing the main workflow prompt. --- experiments : triage_skill : [ triage-fast , triage-deep ] --- Use the `${{ experiments.triage_skill }}` skill to classify this issue. ## skill: `triage-fast` --- description: Fast issue triage --- Classify the issue and suggest the smallest next step. ## skill: `triage-deep` --- description: Detailed issue triage --- Classify the issue, identify missing context, and recommend a fuller follow-up plan.
Statistical balancing Section titled “Statistical balancing” The activation job tracks how often each variant has been selected. The counter is stored using the storage setting in the experiments: block. By default, gh-aw chooses the least-used variant on each run. If multiple variants are tied, including on the first run, one of them is chosen at random. Over time, this keeps usage roughly balanced across variants.
When you provide a weight array, gh-aw uses weighted random selection instead of least-used selection. For example, [70, 30] gives the first variant a 70% selection probability. If start_date or end_date is set and the current date falls outside that range, gh-aw returns the control variant (the first entry) without incrementing any counter. Storage Configuration Section titled “Storage Configuration” The storage key inside the experiments: map controls where experiment state is persisted: experiments : storage : repo # or: cache (default: repo) prompt_style : [ concise , detailed ] Value Behavior repo ( default ) Commits state to a git branch named experiments/{sanitizedWorkflowID} (workflow ID lowercased with hyphens removed, e.g. my-workflow → experiments/myworkflow ).
Durable — survives cache evictions. Requires contents: write permission (added automatically by the compiler). cache Uses GitHub Actions cache (legacy). State may be evicted after 7 days of inactivity. When storage: repo , the compiler adds a push_experiments_state job after the activation job and commits the updated state.json to the
The rest of this page is on the live site.