Text of github.github.com › gh-aw › specs › experiments-specification as Styx crawled it on Nov 19, 2026. The page may have changed since. Only its words are kept, without pictures or layout, and only the first part of a long page.

Open the live page

A/B Experiments Specification | GitHub Agentic Workflows

Skip to main content A/B Experiments Specification Version : 1.0.0 Status : Draft Latest Version : experiments-specification Editors : gh-aw maintainers Abstract Section titled “Abstract” This specification defines the A/B experiment system for GitHub Agentic Workflows. It covers the experiments: frontmatter schema, variant selection algorithms, state persistence backends, expression and template integration, activation job structure, audit CLI integration, and statistical analysis requirements.

Conforming implementations provide operators with a zero-infrastructure mechanism to conduct controlled experiments on agentic workflow behavior using only workflow frontmatter declarations, without any external service dependency. This document consolidates and supersedes the normative sections of ADR-29534, ADR-29618, ADR-29628, ADR-29985, and ADR-29996. It also incorporates corrective requirements identified during an expert review of the implementation in May 2026.

Status of This Document Section titled “Status of This Document” This is a Draft specification. It may be updated, replaced, or made obsolete at any time. A future revision will promote this document to Candidate Recommendation once the reference implementation (gh-aw v1.x) satisfies all conformance requirements below. Promotion from Draft to Candidate Recommendation requires all of the following: Reference implementation completeness : 100% of normative requirements in §§4–12 are implemented in gh-aw and mapped to concrete implementation files ( tracking issue : #31983 ).

Compliance coverage : At least 95% of normative requirements have automated tests, and all MUST/MUST NOT requirements have at least one passing automated test ( tracking issue : #31983 ). CI stability window : The experiments-related test suite passes on the default branch for 30 consecutive days with no unresolved regression in variant selection, persistence, or reporting behavior ( tracking issue : #31983 ). Interoperability evidence : At least two production workflows using experiments: run for a minimum of 500 total assignments each with valid assignment artifacts and reproducible audit output ( tracking issue : #31983 ).

Review sign-off : Written approval from at least two gh-aw maintainers that Sections 10–14 are complete, internally consistent, and suitable for Candidate Recommendation publication ( tracking issue : #31983 ). Sync Section titled “Sync” Who reviews : The experiments specification editors ( gh-aw maintainers ) perform the primary review; one release owner for the current minor version performs final sign-off. When : Review occurs on the first business day of each month and during every minor-release cut.

What triggers an immediate sync update : Any change to experiments: schema fields or validation behavior (§4) Any change to variant selection, gating, or persistence logic (§§5–7) Any change to audit/reporting output contracts (§§10–11) Any incident postmortem that identifies spec/implementation drift When a trigger occurs, spec updates SHOULD be merged in the same PR as the implementation change or in a linked follow-up PR within 3 business days.

Feedback should be filed as GitHub issues against the github/gh-aw repository with the experiments label. Table of Contents Section titled “Table of Contents” Introduction Conformance Definitions Frontmatter Schema Variant Selection Algorithms Date-Range Gating State Persistence Expression and Template Integration Activation Job Structure Audit CLI Integration Statistical Analysis and Reporting Simultaneous Experiments and Interaction Effects Security Considerations Compliance Testing References Appendices Change Log 1.

Introduction Section titled “1. Introduction” 1.1 Purpose Section titled “1.1 Purpose” Agentic workflows compiled by gh-aw use a frontmatter-driven configuration model. Teams running these workflows need a first-class mechanism to test different prompt variants (tone, verbosity, persona, feature flags embedded in the prompt) across successive workflow runs. Without such a mechanism, variant testing is ad-hoc, untracked, and statistically unbalanced.

This specification defines a self-contained A/B experiment system that requires no external service, no manual coordination, and no changes outside the workflow frontmatter. 1.2 Scope Section titled “1.2 Scope” This specification covers: The experiments: frontmatter schema and its two syntactic forms (bare-array, rich-object). Variant selection algorithms: balanced round-robin (least-used), weighted random, and date-gated fallback.

State persistence backends: git-branch ( repo ) and GitHub Actions cache ( cache ). Expression and Handlebars template integration in the compiled workflow prompt. The activation job structure generated by the compiler. The gh aw audit CLI filtering interface for experiment-annotated runs. Requirements for statistical analysis and reporting workflows that consume experiment artifacts. This specification does not cover: The internal compiler architecture beyond what is observable at the compiled YAML boundary.

External analytics dashboards or third-party experiment platforms. Multi-armed bandit or adaptive allocation algorithms (considered future work). 1.3 Design Goals Section titled “1.3 Design Goals” Zero external dependencies — all state is stored within the repository or GitHub Actions infrastructure. Declarative — the complete experiment configuration lives in the workflow frontmatter. Backward compatible — adding experiments: to an existing workflow MUST NOT break any existing compiled output; removing it MUST restore the original output exactly.

Statistically sound — the default selection algorithm guarantees approximate variant balance in the minimum number of runs. Observable — every run produces a durable artifact recording the variant assignment, and OTEL attributes propagate assignments to distributed-tracing backends automatically. 2. Conformance Section titled “2. Conformance” 2.1 Requirements Notation Section titled “2.1 Requirements Notation” The key words MUST , MUST NOT , REQUIRED , SHALL , SHALL NOT , SHOULD , SHOULD NOT , RECOMMENDED , NOT RECOMMENDED , MAY , and OPTIONAL in this document are to be interpreted as described in RFC 2119 .

2.2 Conformance Classes Section titled “2.2 Conformance Classes” This specification defines three conformance classes: Class Requirements Level 1 — Basic Satisfies all MUST/MUST NOT requirements in §4, §5, §8, and §9 Level 2 — Standard Level 1 plus §6 (date gating), §7 (state persistence), §10 (audit CLI) Level 3 — Complete Level 2 plus §11 (statistical analysis and reporting) and §12 (simultaneous experiments) An implementation is considered non-conformant if it fails any MUST or MUST NOT requirement at the level it claims to implement.

2.3 Normative vs. Informative Content Section titled “2.3 Normative vs. Informative Content” Sections containing numbered requirements (e.g., “R-SCHEMA-001”) are normative . Notes, rationale blocks, and appendices are informative and carry no conformance weight. 3. Definitions Section titled “3. Definitions” Term Definition Experiment A named A/B test declared in workflow frontmatter, associating an identifier with two or more variant strings.

Variant A named string value representing one treatment arm in an experiment. Control variant The first variant in the declared variants array; used as baseline and as fallback during date gating. Invocation counter A per-experiment, per-variant integer stored in state.json that records the cumulative number of times a variant has been selected. state.json The JSON file that stores invocation counters and per-run assignment history for all experiments in a single workflow.

Run record An entry in the state.json runs array recording the run ID, timestamp, and variant assignments for one workflow run. Sanitized workflow ID The workflow basename (without .

The rest of this page is on the live site.