Text of github.github.com › gh-aw › specs › graders-specification as Styx crawled it on Nov 19, 2026. The page may have changed since. Only its words are kept, without pictures or layout, and only the first part of a long page.

Open the live page

Graders Specification | GitHub Agentic Workflows

Skip to main content Graders Specification Version : 0.8.0 Status : Draft Specification Feature Status : Experimental 1. Introduction Section titled “1. Introduction” Graders compute deterministic execution metrics without additional model calls. This specification defines configuration, the in-memory interface, execution ordering, artifact output, grader-backed experiment metrics, and read-only reporting of recorded operational-value results.

The key words MUST, MUST NOT, SHOULD, and MAY have the meanings defined by RFC 2119 . 2. Grader interface Section titled “2. Grader interface” Built-in and JavaScript graders MUST receive an in-memory unified session , represented as an array of canonical events with type , data , and provenance . The caller MAY supply non-executable configuration and shared indexes. These graders MUST be pure computation. They MUST NOT read files, spawn processes, access the network, inspect environment variables, or mutate the input.

Identical session and configuration inputs MUST produce identical results. Legacy preprocessed trace payloads are not accepted by the public interface. createSessionLookups(session) materializes event-type indexes, correlated tool calls, and concrete execution statistics once for all graders. Internal trajectory algorithms MAY consume projections derived solely from that session. They MUST NOT reconstruct missing evidence by reading external artifacts.

A Git commit, checkout, or later GitHub state is not an additional grader input. External measurement facts MUST already be present in the supplied session. Historical Git analysis and later lifecycle collection are separate consumers, not mechanisms for adding facts to an immutable session or regrading it. Main-agent execution statistics MUST exclude detection and eval usage. Native tool starts and completions MUST be correlated within their session; unfinished calls count as failures.

Native calls take precedence over duplicate gateway evidence. Proxy token telemetry takes precedence over aggregate agent usage to avoid double-counting. Missing required evidence MUST produce unavailable , not fabricated measurements. 3. Configuration model Section titled “3. Configuration model” Omitting graders , setting it to null, or using graders: {} MUST enable the ten fast built-ins listed below. Other registered built-ins require explicit configuration. graders : loops : enabled : false tool-success-rate : threshold : 0.95 recurrence-determinism : {} Supported per-grader fields are enabled , name , description , unit , direction , threshold , min , max , and config .

These fields carry data only. Custom IDs MUST additionally declare an inline JavaScript script , limited to 256 KiB. Built-in IDs MUST NOT accept scripts or implementation overrides. The reserved operational-value ID MUST declare restricted JavaScript script . run is unsupported for every ID. Unsupported fields and built-in overrides MUST fail validation, even when disabled. Disabling a default with enabled: false MUST persist through manifest normalization and replay.

Disabling all defaults is valid. 4. Built-in graders Section titled “4. Built-in graders” The default set MUST contain only fast, linear-time metrics: ID Measurement tool-success-rate Successful tool calls / total tool calls tool-failure-count Failed or unfinished tool calls retries Retry observations in main-agent MCP events loops Consecutive calls with identical tool names and arguments trajectory-efficiency Distinct tool names / total calls execution-step-count Observed model request count execution-duration Observed duration in milliseconds working-set-rebuild-factor Cumulative input tokens / peak invocation input tokens context-growth Total tokens / first-request tokens artifact-production Safe-output result count Trajectory analyses MUST remain opt-in.

Reference-dependent results MUST be unavailable when the unified session does not contain the required evidence. Potentially quadratic trajectory analyses MUST retain bounded input sizes. The full registry is documented in the Graders Reference . 5. Execution and artifacts Section titled “5. Execution and artifacts” Built-in grading MUST execute in the conclusion job , after agent, detection, safe-output, experiment, and eval observations have been collected into the unified session.

The caller MUST redact the in-memory session before grading. Graders MUST NOT run in the agent job or reread raw logs. Collection, grading, and persistence are separate responsibilities. The persistence boundary MUST: Compute enabled graders from the collected, redacted event array. Write usage/graders/grader_manifest.json and usage/graders/grader_results.json . Append grader.manifest and grader.result events with conclusion provenance.

Persist usage/aw_session.jsonl and upload the usage artifact. Manifest and result schema version is 2. Results MUST preserve stable grader IDs, implementation identity, numeric value or null, unit, status, and threshold outcome. Individual computation failures MUST be recorded as errors and surfaced in conclusion, not converted into successful scores. Conclusion MUST NOT receive or execute workflow-provided JavaScript.

Its manifest MAY register custom grader IDs, but MUST NOT contain custom executable definitions. User-provided conclusion steps MUST be rejected. Custom JavaScript MUST run only in a separate graders job , downstream of a successful conclusion and its published usage artifact. This job MUST use a fresh hosted runner, read-only repository access, no inherited deployment environment, and no App, inference, or OIDC credential minting.

Work-queue participants without a published conclusion session MUST reject custom graders. Authors MUST NOT customize this compiler-owned job. Each custom JavaScript script MUST receive immutable in-memory session/configuration snapshots and session-derived lookup helpers, with no host-object bindings. Ordinary custom IDs MUST be synchronous and return a finite number, null, or a result object with a finite/null value .

Reserved operational-value MUST use the same restricted worker and return a nonempty ordered array of unique {id,value} objects with nonempty string IDs and finite numeric or null values. The first metric MUST supply top-level operational-value.value ; remaining metrics MUST become diagnostics. Schema-v2 results and full result-event projections MUST preserve the complete ordered metrics array. Compact profiles MAY omit per-grader results and MUST NOT be treated as a complete result transport.

Downstream custom results MUST NOT be appended to the already published conclusion session. File, module, environment, process, network, clock, randomness, and timer capabilities MUST NOT be exposed. Separate restricted Node processes MUST bound memory, output, and execution time. Existing adapter projections, including gh-cao JSONL mapping, MUST continue to consume the top-level operational-value source and primary value.

Removed Bash syntax Section titled “Removed Bash syntax” graders.operational-value.run is invalid. Compilers MUST fail with clear migration guidance to graders.operational-value.script , including for disabled or imported declarations. They MUST NOT ignore or translate Bash automatically. Operational-value authority Section titled “Operational-value authority” gh-aw owns producer identity, run-scoped grading, artifacts/events, experiments consumption, and explicit replay.

GitHub is authoritative for later object disposition; gh-aw projects native delivery, adoption, reversal, and survival observations separately from run grades. safe_output.execution records handler application, not human adoption. Workflows in any repository MAY report recorded grades and native lifecycle evidence through gh aw value without a campaign or CAO installation. githubnext/gh-cao and other consumers MAY schedule collection, retain records, and derive repository/campaign views; they MUST NOT

The rest of this page is on the live site.