README

Give your AI coding agent everything it needs to fix a failing test — in a single run.

A Codeception 5 extension that captures structured, deterministic failure context and writes it as machine-readable JSON and plain-text artifacts. Built for the fix-in-a-loop workflow: agent runs tests, reads the report, patches code, repeats — without ever needing a human in the loop.

Why this exists

When an AI agent encounters a failing Codeception test, the default output is a wall of terminal text: ANSI escape codes, PHPUnit XML noise, interleaved suite headers, and a stack trace that buries the actual problem. Agents waste tokens parsing noise instead of fixing bugs.

This extension solves that by producing a clean, stable output file after every test run:

• ai-report.json — structured data ready for programmatic consumption (default)
• ai-report.txt — compact human-readable summary, enable with format: text or format: both

Every failure record contains exactly what an agent needs:

With the --report flag, the same context is also printed inline in the terminal output immediately after each failure — useful for agents that read stdout directly.

Installation

composer require --dev WebProject-xyz/codeception-module-ai-reporter

Requires PHP ^8.3 and Codeception ^5.3.5.

Setup

Add the extension to your codeception.yml:

extensions:
    enabled:
        - WebProject\Codeception\Module\AiReporter\Extension\AiReporter:
              format: json          # text | json | both
              output: tests/_output
              max_frames: 8
              context_lines: 4
              include_steps: true
              include_artifacts: true
              compact_paths: true

Usage

Standard run

vendor/bin/codecept run

Report files are written to tests/_output/ after every run, regardless of whether tests pass or fail.

Agent run mode (--report)

vendor/bin/codecept run --report

Enables inline AI context — a structured block printed directly below each failure in the terminal:

  AI Context
    Test failed: tests/Unit/MyTest.php:checkValue
    Rerun: vendor/bin/codecept run tests/Unit/MyTest.php:checkValue
    Exception: PHPUnit\Framework\ExpectationFailedException
    Message: Failed asserting that two strings are identical.
    Diff:
      --- Expected
      +++ Actual
      @@ @@
      -'expected-value'
      +'actual-value'
    Trace:
      #1 tests/Unit/MyTest.php:42 MyTest->checkValue
    Source tests/Unit/MyTest.php:42:
          40 |     {
          41 |         $value = $this->subject->render();
      >   42 |         self::assertSame('expected-value', $value);
          43 |     }

Recommended agent instruction

Drop this into your agent's system prompt or task description:

Run `vendor/bin/codecept run --report` and use the inline AI Context
plus tests/_output/ai-report.json to identify and fix failures.
Repeat until all tests pass.

Output format

JSON (ai-report.json)

A machine-readable schema is available at schema/ai-report.schema.json (JSON Schema 2020-12).

{
  "run": {
    "generated_at": "2026-02-19T12:00:00+00:00",
    "duration_seconds": 1.23,
    "project_root": "/repo/project",
    "output_dir": "/repo/project/tests/_output"
  },
  "summary": {
    "tests": 10, "successful": 9, "failures": 1,
    "errors": 0, "warnings": 0, "assertions": 42,
    "successful_run": false
  },
  "failures": [
    {
      "status": "failure",
      "suite": "Unit",
      "test": {
        "display_name": "MyTest: check value",
        "signature": "MyTest:checkValue",
        "full_name": "MyTest:checkValue",
        "file": "tests/Unit/MyTest.php"
      },
      "exception": {
        "class": "PHPUnit\\Framework\\ExpectationFailedException",
        "message": "Failed asserting that two strings are identical.",
        "comparison_expected": "'expected-value'",
        "comparison_actual": "'actual-value'",
        "comparison_diff": "--- Expected\n+++ Actual\n@@ @@\n-'expected-value'\n+'actual-value'",
        "previous": []
      },
      "scenario_steps": [],
      "trace": [
        { "file": "tests/Unit/MyTest.php", "line": 42, "call": "MyTest->checkValue" }
      ],
      "source_context": {
        "file": "tests/Unit/MyTest.php",
        "line": 42,
        "start_line": 40,
        "lines": ["    {", "        $value = $this->subject->render();", "        self::assertSame('expected-value', $value);", "    }"]
      },
      "artifacts": {}
    }
  ]
}

Text (ai-report.txt)

Context
generated_at: 2026-02-19T12:00:00+00:00
project_root: /repo/project
totals: tests=10 successful=9 failures=1 errors=0 warnings=0 skipped=0 incomplete=0 useless=0 assertions=42

Failure 1
status: failure
suite: Unit
test: MyTest: check value
rerun: vendor/bin/codecept run tests/Unit/MyTest.php:checkValue
test_file: tests/Unit/MyTest.php
test_signature: MyTest:checkValue

Exception
exception_class: PHPUnit\Framework\ExpectationFailedException
message: Failed asserting that two strings are identical.
comparison_expected: 'expected-value'
comparison_actual: 'actual-value'
comparison_diff:
--- Expected
+++ Actual
@@ @@
-'expected-value'
+'actual-value'

Scenario
none

Trace
#1 tests/Unit/MyTest.php:42 MyTest->checkValue

Source
tests/Unit/MyTest.php:42
  40 |     {
  41 |         $value = $this->subject->render();
> 42 |         self::assertSame('expected-value', $value);
  43 |     }

Artifacts
none

Configuration reference

Platform support

• Linux, macOS, and Windows paths are all handled correctly.
• Paths are normalized to forward slashes in report output for consistency across platforms.

Contributing

Contributions are welcome. Please open an issue before submitting large changes.

composer test:build   # rebuild Codeception actor classes
composer test         # run tests
composer stan         # PHPStan static analysis (level 7)
composer cs:check     # check code style
composer cs:fix       # auto-fix code style

License

MIT