Overview

Relay AI analyzes test history, failure patterns, and timing data across your pipelines. It flags flaky tests, suggests retries, and summarizes failures so you spend less time digging through logs.

ai:
  flaky_detection: true
  auto_retry:
    enabled: true
    max_attempts: 2
  insights: true

jobs:
  test:
    runs-on: relay-linux-medium
    steps:
      - run: npm test -- --reporter=junit --outputFile=results.xml
      - uses: relay/upload-results@v1
        with:
          path: results.xml

Flaky test detection

ai:
  flaky_detection:
    threshold: 0.15
    window: 30d
    min_runs: 10

jobs:
  e2e:
    steps:
      - run: npm run test:e2e
        retry:
          when: relay.ai.flaky
          max: 2
          backoff: exponential

jobs:
  quarantine:
    if: relay.ai.quarantined_tests != ''
    continue-on-error: true
    steps:
      - run: npm run test:quarantine

Failure summaries

Relay AI generates concise failure summaries on pull requests:

- uses: relay/ai-summary@v1
  with:
    post-to: pull_request
    include:
      - root_cause
      - suggested_fix
      - related_commits

Advanced configuration

API Reference

Best Practices

Pro Tip: Let Relay AI collect at least 10 runs on main before tuning quarantine thresholds.

• Upload structured test results on every pipeline run
• Start with auto-retry before quarantining to avoid hiding real failures
• Review AI summaries on failed PRs before re-running blindly
• Combine AI insights with deploy gates for production releases