How to Run an AI Coding Agent in CI

On your laptop, an AI coding agent has your permissions and you are sitting there watching it. In continuous integration it has whatever the workflow grants and nobody is watching at all. That gap is most of how to run an AI coding agent in CI without incident, and it is a configuration problem rather than a prompting one.

The specific failure worth designing around: an agent triggered by a pull request from outside your organisation, holding repository write credentials in its environment, is a shell you have handed to a stranger. It is entirely preventable, and the prevention happens in the workflow file rather than in the prompt.

Decide what the agent is for

Three jobs suit CI. Most others do not.

What does not fit is open-ended feature work. An agent given a vague instruction and no interactive feedback produces a large diff nobody wants to review, which is the worst outcome available. Keep the scope narrow enough that the result is obviously right or obviously wrong, because the reviewer is the bottleneck and a hundred-file diff from a machine gets rubber-stamped or ignored.

How to run an AI coding agent in CI without handing over the keys

This is the security decision, and it is made by the trigger. On GitHub, a workflow triggered by pull_request from a fork runs without access to your secrets, which is the safe default. A workflow triggered by pull_request_target runs in the context of the base repository with secrets available, and if it checks out the pull request's code it will execute untrusted input with your credentials in the environment. GitHub's events reference is worth reading properly on this point rather than copying a workflow from a blog post.

Safe defaults:

• For public repositories, trigger on issue_comment or a manual dispatch, so a maintainer decides when the agent runs.

• For private repositories where everyone with push access is trusted, pull_request is fine.

• Never check out untrusted code in a job that holds a model API key or a write token.

• Keep the agent's job separate from your build job so a compromised dependency in the build cannot reach the agent's credentials.

• Pin third-party actions to a commit SHA rather than a tag, since a tag can be moved to point at different code.

Scope the permissions down

Set the workflow token to read-only at the top of the file and grant write only on the specific scope the job needs, usually pull-requests: write for a commenting agent. An agent that opens pull requests needs contents: write, and that is the point at which you should ask whether it needs to open them at all or whether a comment with a suggested patch is enough.

Model API keys belong in repository secrets, never in the workflow file, and GitHub's guidance on secrets covers the masking behaviour and its limits. Masking hides a secret in logs. It does not stop a process that has the secret from sending it somewhere, which is why the trigger question above comes first. If your provider supports scoped keys, issue a separate one for CI with its own spending limit, so a leak is a contained incident rather than an open-ended one.

Put a ceiling on cost

An agent in CI runs on every push if you let it, and a busy repository will surprise you. Three controls, in order of how much they save:

1. Filter by path. Do not run the agent on documentation-only or lockfile-only changes.

2. Cap the diff size. Above a few hundred changed lines, skip with a comment saying why. Large diffs produce weak review anyway.

3. Set a hard spend limit at the provider, not only in your workflow. A loop that retries on failure is the classic way to spend a month's budget in an afternoon.

Concurrency limits help too: cancel an in-progress agent run when a new commit lands on the same branch, so a developer pushing four times in ten minutes triggers one review rather than four. Add a job timeout as a backstop, because an agent that hangs waiting on a tool call will otherwise burn a runner for six hours. The broader cost patterns are in how to reduce AI API costs.

Keep the review gate human

An agent's pull request is a proposal, not a merge. Require the same approval any contributor needs, and do not give the agent's account the ability to approve or merge anything. This sounds obvious and it is the rule most often quietly relaxed once the agent has been right a few times in a row.

Agent review comments should also be advisory rather than blocking. A required check that fails on a model's judgment call will be routinely overridden within two weeks, and a check everyone overrides is worse than no check, because it trains the team to dismiss the whole category. The standards for reading its output are the same ones in how to review AI-generated code before you ship it.

A workable first configuration

name: agent-review
on:
  issue_comment:
    types: [created]

permissions:
  contents: read
  pull-requests: write

concurrency:
  group: agent-${{ github.event.issue.number }}
  cancel-in-progress: true

jobs:
  review:
    if: github.event.issue.pull_request && contains(github.event.comment.body, '/review')
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
        with:
          ref: refs/pull/${{ github.event.issue.number }}/head
      - name: Review the diff
        env:
          MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }}
        run: ./scripts/agent-review.sh

That configuration runs only when a maintainer comments, holds read access to contents, can only write comments, cancels superseded runs, and times out after ten minutes. Start there, then loosen deliberately rather than starting loose. The one thing to watch is that issue_comment fires on issues as well as pull requests, which is what the condition on the job handles.

Give it the context it needs

A CI agent has no memory of your conventions and cannot ask. Commit the context: a file describing your stack, your testing approach, and the review standards you care about, read by the agent on every run. This is the same practice as writing an AGENTS.md file for interactive use, and it matters more in CI because there is no chance to correct a wrong assumption mid-run.

Be specific about what you do not want flagged, too. An agent that has not been told your project deliberately avoids a pattern will report it as a finding on every single pull request, and that is how teams end up muting the whole thing.

Know when it is working

Track two numbers over the first month: what fraction of the agent's comments led to a change, and what it cost. A useful ratio is somewhere above one in four. Below that, the reading time exceeds the value and you should narrow its scope rather than tolerate the noise. This is a cheap measurement and almost nobody does it, which is why so many teams have an agent running that everyone has learned to scroll past.

FAQ

Should the agent run on every pull request?

Start with manual invocation by comment. Move to automatic once you know the cost per run and have seen the quality of its output on your codebase. Going the other way, from automatic to manual, usually happens after a bill.

Can I let an agent merge its own pull requests?

No. Even with green tests, tests only cover what you thought to test, and an agent optimising for a passing suite is a well-documented failure mode.

How do I stop the agent leaking source code?

Check your provider's data retention terms for API traffic, and if your code cannot leave your infrastructure, this is one of the clearer cases for running a coding model locally on a self-hosted runner.

What if the agent's review is wrong?

Expect a portion to be. That is why comments are advisory. Track roughly how often it is useful, and if the ratio does not justify the reading time, narrow its scope instead of accepting noise.

Does this work outside GitHub Actions?

The principles transfer directly. Every CI system has an equivalent of the trusted-versus-untrusted trigger distinction and an equivalent of scoped tokens. The names differ, the decision does not.

An agent in CI is worth it when it does something a human reviewer reliably skips: the tedious consistency checks, the missing error handling, the test nobody wrote. It is not worth it as a second opinion on design. Picking the right job is most of the value, and choosing among AI coding tools is the sensible place to start if you have not settled on one.