Giving an AI Agent Production Access

The question arrives in roughly this form: the assistant can already read our logs and suggest fixes, so why not let it apply them?

It is a reasonable question and the honest answer is not "never". It is that autonomy is not one decision — it is several, and the ones people conflate are the ones that cause trouble.

This is about the engineering, not the technology. Whether the thing acting on your infrastructure is an LLM agent, a remediation rule in a monitoring tool, or a cron job somebody wrote in 2019, the questions are identical. What changed recently is that the capability became easy to acquire, so the questions are being asked by more people and answered by fewer.

The four levels, and why they are not a slope

These get treated as a gradual ramp. They are not — the gap between the third and fourth is categorically different from the gaps before it.

1. Read and summarise. It looks at telemetry and tells you what it sees. No write access anywhere. Low risk, immediately useful, and where almost everyone should start.

2. Recommend. It proposes a specific action — the command, the manifest change, the rollback. A human reads it and decides. Still no write access.

3. Act, reversibly, within a boundary. It performs a narrow set of actions you have enumerated, all of which can be undone, all of which are logged.

4. Act, generally. It has credentials and judgement about when to use them.

Levels 1 to 3 are engineering problems with known shapes. Level 4 is a different thing: you are no longer bounding what can happen, you are trusting a judgement you cannot inspect in advance. Almost nobody needs level 4, and the people who most want it are usually the least well placed to contain it.

What makes an action safe to automate

Three properties, and an action needs all three. Losing any one of them is what turns automation into an incident.

Reversible. There is a command that undoes it, and you know what it is. Restarting a pod is reversible in the sense that matters — the system returns to a known state. Deleting a PersistentVolumeClaim is not.

Bounded. It acts on a named, finite set of things. "Restart pods in namespace web matching app=frontend" is bounded. "Restart unhealthy pods" is not, because the blast radius depends on how many happen to be unhealthy, and the worst case is every one of them at the moment the cluster is already struggling.

Observable. Every action is recorded with what was done, why, and by which identity — and the record survives the thing that did it. An agent that acts and does not log is worse than no automation, because you now have unexplained changes.

Note what is not on this list: confidence. A high-confidence wrong action is worse than a low-confidence one, because nobody checks it.

The failure mode specific to this

Traditional automation fails predictably. A rule that restarts a pod on OOM either fires or does not; when it misbehaves, it misbehaves the same way every time, and you can reproduce it.

A model-driven agent fails differently. It can produce a confident, well-reasoned, entirely wrong action — and produce a different one on the same input tomorrow. The output looks exactly like the output that was right last week.

Two practical consequences.

Your guardrails cannot live in the prompt. Telling the model not to delete things is a request, not a control. The boundary has to be in the credentials: a role that cannot delete cannot be talked into deleting. This is ordinary least privilege and it is the whole defence.

The reasoning is not evidence. An explanation of why an action is correct is generated alongside the action, not derived from it. A plausible justification for a wrong action is the normal case, not an anomaly.

A worked example

Alerts fire: elevated error rate, one service, starting four minutes ago.

Level 2, recommending:

Error rate on checkout-api rose from 0.1% to 12% at 02:11. A deploy of checkout-api:1.9.0 completed at 02:09. The previous revision ran for six days at baseline.

Suggested: kubectl rollout undo deployment/checkout-api -n production

Confidence: moderate. Correlation is timing only; I have not verified the errors originate in the new code.

That is genuinely useful. It did the first fifteen minutes of triage — correlating the alert against the deploy timeline — and it flagged the weakness of its own inference. A human decides.

Note what makes this tractable: the alert fired on a symptom users would notice, not on a cause. An agent reasoning over cause-level alerts inherits all the noise those produce, which is why symptom vs cause alerting matters more here than it does for a human on call — a person discounts a noisy alert automatically and an agent does not.

Level 3, acting, correctly bounded:

# The narrow, enumerated version
allowed:
  - action: rollout_undo
    namespaces: [production]
    workloads: [checkout-api, cart-api, search-api]
    conditions:
      - error_rate_above: 5%
      - sustained_for: 3m
      - deploy_within: 15m           # only rolls back a *recent* change
    max_per_hour: 2                  # stops a loop
    notify: "#incidents"
    audit: required

Every constraint there is doing a job. The workload list bounds it. The deploy-recency condition stops it rolling back something unrelated to a change. max_per_hour is the one people omit and the one that prevents the worst outcome: an agent that rolls back, sees errors continue because the cause was downstream, rolls back again, and walks the deployment history backwards during an incident.

What level 4 would look like, and why it is different: an agent with cluster-admin deciding for itself what to do about the error rate. The failure is not that it might roll back wrongly — level 3 might too. It is that you cannot enumerate what it might do instead.

Getting there, in an order that works

Run it in shadow for weeks, not days. Let it recommend, log what it would have done, and compare against what humans actually did. You are measuring how often it would have been right — and specifically, how it fails when it fails. If it has never been wrong in your shadow period, your shadow period is too short or your incidents are too samey.

Start with one action, on one workload class. Rolling back a recent deploy is the usual first candidate: reversible, bounded, high-value, and the correlation is unusually strong.

Give it an identity of its own. Not a shared service account, not a human's credentials. A named identity with its own narrow role, so the audit log answers "who did this" and so you can revoke it in one place. The keyless, short-lived pattern applies here as much as anywhere — the OIDC exchange used by CI is the same mechanism.

Make it announce itself. Every action into the incident channel, as it happens. Silent automation during an incident means two actors changing the same system without visibility — which is a worse problem than the one you automated.

Rehearse turning it off. A documented kill switch somebody has actually used, not one that exists in principle. If the procedure for stopping it has never been run, you do not have it.

When not to

During an incident you do not understand. Automation acts on a model of what is wrong. If nobody has that model yet, the agent does not either. A deploy that reports success while users see errors is the shape of incident where this bites hardest: every signal the agent has says the rollout completed.

On anything irreversible. No deletions, no destructive migrations, no scaling to zero for stateful workloads.

Where the action changes something outside your blast radius — a shared dependency, another team's service, anything customer-visible in a way you cannot undo.

When the telemetry is unreliable. Automated action reasons over your signals. If your alerts are noisy or your metrics have gaps, you have automated acting on bad data, which is faster than a human doing it and no better.

To compensate for an unfixed problem. An agent that restarts a service every night is a memory leak with extra steps, and automating it removes the pressure to fix it.

The honest position

Levels 1 and 2 are straightforwardly worth having. Removing the first fifteen minutes of triage from an incident is real value, and the risk is close to zero because nothing is being changed.

Level 3 is worth having for a small number of well-understood actions, once you have run it in shadow long enough to know its failure modes, and only with credentials that make the dangerous cases impossible rather than merely discouraged.

Level 4 is a claim about trust that very few organisations are positioned to make, and the ones making it loudest are usually selling something.

The unglamorous prerequisite for all of it is the same as for AIOps generally: telemetry that is consistent and joinable. An agent reasoning over three monitoring tools with three different names for the same service will be confidently wrong, and it will explain itself beautifully while it is.

Sources worth reading

The engineering guidance that holds up across vendors is consistent on the same few points — narrow scope, reversible actions, a signed audit trail per action, and recommendation-only before autonomy. The Cloud Security Alliance's work on AI agent governance and Google Cloud's blueprint for AI-assisted vulnerability management are both worth reading directly rather than through summaries.

Treat adoption statistics in this area with suspicion. The numbers circulating are largely from vendors whose product is the remedy, and the ones that are not are projections being quoted as measurements.