DevSecOps in the Age of AI Agents: How to Secure, Deploy, Isolate, and Scale AI Agent Infrastructure on Kubernetes and CloudDevSecOps in the Age of AI Agents: How to Secure, Deploy, Isolate, and Scale AI Agent Infrastructure on Kubernetes and Cloud

Most of the software we have spent the last fifteen years securing follows paths someone wrote down. A request comes in, a handler runs, a query goes out, a response comes back. The code can be buggy, and attackers can find paths the authors did not intend, but the set of things the program can do is fixed at build time. Security teams learned to reason about that: enumerate the endpoints, scan the code, lock down the database, watch the logs.

An AI agent breaks that assumption. It is a program whose next action is chosen at runtime by a model reading text, and some of that text comes from people and systems you do not control. Depending on how it is wired, an agent can decide what to do, call tools, hit internal and external APIs, retrieve documents, write files, run code, open pull requests, touch infrastructure, talk to other agents, and generate follow-up actions nobody explicitly asked for.

That changes the security problem more than the model does. A compromised web application might expose a database. A compromised agent may be holding cloud credentials, a connection to your ticketing system, read access to a document store, a code execution sandbox and a deploy pipeline, all inside one session, and it will happily use them in whatever order the latest instruction in its context window suggests.

So DevSecOps for AI agents is not a new discipline that replaces the old one. It is the old discipline with the stakes raised, plus a set of controls that only matter once the workload itself can make decisions. The organising idea of this guide is simple: the blast radius of an agent matters more than the model behind it. Identity, capability, authorization, isolation and observability are how you keep that blast radius small.

We will build up to a full reference architecture, starting with what DevSecOps actually is, then secure cloud and Kubernetes foundations, then the agent-specific threat model, isolation, identity, supply chain, runtime, observability, scale and multi-tenancy.

A note on how to read the standards cited here. Very little of this is legally mandatory for most organisations. NIST, OWASP, CIS and CNCF material is guidance you adopt voluntarily unless a regulator, contract or customer makes it binding. Protocol specifications are different: when the MCP specification says a server MUST NOT accept tokens not issued for it, that is a conformance requirement, not a suggestion. Where this article gives an opinion rather than a documented requirement, it says so.

What DevSecOps actually means

DevSecOps gets reduced to "add a scanner to CI", which is roughly like reducing reliability engineering to "add a health check". The actual idea is that security is a property of the delivery system and the running system, owned by the people who build and operate them, enforced by automation rather than by a review meeting at the end.

NIST's Secure Software Development Framework (SP 800-218) is the most useful vendor-neutral description of the development half. It groups practices into preparing the organisation, protecting the software, producing well-secured software and responding to vulnerabilities. Version 1.1 is the current final release, and NIST published a draft of version 1.2 in December 2025. NIST SP 800-204D applies similar thinking specifically to software supply chain security inside CI/CD pipelines. For operations, the NIST Cybersecurity Framework 2.0 gives the broader govern, identify, protect, detect, respond and recover structure.

In practice, a mature DevSecOps programme covers these areas:

• Secure SDLC and shift-left. Threat modelling during design, security requirements in tickets, and fast feedback in the developer's editor and pull request, where a fix costs minutes.
• Continuous security. Shift-left is necessary but incomplete. New CVEs are published against code you shipped last year. Continuous scanning of what is running matters as much as scanning what is being merged.
• Source, dependency and secret hygiene. Static analysis, software composition analysis and secret detection on every change.
• Infrastructure as Code security. Terraform, Helm, Kustomize and CloudFormation are code, and misconfigurations in them are the most common source of cloud exposure.
• Container and Kubernetes security. Minimal images, image scanning, hardened workload specs, cluster configuration benchmarks, admission control.
• Cloud security, identity and secrets. Account structure, IAM, workload identity, key management and secret delivery.
• Vulnerability management. Not just finding CVEs, but triaging by reachability and exposure, and tracking fixes to closure.
• Runtime security and observability. Detecting what actually happens in production, and having the telemetry to investigate it.
• Compliance and policy-as-code. Turning written policy into machine-enforced rules, so evidence is a by-product of the pipeline rather than a spreadsheet.

Every one of these controls is partial. The table below is the part most tool comparisons leave out: what each category is blind to.

Two observations fall out of this table. First, the controls overlap on purpose: SCA and image scanning both find vulnerable packages, but at different points and with different blind spots. Second, almost nothing in the traditional stack detects a workload using its own legitimate permissions for the wrong reason. That gap is exactly where AI agents live.

DevSecOps is a pipeline, not a tool

The most common failure pattern we see is a team that buys one good scanner, wires it into CI, and considers the problem addressed. No single control has full visibility, so a mature programme places different controls at each point where an artifact changes hands.

The feedback edge at the bottom matters as much as anything else in the diagram. An incident that does not produce a new detection rule, admission policy or test has only been cleaned up, not learned from.

A few principles make this pipeline work in practice rather than in a slide:

• Block on a small set of high-confidence findings, report the rest. A pipeline that fails on every medium-severity CVE gets bypassed within a month.
• Verify at the point of use, not only at the point of creation. Signing in CI is worth little if the cluster does not verify signatures at admission.
• Treat the pipeline itself as production. CI runners hold registry credentials, signing identities and often cloud deploy roles. A compromised pipeline defeats every control upstream of it.

Designing secure cloud infrastructure for Kubernetes

Kubernetes is not automatically secure because it is Kubernetes. A managed cluster with default settings, a public API endpoint and nodes that can reach the instance metadata service is a reasonable place to run a demo and a poor place to run agents holding production credentials. Most of the real decisions happen in the cloud account underneath.

Account structure first. Use separate cloud accounts, projects or subscriptions for production, non-production and shared security tooling, under an organisation with guardrails (AWS Organizations with SCPs, Google Cloud organisation policies, Azure Management Groups with Azure Policy). The account boundary is the strongest isolation primitive most teams have, and it is also the one that limits what a leaked credential can reach. Log archives belong in an account that production workloads cannot write to or delete from.

Network layout. Put nodes in private subnets. Expose workloads through a load balancer in public subnets only when they genuinely serve the internet, and use internal load balancers for everything else. Egress through NAT is convenient, but NAT is not an egress control: it hides your source address and lets everything out. For agents, route outbound traffic through an egress proxy or firewall with an allowlist of destinations, because Kubernetes NetworkPolicy works at the IP and port level and cannot express "only api.github.com" without CNI extensions such as Cilium's FQDN policies.

Control plane access. Private API server endpoints (EKS private endpoint, GKE private clusters, AKS private clusters) or at minimum authorised network ranges. The Kubernetes API is the most privileged interface in the cluster, and there is rarely a good reason for it to accept connections from the whole internet.

Identity. Humans reach the cluster through SSO federated into cloud IAM and mapped to Kubernetes RBAC groups, never through shared kubeconfig files with long-lived certificates. Workloads use workload identity: EKS Pod Identity or IRSA, Workload Identity Federation for GKE, or Microsoft Entra Workload ID on AKS. Each maps a Kubernetes service account to a cloud identity with short-lived credentials, so no static cloud keys live in the cluster.

Metadata service protection. If a pod can reach the node's instance metadata endpoint, it may be able to obtain the node's cloud role. On AWS, require IMDSv2 and set the response hop limit to 1 for nodes where pods should not reach metadata, as the EKS best practices guide recommends. Note that pods relying on the node role will then break, which is the point: move them to Pod Identity or IRSA. On GKE, Workload Identity Federation intercepts metadata requests via the GKE metadata server. A NetworkPolicy blocking 169.254.169.254/32 is a sensible extra layer everywhere.

Nodes. Use a minimal, container-optimised OS (Bottlerocket, Container-Optimized OS, Azure Linux or similar), keep node images patched by replacing nodes rather than patching in place, and separate node groups by workload trust level. System add-ons, general services and untrusted agent execution should not share nodes.

Encryption and keys. Kubernetes Secrets are base64-encoded and, unless you configure encryption at rest, stored unencrypted in etcd. Managed providers encrypt etcd storage, and most support envelope encryption with a customer-managed KMS key. Better still, keep high-value secrets in an external secret manager and deliver them just in time, which we will come back to.

Registries. Pull only from private registries you control, with pull-through caches for upstream images, and restrict node pull permissions to those registries. Enforce this at admission, not by convention.

Logging, audit and recovery. Enable Kubernetes API audit logs and cloud control plane logs (CloudTrail, Cloud Audit Logs, Azure Activity Log), ship them to a separate account, and alert on sensitive operations. Back up cluster state and persistent volumes with tooling such as Velero, and actually rehearse a restore. For clusters that run agents, keep GitOps as the source of truth so a cluster can be rebuilt rather than repaired.

Managed vs self-managed. For most organisations, managed Kubernetes is the more secure choice, because the provider patches and operates the control plane and etcd. Self-managed clusters make sense for air-gapped, sovereign or bare-metal environments, but you then own control plane hardening against the CIS Kubernetes Benchmark yourself.

Multi-cluster. Separate production from non-production at the cluster level at minimum, and ideally at the account level. Beyond that, split clusters by trust and blast radius rather than by team org chart.

The Kubernetes security model

To secure Kubernetes you need a clear picture of what trusts what. The Kubernetes security documentation and the NSA/CISA Kubernetes Hardening Guidance cover the components in depth. The short version:

• API server. Every change goes through it, so authentication, RBAC and admission control here are the core of cluster security.
• etcd. Holds all cluster state, including Secrets. Direct etcd access is equivalent to full cluster control.
• Kubelet. The node agent. Its API must require authentication and authorisation, and the Node authorizer plus the NodeRestriction admission plugin limit what a compromised node can modify.
• Nodes and container runtime. Containers on a node share the host kernel. A kernel vulnerability or a privileged container can turn a single-pod compromise into a node compromise, and from the node, into access to every pod and mounted secret on it.
• Service accounts and RBAC. Every pod runs as a service account. Kubernetes projects short-lived, audience-bound tokens by default now, but a pod that does not need the API should not mount one at all (automountServiceAccountToken: false).
• Admission controllers, CRDs and controllers. Operators often run with cluster-wide permissions. A controller that can create pods in any namespace is a privilege escalation path if it is compromised or tricked.
• Network. By default every pod can talk to every other pod. NetworkPolicy only takes effect if your CNI enforces it, and policies are allow-lists that apply once any policy selects a pod.
• Secrets and persistent volumes. Anyone who can create a pod in a namespace can mount any Secret in that namespace. The Kubernetes RBAC good practices page calls this out explicitly, along with the escalate, bind and impersonate verbs.
• Service mesh. Where used, a mesh such as Istio or Linkerd adds mutual TLS and workload identity for service-to-service calls, which is useful for authorization between services. It does not replace network policy or authorization in the application.

What to trust: the control plane (if it is managed and patched), the admission policies that enforce your baseline, the identity provider, and signed artifacts from your own pipeline.

What not to trust: any container's claim about itself, any image that has not been verified, any input arriving from outside the process, and in the case of agents, any instruction the model derives from content it did not receive from an authenticated principal.

Where boundaries should be: at the edge, at ingress, between namespaces with policy, between node pools with different trust levels, around the secrets manager, and at every tool or API an internal workload calls. The question for each boundary is the same: if the thing on the inside is fully compromised, what can it reach?

The dotted edges are the question to ask for every workload: if this pod were fully compromised, which of these can it reach, and with whose credentials?

What changes when the workload is an AI agent?

Not every AI workload is an agent, and the distinction matters because the controls differ.

• LLM inference service. Takes a prompt, returns tokens. Its security concerns are mostly classic: authentication, rate limiting, resource exhaustion, model artifact integrity, and data handling for prompts and outputs. It does not act on anything.
• AI application. Wraps a model with application logic, such as a chat assistant or a RAG search. The application decides what to do with the output. Risks include prompt injection, sensitive data disclosure, and improper handling of output (for example, rendering model output as HTML).
• AI agent. The model decides which tools to call, with which arguments, in a loop, until it believes the task is done. The application no longer fully determines the control flow.
• Multi-agent system. Several agents delegate to each other, share memory or pass messages. Trust decisions now happen between automated principals, and a single injected instruction can propagate.

An agent in production is rarely just a model. It is a composition:

The arrow that makes this hard is the last one. Tool results go back into the context window, and the model treats everything in its context as potentially instructive. A support ticket, a web page, a README in a repository or a document retrieved by RAG can contain text that reads like an instruction. The model has no reliable, built-in way to separate "data I was given" from "instructions I should follow". This is why the OWASP Top 10 for LLM Applications 2025 still ranks prompt injection first, and why the newer OWASP Top 10 for Agentic Applications, published in December 2025, opens with Agent Goal Hijack.

The practical consequence is an assumption we recommend designing around: treat an agent with meaningful capabilities as a potentially compromised process. Not because the model is malicious, but because its behaviour is steerable by inputs you cannot fully sanitise. Then design so that a steered agent still cannot do much damage.

This also clears up a few common misconceptions:

• Putting an LLM behind an API gateway does not secure an agentic system. The gateway authenticates the caller of the model. It knows nothing about what the agent then does with tools.
• "Human in the loop" is not authorization. An approval prompt is a useful control for high-impact actions, but humans approve what they are shown, get fatigued, and cannot evaluate a hundred tool calls an hour. The system still needs to enforce what the agent is permitted to do regardless of whether a human clicked approve. The OWASP agentic list includes human-agent trust exploitation as its own risk for this reason.
• The model is not the security boundary. System prompts, guardrail classifiers and refusal training reduce risk. They are probabilistic, and they should never be the only thing standing between an injected instruction and a production database.

Threat model for AI agents

A good agent threat model starts from the same place as any other: assets, entry points, trust boundaries and attacker goals. Agents add new entry points (any content that reaches the context window) and new ways to turn access into action (tool calls). The frameworks worth anchoring to are the two OWASP lists above, MITRE ATLAS for adversarial techniques against AI systems, NIST AI 100-2 for its taxonomy of adversarial machine learning, and the NIST AI RMF Generative AI Profile (AI 600-1) for risk management.

Not every threat has equal weight. In our experience the ones that turn into real incidents are the unglamorous combinations: an injected instruction plus an over-privileged tool plus no egress control. Model extraction and training-time poisoning matter a great deal to model producers and much less to most teams deploying agents on top of hosted models.

Every edge from "Core" to "Actions" is where authorization and isolation must sit. You cannot remove the edges on the left completely, so the edges in the middle carry the weight.

If you remember one row, make it the confused deputy. It is the most common design flaw we find in agent systems: a user with read-only access asks an agent, which holds a write-capable service credential, to "fix" something, and the agent does it.

Secure AI agent deployment architecture

The logical architecture that avoids most of the table above looks like this:

User → API gateway → authentication → authorization → agent orchestrator → model gateway → agent runtime → tool gateway → internal and external services.

Each hop has a distinct job:

• API gateway and authentication. Terminates TLS, authenticates the user or calling system through your identity provider, applies coarse rate limits.
• Authorization. Decides whether this principal may start this kind of agent task at all, and with which capability set.
• Agent orchestrator. Owns the agent loop, state, iteration limits and budgets. It is also where a task's capabilities get attached to the run.
• Model gateway. A single egress point for model calls: provider credentials, routing, quotas per tenant, logging of token usage, and content filtering where you use it. Agents never hold model provider API keys directly.
• Agent runtime. Where agent code and any code execution actually run. This is the part to isolate most aggressively.
• Tool gateway. The enforcement point for every action. It validates arguments against schemas, evaluates policy with the user, agent, tool, operation and resource as inputs, obtains a scoped credential for the downstream call, and logs the decision.

On Kubernetes, split these into a control plane (things that decide and enforce) and a workload plane (things that execute on behalf of a request):

• Control plane: identity integration, policy engine, orchestrator, model gateway, tool gateway, secrets manager integration, admission controllers, observability pipeline. These run in dedicated namespaces with tightly held RBAC, on nodes that never run untrusted code.
• Workload plane: agent runtimes and execution sandboxes. These run in per-tenant or per-agent namespaces, with default-deny network policy, the restricted Pod Security Standard, no Kubernetes API token, a sandboxed runtime class where code execution is involved, and egress only to the gateways.

The key design rule: the workload plane should hold as close to zero long-lived credentials as possible. The runtime can ask the tool gateway to do something. It cannot do that thing directly.

Supporting all of this: images come only from a private registry and must carry valid signatures and provenance at admission; secrets flow from the secrets manager into gateways via workload identity; runtime detection runs on every node; and all decisions from the policy engine and gateways feed the same traces as the agent loop.

Isolating AI agents on Kubernetes

Isolation is where people most often overestimate what Kubernetes gives them. The Kubernetes multi-tenancy documentation is candid that the namespace model "requires configuration of several other Kubernetes resources, networking plugins, and adherence to security best practices to properly isolate tenant workloads." A namespace is an organisational boundary. A container is not, by itself, a strong security boundary, because containers on a node share the host kernel.

Think of isolation as levels you add as the risk of the workload rises.

Level 1: Namespace isolation. Gives you a scope for RBAC, quotas, network policies and Pod Security labels. On its own it isolates object names, nothing more. Always use it; never rely on it alone.

Level 2: RBAC isolation. Each agent or tenant gets its own service account with only the Kubernetes permissions it needs, usually none. Agent runtimes almost never need to talk to the Kubernetes API. If an agent's job is to operate on Kubernetes, give that capability to a tool behind the gateway, not to the agent's own service account.

Level 3: NetworkPolicy. Start with default deny for ingress and egress in every agent namespace, then allow DNS and the specific gateways the runtime needs.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: tenant-a-agents
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: agent-runtime-egress
  namespace: tenant-a-agents
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/component: agent-runtime
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: platform-gateways
          podSelector:
            matchLabels:
              app.kubernetes.io/name: tool-gateway
      ports:
        - protocol: TCP
          port: 8443

NetworkPolicy alone does not solve agent isolation. It cannot inspect what an allowed connection carries, it cannot express domain names, and it does nothing about a kernel exploit. It is the layer that stops a compromised runtime from wandering across the cluster network.

Level 4: Pod Security Standards and admission policies. Label agent namespaces to enforce the restricted Pod Security Standard: non-root, no privilege escalation, all capabilities dropped, a seccomp profile. Add admission rules for what PSS does not cover: allowed registries, required signatures, required runtime class for execution workloads, no hostPath, mandatory resource limits. ValidatingAdmissionPolicy (GA since Kubernetes 1.30) handles many of these natively with CEL; Kyverno and Gatekeeper cover the rest, including image signature verification. User namespaces, which reached GA in Kubernetes 1.36, are worth enabling with hostUsers: false for runc-based agent pods, since they map root in the container to an unprivileged user on the host.

Level 5: Dedicated node pools. Run agent runtimes, and especially code execution, on nodes that never host control plane components, gateways or anything holding secrets. If a sandbox escapes to the node, the only things on that node are other sandboxes.

Level 6: Taints, tolerations and scheduling isolation. Dedicated pools only work if scheduling enforces them. Taint the untrusted pool so nothing lands there without a toleration, and use node selectors or affinity so untrusted workloads cannot land anywhere else. Enforce the toleration and selector at admission, rather than trusting each manifest to include them.

Level 7: gVisor. gVisor implements a large part of the Linux system call interface in a user-space kernel, so the workload does not talk to the host kernel directly. It is well suited to running model-generated or user-supplied code, with a performance cost that is most visible on syscall-heavy and I/O-heavy workloads. GKE Sandbox is built on it.

Level 8: Kata Containers. Kata Containers runs each pod inside a lightweight virtual machine with its own kernel, giving hardware virtualisation boundaries while keeping the pod interface. It costs more memory per pod and needs nodes that support nested or bare-metal virtualisation, and it is a good fit when you need a VM-grade boundary per execution. AKS Pod Sandboxing uses Kata. Both gVisor and Kata plug in through a RuntimeClass, and the Kubernetes SIG Apps Agent Sandbox project builds a declarative Sandbox API on top of these for isolated, stateful, singleton agent workloads.

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc
scheduling:
  nodeSelector:
    workload-class: untrusted
  tolerations:
    - key: workload-class
      operator: Equal
      value: untrusted
      effect: NoSchedule
---
apiVersion: v1
kind: Pod
metadata:
  name: code-exec-sandbox
  namespace: tenant-a-exec
spec:
  runtimeClassName: gvisor
  automountServiceAccountToken: false
  enableServiceLinks: false
  securityContext:
    runAsNonRoot: true
    runAsUser: 10001
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: sandbox
      image: registry.example.com/agents/sandbox@sha256:<digest>
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop: ["ALL"]
      resources:
        requests:
          cpu: 250m
          memory: 256Mi
        limits:
          cpu: "1"
          memory: 1Gi
          ephemeral-storage: 2Gi
      volumeMounts:
        - name: workspace
          mountPath: /workspace
  volumes:
    - name: workspace
      emptyDir:
        sizeLimit: 1Gi

Notice what is absent: no service account token, no secrets, no environment variables with credentials, and a pinned image digest. The sandbox is designed on the assumption that whatever runs inside it is hostile.

Level 9: Separate clusters. When agents execute untrusted code at scale, serve mutually untrusted customers, or operate in a regulated context, put high-risk execution in its own cluster. A cluster boundary removes shared control plane objects, shared CRDs and cluster-scoped operators from the blast radius.

Level 10: Separate cloud accounts, projects or subscriptions. For the highest-risk workloads, or tenants with contractual isolation requirements, place the cluster in its own account. Now a compromise cannot reach your other accounts' IAM, networks or KMS keys without crossing an explicit trust relationship.

Our recommendation for most teams: levels 1 to 6 for every agent, level 7 or 8 for anything that executes code or browses arbitrary content, and levels 9 to 10 decided by tenant trust and regulatory need rather than by default.

Identity, secrets, and authorization

In a traditional service, identity answers "which service is calling?". With agents that question is insufficient. Every action has at least three principals: the user who initiated the task, the agent acting on their behalf, and the tool executing the operation. Authorization has to consider all of them.

The distinction that matters most:

• "The agent can call the API" is a network and credential fact.
• "The agent is authorized to perform this operation, on this resource, for this user, right now" is a policy decision.

Most agent systems in the wild only establish the first. That is how confused deputy bugs happen.

A useful way to think about the progression:

• Bad: the agent runtime has broad cloud credentials or a powerful API key in an environment variable. Any injection or code execution inherits all of it.
• Better: the agent has permission to call a specific set of tools, and the tools hold the credentials. The agent's reach is limited to those tools.
• Best: each tool call is authorized at the tool gateway against the user's own permissions, the agent's identity and allowed capabilities, the specific operation and resource, and context (tenant, time, data classification, approval state). Only then does the gateway obtain a short-lived credential scoped to that operation and make the call.

That last pattern is capability-based security applied to agents: the agent holds references to narrowly defined capabilities, never ambient authority.

The building blocks for this are mature:

• Kubernetes service accounts give each agent runtime and gateway a distinct workload identity. Use one service account per agent type or per tenant agent, never a shared "agents" account.
• Cloud workload identity (EKS Pod Identity or IRSA, GKE Workload Identity Federation, Entra Workload ID) turns that service account into short-lived cloud credentials. The tool gateway's identity, not the runtime's, should carry cloud permissions.
• SPIFFE and SPIRE provide portable, attested workload identities across clusters and clouds when a mesh or multi-cluster setup needs them.
• HashiCorp Vault or a cloud secrets manager issues dynamic credentials: database users created per lease, cloud credentials with TTLs, and automatic revocation. A leaked credential that expires in fifteen minutes is a very different incident from a leaked static key.
• OAuth 2.0 Token Exchange (RFC 8693) lets a gateway exchange the user's token for a downscoped token for a specific downstream audience, preserving the user as subject and the agent as actor. The MCP authorization specification builds on OAuth 2.1, and its security best practices state that MCP servers MUST NOT accept tokens that were not explicitly issued for them, which rules out the token passthrough shortcut.
• Policy engines such as OPA, Cedar or OpenFGA keep the authorization logic declarative, testable and auditable, rather than scattered across tool implementations.

A simplified tool gateway policy in Rego shows the shape of the decision:

package toolgateway.authz
 
import rego.v1
 
default allow := false
 
allow if {
	tool := data.tools[input.tool]
	input.agent.id in tool.allowed_agents
	input.operation in tool.allowed_operations
	input.tenant == input.resource.tenant
	user_has_permission(input.user, input.operation, input.resource)
	not needs_approval
}
 
needs_approval if {
	input.operation in data.tools[input.tool].approval_required
	not input.approval.granted
}
 
user_has_permission(user, operation, resource) if {
	some grant in data.grants[user.id]
	grant.resource == resource.id
	operation in grant.operations
}

On secrets specifically: do not inject secrets everywhere just because the application eventually needs them. An environment variable is readable by every process in the container, visible to anything that can exec into the pod, and one prompt injection away from appearing in a tool output. Keep secrets in the gateways, deliver them through workload identity and short TTLs, rotate automatically, and scan agent outputs and logs for credential patterns. Our earlier piece on building MCP tools for production goes into token masking, auth patterns and SSRF handling at the individual tool level.

Securing the AI supply chain

The supply chain for a normal service is source, dependencies, base image, build and artifact. For an agent, several more things change behaviour without a code change:

• Source code and dependencies, including fast-moving Python and Node ecosystems where agent frameworks pull in large dependency trees.
• Base images, which should be minimal, pinned by digest and rebuilt regularly.
• Models and model artifacts. Weights downloaded from a hub are code-adjacent: some serialisation formats, notably Python pickle, can execute arbitrary code on load. Prefer formats like safetensors, scan artifacts with tools such as ModelScan, pin exact revisions and mirror approved models internally.
• Datasets and retrieval corpora. Poisoned data in a fine-tuning set or a RAG index changes behaviour. The NSA, CISA and partners' AI Data Security guidance from May 2025 covers provenance, integrity and access controls for this data.
• Prompts and agent configuration. System prompts, tool descriptions and routing rules are behaviour. They belong in version control, with review, not in a database row someone edits in production.
• Tools and MCP servers. A third-party MCP server is a dependency with execution rights. Review it, pin it, and run it behind your own gateway.
• CI/CD and registries, which have to be as protected as production.

The standards to build on:

• SBOMs. SPDX (an ISO/IEC standard) and CycloneDX (an Ecma standard, ECMA-424) are the two established formats. Both now describe AI components: SPDX 3.0 added AI and Dataset profiles, and CycloneDX has supported ML-BOMs since version 1.5. Generate them in the build, attach them to the image as attestations, and actually query them when a new CVE lands.
• Provenance and integrity. SLSA defines levels for build integrity and, since version 1.2 (November 2025), a Source track. Sigstore and Cosign provide keyless signing tied to CI identity, with a public transparency log. The OpenSSF model signing project (v1.0 in April 2025) applies Sigstore-style signing to model artifacts.
• Admission verification. Signatures only matter when something checks them. Kyverno image verification, the Sigstore policy-controller, or a registry-side gate should refuse to run images whose signature or provenance does not match your pipeline identity.
• Secure development for AI. NIST SP 800-218A extends the SSDF with practices specific to generative AI and foundation model development, and is a reasonable checklist for the model-facing parts of your pipeline.

The broader point: an agent's behaviour is a function of code, model, prompts, tools, retrieval sources and configuration. If only the code goes through review, signing and admission, the other five are an unguarded path to production.

Secure CI/CD pipeline for AI agents

The pipeline below extends the standard DevSecOps flow with the checks that are specific to agents.

Where the AI-specific checks belong:

• Prompts, tool manifests and policies in the same repository and review flow as code. A change that adds a tool to an agent or widens its scope is a permissions change and should get the same scrutiny as an IAM policy diff. Many teams require security approval specifically when a tool allowlist changes.
• Policy unit tests. Authorization policies for the tool gateway should have tests: this agent cannot call this operation, this tenant cannot reach that resource. OPA and Cedar both support this well.
• Model and artifact validation before signing: verify the source and exact revision, check hashes against the approved list, reject unsafe serialisation formats, and record the result in the SBOM.
• Adversarial testing in staging. Run a maintained suite of direct and indirect injection cases against the real toolset, with canary secrets and canary resources, and assert that the agent cannot reach them. Treat a regression here like a failing integration test.
• Evals that include security properties. Accuracy evals alone will happily pass an agent that completes tasks by doing things it should not be allowed to do.
• GitOps for deployment. Argo CD or Flux pulling signed manifests means the cluster changes only through reviewed commits, and the CI system does not need cluster admin credentials.

Runtime security

Everything before deployment reduces the likelihood of a problem. Runtime security is how you find out when one gets through anyway.

At the infrastructure layer, eBPF-based tools are the current standard. Falco, a CNCF graduated project, detects suspicious behaviour from kernel events and Kubernetes audit logs using rules. Tetragon, part of the Cilium project, provides eBPF-based observability and can also enforce, for example killing a process that performs a forbidden action. Typical detections worth enabling for agent nodes:

• Shells or interpreters spawned in containers that should not run them
• Unexpected binaries executed, especially ones written at runtime
• Writes to sensitive paths, reads of service account tokens or cloud credential files
• Privilege escalation attempts, capability changes, setuid binaries
• Connections to the instance metadata address or to unexpected external domains
• Crypto mining indicators: known pool domains, sustained CPU with unusual processes
• Container escape indicators such as mounting host filesystems or accessing the container runtime socket

The Kubernetes API audit log covers the control plane side: secrets read by unusual identities, exec into pods, new role bindings, pods created with privileged settings.

None of this sees agent behaviour. From the kernel's point of view, an agent exfiltrating a customer list through an approved API call looks exactly like an agent doing its job. That is why agent runtime monitoring has to happen at the orchestrator, model gateway and tool gateway too:

• Tool calls: which tools, which operations, which resources, how often, and in what order.
• Permission denials and failed tool calls: a spike in denials often means an agent is being steered toward something it should not do.
• Token usage and model calls: sudden growth in context size or call count per task.
• Agent loops: iteration counts approaching limits, repeated identical tool calls.
• Sensitive data access: reads of classified data followed by any outbound action.
• Unusual destinations: fetches to domains never seen before for that agent.
• Abnormal execution chains: a read from an untrusted source immediately followed by a high-impact write is the classic indirect injection pattern, and it is worth an alert on its own.
• High-cost behaviour: per task, per agent, per tenant.

Normal infrastructure monitoring asks "is the system healthy?". Agent runtime monitoring asks "is the system doing what it was asked to do, for the person who asked?". You need both.

Observability for AI agents

Good AI agent observability layers five kinds of signal:

• Infrastructure: CPU, memory, GPU utilisation and memory, node pressure, pod restarts.
• Application: request rate, latency, error rate, queue depth.
• LLM: tokens in and out, time to first token, model and version, cost per request, provider errors and rate limits.
• Agent: iterations per task, tool calls per task, tool failure rate, retrieval events, task completion and abandonment.
• Security: authorization denials, approval requests and outcomes, prompt injection detections, sensitive data events, policy violations, egress blocks.

The main shift is that a single agent task is a tree of work: a user request, several model calls, many tool calls, retrievals, maybe sub-agents, each of which may call other services. Metrics alone cannot tell you why a task did something. You need a trace that follows the whole execution chain with consistent IDs from the API gateway through the policy decision to the downstream service.

OpenTelemetry's GenAI semantic conventions define spans such as invoke_agent, chat and execute_tool, with attributes for model, token usage and tool names. They are still marked as in development, so expect attribute names to shift, but adopting them now is far better than inventing a private schema. Be deliberate about content capture: full prompts and tool outputs in traces are invaluable for debugging and a data protection problem if they contain personal or secret data. Capture metadata by default and sample or redact content.

With this in place, an investigation becomes a query: show every task for tenant B in the last day where an execute_tool span with a write operation followed a retrieval from an external source. Without it, you are grepping application logs from four services and guessing. Our write-up on silent failures in document pipelines makes a related point: the dangerous failures are the ones that report success, and agents are very good at reporting success.

Scaling secure AI agent systems

Scaling agents securely is mostly about making sure controls are properties of the platform rather than of individual deployments. What works for ten agents managed by hand falls apart at a hundred and becomes dangerous at a thousand.

From 10 to 100 agents. Hand-written manifests and per-agent secrets stop scaling. Introduce a platform abstraction: an agent is declared (tools, model access, tenant, risk tier, budgets) and the platform generates the namespace, service account, network policy, quotas and gateway policy entries from that declaration. Security review shifts from reviewing each deployment to reviewing the templates and the declarations.

From 100 to 1,000+ agents. Per-agent policies need to be data, not code. Tool allowlists and capability grants live in the policy engine's data store, versioned and audited. Admission policies enforce that nothing runs outside the platform's templates. Isolation tiers are selected by risk tier automatically.

The scaling mechanics themselves:

• Horizontal Pod Autoscaling and cluster autoscaling (Cluster Autoscaler or Karpenter) for stateless components such as gateways. Scale on meaningful signals like queue depth or in-flight requests rather than CPU alone.
• Queues and asynchronous execution. Long-running agent tasks belong on a queue with workers, not in synchronous HTTP requests. KEDA can scale workers on queue length. Queues also give you natural backpressure and a place to enforce per-tenant fairness.
• Separate node pools for gateways, CPU agent runtimes, sandboxed execution and GPU inference, each scaling independently. GPU nodes should be tainted so only inference workloads land on them.
• Model gateways for rate limiting and routing across providers and self-hosted models, with per-tenant and per-agent token quotas. Our analysis of a year of production LLM serving traffic is a useful reminder that workload shape shifts over months, so capacity limits need revisiting.
• Concurrency control and backpressure. Cap concurrent tasks per tenant and per agent. Reject or queue rather than letting load cascade into downstream services.
• Budgets at every level. Maximum iterations per task, maximum tool calls per task, maximum tokens per task, maximum cost per tenant per day. These are safety controls as much as financial ones: a runaway loop and a successful injection often look the same in the metrics.
• Circuit breakers and retry budgets. Agents retry enthusiastically. Without retry budgets at the tool gateway, one failing downstream API turns into a self-inflicted denial of service.
• Tool execution limits. Timeouts, maximum response sizes and rate limits per tool, enforced at the gateway rather than trusted to the agent.

The security invariant at scale: adding an agent should never require adding a privilege by hand. If it does, the platform has a gap, and that gap will be filled by whoever is in a hurry.

Multi-tenant AI agent platform

If you run agents on behalf of multiple customers or business units, tenant isolation is the requirement everything else serves. A SaaS-style layout typically looks like this:

The dimensions of tenant isolation:

• Identity: tenant is a claim in every token and a required input to every policy decision. Never derive tenant from something the client or the model can set.
• Data: separate memory stores and retrieval indexes per tenant, or at minimum enforced tenant filters applied by the retrieval service, not by the prompt. Poisoned memory in a shared index is a cross-tenant injection channel. Our guides on agent memory in production and permission-aware enterprise RAG cover the data layer in more depth.
• Credentials: per-tenant secret paths and per-tenant credentials for downstream integrations, so a bug cannot use tenant A's Salesforce token for tenant B.
• Namespaces and network: one namespace per tenant (or per tenant and agent class), default-deny policies, and no direct pod-to-pod traffic across tenants.
• Storage: per-tenant persistent volumes and storage classes, encryption with per-tenant keys where contracts require it.
• Model access: per-tenant model allowlists and quotas at the model gateway, and clarity about whether tenant data may be sent to which providers.
• Rate limits and quotas: ResourceQuota and LimitRange in the namespace, plus token, cost and concurrency limits in the gateways, so one tenant cannot starve the rest.
• Observability and audit: tenant labels on every span, metric and log, and tenant-scoped views if customers get access to their own audit trail.

When are namespaces enough? Namespace-based soft multi-tenancy, done properly with every control above, is a reasonable choice when tenants are internal teams or when agents do not execute arbitrary code and hold no direct credentials. Add sandboxed runtimes and dedicated node pools once agents run code. Move to separate clusters (or virtual clusters such as vCluster for control plane separation) when tenants are mutually untrusted external customers running code, and to separate accounts when contracts, regulators or data residency demand it. Many platforms end up with a tiered model: most tenants on shared infrastructure, a few on dedicated clusters.

A practical security reference architecture

Pulling everything together, this is the architecture we would start from for a production agent platform on Kubernetes.

The properties to verify in a design review:

1. The agent runtime holds no long-lived credentials and cannot reach anything except the model and tool gateways.
2. Every tool call passes through a policy decision that includes the user, not just the agent.
3. Nothing reaches the cluster without a verified signature and provenance from your pipeline.
4. Code execution happens in a sandboxed runtime on nodes that host nothing else.
5. A single trace ID connects the user request, every model call, every policy decision and every downstream call.
6. Budgets and limits exist per task, per agent and per tenant, and are enforced outside the agent.
7. Losing any one control (a missed injection, a scanner gap, an escaped sandbox) does not by itself lead to a serious breach.

The DevSecOps checklist for AI agents

Use this as a starting point for a production readiness review. Not every row applies to every agent, but every skipped row should be a conscious decision.

What a mature AI DevSecOps platform looks like

Maturity models are always a simplification, but they help teams see where they are and what the next step is.

Level 1: Basic. Agents run as ordinary deployments. Credentials sit in environment variables or Kubernetes Secrets. Some scanning exists in CI but does not block. No network policies. Agents call tools and APIs directly. Observability is application logs and a model provider dashboard. Security review happens once, before launch.

Level 2: Automated. SAST, SCA, secret, IaC and image scanning run on every change with a blocking policy for critical findings. Images come from a private registry. Workload identity replaces static cloud keys. Pod Security Standards are enforced. Model calls go through a gateway with usage tracking. Basic iteration and token limits exist.

Level 3: Policy-driven. Admission control verifies signatures and provenance. Default-deny network policies everywhere. A tool gateway enforces per-agent tool allowlists. Prompts, tool manifests and policies are versioned and reviewed like code. SBOMs cover models and datasets. Runtime detection runs on all nodes. Security tests for injection and tool abuse run in staging.

Level 4: Zero-trust agent infrastructure. Every tool call is authorized against user, agent, operation, resource and context, in line with the zero trust principles in NIST SP 800-207. No workload holds long-lived credentials. Code execution is sandboxed on dedicated node pools. End-to-end traces link user requests to downstream actions. Budgets exist per task, agent and tenant. Anomaly detection covers agent behaviour, not just infrastructure.

Level 5: Fully governed multi-tenant AI platform. Agents are declared, and the platform generates isolation, identity and policy from the declaration and risk tier. Isolation tiers range from namespaces to dedicated accounts, selected automatically. Tenant-scoped audit trails are available to customers. Risk management maps to frameworks such as the NIST AI RMF with evidence generated continuously. Incidents feed back into policies, detections and adversarial test suites as a matter of routine.

Most organisations we talk to are at level 1 or 2 for agents, even when their traditional services sit at level 3. That gap is the practical risk: the newest, most capable workloads often run with the least mature controls.

AI agents make DevSecOps more important, not less

It is tempting to treat AI agent security as a model problem: better guardrails, better system prompts, better classifiers. Those help, and you should use them. But the incidents that hurt will come from the same places they always have: over-broad credentials, flat networks, unverified artifacts, missing authorization checks and nobody watching what a workload actually does. Agents simply reach all of those faster, and in combinations nobody wrote down.

That is the core of DevSecOps for AI agents. The more autonomy you give an agent, the more you depend on identity, authorization at every tool, isolation matched to risk, a supply chain that covers models, prompts and tools, runtime detection and traces that follow the whole execution chain. The goal is not an agent that can never be manipulated. The goal is a platform where a manipulated agent still cannot do much. CISA and its international partners put it plainly in their May 2026 guidance on careful adoption of agentic AI services: avoid granting broad or unrestricted access, and start with low-risk use cases.

Design for the compromised agent, and the capable agent becomes something you can safely ship.

If you are building AI agents and need help designing the infrastructure underneath them, from the Kubernetes platform and cloud foundations to security controls and compliance and the agent systems themselves, Covaratech does this work end to end, including DevSecOps consulting for teams taking agents into production. We are happy to review an architecture, pressure-test a threat model or help build the platform. Talk to an engineer.

References

1. NIST: Secure Software Development Framework (SP 800-218)
2. NIST: SP 800-218A, Secure Software Development Practices for Generative AI and Dual-Use Foundation Models
3. NIST: SP 800-204D, Strategies for the Integration of Software Supply Chain Security in DevSecOps CI/CD Pipelines
4. NIST: Cybersecurity Framework 2.0
5. NIST: AI Risk Management Framework
6. NIST: AI 600-1, Generative AI Profile
7. NIST: AI 100-2 E2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
8. NIST: SP 800-207, Zero Trust Architecture
9. OWASP: Top 10 for LLM Applications 2025
10. OWASP: Top 10 for Agentic Applications for 2026
11. MITRE: ATLAS, Adversarial Threat Landscape for AI Systems
12. NSA and CISA: Kubernetes Hardening Guidance v1.2
13. NSA, CISA and partners: Deploying AI Systems Securely
14. NSA, CISA and partners: AI Data Security: Best Practices for Securing Data Used to Train and Operate AI Systems
15. CISA and partners: Careful Adoption of Agentic AI Services
16. CIS: Kubernetes Benchmark
17. Kubernetes: Security concepts, Multi-tenancy, Pod Security Standards, RBAC good practices, Network Policies
18. Kubernetes: User Namespaces GA in v1.36 and Validating Admission Policy GA in 1.30
19. Kubernetes SIG Apps: Agent Sandbox
20. gVisor and Kata Containers
21. OpenSSF: SLSA v1.2 specification and Model Signing specification
22. Sigstore, SPDX, CycloneDX
23. Model Context Protocol: Authorization and Security Best Practices
24. IETF: RFC 8693, OAuth 2.0 Token Exchange
25. OpenTelemetry: Semantic conventions for generative AI systems
26. Falco and Tetragon
27. AWS: EKS Pod Identity and EKS identity and access management best practices; Google Cloud: Workload Identity Federation for GKE; Microsoft: Entra Workload ID on AKS

Tool names are examples of categories, not endorsements, and capabilities change between releases. Kubernetes feature maturity is stated as of the versions cited. Check your provider's current documentation before relying on a specific default.