Bedrock Agents in Production: Lessons from the Field

We’ve spent the first half of 2025 helping clients build and operate Bedrock Agents in production. Some of what we expected held up. Quite a bit didn’t.

This post is a candid summary of what we’ve learned: the patterns that work, the failure modes that surprised us, and the operational disciplines you need in place before you turn an agent loose on real users.

What’s actually working

Three classes of agents have been consistently successful:

Retrieval-augmented internal tools. Agents that wrap a knowledge base (internal docs, support tickets, runbooks) and answer employee questions. Low risk, high leverage. The failure mode is “I don’t know,” which is acceptable.

Workflow orchestrators with tight tool definitions. Agents that take a user request and call a small, well-defined set of internal APIs. Onboarding flows, status checks, ticket triage. The structure of the tools constrains the agent’s behavior in useful ways.

Code review and IaC drafting assistants. Less surprising, but worth noting: agents that work alongside engineers on code review and CloudFormation drafting have been our most consistently used internal tooling. Lower stakes than user-facing work, immediate feedback loops.

What’s failed in interesting ways

Three patterns we’ve seen go wrong:

Tool sprawl

The temptation to give an agent twenty tools and let it figure out which to use. In practice, agents handle five to seven tools well; beyond that, behavior degrades, especially in tool selection, where the agent reaches for the wrong tool in confusing ways.

The fix is unsexy: split a single agent into multiple specialized agents with five to seven tools each, orchestrated by a router. Painful to architect, but the reliability improvement is substantial.

Silent prompt drift

We had a client whose agent gradually got worse over a six-week period. No code changes. No model version changes. The cause: their knowledge base had been updated, and the new content had different formatting that confused the retrieval-augmented prompt.

Lesson: agent quality is a function of every input: model, prompt, tools, knowledge base content. You need monitoring across all of them, not just the model version.

Cost spikes from runaway loops

An agent given the ability to call itself recursively (or to call tools that trigger the agent again) can produce expensive failure modes. We’ve seen single user sessions cost hundreds of dollars when a tool returned ambiguous output that caused the agent to retry indefinitely.

Mitigations: hard step limits, per-session cost ceilings, and circuit breakers on tool failure rates.

Operational disciplines you need

Before you put a Bedrock Agent into production, we’d recommend having all of the following in place:

Evaluation pipeline. A repeatable test suite of representative inputs with expected outcomes, runnable against any prompt or model change. Without this you can’t tell if a change improved the system or made it worse.

Cost observability per session. Tag every agent invocation with a session ID and propagate it through tool calls. You need to be able to identify the most expensive sessions and understand why they were expensive.

Tool call audit log. Every tool invocation by every agent, persisted. When a user reports a bad experience, you need to be able to reconstruct what the agent did.

Manual override path. Some way for a human to step in mid-session for high-stakes interactions. Not always relevant, but for anything customer-facing it’s important.

Deployment process with model version pinning. Agent behavior changes with model versions. You need to be able to deploy a model upgrade as a deliberate change with a rollback path, not as an automatic upgrade.

Where we’d go cautiously

We’ve grown more careful about a few patterns:

• Customer-facing autonomous decisions (refunds, account changes, anything with user-visible consequences). Keep humans in the loop.
• Agents that act on production infrastructure (triggering deploys, modifying resources). The blast radius is too high relative to the failure modes we see.
• Long-running agents with persistent state. The operational complexity grows quickly; for now, prefer stateless or short-session designs.

Closing

Bedrock Agents are genuinely useful for a real set of use cases, and they’re maturing fast. But they’re not a “turn it on and walk away” technology. The teams getting real value from them are the ones treating agent operations as a discipline, with eval suites, cost monitoring, and tool audit logs, rather than a feature.

If you’re planning a production Bedrock Agent deployment, we’d be glad to talk through what we’ve learned.