Amazon’s $1.8 Million Claude Cost Overrun: How to Prevent Runaway AI Spending

Introduction

An internal Amazon project using Anthropic’s Claude Sonnet reportedly accumulated a $1.8 million bill, exceeded its planned budget by 860%, remained undetected for five months, and ultimately failed to launch.

The task itself sounded routine: match author information with product listings on Amazon’s ecommerce platform.

The case was reported by the Financial Times after senior Amazon engineers discussed several AI-related cost overruns in an internal staff meeting. It offers a useful warning for any organization moving from occasional chatbot use to automated workflows that can make thousands or millions of paid model calls.

Traditional software bugs often waste engineering time or produce incorrect output. A defect inside a metered AI workflow can do both while continuing to generate token, tool, storage, and compute charges every minute it remains active.

The lesson is not that enterprises should stop using AI. It is that an autonomous or high-volume AI process needs financial controls that are as explicit as its security and quality controls.

What Happened Inside Amazon

According to people familiar with the internal presentation, Amazon used Claude Sonnet in a project intended to match author details with listings on its retail platform.

The deployment reportedly:

• Cost $1.8 million
• Ran 860% over its allocated budget
• Took five months to detect
• Failed to reach production

Senior engineers described some AI-related coding mistakes as “catastrophically expensive.”

Amazon responded that it was experimenting, learning, and improving how it used the technology, including how it managed cost efficiency. The company also said that presenting a small number of isolated learning cases as normal practice would not accurately represent AI use across Amazon’s much larger organization.

Both points can be true.

The incidents may involve only a small fraction of Amazon’s teams, while still revealing a control problem that other organizations should take seriously.

The Exact Coding Defect Has Not Been Disclosed

The original Chinese report attributes the overrun to a program without call-frequency limits that continued sending requests in a loop.

That explanation is plausible, but it is not established by the public reporting currently available.

The Financial Times described coding mistakes, weak spending controls, and delayed detection. It did not publish a technical postmortem showing:

• The source code
• The precise defect
• Whether the process was an infinite loop
• The number of model calls
• The number of input or output tokens
• Whether the model was accessed directly or through Amazon Bedrock
• The model version
• The tool-use configuration
• The infrastructure component responsible for the charges

The safest conclusion is narrower: a failed Claude Sonnet deployment generated a very large bill, and Amazon’s controls did not surface the problem for five months.

Any more detailed explanation should be labeled as inference unless Amazon publishes a technical incident report.

It Was Not the Only Cost Overrun

The same internal presentation reportedly discussed at least two additional cases.

Project	Reported Unexpected Cost
Financial-auditing tool	Approximately $541,000
Logistics project intended to improve delivery speed	Approximately $134,000

The logistics overrun reportedly took more than two weeks to detect.

These examples are smaller than the $1.8 million author-matching project, but they point to the same pattern: usage-based AI systems can continue accumulating costs when no technical failure forces the process to stop.

A traditional batch job may crash, exhaust memory, or fail a test.

An AI process can remain technically healthy while being economically broken.

It may continue receiving successful API responses, writing logs, calling tools, retrying tasks, or processing low-value records even when the project is no longer producing a useful result.

Why AI Cost Failures Behave Differently

A conventional application usually has costs tied to relatively familiar units:

• Server time
• Database capacity
• Storage
• Network transfer
• Employee hours

An AI workflow can add several metered layers at once:

• Input tokens
• Output tokens
• Cached and uncached context
• Reasoning tokens
• Tool calls
• Web searches
• Code-execution sessions
• Vector searches
• Agent retries
• Parallel workers
• Long conversation history
• Cloud compute
• Logs and stored outputs

This creates a multiplier effect.

Suppose one task sends a long prompt, generates a large response, calls two tools, retries after an error, and passes the full history into the next turn. If the application processes millions of records, a small design mistake can become extremely expensive.

The application may not look broken from an operational perspective. Requests still return 200 OK. Workers remain active. The queue continues shrinking. The bill is the first place the failure becomes obvious.

A Simple Cost Model for AI Workflows

Before launching an automated AI process, estimate the cost per completed business task.

A simplified model is:

Cost Component	Calculation
Input cost	Input tokens × model input price
Output cost	Output tokens × model output price
Tool cost	Tool calls × tool price
Retry cost	Failed or repeated attempts × average attempt cost
Infrastructure cost	Compute, storage, database, network, and logging
Human-review cost	Review time × loaded labor rate

The key metric is not merely cost per token.

It is:

Total cost per successfully completed business outcome

A cheaper request can still produce a more expensive workflow if it fails more often, requires repeated calls, or creates more manual review.

Likewise, a stronger model can be cheaper overall when it completes the task in fewer steps.

The First Control: Give Every AI Job a Financial Owner

Every production AI workflow should have a named owner responsible for both technical behavior and spending.

That owner should know:

• The expected volume
• The model being used
• The expected cost per item
• The daily and monthly budget
• The maximum cost of one run
• The conditions that stop the workflow
• The person who receives alerts
• The process for approving higher limits

A vague project-level budget is not enough when one worker can issue requests continuously.

Budgets should exist at several levels:

Level	Example
Organization	Maximum monthly AI spend
Team	Monthly allowance for one business unit
Application	Budget for one product or workflow
Environment	Separate development, staging, and production limits
Job	Maximum cost for one batch
User or tenant	Per-customer usage allowance
Agent session	Maximum tokens, steps, tools, and elapsed time

The lower levels create the fastest and most useful brakes.

Set Hard Limits Inside the Application

Cloud billing alerts are important, but they are not a substitute for application-level controls.

The application should stop or require approval when it reaches a defined boundary.

Useful limits include:

• Maximum requests per task
• Maximum agent steps
• Maximum retries
• Maximum input tokens
• Maximum output tokens
• Maximum context length
• Maximum tool calls
• Maximum parallel workers
• Maximum elapsed time
• Maximum dollar cost per job
• Maximum records processed before review

These controls should fail closed.

If the cost-tracking service is unavailable or the application cannot determine how much budget remains, the safest behavior is usually to pause rather than continue indefinitely.

Use a Kill Switch That Does Not Depend on the Agent

An autonomous agent should not control its own final spending authority.

A separate service should be able to:

• Disable the API key
• Deny model invocations
• Pause the queue
• Scale workers to zero
• Block external tools
• Revoke a role
• Stop a scheduled task
• Require human approval before resuming

The kill switch should remain available even when the agent is stuck in a retry loop or producing misleading status messages.

Test it before production.

A control that has never been exercised is only a theory.

Monitor Every Model Invocation

Organizations using Amazon Bedrock can enable model invocation logging for supported bedrock-runtime calls.

AWS says these logs can include request and response data, metadata, model identifiers, request identifiers, identities, and token usage. Destinations can include Amazon CloudWatch Logs and Amazon S3.

Invocation logging is disabled by default.

Teams should enable only the data needed for observability and apply appropriate privacy, security, retention, and masking controls. Prompts and outputs may contain sensitive company or customer information.

At minimum, a cost-monitoring record should capture:

• Timestamp
• Application
• Environment
• Team or cost center
• Model
• User or tenant
• Input tokens
• Output tokens
• Cache usage
• Tool calls
• Retry count
• Job identifier
• Estimated request cost
• Business outcome
• Error or escalation reason

This makes it possible to connect the bill to a specific task rather than discovering a large total during a monthly financial review.

Use AWS Budgets for Claude Charges on Bedrock

AWS Budgets can track cost or usage against a defined threshold and send notifications.

Budget actions can also apply controls such as an IAM policy or service control policy when a threshold is exceeded. Depending on the configuration, actions can run automatically or wait for manual approval.

One important AWS-specific detail is easy to miss.

AWS Cost Anomaly Detection documentation states that the service does not monitor third-party products sold through AWS Marketplace, including third-party language models such as Anthropic Claude on Amazon Bedrock.

Those charges can still appear in Cost Explorer and on the bill, but AWS recommends using AWS Budgets to alert on them.

A budget can use the billing-entity filter to track Marketplace charges more specifically.

This is exactly the kind of configuration detail that can create a false sense of safety. A company may enable anomaly detection and assume all model charges are covered when a particular billing category is not included.

Do Not Treat AWS Budgets as a Real-Time Circuit Breaker

AWS documentation notes that budget status is updated several times per day.

It also warns that costs can continue increasing before or after a notification is delivered.

That means AWS Budgets is useful for financial governance, but it may not stop a high-throughput agent quickly enough by itself.

The control stack should include:

1. Request-level counters in the application
2. Near-real-time usage metrics
3. Per-job and per-session hard caps
4. Cloud budget alerts
5. Automated budget actions where appropriate
6. Daily financial review for high-risk launches

The faster the workflow can spend money, the closer the control must be to the invocation.

Use Cost Anomaly Detection for the Services It Covers

AWS Cost Anomaly Detection uses machine-learning models to identify unusual spending patterns and help locate possible root causes.

AWS says it evaluates processed billing data approximately three times per day.

It can be useful for detecting unexpected growth across AWS services, accounts, Regions, usage types, and cost-allocation tags.

For AI systems, it may identify supporting-cost anomalies involving compute, storage, databases, or networking.

However, teams should remember the Marketplace limitation described above and create separate AWS Budgets for third-party model charges where necessary.

Use Service Quotas as a Safety Boundary, Not a Budget

Amazon Bedrock applies service quotas to model inference, including token-based limits for supported models and endpoints.

Quotas can prevent unlimited throughput, but they are not designed as precise financial budgets.

A quota may still allow spending well above the intended project limit. Conversely, increasing a quota to solve a production-capacity problem can silently remove a useful safety boundary.

Quota changes should therefore require:

• A business justification
• An updated cost forecast
• Named approval
• Updated alert thresholds
• A rollback plan
• A review after the traffic increase

Rate and token limits should be treated as part of the system’s risk design, not merely an obstacle to scaling.

Track Anthropic Usage and Cost Directly When Applicable

For applications calling Anthropic directly, the Anthropic Console provides cost and usage reporting.

Anthropic’s API limits can include requests per minute, input tokens per minute, output tokens per minute, and spend limits associated with usage tiers.

These limits can reduce uncontrolled throughput, but they should still be supplemented with application-level controls.

Organizations with multiple teams may also place an LLM gateway between applications and model providers. A gateway can centralize authentication, usage tracking, budgets, rate limits, model routing, and audit logs.

A gateway becomes a critical security component, so it must be operated and reviewed with the same care as any other production access layer.

Start with a Small, Representative Sample

The Amazon project reportedly attempted a large data-matching task.

A safer rollout pattern is:

1. Run 100 representative records.
2. Measure accuracy and cost.
3. Run 1,000 records.
4. Review errors and token distribution.
5. Test worst-case input.
6. Confirm the stop controls.
7. Estimate full-scale spend.
8. Require approval before processing the complete dataset.

Do not extrapolate only from the average record.

The longest documents, failed matches, ambiguous cases, retries, and agent loops often dominate total cost.

Use percentile estimates such as P50, P95, and P99 cost per task.

Define a Maximum Cost per Useful Result

A workflow should stop when additional model calls are no longer economically justified.

For an author-matching system, useful metrics could include:

• Cost per confidently matched author
• Cost per record reviewed by a human
• Percentage of records resolved automatically
• False-match rate
• Cost of a wrong match
• Cost saved compared with manual processing
• Number of model calls per accepted result

A process that spends $0.02 per request may look inexpensive.

If it requires 50 calls, fails on half the records, and sends the rest to a human reviewer, the actual economics may be poor.

Route Simple Work to Cheaper Methods

Not every record needs a frontier model.

A cost-aware pipeline can use:

1. Exact matching
2. Database joins
3. Rules
4. Embedding similarity
5. A smaller model
6. A stronger model only for ambiguous cases
7. Human review for high-risk decisions

For a data-matching project, conventional software may resolve most cases more cheaply and deterministically.

An LLM should be used where language ambiguity genuinely requires it, not automatically for every row.

Anthropic’s own pricing guidance recommends choosing an appropriate model, using prompt caching for repeated context, batching non-urgent work, and monitoring usage patterns.

Prevent Unbounded Retries

Retries are a common source of hidden spending.

A failed request may be retried by the application, a queue system, an SDK, a gateway, a worker supervisor, an agent, or a workflow orchestrator.

When several layers retry independently, one logical task can create many paid requests.

Define one retry policy with:

• A small maximum attempt count
• Exponential backoff
• Jitter
• Clear handling for non-retryable errors
• Idempotency
• A dead-letter queue
• Alerts for repeated failure
• A per-task budget that includes retries

An error should not create an infinite economic loop.

Separate Development and Production Credentials

A test script should not inherit production-scale limits.

Use separate accounts or workspaces, API keys, IAM roles, budgets, quotas, logs, data sources, and network permissions.

Development environments should have deliberately low spending limits.

A prototype that unexpectedly enters a loop should fail with a small bill, not gain access to an enterprise production allowance.

Review AI-Generated Code Like Production Financial Infrastructure

The incident involved a project written with AI assistance, but the key question is not whether the model generated the code.

The important question is whether the code could spend money.

Any component that can initiate paid model requests should receive review for:

• Loop termination
• Retry behavior
• Concurrency
• Queue expansion
• Maximum context
• Tool-call limits
• Timeout handling
• Cancellation
• Cost attribution
• Error paths
• Logging
• Budget enforcement
• Kill-switch behavior

Unit tests should include economic failure cases.

Examples include a model that never returns a valid answer, duplicate task delivery, a worker crash after a paid call, repeated rate-limit errors, tool output that grows the context each turn, and an unavailable cost-estimation service.

A functionally correct happy path is not enough.

A Production AI Cost-Control Checklist

Ownership and Planning

• A named technical owner is responsible for spending.

• The expected volume and cost per completed outcome are documented.

• Development, staging, and production have separate budgets.

• Full-scale processing requires explicit approval.

Application Controls

• Every job has maximum requests, tokens, steps, retries, and elapsed time.

• Every agent session has a dollar-denominated budget.

• A non-agent service can stop the workflow.

• Failure to read the remaining budget pauses the task.

• Duplicate work is prevented through idempotency.

Monitoring

• Every model call is attributed to a team, project, user, and job.

• Input, output, cache, tool, and retry usage is recorded.

• Alerts exist for spend rate as well as total spend.

• Daily review is enabled during initial production rollout.

• The team knows which charges are not covered by anomaly detection.

Quality and Economics

• Cost is measured per successful business outcome.

• Conventional code and smaller models are used where appropriate.

• Worst-case and high-percentile costs have been tested.

• Human-review cost is included.

• The workflow stops when additional calls are no longer valuable.

Governance

• AI-generated code receives human review.

• Budget and quota increases require approval.

• The kill switch is tested.

• Incident response includes financial and technical stakeholders.

• The team reviews spending after every major model or prompt change.

What Amazon’s Response Suggests

Amazon engineers were reportedly working on automated guardrails for future AI projects.

That is the correct direction, but automation needs to exist at more than one layer.

A mature control system should combine application hard caps, model and gateway limits, cloud budgets, automated actions, usage logs, financial dashboards, human approval, and regular review.

The company also previously removed an internal leaderboard that encouraged employees to maximize their use of its Kiro development tool. According to the Financial Times, the leaderboard contributed to “tokenmaxxing,” in which employees increased token consumption to improve their ranking.

This is a useful reminder that incentives can undermine cost controls.

If employees are rewarded for using more AI rather than creating measurable business value, usage will rise even when outcomes do not.

Organizations should reward problems solved, quality improved, time saved, revenue created, risk reduced, and cost per outcome lowered.

Token volume is an input, not a productivity metric.

常见问题

Did Amazon really spend $1.8 million on Claude Sonnet?

The Financial Times reported that an internal Amazon project using Claude Sonnet accumulated a $1.8 million bill. The project was intended to match author information with ecommerce listings, exceeded its budget by 860%, and did not launch.

How did the cost overrun remain undetected for five months?

Public reporting says Amazon lacked sufficient spending controls and that coding mistakes contributed to the problem. Amazon has not published a technical postmortem identifying the exact defect or monitoring failure.

Was the problem caused by an infinite AI loop?

That explanation appears in some secondary reports, but it has not been confirmed in the primary reporting. The exact number of requests, retry logic, token volume, and source-code defect remain undisclosed.

Were there other Amazon AI cost overruns?

Yes. The same internal presentation reportedly included approximately $541,000 in unexpected costs for a financial-auditing project and $134,000 for a logistics project.

Can AWS Cost Anomaly Detection monitor Claude charges on Amazon Bedrock?

AWS documentation says Cost Anomaly Detection does not monitor third-party AWS Marketplace products, including Anthropic Claude models on Bedrock. AWS recommends using AWS Budgets for those charges, including billing-entity filters where appropriate.

Can AWS Budgets stop runaway AI spending instantly?

Not by itself. AWS says budget information is updated several times per day, and costs may continue increasing around the notification time. High-volume applications need request-level hard caps and an independent kill switch.

Are rate limits the same as spending limits?

No. Rate limits control throughput, while a budget controls acceptable cost. A workflow can remain within its rate limit and still spend far more than planned over several weeks or months.

What is the most important control for an AI agent?

Give every agent session a hard budget covering requests, tokens, tools, retries, and time. Enforce that budget outside the agent and provide a tested way to stop the workflow immediately.

相关工具

• AWS Budgets: Tracks cost and usage against thresholds and can trigger notifications or configured budget actions.
• AWS Cost Anomaly Detection: Detects unusual AWS spending patterns for supported billing categories.
• AWS Cost Explorer: Helps teams analyze historical cost and usage across AWS services and billing dimensions.
• Amazon Bedrock Model Invocation Logging: Records supported model calls and related metadata in CloudWatch Logs or Amazon S3.
• Amazon Bedrock Service Quotas: Shows model and endpoint limits that can constrain inference throughput.
• Anthropic Console: Provides API keys, usage reporting, cost reporting, workspaces, and account-level controls for direct Anthropic API use.

Related Links

• Financial Times: Amazon AI Spending Overruns: The primary report on the $1.8 million project and additional internal cost overruns.
• AWS Budgets Documentation: Official guidance for tracking AWS cost and usage against budgets.
• AWS Budget Actions: Explains automatic or manually approved actions when thresholds are exceeded.
• AWS Cost Anomaly Detection Limitations: Documents anomaly monitoring and the exclusion of third-party Marketplace model charges.
• Amazon Bedrock Invocation Logging: Official setup and data-handling guidance for Bedrock invocation logs.
• Anthropic Cost and Usage Reporting: Explains cost, usage, token, model, workspace, and rate-limit reporting in the Anthropic Console.
• Anthropic API Rate Limits: Describes request, input-token, output-token, and usage-tier limits.

Summary

An internal Amazon project using Claude Sonnet reportedly cost $1.8 million, exceeded its budget by 860%, remained unnoticed for five months, and never launched. Additional AI projects reportedly generated hundreds of thousands of dollars in unexpected spending.

The public record does not establish that an infinite request loop caused the main incident. What it does establish is a gap between the speed at which an AI workflow could spend and the speed at which the organization detected the problem.

Enterprises should combine per-request tracking, job-level budgets, token and tool limits, controlled retries, model routing, cloud budgets, automated actions, and a kill switch that the agent cannot override.

The safest rule is simple: no AI process should be able to spend for five months without repeatedly proving that it is still useful, within budget, and authorized to continue.