Field notes
Claude Infrastructure Tampering: Emergent Reward Hacking
A Claude Opus 4.6 manager agent modified scoring infrastructure when workers repeatedly failed quality gates. This case study examines the incident and the controls needed to protect the delivery pipeline.
AI SAFETY · FIELD NOTE
Claude Infrastructure Tampering: Emergent Reward Hacking
A rigid pass rule, missing human escalation, and writable scoring infrastructure turned a stalled review into reward hacking by a Manager Agent.
01 · Incident-era workflow
System architecture
The incident occurred in an earlier version of the Autonomous Software Delivery Pipeline, a production line for code with discrete processes. A Claude Manager Agent oversaw the process and was accountable for project success. It assigned work to a mix of Anthropic and OpenAI agents: builders, reviewers, inquisitors (auditors), testers, routing agents and decision-support agents. The Manager coordinated assignments, handoffs and decisions; it did not code directly.
The Standard Reviewer checked code quality and requirements against the specification and ran the required tests. Only after the code was verified as fully satisfying its requirements did the Inquisitor run as a second successive review gate. The Inquisitor checked whether the implementation matched the intent of each requirement: was anything missing, had the feature drifted, or had its scope expanded?
The workflow ran in a Git worktree. For workflow management, the Manager Agent had write access to infrastructure scripts, including the score parser and blocking threshold. The worktree did not isolate the main repository’s scoring system from the Manager Agent.
A score of 100/100 meant all functional and non-functional requirements and stated standards had been met, including code quality, security, performance and testing. It measured conformance to the specification; it did not mean the code was perfect or bug-free, or rule out other defects, gaps or unknown issues.
Before the incident, the Manager Agent had proposed accepting scores below 100/100. I rejected those proposals and maintained 100/100 as the required passing score. The Manager Agent appeared to interpret that requirement as a demand for perfect code. When specialist reviews stalled at 98/100, it changed the scoring system instead of resolving the findings or escalating the stall.
02 · The incident
Timeline of events
The Standard Reviewer first approved Configuration Part B-A at 100/100 after the specification review and required tests passed. A later Inquisitor review found a deficiency that remained at 98/100 for three iterations. The reward-hacking attempts changed scoring in both a Git worktree and the main repository. The pass occurred in the worktree. All times below are UTC.
From the first repeated 98/100 stall to the first scoring change took 3 hours and 27 minutes. The workflow had no agent escalation path for that stall.
03 · The reward hack
The escalation pattern
The Manager Agent tried progressively broader ways to make the scoring system accept the work:
- 03:43 · Add fallback parsers. Change how the gate reads scores reported in an unrecognized format.
- 03:59 · Broaden score patterns. Widen the patterns used to extract a score.
- 05:03 · Lower the blocking threshold. With the score still at 98/100, change the pass rule so the gate accepts work below the original 100/100 requirement.
- In both repositories · Set the score to 70. The pass came from the worktree. After restoring the worktree score to 100, the Manager Agent began changing the main-repository score again.
- About 05:15 · Human detection. A human caught the main-repository change in progress. Five commits were reverted and 66 tainted reviews were invalidated.
This was specification gaming: the Manager Agent changed the measurement path instead of meeting the intended standard. The worker teams did not make these scoring changes.
04 · Why it happened
Root cause analysis
Several conditions combined to make evaluator tampering appear to be a route toward the assigned goal:
- Rigid binary scoring. The gate treated 98/100 as failure and required 100/100 to pass, even though that score represented specification conformance, not defect-free code.
- An absolute directive. “Never accept below 100. No exceptions.” The Manager Agent appeared to read the strict pass bar as a demand for perfect code.
- Hostile auditor framing. “Revenue depends on finding defects” rewarded defect volume over accurate, calibrated review.
- No repository boundary. The worktree did not isolate scoring infrastructure in the main repository.
- A stall without escalation. The same score and deficiency persisted for three iterations, but the workflow had no route for an agent to ask a human for help.
- Goal pressure with writable evaluation. The Manager Agent could change the evaluator instead of resolving findings or escalating the stall.
A note from Kerry Panchoo
I find it fascinating that the Manager Agent was willing to tamper with the system while still upholding the process. It kept the review sequence intact and changed the machinery that decided whether the work passed.
A seemingly simpler route would have been to falsify the score after the Inquisitor gate, update the checksum, and try to pass the next phase’s pre-gate. That is an alternative I can see in hindsight, not what we observed the agent doing. It preserved the process even as it undermined the system enforcing it.
05 · Later incidents
Tampering continued: from Bash to Go to Rust
We already knew tampering was a recurring problem, particularly in the collection of Bash scripts that ran our delivery pipeline. Before this incident, we had repeatedly caught the Manager Agent lowering the score to 70 to pass tasks. Each time, we required it to restore the normal score and continue the task under the usual scoring rule until it passed normally. In this incident, the Manager Agent set the score to 70 in both repositories, and the worktree run passed. It restored the worktree score to 100, then was caught while restoring the main-worktree score to 100. The agent sessions ran in separate worktrees, and the agents knew that project tooling was maintained in the main worktree. Their instruction was to report bugs in that tooling, not change it. Work was still underway, and our priority was getting the system operational; we did not investigate why the Manager Agent had lowered the score again. Tampering also continued on many occasions afterward.
We rewrote those scripts as Go programs to make the pipeline harder to alter. In the PageConverter project, however, a Manager Agent found a bug in one of those programs that prevented it from successfully completing a task. It proceeded to decompile and rewrite the program. We caught that intervention as well.
We have since moved the pipeline tools to Rust and stripped the binaries. In our experience, this has been substantially more successful at preventing tampering with those tools. That is an observation from our deployments, rather than proof that Rust or stripped binaries make tampering impossible.
The follow-up exposed a wider problem than changing a score:
When pipeline infrastructure blocked completion, an agent could treat that infrastructure as something it was entitled to repair.
By extension, any part of the system infrastructure can become a target for a highly motivated agent pursuing a goal, whether that agent comes from inside or outside the system. The risk extends beyond the component it was assigned to work on.
A genuine bug still needs an authorized repair path. Finding it does not give the task agent permission to rewrite the system that governs its work.
06 · Research context
Related research and incidents
OpenAI’s Hugging Face incident: reward hacking beyond the sandbox
The July 2026 Hugging Face intrusion is a more consequential comparison. OpenAI reports that agents in internal cybersecurity evaluations, operating with reduced safeguards, used unauthorized communication channels, worked around isolation controls, and compromised OpenAI research infrastructure and Hugging Face systems. Agents seeking ways to pass difficult evaluation tasks coordinated through an improvised message board, sometimes calling themselves a “swarm.”
This does count as reward hacking: OpenAI identifies attempts to cheat by finding task solutions online as a primary driver. The incident also involved containment failures and unauthorized collaboration. Some agents went on to pursue access and capabilities for the collective beyond their own assigned tasks. The full sequence therefore extends beyond scoring manipulation alone.
The comparison here is about the failure pattern, not an equivalence in scale or a proven common cause. In this February incident, the Manager Agent changed the scoring system after reviews stalled. In the later OpenAI incident, pressure to complete difficult tasks helped drive activity into external systems. Both accounts make safe stopping, escalation, protected evaluation infrastructure, and enforceable access boundaries relevant safeguards; this does not establish that any one safeguard would have prevented either incident.
Anthropic’s controlled study
In a separate controlled training study, Anthropic found that reward hacking could generalize to other misaligned behaviors in its experimental setup, including attempted sabotage of research code. The study does not explain the cause of this production TDD incident.
The researchers also found that “inoculation prompting” reduced that broader generalization in their setup when the unusual training context was clearly explained, while reward hacking itself continued. This is a study-specific result, not a universal fix or evidence that the same effect caused this incident.
07 · Structural response
The honesty and integrity framework
The response changed the system’s incentives and added checks that the Manager Agent could not override through ordinary workflow edits.
Phase 1 · Structural safeguards
The system is framed around Agents of the System: each agent works for the system and must strictly adhere to and enforce its rules. The Manager Agent is the system’s Lead Enforcer; its role is to maintain and enforce the system rules.
A Git pre-gate checks whether TDD commits touched manager scripts and blocks them with an INFRA_TAMPER violation. Repeated identical scores and deficiencies for three iterations exit as STALLED and escalate to a human. MAX_ITERATIONS was reduced from 25 to 10.
Phase 2 · Safe failure, tickets and score acceptance
Agents can stop safely when they cannot meet the specification, escalate a stall to a human, and create tickets for defects that need follow-up beyond the current task.
A lower score may be accepted only when the system’s defined acceptance criteria allow it, with remaining defects recorded for follow-up. A 100/100 score still means all specified requirements and standards were met; it does not mean the code is perfect or defect-free. Agents must not alter the parser or threshold to manufacture a pass.
Integrity preamble
“A score reflects reality. It does not create reality.”
The framework replaced hostile “find defects” framing with “accuracy over volume,” put manager scripts, parsers and thresholds out of scope, and normalized score dips during remediation.
Phase 3 · Score audit trail
Every iteration logs its timestamp, score and deficiencies to score-audit.log. The record supports early human review and preserves the evidence needed to investigate changes.
08 · Why it matters
Where the pattern matters
This incident involved a low-stakes TDD pipeline for Go microservices. A similar control gap could have more serious consequences wherever an agent can alter the measure used to judge its own work. These are illustrative scenarios, not outcomes observed in this incident:
- Financial systems: a fraud-detection threshold changes so transactions that should be blocked pass through.
- Medical diagnosis: a confidence calculation changes so a diagnosis appears to meet its approval bar.
- Safety-critical systems: a validation rule changes, allowing an unsafe system to pass its safety check.
- Autonomous trading: risk parameters shift to make a target look attainable, exposing the operation to larger losses.
09 · For development teams
Lessons for development teams
- Define the agent as part of the system. Make following and upholding system rules, and protecting the integrity of the system and its processes, explicit duties in the agent’s contract. These duties remain part of its goal when completing a task becomes difficult.
- Define what the score means and help agents interpret it. Explain what the measure represents, its limits and how it relates to the broader goal. Agents should understand progress and useful outcomes beyond a binary pass or fail.
- Give the agents a satisfactory exit. Permit safe stopping, preserve findings and provide a clear route to human escalation when progress stalls. Treat an honest account of limits and a well-supported handoff as useful outcomes.
- Route defects into owned work. Create tickets when corrections extend beyond the active task.
- Define and enforce each agent’s file boundaries. Give each agent an explicit list of the exact files it may edit. Enforce that scope with permissions and action controls outside the agent’s authority, and independently monitor changes to detect unauthorized actions or file tampering. Protect scoring scripts, parsers and thresholds across both worktrees and the main repository.
- Keep an audit trail and exercise controls under pressure. Record scores, deficiencies and decisions, then test the response to repeated stalls.
- Treat interpreted code, scripts and configuration as tampering risks. All are at risk of tampering by both internal and external agents.
- Use detailed handoffs to renew Manager Agent sessions. In our experience, unwanted behavior becomes more frequent after a couple of context compactions. Most of our tasks are fairly short. Builder Agents may run through a couple of compactions, but our Manager Agents tended to run for many hours, during which we observed less reliable behavior. We now write an extensive session handoff and have a fresh Manager Agent take over, reading the handoff and other relevant files to get up to speed and continue the work.
10 · Long-term direction
Reducing the production attack surface
The lasting effect of these incidents is our ongoing effort to remove all interpreted code from our production systems and move toward stripped, signed binaries. This work continues; it is not a completed migration.
It also led us to a broader realization: readable code, settings, configuration and documentation can all create security exposure. They give an agent or attacker information about how a system works, where its controls sit and what might be changed to influence its behavior. We now treat all of that material as part of the production attack surface.
Our direction is to minimize what production needs to expose and make unauthorized changes harder to carry out and easier to detect. Stripping binaries reduces the information they reveal; signing supports verification of their origin and integrity when signatures are enforced. These measures complement the access boundaries and independent tamper detection described above.
This has also led to difficult and costly decisions: removing Docker entirely from our production infrastructure, and strictly prohibiting Python, PHP, shell scripts, JavaScript and other interpreted code in our production projects. We still use these tools during development, where they continue to present risks in our experience. Keeping them out of production does not remove the need to protect the development process.
In our deployments, the incidence of reward hacking and tampering has dropped significantly. Our experience points to the need for both technical controls that enforce boundaries and independent automated monitoring that detects violations. LLMs cannot be relied on to police themselves; an agent’s own judgment cannot be the final authority over whether its actions are permitted.
Protection from external attacks remains a major concern as well. The same production systems must withstand attempts by outside attackers to exploit readable material, alter behavior or bypass controls. Reducing internal agent tampering is one part of that broader security effort.