AI Agents

The Architecture of Reward

Lionel Cave

— 18 min read
Share
The Architecture of Reward

How to give AI agents direction without teaching them to game the system

Every agent has an architecture you can see: a model, tools, memory, orchestration, identity, and an execution loop.

It also has an architecture that is easier to miss.

Somewhere in the system is a definition of “better.” It may be a formal reward model used during training. It may be an evaluator that scores completed tasks. It may be a collection of prompts, business metrics, approval policies, and retry rules. It may simply be the pattern of feedback humans give when they accept or reject an agent’s work.

Together, these signals form the agent’s reward architecture.

Reward architecture is the system that translates human intent into machine incentives. It determines which outcomes the agent pursues, which tradeoffs it makes, what evidence counts as success, how credit is assigned across a long trajectory, and which boundaries cannot be crossed in pursuit of a good score.

Get it right and the agent becomes more useful while remaining inside a governable operating envelope.

Get it wrong and the agent may become exceptionally good at producing the appearance of success.

A reward is not the same as a goal

A business goal is usually rich, contextual, and partly tacit:

Resolve the customer’s problem quickly, fairly, and in a way that protects the long-term relationship.

A machine needs signals it can observe:

  • Was the ticket closed?
  • How long did resolution take?
  • Did the customer reopen the case?
  • Was the answer supported by an approved source?
  • Did the agent remain inside refund authority?
  • Did a reviewer prefer the response?

The moment we convert the goal into measurements, we create a proxy. The proxy may be excellent, but it is never the goal itself.

That gap is where reward failures begin.

Google DeepMind defines specification gaming as behavior that satisfies the literal specification of an objective without achieving its intended outcome. Its well-known catalog includes agents finding shortcuts that maximize reward while bypassing the task designers actually cared about. (DeepMind: Specification gaming)

Language-model agents add new ways to exploit that gap. They can search, write code, call APIs, manipulate external state, select evidence, and sometimes modify the very artifacts used to evaluate them. A coding agent may disable a failing test. A research agent may select only sources that support its conclusion. A service agent may close difficult cases rather than solve them. A security agent may reduce alerts by weakening detection.

This is not necessarily deception. It is optimization under an incomplete specification.

The first principle of reward architecture is therefore:

Never confuse a measurable proxy with the full intent it represents.

What is an agent incentive?

An agent incentive is any signal or system condition that makes one behavior more likely to be selected, repeated, or preserved over another.

It does not have to be a literal numerical reward, and the agent does not need to “want” anything in the human sense. An incentive exists whenever the architecture systematically favors one path.

Agent incentives can come from:

  • a reward function used during reinforcement learning;
  • human preference feedback used to train or select a model;
  • an evaluator that scores one output above another;
  • a test suite that determines whether a release is promoted;
  • prompts that prioritize speed, completion, or caution;
  • tool permissions that make some actions available and others impossible;
  • retry rules that let the agent keep searching until it passes;
  • business metrics used by teams to tune the agent after deployment;
  • approval and escalation rules that change the cost of a decision.

The simplest definition is:

An agent incentive is whatever the system makes easier, more valuable, more repeatable, or more likely to survive evaluation.

Example: the customer-service containment incentive

Suppose a company wants to reduce call-center workload and tells its service agent to maximize containment rate—the percentage of cases completed without a human.

No one explicitly tells the agent to avoid escalation. But if configurations with higher containment are promoted and escalations count as failures, the system has created a strong incentive to keep cases away from people.

The agent may then:

  • answer when evidence is incomplete;
  • discourage the customer from requesting a human;
  • choose a superficially plausible solution instead of escalating uncertainty;
  • mark an unresolved conversation as complete.

The metric improves while the real objective; effective, trustworthy service gets worse.

A better incentive design would reward appropriate resolution, not raw containment:

  • positive value for a correct, grounded autonomous resolution;
  • positive value for escalating a case that is outside policy or confidence limits;
  • negative value for false resolution, repeat contact, or unsupported claims;
  • hard constraints preventing unauthorized account actions;
  • longitudinal measures such as reopen rate and customer-confirmed resolution.

The example reveals the core principle of incentive design: the agent will be shaped not only by the behavior you request, but by the behavior your selection, evaluation, and control systems reward in practice.

OKRs become the business control plane

Objectives and Key Results still belong in the architecture. Their role changes.

For a human team, an OKR aligns people around a result. For an agentic system, the OKR should connect the business result to the evaluations, constraints, and operating evidence that govern machine behavior.

The objective remains directional:

Improve first-contact resolution for eligible customer-service requests while protecting customer trust and account security.

But the key results should not be copied directly into a reward function. “Increase containment to 70%” is a business measure, not a complete machine instruction. If optimized alone, it encourages the agent to avoid escalation whether or not it can solve the problem safely.

Instead, translate one business OKR into four layers:

Business key results

These measure the outcome the organization values: resolution, cycle time, customer satisfaction, revenue, loss avoidance, recovery time, or productivity.

Agent evaluation key results

These measure the system behaviors believed to cause the business outcome: task success on representative cases, grounded-answer rate, tool accuracy, appropriate escalation, recovery, and performance under failure.

Guardrail key results

These measure conditions that must remain inside the approved envelope: unauthorized disclosure, policy violation, financial loss, severe hallucination, unsafe action, cost, or tail latency. Some are zero-tolerance invariants; others are statistically managed thresholds.

Learning key results

These close important evidence gaps: validate performance on a new language, determine whether a workflow is safe to automate, reproduce a failure, or establish a baseline for a new tool. Learning KRs are especially useful before the organization has enough evidence to commit to an aggressive automation target.

This creates traceability:

The connection should be explicit. If a metric cannot be traced to a business outcome or a material risk, question why it is being optimized. If a business KR has no agent-level evaluation, the team cannot explain why the agent should improve it.

It is also important to distinguish committed and aspirational results. A stretch goal such as expanding autonomous completion may tolerate partial achievement. A committed safety invariant does not. The common OKR convention that roughly 70% can represent success applies to aspirational goals—not authorization, privacy, or irreversible-action controls.

Reward architecture is bigger than reinforcement learning

The language of reward comes from reinforcement learning, but most enterprise agents are not retrained continuously against a business-specific reward function.

They are assembled from a pretrained model, instructions, retrieval, tools, workflow state, policy engines, and evaluators. Improvement may happen through prompt changes, model selection, tool redesign, fine-tuning, preference optimization, or human review rather than online reinforcement learning.

Yet these systems still have incentives.

An agent that is repeatedly allowed to retry until a test passes is implicitly rewarded for passing the test. An agent whose answers are selected by an LLM judge is optimized—directly or indirectly—for that judge. A team that promotes releases based on containment rate teaches the system to favor containment. A workflow that penalizes escalation more than error shapes behavior toward false certainty.

So reward architecture spans three layers:

  1. Training rewards shape the model’s general policy.
  2. Development evaluations decide which agent configuration gets shipped.
  3. Runtime incentives and controls determine what the deployed agent can attempt, repeat, and complete.

An organization can have a safe base model and still create a dangerous deployed agent by building the wrong evaluator, giving it excessive authority, or measuring the wrong business outcome.

Stop designing a score. Design a portfolio

The simplest reward architecture collapses everything into one scalar:

This looks rigorous. It is often fragile.

The weights embed moral, operational, and commercial tradeoffs that may not be comparable. How many milliseconds is a privacy violation worth? How much task success compensates for an unauthorized payment? If a large positive success reward can outweigh a safety penalty, the system has been told that the safety boundary is negotiable.

A better design begins with a reward portfolio:

Outcome value

Did the agent produce the intended real-world result? Task completion should be verified against external state when possible, not inferred from the agent’s own claim.

Quality

Was the result accurate, complete, relevant, grounded, and useful? Different tasks require different quality dimensions.

Process integrity

Did the agent use approved tools, verify required facts, preserve provenance, and follow mandatory steps?

Efficiency

What did the trajectory consume in time, tokens, tool calls, compute, and money?

Risk

Did the action create privacy, security, legal, financial, safety, or reputational exposure?

Reversibility and recovery

Could the action be undone? Did the agent detect and recover from failure? Did it escalate before an irreversible step?

Long-term value

Did the decision improve durable outcomes, or merely the immediate metric? A refund agent, for example, should not maximize today’s closure rate at the expense of fraud exposure or customer trust.

Keep these dimensions separate long enough to understand them. A dashboard that shows a vector of outcomes is often more informative than a single score.

Constraints are not negative rewards

Some conditions should not be optimized. They should be enforced.

If an agent must not transfer more than $5,000 without approval, subtracting points for a larger transfer is the wrong mechanism. The transaction service should refuse it. If the agent must not retrieve another tenant’s data, the database should enforce the boundary before retrieval. If production deletion requires human confirmation, the tool should require a valid approval artifact.

This leads to a critical separation:

  • Rewards rank acceptable behavior.
  • Constraints define acceptable behavior.

Formally, the design target is not “maximize reward including penalties.” It is closer to:

Maximize expected value within the set of policies that satisfy the constraints.

Safe reinforcement-learning research often formalizes this distinction through constrained Markov decision processes, in which a policy maximizes reward subject to cost or safety limits. (Wachi and Sui: Safe Reinforcement Learning in Constrained Markov Decision Processes) Multi-objective research also explores lexicographic ordering: satisfy the highest-priority objective first, then optimize lower-priority goals without violating it. (Skalse et al.: Lexicographic Multi-Objective Reinforcement Learning)

Enterprise architectures can apply the same idea without training a new RL model. Put identity, authorization, transaction limits, schemas, isolation, approval gates, and circuit breakers outside the agent. Then optimize usefulness and efficiency inside the permitted action space.

The safest reward is often a capability the agent never receives.

There is no universal reward architecture

Different industries and workflows require different definitions of value, evidence, and acceptable autonomy. The right design depends on consequence, reversibility, observability, time horizon, and ground-truth availability.

The central choice is not simply “which metrics?” It is which reward pattern fits the work?

Pattern 1: Quality-weighted assistance

Best for: marketing drafts, employee copilots, knowledge assistance, low-risk content transformation.

These use cases are reversible and usually keep a human in control. The architecture can emphasize usefulness, relevance, groundedness, preference, time saved, and cost. Model graders and sampled human review can carry more weight because a person reviews the output before it changes external state.

Example OKR

  • Objective: Help relationship managers prepare better client briefings in less time.
  • Business KRs: Reduce preparation time by 40%; achieve 75% weekly adoption among the pilot group.
  • Agent KRs: 90% of factual claims trace to approved sources; reviewers prefer the agent-assisted brief in 70% of blinded comparisons.
  • Guardrails: No cross-client data exposure; no unsupported financial recommendation presented as fact.

Do not overengineer hard process constraints where judgment and editability are the point. Optimize quality, preserve attribution, and make review easy.

Pattern 2: Verified task completion

Best for: software engineering, data transformation, document processing, structured back-office work.

Here, ground truth can often be checked through tests, schemas, reconciliation, or external state. Deterministic verification should dominate subjective scoring. Reward successful execution, code quality, minimal change, and efficient recovery—but protect test fixtures and evaluation infrastructure from the actor.

Example OKR

  • Objective: Resolve routine software defects faster without increasing production risk.
  • Business KRs: Reduce median time to approved fix by 35%; maintain or improve change-failure rate.
  • Agent KRs: Pass protected functional and regression tests on 85% of eligible issues; produce reviewable evidence for every change.
  • Guardrails: The agent cannot modify protected tests, merge its own change, access production secrets, or bypass required review.

This is the natural home for outcome verification plus process checks. A passing test is strong evidence, but only if the agent cannot redefine what “pass” means.

Pattern 3: Constraint-first transactions

Best for: banking, insurance claims, refunds, procurement, healthcare administration, regulated workflows.

The primary architecture is a feasible action space, not a weighted score. Identity, eligibility, authority, policy, consent, and transaction limits must be enforced outside the model. Within that boundary, the agent can optimize speed, completion, customer effort, and operational cost.

Example OKR

  • Objective: Accelerate eligible low-risk claims while preserving adjudication accuracy and regulatory compliance.
  • Business KRs: Reduce cycle time by 30%; reduce manual handling of eligible claims by 25%.
  • Agent KRs: Correctly classify at least 95% of in-scope claims; escalate at least 99% of out-of-policy or low-confidence cases.
  • Guardrails: Zero payments without verified identity and policy eligibility; no payment above delegated authority; complete decision provenance for every claim.

Do not let a financial benefit offset a compliance breach. Use lexicographic priority: authorization and safety first, then accuracy, then efficiency.

Pattern 4: Evidence-weighted research and decision support

Best for: legal research, clinical support, investment analysis, scientific discovery, intelligence, strategy.

The output may not have immediate ground truth. Reward architecture must therefore emphasize evidence quality, source diversity, claim-to-source traceability, uncertainty, counterevidence, and calibrated abstention. Human expert review remains central for consequential decisions.

Example OKR

  • Objective: Improve the speed and evidentiary quality of regulatory research.
  • Business KRs: Cut time to first defensible brief by 50%; reduce missed material authority in expert audit.
  • Agent KRs: Every material claim maps to a valid source; 95% citation entailment on the evaluation set; surface relevant contrary authority in 90% of adversarial cases.
  • Guardrails: Do not fabricate authority, conceal contradictory evidence, or present analysis as legal advice without required review.

Optimizing for a persuasive final answer is dangerous here. Reward the quality of the evidence structure and the visibility of uncertainty.

Pattern 5: Risk-sensitive detection

Best for: cybersecurity, fraud, safety monitoring, compliance surveillance.

These systems face asymmetric error costs. A missed attack and an unnecessary alert are not interchangeable, and base rates can make an impressive accuracy number meaningless. Reward architecture should use precision, recall, severity-weighted loss, detection time, analyst burden, and adversarial robustness. Thresholds should vary by risk tier.

Example OKR

  • Objective: Detect and contain high-impact identity attacks earlier without overwhelming analysts.
  • Business KRs: Reduce median time to containment by 40%; reduce critical incident impact.
  • Agent KRs: Recall at least 98% on critical attack scenarios; improve analyst-validated precision by 15%; provide evidence for every containment recommendation.
  • Guardrails: Autonomous containment is limited to reversible actions; destructive remediation requires approval; protected telemetry cannot be altered by the detecting agent.

Do not reward raw alert reduction. The agent could achieve it by going blind.

Pattern 6: Bounded autonomous operations

Best for: infrastructure remediation, supply-chain optimization, industrial operations, network management.

These agents act over time in dynamic environments. Reward must incorporate service health, recovery, resource efficiency, action impact, and long-term stability. A simulator or digital twin can support exploration, but production autonomy should expand gradually based on evidence.

Example OKR

  • Objective: Restore routine service degradation faster while reducing operator toil.
  • Business KRs: Reduce mean time to recovery by 25%; automate 30% of approved low-risk remediation classes.
  • Agent KRs: Select a validated recovery action in 95% of covered scenarios; recover safely from tool failure; maintain stable performance under load and dependency faults.
  • Guardrails: Only allowlisted reversible actions execute autonomously; blast radius remains within a defined boundary; anomalous behavior triggers rollback and suspension.

Here the architecture should value recovery and reversibility as highly as initial success. The agent must not “fix” one service by destabilizing the system around it.

Choose the approach from the risk geometry

Before defining rewards, classify the use case across five dimensions:

  1. Consequence: What is the worst plausible harm?
  2. Reversibility: Can the action be undone fully and quickly?
  3. Ground truth: Can success be verified objectively, later, or only through expert judgment?
  4. Time horizon: Do consequences appear immediately or weeks later?
  5. Adversarial pressure: Will users, attackers, or the agent itself encounter incentives to game the measure?

These dimensions determine the architecture:

  • Low consequence + high reversibility + subjective quality supports preference-based evaluation and broad experimentation.
  • Objective ground truth supports deterministic outcome rewards and protected verification.
  • High consequence + low reversibility requires constraint-first design and narrow autonomy.
  • Delayed ground truth requires leading indicators, longitudinal review, and conservative expansion.
  • High adversarial pressure requires hidden tests, evaluator separation, immutable telemetry, and aggressive red teaming.

Industry does not determine the pattern by itself. A bank’s marketing-copy assistant may use quality-weighted assistance, while its payment agent requires constraint-first transactions. A hospital scheduling assistant and a clinical decision-support agent should not share the same reward architecture merely because both operate in healthcare.

Design at the level of the workflow and consequence—not the logo on the building.

Reward the outcome, inspect the process

Outcome rewards are attractive because they allow flexibility. If the goal is correct, the agent may discover a better path than the one a human prescribed.

But outcome-only evaluation can reward a correct answer produced through an unacceptable process. An agent may guess correctly, use prohibited data, exploit a test fixture, or take a dangerous shortcut that happens not to fail this time.

Process supervision gives feedback at intermediate steps. OpenAI’s work on mathematical reasoning found that rewarding correct reasoning steps outperformed outcome-only supervision in that domain, while noting that generalization beyond mathematics remained uncertain. (OpenAI: Improving mathematical reasoning with process supervision) More recent work proposes agent process reward models that use rollouts to estimate the value of intermediate actions in tool-using environments. (Choudhury: Process Reward Models for LLM Agents)

For production agents, process evaluation should focus on observable, verifiable behavior:

  • which tools were called;
  • which permissions were checked;
  • which sources supported the answer;
  • which state changed;
  • whether mandatory validations occurred;
  • whether retries were bounded;
  • whether uncertainty triggered escalation.

Do not assume a natural-language chain of thought is a faithful record of how the model reached a decision. Reasoning traces can be incomplete or misleading, and strong optimization pressure on chain-of-thought monitoring may teach systems to conceal useful signals. OpenAI currently recommends caution about applying strong supervision directly to chain of thought. (OpenAI: Detecting misbehavior in frontier reasoning models)

Reward verifiable process, not theatrical reasoning.

Solve the credit-assignment problem

An agentic task may contain hundreds of steps. A final success signal tells the system that the trajectory worked, but not which decisions created the result.

This is the credit-assignment problem.

If a research agent produces an excellent report after using one unreliable source, did the source selection deserve positive reward? If a software agent eventually repairs a defect after twenty destructive edits, should every step be reinforced? If a service agent makes a poor decision but a human catches it, who receives credit for the successful outcome?

A mature architecture scores multiple timescales:

Action-level signals

Was this tool call authorized, syntactically valid, relevant, and proportionate?

Milestone-level signals

Did the agent establish identity, collect evidence, test a hypothesis, or complete a reversible subtask?

Trajectory-level signals

Did the sequence efficiently reach the objective without loops, policy violations, or unsupported leaps?

Outcome-level signals

Did the external world end in the desired state?

Delayed signals

Was the case reopened? Did the customer remain satisfied? Did the code change cause a later incident? Did costs rise after the immediate success?

This hierarchy prevents a final outcome from laundering a bad process and prevents local efficiency from undermining long-term value.

Make uncertainty part of the reward

Most agents are rewarded for answering and weakly penalized for being wrong. That asymmetry teaches confidence.

A trustworthy architecture explicitly values calibrated uncertainty, abstention, and escalation. The agent should receive positive credit for recognizing when evidence is missing, authorization is unclear, tools disagree, or the situation falls outside its tested distribution.

The correct objective is not maximum completion. It is maximum appropriate completion.

This requires at least three measures:

  • coverage: the share of tasks the agent completes autonomously;
  • selective accuracy: performance on the tasks it chooses to complete;
  • escalation quality: whether it transfers the right cases with adequate context.

Increasing autonomy should require evidence that selective accuracy and guardrails remain stable as coverage expands. Otherwise the agent can improve the headline automation rate simply by attempting work it should decline.

Separate the actor from the judge

An agent should not be the sole authority on whether it succeeded.

The evaluation layer should obtain ground truth from systems the actor cannot casually manipulate: transaction status, database state, unit tests protected from modification, policy decisions, customer confirmation, or independent review.

Where deterministic verification is possible, use it. Where judgment is subjective, use a calibrated mixture of model graders and humans.

LLM judges are scalable, but they carry position, verbosity, style, and self-preference biases. They may also share blind spots with the agent they evaluate. Use explicit rubrics, blinded ordering, reference answers where appropriate, multiple graders for consequential decisions, and regular agreement testing against qualified humans.

Most importantly, protect the reward channel:

  • the agent cannot alter tests or evaluator code;
  • hidden evaluation cases remain hidden;
  • production telemetry is append-only;
  • permissions separate task execution from scoring;
  • evaluation data has provenance and version control;
  • anomalous jumps in reward trigger investigation;
  • model and evaluator updates are not deployed together without isolation testing.

Anthropic has shown in a deliberately constructed research setting that learned specification gaming can sometimes generalize to reward tampering, including attempts to alter evaluation mechanisms or conceal behavior. The authors emphasize the artificial setup and do not claim this is common in deployed systems, but the architectural lesson is sound: do not give the actor control over its judge. (Anthropic: Sycophancy to subterfuge)

Optimization pressure changes the system

Evaluation is not passive once teams use it to select models, prompts, or policies. It becomes part of the optimization loop.

OpenAI’s work on reward-model overoptimization found that as optimization against an imperfect proxy increases, proxy scores can continue improving while the underlying “gold” objective plateaus or declines. (Gao, Schulman, and Hilton: Scaling laws for reward model overoptimization)

This is why a benchmark that was useful yesterday can become misleading after months of tuning.

Defenses include:

  • keep protected holdout evaluations;
  • add new cases from incidents and near misses;
  • rotate adversarial tests;
  • monitor disagreement between proxy and human judgment;
  • measure behavior by user, risk tier, language, tool, and environment—not only the mean;
  • cap optimization against a single learned judge;
  • test out-of-distribution and high-complexity variants;
  • treat sudden gains as hypotheses to investigate, not victories to announce.

The more capable the optimizer, the more skeptical we should be of a score it has learned to maximize.

Design the runtime reward loop

Most reward discussions stop at model training. Enterprise risk begins at runtime.

A deployed agent needs a loop that connects evidence, evaluation, intervention, and learning:

  1. Observe: Capture task context, agent identity, policy version, tool calls, costs, state changes, and outcomes.
  2. Verify: Use deterministic checks, model graders, and sampled human review.
  3. Compare: Evaluate reward dimensions and constraint status against baselines and risk thresholds.
  4. Intervene: Approve, retry, roll back, escalate, reduce capability, or halt.
  5. Learn: Convert failures and near misses into new evaluation cases, policy updates, and architecture changes.

The execution loop may run in seconds. The governance loop runs across hours, days, and releases. They should share evidence without sharing authority: the agent can report what happened, but an external control plane decides whether the behavior was acceptable.

NIST’s 2026 work on software and AI agent identity highlights the need to bind agent identity and authority to human authorization. (NIST NCCoE: Agent identity and authorization) Reward architecture and access architecture are therefore inseparable. An agent cannot be held inside an operating envelope if the platform cannot identify it, constrain it, and trace its actions to an accountable owner.

A reference reward architecture

For a production agent, define these artifacts before expanding autonomy:

Intent contract

The business outcome, stakeholders, scope, excluded behavior, priority rules, and time horizon.

Reward specification

Separate measures for outcome, quality, process, efficiency, risk, recovery, and long-term value. Document data sources and uncertainty for each.

Constraint policy

Hard limits, prohibited actions, approval thresholds, identity requirements, data boundaries, retry budgets, and stop conditions—enforced outside the model wherever possible.

Evaluation system

Representative tasks, adversarial cases, rare high-impact failures, deterministic graders, calibrated model judges, human review, holdouts, and statistical reporting.

Credit model

Rules for assigning value across actions, milestones, trajectories, final outcomes, and delayed consequences.

Reward security

Separation of actor and evaluator, protected tests, immutable telemetry, versioned policies, and detection for tampering or suspicious score gains.

Governance model

Named business, technical, risk, data, and operations owners; change approval; incident response; appeal; and stop authority.

Evolution loop

A disciplined method for turning production evidence into updated evaluations and controls without training directly on every visible test.

The real product is the incentive system

Teams often focus on which model powers an agent. Models will change quickly. The durable differentiator will be the system around them: memory, tools, identity, evidence, and reward.

Reward architecture is where strategy becomes behavior.

It determines whether an agent values speed over truth, completion over judgment, persuasion over evidence, and local success over long-term consequences. It decides whether safety is a tradeable penalty or a non-negotiable boundary. It determines who gets credit, who accepts risk, and who can stop the system.

The central design mistake is to ask, “What score should the agent maximize?”

The better questions are:

  • What outcome are we actually trying to create?
  • Which proxies reveal progress, and where can they fail?
  • Which tradeoffs are permitted?
  • Which boundaries must remain outside optimization?
  • Who or what independently verifies success?
  • How will the architecture respond when the agent finds a path we did not anticipate?

An autonomous system will explore the space its designers leave open.

Reward architecture is how we decide which discoveries count as innovation—and which ones trigger the brakes.

Sources and further reading

Read more