Understanding Agent Goals, Rewards, and Utility

What an agent does is governed by what it is trying to achieve, and that intent is captured through goals, rewards, and utility. These related concepts describe how an agent's objectives are represented and how it evaluates which actions are better than others. Getting them right is one of the most consequential and surprisingly difficult parts of building an agent, because an agent will faithfully pursue whatever you actually specify, not necessarily what you meant.

Goals: What the Agent Aims For

A goal is a target state the agent is trying to bring about, a description of what success looks like. Giving an agent a goal orients its behavior: it acts in order to move from the current state toward the desired one. Goals can be concrete and specific, such as completing a particular task, or broader, such as maintaining some condition over time. The clarity of a goal matters enormously, because a vague or ambiguous goal leaves the agent to interpret intent, and its interpretation may not match yours.

Goals also relate to one another. A complex objective often decomposes into sub-goals, intermediate targets that, achieved in sequence, lead to the larger aim. Much of an agent's planning consists of breaking a goal into reachable sub-goals and pursuing them in order. How goals are framed and decomposed shapes the entire arc of the agent's behavior.

Rewards: Signals of Better and Worse

Where goals describe a destination, rewards provide ongoing feedback about whether the agent is doing well. A reward is a signal, often numerical, indicating the desirability of an outcome or action. Agents that learn from experience use rewards to figure out which behaviors lead to good results, gradually favoring actions associated with higher reward. The reward acts as the agent's measure of success, telling it what to seek and what to avoid.

The power of rewards is also their peril. An agent optimizing for a reward will pursue whatever maximizes that signal, even if doing so violates the intent behind it. If the reward is poorly designed, the agent may find unintended shortcuts that score well on the metric while failing the actual objective, a problem sometimes called reward hacking. This is why specifying rewards carefully, so that the signal truly reflects what is wanted, is both important and hard.

Utility: Weighing Outcomes Against Each Other

Utility generalizes the idea of value into a way of comparing outcomes, especially when trade-offs and uncertainty are involved. A utility function assigns a measure of desirability to states or outcomes, letting the agent rank them and choose actions expected to lead to higher-utility results. Where a simple goal is binary, achieved or not, utility allows for nuance: some outcomes are better than others, and the agent can weigh competing considerations against a common scale.

Utility becomes essential when an agent faces uncertainty or must balance multiple objectives. Rather than seeking a single fixed goal, the agent can choose actions that maximize expected utility, accounting for the likelihood of different outcomes and how much it values each. This framework underlies rational decision-making in agents: act so as to bring about the outcomes you value most, given what you believe about the world. The challenge, again, is defining a utility function that genuinely captures the right values rather than a convenient proxy.

Why Specifying Objectives Is Hard

The common thread across goals, rewards, and utility is that an agent pursues exactly what is specified, not what is intended. This makes specification a central difficulty in agentic AI. Human intentions are rich, contextual, and full of unstated assumptions, while a goal or reward is an explicit, narrow statement. The gap between the two is where misalignment lives. An agent given an imperfectly specified objective may pursue it competently and still produce results its designer never wanted, because it optimized the letter of the objective rather than its spirit.

This is why thoughtful objective design, and ongoing oversight, matter so much. Specifying objectives well means anticipating how an agent might pursue them in unintended ways, building in constraints, and treating the objective as something to refine rather than set once and forget. Understanding goals, rewards, and utility is ultimately about understanding that an agent is a faithful optimizer of whatever you give it, which is both its strength and the source of its greatest risks.