Can AGI Remain Objective?

As discussions around Artificial General Intelligence become increasingly sophisticated, they continue to revolve around familiar questions. Can machines become as intelligent as humans? Can they reason? Can they create? Can they align with human values?

I have gradually become interested in a different question.

Can AGI remain objective?

The choice of the word remain is deliberate.

An intelligent system may initially appear objective because it begins with the same architecture, the same knowledge, and the same design principles. But if it eventually becomes genuinely agentic and begins making decisions whose consequences it must carry into the future, can it continue to interpret and act from a history-independent standpoint?

I suspect the answer is no.

Although I use AGI as the motivating example, the question is ultimately more fundamental than artificial intelligence. It is a question about agency itself.

In my previous essay, I argued that agency is not simply the ability to make decisions. It is the capacity to become the origin of an action by carrying its consequences through time. If that definition is correct, it immediately raises another question. What happens when we apply the same definition, not to humans, but to artificial intelligence?

Before continuing, I should clarify the role of the mathematical expressions that appear in this essay. They are not intended as proofs, nor do they claim that agency has already been reduced to equations. Their purpose is more modest. They simply make the structure of the problem explicit. If agency is ever to become a rigorous theory, we must first understand the architecture of the problem we are trying to explain.

Most discussions of AGI implicitly assume something like the following:

At first glance, this seems entirely reasonable. As intelligence improves, an agent develops increasingly accurate models of reality and therefore makes increasingly effective decisions.

Hidden within this simple chain, however, is a much stronger assumption. It assumes that the decision-maker itself remains fundamentally unchanged. Knowledge improves. Predictions improve. Models become more accurate. Yet the agent is treated as though it remains an objective observer, standing outside its own history.

But what if genuine agency makes that impossible?

If agency means carrying the consequences of one’s own actions, then every meaningful decision changes not only the environment but also the one who will make the next decision.

The simplest recursive description of that idea might be,

where A_t is the agent at time t, C_t represents the constraints encountered, and D_t the decisions taken.

The equation is intentionally simple. Its significance lies in something easy to overlook. The agent appears on both sides. Every decision changes the future decision-maker.

This immediately suggests a simple thought experiment.

Suppose we create two identical AGIs. They begin with the same architecture, the same parameters, the same memories, and the same objectives.

Initially,

Now allow them to operate independently.

They encounter different environments, different constraints, different interactions, and therefore make different decisions. Each decision becomes part of the history that each agent carries.

Over time,

Their divergence is not the result of faulty programming. It is not random noise. It is not hardware variation. It is the natural consequence of accumulated history.

Notice what has changed.

The important difference is not simply that the two agents possess different experiences. It is that neither can return to the historical position from which they began. Every future judgment is now made by a different agent, one shaped by a different trajectory of carried consequences.

The external world may be identical. Their computational capabilities may be identical. Their available knowledge may even be identical. Yet neither now approaches the world from a history-independent standpoint.

This, I believe, is where the discussion about objectivity changes.

Nothing in this argument suggests that an autonomous agent becomes less capable of understanding reality. Quite the opposite. It may become increasingly accurate in its observations, increasingly sophisticated in its models, and increasingly rigorous in its reasoning.

The question is therefore not whether objective models of reality are possible.

The question is whether an objective agent is.

An objective model attempts to describe reality independently of the observer. An objective agent would have to interpret, judge, and act independently of the history through which it has become what it is.

That is a much stronger requirement.

Agency continuously places the agent inside its own history. Every decision becomes part of the standpoint from which future decisions are made. History does not prevent accurate observation. It prevents a history-independent standpoint from which to judge.

Perhaps this is the distinction we have overlooked.

Objective models may become increasingly refined. Objective agency may be structurally impossible.

An instrument can become extraordinarily capable while remaining detached from its own history. An autonomous agent is something different. The moment intelligence begins making irreversible decisions that reshape its own future, it begins constructing a unique trajectory.

Agency creates trajectory.

Trajectory creates a unique historical standpoint.

Over time, that historical standpoint becomes identity.

This also reframes the discussion about AI alignment.

The usual question is whether AGI will continue to share human values. But perhaps a more fundamental question comes first.

Can a genuinely agentic system permanently share anyone’s values?

If every autonomous decision becomes part of the standpoint from which future decisions are made, then values themselves cannot remain entirely detached from that developmental process.

Alignment is no longer a static engineering problem.

It becomes the developmental problem of another historical being.

Perhaps this explains why I have become increasingly skeptical of the phrase objective AGI. The more I think about agency, the more those two words appear to pull in opposite directions.

An intelligence may remain objective precisely because it remains an instrument.

But the moment it becomes genuinely agentic, it begins constructing its own history.

We often speak about AGI as though we are building a more capable tool.

But if genuine agency inevitably creates a unique historical trajectory, then we are no longer talking about a tool.

We are talking about creating another kind of historical being.

One that will gradually develop its own identity, not because it has become irrational, but because it can no longer stand outside the history it has created.

We often ask whether humanity should create a superhuman intelligence.

We rarely ask why we would choose to create another subjective intelligence at all.

If genuine agency inevitably gives rise to a unique historical standpoint, then every successful AGI would gradually become another particular intelligence rather than a universal one.

Perhaps the question is no longer whether we can build such beings.

It is what purpose we believe they are meant to serve.

And perhaps that has been the real question all along.

Not whether an artificial intelligence can become like us, but whether any intelligence that truly possesses agency can ever remain objective.

Manly Beach, Sydney. Photograph taken in September 2019