Bilateral Underdetermination and the Anthropomorphism Trap

Most commentary on the 2016 match between AlphaGo and Lee Sedol focuses on the machine. Move 37 in the second game, a placement so unexpected that Go experts initially dismissed it as an error, is routinely cited as the moment artificial intelligence demonstrated superhuman strategic capability. The subsequent development of AlphaGo Zero, which achieved mastery through self-play without any human data, reinforced the narrative.

But something else happened in that match. Lee Sedol sat across from a human opponent who was not his opponent. The person placing stones on AlphaGo's behalf was a machine operator, a proxy whose body language, facial expressions, and physical responses carried no strategic information whatsoever. In any human match at that level, the opponent's posture after a difficult move, a hesitation, a glance, the tempo and rhythm of play itself, all constitute a bandwidth of non-verbal communication that experienced players read as they read the board. Sedol was denied those signals.

In machine learning, an ablation study investigates a system's performance by systematically removing components to assess their contribution. What the AlphaGo team performed on Sedol was, in effect, an ablation study on the human. The bandwidth of face-to-face interaction was attenuated to zero. Every informal, adaptive, context-sensitive resource that a human player draws upon when reading an opponent was stripped from the environment. Sedol was not simply playing a stronger opponent. He was playing in an environment from which an entire dimension of human capability had been removed.

He won the fourth game anyway.

That fact deserves more attention. Not because it reclaims some level of human performance against machine capability, but because it exposes a gap in the vocabulary available for describing what happened. The existing discussion is organised around a single axis. Either the machine is better at the task, or it is not. The framing risks treating the interaction as a contest between two actors and evaluates the outcome by comparing their performance. What this framing cannot reach is the structural alteration of the environment in which the human was performing. The question is not "how good is the AI?" but "what was taken from the human, and what does it mean that nobody had a name for it?"

The Acumen Problem

The vulnerability Sedol experienced can be mapped more precisely. Consider human-AI interaction along two dimensions (Figure 1): acumen (the capacity to discern, adapt, and exert influence) and the degree to which the AI system presents with human likeness.

When human likeness is low and acumen is high, the interaction is relatively navigable. Clear boundaries make assessment easier. Discernment relies on understanding the AI's limitations rather than projecting familiar human concepts onto machines. When human likeness is high and acumen is also high, the terrain becomes challenging but remains manageable. Recognising genuine patterns of behaviour despite human-like traits is where acumen can thrive.

The danger zone sits at the intersection of high human likeness and low acumen. The human capacity to assess intentions and capabilities falters when dealing with human-like machines. Vulnerability arises from over-trusting appearances and ignoring subtle, uncanny cues. Acumen is undermined precisely when it is misapplied as trust.

This is a structural vulnerability, not an individual failing. As AI systems present with increasing human likeness, the proportion of interactions falling within the danger zone increases regardless of how informed or well-intentioned the participants are.

The Sedol case sits at an interesting angle to this mapping. AlphaGo did not present with high human likeness. It was, conspicuously, a machine. Yet the match environment was structured as though it were a human contest, complete with a proxy opponent, a physical board, the social choreography of competitive play. The human likeness was not in the machine. It was in the protocol of the interaction. Nobody noticed what had been taken from Sedol because the interaction still looked like a Go match between two players. The protocol's surface preserved the appearance of a human contest while its substance had been fundamentally altered.

The vulnerability does not require the AI to resemble a human. It requires the interaction to resemble a human interaction. The protocol carries the anthropomorphism, not the machine.

The Anthropomorphism Trap

That observation points to something structural. When practitioners, researchers, or policymakers describe AI behaviour, the vocabulary they reach for is overwhelmingly indexed to human experience. AI systems are said to understand, learn, decide, reason, hallucinate, align. Each term borrows from human cognition to describe operations that are not cognitive in the human sense. The borrowing is so pervasive that it has become almost invisible.

The anthropomorphism trap names this structural feature. It is not a criticism of any particular vocabulary or framework. It is a diagnosis of a property that belongs to any vocabulary indexed to human experience when that vocabulary is extended to nonhuman actors.

The mechanism is pre-reflective. If orientation shapes perception before deliberate attention begins, then practitioners whose orientation has been formed through work with human social systems will perceive AI behaviour through human social categories before they are aware of doing so. A practitioner trained to read peer recognition, informal networks, and coded language in human organisations will perceive those same patterns in AI-mediated environments. Not because the patterns are there in the same sense, but because the perceptual apparatus has been calibrated to find them. The trap is not a choice. It is a consequence of the vocabulary through which orientation has been formed.

The trap migrates. When a new concept is developed from vocabulary that carries the indexing, the new concept inherits the trap regardless of how carefully it has been constructed. A framework that describes AI behaviour using terms like "sapience" or "sentience" imports the anthropomorphism at the moment of formulation. The vocabulary renews itself. The indexing persists.

This persistence is what makes the trap dangerous rather than merely imprecise. Imprecision can be corrected by refining definitions. The anthropomorphism trap cannot be corrected by refining the vocabulary, because the problem is not in any specific term but in the relationship between the vocabulary and the domain it is being applied to. The vocabulary describes the human side of the coordination adequately and renders the rest invisible.

The Bundling Problem

The philosopher Simon Blackburn, discussing Hume's account of personal identity in Think ¹, offers a formulation that illuminates why this persistence is structural.

Hume observed that when you look inward, you find individual thoughts, experiences, and passions, but no "you" holding them together. The self is nothing but a bundle of its perceptions. Content but no container. Blackburn identifies the standard objection: experiences are parasitic on persons who have them. You cannot have an unowned experience any more than you can have a dent without a surface. In the beginning there is a surface, the surface is changed by becoming dented, and then an abstraction is performed. A noun is extracted, and the dent is discussed as though it were a thing in its own right. But the noun is logically downstream of the adjective. There is no dent without a dented surface. There is no grin without a grinning face.

This applies directly to the conceptual vocabulary used in complexity practice when it addresses AI. Consider a bundle of properties frequently invoked: entangled agency, ongoing underdetermination, distributed sensemaking, narrative capacity, moral reasoning. These properties cohere because they share a common grounding. They are all indexed to human experience. The human actor is the surface in which these properties are dented. Entanglement means entanglement between human meaning-makers. Agency means human agency. Underdetermination means the underdetermination of human complex systems, produced by human meaning-making operating below the threshold of deliberate reflection.

Remove the human surface and the bundle loses its grounding. The properties retain the appearance of coherence, but the thing holding them together is no longer there. In Blackburn's terms, they become like the Cheshire cat's grin persisting after the face has disappeared.

Recent work in economics has formalised a structurally parallel problem. Garicano, Li, and Wu ² argue that the effect of AI on an occupation depends not only on which tasks AI can perform, but on how costly it is to unbundle those tasks from the job. Labour markets price jobs, not tasks. Jobs bundle tasks together, and the critical variable is the coordination cost of breaking the bundle. In a strong bundle, splitting the job destroys enough value that the job survives intact and AI assists within it. In a weak bundle, the cost of splitting is low, AI replaces some tasks, and the human role narrows.

The parallel is not a metaphor. The question Garicano et al pose for labour markets (whether performing a task separately destroys enough value to keep the job intact) is structurally identical to the question the Blackburn critique poses for conceptual vocabularies (whether extending a concept beyond its grounding surface destroys enough meaning to keep the conceptual bundle coherent). In both cases, bundling protects coherence until the cost of maintaining it is exceeded. And in both cases, the moment the bundle breaks, the consequences are discrete rather than incremental.

The anthropomorphism trap, read through this lens, is a bundling problem. The conceptual vocabulary holds together because its properties share a human surface. Extending any property beyond that surface is an act of unbundling. If the property's meaning genuinely depends on the human grounding, the extension fails silently. The term persists, the usage feels natural, and the diagnostic power disappears.

The Diagnostic Shift and the Consent to Opacity

If the anthropomorphism trap is structural, it cannot be escaped by refining the vocabulary that carries it. A different move is required: shifting the unit of analysis entirely.

Rather than asking what an AI system is, the diagnostic question becomes what its coordination structures do. Rather than treating an AI as an agent to be understood (which inevitably invokes the human-indexed vocabulary of intention, understanding and decision), the AI is treated as an environment or substrate whose operational patterns must be discovered and learned. The focus shifts from categorising the actors to diagnosing the coordination between them.

The philosophical warrant for this shift comes from an unexpected direction. Edouard Glissant's concept of the right to opacity holds that the demand for transparency, the insistence that the other be legible in the observer's categories, is the fundamental epistemological error. Glissant developed this principle in the context of creolisation and the politics of cultural identity. Its application here is structural, not analogical.

The anthropomorphism trap is a specific instance of the refusal of opacity. When AI behaviour is described using human social categories, what is being performed is a demand that the machine become legible in the observer's terms. The machine's actual operations are refused. Its opacity is overridden by the projection of familiar categories onto unfamiliar processes. This produces the same consequence Glissant identifies in every other context where opacity is refused: the coordination between the parties becomes invisible to the party that insisted on transparency.

Shifting the unit of analysis from the actor to the coordination is, in Glissant's terms, the consent to opacity. It accepts that the actors are opaque to each other. It does not demand that either become transparent to the other's categories. It diagnoses what emerges from their contact.

This consent reveals three distinct forms of opacity. Actor opacity, in Glissant's original sense, is the irreducible character of each actor, not to be penetrated by the demand for transparency. Asserted transparency is what an AI system performs when it presents output with quantified authority that obscures its own limitations. And coordination opacity is the structural invisibility of the patterns that emerge from contact between actors whose operations are not mutually legible. Coordination opacity is not a property of either actor. It is a property of the coordination itself, and it is the phenomenon that actor-level vocabulary cannot reach.

Bilateral Underdetermination

The concept that names the territory beyond coordination opacity is bilateral underdetermination.

In human-only complex systems, underdetermination has a specific character. The system cannot be "solved" because human meaning-making operates below the threshold of deliberate reflection. Practitioners navigate by attunement, pattern recognition, and situated judgement that resists formalisation. The system is underdetermined not because it is intractable in the computational sense, but because the actors who constitute it are opaque to full rational reconstruction.

The scope, however, is defined by the surface on which it is grounded. The underdetermination is constituted by human meaning-making. The practitioner attunes to the landscape by reading patterns that are, ultimately, patterns of human activity.

When the system includes AI actors, the underdetermination acquires a second source. The AI system's operations are not accessible to the attentive practices that serve the practitioner in human-only contexts. The practitioner cannot attune to what the AI system is doing in the way one attunes to what a person in the landscape needs. Not opaque in the sense of being hidden or encrypted, but opaque in the sense that the perceptual apparatus calibrated for human coordination simply does not reach it. Simultaneously, the AI system cannot access human meaning-making. Each side's operations are underdetermined from the other's perspective. The underdetermination is bilateral.

This is not simply a doubling of the original problem. In human-only systems, the underdetermination is at least constituted by actors who share the same basic kind of opacity. They share the capacity for narrative, for moral reasoning, for the forms of attunement that complexity practice cultivates. In human-AI systems, the two sources of underdetermination are structurally different. The human's opacity is constituted by meaning-making. The machine's opacity is constituted by computational operations that have no analogue in meaning-making.

Garicano, Li, and Wu's model of job bundling provides a structural parallel. In their model, unbundling a job produces a capacity shock. Each surviving worker, freed from the task now handled by AI, floods the market with output on the remaining task, depressing its price and displacing marginal workers. The conceptual analogue is precise. When human-indexed vocabulary is unbundled from its proper scope and applied to AI-mediated coordination, it produces a flood of apparently adequate descriptions that depress the perceived need for new diagnostic vocabulary. Practitioners equipped with familiar categories continue to produce analyses that feel complete. The analyses describe the human side and leave the rest structurally invisible.

This is why the concrete failure modes that recur in human-AI systems prove so resistant to correction. Automation bias (humans deferring to machine judgement even when warned against doing so), performative oversight (human review becoming ritual rather than substantive), and the structural displacement of accountability (individuals absorbing blame for systemic failures they had no meaningful capacity to prevent) are not properties of the human actor or of the AI system. They are properties of the coordination between them. They persist because the vocabulary available for diagnosing them reaches only one side of a bilateral condition.

Concluding

The AlphaGo match stripped from Lee Sedol every informal, adaptive, context-sensitive resource that a human player draws upon when reading an opponent. Nobody named what had been taken. That absence of vocabulary is the symptom this essay has tried to diagnose.

The anthropomorphism trap explains why the vocabulary does not develop naturally from existing frameworks. The bundling problem explains why extending existing concepts does not fill the gap. And bilateral underdetermination names the condition that any adequate vocabulary must eventually reach: a coordination in which both sides are opaque to each other, in structurally different ways, simultaneously.

The ablation was performed on the human, not on the machine. The question for any framework claiming to address human-AI coordination is whether its vocabulary can diagnose what was taken, or only describe what was left.