Adaptive Systems and the Illusion of Control
There is a growing assumption in technology circles that if we can just capture enough context, enough telemetry, enough workflow and enough behavioural data, uncertainty itself starts to disappear. This assumption sits underneath a surprising number of current AI strategies, even when nobody says it out loud. If we can observe enough, model enough and store enough information, then eventually the organisation becomes understandable, predictable and optimisable.
At first glance, this sounds perfectly reasonable. After all, modern civilisation depends on highly successful modelling. Trading systems work. Aircraft systems work. Industrial control systems work. Modern logistics systems work. Most of us would prefer not to discover halfway through a flight that the aviation industry has decided that exact calculations and bounded tolerances are now merely “guidance”.
So the argument here is not that modelling fails. Clearly it does not.
The difficulty emerges when organisations attempt to extend deterministic assumptions into adaptive systems containing recursive behaviour, hidden state, incomplete information and evolving incentives. That is a very different problem space entirely.
Even the interaction between two people becomes surprisingly messy once you examine it properly. Human beings rarely communicate in straight lines. They communicate through layers of signalling, interpretation, emotional state, self-interest, memory, partial information and strategic behaviour. Some people deliberately mislead. Others are sincere but incorrect. Many interpolate between incomplete facts and unknowingly construct false internal models that feel coherent to them. Confidence is not accuracy. Fluency is not truth. Consistency is not correctness.
Then recursion enters the system.
Each participant is attempting to model the other person while simultaneously modelling themselves and adjusting behaviour based on how they believe they are being perceived. The moment observation enters the system, the system itself changes behaviour. Anyone who has worked inside large organisations for long enough has seen this happen repeatedly. Introduce a metric and suddenly people optimise for the metric. Introduce monitoring and behaviour adapts around the monitoring. Measure ticket closure times and somehow tickets begin closing at suspiciously convenient moments five minutes before SLA breach.
This is not necessarily malicious behaviour. It is adaptive behaviour.
Underneath modern organisational structures sit much older behavioural dynamics. Technology evolves. Abstractions evolve. Civilisation becomes increasingly sophisticated. Yet the underlying incentives remain remarkably consistent. Humans still seek security, resources, certainty, status, survival advantage and reward. Organisations are simply scaled coordination systems built on top of those same behaviours. Markets, political systems, corporations, alliances and technology races all propagate upward from these foundational dynamics.
This is why the same patterns repeat throughout history regardless of technological era. A successful strategy becomes visible. Observation creates imitation. Imitation creates competition. Competition drives optimisation. Optimisation creates escalation. Eventually organisations form in order to coordinate resources more efficiently and protect advantage against competitors. The technologies change, but the behavioural mechanics remain recognisable.
This becomes highly relevant when discussing AI because many organisations currently believe they are solving a technology problem when in reality they are confronting a systems problem. The assumption is often that if enough workflows are mapped, enough interactions are captured and enough telemetry is stored, then the enterprise itself becomes understandable in its entirety.
But every model has a boundary, and anything outside that boundary can invalidate the forecast.
Alfred Korzybski summarised this decades ago with the phrase, “the map is not the territory.” The representation is never the system itself. The moment organisations forget this, they begin mistaking increased visibility for genuine understanding.
Suppose, hypothetically, we possessed technology capable of observing every particle and every state transition within two individuals from birth onward. Every memory, every biological interaction and every environmental influence within a bounded radius. Even then, the model remains vulnerable to externalities beyond its observation boundary. Expand the boundary and the complexity rises exponentially. Observation itself has a cost. Storage has a cost. Interpretation has a cost. Maintaining dimensional awareness across evolving systems has a cost.
At some point the pursuit of fidelity becomes economically irrational.
This is one of the major mistakes organisations make when approaching AI implementation. They focus heavily on external observation while largely ignoring the operational cost of observation itself. More telemetry is assumed to be inherently valuable. More dashboards. More workflow capture. More monitoring. More context. Yet every additional layer consumes computational resources, storage, governance overhead, organisational attention and interpretation capacity. Much of that information rapidly becomes noise rather than signal.
Herbert Simon observed decades ago that “a wealth of information creates a poverty of attention.” That problem has not disappeared in the AI era. If anything, it has intensified.
Worse still, the observation itself is already partially outdated the moment it is captured. Organisations evolve continuously. Incentives shift. Staff adapt around processes. Informal behaviours emerge. Political structures form internally. External market conditions change. Edge cases appear that no original workflow anticipated. The organisation being modelled today is already diverging from the version that existed yesterday.
At this point some readers will probably think this all sounds highly abstract or philosophical. Others may reasonably point out that organisations have always operated under uncertainty and that iterative deployment, bounded experimentation and adaptive operational models are hardly revolutionary concepts. That is also true.
The argument here is not against modelling, telemetry or AI. Nor is it an argument against ambition. It is an argument against the assumption that unlimited contextual modelling produces unlimited certainty.
It does not.
In practice, the organisations most likely to succeed with AI will probably not be the ones attempting to fully model reality or eliminate uncertainty entirely. They will be the organisations capable of identifying where visibility genuinely creates operational leverage and where additional observation begins producing diminishing returns.
That usually means bounding the problem space first. Run pilots. Identify where cognitive friction actually exists. Focus on augmentation before replacement. Observe where repetitive pattern recognition creates operational drag. Assume edge cases exist even when they have not yet appeared. Treat telemetry as an operational cost rather than an inherently positive asset. More visibility is only useful if the organisation can meaningfully interpret and act upon the resulting signal.
Most importantly, understand that people adapt to systems. The moment workflows become measured, behaviours begin to change. Metrics become targets. Processes become political. Staff route around friction. Informal structures emerge. Anyone who believes otherwise has probably not spent enough time inside real organisations where half the process documentation was already out of date before the meeting about updating the process documentation had even finished.
The goal therefore should probably not be perfect prediction. That is a pursuit without an endpoint. Every additional layer of observation reveals further complexity, further unknowns and further abstraction leakage.
The more realistic objective is improved adaptive capability under uncertainty.
The organisations that understand that distinction early are likely to have a considerable advantage over those still pursuing the illusion of control.