Being Human in Safety-Critical Organisations

By Dik Gregory and Paul Shanahan

If human error only starts to explain how accidents happen in complex, adaptive systems, what does the rest of the explanation look like? And what can be done as a result? If complex systems are fundamentally different from merely complicated ones, what does this mean for us – the people who have to live and work in them?

Through a re-analysis of real events, this book integrates recent thinking from psychology, resilience engineering, complexity theory and cybernetics.

Intimidated? Don’t be.

The result is a clear story of why people do what they do, how they mostly get it right, why they sometimes get it wrong, where safety really comes from, and why and how organisations need to fundamentally change their assumptions about people if they want to become safer.

The book is aimed at all safety-critical sectors, including aviation, chemical, defence, healthcare, highways, maritime, nuclear, rail and space. Throughout, the authors – both of whom are organisational psychologists – provide insight and clear practical guidance on how individuals and organisations can achieve greater resilience by acknowledging the true nature of human beings operating in a world of complexity.

• Language: English
• Publisher: TSO
• Release date: Sep 11, 2017
• ISBN: 9780115535451

Book preview

1

About this book

how people create safety, what stops them and what to do about it

Being human

For most of us, most of the time, going to work is a safe and uneventful activity. Sometimes, though, despite doing our usual best to act responsibly, we may end up as unwitting participants or bystanders – or perhaps victims – in things that go wrong. This book provides a structured approach to understanding how and why this happens, and how to adjust our thinking at both individual and organisational levels to deal with the many, ever-present influences on our behaviour at work.

The maritime industry is one of the most dangerous business activities that humans undertake. This book is the result of the responsible action of a consortium of maritime companies to do what they can to understand, accommodate, harness and assure safe human behaviour at work. As such, it is highly relevant to anyone – at any level – who is involved in the operation of safety-critical activities, whether in maritime or in sectors such as aviation, chemical, defence, healthcare, highways, nuclear, rail or space. The many different examples used in the book reflect the behavioural common denominators relevant to all these areas of people at work.

The underlying drivers of our behaviour – evolved over millions of years – will not go away and cannot be ignored.

It is our contention that dealing with them effectively and responsibly will not only make work a genuinely and sustainably safer activity; it will also make it a more honestly human one. We also believe the reverse is true: making work more human will make it safer.

A new perspective

About 40 years ago, a new understanding of major industrial accidents emerged. It crystallised out of the incident at the Three Mile Island nuclear plant in 1979. Such disasters have occurred at regular intervals: the Deepwater Horizon oil-rig explosion and subsequent pollution in 2010 is one among many more recent examples. What was new about our understanding of these accidents was the realisation that they could not be explained in terms of simple cause and effect. The complexity of the operations defied full understanding, both by those working at the time and, later, by accident investigators.

In recent years, a number of engineers, system designers, safety specialists, psychologists and organisational theorists have grown increasingly curious about the character of such accidents. To solve the mystery, they have had to break out of older ways of thinking. The result is a new approach to understanding how and why complex systems go wrong, known as resilience engineering.

A key aspect of resilience engineering is how it looks at the role of humans. Traditionally in safety thinking, people have been viewed as the weak link in otherwise soundly engineered systems. This is reflected in the great majority of accident reports, which usually implicate ‘human error’. Resilience engineering starts by recognising that today’s complex systems are necessarily underspecified. It is simply impossible to predict all the states the system can be in and how all the numerous components will work in combination. This problem becomes even more intractable when the system is embedded in a highly variable environment that can – and frequently does – throw up novel demands.

This inherent unpredictability means that the people working in the system must develop and apply deep expertise so that they can continually adjust the working of the system to cope with unexpected technical glitches and environmental demands. Far from being the weak link, humans literally create safety. But of course, being human, they will sometimes make errors or be confronted with challenges beyond their capacities.

Human beings: it seems it’s difficult to live with them and even harder to live without them.

At the same time as our understanding of the nature of complex systems has increased, there has been rapid growth in our knowledge of human behaviour. Psychologists, human-factors specialists, sociologists, neuroscientists, physiologists, evolutionary biologists and anthropologists have developed a vast and fascinating body of behavioural knowledge. We understand much better how humans perceive and make sense of the world; how we make decisions; what really motivates us at work; how we are affected by technological, environmental and organisational factors; how we communicate and work cooperatively with each other; and much more.

This book brings together the emerging knowledge of resilience engineering with recent insights into our behaviour. By understanding how resilience engineering and human nature fit with each other, operators, managers, regulators and others involved in safety-critical industries can adopt a more proactive approach to preventing industrial disasters and can have greater insight into what goes wrong when accidents do happen.

The book draws on many case studies and experiences from a range of safety-critical sectors. The lessons learned can be applied everywhere that humans are to be found creating safety in the midst of uncertainty.

A new framework

Being Human is constructed around a simple but powerful model of human behaviour – SUGAR – that helps people at all levels put these ideas into practice. The SUGAR model is introduced and developed in Chapter 3 (Being framed) and Chapter 4 (Being sufficient).

Together, Chapters 2, 3 and 4 describe the approach and framework for the book, exposing what we believe to be the true problems of safety in a complex, globalised world that is here to stay.

Chapters 13 and 14 give practical guidance and tools that will help individuals and organisations address their safety issues more effectively – and with proper regard to the humans at the centre of these issues.

In between, Chapters 5 to 12 provide evidence, examples and stories that amply illustrate the perspective that this book takes.

The authors are both organisational psychologists who have spent many years learning at first hand the realities of working in and with complex, safety-critical systems. We are grateful to the many seafarers, aviators, soldiers, railway signallers, healthcare workers, air traffic controllers and others who, over the years, have helped us understand their work and their lives.

We offer our deep respect to the vast majority who continuously sustain each other with their professionalism and mindfulness. And we pay tribute to those who suffer when things go wrong somewhere in the system of which they are a part. There will be more. This book aims to make their number less than it would have been.

2

Being at work

the curiousness of the problems we really face

What’s the real issue with people?

What does normal work look like?

Why can’t we eliminate human error?

Why can’t we predict and eliminate accidents?

The puzzle of people

In the past few decades, the proportion of accidents due to equipment failure has steadily decreased. We have got a lot better at understanding how materials work and how they fail. But over the same period, the proportion of accidents that implicate human behaviour has stayed stubbornly the same. Around half of all work-related accidents involve people doing something – or failing to do something – that contributes to a bad outcome. This is despite all the training, safety campaigns, lessons learned, and new rules and procedures – not to mention the distress that is caused when yet another accident impacts another industry, another business, and another set of family lives for ever.

What are we to do? How are we ever going to improve this stubborn statistic? Or are we looking at this the wrong way – measuring the wrong thing?

The authors of this book think so. This book is all about looking at the human relationship with safety in a different way. A way that is much more constructive, much more effective, much more consistent with our everyday experience of safety – and, in the end, much more human.

What do humans actually do?

Sidney Dekker is an airline pilot, professor of human factors and psychology, and author of several books on safety. Let’s start with his idea about the true place of humans in modern systems. Systems that involve humans are best characterised as both complex and adaptive. That is to say, they are composed of lots of smart, interacting components – including humans. Such components are ‘smart’ not only because they communicate with each other, but because each may have a different agenda, a different capability and a different range of applicability. Within their respective limitations, each component seeks to achieve its goals by coordinating and adjusting its activities with other components – all with the least effort, so as to conserve limited energy and resources.

So, for example, a surgeon with a particular range of knowledge, skills, experience and expectations – and in a particular emotional or physical state – prepares to operate on a patient. The patient has a certain set of capabilities to deal with the forthcoming surgical assault. The surgeon is supported by a number of medical staff, each with their own goals, knowledge and attentiveness. The surgical team members are all using sets of instruments with a particular functionality, readiness and reliability. And all this happens within an operating space that imposes its own demands and constraints, in a context in which the operating schedule is routinely paced or perhaps highly pressured by an external calamity that is producing more patients by the minute.

As the surgery proceeds, attention is focused and switched, information is requested and provided, assumptions are made and challenged, decisions are considered and enacted, suggestions are solicited and offered, techniques are tried and evaluated, and priorities are created and changed. The whole process is mutually coordinated in a fluid dance in which every element plays its part. This is true even of inanimate tools. A scalpel is either sharp, sterile and to hand when it is needed or it is not – in which case the drama takes a new turn as the team adapts to changed circumstances and reconfigures its actions and priorities.

Although this is a surgical example, the same underlying principles apply to every other human system. Recognition of these principles is especially important in safety-critical enterprises such as maritime, aviation, rail, construction, mining, space, nuclear and the highways.

A system may comprise many humans and pieces of equipment, or just a few. Whether it’s a bridge team, a cockpit crew, a military fire team, a construction gang, a surgical team or a road full of cooperating car drivers, the characteristics are the same. The space in which the system plays out is prescribed by sets of rules, procedures, processes and techniques that specify part, but never all, of what, how and when things get done. The space of operations – be it a ship’s bridge, a flight deck, a building site, an operating theatre or a highway – is always filled with uncertainty. Its users are always uncertain about who exactly will do what and when, and what this might mean in the moment in which it is done, given the many other things that may also arise in that moment. Our task within the uncertainty of the space in which we operate is always underspecified.

This is why Dekker says that the essential place of humans in systems is to complete the design.¹

The story that unfolded on 17 January 2008 at London’s Heathrow Airport is a wonderful example of humans completing the design. It is also an example of an accident that was due entirely to equipment failure. The fact that the situation was recovered by the humans involved is something that occurs frequently in the vast amount of everyday work in which there are no accidents. We hardly ever notice them – but the stories are there, if we only care to look.

Accident investigations often focus on the shortcomings of system components – and people are an easy target. So it is especially interesting to find an accident that involves an extended and disparate team of people doing almost everything right to successfully avert what would otherwise have been a monumental disaster.

This is the story of British Airways ‘Speedbird’ 38.², ³

Speedbird 38 – plucked from the jaws of disaster by human agency

As British Airways (BA) flight 38 lifted off from Beijing, China on 17 January 2008, its pilots had no idea that less than 11 hours later they would be at the centre of one of the most remarkable dramas ever to unfold at London Heathrow.

In fact, no-one had any idea about the impending drama until just one minute before touchdown at 12.42 on runway 27L. It was a cloudy, dry London winter’s day and, at 10°C, a mild day for the time of year. The aircraft – call sign Speedbird 38 on this flight – was a Boeing 777. It had been in uneventful service for seven years, clocking 28,675 hours over the course of nearly 4,000 take-off/landing cycles.

In the commander’s seat was 43-year-old Captain Peter Burkill, a married father-of-three from Worcester – his youngest just ten weeks old. He had 8,450 hours of flying experience on Boeing 777s. Alongside him was co-pilot John Coward, aged 41, with 7,000 hours. When not working, Coward lived with his wife at their home in France.

They were both highly experienced and highly trained, having met the rigorous and regular simulation and certification requirements demanded of all BA pilots.

For all but the last minute of Speedbird 38’s flight that day, the trip was apparently routine and uneventful. What the crew did not know was that a peculiar and previously unknown malfunction had been developing in the fuel system throughout the entire flight.

Abnormally high levels of soft ice had built up in the fuel. Ice in fuel is quite normal and comes from water that occurs naturally in the aviation mix. The difference here lay in the combination of very cold conditions at the start of the flight, a long period of cruising with low fuel flow, a fuel temperature that increased significantly in the latter part of the flight, and a fuel/oil heat exchanger whose design could not cope with the resulting large amounts of soft ice that detached from inside the fuel lines as the aircraft manoeuvred for its final approach. The effect of these circumstances had never been simulated or imagined by the aircraft’s designers – or its pilots.

As Speedbird 38 descended towards London, Captain Burkill had the controls. Under the guidance of the UK en route air traffic controller (ATCO) at Swanwick, 60 miles (100 km) away on England’s south coast, Burkill descended the aircraft from 4,000 feet. The Swanwick ATCO handed over the aircraft to the London Heathrow control tower, and Burkill responded to new ATCO instructions to enter a hold pattern at 1,100 feet. After five minutes, the tower instructed a further descent to 900 feet. As he did so, Burkill prepared the aircraft for an Instrument Landing System (ILS) approach to Runway 27L. This put the aircraft on an automated glide path and set things up for the aircraft’s control to be handed to co-pilot Coward for a manual landing.

At 1,000 feet, and 83 seconds before touchdown, the aircraft was fully configured for the landing. The landing gear was down and flap 30 selected, as normal. This flap setting would increase lift in exchange for the lower speed required for a safe and smooth landing.

Around this time, as planned, Burkill handed over control to Coward, who now became the ‘pilot flying’, releasing the captain to maintain a trouble-shooting, supervisory role for the landing. It was just as well.

In the Heathrow control tower, the duty ATCO was in full charge of the normal, high-intensity operations at one of the world’s busiest airports. Here, there is a major aircraft movement every 60 seconds or so, a tempo that is enabled by the support, research and training provided by NATS (National Air Traffic Services – the British ATC organisation).

Coincidentally, at that exact moment, the authors of this book were at NATS in Swanwick, having just finished a meeting with en route ATCO managers. In the spirit of continuous improvement that makes NATS world-class, we had been discussing ways in which human-element thinking could be even more deeply entrenched into NATS’ already sophisticated operational practices.

At a height of 720 feet, 3 miles (5 km) and now less than a minute from touchdown at London Heathrow, the fuel flow to the right engine of the 777 suddenly stopped. Seven seconds later, at 620 feet, the same thing happened to the left engine. The pilots didn’t know it, but ice in the fuel had starved the engines just as a final demand for thrust was needed.

At 590 feet, and 48 seconds before impact, Coward became acutely aware of a serious thrust failure. The problem distracted Coward from disengaging the autopilot at this point, as he had intended.

At 430 feet, with 34 seconds to go, as Coward struggled to comprehend the problem, the following calm and understated exchange took place:

Captain Burkill: [Is the aircraft] Stable?

First Officer Coward: Well, not exactly. I can’t get any power from the engines … Looks like we have a double engine failure.

The aircraft was not going to make the runway. In fact, it was starting to stall as the autopilot lifted the nose to try to maintain the glide slope.

At 240 feet, Burkill retracted the flaps. The action was instinctive and not taught in the simulators. It had the desired effect. It reduced drag and (as it turned out) increased the distance to touchdown by a vital 50 metres. Although Burkill knew it would have little impact on the forthcoming stall, he had done what he could to clear the perimeter fence. Accident investigators later calculated that if the flap setting had been left alone, the impact would still have been within the airfield boundary – just – but the aircraft would have collided with the ground-based ILS antenna, causing substantial structural damage and potentially multiple passenger injuries.

At 200 feet, the stick shaker activated, telling the pilots that the aircraft was stalling. They were dropping like a stone. At 150 feet, Coward pushed hard on the controls. His action caused the autopilot to disconnect and the aircraft to keep flying with the wings level. As the aircraft glided very low over Hatton Cross tube station, to the left of a busy petrol station and to the right of a dense residential area and school, the pilots had an unusual and terrifying view.

They were diving towards the ground at 125 mph. Burkill transmitted MAYDAY … MAYDAY … Speedbird … Speedbird … 95 … 95. As the pilots continued their struggle with the stalling aircraft, there was no time to instruct the passengers to brace, brace.

It was three seconds before impact.

In the tower, the final-approach ATCO had been managing several aircraft in various stages of transit. Included in these had been a series of transmissions to Speedbird 38, giving clearance to land and post-touchdown taxiing instructions. After hearing the MAYDAY transmission, the ATCO took a few beats to comprehend the situation.

He ignored the fact that, under extreme pressure, Burkill had used the wrong call sign (Speedbird 95 instead of Speedbird 38). Interestingly, ‘Speedbird 95’ is the call sign reserved for use in BA’s 777 simulator. Under stress, people tend to revert to what they know best. It is a measure of the quality of BA pilot training that what Burkill knew best was his training, although even that had not anticipated the present circumstances.

The aircraft skimmed the perimeter fence, missed the ILS tower and finally stalled 10 feet from the ground, dropping to the soft, rain-soaked grass. Coward flared the nose of the plane to reduce the speed of impact. Initial touchdown was 1,000 feet short of Runway 27L. The main landing gear tore gouges 18 inches deep before the aircraft bounced and touched again, breaking up as it slid along the grassy apron and up onto the southern edge of the runway itself. The left main landing gear had seriously bent, puncturing the wing. The right main landing gear had torn off at first impact, causing the aircraft to veer to the right and puncturing the fuselage. The penetrating wheel assembly broke the leg of the unlucky passenger in seat 30K – the only serious injury sustained among the 152 people on board.

Now aware of the crisis, the tower ATCO instructed another aircraft preparing to depart, Speedbird 229, to hold position. He then transmitted clearly, but with arresting urgency, Aircraft accident … Aircraft accident … the position is the threshold runway two seven left … aircraft type is a triple seven … nature of the problem is ‘crash’ … aircraft has crashed. Rendezvous point is south.

This message contained all the essential information for the fastest possible fire, medical and police response.

As the plane came to a stop, there was an eerie silence in the cockpit. To the pilots’ surprise, they appeared to be uninjured. They were not so sure about the passengers, many of whom they feared must be dead. But now the risk was fire and Burkill made the announcement This is the captain. This is an emergency: evacuate, evacuate.

Hearing this, the ATCO immediately realised that, in the continued stress of the moment, the captain had made the announcement on the wrong circuit – on the VHF radio to the tower, rather than on the public-address (PA) system to the passengers. The ATCO responded calmly and effectively: Transmitted on ATC, sir. Fire service on the way. Without missing a beat, he immediately switched his attention to another aircraft on approach: Qatari 011 go around … I say again, go around, acknowledge.

The approaching Qatari aircraft acknowledged immediately and the ATCO spent the next few minutes on VHF, intercom and telephone, coordinating the emergency services, managing the multiple approaching aircraft and liaising with his ATC colleagues in the tower and at Swanwick.

Meanwhile, at Swanwick, high-alert procedures were being implemented. All ATCOs were recalled from their breaks to the operations room and visitors (including ourselves) were asked to vacate the premises, pending a potential lockdown, until the source of the problem at Heathrow was understood. The possibility of a terrorist attack was clear – especially since one of the passengers in a waiting aircraft, just a few hundred metres from the crashed 777, was the British Prime Minister, Gordon Brown, on his way to China for an official five-day visit.

As soon as the crashed aircraft had come to rest, some of the passengers left their seats to seek an exit. Under the leadership of Cabin Service Director Sharon Eaton-Mercer, the cabin crew took immediate control, firmly instructing people to stay in their seats. Moments later, the captain gave the evacuate command and flashing red lights picked out the exits.

The cabin crew held back the passengers until the escape slides were fully inflated, blocking access to slides that had too much debris at the bottom. They then helped the passengers to jump. Passengers began leaving the aircraft just 58 seconds after first impact. The passenger with the broken leg in seat 30K was helped by a neighbouring passenger, who accompanied him down the slide. The cabin crew were the last out, following an operation carried out with textbook clarity and care.

Even though they had no prior warning of a developing emergency, the fire services arrived at the crash site within two minutes of first impact. This was just as well. Nearly 7,000 kg of fuel had leaked out of the aircraft and oxygen was escaping from ruptured passenger emergency cylinders. The attending fire officers neutralised the problem with 300,000 litres of water and 17,000 litres of foam.

As might be expected, London Heathrow was severely disrupted for days afterwards. But this impact was nothing compared with what might have been.

What was good here?

The crew were presented with a potentially catastrophic situation at a critical moment and with very little time to sort it out. Sudden double engine failure 2 miles (3 km) from landing was unprecedented, unimagined and unrehearsed.

What was needed were high levels of professionalism and knowledge. Deep technical knowledge – in this case about flight – was required so that effective action could be generated. A high degree of emotional training was also required, so that panic could be averted and minds could remain analytically focused on the problem.

Read more about the nature of emotions and understanding in Chapter 7, and the nature of expertise in Chapter 10

But far more than this was delivered on the day that Speedbird 38 crashed. High levels of teamwork, coordination and shared understanding were displayed by the cabin crew, the London tower ATCOs, their en route colleagues miles away in Swanwick, the airport emergency services and ground staff, and the flight crews of other aircraft in the area. Fundamental to this coordination was the use of effective, unambiguous and crystal-clear communication made at the right time and in the right way, allowing priorities to be correctly set and actions to be synchronised.

Read more about the nature of teamwork and communications in Chapter 11

The coordination was not perfect, of course. The evacuation checklist was split between the captain and first officer. While one operated the engine cut-off switch, the other operated the engine fire switches. But they were done in the wrong order. In the stress of the moment, the fire switches were thrown first, and a large amount of fuel leaked out of the engines. Fortunately, there was no fire, even though the risk was heightened due to leaking oxygen.

All unintended results within complex systems occur when many components (including humans), each with their own variability, combine in an unimagined way to create a unique event. When the event is negative and people are hurt or property is damaged, we call it an accident. When the event is positive and a new material is discovered or a new insight is glimpsed, we call it an invention. When the event is neutral and (for example) aviation fuel leaks but there is no fire, we call it luck. If we are wise, we attempt to learn by removing the reliance on luck via better design.

Humans have deployed such learning throughout their evolution. What we need to understand, however, is that multiple interacting components – each with their own agendas and capabilities – will always generate unique combinations with unintended consequences. No amount of learning will result in a finite space of operations in which all combinations are knowable and all eventualities can be catered for. In fact, the opposite is the case: the more we learn, the more new components emerge, with new degrees of freedom and potential new combinations. That means unimagined, unprecedented possibilities.

It is only when an accident occurs, such as the one involving Speedbird 38, and forensic analysis follows, that we suddenly see humans doing what they do unnoticed the rest of the time. When people go about their work apparently uneventfully, or when ‘luck’ works in our favour, we usually fail to think of them as doing what the crew of Speedbird 38 did: ‘completing the design’ of underspecified processes.

Instead, our perception of what humans spend most of their time doing is often distorted. When, with hindsight, we see people making decisions that appear to be on the critical path to catastrophe, we can become seduced by the desire to blame, re-train or eliminate them. Such temptation often goes hand in hand with the desire to create a new procedure, which we then insist everyone else uses.

Read more about the problems of hindsight in Chapter 9

Get help with enhancing hindsight analysis in Chapter 14

Unintended consequences – what happens when you poke a system with a stick

If it seems clear to us with hindsight that a new procedure would have prevented the observed accident, it is easy to convince ourselves that adopting the new procedure will create a safer workplace. Unfortunately, this logic only works to the extent that the unique combination of degrees of freedom that created the bad results will occur again. The logic can be further undermined to the extent that the new procedure produces a new range of possibilities. If its use interferes with the ability of other system components to achieve their function or goals, then the behaviour of those other components will also change, leading to a new set of uncertainties.

This dynamic was established in chemistry by Le Chatelier in 1898, and it has since been generalised for use in biology and other realms. Le Chatelier’s principle basically says that any change in the stability of a system provokes an opposing reaction, leading to a new status quo. If you fuss with the system by changing how one of its components operates, it will respond by trying to find a new stability. All sorts of unintended consequences may then follow as other components adjust their behaviour to compensate. In human circles, this is known as ‘playing the system’, but in many cases it is a natural and organic phenomenon in which the system is simply trying to re-optimise itself in the interests of efficiency.

In 2002, the UK government tried to reduce the time that casualty patients were waiting to be seen by a doctor. It decided that 98% of patients should be dealt with within four hours. In 2010, this was eased to 95%. Both targets were virtually impossible to achieve without radical system redesign and extra resources. Despite this, average waiting times appeared to get a lot better. How come?

What is interesting is what the highly pressured hospital system did to help itself. One response was to delay the four-hour clock from starting by holding patients in ambulances until the hospital staff could take them. Another was to transfer casualties out of the queue into ‘assessment wards’, so stopping the clock. A third was to discharge patients and then immediately re-admit them.

It would be easy to assume that staff were simply evading the means of measuring their performance. However, this view assumes a world of simple systems, governed by linear cause and effect. In the real, complex, adaptive world consisting of many constantly negotiating parts, these are effective strategies that introduce a new degree of freedom to replace the one that was taken away.

It is important to understand that such compensations do not return the world to the way it was. Instead, they move the system into a new space, leading to new negotiations and a new balance that may or may not be fundamentally more dangerous than the one it replaced.

Detailed scrutiny of an event with a bad outcome seems to reveal problems that then skew our perception of the role of people in their everyday work. When we see people make apparently avoidable mistakes, we want to blame them. Alternatively, we set about creating procedures that are designed to prevent others from making the same mistakes.

The curiousness of the problems we really face

The curiousness of the problems we really face at work is twofold.

The uncertainty of operations First, as we have just seen, we operate in an uncertain, underspecified world where we must function as designers in order to complete the specification in an operational setting. This means it is of limited value to keep adding more rules and procedures.

The capacity for surprise Second, it is often not obvious how the human behaviour at the centre of accidents is any different from that on days when no accident occurs. This means that we must look not so much at what people do, as at what influences them to do the things they do at the time they do them.

Underlying both of these problems is the curiousness of the kind of world in which we live and work. A fundamental distinction needs to be made between complex, adaptive systems on the one hand and systems that are simple – or merely complicated – on the other.⁴

Simple systems have few elements, variables and states. Here, checklists, stepwise procedures and simple look-up tables can be applied by almost anyone to diagnose and fix problems.

Complicated systems may have very large numbers of elements, variables and states, but, like simple systems, they have been designed and assembled. Problems will occur, but their solutions will be found in some giant virtual look-up table (although it will normally take an expert to analyse and find them).

Complex systems are radically different from those that are simple or merely complicated. Modern humans live and work as agents in complex, adaptive systems where everything is tightly connected and small changes in one part can have huge implications elsewhere, producing lock-up and confusion.

Read more about complex systems in Chapter 12

A good example of lock-up is when a car driver in heavy traffic lightly touches their brake pedal, causing the driver behind to do the same. The knock-on effect results in total stoppage several kilometres behind – and the potential danger of rear-end shunts. As the bunched-up traffic gets going again, drivers are mystified to discover that there is no accident ahead – just free-moving traffic. Naturally, there is no trace of the briefly illuminated brake light.

The answer to such problems does not lie in more complicated layers of rules and procedures, but in some means of detecting when the system is becoming fragile and then damping the particular behaviour that is responsible. In the UK, Highways England⁵ found a great way to do this. Sensors detect when the density of traffic has built up to a critical threshold, at which point compulsory variable speed limits come on. The traffic controls itself by means of a damping feedback loop. The result is that traffic slows, but moves much more continuously – and much more safely – with greatly reduced ‘stop–start’ effects.

There are many examples of the confusion that arises when overcomplication develops. Hundreds of alarms were triggered in the first few seconds of the near meltdown at Three Mile Island nuclear power plant in 1979.⁶ In a more recent case, engineers were puzzled by negative-pressure test results on BP’s Deepwater Horizon platform in the Gulf of Mexico in 2010.⁷ What they didn’t realise was that a giant bubble of hydrocarbons was heading towards them from the seabed with the force of a 500-ton train. The resulting explosion killed 11 people and produced huge environmental damage, five years of lawsuits, worldwide corporate opprobrium and a bill for $60 billion.

Is it possible to turn a complex system into a simple one? It might seem so, but this is just an illusion.

Prosecution lawyers, the media, accident enquiries and gossiping onlookers do it all the time. By applying hindsight, it is possible to manufacture a coherent and clear causal chain that tells just one story of a single trajectory through a simple space that inevitably ends in what was observed to happen. The problem, of course, is that the imagined simplicity of the space didn’t exist at the time the participants were involved in it. Instead, there was highly selective attention, prioritising in the face of deep uncertainty, imperfect knowledge, calculated guesswork, intuition, the guidance (and possible misdirection) produced by previous experience, and the attempt to predict future interactions between components that are in principle unpredictable.

Explanations of things as simple causal chains of events are compelling, but not very useful. In the complex, adaptive systems exemplified by safety-critical industries such as aviation, chemicals, defence, healthcare, highways, maritime, nuclear, rail and space, we need a different approach, a different way of framing our understanding: a matter to which we turn in the next chapter.

Footnotes

1Dekker (2005)

2AAIB (2010)

3Wikipedia

4Dave Snowden (2016) distinguishes a further type of system – the chaotic – in which agents are unconstrained and behave independently of each other, with little or no mutual regard. Chaotic systems lend themselves to modelling by probability theory, which is good at describing the world of unconnected random events. Chaos, random events and probability inhabit a strange world that is extremely difficult for most humans to understand – as we shall see in Chapter 8 (Being in the know – part III). See http://cognitive-edge.com [accessed April 2017].

5Highways England is the UK government authority that operates and maintains England’s motorways and major roads.

6Kemeny (1979)

7National Commission on the BP Deepwater Horizon Oil Spill and Offshore Drilling (2011). Deepwater Horizon was the name of the platform from which drilling operations had been taking place at BP’s Macondo Prospect oil well on the seabed 1 mile (1,600 metres) below.

3

Being framed

how context makes us blind

How is it possible to hide in plain sight?

Why do things surprise us?

Why do different people see different things?

Why do the same things have different meanings for different people?

Where do context and meaning come from?

A question of perspective

Here’s an easy question. Look at everything around you. What do you see? For everything that you silently pick