> Automation doesn't make operators more careful. It makes them forget how to be. The more reliable the system, the less ready the human.
The entire premise of a system is that it removes the need for careful attention.
system: signal lights tell me whether or not I can pass through an intersection, so that I do not have to attend to potentially high speed traffic from a variety of directions.
system: the side my knife blade sits on my arched guide fingers, so that I do not have to attend to the edge of the blade or the location of my fingers.
>The entire premise of a system is that it removes the need for careful attention.
I think this premise is flawed or, at best, too narrow. A system is just a logical grouping of items that perform a function. Sometimes that function can be to reduce cognitive burden, but it doesn't have to be. A "vision system" like what humans use does not reduce attention, but increases/enables it, while a autonomic nervous system can reduce attention. The ability to increase/reduce attention is not the central principle of a system.
What I'd say you're pointing out is that the word "system" is overloaded.
A vision system does allow you to pay less attention: you don't need to carefully remember how far away the door is, you just need to look! I tried this often as a kid: if you want to navigate a hallway with your eyes closed, you need to pay far more attention to your other senses than you need to pay with your eyes -- where attention here is not the volume of data, but rather the complexity of conscious bookkeeping -- I can (ironically) "play it by ear" with my eyes open, but eyes closed I must plan every step!
It just so happens to be that the ability to pay less attention makes more things possible and hence the demand for attention overall may increase -- if not intrinsically, due to your competitors (who can also see!)
I would argue this take conflates attention with cognitive overhead required because of a lack of training. Navigating with our closed feels like it takes more attention because we’ve practiced so many hours navigating by sight that it no longer feels cognitively burdensome. A bat would have no trouble navigating without sight for the same reason. I don’t think most people would say giving up our sight for echolocation would reduce our attention, it just transforms it.
> system: signal lights tell me whether or not I can pass through an intersection, so that I do not have to attend to potentially high speed traffic from a variety of directions.
You know I noticed this... I lived in a country where people obey traffic laws, and in a country where they very much don't.
I witnessed many more traffic accidents in the country where people are used to relying on the traffic lights to tell them if it's safe or not.
Whereas in the other country, everyone correctly assumes that the other drivers are completely insane, and so they stay vigilant.
Other than the data of road fatalities that disproves this anecdote, my own anecdote is this is the false sense of security people get in other countries that don’t have traffic laws. Oh see the people have to look all the time so it’s much safer. When you start to live it for a long time you realize it’s not true. Many more fatalities.
Now I do think the science shows if you design roads and systems to make drivers more thoughtful it can improve outcomes. Size roads for the speed limit, roundabouts, etc. these can make a difference as it balances the system.
Having lived, driven, and crossed roads in both -- what I find is essentially that drivers from poor systems pay far more attention, but the system is a lot more effective than attention.
The difference here is one of stability: in a developing country, I can just walk across a street (often there is no traffic light) by essentially signalling with my body language -- both I and the drivers are paying attention. And if one party fails, the other has a good chance of catching that mistake.
Now, in a developed country, neither side is paying attention. If I walk across the street, I'm in danger, no matter how clear my body language (I tried it on British streets a few times -- it works in some areas, but usually very poorly!), and no one expects a crazy driver to come barreling through a red light.
The developing countries fall behind because in the crazy * sane intersection, sometimes the sane person is just not fast enough -- whereas the crazy * crazy intersection is extremely dangerius and happens often enough.
On the other hand, a developed country makes every interaction sane * sane regardless of the personalities or moods of those involved -- but God forbid a bit of crazy leaks out!
>Other than the data of road fatalities that disproves this anecdote,
You can't make that assertion (well you can, it's called "lying with statistics" but that's beside the point) without knowing if the fatalities the result of the accident rate or just a higher conversion ratio as a result of reduced safety equipment, reduced seatbelt usage, more motorcycles, etc, etc, worse emergency services, etc, etc.
INB4 other people start whining on your behalf, I'm not saying those countries aren't less safe to drive, just that you can't do a straight comparison of accident rates and fatalities without considering the conversion ratio.
Of course you have to consider confounders. That’s why transportation data usually includes best efforts.
But at some point you have to look at the totality of the evidence. Countries with better road infrastructure, enforcement, vehicle standards, and driving behavior generally produce better safety outcomes. The fact that multiple factors contribute doesn’t make the observed outcome meaningless.
As I already stated there is absolutely systems that increase the perceived sense of risk that can help outcomes (road width sizing, roundabouts, minimal signs/lines) but those typically work best in a system where there is already some sense of order.
There's some documented studies of removing all the street clutter and lines from residential area intersections forcing drivers to be more careful, especially around pedestrians, reducing overall accidents. But this does reduce throughput slightly.
This is an example of risk compensation. When people perceive greater protections around themselves, they tend to become more aggressive at the margin, such as with the driving habits that you mentioned or hitting more violently in American football because of improvements in helmets and padding.
Football might be a weak example, because being able to hit harder is an overwhelming competitive advantage. A player who acted like they were not wearing a helmet would be effectively dysfunctional.
In contrast, most careless driving habits don't actually get anybody to their destination any quicker.
Not sure why this would downvoted. Go to a less developed nation where traffic laws are not important and it’s one of those sense of false security ideas.
When I lived in India everyone would always tell me how everyone drives so much safer there because they're more aware etc (similar reasoning as we're seeing in this thread), but man, the national crash statistics say otherwise.
Not to mention I lost count of how many dozens of accidents I witnessed in my year there. I've personally been in 3 rickshaw crashes.
My experience as well in Vietnam. the first time I was there I figured they were right but it’s just a false narrative people sell to make themselves feel safe. Don’t even get me started on methed out American size semi trucks speeding with no concern of running you over.
No they don't, they tell you and other vehicles to stop. You would fail your driving test if you depend only on the traffic lights and don't bother to verify it is safe to pass yourself.
The main function of a traffic signal is the green phase, not the red phase. A traffic signal increases throughput by allowing drivers to ignore crossing traffic.
(If safety/the red phase was the purpose, the intersection would use a roundabout instead.)
I mean, it depends on where you take your driving test. In a lot of places in the US (especially in some rural areas), you may still pass. In some cases you might not even drive near a stoplight during the test.
If you "know a guy" you can even pass a driving test without ever getting behind the wheel of a car. Road licensing is in complete shambles in the last 10 years. A lot of "workarounds" and corruption.
You definitely still should be paying attention to cross traffic, regardless of what the lights indicate. The lights just make it easier by stopping most traffic for you, so you only have to do a quick scan for outliers.
Someone learning to fly may be described as paying careful attention: to every little sound, vibration, and sensation. A common tactic by student pilots is overcontrolling the aircraft, e.g., large sudden changes rather than smooth pressures from flying with a light touch.
Automation requires active, intentional attention particularly when flying in clouds. What are my instruments telling me? Are they all telling the same story? Have any failed? Which ones?
A significant part of flight training and testing emphasizes the ability to divide attention between multiple competing needs, being able to correctly prioritize them, and responding promptly and safely in order of priority.
There's been many studies around the concept "how do you get expertise? In what field can you actually develop expertise?" such as Naturalistic Decision Making [1] [2]. The wisdom from there can be applied to the question of automation.
If your task is for example, to pilot a preset route with stable condition and very low surprise, you will fall for the "getting too comfortable" trap and we tend to start to get lazy(or efficient) and offload the mental effort and skills atrophy. A common workaround to this is to have regular training(deliberate practice) that introduce the "tricky" situation to keep the skill up. Problem ofc arises when people don't keep this up.
This can be seen in the diagnostic performance differences between junior and senior doctors, not always in the favor to the senior [3]. If you add a layer of automation but the insight gathered by working on that layer is great (and falls off) then deliberate practice start to become a requirement
I think part of the point of the article is that it also makes edge-cases more dangerous and catastrophic than if there was no autopilot at all. From the article:
>The argument for automation is that it frees up cognitive bandwidth. Fewer routine decisions means more headroom to think carefully about the ones that matter.
So if the expectation is that the human pilot is expected to pay attention to mitigate the dangerous edge cases "that matter", there is a contradiction: the tool that promises to free up the bandwidth for that attention creates a complacency that prevents that attention from being applied.
In other words, it makes the normal situations safer but the abnormal situations more dangerous.
I remember reading an internal memo something to the effect that with the assist systems off there were large groups of pilots that couldn't physically control the aircraft to the point of landing, they just didn't have the strength for it because the physicality had stopped being a day to day aspect of the job.
Landing is an abnormal situation for an aircraft which we make SUBSTANTIALLY less dangerous through intense automation. Do you want to rip out automated landing systems?
The FAA describes taxi, takeoff, landing, and operations other than cruise flight below 10,000 MSL as critical phases of flight because of increased risk. The aircraft is closer to the ground, other aircraft, and hazards such that prompt, correct responses are essential to the safe outcome of the flight.
Any equipment on the aircraft can and will fail. Becoming dependent on autoland — not a worry on most general aviation aircraft — is terrible risk management. Every pilot must maintain hand flying skills. Automation is nice and reduces workload, but the pilot must actively manage it.
Not only is landing not an "abnormal situation", contrary to armchair internet wisdom pilots of airliners in fact do not use autoland all the time and don't even always fly a precision approach at all.
Not to mention that they get mandated regular reviews of their ability to fly manually. And even with that, there's still a reason why "children of the magenta line" (i.e. pilots who passively follow automated systems into danger and/or have seriously degraded stick-and-rudder skills) has become a term.
2 things. First, landing a plane is abnormal! You're shedding a huge amount of energy and you're transitioning from a state where you can keep flying due to your speed to one where flight is impossible as you lower speed. That's an abnormal state to put an aircraft in, regardless of how often it happens. Second, what exactly is the level of automation you're saying is not necessary? Should we rip out radar systems that mark glide paths?
This is about as silly as feverishly claiming that e.g. deploying a web app to production is an "abnormal situation".
On top of that I'm sorry but you seem to have skimmed over both the article and what I said in favour of clutching pearls at some nebulous entity apparently claiming that "automation should be ripped out" when what is actually being explained to you is that without actual, manual, hands-on, current experience the "human in the loop" loses the ability to properly control or take over from an automated system - and worse, the ability to even understand when it is doing something nonsensical and/or dangerous.
As an aside, I assume that by "radar systems" you are referring to radio navigation aids. Like I've already mentioned (though in fairness not everyone knows what a non-precision approach means), pilots of airliners are still trained to fly without them, are expected to know how to fly without them, and shockingly enough DO fly without them in the real world where equipment fails or cannot even be installed at all. I know most of the software that people write here is insulated by several layers of abstraction from the hardware, but surely we haven't already lost the understanding that automated systems are not in fact magic - that they depend on real world hardware with real world physical constraints?
First, snark goes against HN guidelines so you might want familiarize yourself with them.
Secondly, MCAS autonomously adjusts trim based on sensor inputs to avoid a hazard. It is not advisory and directly controls flight surfaces. This would make it automation according to how organizations like NASA categorize flight software taxonomy.
> That's an abnormal state to put an aircraft in, regardless of how often it happens
Taking off, flying, and landing are all absolutely required in the normal operation of a plane. If your plane is not engineered in such a way that landing is normal, it won't last long
The circumstance doesn’t have to be that dramatic to be abnormal.
Landing after a merely unstable approach, too many significant changes too close to landing, increases risk.
Landing too fast may result in overrunning the end of the runway, pilot induced oscillation, or loss of control. Energy being proportional to the square of velocity means the margin doesn’t have to be huge to pose significant danger. Landing too slow risks an aerodynamic stall or worse a spin, which at low altitude is nearly certain to be fatal.
Landing safely with a crosswind requires technique changes. Too much crosswind or “running out of rudder” is extremely dangerous.
Landing after accumulating airframe icing is triply bad because the ice reduces the control surfaces’ aerodynamic effectiveness, makes the airplane heavier, and requires a faster landing.
As the article already states, there is a well known phenomenon in aviation called automation-induced complacency. So, yes, if you automate landing to the extent that human pilots no longer pay attention to abnormal signals that indicate something is wrong, or no longer feel the need to train or stay vigilant, it can make things more dangerous. There is plenty of research on this, but here's the first that came up in a cursory search:
A more recent example is the Boeing 737-Max where there was a focus on automating trim control. In that case, the automation made the system more complex, to the detriment of a pilot understanding and reacting to an abnormal operation.
We should also be careful that we don't create a false dichotomy between "all automated or no automation", or an expectation that more automation is always better. The goal should be the right balance that increases reliability/safety.
> A more recent example is the Boeing 737-Max where there was a focus on automating trim control. In that case, the automation made the system more complex, to the detriment of a pilot understanding and reacting to an abnormal operation
To be fair this is not entirely accurate: a focus was made on stall prevention in a very specific mode of flight given the variant's increased susceptibility to the pitch-power couple. It did not make the system any more complex per se than other airliners - see e.g. Airbus aircraft which do actually have autotrim in normal flight. The actual kicker was that the existence of MCAS was hidden to avoid the need for lengthy re-training of pilots if the 737 MAX was deemed sufficiently different from its predecessor variants (on top of MCAS being rather poorly implemented in its first iteration).
Fair enough. The “hidden” aspect is what I was alluding to…ie, control that exists but isn’t apparent to the pilots (and worse, intermittent). In the human factors world, it was more complex than the pilots assumed, but you’re right that it’s probably not the best description.
(As an aside, the hazard being mitigated, ie stall, has little bearing on whether or not it’s autonomous or complex, although it does impact whether its safety critical)
The fact that the autopilot will loudly disengage if there is a serious enough control surface failure to cause an upset is more than enough support IMO.
> I don't think we believe that the automation makes the human operators safer, but that it is overall safer than the human operator alone.
If the automation is for the easy/routine stuff, then no. The automation doesn't work in exactly the most safety-critical situations, and then the human operator is thrust into fixing the situation without the full context.
I don't know about flying, but in machining it is the easy/routine stuff that will take your hand off. New machinists rarely get the serious injuries that more experienced ones do (they get more minor injuries) - the experienced machinist gets complacent about how dangerous the machine is and stops thinking to keep their hands away until they lose it. Automation has saves a lot of limbs (and lives) because you don't need to get close to the moving parts anymore.
Well, it's differently safer. It's impossible to make categorical statements of more or less safe with automation because it depends massively on the design.
Sure you can; right now, planes fly 90% on automation. They fall back to pilots to correct issues, engine outs or touchdowns. Right now, that fallback condition is mostly reasonable: pilots are trained extensively and have thousands of hours of flight experience to manage it. But as that automation approaches 100%, that fall back condition crosses a line where it's more likely the pilot is both not well trained because of it and their instinct in an emergency is entirely untrustable.
So too is what's happening with LLMs: they're writing code that the programmer is increasinging unaware of and the programmer is increasingly not capable of understanding because they dont have the experience of writing and navigating complex codes. So in the event of a permissions prompt, the fallback condition, you have the same race condition between increasing automation removing the knowledge generation of the operator.
So there's obviously a rubicon where automation _has_ to be 100% because no operator can be a fallback.
Automation should always be the serialization of understanding of the system, which can execute on its own under normal conditions. When it breaks, your reaction should not be removing the automation and asking humans to get back to manual operation, that is the wrong approach. Instead, you should ask humans to leverage their understanding to step in, reason about the automation system, and fix it.
This is how I have been doing SRE for the past decade. And the inability to practice this under-the-hood reasoning is the main reason why I don't trust any modern AI based automation systems. They are helpers for some manual work, but they can't be automation.
There are two other solutions, make the automation so reliable that the human skill is not needed, or make the consequences of failure low enough that you can just accept failure.
As far as I know, Places that actually care about safety apply all of them at once.
They add layers of failsafes that don't rely on humans, they make the automation better and better, and for the most critical stuff, they continue training for things that will almost certainly not happen.
And for the rare(depending on personality) cases where we care about the pilot's ability more than the result, just don't automate at all.
From my manufacturing experience on both ends of the spectrum of highly automated versus manual, the best analogy that comes to mind is driving a manual/stick versus an automatic car. If you know how to drive a manual, you can adapt to an automatic very quickly. The reverse, of course, is not true.
There are, of course, many benefits to automation such standardisation, measurability and the list goes on. Plus cuurrently we have this sweet spot
where the workforce contains several generations who have experienced both very manual and highly automated processes. This dual experience is invaluable for investigation and continuous improvement. It makes me wonder what will happen when the workforce consists entirely of operators and engineers who simply press start most of the time.
I drive a manual Mazda 3 because I like the greater feeling of connection to the machine, but also because I think it makes me pay more attention. When I rent an automatic, it feels like a (literal) blunt instrument in comparison. I understand the appeal of (semi-) self driving, but the whole idea still makes me nervous. I can't really imagine sitting in a car doing nothing for long stretches of time but still being alert and ready to take control at any random moment.
It's interesting living through all this civilizational flailing. Remarkable. I read that when people invented writing it was very controversial because the facility of memory was the prized intellectual capability of the time. I never thought I'd see this kind of thing in person
I'm not convinced that this is necessarily always a bad thing on balance. It seems to me that automation in safety critical situations might make things worse when they finally go wrong but also that the number of times that they go wrong at a lower level is so much reduced that the result is net positive.
Also if you know that automation induced complacency is a thing then it must surely become a target for training, surveillance, and adaptation not mere hand wringing.
I think the big question here is how effective can you make training and monitoring across a widespread population in practice?
In aviation, commercial pilots have very strict and extensive training and monitoring and as a result are generally able to utilize automation effectively while keeping up their manual skills. There are very rarely CFIT incidents in major commercial airlines.
The opposite is true in general aviation (small private Cessnas, etc), where it’s extremely common for pilots to buy more plane than they can handle and then rely on automation to bridge their skill gap. CFIT is much more common in general aviation, along with incorrect actions in response to real system failures that should have been recoverable. Automation complacency regularly kills in general aviation.
A key thing to notice is that automation isn’t outright prohibited in either commercial or general aviation, but there are distinct regulatory frameworks based on potential impact.
We accept looser rules for general aviation because the failures are societally less severe and because the population is much larger so effective training and enforcement would be significantly harder. In commercial airliners where failures are catastrophic, we have much stricter policies and require training and testing regularly to avoid automation complacency.
Will we start to see this practice in software? Probably, but only if/when the societal cost of NOT doing it becomes more clear. We regulated aviation because crashing planes are obviously bad. We license structural engineers because collapsing bridges are obviously bad. Will automation-induced software failures hit a similar tipping point?
Am I missing something? Is TFA only 2-3 paragraphs of a generic metaphor, with no actual data/research from aviation (or other fields) to back up the core thesis?
In Munich, the U-Bahn subway normally drives semi-automatically (i.e. the driver only opens and closes the doors, then hands over control to the computer), but every driver has to regularly drive without automation assistance ("Fahren nach ortsfesten Signalen") to practice.
The crew and the plane are a single system. It is meaningless to imagine pilot “skill” without the plane. Further, we killed so many pilots in the 20th century through impossible workloads which we have now automated, it’s almost cruel to be wistful for it.
I know this is an analogy to AI, but I wish we would dispense with this idea that there’s some appropriate level of machinery which was reached just a hair before right now. There is no appropriate level of machinery, no point at which the nature of the system itself will unambiguously say “that’s enough.”
And that's the whole freaking point. If the humans have to obtain and maintain the exact same skills as before than what's the point of automation?
Actually this could be a side channel to measure the efficiency of automation. If the human operators are just as good in every aspect as before, we know all the money put into automation is wasted.
This is the concern I have with all the driving aids: adaptive cruise control, automatic braking, lane departure assist, et cetera. It all leads to drivers paying less attention to the road and more attention to their @#$%&*! phones because "Muh car drives itself."
Human drivers have been day dreaming for decades. It is not hard to move your car down the road without paying attention - until you run right through a red light or worse into a car.
American Airlines captain Warran VanderBurgh once called this phenomenon "the children of the magenta"[1] in his talk on automation dependency in the 90s.
I wonder what we'd call the children today in hindsight and what line they're chasing now...
The more the pilot flies, the worse the pilot- long distance vs short distance, can create the absurd problem that you forget how to land and start, the most important pilot tasks, while flying long haul.
> Automation doesn't make operators more careful. It makes them forget how to be. The more reliable the system, the less ready the human.
The entire premise of a system is that it removes the need for careful attention.
system: signal lights tell me whether or not I can pass through an intersection, so that I do not have to attend to potentially high speed traffic from a variety of directions.
system: the side my knife blade sits on my arched guide fingers, so that I do not have to attend to the edge of the blade or the location of my fingers.
etc etc.
>The entire premise of a system is that it removes the need for careful attention.
I think this premise is flawed or, at best, too narrow. A system is just a logical grouping of items that perform a function. Sometimes that function can be to reduce cognitive burden, but it doesn't have to be. A "vision system" like what humans use does not reduce attention, but increases/enables it, while a autonomic nervous system can reduce attention. The ability to increase/reduce attention is not the central principle of a system.
What I'd say you're pointing out is that the word "system" is overloaded.
A vision system does allow you to pay less attention: you don't need to carefully remember how far away the door is, you just need to look! I tried this often as a kid: if you want to navigate a hallway with your eyes closed, you need to pay far more attention to your other senses than you need to pay with your eyes -- where attention here is not the volume of data, but rather the complexity of conscious bookkeeping -- I can (ironically) "play it by ear" with my eyes open, but eyes closed I must plan every step!
It just so happens to be that the ability to pay less attention makes more things possible and hence the demand for attention overall may increase -- if not intrinsically, due to your competitors (who can also see!)
I would argue this take conflates attention with cognitive overhead required because of a lack of training. Navigating with our closed feels like it takes more attention because we’ve practiced so many hours navigating by sight that it no longer feels cognitively burdensome. A bat would have no trouble navigating without sight for the same reason. I don’t think most people would say giving up our sight for echolocation would reduce our attention, it just transforms it.
> system: signal lights tell me whether or not I can pass through an intersection, so that I do not have to attend to potentially high speed traffic from a variety of directions.
You know I noticed this... I lived in a country where people obey traffic laws, and in a country where they very much don't.
I witnessed many more traffic accidents in the country where people are used to relying on the traffic lights to tell them if it's safe or not.
Whereas in the other country, everyone correctly assumes that the other drivers are completely insane, and so they stay vigilant.
Other than the data of road fatalities that disproves this anecdote, my own anecdote is this is the false sense of security people get in other countries that don’t have traffic laws. Oh see the people have to look all the time so it’s much safer. When you start to live it for a long time you realize it’s not true. Many more fatalities.
Now I do think the science shows if you design roads and systems to make drivers more thoughtful it can improve outcomes. Size roads for the speed limit, roundabouts, etc. these can make a difference as it balances the system.
Having lived, driven, and crossed roads in both -- what I find is essentially that drivers from poor systems pay far more attention, but the system is a lot more effective than attention.
The difference here is one of stability: in a developing country, I can just walk across a street (often there is no traffic light) by essentially signalling with my body language -- both I and the drivers are paying attention. And if one party fails, the other has a good chance of catching that mistake.
Now, in a developed country, neither side is paying attention. If I walk across the street, I'm in danger, no matter how clear my body language (I tried it on British streets a few times -- it works in some areas, but usually very poorly!), and no one expects a crazy driver to come barreling through a red light.
The developing countries fall behind because in the crazy * sane intersection, sometimes the sane person is just not fast enough -- whereas the crazy * crazy intersection is extremely dangerius and happens often enough.
On the other hand, a developed country makes every interaction sane * sane regardless of the personalities or moods of those involved -- but God forbid a bit of crazy leaks out!
>Other than the data of road fatalities that disproves this anecdote,
You can't make that assertion (well you can, it's called "lying with statistics" but that's beside the point) without knowing if the fatalities the result of the accident rate or just a higher conversion ratio as a result of reduced safety equipment, reduced seatbelt usage, more motorcycles, etc, etc, worse emergency services, etc, etc.
INB4 other people start whining on your behalf, I'm not saying those countries aren't less safe to drive, just that you can't do a straight comparison of accident rates and fatalities without considering the conversion ratio.
Of course you have to consider confounders. That’s why transportation data usually includes best efforts.
But at some point you have to look at the totality of the evidence. Countries with better road infrastructure, enforcement, vehicle standards, and driving behavior generally produce better safety outcomes. The fact that multiple factors contribute doesn’t make the observed outcome meaningless.
As I already stated there is absolutely systems that increase the perceived sense of risk that can help outcomes (road width sizing, roundabouts, minimal signs/lines) but those typically work best in a system where there is already some sense of order.
Less Reddit style snark would go a long way too.
Whether you witness something or not is a function of a ton of other things too, so much that it makes your anecdotes useless if not actively harmful.
For example, if you live somewhere where you use the highway more often, that sure as heck can skew the result.
Or if you live(d) somewhere where people tend to hit and run instead of waiting... you're obviously not going to witness them as often.
Also, note that accidents and injuries are not the same thing. You can totally have fewer accidents but more injuries or fatalities.
Without knowing the neighborhoods you've lived in (so people can compare the data for themselves) you're really not going to make a compelling case.
There's some documented studies of removing all the street clutter and lines from residential area intersections forcing drivers to be more careful, especially around pedestrians, reducing overall accidents. But this does reduce throughput slightly.
This is an example of risk compensation. When people perceive greater protections around themselves, they tend to become more aggressive at the margin, such as with the driving habits that you mentioned or hitting more violently in American football because of improvements in helmets and padding.
https://en.wikipedia.org/wiki/Risk_compensation
Football might be a weak example, because being able to hit harder is an overwhelming competitive advantage. A player who acted like they were not wearing a helmet would be effectively dysfunctional.
In contrast, most careless driving habits don't actually get anybody to their destination any quicker.
Update: someone replied below with a link to traffic deaths statistics. Turns out the data shows the opposite of what I witnessed.
The insane driving country has double the traffic related deaths as the chill, lawful driving country.
Except if you look at this map: https://en.wikipedia.org/wiki/List_of_countries_by_traffic-r...
People die on road more in countries that conventionally don't follow traffic laws.
Not sure why this would downvoted. Go to a less developed nation where traffic laws are not important and it’s one of those sense of false security ideas.
When I lived in India everyone would always tell me how everyone drives so much safer there because they're more aware etc (similar reasoning as we're seeing in this thread), but man, the national crash statistics say otherwise.
Not to mention I lost count of how many dozens of accidents I witnessed in my year there. I've personally been in 3 rickshaw crashes.
My experience as well in Vietnam. the first time I was there I figured they were right but it’s just a false narrative people sell to make themselves feel safe. Don’t even get me started on methed out American size semi trucks speeding with no concern of running you over.
>signal lights tell me whether or not I can pass
No they don't, they tell you and other vehicles to stop. You would fail your driving test if you depend only on the traffic lights and don't bother to verify it is safe to pass yourself.
The main function of a traffic signal is the green phase, not the red phase. A traffic signal increases throughput by allowing drivers to ignore crossing traffic.
(If safety/the red phase was the purpose, the intersection would use a roundabout instead.)
I mean, it depends on where you take your driving test. In a lot of places in the US (especially in some rural areas), you may still pass. In some cases you might not even drive near a stoplight during the test.
If you "know a guy" you can even pass a driving test without ever getting behind the wheel of a car. Road licensing is in complete shambles in the last 10 years. A lot of "workarounds" and corruption.
You definitely still should be paying attention to cross traffic, regardless of what the lights indicate. The lights just make it easier by stopping most traffic for you, so you only have to do a quick scan for outliers.
The distinction is subtle.
Someone learning to fly may be described as paying careful attention: to every little sound, vibration, and sensation. A common tactic by student pilots is overcontrolling the aircraft, e.g., large sudden changes rather than smooth pressures from flying with a light touch.
Automation requires active, intentional attention particularly when flying in clouds. What are my instruments telling me? Are they all telling the same story? Have any failed? Which ones?
A significant part of flight training and testing emphasizes the ability to divide attention between multiple competing needs, being able to correctly prioritize them, and responding promptly and safely in order of priority.
There's been many studies around the concept "how do you get expertise? In what field can you actually develop expertise?" such as Naturalistic Decision Making [1] [2]. The wisdom from there can be applied to the question of automation.
If your task is for example, to pilot a preset route with stable condition and very low surprise, you will fall for the "getting too comfortable" trap and we tend to start to get lazy(or efficient) and offload the mental effort and skills atrophy. A common workaround to this is to have regular training(deliberate practice) that introduce the "tricky" situation to keep the skill up. Problem ofc arises when people don't keep this up.
This can be seen in the diagnostic performance differences between junior and senior doctors, not always in the favor to the senior [3]. If you add a layer of automation but the insight gathered by working on that layer is great (and falls off) then deliberate practice start to become a requirement
[1] https://commoncog.com/putting-mental-models-to-practice/
[2] https://youtu.be/5eW6Eagr9XA?si=Y9exacaW-F4PDOKF
[3] https://pubmed.ncbi.nlm.nih.gov/26375267/
I don't think we believe that the automation makes the human operators safer, but that it is overall safer than the human operator alone.
I think part of the point of the article is that it also makes edge-cases more dangerous and catastrophic than if there was no autopilot at all. From the article:
>The argument for automation is that it frees up cognitive bandwidth. Fewer routine decisions means more headroom to think carefully about the ones that matter.
So if the expectation is that the human pilot is expected to pay attention to mitigate the dangerous edge cases "that matter", there is a contradiction: the tool that promises to free up the bandwidth for that attention creates a complacency that prevents that attention from being applied.
In other words, it makes the normal situations safer but the abnormal situations more dangerous.
I remember reading an internal memo something to the effect that with the assist systems off there were large groups of pilots that couldn't physically control the aircraft to the point of landing, they just didn't have the strength for it because the physicality had stopped being a day to day aspect of the job.
Landing is an abnormal situation for an aircraft which we make SUBSTANTIALLY less dangerous through intense automation. Do you want to rip out automated landing systems?
The FAA describes taxi, takeoff, landing, and operations other than cruise flight below 10,000 MSL as critical phases of flight because of increased risk. The aircraft is closer to the ground, other aircraft, and hazards such that prompt, correct responses are essential to the safe outcome of the flight.
Any equipment on the aircraft can and will fail. Becoming dependent on autoland — not a worry on most general aviation aircraft — is terrible risk management. Every pilot must maintain hand flying skills. Automation is nice and reduces workload, but the pilot must actively manage it.
Not only is landing not an "abnormal situation", contrary to armchair internet wisdom pilots of airliners in fact do not use autoland all the time and don't even always fly a precision approach at all.
Not to mention that they get mandated regular reviews of their ability to fly manually. And even with that, there's still a reason why "children of the magenta line" (i.e. pilots who passively follow automated systems into danger and/or have seriously degraded stick-and-rudder skills) has become a term.
2 things. First, landing a plane is abnormal! You're shedding a huge amount of energy and you're transitioning from a state where you can keep flying due to your speed to one where flight is impossible as you lower speed. That's an abnormal state to put an aircraft in, regardless of how often it happens. Second, what exactly is the level of automation you're saying is not necessary? Should we rip out radar systems that mark glide paths?
You land a plane as part of most flights, I'd say that's not particularly abnormal as an event...
Very few flights end without a landing.
This is about as silly as feverishly claiming that e.g. deploying a web app to production is an "abnormal situation".
On top of that I'm sorry but you seem to have skimmed over both the article and what I said in favour of clutching pearls at some nebulous entity apparently claiming that "automation should be ripped out" when what is actually being explained to you is that without actual, manual, hands-on, current experience the "human in the loop" loses the ability to properly control or take over from an automated system - and worse, the ability to even understand when it is doing something nonsensical and/or dangerous.
As an aside, I assume that by "radar systems" you are referring to radio navigation aids. Like I've already mentioned (though in fairness not everyone knows what a non-precision approach means), pilots of airliners are still trained to fly without them, are expected to know how to fly without them, and shockingly enough DO fly without them in the real world where equipment fails or cannot even be installed at all. I know most of the software that people write here is insulated by several layers of abstraction from the hardware, but surely we haven't already lost the understanding that automated systems are not in fact magic - that they depend on real world hardware with real world physical constraints?
You seem to confuse "high consequence" with "abnormal".
I mean you confused the 737-max problems with the general notion of automation, so...
First, snark goes against HN guidelines so you might want familiarize yourself with them.
Secondly, MCAS autonomously adjusts trim based on sensor inputs to avoid a hazard. It is not advisory and directly controls flight surfaces. This would make it automation according to how organizations like NASA categorize flight software taxonomy.
> That's an abnormal state to put an aircraft in, regardless of how often it happens
Taking off, flying, and landing are all absolutely required in the normal operation of a plane. If your plane is not engineered in such a way that landing is normal, it won't last long
If you expect every aircraft to land, it seems to meet the very definition of "normal" operation.
An abnormal landing would be something like trying to land with a broken elevator surface.
The circumstance doesn’t have to be that dramatic to be abnormal.
Landing after a merely unstable approach, too many significant changes too close to landing, increases risk.
Landing too fast may result in overrunning the end of the runway, pilot induced oscillation, or loss of control. Energy being proportional to the square of velocity means the margin doesn’t have to be huge to pose significant danger. Landing too slow risks an aerodynamic stall or worse a spin, which at low altitude is nearly certain to be fatal.
Landing safely with a crosswind requires technique changes. Too much crosswind or “running out of rudder” is extremely dangerous.
Landing after accumulating airframe icing is triply bad because the ice reduces the control surfaces’ aerodynamic effectiveness, makes the airplane heavier, and requires a faster landing.
And you contend that autopilot makes that situation more dangerous? Do you have any support for this?
As the article already states, there is a well known phenomenon in aviation called automation-induced complacency. So, yes, if you automate landing to the extent that human pilots no longer pay attention to abnormal signals that indicate something is wrong, or no longer feel the need to train or stay vigilant, it can make things more dangerous. There is plenty of research on this, but here's the first that came up in a cursory search:
https://ntrs.nasa.gov/api/citations/20020021642/downloads/20...
A more recent example is the Boeing 737-Max where there was a focus on automating trim control. In that case, the automation made the system more complex, to the detriment of a pilot understanding and reacting to an abnormal operation.
We should also be careful that we don't create a false dichotomy between "all automated or no automation", or an expectation that more automation is always better. The goal should be the right balance that increases reliability/safety.
> A more recent example is the Boeing 737-Max where there was a focus on automating trim control. In that case, the automation made the system more complex, to the detriment of a pilot understanding and reacting to an abnormal operation
To be fair this is not entirely accurate: a focus was made on stall prevention in a very specific mode of flight given the variant's increased susceptibility to the pitch-power couple. It did not make the system any more complex per se than other airliners - see e.g. Airbus aircraft which do actually have autotrim in normal flight. The actual kicker was that the existence of MCAS was hidden to avoid the need for lengthy re-training of pilots if the 737 MAX was deemed sufficiently different from its predecessor variants (on top of MCAS being rather poorly implemented in its first iteration).
Fair enough. The “hidden” aspect is what I was alluding to…ie, control that exists but isn’t apparent to the pilots (and worse, intermittent). In the human factors world, it was more complex than the pilots assumed, but you’re right that it’s probably not the best description.
(As an aside, the hazard being mitigated, ie stall, has little bearing on whether or not it’s autonomous or complex, although it does impact whether its safety critical)
The fact that the autopilot will loudly disengage if there is a serious enough control surface failure to cause an upset is more than enough support IMO.
> I don't think we believe that the automation makes the human operators safer, but that it is overall safer than the human operator alone.
If the automation is for the easy/routine stuff, then no. The automation doesn't work in exactly the most safety-critical situations, and then the human operator is thrust into fixing the situation without the full context.
I don't know about flying, but in machining it is the easy/routine stuff that will take your hand off. New machinists rarely get the serious injuries that more experienced ones do (they get more minor injuries) - the experienced machinist gets complacent about how dangerous the machine is and stops thinking to keep their hands away until they lose it. Automation has saves a lot of limbs (and lives) because you don't need to get close to the moving parts anymore.
Well, it's differently safer. It's impossible to make categorical statements of more or less safe with automation because it depends massively on the design.
ijterestingly, AI tools are improving code theoughput but also elevating catastrophic error rates.
even if the average rate goes up tor net benefit, are organizations prepared for increased carastrophic failures?
For sure I not think you can compare agentic coding to automation in this context
Sure you can; right now, planes fly 90% on automation. They fall back to pilots to correct issues, engine outs or touchdowns. Right now, that fallback condition is mostly reasonable: pilots are trained extensively and have thousands of hours of flight experience to manage it. But as that automation approaches 100%, that fall back condition crosses a line where it's more likely the pilot is both not well trained because of it and their instinct in an emergency is entirely untrustable.
So too is what's happening with LLMs: they're writing code that the programmer is increasinging unaware of and the programmer is increasingly not capable of understanding because they dont have the experience of writing and navigating complex codes. So in the event of a permissions prompt, the fallback condition, you have the same race condition between increasing automation removing the knowledge generation of the operator.
So there's obviously a rubicon where automation _has_ to be 100% because no operator can be a fallback.
[flagged]
Also, ironies of automation[1], etc.
[1] https://static1.squarespace.com/static/644321e78cd2dd37613af...
Automation should always be the serialization of understanding of the system, which can execute on its own under normal conditions. When it breaks, your reaction should not be removing the automation and asking humans to get back to manual operation, that is the wrong approach. Instead, you should ask humans to leverage their understanding to step in, reason about the automation system, and fix it.
This is how I have been doing SRE for the past decade. And the inability to practice this under-the-hood reasoning is the main reason why I don't trust any modern AI based automation systems. They are helpers for some manual work, but they can't be automation.
There are two other solutions, make the automation so reliable that the human skill is not needed, or make the consequences of failure low enough that you can just accept failure.
As far as I know, Places that actually care about safety apply all of them at once.
They add layers of failsafes that don't rely on humans, they make the automation better and better, and for the most critical stuff, they continue training for things that will almost certainly not happen.
And for the rare(depending on personality) cases where we care about the pilot's ability more than the result, just don't automate at all.
From my manufacturing experience on both ends of the spectrum of highly automated versus manual, the best analogy that comes to mind is driving a manual/stick versus an automatic car. If you know how to drive a manual, you can adapt to an automatic very quickly. The reverse, of course, is not true.
There are, of course, many benefits to automation such standardisation, measurability and the list goes on. Plus cuurrently we have this sweet spot where the workforce contains several generations who have experienced both very manual and highly automated processes. This dual experience is invaluable for investigation and continuous improvement. It makes me wonder what will happen when the workforce consists entirely of operators and engineers who simply press start most of the time.
I drive a manual Mazda 3 because I like the greater feeling of connection to the machine, but also because I think it makes me pay more attention. When I rent an automatic, it feels like a (literal) blunt instrument in comparison. I understand the appeal of (semi-) self driving, but the whole idea still makes me nervous. I can't really imagine sitting in a car doing nothing for long stretches of time but still being alert and ready to take control at any random moment.
>The reverse, of course, is not true.
5min to learn. A week of normal driving to get not bad. I wouldn't say it's hard.
Yes, but the magnitude of the difference is clear.
How long did it take you to understand what good code/engineering designs looked like. How long to debug tricky bugs?
It certainly takes more than a week!
It's interesting living through all this civilizational flailing. Remarkable. I read that when people invented writing it was very controversial because the facility of memory was the prized intellectual capability of the time. I never thought I'd see this kind of thing in person
They should get an auotpilot to fix their website to render on desktops and not just phones.
Just `commmand` + `+`
I'm not convinced that this is necessarily always a bad thing on balance. It seems to me that automation in safety critical situations might make things worse when they finally go wrong but also that the number of times that they go wrong at a lower level is so much reduced that the result is net positive.
Also if you know that automation induced complacency is a thing then it must surely become a target for training, surveillance, and adaptation not mere hand wringing.
I think the big question here is how effective can you make training and monitoring across a widespread population in practice?
In aviation, commercial pilots have very strict and extensive training and monitoring and as a result are generally able to utilize automation effectively while keeping up their manual skills. There are very rarely CFIT incidents in major commercial airlines.
The opposite is true in general aviation (small private Cessnas, etc), where it’s extremely common for pilots to buy more plane than they can handle and then rely on automation to bridge their skill gap. CFIT is much more common in general aviation, along with incorrect actions in response to real system failures that should have been recoverable. Automation complacency regularly kills in general aviation.
A key thing to notice is that automation isn’t outright prohibited in either commercial or general aviation, but there are distinct regulatory frameworks based on potential impact.
We accept looser rules for general aviation because the failures are societally less severe and because the population is much larger so effective training and enforcement would be significantly harder. In commercial airliners where failures are catastrophic, we have much stricter policies and require training and testing regularly to avoid automation complacency.
Will we start to see this practice in software? Probably, but only if/when the societal cost of NOT doing it becomes more clear. We regulated aviation because crashing planes are obviously bad. We license structural engineers because collapsing bridges are obviously bad. Will automation-induced software failures hit a similar tipping point?
In this case the only reason I'm a pilot is the existence of autopilot haha
Am I missing something? Is TFA only 2-3 paragraphs of a generic metaphor, with no actual data/research from aviation (or other fields) to back up the core thesis?
Well, don't discount the equal number of affiliate links to tryware.
In Munich, the U-Bahn subway normally drives semi-automatically (i.e. the driver only opens and closes the doors, then hands over control to the computer), but every driver has to regularly drive without automation assistance ("Fahren nach ortsfesten Signalen") to practice.
The crew and the plane are a single system. It is meaningless to imagine pilot “skill” without the plane. Further, we killed so many pilots in the 20th century through impossible workloads which we have now automated, it’s almost cruel to be wistful for it.
I know this is an analogy to AI, but I wish we would dispense with this idea that there’s some appropriate level of machinery which was reached just a hair before right now. There is no appropriate level of machinery, no point at which the nature of the system itself will unambiguously say “that’s enough.”
And that's the whole freaking point. If the humans have to obtain and maintain the exact same skills as before than what's the point of automation?
Actually this could be a side channel to measure the efficiency of automation. If the human operators are just as good in every aspect as before, we know all the money put into automation is wasted.
This is the concern I have with all the driving aids: adaptive cruise control, automatic braking, lane departure assist, et cetera. It all leads to drivers paying less attention to the road and more attention to their @#$%&*! phones because "Muh car drives itself."
Human drivers have been day dreaming for decades. It is not hard to move your car down the road without paying attention - until you run right through a red light or worse into a car.
[dead]
American Airlines captain Warran VanderBurgh once called this phenomenon "the children of the magenta"[1] in his talk on automation dependency in the 90s.
I wonder what we'd call the children today in hindsight and what line they're chasing now...
[1] https://www.youtube.com/watch?v=5ESJH1NLMLs
Yet another AI written slop article hitting the top of HN.
Evidence: Look at the most recent article on this blog: https://julienreszka.com/blog/difficult-conversations-don-t-...
"Memory is reconstructive. When someone recalls events differently, it feels like gaslighting. It usually isn't. Document first, then negotiate."
Bleugh
The more the pilot flies, the worse the pilot- long distance vs short distance, can create the absurd problem that you forget how to land and start, the most important pilot tasks, while flying long haul.
https://theaircurrent.com/aviation-safety/emirates-777-ek231...