Dissecting the End of the World
I’m sure you’ve all heard by now that, on Saturday, January 13th, Hawaii’s Emergency Alert system sent out an SMS message to everybody that said:
Emergency Alert
BALLISTIC MISSILE THREAD INBOUND TO HAWAII. SEEK IMMEDIATE SHELTER. THIS IS NOT A DRILL.
A half hour later, a message was sent out that this message was a false alarm and there was no missile attack at all. (You can read Michelle Broder Van Dyke’s twitter thread that has pictures and everything here.)
Now, according to the governor, this was caused by an employee pushing the wrong button. Here’s the quotation from the article:
“It was a mistake made during a standard procedure at the change over of a shift, and an employee pushed the wrong button,” he said.
I am very relieved that the end of the world appears to be delayed for a couple more weeks but I have questions about this sort of thing and find it far too easy to question the official narrative.
Here’s what we know: a message saying “missile warning, this is not a drill” got sent out and it took 38 minutes to send out the message that said “whoops, yeah, this is a drill”. When asked about it, the authorities said that this was a mistake that happened when an employee pushed a button.
So let’s look at the possibilities:
1. The message was sent out mistakenly
2. The message was sent out deliberately but the sender knew that there wasn’t a missile
3. The message was sent out deliberately but the sender thought that there was a missile
4. The missile was sent, exploded, and we’re all living in a simulation or dead and in the afterlife or something
So let’s look at the first one:
As buttons go, having a single button that sends out a *VERY* specific message dealing with missile attacks makes me wonder exactly how many messages they have assigned to how many buttons. I mean, if a button got pushed that sent out an email that said “there’s a storm coming and it’s going to be a doozy, get to shelter”, that would make sense to me. Hawaii probably has doozies of a storm often enough that you’d want a message like that to be sendable with a single push of a button.
But a missile attack?
And, on top of that, it’s a button that can be pushed accidentally during a routine shift change? Maybe we’d want a plastic thing over that button that you have to flip up before the button is pressed. Perhaps two buttons (on two different sides of the room) that require being pressed at the same time. Maybe a key that has to be turned before the button is pushable.
I think that we’re lucky that the mistake does not yet appear to have a death toll because I could easily see how something awful could have happened by panicky people responding to a “THIS IS NOT A DRILL” message. Though I suppose that some of the panicky responses could result in a net positive number of lives due to the message… something for an intrepid journalist to investigate 9 months from now.
But if this was a mistake, there needs to be a full accounting, with maps, and descriptions of fixes, and somebody getting fired. This has created one heck of a callus for not only Hawaii but the entire country: the next time that there is an SMS message being sent out saying that there is an attack incoming and THIS IS NOT A DRILL, there will be a huge number of people who just won’t believe it.
So let’s look at the second one:
Off the top of my head, there are a couple of ways that a message could be sent out deliberately with the sender knowing that there wasn’t a missile. The first is that it was a “prank” by a stupid/malicious actor who hacked into the system. The second is that it was a “prank” by a stupid/malicious actor who had legitimate access to the system.
Given that the former exposes a *HUGE* vulnerability, the best response is the government covering up by saying something like “yeah, Joe Blow bumped the button with his elbow during shift change. Oh that Joe!” and figure out, right freaking now, what other systems have these vulnerabilities and patch those holes closed as quickly as possible and start pouring money into computer security.
The latter strikes me as being less likely for the authorities to be willing to cover up (but, I suppose, theoretically possible). If you have a prankster, it’s going to be turned into political hay by somebody. Better to deny the political hay and have the official story be Joe Blow “accidentally” bumped the “button”. Then you fire Joe if he’s firable and you transfer him if he’s not.
And the official story can remain that this was not done deliberately.
The third one is where we *REALLY* get into tin foil hat territory.
Somebody thought that there was a missile. The only proper response is to send out the alert. Maybe it’s because there *WAS* a missile! And our Star Wars program shot it down! But if The People knew that there was an attack, it’d demand a response! And so in order to prevent megadeaths (or even gigadeaths), we say that the SMS message was sent out in error. Whoops. Oh that Joe. Then we go to bed knowing that we saved the world until the next time.
(This strikes me as really, really, really, really unlikely. For one, South Korea and Japan would have known that there was a missile and this would have been all over Korean and Japanese twitter a few minutes before the SMS message went out. Korean and Japanese twitter was not all over this a few minutes before the SMS message went out. Therefore, there was no missile. Q.E.D.)
But maybe they thought there was a missile *MISTAKENLY*. Like they thought that a plane or a drone or a UFO or an undigested bit of beef, a blot of mustard, a crumb of cheese, a fragment of underdone potato. They responded in the only moral way they knew how: they pushed the button and warned their friends and loved ones. And, wouldn’t you know it, it was a bird.
And, at that point, well… there were a number of institutional failures there that need protections. Two-man verification from now on. No sending the SMS unless you’ve got an okay from the supervisor and the supervisor’s manager and the supervisor’s manager’s boss too. And the official story is that the button was pushed by accident.
As for the fourth one, that strikes me as not likely at all but, hey, it’s technically a possibility.
What *REALLY* happened? Well, we don’t know exactly quite yet and probably won’t for at least a few weeks until the official narrative coalesces fully.
As for “why did it take 38 minutes to send out the correction?”, we spoke to friends who had recently taken an extended vacation in Hawaii. They shrugged and said “Island time”. So maybe that’s the explanation there.
All that to say: I’m glad it’s not the end of the world. Yet, anyway.
It probably wasn’t a button like on some control console somewhere, it was probably on a drop down list in the software they use to manage shift changes and what not, and someone had the wrong screen up, or clicked in the wrong ComboBox, or selected the wrong item in the list, and it’s wasn’t really Joe’s fault, it was whoever designed the software interface.Report
Yeah, you’re right. Here’s a paragraph from the Fox article:
So someone had the wrong screen up and clicked in the wrong ComboBox.
And then did it again.Report
Are you incredulous that he did it again?Report
If I knew how much his salary was, it would help.
If he was making $12/hr? I’d see his pushing the button a second time as pretty much inevitable and would immediately begin wondering why it took so long for this to happen.
If he’s a salaried employee who had to do stuff like pass a background check to get his job?
Yeah. I’m willing to be incredulous at that point.Report
Having done end user support for long enough (a decade or so) amongst people, young and old, who were ostensibly the best and brightest, I’m not incredulous at all. A badly designed GUI can readily trip up the smartest kinds of people, and badly designed GUIs are a chronic problem, especially in software marketed to government agencies, or built to spec for them.
So, I want to see the UI, not the employee stats.Report
If the second push is
Confirm? Yes/No/Cancel
as opposed to
Type the Name of the Emergency and Hit Enter
I can see it very easily.Report
This isn’t even the first time. In 2005, an operator error caused an evacuation order for the entire state of Connecticut to go out. I have seen speculation that some of the state systems are running 20+ year old software.
When I was a legislative budget analyst, it had become unfortunately common to see policy changes voted down because of the cost of putting those changes into the software systems (think tens of millions of dollars). Trump’s infrastructure dollars could possibly produce greater benefit if they were routed to the states for software replacements.Report
Yes, this. A long shift, too much coffee, a bad UI — really there is no need for conspiracy theories here.Report
Suspiciously implausible. System was hacked. QED. 🙂Report
And actually, I’m not kidding except for the QED. As greg notes below, there’s not only no reason (a priori!) to assume the system wasn’t hacked, we’ve been presented with lots of evidence (or reports of lots of evidence…) that foreign state actors are actively trying to hack into exactly these types of US state and federal systems. So the rational explanation given that we don’t know all the facts will be a probability assignment of each account based on what we do know.Report
The two things that make me most suspicious: the 38 minute response time and the somewhat wacky phrasing of the original message.
As Kohole points out, “this is not a drill” doesn’t sound like official language (I think it sounds like something that a prankster would add). The 38 minute response time makes sense if nobody was at the console and then had to get to it… less sense if someone was sitting at the console and pressed the button himself twice.Report
The 38 minute reponse time makes sense in this scenario –
Shift change, *saturday morning*. So the ‘regular’ people aren’t at work.
Someone errs right as they are walking out the door, and nobody catches that they send a real world outgoing alert. (I have no idea how this system works, but there are similar systems that have ‘safe’ or ‘training’ modes where it’s not necessarily clear to certain users if you are in a training mode or not.
There possibly are multiple locales that can send such an alert (e.g. a terminal at the Tsunami Warning Center, a terminal at Hawaiian Civil Defense HQ, a terminal at one of the military commands on the island).
So when something went out, it takes a while for the watchcenters to know something went out. They don’t have their personal cellphones on them for security and paying attention reasons. The bosses at home do get an alert, (eventually), they call into their respective people, but it takes a while to determine who sent the alert, and to verify with ‘ground truth’ watchstanders (who only a limited number of people have access too) that indeed, the status boards are all green and nothing is actually hapenning in the real world.
I could easily see this entire process taking 38 minutes from the time the alert went out, until when the governor has enough info to verify ‘all clear, false alarm’.
Remember, it was a Saturday.Report
Re the shift change hypothesis: why would the State run a test of the emergency systems during a shift change? Does that seem plausible?Report
People on the previous shift doing a favor (or following procedures) to queue stuff up for a drill to be run on the next shift.
People on previous shift scheduled to run a drill and had stuff queued up but the drill was cancelled because of whatever reason.
People futzing around on the previous shift, loading message into the bin for training/demonstration (or for sh**s & giggles) and not clearing the buffer when they were done.
In either scenario that stuff could have been activated during the shift turnover when it wasn’t supposed due to not following correct turnover procedures or a flaw in the procedures that allowed this message to go live for real.
Further thought. Someone on previous shift showing an FNG on next shift ‘this is how you do this’ and it went too far.Report
Also, you sometimes run a drill near a shift change because one shift is the one being drilled, and another shift is also on site conducting, monitoring, and evaluating the drill.Report
OK, not implausible. Thanks.Report
Running one? No. Ending one? Yeah.
Say it’s the last test run of a long list (like, every part of the system) because it’s the least probable and the least used.
Then there are multiple delays. Joe didn’t know it was “live”. Jim didn’t know where the “kill this” switch even was (an even more unused part of the system), or maybe he even doesn’t have security access to this part of the system.
38 minutes is long enough for someone to walk out to his car, turn on the radio, realize what he’d done (panic on live news), and then run back to his desk.Report
Ok scratch that. The news is saying now that Hawaiian officials knew everything was ok in 3 minutes.Report
Mr. Miyagi needs to send that employee back to waxing the cars and painting the fences.Report
I sort assumed that “pushed the wrong button” is managerial shorthand for “opened a couple of drop down menus, pushed the wrong buttons or three, and didn’t notice the difference between menu (a) and (III) argle bargle…:” since I doubt there are many systems left that are analog and are physical buttons.Report
“Pushed the wrong button” is something that gets me to say “hey, that could have happened to anybody”.
“Opened a couple of drop down menus, pushed the wrong buttons or three, and didn’t notice the difference between menu (a) and (III) argle bargle” does not get me to say “hey, that could have happened to anybody” but “what the hell is going with training and procedures over there?”Report
Well, yeah, its a very good question as to who, and how, and under what circumstances a ‘”OMG WE ARE AT WAR !! WOOP WOOP WOOP” message is allowed to be sent;
Like I mentioned below, I don’t get why any local entity is empowered to unilaterally decide to send this kind of message, even if they sincerely believed it to be real.
I mean like,even since ancient times the military has an entire command and control system to control who is allowed to alert the troops and assemble them into fighting formation.Report
I’ve made wrong selections on drop-down menus before, but in my case, that meant the wrong statistical test got run, I cursed about it, and went back and did the right one. But if clicking the wrong thing on a drop-down menu would, for example, erase months of data I had spend weeks entering? I’d make DAMN sure of what I was doing before I clicked anything.
That 38 minute gap is what I find so horrifying. I can’t imaging what would have been going through my mind, had I been in Hawaii. I hope no one died (heart attack, stroke….) as a result of the stress of those 38 minutes.Report
I did ponder that there may be a spike in births in 9 months.
There have been a bunch of report that Russian associated hackers have been working at getting access to more than just e-mail accounts.
https://techcrunch.com/2018/01/12/russian-hackers-senate-pawn-storm-fancy-bear/
Or course that is just the senate.
Allthough in this bit, in the middle of the piece:
“The U.S. is vulnerable in other areas, too. When Attorney General Jeff Sessions testified before the Senate Intelligence Committee in June, Senator John McCain turned his attention to an even more worrisome possibility: “Quietly, the Kremlin has been trying to map the United States telecommunications infrastructure,” he said, describing a series of steps hackers have taken to develop “a cyber weapon that can disrupt the United States power grids and telecommunications infrastructure.” When McCain asked Sessions if the administration had a plan to deal with such an eventuality, Sessions admitted that it did not.”
https://www.vanityfair.com/news/2018/01/russian-hackers-may-be-preparing-another-major-us-attack
So that is all scary especially since dealing with hacking by foreign actors has, perversely, become a deeply partisan issue. On the other hand i’d still bet this was a simple mistake since dumb errors are far more common than anything else in the world.Report
I said already on twitter that the biggest problem is that the combox has a setting for ‘this is not a drill’ (and/or there are procedures that allow/require you to input that)
Drill messages should be clearly marked as such, and real-world messsages should just have the message without any fluff, emphasis, or editorializing.Report
You’d think that there’d be a meeting or fourteen where they sat down and hammered out the wording of the battery of pre-written SMS emergency messages.
“How do we want to phrase the hurricane one?”
(Two whole hours devoted to whether all caps should be used, whether punctuation should be used, whether adjectives should be used…)
“Okay. Break for lunch, then back here and we’ll figure out how to phrase the missile attack messages.”Report
I would think that the alert messages for the entire US Pacific rim would have enough commonalities that Hawaii didn’t need a bespoke home grown system.
(and/or they would be the ones that innovated it and then sent it along to the other Pacific coast states)Report
I would like to know more about this myself.
Wouldn’t it make sense that any messages about foreign attack should be tightly controlled by the Defense Dept in Washington?
Its one thing to have localized conditions controlled locally, but it seems like the binary nature of “we are at war/ we are NOT at war” should be part of the command and control system.
So like, instead of a warning screen like “Are you sure?” it should be “enter the DoD authorization code”.
But then, I am not sure how it works to begin with.Report
I doubt it. An in bound missile would trump any lack of message from Washington. Hawaii is (historically) pretty isolated and could expect to know of some types of attacks (including a missile from NK) before Washington does. Think Pearl Harbor.Report
Maybe there was no alert. Mass hysteria. Caused by chemtrails.Report
Btw, this is also why ‘zombie apocalypse’ is now used so frequently in emergency response & mass casualities drills, because it can’t be mistaken for ‘the real thing’.Report
One day, ninety-nine balloons
were released into the air
and one-by-one they crossed the wall
from over here to over there
and on the other side, they saw
a blip upon a radar screen
the operator said it was
the strangest thing he’d ever seenReport
Occam’s Razor says it was a stupid mistake during a shift change and nothing more. Speculating otherwise might be amusing but it is dangerous for political sanity and amusing ourselves to death.Report
Occam’s Razor has its advantages.Report
That’s a weird response when the NatSec community and lots of Dems on the Hill think we’re potentially amusing ourselves to death by not addressing Russian efforts to penetrate US security and infrastructure systems.Report
Oh, please….every “right thinking person” knows it’s Trump’s fault.Report
Yes, yes, yes. We already have too many hair brained conspiracy theories spun out of practiced ignorance.Report
I need a minute to catch up on the narrative. Aren’t we supposed to believe the Russians interfered with our elections, going so far as to hack into DNC email accounts, various Senators personal email accounts, and try to penetrate 26 states election systems? Is that just a conspiracy theory spun out of practiced ignorance?
Maybe Trump is right about the Mueller investigation after all.Report
Last I heard, the Russians appear to have illegally placed roughly $200k(?) in advertising on FB… in an election where Billions of dollars were spent on advertising.
Did it happen? Yes. Did it matter? Probably not.
In theory that illegal advertising purchases the 20k(?) votes who swung the election. In practice we have lots of media professionals who are supposed to be good at influencing elections here in the US. If we can’t predict which 20k votes matter then I seriously doubt the Russians can.
This is independent of Trump doing something heinous because Trump is always doing something heinous and none of his people were experienced at knowing the rules. So the rules were probably broken, and I expect Mueller can find something Vile about Trump.
I also expect it didn’t really matter more than what we already know and it didn’t swing the election more than HRC’s email server and/or Trump’s twitter feed.
This fascination with the Russians is an attempt to externize an internal problem.Report
Dark,
If the Russia-Trump-election connection was limited to Russain purchases of Facebook ads you would maybe (maybe!) have a point.Report
WaPo has an article out. They’re saying that it’s a design fail where the choice “Missile Alert” was chosen instead of “Test Missile Alert”.Report
“Please choose what you want to do…
1. Text mom
2. Flick lights in bathroom to confuse coworker
3. Missile alert
4. Test missile alert
5. Launch missile
6. Cat videos”
“Don’t you think maybe we should have separate menus?”
“It’s more efficient this way.”Report
This is why you should go with the second lowest bidder instead of just the lowest one.Report
Or, if it is anything like the 90s era software the govt often uses it is a cropped dropdown and all the options start with Missile:
* Missile Launch, Test
* Missile Launch, Alert
* Missile Launch
But the only thing that shows in the dropdown (unless you select and read it) is:
| Missile Lau |
The regular guys just know to always select #1 (and most don’t even know what #3 even does).Report
One hopes they are not using 90’s era software.
One hopes they are not using stuff from the oughts/naughts/I don’t know what we’re going to call it/the decade right after the 90’s.Report
I read Van Dyke’s article and a few things stood out.
She included a video of an alert being broadcast on the TV over a basketball game. The content of that alert was different than the SMS message. So, presumably, the “wrong button” triggered some sort of chain reaction of alerts. She also mentioned that these were 2 of the 3 ways they’d be alerted, with the third being sirens. She didn’t hear the sirens but lives far from them. She reached out to others who were closed and they said there were no sirens, but apparently others have claimed to have heard them.
So this leaves me wondering…
If we accept the official story, someone somewhere pushed the wrong button. What happened next? Did that button directly cause the SMS messaging? Did it cause the television alert? Or did that button being pushed cause a light to go off or a bell to ring elsewhere and somewhere who saw that light or heard that bell then pushed a button to send the SMS message and television alert? Or did someone at the TV station see the SMS message and decide to put out a television alert? Why were the alerts different? Why didn’t (or did?) the sirens go off? If all of these alerts aren’t automatically triggered by the wrong button (and I assume they are not if the sirens did indeed remain silent) why did no one double-check that the light or bell or whatever was indeed correct before pushing their buttons to send their alerts?
I’m really curious to know more about the chain of events that begins with someone pushing a wrong button and ends with 2 alerts being sent but not a third.Report
Yeah. I’m 90% satisfied with the official explanation.
That 10%, though. Man.Report
I hadn’t even considered that the official story wasn’t more or less true until I read this post and I’m still inclined to believe it’s true or true enough.
I think it is easy to think, “HOW CAN A MISTAKE LIKE THIS OCCUR?!” But if you assume there’s probably 3 shift changes a day every day for however many years and probably lots of button pushes every shift, the error rate starts to look very tiny and probably approaching what is reasonable. Layer on what sound like genuine design flaws (which seem less reasonable) and shit happens.
So I don’t offer this to challenge the official story as much to say there is lots that is still unexplained to me (and maybe there are perfectly good answers that I just haven’t seen yet).Report
The employee who screwed up has been reassigned. That confirms that it was an employee screw up and confirms that we’re in a new era of government accountability. Previously a government employee could’ve accidentally triggered a global nuclear war and the most punishment they’d face was a paid leave of absence.Report
After all these posts, this seems obligatory:
Shall. We. Play. A. Game?Report
When I was an undergrad at UC Santa Barbara–this would have been around 1982, a/k/a the height of the Cold War, a/k/a The Good Old Days, I was up late studying in the middle of the night with the radio on. The Emergency Broadcast System warning tone came on without the usual “This is a test” language before it. That was startling. Back in those days we all knew the nearest best target for nukes. Santa Barbara itself would be a waste of a good warhead, but it is down the coast from Vandenberg Air Force Base, which would have been prime. So I sat there for a couple of minutes contemplating the prevailing wind direction, then the warning tone stopped and the regular program resumed with no mention of it.Report
Hoo boy.Report
Doh…poor man that’s going to be in every trading video from here on out.Report
here’s an article with screen caps of the gui.
Japan had a Snafu of their own
Eta and this is relevant https://twitter.com/politicalmath/status/953308164539138048Report
Oooh, thank you. The gui is something I’ve been wondering about…
Ugh, that gui is bowling-shoe ugly.Report
Looks like damn near every government intranet site I’ve had to navigate for Navy/VA benefits, etc.Report
Very often software like this is customizable, with sets of menu options read from some list of customer-specified “actions”, and thus just listed together without a lot of thought. In other words, a well constructed application might have very clear “test mode” options, color coded, with another set of “for real” options, which then have extra fail-safes. But if these are just added modules hacked together onsite, then you will see UIs like this.Report
There’s a fifth possibility, which is that they knew there was not a missile, but wanted to test the system AND wanted to test the public response. Joe Blow is just the fall guy.Report
Oh, and to tie up the last loose end:
Hawaii governor took long to post on Twitter about missile alert because he forgot username, password
So there’s that.Report