Dissecting the End of the World

Jaybird

Jaybird is Birdmojo on Xbox Live and Jaybirdmojo on Playstation's network. He's been playing consoles since the Atari 2600 and it was Zork that taught him how to touch-type. If you've got a song for Wednesday, a commercial for Saturday, a recommendation for Tuesday, an essay for Monday, or, heck, just a handful a questions, fire off an email to AskJaybird-at-gmail.com

Related Post Roulette

59 Responses

  1. Oscar Gordon says:

    It probably wasn’t a button like on some control console somewhere, it was probably on a drop down list in the software they use to manage shift changes and what not, and someone had the wrong screen up, or clicked in the wrong ComboBox, or selected the wrong item in the list, and it’s wasn’t really Joe’s fault, it was whoever designed the software interface.Report

    • Jaybird in reply to Oscar Gordon says:

      Yeah, you’re right. Here’s a paragraph from the Fox article:

      Rather than triggering a test of the system, it went into actual event mode. He confirmed that to trigger the alert, there is a two-step process involving only one employee — who both triggers the alarm, then also confirms it.

      “There is a screen that says, ‘Are you sure you want to do this?'” Miyagi said. The employee confirmed the alert, inadvertently causing a panic in a state already on edge over saber-rattling missile threats from North Korea.

      So someone had the wrong screen up and clicked in the wrong ComboBox.

      And then did it again.Report

      • Oscar Gordon in reply to Jaybird says:

        Are you incredulous that he did it again?Report

        • Jaybird in reply to Oscar Gordon says:

          If I knew how much his salary was, it would help.

          If he was making $12/hr? I’d see his pushing the button a second time as pretty much inevitable and would immediately begin wondering why it took so long for this to happen.

          If he’s a salaried employee who had to do stuff like pass a background check to get his job?

          Yeah. I’m willing to be incredulous at that point.Report

          • Oscar Gordon in reply to Jaybird says:

            Having done end user support for long enough (a decade or so) amongst people, young and old, who were ostensibly the best and brightest, I’m not incredulous at all. A badly designed GUI can readily trip up the smartest kinds of people, and badly designed GUIs are a chronic problem, especially in software marketed to government agencies, or built to spec for them.

            So, I want to see the UI, not the employee stats.Report

            • Mike Schilling in reply to Oscar Gordon says:

              If the second push is

              Confirm? Yes/No/Cancel

              as opposed to

              Type the Name of the Emergency and Hit Enter

              I can see it very easily.Report

              • This isn’t even the first time. In 2005, an operator error caused an evacuation order for the entire state of Connecticut to go out. I have seen speculation that some of the state systems are running 20+ year old software.

                When I was a legislative budget analyst, it had become unfortunately common to see policy changes voted down because of the cost of putting those changes into the software systems (think tens of millions of dollars). Trump’s infrastructure dollars could possibly produce greater benefit if they were routed to the states for software replacements.Report

            • veronica d in reply to Oscar Gordon says:

              Yes, this. A long shift, too much coffee, a bad UI — really there is no need for conspiracy theories here.Report

      • Stillwater in reply to Jaybird says:

        Suspiciously implausible. System was hacked. QED. 🙂Report

        • Stillwater in reply to Stillwater says:

          And actually, I’m not kidding except for the QED. As greg notes below, there’s not only no reason (a priori!) to assume the system wasn’t hacked, we’ve been presented with lots of evidence (or reports of lots of evidence…) that foreign state actors are actively trying to hack into exactly these types of US state and federal systems. So the rational explanation given that we don’t know all the facts will be a probability assignment of each account based on what we do know.Report

          • Jaybird in reply to Stillwater says:

            The two things that make me most suspicious: the 38 minute response time and the somewhat wacky phrasing of the original message.

            As Kohole points out, “this is not a drill” doesn’t sound like official language (I think it sounds like something that a prankster would add). The 38 minute response time makes sense if nobody was at the console and then had to get to it… less sense if someone was sitting at the console and pressed the button himself twice.Report

            • Kolohe in reply to Jaybird says:

              The 38 minute reponse time makes sense in this scenario –

              Shift change, *saturday morning*. So the ‘regular’ people aren’t at work.

              Someone errs right as they are walking out the door, and nobody catches that they send a real world outgoing alert. (I have no idea how this system works, but there are similar systems that have ‘safe’ or ‘training’ modes where it’s not necessarily clear to certain users if you are in a training mode or not.

              There possibly are multiple locales that can send such an alert (e.g. a terminal at the Tsunami Warning Center, a terminal at Hawaiian Civil Defense HQ, a terminal at one of the military commands on the island).

              So when something went out, it takes a while for the watchcenters to know something went out. They don’t have their personal cellphones on them for security and paying attention reasons. The bosses at home do get an alert, (eventually), they call into their respective people, but it takes a while to determine who sent the alert, and to verify with ‘ground truth’ watchstanders (who only a limited number of people have access too) that indeed, the status boards are all green and nothing is actually hapenning in the real world.

              I could easily see this entire process taking 38 minutes from the time the alert went out, until when the governor has enough info to verify ‘all clear, false alarm’.

              Remember, it was a Saturday.Report

              • Stillwater in reply to Kolohe says:

                Re the shift change hypothesis: why would the State run a test of the emergency systems during a shift change? Does that seem plausible?Report

              • Kolohe in reply to Stillwater says:

                People on the previous shift doing a favor (or following procedures) to queue stuff up for a drill to be run on the next shift.

                People on previous shift scheduled to run a drill and had stuff queued up but the drill was cancelled because of whatever reason.

                People futzing around on the previous shift, loading message into the bin for training/demonstration (or for sh**s & giggles) and not clearing the buffer when they were done.

                In either scenario that stuff could have been activated during the shift turnover when it wasn’t supposed due to not following correct turnover procedures or a flaw in the procedures that allowed this message to go live for real.

                Further thought. Someone on previous shift showing an FNG on next shift ‘this is how you do this’ and it went too far.Report

              • Kolohe in reply to Kolohe says:

                Also, you sometimes run a drill near a shift change because one shift is the one being drilled, and another shift is also on site conducting, monitoring, and evaluating the drill.Report

              • Stillwater in reply to Kolohe says:

                OK, not implausible. Thanks.Report

              • Dark Matter in reply to Stillwater says:

                Re the shift change hypothesis: why would the State run a test of the emergency systems during a shift change? Does that seem plausible?

                Running one? No. Ending one? Yeah.

                Say it’s the last test run of a long list (like, every part of the system) because it’s the least probable and the least used.

                Then there are multiple delays. Joe didn’t know it was “live”. Jim didn’t know where the “kill this” switch even was (an even more unused part of the system), or maybe he even doesn’t have security access to this part of the system.

                38 minutes is long enough for someone to walk out to his car, turn on the radio, realize what he’d done (panic on live news), and then run back to his desk.Report

              • Kolohe in reply to Kolohe says:

                Ok scratch that. The news is saying now that Hawaiian officials knew everything was ok in 3 minutes.Report

      • Kolohe in reply to Jaybird says:

        Mr. Miyagi needs to send that employee back to waxing the cars and painting the fences.Report

      • Chip Daniels in reply to Jaybird says:

        I sort assumed that “pushed the wrong button” is managerial shorthand for “opened a couple of drop down menus, pushed the wrong buttons or three, and didn’t notice the difference between menu (a) and (III) argle bargle…:” since I doubt there are many systems left that are analog and are physical buttons.Report

        • Jaybird in reply to Chip Daniels says:

          “Pushed the wrong button” is something that gets me to say “hey, that could have happened to anybody”.

          “Opened a couple of drop down menus, pushed the wrong buttons or three, and didn’t notice the difference between menu (a) and (III) argle bargle” does not get me to say “hey, that could have happened to anybody” but “what the hell is going with training and procedures over there?”Report

          • Chip Daniels in reply to Jaybird says:

            Well, yeah, its a very good question as to who, and how, and under what circumstances a ‘”OMG WE ARE AT WAR !! WOOP WOOP WOOP” message is allowed to be sent;
            Like I mentioned below, I don’t get why any local entity is empowered to unilaterally decide to send this kind of message, even if they sincerely believed it to be real.

            I mean like,even since ancient times the military has an entire command and control system to control who is allowed to alert the troops and assemble them into fighting formation.Report

          • fillyjonk in reply to Jaybird says:

            I’ve made wrong selections on drop-down menus before, but in my case, that meant the wrong statistical test got run, I cursed about it, and went back and did the right one. But if clicking the wrong thing on a drop-down menu would, for example, erase months of data I had spend weeks entering? I’d make DAMN sure of what I was doing before I clicked anything.

            That 38 minute gap is what I find so horrifying. I can’t imaging what would have been going through my mind, had I been in Hawaii. I hope no one died (heart attack, stroke….) as a result of the stress of those 38 minutes.Report

  2. greginak says:

    I did ponder that there may be a spike in births in 9 months.

    There have been a bunch of report that Russian associated hackers have been working at getting access to more than just e-mail accounts.

    https://techcrunch.com/2018/01/12/russian-hackers-senate-pawn-storm-fancy-bear/

    Or course that is just the senate.

    Allthough in this bit, in the middle of the piece:

    “The U.S. is vulnerable in other areas, too. When Attorney General Jeff Sessions testified before the Senate Intelligence Committee in June, Senator John McCain turned his attention to an even more worrisome possibility: “Quietly, the Kremlin has been trying to map the United States telecommunications infrastructure,” he said, describing a series of steps hackers have taken to develop “a cyber weapon that can disrupt the United States power grids and telecommunications infrastructure.” When McCain asked Sessions if the administration had a plan to deal with such an eventuality, Sessions admitted that it did not.”

    https://www.vanityfair.com/news/2018/01/russian-hackers-may-be-preparing-another-major-us-attack

    So that is all scary especially since dealing with hacking by foreign actors has, perversely, become a deeply partisan issue. On the other hand i’d still bet this was a simple mistake since dumb errors are far more common than anything else in the world.Report

  3. Kolohe says:

    I said already on twitter that the biggest problem is that the combox has a setting for ‘this is not a drill’ (and/or there are procedures that allow/require you to input that)

    Drill messages should be clearly marked as such, and real-world messsages should just have the message without any fluff, emphasis, or editorializing.Report

    • Jaybird in reply to Kolohe says:

      You’d think that there’d be a meeting or fourteen where they sat down and hammered out the wording of the battery of pre-written SMS emergency messages.

      “How do we want to phrase the hurricane one?”
      (Two whole hours devoted to whether all caps should be used, whether punctuation should be used, whether adjectives should be used…)
      “Okay. Break for lunch, then back here and we’ll figure out how to phrase the missile attack messages.”Report

      • Kolohe in reply to Jaybird says:

        I would think that the alert messages for the entire US Pacific rim would have enough commonalities that Hawaii didn’t need a bespoke home grown system.

        (and/or they would be the ones that innovated it and then sent it along to the other Pacific coast states)Report

        • Chip Daniels in reply to Kolohe says:

          I would like to know more about this myself.

          Wouldn’t it make sense that any messages about foreign attack should be tightly controlled by the Defense Dept in Washington?

          Its one thing to have localized conditions controlled locally, but it seems like the binary nature of “we are at war/ we are NOT at war” should be part of the command and control system.
          So like, instead of a warning screen like “Are you sure?” it should be “enter the DoD authorization code”.

          But then, I am not sure how it works to begin with.Report

          • Dark Matter in reply to Chip Daniels says:

            Wouldn’t it make sense that any messages about foreign attack should be tightly controlled by the Defense Dept in Washington?

            I doubt it. An in bound missile would trump any lack of message from Washington. Hawaii is (historically) pretty isolated and could expect to know of some types of attacks (including a missile from NK) before Washington does. Think Pearl Harbor.Report

  4. Kazzy says:

    Maybe there was no alert. Mass hysteria. Caused by chemtrails.Report

  5. Kolohe says:

    Btw, this is also why ‘zombie apocalypse’ is now used so frequently in emergency response & mass casualities drills, because it can’t be mistaken for ‘the real thing’.Report

  6. Alan Scott says:

    One day, ninety-nine balloons
    were released into the air
    and one-by-one they crossed the wall
    from over here to over there
    and on the other side, they saw
    a blip upon a radar screen
    the operator said it was
    the strangest thing he’d ever seenReport

  7. Saul Degraw says:

    Occam’s Razor says it was a stupid mistake during a shift change and nothing more. Speculating otherwise might be amusing but it is dangerous for political sanity and amusing ourselves to death.Report

    • LeeEsq in reply to Saul Degraw says:

      Occam’s Razor has its advantages.Report

    • Stillwater in reply to Saul Degraw says:

      That’s a weird response when the NatSec community and lots of Dems on the Hill think we’re potentially amusing ourselves to death by not addressing Russian efforts to penetrate US security and infrastructure systems.Report

    • Damon in reply to Saul Degraw says:

      Oh, please….every “right thinking person” knows it’s Trump’s fault.Report

    • veronica d in reply to Saul Degraw says:

      Yes, yes, yes. We already have too many hair brained conspiracy theories spun out of practiced ignorance.Report

      • Stillwater in reply to veronica d says:

        I need a minute to catch up on the narrative. Aren’t we supposed to believe the Russians interfered with our elections, going so far as to hack into DNC email accounts, various Senators personal email accounts, and try to penetrate 26 states election systems? Is that just a conspiracy theory spun out of practiced ignorance?

        Maybe Trump is right about the Mueller investigation after all.Report

        • Dark Matter in reply to Stillwater says:

          Last I heard, the Russians appear to have illegally placed roughly $200k(?) in advertising on FB… in an election where Billions of dollars were spent on advertising.

          Did it happen? Yes. Did it matter? Probably not.

          In theory that illegal advertising purchases the 20k(?) votes who swung the election. In practice we have lots of media professionals who are supposed to be good at influencing elections here in the US. If we can’t predict which 20k votes matter then I seriously doubt the Russians can.

          This is independent of Trump doing something heinous because Trump is always doing something heinous and none of his people were experienced at knowing the rules. So the rules were probably broken, and I expect Mueller can find something Vile about Trump.

          I also expect it didn’t really matter more than what we already know and it didn’t swing the election more than HRC’s email server and/or Trump’s twitter feed.

          This fascination with the Russians is an attempt to externize an internal problem.Report

  8. Jaybird says:

    WaPo has an article out. They’re saying that it’s a design fail where the choice “Missile Alert” was chosen instead of “Test Missile Alert”.Report

    • Kazzy in reply to Jaybird says:

      “Please choose what you want to do…
      1. Text mom
      2. Flick lights in bathroom to confuse coworker
      3. Missile alert
      4. Test missile alert
      5. Launch missile
      6. Cat videos”

      “Don’t you think maybe we should have separate menus?”
      “It’s more efficient this way.”Report

      • Jaybird in reply to Kazzy says:

        This is why you should go with the second lowest bidder instead of just the lowest one.Report

        • Marchmaine in reply to Jaybird says:

          Or, if it is anything like the 90s era software the govt often uses it is a cropped dropdown and all the options start with Missile:

          * Missile Launch, Test
          * Missile Launch, Alert
          * Missile Launch

          But the only thing that shows in the dropdown (unless you select and read it) is:

          | Missile Lau |

          The regular guys just know to always select #1 (and most don’t even know what #3 even does).Report

  9. Kazzy says:

    I read Van Dyke’s article and a few things stood out.

    She included a video of an alert being broadcast on the TV over a basketball game. The content of that alert was different than the SMS message. So, presumably, the “wrong button” triggered some sort of chain reaction of alerts. She also mentioned that these were 2 of the 3 ways they’d be alerted, with the third being sirens. She didn’t hear the sirens but lives far from them. She reached out to others who were closed and they said there were no sirens, but apparently others have claimed to have heard them.

    So this leaves me wondering…

    If we accept the official story, someone somewhere pushed the wrong button. What happened next? Did that button directly cause the SMS messaging? Did it cause the television alert? Or did that button being pushed cause a light to go off or a bell to ring elsewhere and somewhere who saw that light or heard that bell then pushed a button to send the SMS message and television alert? Or did someone at the TV station see the SMS message and decide to put out a television alert? Why were the alerts different? Why didn’t (or did?) the sirens go off? If all of these alerts aren’t automatically triggered by the wrong button (and I assume they are not if the sirens did indeed remain silent) why did no one double-check that the light or bell or whatever was indeed correct before pushing their buttons to send their alerts?

    I’m really curious to know more about the chain of events that begins with someone pushing a wrong button and ends with 2 alerts being sent but not a third.Report

    • Jaybird in reply to Kazzy says:

      Yeah. I’m 90% satisfied with the official explanation.

      That 10%, though. Man.Report

      • Kazzy in reply to Jaybird says:

        I hadn’t even considered that the official story wasn’t more or less true until I read this post and I’m still inclined to believe it’s true or true enough.

        I think it is easy to think, “HOW CAN A MISTAKE LIKE THIS OCCUR?!” But if you assume there’s probably 3 shift changes a day every day for however many years and probably lots of button pushes every shift, the error rate starts to look very tiny and probably approaching what is reasonable. Layer on what sound like genuine design flaws (which seem less reasonable) and shit happens.

        So I don’t offer this to challenge the official story as much to say there is lots that is still unexplained to me (and maybe there are perfectly good answers that I just haven’t seen yet).Report

      • George Turner in reply to Jaybird says:

        The employee who screwed up has been reassigned. That confirms that it was an employee screw up and confirms that we’re in a new era of government accountability. Previously a government employee could’ve accidentally triggered a global nuclear war and the most punishment they’d face was a paid leave of absence.Report

  10. Chip Daniels says:

    After all these posts, this seems obligatory:

    Shall. We. Play. A. Game?Report

  11. When I was an undergrad at UC Santa Barbara–this would have been around 1982, a/k/a the height of the Cold War, a/k/a The Good Old Days, I was up late studying in the middle of the night with the radio on. The Emergency Broadcast System warning tone came on without the usual “This is a test” language before it. That was startling. Back in those days we all knew the nearest best target for nukes. Santa Barbara itself would be a waste of a good warhead, but it is down the coast from Vandenberg Air Force Base, which would have been prime. So I sat there for a couple of minutes contemplating the prevailing wind direction, then the warning tone stopped and the regular program resumed with no mention of it.Report

    • Jaybird in reply to Kolohe says:

      Oooh, thank you. The gui is something I’ve been wondering about…

      Ugh, that gui is bowling-shoe ugly.Report

    • veronica d in reply to Kolohe says:

      Very often software like this is customizable, with sets of menu options read from some list of customer-specified “actions”, and thus just listed together without a lot of thought. In other words, a well constructed application might have very clear “test mode” options, color coded, with another set of “for real” options, which then have extra fail-safes. But if these are just added modules hacked together onsite, then you will see UIs like this.Report

  12. Jacob says:

    There’s a fifth possibility, which is that they knew there was not a missile, but wanted to test the system AND wanted to test the public response. Joe Blow is just the fall guy.Report