Dissecting the End of the World

Avatar

Jaybird

Jaybird is Birdmojo on Xbox Live and Jaybirdmojo on Playstation's network. He's been playing consoles since the Atari 2600 and it was Zork that taught him how to touch-type. If you've got a song for Wednesday, a commercial for Saturday, a recommendation for Tuesday, an essay for Monday, or, heck, just a handful a questions, fire off an email to AskJaybird-at-gmail.com

Related Post Roulette

59 Responses

  1. Avatar Oscar Gordon
    Ignored
    says:

    It probably wasn’t a button like on some control console somewhere, it was probably on a drop down list in the software they use to manage shift changes and what not, and someone had the wrong screen up, or clicked in the wrong ComboBox, or selected the wrong item in the list, and it’s wasn’t really Joe’s fault, it was whoever designed the software interface.Report

    • Avatar Jaybird in reply to Oscar Gordon
      Ignored
      says:

      Yeah, you’re right. Here’s a paragraph from the Fox article:

      Rather than triggering a test of the system, it went into actual event mode. He confirmed that to trigger the alert, there is a two-step process involving only one employee — who both triggers the alarm, then also confirms it.

      “There is a screen that says, ‘Are you sure you want to do this?'” Miyagi said. The employee confirmed the alert, inadvertently causing a panic in a state already on edge over saber-rattling missile threats from North Korea.

      So someone had the wrong screen up and clicked in the wrong ComboBox.

      And then did it again.Report

      • Avatar Oscar Gordon in reply to Jaybird
        Ignored
        says:

        Are you incredulous that he did it again?Report

        • Avatar Jaybird in reply to Oscar Gordon
          Ignored
          says:

          If I knew how much his salary was, it would help.

          If he was making $12/hr? I’d see his pushing the button a second time as pretty much inevitable and would immediately begin wondering why it took so long for this to happen.

          If he’s a salaried employee who had to do stuff like pass a background check to get his job?

          Yeah. I’m willing to be incredulous at that point.Report

          • Avatar Oscar Gordon in reply to Jaybird
            Ignored
            says:

            Having done end user support for long enough (a decade or so) amongst people, young and old, who were ostensibly the best and brightest, I’m not incredulous at all. A badly designed GUI can readily trip up the smartest kinds of people, and badly designed GUIs are a chronic problem, especially in software marketed to government agencies, or built to spec for them.

            So, I want to see the UI, not the employee stats.Report

            • Avatar Mike Schilling in reply to Oscar Gordon
              Ignored
              says:

              If the second push is

              Confirm? Yes/No/Cancel

              as opposed to

              Type the Name of the Emergency and Hit Enter

              I can see it very easily.Report

              • Avatar Michael Cain in reply to Mike Schilling
                Ignored
                says:

                This isn’t even the first time. In 2005, an operator error caused an evacuation order for the entire state of Connecticut to go out. I have seen speculation that some of the state systems are running 20+ year old software.

                When I was a legislative budget analyst, it had become unfortunately common to see policy changes voted down because of the cost of putting those changes into the software systems (think tens of millions of dollars). Trump’s infrastructure dollars could possibly produce greater benefit if they were routed to the states for software replacements.Report

            • Avatar veronica d in reply to Oscar Gordon
              Ignored
              says:

              Yes, this. A long shift, too much coffee, a bad UI — really there is no need for conspiracy theories here.Report

      • Avatar Stillwater in reply to Jaybird
        Ignored
        says:

        Suspiciously implausible. System was hacked. QED. 🙂Report

        • Avatar Stillwater in reply to Stillwater
          Ignored
          says:

          And actually, I’m not kidding except for the QED. As greg notes below, there’s not only no reason (a priori!) to assume the system wasn’t hacked, we’ve been presented with lots of evidence (or reports of lots of evidence…) that foreign state actors are actively trying to hack into exactly these types of US state and federal systems. So the rational explanation given that we don’t know all the facts will be a probability assignment of each account based on what we do know.Report

          • Avatar Jaybird in reply to Stillwater
            Ignored
            says:

            The two things that make me most suspicious: the 38 minute response time and the somewhat wacky phrasing of the original message.

            As Kohole points out, “this is not a drill” doesn’t sound like official language (I think it sounds like something that a prankster would add). The 38 minute response time makes sense if nobody was at the console and then had to get to it… less sense if someone was sitting at the console and pressed the button himself twice.Report

            • Avatar Kolohe in reply to Jaybird
              Ignored
              says:

              The 38 minute reponse time makes sense in this scenario –

              Shift change, *saturday morning*. So the ‘regular’ people aren’t at work.

              Someone errs right as they are walking out the door, and nobody catches that they send a real world outgoing alert. (I have no idea how this system works, but there are similar systems that have ‘safe’ or ‘training’ modes where it’s not necessarily clear to certain users if you are in a training mode or not.

              There possibly are multiple locales that can send such an alert (e.g. a terminal at the Tsunami Warning Center, a terminal at Hawaiian Civil Defense HQ, a terminal at one of the military commands on the island).

              So when something went out, it takes a while for the watchcenters to know something went out. They don’t have their personal cellphones on them for security and paying attention reasons. The bosses at home do get an alert, (eventually), they call into their respective people, but it takes a while to determine who sent the alert, and to verify with ‘ground truth’ watchstanders (who only a limited number of people have access too) that indeed, the status boards are all green and nothing is actually hapenning in the real world.

              I could easily see this entire process taking 38 minutes from the time the alert went out, until when the governor has enough info to verify ‘all clear, false alarm’.

              Remember, it was a Saturday.Report

              • Avatar Stillwater in reply to Kolohe
                Ignored
                says:

                Re the shift change hypothesis: why would the State run a test of the emergency systems during a shift change? Does that seem plausible?Report

              • Avatar Kolohe in reply to Stillwater
                Ignored
                says:

                People on the previous shift doing a favor (or following procedures) to queue stuff up for a drill to be run on the next shift.

                People on previous shift scheduled to run a drill and had stuff queued up but the drill was cancelled because of whatever reason.

                People futzing around on the previous shift, loading message into the bin for training/demonstration (or for sh**s & giggles) and not clearing the buffer when they were done.

                In either scenario that stuff could have been activated during the shift turnover when it wasn’t supposed due to not following correct turnover procedures or a flaw in the procedures that allowed this message to go live for real.

                Further thought. Someone on previous shift showing an FNG on next shift ‘this is how you do this’ and it went too far.Report

              • Avatar Kolohe in reply to Kolohe
                Ignored
                says:

                Also, you sometimes run a drill near a shift change because one shift is the one being drilled, and another shift is also on site conducting, monitoring, and evaluating the drill.Report

              • Avatar Stillwater in reply to Kolohe
                Ignored
                says:

                OK, not implausible. Thanks.Report

              • Avatar Dark Matter in reply to Stillwater
                Ignored
                says:

                Re the shift change hypothesis: why would the State run a test of the emergency systems during a shift change? Does that seem plausible?

                Running one? No. Ending one? Yeah.

                Say it’s the last test run of a long list (like, every part of the system) because it’s the least probable and the least used.

                Then there are multiple delays. Joe didn’t know it was “live”. Jim didn’t know where the “kill this” switch even was (an even more unused part of the system), or maybe he even doesn’t have security access to this part of the system.

                38 minutes is long enough for someone to walk out to his car, turn on the radio, realize what he’d done (panic on live news), and then run back to his desk.Report

              • Avatar Kolohe in reply to Kolohe
                Ignored
                says:

                Ok scratch that. The news is saying now that Hawaiian officials knew everything was ok in 3 minutes.Report

      • Avatar Kolohe in reply to Jaybird
        Ignored
        says:

        Mr. Miyagi needs to send that employee back to waxing the cars and painting the fences.Report

      • Avatar Chip Daniels in reply to Jaybird
        Ignored
        says:

        I sort assumed that “pushed the wrong button” is managerial shorthand for “opened a couple of drop down menus, pushed the wrong buttons or three, and didn’t notice the difference between menu (a) and (III) argle bargle…:” since I doubt there are many systems left that are analog and are physical buttons.Report

        • Avatar Jaybird in reply to Chip Daniels
          Ignored
          says:

          “Pushed the wrong button” is something that gets me to say “hey, that could have happened to anybody”.

          “Opened a couple of drop down menus, pushed the wrong buttons or three, and didn’t notice the difference between menu (a) and (III) argle bargle” does not get me to say “hey, that could have happened to anybody” but “what the hell is going with training and procedures over there?”Report

          • Avatar Chip Daniels in reply to Jaybird
            Ignored
            says:

            Well, yeah, its a very good question as to who, and how, and under what circumstances a ‘”OMG WE ARE AT WAR !! WOOP WOOP WOOP” message is allowed to be sent;
            Like I mentioned below, I don’t get why any local entity is empowered to unilaterally decide to send this kind of message, even if they sincerely believed it to be real.

            I mean like,even since ancient times the military has an entire command and control system to control who is allowed to alert the troops and assemble them into fighting formation.Report

          • fillyjonk fillyjonk in reply to Jaybird
            Ignored
            says:

            I’ve made wrong selections on drop-down menus before, but in my case, that meant the wrong statistical test got run, I cursed about it, and went back and did the right one. But if clicking the wrong thing on a drop-down menu would, for example, erase months of data I had spend weeks entering? I’d make DAMN sure of what I was doing before I clicked anything.

            That 38 minute gap is what I find so horrifying. I can’t imaging what would have been going through my mind, had I been in Hawaii. I hope no one died (heart attack, stroke….) as a result of the stress of those 38 minutes.Report

  2. Avatar greginak
    Ignored
    says:

    I did ponder that there may be a spike in births in 9 months.

    There have been a bunch of report that Russian associated hackers have been working at getting access to more than just e-mail accounts.

    https://techcrunch.com/2018/01/12/russian-hackers-senate-pawn-storm-fancy-bear/

    Or course that is just the senate.

    Allthough in this bit, in the middle of the piece:

    “The U.S. is vulnerable in other areas, too. When Attorney General Jeff Sessions testified before the Senate Intelligence Committee in June, Senator John McCain turned his attention to an even more worrisome possibility: “Quietly, the Kremlin has been trying to map the United States telecommunications infrastructure,” he said, describing a series of steps hackers have taken to develop “a cyber weapon that can disrupt the United States power grids and telecommunications infrastructure.” When McCain asked Sessions if the administration had a plan to deal with such an eventuality, Sessions admitted that it did not.”

    https://www.vanityfair.com/news/2018/01/russian-hackers-may-be-preparing-another-major-us-attack

    So that is all scary especially since dealing with hacking by foreign actors has, perversely, become a deeply partisan issue. On the other hand i’d still bet this was a simple mistake since dumb errors are far more common than anything else in the world.Report

  3. Avatar Kolohe
    Ignored
    says:

    I said already on twitter that the biggest problem is that the combox has a setting for ‘this is not a drill’ (and/or there are procedures that allow/require you to input that)

    Drill messages should be clearly marked as such, and real-world messsages should just have the message without any fluff, emphasis, or editorializing.Report

    • Avatar Jaybird in reply to Kolohe
      Ignored
      says:

      You’d think that there’d be a meeting or fourteen where they sat down and hammered out the wording of the battery of pre-written SMS emergency messages.

      “How do we want to phrase the hurricane one?”
      (Two whole hours devoted to whether all caps should be used, whether punctuation should be used, whether adjectives should be used…)
      “Okay. Break for lunch, then back here and we’ll figure out how to phrase the missile attack messages.”Report

      • Avatar Kolohe in reply to Jaybird
        Ignored
        says:

        I would think that the alert messages for the entire US Pacific rim would have enough commonalities that Hawaii didn’t need a bespoke home grown system.

        (and/or they would be the ones that innovated it and then sent it along to the other Pacific coast states)Report

        • Avatar Chip Daniels in reply to Kolohe
          Ignored
          says:

          I would like to know more about this myself.

          Wouldn’t it make sense that any messages about foreign attack should be tightly controlled by the Defense Dept in Washington?

          Its one thing to have localized conditions controlled locally, but it seems like the binary nature of “we are at war/ we are NOT at war” should be part of the command and control system.
          So like, instead of a warning screen like “Are you sure?” it should be “enter the DoD authorization code”.

          But then, I am not sure how it works to begin with.Report

          • Avatar Dark Matter in reply to Chip Daniels
            Ignored
            says:

            Wouldn’t it make sense that any messages about foreign attack should be tightly controlled by the Defense Dept in Washington?

            I doubt it. An in bound missile would trump any lack of message from Washington. Hawaii is (historically) pretty isolated and could expect to know of some types of attacks (including a missile from NK) before Washington does. Think Pearl Harbor.Report

  4. Avatar Kazzy
    Ignored
    says:

    Maybe there was no alert. Mass hysteria. Caused by chemtrails.Report

  5. Avatar Kolohe
    Ignored
    says:

    Btw, this is also why ‘zombie apocalypse’ is now used so frequently in emergency response & mass casualities drills, because it can’t be mistaken for ‘the real thing’.Report

  6. Avatar Alan Scott
    Ignored
    says:

    One day, ninety-nine balloons
    were released into the air
    and one-by-one they crossed the wall
    from over here to over there
    and on the other side, they saw
    a blip upon a radar screen
    the operator said it was
    the strangest thing he’d ever seenReport

  7. Avatar Saul Degraw
    Ignored
    says:

    Occam’s Razor says it was a stupid mistake during a shift change and nothing more. Speculating otherwise might be amusing but it is dangerous for political sanity and amusing ourselves to death.Report

    • Avatar LeeEsq in reply to Saul Degraw
      Ignored
      says:

      Occam’s Razor has its advantages.Report

    • Avatar Stillwater in reply to Saul Degraw
      Ignored
      says:

      That’s a weird response when the NatSec community and lots of Dems on the Hill think we’re potentially amusing ourselves to death by not addressing Russian efforts to penetrate US security and infrastructure systems.Report

    • Avatar Damon in reply to Saul Degraw
      Ignored
      says:

      Oh, please….every “right thinking person” knows it’s Trump’s fault.Report

    • Avatar veronica d in reply to Saul Degraw
      Ignored
      says:

      Yes, yes, yes. We already have too many hair brained conspiracy theories spun out of practiced ignorance.Report

      • Avatar Stillwater in reply to veronica d
        Ignored
        says:

        I need a minute to catch up on the narrative. Aren’t we supposed to believe the Russians interfered with our elections, going so far as to hack into DNC email accounts, various Senators personal email accounts, and try to penetrate 26 states election systems? Is that just a conspiracy theory spun out of practiced ignorance?

        Maybe Trump is right about the Mueller investigation after all.Report

        • Avatar Dark Matter in reply to Stillwater
          Ignored
          says:

          Last I heard, the Russians appear to have illegally placed roughly $200k(?) in advertising on FB… in an election where Billions of dollars were spent on advertising.

          Did it happen? Yes. Did it matter? Probably not.

          In theory that illegal advertising purchases the 20k(?) votes who swung the election. In practice we have lots of media professionals who are supposed to be good at influencing elections here in the US. If we can’t predict which 20k votes matter then I seriously doubt the Russians can.

          This is independent of Trump doing something heinous because Trump is always doing something heinous and none of his people were experienced at knowing the rules. So the rules were probably broken, and I expect Mueller can find something Vile about Trump.

          I also expect it didn’t really matter more than what we already know and it didn’t swing the election more than HRC’s email server and/or Trump’s twitter feed.

          This fascination with the Russians is an attempt to externize an internal problem.Report

  8. Avatar Jaybird
    Ignored
    says:

    WaPo has an article out. They’re saying that it’s a design fail where the choice “Missile Alert” was chosen instead of “Test Missile Alert”.Report

    • Avatar Kazzy in reply to Jaybird
      Ignored
      says:

      “Please choose what you want to do…
      1. Text mom
      2. Flick lights in bathroom to confuse coworker
      3. Missile alert
      4. Test missile alert
      5. Launch missile
      6. Cat videos”

      “Don’t you think maybe we should have separate menus?”
      “It’s more efficient this way.”Report

      • Avatar Jaybird in reply to Kazzy
        Ignored
        says:

        This is why you should go with the second lowest bidder instead of just the lowest one.Report

        • Avatar Marchmaine in reply to Jaybird
          Ignored
          says:

          Or, if it is anything like the 90s era software the govt often uses it is a cropped dropdown and all the options start with Missile:

          * Missile Launch, Test
          * Missile Launch, Alert
          * Missile Launch

          But the only thing that shows in the dropdown (unless you select and read it) is:

          | Missile Lau |

          The regular guys just know to always select #1 (and most don’t even know what #3 even does).Report

  9. Avatar Kazzy
    Ignored
    says:

    I read Van Dyke’s article and a few things stood out.

    She included a video of an alert being broadcast on the TV over a basketball game. The content of that alert was different than the SMS message. So, presumably, the “wrong button” triggered some sort of chain reaction of alerts. She also mentioned that these were 2 of the 3 ways they’d be alerted, with the third being sirens. She didn’t hear the sirens but lives far from them. She reached out to others who were closed and they said there were no sirens, but apparently others have claimed to have heard them.

    So this leaves me wondering…

    If we accept the official story, someone somewhere pushed the wrong button. What happened next? Did that button directly cause the SMS messaging? Did it cause the television alert? Or did that button being pushed cause a light to go off or a bell to ring elsewhere and somewhere who saw that light or heard that bell then pushed a button to send the SMS message and television alert? Or did someone at the TV station see the SMS message and decide to put out a television alert? Why were the alerts different? Why didn’t (or did?) the sirens go off? If all of these alerts aren’t automatically triggered by the wrong button (and I assume they are not if the sirens did indeed remain silent) why did no one double-check that the light or bell or whatever was indeed correct before pushing their buttons to send their alerts?

    I’m really curious to know more about the chain of events that begins with someone pushing a wrong button and ends with 2 alerts being sent but not a third.Report

    • Avatar Jaybird in reply to Kazzy
      Ignored
      says:

      Yeah. I’m 90% satisfied with the official explanation.

      That 10%, though. Man.Report

      • Avatar Kazzy in reply to Jaybird
        Ignored
        says:

        I hadn’t even considered that the official story wasn’t more or less true until I read this post and I’m still inclined to believe it’s true or true enough.

        I think it is easy to think, “HOW CAN A MISTAKE LIKE THIS OCCUR?!” But if you assume there’s probably 3 shift changes a day every day for however many years and probably lots of button pushes every shift, the error rate starts to look very tiny and probably approaching what is reasonable. Layer on what sound like genuine design flaws (which seem less reasonable) and shit happens.

        So I don’t offer this to challenge the official story as much to say there is lots that is still unexplained to me (and maybe there are perfectly good answers that I just haven’t seen yet).Report

      • Avatar George Turner in reply to Jaybird
        Ignored
        says:

        The employee who screwed up has been reassigned. That confirms that it was an employee screw up and confirms that we’re in a new era of government accountability. Previously a government employee could’ve accidentally triggered a global nuclear war and the most punishment they’d face was a paid leave of absence.Report

  10. Avatar Chip Daniels
    Ignored
    says:

    After all these posts, this seems obligatory:

    Shall. We. Play. A. Game?Report

  11. Avatar Richard Hershberger
    Ignored
    says:

    When I was an undergrad at UC Santa Barbara–this would have been around 1982, a/k/a the height of the Cold War, a/k/a The Good Old Days, I was up late studying in the middle of the night with the radio on. The Emergency Broadcast System warning tone came on without the usual “This is a test” language before it. That was startling. Back in those days we all knew the nearest best target for nukes. Santa Barbara itself would be a waste of a good warhead, but it is down the coast from Vandenberg Air Force Base, which would have been prime. So I sat there for a couple of minutes contemplating the prevailing wind direction, then the warning tone stopped and the regular program resumed with no mention of it.Report

    • Avatar Jaybird in reply to Kolohe
      Ignored
      says:

      Oooh, thank you. The gui is something I’ve been wondering about…

      Ugh, that gui is bowling-shoe ugly.Report

    • Avatar veronica d in reply to Kolohe
      Ignored
      says:

      Very often software like this is customizable, with sets of menu options read from some list of customer-specified “actions”, and thus just listed together without a lot of thought. In other words, a well constructed application might have very clear “test mode” options, color coded, with another set of “for real” options, which then have extra fail-safes. But if these are just added modules hacked together onsite, then you will see UIs like this.Report

  12. Avatar Jacob
    Ignored
    says:

    There’s a fifth possibility, which is that they knew there was not a missile, but wanted to test the system AND wanted to test the public response. Joe Blow is just the fall guy.Report

Leave a Reply

Your email address will not be published. Required fields are marked *