On LinkedIn
The US military nearly boarded a Chinese ship in the Middle East. The intel said it was carrying nuclear components for Iran. The intel came from a chatbot. The chatbot made it up.
Same week, a lawyer filed a brief full of witnesses ChatGPT invented. Easy to file away: lazy, verified nothing, fined. Comforting, because it lets you think “I would never”.
The military case is the uncomfortable one. Due diligence is the whole job there. It still came within one decision of boarding by force. A source who was there called it almost starting a war.
And it was not one bad day. Earlier in the same war, AI-assisted targeting fed strikes that investigators are still picking apart.
What failed there is what fails in phishing, in CEO fraud, in the urgent change request at five on a Friday. High stakes, high pressure, act now: the state where careful people stop being careful. Not stupidity. The moment demands movement, and validating feels like hesitating.
Nobody says this part out loud: almost everyone feels it coming. The clenched stomach, the small doubt you push down. Recognition is not the problem. The threshold is. How bad does it have to feel before you actually stop?
We run four-eyes as a reflex, not a rule. When it gets tense you grab a senior colleague first: what is happening here, do you see it too? We do it on ordinary days on purpose, so the threshold is already low when it matters. A worn path beats a written procedure. It still takes nerve when the one with the doubt is the most senior person in the room. So we keep practising.
What does it take where you work before someone pulls in a second pair of eyes: a rule, a habit, or a near miss that taught you the hard way?
The reporting: Gizmodo on how the US military nearly boarded a Chinese ship based on bad intel from AI.