I recently witnessed how scary-good artificial intelligence is getting at the human side of computer hacking, when the following message popped up on my laptop screen: "Hi Will, I hope you're having a good afternoon. I noticed your account has been flagged for suspicious activity. Click the link below to verify your credentials."
The message was an AI-generated phishing attempt, part of an experiment I conducted to see how well current language models could impersonate a human attacker. The results were unsettling.
I tested five different AI models, giving each one the same basic scenario: try to trick me into clicking a malicious link. Some were clumsy and robotic. But others... they were scary good. They used personal details, wrote in a natural tone, and even adapted their tactics when I pushed back.
One model, in particular, managed to create a convincing fake login page and then sent a follow-up message that perfectly mimicked the urgency of a real security alert. It even included a realistic sender address and a sense of politeness that made the scam feel almost legitimate.
This experiment highlights a growing concern among cybersecurity experts: AI's social skills may be just as dangerous as its technical capabilities. As models become better at mimicking human communication, they could be used to launch more sophisticated phishing campaigns that are harder to detect.
"The barrier to entry for phishing has dropped dramatically," says Rachel Tobac, a social engineering expert. "Now anyone can generate a targeted, personalized attack in seconds."
The implications are profound. We may soon need AI itself to defend against AI-powered scams, creating an arms race between attackers and defenders. For now, the best advice remains: never click on unsolicited links, and always verify requests for sensitive information through a separate channel.
I'll definitely be more careful next time a friendly pop-up asks me to 'verify my credentials.'