Why you should never ask AI to generate passwords

If you have ever asked an AI chatbot to generate a secure password because you were too lazy to think of one, you might want to change it immediately.

Security firm Irregular recently tested passwords generated by ChatGPT, Claude, and Gemini. The findings are alarming: while these AI-generated strings look complex, they are highly predictable, mathematically weak, and surprisingly easy to crack.

Favorite letters and identical strings

Researchers asked the frontier models to create secure six-character passwords containing letters, numbers, and special symbols. With a proper random generator, collisions should be extremely rare. With AI, it was a mess.

In a test batch of 50 passwords generated by Claude Opus 4.6, only 30 were unique. There were 20 duplicates, and 18 of them were the exact same string repeated over and over.

The predictability went much deeper than simple repetition. Every password Claude generated started with a letter, usually an uppercase "G," followed almost always by the digit "7." Characters like "L," "9," "m," "2," "$," and "#" showed up in every single password, while most of the alphabet was never used.

ChatGPT had its own tell: it liked to start almost every password with the letter "v," with nearly half using an uppercase "Q" as the second character. Gemini favored starting with "k," routinely following it with "#," "P," or "9."

Interestingly, none of the 50 passwords in Irregular's sample contained repeating characters. That might sound diverse to a human eye, but statistically, zero duplicate characters across dozens of random strings proves the process is anything but random.

The randomness paradox

Why are state-of-the-art models so bad at this? The answer lies in their fundamental architecture.

Large language models are trained on statistical probability to predict plausible text. They excel at producing outputs that make sense based on patterns. Real cryptographic security, however, requires maximum entropy and complete unpredictability, which is the exact opposite of what an LLM does.

Humans are notoriously bad at randomness, often reusing habits like substituting "3" for "E." It turns out AI has inherited that exact weakness. Irregular noted that no amount of prompt engineering or temperature tweaking can fix this, because asking an LLM for true randomness goes against how it is built.

Why it matters

This research is a crucial reality check for anyone using AI as an all-purpose utility knife. If you need a password, do not ask a chatbot. Stick to dedicated password managers or built-in OS keychains that rely on cryptographically secure pseudorandom number generators.

The bigger and more immediate danger lies in automated workflows. As software teams increasingly rely on autonomous coding agents, those agents might ask LLMs to generate default credentials, tokens, or secret keys during code scaffolding. If that happens, attackers will not need brute-force supercomputers; they will just generate targeted dictionaries based on these known AI character habits.

We think this highlights a classic AI trap: models are masters at mimicking the aesthetic of a solution without understanding its core requirement. A password that looks complex but lacks entropy is worse than a simple one, because it gives everyone a false sense of safety.