The Rising Threat of AI-Hallucinated Developer Packages in Supply Chain Attacks
Large Language Models (LLMs) have shown remarkable capabilities in assisting developers with coding tasks, but their tendency to hallucinate non-existent packages poses a growing security risk. A recent multi-university study, one of the most comprehensive analyses on this issue to date, has uncovered alarming patterns in how LLMs generate references to phantom software packages that do not actually exist. This vulnerability could be exploited by malicious actors to introduce harmful code into the software supply chain.
The study, which tested 16 LLMs for Python and 14 for JavaScript, found that 19.7% of the 2.23 million generated code samples contained references to non-existent packages. This means that nearly 440,445 instances of AI-generated code suggested dependencies that developers might unknowingly try to install—potentially opening the door for attackers to create malicious lookalike packages. If unsuspecting developers trust and use these hallucinated package names, they could unintentionally introduce security vulnerabilities into their projects.
Even more concerning, researchers found that LLMs generated 205,474 unique hallucinated package names, demonstrating how widespread and unpredictable the issue is. The sheer volume of these non-existent dependencies suggests that bad actors could easily preemptively register these package names in repositories like PyPI or npm, filling them with malicious payloads designed to compromise software integrity. This method, known as typosquatting or dependency confusion attacks, has already been exploited in real-world scenarios.
As LLMs become more embedded in developer workflows, mitigating this risk is crucial. Developers should implement strict package validation, verify dependencies through trusted sources, and rely on automated security tools to detect suspicious dependencies before they are installed. Meanwhile, AI researchers and model developers must work on reducing hallucinations in generated code, ensuring that AI-assisted development remains a tool for efficiency rather than a new vector for supply chain attacks.