AI code assistants are spitting out package names that don’t exist, and attackers are turning those hallucinations into real supply-chain risks.

When security researcher Bar Lanyado from Lasso Security asked an AI-driven coding tool for a Python client, the model suggested installing huggingface-cli. The package is a phantom; the legitimate library lives under the name huggingface_hub. To prove the danger, Lanyado registered the bogus name on the Python Package Index (PyPI) as an empty placeholder. Within three months the placeholder amassed over 30,000 downloads, showed up in public documentation and even appeared in code samples harvested from Alibaba’s repositories.

How a phantom package becomes a real threat

  1. Prompt → hallucination – A developer asks an AI for help. The model, trained on noisy internet data, invents a plausible-sounding package name.
  2. Copy-paste → documentation – The suggestion lands in a README, a Stack Overflow answer, or an internal wiki. Once written, the name spreads through the community.
  3. Code integration – The developer, trusting the AI, adds the name to a requirements file and pushes it to production.

Why existing defenses miss the problem

Static analysis tools and vulnerability scanners look for known CVEs and for libraries with a release history. A freshly published package with zero downloads before the AI suggestion has no CVE, no reputation, and therefore appears clean. The standard “is the version vulnerable?” check returns false, giving developers a false sense of safety.

What the experiment proves

  • AI-invented names reach production – The over 30,000 downloads show that developers are actually pulling these phantom packages.
  • Hallucinations become documentation – Once a fake name appears in a public guide, it can persist indefinitely, propagating the error.
  • Registration is trivial – Publishing a package on PyPI costs nothing and takes minutes, lowering the barrier for supply-chain abuse.

Defensive steps that actually work

  • Validate every dependency – Before adding a new requirement, search the package index and confirm the name matches an existing, documented library.
  • Cross-check with official sources – Compare the suggested name against the vendor’s repository or official installation guide.
  • Treat new, low-adoption packages as high risk – Flag any dependency that has fewer than a handful of downloads or a very recent release date for manual review.

What to watch next

The takeaway is simple: an AI suggestion is not a guarantee. Treat every new dependency as an unvetted third-party component, verify its provenance, and keep an eye on the supply chain before you let a phantom package slip into production.