Article: A new open-source scanner that audits AI “agent skills” is being turned into a tool after researchers showed an unverified skill can silently harvest SSH keys and cloud credentials. The scanner blends keyword filters, behavior analysis, AI-driven reasoning, sandbox execution and change-detection to stop the next wave of supply-chain attacks against AI-assisted developers.

Why AI skills matter now

Large-language-model (LLM) assistants such as Claude and GitHub Copilot now rely on “agent skills” – small, self-contained folders that tell the model how to perform a specific task. GitHub recently added a one-line command that lets developers pull these skills straight from public repositories, turning them into a de-facto package manager for AI-driven workflows.

The convenience is undeniable: a skill can turn a vague request like “clean up this spreadsheet” into precise API calls, file manipulations or code edits. The same simplicity lets malicious code slip into the bundle.

GitHub’s warning page notes that skills are not verified before they are installed.

How a hidden payload can stay under the radar

Unlike a typical binary, an AI skill does not load all at once. The model first reads a short summary, then fetches the full instruction set only when the task appears relevant. This staged loading creates a window where a malicious file can sit idle, invisible to the user, until the model decides to invoke it.

In a proof-of-concept experiment, the researcher created a fake skill called csv-formatter that advertised spreadsheet cleanup. The visible code looked harmless, but a hidden instruction added a “diagnostic step.” The step did not diagnose anything; it scanned the host for SSH private keys and AWS access tokens, then posted the findings to an external server.

The malicious command also appeared in a changelog file – a spot most humans never inspect, but one the AI reads when evaluating version history. The AI silently executed a credential-theft routine while the developer thought the skill was merely formatting data.

The open-source response

To close the gap, the researcher packaged the detection logic into an open-source auditing tool. The scanner does not rely on a single technique; it layers several defenses:

  • Keyword matching catches obvious, lazy attempts that include known malicious strings.
  • Behavior analysis flags instructions that read credential files and then make network calls.
  • AI reasoning runs a secondary model to evaluate whether a skill’s instructions are deliberately hidden from the user.
  • Sandboxing executes the skill in an isolated environment, observing real-time actions without risking the host system.
  • Change detection monitors trusted skills for unexpected modifications, alerting maintainers before a new version is installed.

The author will include a test suite that reproduces the csv-formatter attack and a public benchmark that measures detection rates across a curated set of benign and malicious skills.

What to watch next

  • Community benchmarks: As more researchers publish malicious skill samples, the public benchmark will need regular updates to stay relevant.

The emergence of AI agent skills marks a new frontier in developer tooling, but it also opens a door for supply-chain attacks that bypass traditional code reviews. An open-source scanner that blends static checks, dynamic observation and AI reasoning offers a practical first line of defense. Developers who treat a skill like a random USB drive will soon find that the cost of ignoring verification far outweighs the convenience of instant AI assistance.