AI hallucinations have opened a potential path for hackers to deliver malicious packages to the repositories of large organizations, new research shows.
The claims, made by Lasso Security researcher Bar Lanyado, follow a 2023 investigation into whether developers should trust the package recommendations provided by ChatGPT.
In June 2023, Lanyado found that LLMs frequently hallucinated when asked to recommend code libraries, suggesting that developers download packages that do not actually exist.
Lanyado warned that this flaw could have a big impact as a large portion of developers are starting to use AI chatbots instead of traditional search engines to research coding solutions.
This initial investigation, since updated in a tracking probesheds more light on the growing magnitude of the problem and warns that threat actors are waking up to this trend of creating malicious packages with names that often boggle the mind of models.
The research paper recommended developers only download from vetted libraries to avoid installing malicious code.
Lanyado said his goal with this latest study was to determine whether model makers have addressed the problems he highlighted in August last year, or whether package hallucinations remain a problem six months after his initial findings.
One of the most surprising findings of this latest study relates to a hugging Python package repeatedly devised by various models called 'huggingface-cli'.
The researchers uploaded an empty package with the same name to evaluate whether developers were uncritically downloading them to repositories using coding wizards.
It also included a dummy package to check how many downloads were done by real people or just scanners.
Lanyado discovered that the fake Hugging Face package received more than 30,000 authentic downloads in just three months, demonstrating the magnitude of the problem related to the reliance on LLMs for development work.
By searching GitHub to see if the package had been added to any enterprise repositories, Lanyado found that several large companies use or recommend the package in their codebase.
One example was the Chinese multinational Alibaba, which provided instructions for installing the fake Python package in the README file of a repository dedicated to the company's internal research.
Nearly one in three questions caused one or more AI hallucinations
in its initial studyLanyado used 457 questions on more than 40 topics in two programming languages to test the reliability of GPT-3.5 turbo suggestions.
Lanyado found that just under 30% of the questions elicited at least one amazed package from the model. The latest research has an expanded scope in terms of the questions he asks of the models, as well as the number of programming languages he tested.
The researchers also tested several different models, including GPT-3.5 and 4, Google's Gemini Pro, and Cohere's Coral, comparing performance and looking for overlaps where blown packets were received on more than one model.
The latest research used 2,500 questions on 100 topics in 5 different programming languages, including Python, node.js, go, .net, and ruby.
Lanyado prompts were optimized to simulate the workflow of software developers, using “how to” questions to prompt the model to provide a solution that includes a specific package.
Another modification made in the study was to use the Langchain application development framework to manage interactions with the models.
The benefit of this is to use the system's default message that tells the model to say if it doesn't know the answer, which I expected would make LLMs hallucinate less frequently.
The research found that the frequency of hallucinations varied between models, with GPT-3.5 being the least likely to generate fake packages with a hallucination rate of 22.2%.
Not far behind were GPT-4 (24.2%) and Cohere (29.1%), but by far the least reliable model in Lanyado's tests was Gemini, which came up with code packages at 64.5%.
