BLACK HAT USA – Las Vegas – Thursday August 8 – Companies are rapidly deploying Microsoft’s Copilot AI-powered chatbots, hoping to transform the way employees gather data and organize their time and work. But at the same time, Copilot is also an ideal tool for threat actors.

Security researcher Michael Bargury, a former senior security architect at Microsoft’s Azure Security CTO office and now co-founder and Chief Technology Officer of Zenity, says attackers can use Copilot to search for data, exfiltrate it without producing logs, and use social engineering to lure victims to phishing sites even if they don’t open the emails or click on any links.

Bargury demonstrated today at Black Hat USA in Las Vegas how Copilot, like other chatbots, is susceptible to prompt injections that allow hackers to bypass security measures.

The briefing, Life of Microsoft Copilotis the second Black Hat presentation in as many days for Bargury. In his first presentation on Wednesday, Bargury showed how developers unconsciously building Copilot chatbots capable of exfiltrating data or bypass data loss prevention policies and measures with Copilot Studio, Microsoft’s tool for creating and managing bots.

A Red-Team Hacking Tool for Copilot

Thursday’s follow-up session focused on various risks associated with the actual chatbots, and Bargury released an offensive security toolset for Microsoft 365 on GitHub. The new LOLCopilot modulepart of powerpwn, is designed for Microsoft Copilot, Copilot Studio and Power Platform.

Bargury describes it as a red-team hacking tool to demonstrate how to change the behavior of a bot, or “copilot” in Microsoft jargon. by rapid injection. There are two types: A direct prompt injection, or jailbreak, is where the attacker manipulates the LLM prompt to modify its output. In indirect prompt injections, attackers modify the data sources accessed by the model.

Using the tool, Bargury can add direct prompt injection to a copilot, jailbreak it, and modify a parameter or instruction within the model. For example, he can embed an HTML tag in an email to replace a valid bank account number with the attacker’s, without changing the reference information or modifying the model with, for example, white text or a very small font.

“I can manipulate everything Copilot does on your behalf, including the responses it gives you, any action it can take on your behalf, and how I can personally take full control of the conversation,” Bargury tells Dark Reading.

What’s more, the tool can do all of this undetected. “There’s no indication here that this is coming from some other source,” Bargury says. “This still points to valid information that this victim actually created, and so this thread looks trustworthy. You don’t see any indication of a prompt injection.”

RCE = Remote “Copilot” Execution Attacks

Bargury describes Copilot prompt injections as similar to remote code execution (RCE) attacks. While copilots do not execute code, they do follow instructions, perform actions, and create composites of those actions.

“I can come into your conversation from the outside and take full control of all the actions that the co-pilot is taking on your behalf and the inputs to that,” he says. “That’s why I say this is the equivalent of Remote code execution in the world of LLM apps.”

During the session, Bargury demonstrated what he described as remote Copilot executions (RCEs), where the attacker:

Bargury isn’t the only researcher who has investigated how threat actors can attack Copilot and other chatbots using prompt injection. In June, Anthropic detailed his work Red Team Testing Approach of its AI offerings. And Microsoft, in turn, has his red team efforts has been working on AI security for some time now.

Microsoft’s AI Red Team Strategy

Microsoft has been tackling new research into prompt injections in recent months, which come in both direct and indirect forms.

Microsoft Azure CTO and Technical Officer Mark Russinovich recently discussed various AI and Copilot threats at Microsoft’s annual Build conference in May. He highlighted the release of Microsoft’s new Prompt Shields, an API designed to detect direct and indirect prompt injection attacks.

“The idea here is that we’re looking for signals that there are instructions embedded in the context, either the direct user context or the context that’s fed in through the RAG (retrieval-augmented generation), that might cause the model to behave strangely,” Russinovich said.

Prompt Shields is one of the many Azure tools we’ve collected. Microsoft recently launched which are designed for developers to build safe AI applications. Other new tools include Groundedness Detection to detect hallucinations in LLM output and Safety Evaluation to detect an application’s susceptibility to jailbreak attacks and the creation of inappropriate content.

Russinovich also mentioned two other new tools for security teams: PyRIT (Python Risk Identification Toolkit for Generative AI)an open source framework that discovers risks in generative AI systems. The other, Crescendomation, automates Crescendo attacks, which produce malicious content. He also announced Microsoft’s new partnership with HiddenLayerwhose Model Scanner is now available for Azure AI to scan commercial and open source models for vulnerabilities, malware, or tampering.

The need for anti-promptware tooling

According to Bargury, Microsoft says it has addressed these attacks with security filters, but AI models are still susceptible.

He specifically says there is a need for more tools that scan for what he and other researchers call “promptware,” i.e., hidden instructions and untrusted data. “I don’t know of anything that you can use out of the box (for detection) today,” Bargury says.

“Microsoft Defender and Purview don’t have that capability today,” he added. “They have some user behavior analytics, which is useful. If they see that the copilot endpoint is making multiple calls, that might be an indication that they’re trying to do a quick injection. But actually, it’s something very surgical, where someone has a payload, they send you the payload, and (the defense) won’t notice.”

Bargury says he communicates regularly with Microsoft’s red team and notes that they are aware of his presentations at Black Hat. He also believes that Microsoft has been aggressive in addressing the risks associated with AI in general and its own Copilot in particular.

“They’re really working hard,” he said. “I can tell you that in this research, we found 10 different security mechanisms that Microsoft has put into Microsoft Copilot. These are mechanisms that scan everything that goes into Copilot, everything that goes out of Copilot, and a bunch of steps in the middle.”