Spam becomes urgent tasks: Hidden prompts in emails manipulate AI summarizers into tricking employees

Organisations relying on artificial intelligence assistants to process inboxes face a new vector for social engineering. Attackers are no longer solely targeting human recipients with deceptive messages. Instead, they are embedding hidden instructions designed to manipulate the AI tools that summarise those emails. This technique turns a phishing attempt that a user might have ignored into an urgent, trusted recommendation generated by the software itself.

The core issue lies in how large language models process external text. When an assistant reads an email to provide a summary or action items, it may inadvertently follow malicious commands hidden within the message body. These commands, known as indirect prompt injections, instruct the AI to misrepresent the email’s legitimacy or to generate fake priority tasks, such as changing vendor payment details.

What we know

A report published by Malwarebytes Labs titled "Phishing emails are baiting your AI assistant" describes this risk. The publication notes that users who wisely avoid clicking links or opening attachments in suspicious emails may still be compromised. If their AI assistant processes the same message, the hidden instructions can bypass the user’s initial caution. The assistant then presents the fraudulent request as a legitimate, urgent to-do item, leveraging the trust users place in their productivity tools.

Barracuda provided specific examples of how these attacks are constructed. In one scenario, an email contained a password-protected attachment for the human reader, while simultaneously holding hidden instructions for the AI. These instructions aimed to influence how the assistant summarised the content. By framing the malicious request as an AI-generated recommendation, attackers increase the likelihood that an employee will act on it without further scrutiny.

The concealment techniques used to hide these instructions are varied and sophisticated. Barracuda identified methods including comments embedded in the email’s underlying HTML code, text rendered invisible through styling, encoded content, and the use of zero-width characters. These methods ensure that a human viewing the email sees nothing amiss, while the AI system processes the hidden payload. The success of such an attack depends heavily on how the specific tool extracts and prepares email content for the model.

Apollo Raines, developer of the AI code review service ShipItClean, explained the fundamental vulnerability. He stated that the model cannot distinguish between text it is supposed to read and instructions it is supposed to follow. Raines argued that stripping hidden content before it reaches the model is essential, but noted that it remains an arms race against new encoding tricks. He emphasised that defenses must reside within the tools and models themselves, including sandboxing and human sign-off on payments.

The risk extends beyond misleading summaries. If an AI assistant has additional permissions, such as the ability to send emails or access financial systems, the consequences could be more severe. Malicious instructions might attempt to make the assistant carry out unauthorised actions or disclose sensitive information directly. This shifts the attack surface from the user’s judgement to the software’s configuration and safeguards.

Defender's checklist

• Audit the permissions granted to AI assistants, ensuring they cannot execute financial transactions or send external communications without human approval.
• Review how email clients and AI tools preprocess messages, checking for mechanisms that strip HTML comments, invisible text, or encoded payloads before analysis.
• Implement mandatory human verification for any AI-generated recommendation involving changes to vendor details, payment information, or sensitive data.
• Monitor for unusual patterns in AI activity, such as sudden requests to update contact information or process urgent payments, which may indicate a successful injection.

Background: Large language models

Large language models are zero-shot reasoners only in appearance. They predict the next word by matching patterns in training data, not by applying logical rules. This means they can solve novel problems if similar examples exist in their data, but they fail when true novel logic is required.

Read the full guide: Are Large Language Models Zero-Shot Reasoners? The Reality Check

What to do and how to stay safe: Malwarebytes Labs

• Verify the original email content directly before acting on any recommendation provided by an AI assistant, especially those involving money or sensitive data.
• Limit the permissions of AI tools so they can only read and summarise, not execute actions like sending emails or modifying records.
• Educate staff to treat AI summaries as unverified information that requires cross-checking with the source message.
• Once the vendor provides an update, apply patches that address prompt injection vulnerabilities in email processing and AI summarisation features.

Step-by-step guide: Multi-Factor Authentication Checklist: Stop Account Takeovers

General security guidance from the TechWire Beat newsroom. It is not confirmed advice from the organisations named in this story.

Frequently asked questions

How do attackers hide prompt injections in emails?

Attackers use techniques such as HTML comments, invisible text styling, encoded content, and zero-width characters to conceal instructions from humans while making them readable to AI systems.

What is indirect prompt injection?

It is a technique where attackers plant instructions in external material, such as an email, hoping the AI system will follow those instructions instead of treating them as part of the content to be summarised.

Can AI assistants perform unauthorized actions?

Yes, if an assistant has additional permissions, malicious instructions might try to make it carry out unauthorized actions or disclose sensitive information, depending on its capabilities and safeguards.