UK security agency maps AI risks in new research framework for organisations

The National Cyber Security Centre has introduced a new chapter to its research problem book, focusing specifically on the intersection of artificial intelligence and cyber security. According to the official advisory, this move aims to focus innovation efforts on the difficult questions surrounding system design, assurance and trust that arise as AI capabilities advance. The agency states that a revolution is underway, with frontier AI capabilities rewriting what is possible for both attackers and defenders. This publication seeks to highlight the opportunities and risks presented by these new technologies, ensuring that the sector addresses the fundamental challenges ahead.

The NCSC notes that while AI and machine learning have long helped defenders analyse data and automate tasks, the new generation of systems is different. Generative and agentic AI can now plan, reason and use tools with increasing autonomy. This shift creates new problems to overcome, making some automation easier but introducing complex risks. The agency argues that many current best practices were developed before this kind of AI existed, meaning concepts like vulnerability management and risk assessment must evolve to account for AI-enabled systems and attackers.

What we know

The Five Eyes cyber security agencies have previously warned that AI is accelerating the pace and scale of cyber threats. The NCSC’s Cyber Shield initiative sets out a vision for harnessing AI to strengthen cyber defence, but turning this into reality requires more than just technical advances. The new problem book chapter identifies seven areas spanning three broad themes: the security of AI systems themselves, the use of data-driven methods to improve resilience, and human-AI collaboration. A common thread is the need to understand where autonomy adds value and where human judgement remains essential.

The agency emphasises that addressing these challenges requires a collective effort across the cyber security and AI communities. It welcomes feedback on the identified challenges, suggestions for additional research questions and examples of ongoing research from academia, industry and government. The goal is to ensure future cyber security approaches remain effective and trustworthy in an AI-enabled world. The publication serves as a call to action for leaders to consider the choices they make today, which will determine whether the future is secure.

Defender's checklist

• Review current vulnerability management processes to ensure they account for AI-enabled systems.
• Evaluate incident response plans for scenarios involving autonomous AI agents.
• Assess risk management frameworks for the specific threats posed by generative AI.
• Determine where human oversight is essential versus where machine autonomy is safe.

The background in plain language

Organisations must understand that AI systems are no longer just passive tools; they can act with autonomy. This means security teams need to verify that these systems behave as intended, even when faced with unexpected inputs. It is crucial to distinguish between routine tasks that machines can handle and complex decisions requiring human judgement. Teams should ensure that oversight mechanisms are in place to monitor AI actions continuously, preventing unintended consequences while maintaining operational speed and efficiency.

Background: Agentic AI

Agentic AI refers to artificial intelligence systems capable of autonomous decision-making and action. Unlike standard models that only generate text, these agents perceive their environment, break down goals into steps, and use tools to achieve outcomes. They operate with a degree of independence, managing workflows and executing tasks until the objective is met.

Read the full guide: Agentic AI Explained: How Autonomous Systems Change Work

What to do and how to stay safe: NCSC

• Audit existing AI tools to identify where they operate with high autonomy.
• Establish clear protocols for human review of critical AI-driven decisions.
• Monitor vendor advisories for updates on AI security best practices.
• Train staff to recognise signs of AI-generated threats or anomalies.

Step-by-step guide: AI Governance for Small Teams: Practical Controls and Cost

General security guidance from the TechWire Beat newsroom. It is not confirmed advice from the organisations named in this story.

Frequently asked questions

What is the NCSC research problem book?

It is a document that identifies key challenges to focus innovation and research efforts in the cyber security sector.

Why is the NCSC adding an AI chapter?

To address the new risks and opportunities created by generative and agentic AI systems that can act with increasing autonomy.

Who has warned about AI accelerating cyber threats?

The Five Eyes cyber security agencies have warned that AI is increasing the pace and scale of cyber threats.