Summarize concepts related to the use of AI in security operations
New in V4: using AI for analysis and enrichment, the risks it introduces, and recognising AI-enabled threats.
- Reading time
- 20 min read
- Flashcards
- 18 cards
- Practice questions
- 8 questions · 2 PBQs
This objective is new in V4 and has no CS0-003 equivalent, so no legacy study material covers it. It splits three ways, and keeping them separate is most of the work:
- AI as a tool you use in the SOC
- AI as a risk you introduce by using it
- AI as a threat your adversaries use against you
Exam stems will describe a situation and expect you to know which of the three you are in.
AI as a tool
Where machine learning genuinely helps in security operations:
- Anomaly detection — learning a baseline of normal behaviour and flagging deviation. This is the oldest and most established use, and it is what user and entity behaviour analytics does.
- Alert triage and prioritisation — ranking a queue so the analyst starts with what matters.
- Enrichment and summarisation — pulling context together and condensing a long incident into something a manager can read.
- Correlation across sources — spotting the relationship between events no single rule was written for.
- Drafting — detection logic, queries, and report first drafts that a human then verifies.
What AI is good at
- Volume — reading more than a human can
- Pattern recognition across dissimilar sources
- Summarising and drafting
- Consistency — no fatigue at 4 a.m.
What it is bad at
- Being right when it is confident and wrong
- Explaining why, in a way that survives review
- Novel attacks unlike its training data
- Anything you cannot verify
The tellUse AI where a wrong answer is cheap and checkable. Avoid it where a wrong answer is expensive and invisible.
AI as a risk
Hallucination
The model produces fluent, confident output that is simply wrong — a CVE that does not exist, a log field that was never in the schema, an attribution with no basis. The danger is not the error itself but that it arrives phrased exactly like a correct answer.
Data leakage through prompts
Whatever an analyst pastes into a third-party AI service has left the organisation. Log excerpts, incident details, source code, and customer data pasted for "help analysing this" are a disclosure, and in a regulated environment potentially a reportable one.
Shadow AI
Staff using unapproved AI tools without the security team's knowledge — the same problem as shadow IT, with sensitive data in the prompts. The control is an acceptable use policy plus an approved tool that is good enough that people use it.
Prompt injection
Malicious instructions hidden in content the model processes — a web page, a document, a log entry, an email — that the model then follows as though they came from the user. If an AI assistant reads attacker-controlled text, the attacker gets a say in what it does.
Model poisoning
Corrupting training data so the model learns the wrong thing. Against a security model the goal is usually to teach it that the attacker's behaviour is normal, so it never alerts.
Over-reliance and automation bias
Analysts accepting AI output without verification, and skills atrophying because the tool always answers first. The failure is quiet: nobody notices until the day the model is confidently wrong about something that matters.
Check yourself
An analyst pastes a customer database excerpt into a public AI chatbot to help identify anomalies. What is the primary security concern?
AI as a threat
What adversaries gain from the same technology:
AI-generated phishing
Fluent, personalised, error-free messages at scale. The practical consequence: "look for bad grammar" is dead as user-awareness advice, and training has to move to verifying requests through a second channel.
Deepfakes and voice cloning
Synthetic audio or video of a real person, from a small sample of their public speech. Used for business email compromise by phone and for approving fraudulent transfers. The countermeasure is procedural, not technical — a call-back on a known number, or an out-of-band approval step that a convincing voice cannot bypass.
- Accelerated malware development — lowering the skill floor and shortening the time from vulnerability to working exploit
- Automated reconnaissance — gathering and correlating target information at scale
- Evasion — generating variants faster than signature-based detection can keep up, which is another argument for behavioural detection
Check yourself
A finance employee receives a phone call from someone with the CFO's voice authorising an urgent wire transfer. Which control most directly addresses this?
Governing it
The exam expects awareness of the guardrails, not a policy-writing exercise:
- Acceptable use policy for AI tools, naming what may and may not be entered
- Data classification before submission — the question is not "is this tool good" but "may this data leave"
- Human review of AI output before it drives an action or reaches a stakeholder
- Approved tooling, since banning AI outright produces shadow AI rather than no AI
- Logging AI usage, so the organisation knows what was asked and by whom