You Asked. Savannah Answered. Live SOC Session 3 Recap

Thank you to everyone who joined our third live Security Operations Center (SOC) session and sent in questions ahead of time. Missed the earlier ones? Catch up on Session 1 with Wyatt Quintero (recap | watch) and Session 2 with Seth Thompson (recap | watch).

This time, Savannah Overly, Senior SOC Engineer, answered the questions below, which we grouped by theme. She joined Critical Start more than five years ago as a security analyst, became a lead analyst managing seven people, and now works on SOC training, workflow improvements and SOC AI architecture.

Working With Your SOC

Q1: What's the hardest part of the job outsiders don't see, and what do you wish customers understood?

Every environment differs in risk appetite, tools, team size and alert volume. Two things help the SOC tailor its work. Investigation procedures set the steps analysts follow for each alert, customized per customer. Rules of engagement define which response actions are permitted. Feedback matters too: telling us an activity is expected feeds the TBR.

Savannah's wish is that customers share more context up front. More context means deeper investigations, better escalations, and more proactive tuning. In her words, “The SOC really flourishes when we have a wealth of information.”

“The SOC really flourishes when we have a wealth of information.”— Savannah Overly

Q2: What's the biggest misconception about what the SOC does?

That the SOC already knows your internal IT context, such as change requests, policies or job functions. Analysts can only decide from the data they have. If they cannot tell whether activity is normal, they ask you rather than assume.

Savannah put it this way:

“We are, at the end of the day, the experts on the threats, and the customers help us out by being the experts on their environments.”— Savannah Overly

Q3: What should customers share, and what does success look like?

Start with known-good activity: upcoming user travel, vulnerability scanner IP ranges, approved tools and admins who are expected to run certain activity. Then keep the conversation going. Analysts read every comment sent back on escalated alerts, so your replies shape the next investigation.

Success means the right investigation depth, a confident verdict and a clear path to resolution, whether that is remediation, a TBR entry or no escalation.

AI in the SOC

Q4: How has AI changed investigations, and where is human judgment still essential?

Manual open-source intelligence (OSINT) lookups once took 10 to 20 minutes. SOC AI now automates that context gathering and catches small details people miss. Analysts then apply critical thinking, add customer-specific context and reach the verdict. Read more in AI Belongs in the SOC. The Shortcut Doesn't and The AI MDR Shortcut and Why We're Not Taking It.

Q5: How do you measure whether the AI is right?

AI never closes Tier 1 alerts on its own. A human analyst investigates and decides every one. As Savannah said,

“Every single Tier 1 alert that comes across our desk is going to be investigated and verified and validated by a human analyst.”— Savannah Overly

To measure accuracy, every SOC AI message has a thumbs up and thumbs down. The team reviews feedback regularly, especially negatives, and studies what analysts did instead (the override). Those findings tune the agent's prompting and configuration. Seth covered how the team checks for drift in Session 2. For the risks of AI itself, see AI By Design.

Threats and Triage

Q6: What do attackers do that looks harmless but raises red flags?

Savannah pointed to two phishing techniques:

1. Adversary-in-the-middle (AiTM) phishing: a lookalike sign-in page, often on a typosquatted domain, captures credentials and the multi-factor authentication (MFA) session, then redirects to a legitimate site like Outlook or Wikipedia so the user suspects nothing.
2. Device code phishing: the user signs in on a genuine Microsoft page and enters a code the attacker reuses elsewhere.

A suspicious email, a sign-in page and an odd redirect together is an immediate red flag. See it in action in AiTM Credential Harvesting Attack Contained in 17 Minutes and Strategic Phishing Campaigns Leverage CAPTCHA and Email Routing Misconfigurations.

Q7: How do you avoid alert fatigue while investigating every threat?

Through the Trusted Behavior Registry. Every event is compared against the TBR, a collection of behavior analysts have already judged benign, either universally or for a specific customer. Matches need no review. What remains is the Tier 1 queue: activity that matches nothing known good, which analysts investigate deeply. When an analyst finds new benign activity, they add a TBR entry, so the registry keeps improving. Learn more in The Right Layer for AI Is Rarely First.

Q8: What happens when something is suspicious but not provably malicious?

Trust but verify, and err on the side of caution. Take a scanning tool like Nmap: routine for an IT admin, but possibly an attacker mapping your network. Without context, analysts send you the evidence and the specific gaps. Nothing is closed or filtered unless the team is confident. Wyatt described the same instinct in Session 1: if this were your own environment, would you want to know?

See how correlation works in Email Bombing & Account Compromise, where five alerts became one campaign.

“Trust but verify, and we err on the side of caution.”— Savannah Overly

---

What's Next