How SOC Analysts Use AI: Q&A on Speed, Validation & PHP2Shell Response

Q&A recap from our fourth live SOC session | October 30, 2026

---

Thank you to everyone who joined our fourth live Security Operations Center (SOC) session, our last one for October, and sent in questions. Missed the earlier ones? Catch up on Session 1 with Wyatt Quintero, Session 2 with Seth Thompson and Session 3 with Savannah Overly.

This time all three joined as a panel:

• Wyatt Quintero, VP of Security Operations, has been with Critical Start for more than 10 years.
• Savannah Overly, Senior SOC Engineer and AI architect, has spent five and a half years in the SOC.
• Seth Thompson, Senior Manager of Security Operations, has been here more than eight years.

AI in the SOC

Q1: AI investigations dropped from 30 to 45 minutes to under 20. Where does the time go, and how do you keep speed from becoming a shortcut?

Answered by Seth Thompson

People can only look at so many consoles and so much data at once. Automations and AI agents gather all of it faster and prepare a written verdict for the analyst. The analyst then reviews every step the AI took to confirm it pulled the right data and reached the right conclusion.

The time saved goes into measuring the AI: did analysts agree with the verdict, and how is that feedback captured and fed back into the model? It also goes into the process and training work a SOC does to stay sharp. Learn more about SOC AI, read SOC AI: AI-Accelerated, Human-Validated MDR or see the SOC AI press release.

Q2: When SOC AI and analysts disagree, what does the data show?

Answered by Savannah Overly

Every SOC AI thread has a thumbs up and thumbs down option, and Savannah and other engineers review the feedback constantly. For negative feedback they open the thread and ask where the agent missed data or misread something, then use the answer to tune the agent and its prompting. She didn’t quote a specific accuracy percentage on the session.

Trends from that feedback also go into tuning the Trusted Behavior Registry (TBR) and orchestration.

“When our team highlights an opportunity for tuning and disagrees with the agent, we can turn that into a positive permanent piece of orchestration.”— Savannah Overly

Q3: Which investigation tasks have you chosen not to give to AI, and why?

Answered by Wyatt Quintero

Critical Start applies AI where it cuts time to respond most, while staying accurate and consistent:

• Evidence collection and triage
• Blast radius across multiple tools in a customer environment
• Campaigns that span many organizations, such as widespread vulnerabilities or AI-enriched phishing

We also bring each customer’s own investigation paths into the platform as tools, calls and playbooks the investigation agent follows, rather than adding manual steps for analysts.

“Where we’re keeping the analysts completely involved is when we’re making that decision and taking that action.”— Wyatt Quintero

That means response actions, playbooks and any TBR entry that is created or updated. Most customers are still hesitant to automate everything where results can’t be tied to deterministic values. The framework to support faster automated response is being built, and Wyatt expects the balance to shift over the next 3, 6 or 12 months as AI speeds up attacks. See AI By Design, AI Belongs in the SOC. The Shortcut Doesn’t and MDR vs. AI SecOps: Field Notes from Gartner SRM 2026.

Working With Your SOC

Q4: What’s the most common rules of engagement mistake, and what does it cost?

Answered by Seth Thompson

Two mistakes stand out:

• Skipping the “who and what matters” forms at onboarding. Analysts can’t tell your CEO from someone with the same name in HR. Knowing your crown jewels (executives, admins, domain controllers, sensitive hosts) lets us build stronger rules of engagement and escalate faster.
• Not following onboarding best practice guides. Missing log sources, endpoint agents that were never deployed and tools configured without the right access all create blind spots. Check what we support on our integrations page.

“Every time that we don’t configure something correctly… we create a gap.”— Seth Thompson

Seth didn’t put a dollar or minute figure on it. The cost is the time it takes to work out what we should have known up front, and in an incident your time can cost real money. See what fast containment looks like in Rapid Detection and Containment of a Qilin Ransomware Attack, and hear more on accountability in No Asterisk, Ep 1: Building Accountability in MDR.

Threats and Triage

Q5: AiTM and device code phishing bypass MFA. What should customers change in their identity setup?

Answered by Savannah Overly and Wyatt Quintero

Savannah explained that adversary-in-the-middle (AiTM) attacks interrupt a normal login and capture a session token, cookie or authentication code. They’re stealthy, so analysts look at sign-in data, whether the device is trusted or registered, and whether the location and IP address are typical.

Wyatt’s posture advice:

• Know the baseline of your environment and users, including where sensitive accounts or business units have different risk.
• Deploy MFA broadly, enable conditional access where supported and use short session lifetimes.
• Find policy gaps before an incident, not during one.

Identity accounts for roughly 90% of the high and critical true-positive escalations with response actions each week. Endpoint, email and identity are the strongest foundation for spotting risk and responding quickly; network and perimeter monitoring help, but they’re the cherry on top. For real examples, read AiTM Credential Harvesting Attack Contained in 17 Minutes, Email Bombing & Account Compromise and The Hidden Threat of Unmanaged Machine Identities in Enterprises.

Q6: What breaks first when alert volume spikes 10x during a mass exploitation event or zero-day?

Answered by Wyatt Quintero

Spikes come from two places: a major endpoint or identity vendor reclassifying a domain as malicious (it happens quarterly), and a new zero-day that triggers a swarm of vendor detections.

The Trusted Behavior Registry was built before our AI automations to handle this. It groups alerts by shared attributes such as domain, URL, sender email or network configuration. That confirms false positives with confidence and narrows what analysts and agents need to look at. The team excludes only the specific attributes triggering the detection, not the entire domain, because attackers use noisy moments to slip in.

“The Trusted Behavior Registry is our number one assistant when volume spikes occur.”— Wyatt Quintero

During a zero-day, the same approach trims noise so the team can focus on real exploitation signals. It also lets them compare against vulnerability management and other log sources to answer what’s running, where and who is exposed. See Understanding and Defending Against Zero-Day Vulnerabilities, No Asterisk, Ep2: Hunt Less, Detect More, Beyond the Noise and the platform.

Q7: How was Critical Start affected by the PHP vulnerability, and how did you inform customers?

Answered by Seth Thompson

Critical Start wasn’t directly affected. After our Cyber Research Unit published its bulletin on October 21, we triggered our live attack response and checked our own exposure with software inventory, vulnerability scanning and exposure management.

Customer communication went out in two steps:

• October 21: one email to all customers explaining the vulnerability, the CVEs, the fixed versions, what to expect from us and what to do next (inventory, update and restart hosts).
• October 22: targeted emails to each affected customer with the specific host names found in our scans.

Everyone was informed within 24 hours, with exact exposure details for those affected. Read the full PHP2Shell RCE Vulnerabilities bulletin.

What’s Next

Join us for Live SOC Session 5 on Wednesday, November 4, at 1 pm CT with our CTO, Randy Watkins.

Register Now

Browse more on the Resources page, or request a demo to see MDR in action.