Q&A recap from our second live SOC session | October 16, 2026
Thank you to everyone who joined our second live Security Operations Center (SOC) session and sent in questions ahead of time. Seth Thompson, Senior Manager of Security Operations, started as a Critical Start analyst in 2018 and has worked nearly every role in the SOC since. He answered the questions below, which we grouped by theme.
What you'll get from this recap
Every verdict is checked by a person. The team records the verdict SOC AI suggests for an alert and compares it with the verdict the analyst reaches. That shows how often the AI needs a correction and confirms its output reaches you in a format you can use.
On AI-closed alerts that turned out to be true positives, the answer is none, because SOC AI does not close alerts. When it suggests a closure, a human analyst still runs the full investigation and decides whether the alert should close. SOC AI is conversational, so the analyst can ask it to pull in open-source intelligence (OSINT) or another context before deciding.
Drift is tracked in two ways. First, the team samples verdicts every week and reviews them objectively to see whether the AI was right and the analyst was not, and the reverse. Those samples help improve the model.
Second, new features go through change control. SOC engineers, senior SOC engineers, and the engineering and product teams make one change at a time, measure it, refine it, and then move it to full production. Whether a change adds context, adds speed, or adjusts what the AI considers, this process keeps drift out and lets the team measure every update.
Yes, though Seth does not see AI as a new kind of threat actor right now. In his view it lets attackers move faster and at a larger scale, hitting more people and more hosts and spreading more quickly. The attack patterns themselves are the same, so the identity, endpoint, network, and email detections that have always mattered are still the ones to rely on.
The goal is to catch behaviors rather than chase individual tools. Tools and payloads will keep changing, and many in the industry are busy trying to spot specific AI tools. Reconnaissance patterns, automated credential attacks, changing scripts, and fast movement through the attack chain still show up in our detections. What changes is how we use them together. Small, single-purpose detections aimed at specific indicators can be deployed quickly. Looking for a rapid succession of them along an attack chain can point to AI at work, since a person typically cannot move that fast across that much scope.
Detections are only as good as the people investigating them, and the old pattern of triaging one alert and moving to the next does not hold up against attackers at this speed. When a phishing email comes in, analysts need to see more than the person who received it. Do the same headers, senders, or links appear elsewhere in the environment? Do they appear across other customer environments? Is it a campaign, and can it be tied to a threat actor? Threat actors are using these AI tools too.
Our approach pairs stronger correlation and AI on our side with continued human review. We also use the automation and playbooks that come with the Trusted Behavior Registry® (TBR®), so alerts are resolved faster and lessons from one environment apply to all.
There is no single answer, but there are three things that rarely show up in a final write-up.
TBR is doing a lot of work. If your environment generates 10,000 to 12,000 events, you might see only about 10 alerts. TBR draws on years of experience, multiple customer environments, our cyber threat intelligence (CTI), and OSINT sources to make determinations before a person has to look at anything.
SOC AI shortens investigations. An investigation that historically took 30 to 45 minutes because of the sheer volume of content can now take under 20 minutes in many cases. The AI gathers hundreds of data points at once, summarizes them, and then works through the investigation conversationally with the analyst, pulling in more OSINT or context as needed.
Analysts collaborate constantly. Your alert runs through TBR and SOC AI and is verified by a person, who then talks it over with the analysts in the room. Has anyone seen this before? Is there more context? The same happens when 500 alerts arrive at once because a customer updated its whole environment, or when a phishing campaign shows up across several organizations. Analysts compare notes on behaviors and entities, build a larger case, and work with our CTI team to get the findings out to customers.
The team is notified when anything in our console hits a rate limit or logs stop coming in. Because we keep direct access to your consoles through our direct integrations, we can keep investigating there even if API calls are not arriving. The analyst starts from the core alert and investigates manually, and our platform makes that quicker with single-click queries that pull back a lot of data at once.
Two SOCs in two different locations stand ready to cover for each other during an incident. Working directly in a console is also how many of our analysts started, so it is a skill that is still very much alive in the SOC.
Retention is a long-term goal. The longer an analyst stays, the better they know each customer environment. That means less time spent training people on the details and faster recognition of patterns and rules of engagement. Seth called it speed through experience. Mentoring, promotions, and career pathing are things he cares a lot about.
The progression plan has three levels: analyst, senior analyst, and principal analyst. A new analyst completes about 300 hours of training over roughly eight weeks and then tests out. Alerts are generated, the analyst works them while talking through the process with a proctor, and the results are graded. Once an analyst passes, a senior or principal checks everything they do. That gives direct feedback, measures readiness for the next level, and helps management spot future leaders.
The data drives promotion. An analyst is ready to be considered for senior after meeting three requirements: getting alerts right the first time 90 percent of the time over 90 days, 6 to 12 months of tenure, and a certification, usually a CompTIA and a Microsoft certification. After two more weeks of training and another test, the analyst becomes a senior, with a pay raise, a new title, and more responsibility. The same path applies from senior to principal. Most of it happens while analysts are working, so they are often eligible before they realize it.
Promotions are self-paced. Nobody has to wait for an opening to move up, and Seth would love for every analyst in his SOC to become a principal.
Critical Start currently has 90 percent retention, which Seth contrasted with the 50 to 60 percent churn he usually sees in SOCs. Two SOCs, in Dallas and Salt Lake City, add redundancy and more openings for lead roles. Career paths also lead beyond the SOC to onboarding, implementations, incident response, technical account manager (TAM) roles, and technical sales, with defined requirements and a clear starting point for each.
Seth pointed to an incident early in Critical Start's history that led the company to rebuild how the SOC works. It showed the need for validation and two-person integrity, so the team restructured the SOC, reworked its workflows, and added two-person verification across the board. The analyst levels described above came out of that change.
He was frank that everyone in security misses something, and that claiming otherwise is silly. What matters is whether you turn misses into lessons learned. If Critical Start gets something wrong, the first step is to tell you, quickly and plainly, with answers already in hand about what we are doing next.
We will host another live SOC session with a different member of our SOC team, answering questions you submit. Watch our LinkedIn page for details and we will follow up by email.
Still curious? Send your questions to info@criticalstart.com and we will get back to you as soon as we can.
