Incident Detection: Identifying Security Events

The phone rang at 2:47 AM. I don't remember the exact date anymore—after fifteen years of incident response, the midnight calls all blur together—but I remember exactly what the VP of Engineering said when I answered.

"We think we've been breached. Maybe. We're not sure."

"What makes you think that?" I asked, already pulling up my laptop.

"Our customer support team noticed some weird login patterns this morning. Like, customers calling saying they never requested password resets. But we didn't think much of it until about an hour ago when our AWS bill spiked by $47,000 in three hours."

I felt my stomach drop. "How long have the weird login patterns been happening?"

Long pause. "We're not... entirely sure. Maybe a week? Maybe longer?"

By the time we finished the investigation six weeks later, we discovered the breach had started 73 days earlier. The attackers had exfiltrated 2.4 terabytes of customer data, deployed cryptomining malware across 340 EC2 instances, and established persistence in 17 different systems.

Total damage: $8.7 million in direct costs, $23 million in customer churn, and a class-action lawsuit that's still ongoing.

The kicker? Every single attack technique they used was detected by the company's security tools. Every. Single. One. The SIEM had logged it. The IDS had flagged it. The endpoint detection had alerted on it.

But nobody was watching. Nobody had tuned the alerts. Nobody knew what "normal" looked like, so they couldn't recognize "abnormal."

After fifteen years of building detection programs for Fortune 500 companies, federal agencies, healthcare systems, and startups, I've learned one brutal truth: having security tools doesn't mean you're detecting security events. Most organizations are drowning in alerts while simultaneously blind to actual attacks.

And it's costing them everything.

The $23 Million Question: Why Incident Detection Matters

Let me tell you about two companies I consulted with in 2022. Both were SaaS platforms, similar size (around 400 employees), similar tech stack, similar customer base. Both got breached within three months of each other.

Company A detected the breach in 4 hours and 23 minutes. They contained it in 6 hours, eradicated the threat in 12 hours, and notified affected customers within 24 hours. Total damage: $340,000 in incident response costs, zero customer data exfiltrated, minimal reputation impact.

Company B detected the breach 47 days after initial compromise. By then, the attackers had exfiltrated 890GB of customer data, established backdoors in 23 systems, and sold the data on dark web markets. Total damage: $11.4 million in direct costs, 34% customer churn, regulatory fines, and a damaged reputation that still hasn't recovered.

The difference between these companies wasn't their security budget. Company B actually spent more on security tools. The difference was detection capability.

Company A knew what to look for, how to look for it, and who was looking. Company B had all the tools but no coherent detection strategy.

"Incident detection isn't about having the most expensive tools or the largest security team—it's about having the right visibility, the right baselines, and the right people asking the right questions at the right time."

Table 1: Impact of Detection Speed on Breach Costs

The data is clear: every hour matters. Every day matters. The difference between detecting a breach in 4 hours versus 4 weeks is literally the difference between a manageable incident and an existential threat.

Understanding the Incident Detection Landscape

Before we dive into how to detect security events, you need to understand what you're actually trying to detect. This sounds obvious, but I've consulted with organizations that couldn't articulate the difference between an event, an alert, an incident, and a breach.

I worked with a financial services company in 2020 that was generating 847,000 "security incidents" per day. Except they weren't incidents—they were events. Their SOC analysts were drowning in noise, spending 94% of their time on false positives and 6% on actual investigation.

We rebuilt their detection framework from the ground up. Within six months, they were down to 1,200 meaningful alerts per day, with 78% true positive rate. Their mean time to detect dropped from 14 days to 3.7 hours.

The key was understanding the detection hierarchy.

Table 2: Security Detection Hierarchy

I worked with a healthcare provider that didn't understand this hierarchy. They were treating every failed login attempt as an incident requiring formal investigation. They had a 12-person SOC team that couldn't keep up with 50,000+ "incidents" daily.

We implemented proper filtering: events → indicators → alerts → notable events → incidents. Within 90 days, their SOC was investigating an average of 47 actual incidents daily instead of drowning in 50,000 meaningless alerts. Detection quality went up. Analyst burnout went down. Actual threats got addressed.

The Three Pillars of Effective Detection

After building detection programs for 40+ organizations, I've identified three fundamental pillars that separate effective detection from security theater.

Every successful detection program I've implemented has had all three. Every failed program I've fixed was missing at least one.

Pillar 1: Comprehensive Visibility

You cannot detect what you cannot see. Sounds obvious, but I've responded to breaches where the attackers operated in blind spots for months.

I investigated a breach at a manufacturing company in 2021 where attackers accessed the network through a forgotten VPN concentrator that nobody was monitoring. The device had been installed six years earlier for a temporary contractor project and never decommissioned. No logs were being collected. No alerts were configured. Perfect blind spot.

The attackers used it for 127 days before we discovered it during the forensic investigation.

Table 3: Critical Visibility Domains

I consulted with a company that had invested $2.4 million in a state-of-the-art SIEM but wasn't collecting logs from their most critical application—a custom-built order processing system handling $400 million in annual transactions. The SIEM was beautiful and completely useless for detecting attacks against their most valuable asset.

We spent $67,000 integrating the application logs. Within three weeks, we detected a sophisticated fraud scheme that had been running for 14 months, costing the company an estimated $8.4 million.

ROI on that $67,000 investment: immediate and massive.

Pillar 2: Behavioral Baselines

The second pillar is understanding normal so you can recognize abnormal. This is where most organizations fail spectacularly.

I worked with a SaaS platform in 2019 that had excellent visibility—they collected everything. But when I asked, "What does normal look like?", nobody could answer. They had two years of security logs and zero understanding of baseline behavior.

When unusual activity occurred, they had no context. Was 47 failed logins in an hour normal? They didn't know. Was 2.3GB of outbound traffic from the database server normal? They didn't know. Was a finance employee accessing the engineering code repository normal? They didn't know.

We spent four months establishing baselines across 23 critical dimensions. Once we knew "normal," the abnormal became obvious.

Table 4: Critical Behavioral Baselines

Here's a real example: We established that a specific database administrator typically executed between 12-28 queries per day, always during business hours (8 AM - 6 PM EST), always from two specific IP addresses (office and home).

One Tuesday at 3:47 AM, the baseline detected 147 queries from an IP address in Romania. The SOC analyst investigating found the DBA's credentials had been compromised via a phishing attack three days earlier.

Because we had the baseline, we detected the anomaly in 14 minutes. Without the baseline, it would have looked like normal database activity.

Total data accessed before we locked down the account: 47 records. Total data accessed in similar breaches without behavioral detection: tens of thousands to millions of records.

Pillar 3: Skilled Analysis

The third pillar is having people who know what they're looking for and how to investigate what they find.

I cannot count the number of times I've seen organizations spend millions on security tools and hire entry-level analysts with zero training to operate them. It's like buying a Formula 1 race car and asking someone who just got their learner's permit to drive it.

I consulted with a financial services company in 2023 that had a six-person SOC operating 24/7. Average experience level: 8 months in cybersecurity. They were overwhelmed, constantly escalating false positives, and missing real threats.

We restructured the team: two senior analysts (5+ years experience), three mid-level analysts (2-4 years), and three junior analysts. We implemented a tier structure with defined escalation paths and intensive training programs.

Within six months:

• Mean time to detect: 14.3 hours → 2.1 hours

• False positive rate: 87% → 23%

• Analyst retention: 40% annual turnover → 8%

• Critical incidents missed: 3-4 per quarter → 0

The investment in experience and training paid for itself in the first two months through reduced incident response costs.

Table 5: Detection Team Structure and Capabilities

Detection Methods and Technologies

Now let's talk about the actual methods and technologies used for detection. I'll save you from the vendor marketing nonsense and tell you what actually works based on real implementations.

I've deployed every category of detection technology available. Some are essential. Some are nice-to-have. Some are expensive mistakes.

Table 6: Detection Technology Categories

Let me share real-world effectiveness data from a company I worked with that implemented all of these over a three-year period:

Year 1: Deployed SIEM, EDR, Email Security (essentials)

• Total investment: $340,000

• Detection capability: 65% of attack techniques

• Mean time to detect: 18.4 hours

Year 2: Added NDR, UEBA, Threat Intelligence

• Additional investment: $280,000

• Detection capability: 87% of attack techniques

• Mean time to detect: 4.7 hours

Year 3: Added CASB, DLP, Deception Technology

• Additional investment: $310,000

• Detection capability: 94% of attack techniques

• Mean time to detect: 2.3 hours

The key insight: the first 65% of detection capability cost $340,000. Getting from 65% to 94% cost an additional $590,000. But that last 29% of coverage detected the most sophisticated attacks—the ones that matter most.

Building Detection Use Cases

Here's where theory meets practice. You need specific detection use cases that map to real attack techniques.

I worked with a government contractor in 2022 that had a SIEM with exactly one detection rule: "Alert if login fails more than 10 times." That was it. One rule. They were paying $240,000 annually for a SIEM with one detection rule.

We built out 147 detection use cases covering the MITRE ATT&CK framework. Within the first month, we detected:

• 3 instances of credential dumping

• 7 lateral movement attempts

• 2 data staging operations

• 12 persistence mechanisms

• 5 defense evasion techniques

None of these would have triggered the "10 failed logins" rule. They were operating completely undetected.

Table 7: Essential Detection Use Cases by Attack Phase

I'll give you a specific example from a healthcare company I worked with in 2021:

Use Case: Detect credential dumping via LSASS access

Data Sources:

• Windows Event ID 4656 (handle to object requested)

• Windows Event ID 4663 (attempt to access object)

• EDR process monitoring

Detection Logic:

(EventID=4656 OR EventID=4663) AND ObjectName="*lsass.exe" 
AND ProcessName!="C:\Windows\System32\wbem\WmiPrvSE.exe" 
AND ProcessName!="C:\Windows\System32\svchost.exe"
AND AccessMask="0x1410"

Tuning: Excluded legitimate system processes, adjusted to known good access patterns

Results:

• Detected 3 actual credential dumping attempts in first 90 days

• False positives: 2 per week (manageable)

• Prevented one lateral movement campaign that could have escalated to full network compromise

The estimated cost of that prevented breach: $4.7 million based on similar incidents in their industry.

Cost to develop and maintain that detection use case: $8,400 over 12 months.

ROI: absolutely massive.

The Detection Maturity Model

Not every organization needs the same level of detection maturity. A 50-person startup doesn't need the same program as a Fortune 500 bank.

I developed this maturity model after working with organizations at every stage of detection capability. It helps companies understand where they are and what the next step should be.

Table 8: Detection Maturity Progression

I worked with a company that jumped from Level 1 to Level 4 in 18 months. They spent $3.2 million doing it. Six months later, they got breached anyway because they didn't have the operational maturity to use the tools effectively.

Meanwhile, I worked with another company that went from Level 2 to Level 4 over 36 months, spending $1.8 million total. They haven't had a successful breach in four years because they built capability gradually with operational excellence at each stage.

The lesson: maturity takes time. Tools are easy to buy. Capability is hard to build.

Framework-Specific Detection Requirements

Every compliance framework has opinions about incident detection. Let me cut through the confusion and tell you what each framework actually requires.

Table 9: Framework Detection Requirements

Here's what this looks like in practice. I worked with a healthcare SaaS company that needed to comply with HIPAA, SOC 2, and PCI DSS simultaneously.

Their detection requirements ended up being:

• SIEM with 1-year online retention (most stringent: PCI DSS)

• File integrity monitoring on all systems with ePHI or cardholder data

• Daily log review (PCI DSS minimum)

• Incident response procedures meeting 72-hour notification (GDPR, even though not explicitly required, became the de facto standard)

• Documented monitoring procedures across all in-scope systems

Instead of implementing three separate detection programs, we built one that satisfied the most stringent requirement from each framework. Total cost: $680,000 over 12 months. Cost of three separate programs: estimated $1.9 million.

Common Detection Failures and How to Avoid Them

I've investigated hundreds of breaches. The vast majority could have been detected earlier—sometimes much earlier—if not for common, predictable failures.

Let me share the top 10 detection failures I see repeatedly, along with real costs from actual incidents.

Table 10: Top 10 Detection Failures

Let me tell you about the most expensive detection failure I personally investigated.

A financial services company had a world-class SIEM generating about 40,000 alerts daily. They had a six-person SOC working 24x7. Everything looked good on paper.

But they had massive alert fatigue. The SOC had learned to ignore certain alert categories because they were "always false positives." One of those categories was "unusual database access patterns."

An insider—a database administrator—began slowly exfiltrating customer financial records. The SIEM detected it immediately and generated alerts. For 89 days. Every single day, the alert was generated. Every single day, it was ignored.

When we investigated, we found 89 consecutive alerts, all marked as "false positive - ignore" by SOC analysts who never actually investigated.

Total records exfiltrated: 840,000 customer accounts Total data: 2.1 TB Direct breach costs: $23 million Regulatory fines: $14 million Lawsuits: ongoing, estimated $50+ million Total impact: $87+ million and counting

All because they had trained themselves to ignore alerts.

The fix isn't complicated: if an alert fires repeatedly and is always a false positive, tune the rule or delete it. Never train your team to ignore alerts.

Building an Effective Detection Program: 180-Day Roadmap

When organizations ask me, "How do we build detection capability from scratch?", I give them this 180-day roadmap. It's based on successful implementations at organizations ranging from 200 to 20,000 employees.

Table 11: 180-Day Detection Program Implementation

I implemented this exact roadmap at a healthcare technology company with 1,200 employees in 2022.

Starting point:

• No SIEM

• No EDR

• No formal detection capability

• Mean time to detect: 45+ days (when they detected anything at all)

After 180 days:

• Full SIEM deployment (Splunk)

• EDR on 100% of endpoints (CrowdStrike)

• 67 production detection use cases

• Mean time to detect: 3.2 hours

• False positive rate: 17%

• Zero successful breaches in 18 months since implementation

Total investment: $680,000 Annual operating cost: $840,000 (including full SOC team) Avoided breach costs (based on industry averages): $8-12 million over 18 months

ROI: massive and immediate.

Advanced Detection: Threat Hunting

Once you have solid foundation detection in place, the next evolution is proactive threat hunting—looking for threats before alerts fire.

I started doing threat hunting in 2013 before it had a formal name. We just called it "looking for bad stuff that the tools didn't catch."

The best threat hunting program I built was for a financial services company in 2020. We started with hypothesis-driven hunts based on threat intelligence, evolved to data-driven hunts based on anomalies, and eventually built a continuous hunting program.

Table 12: Threat Hunting Maturity and Results

At that financial services company, our threat hunting program found:

• Month 1: 2 findings (1 true positive - unauthorized admin account)

• Month 6: 7 findings per month average (4.2 true positives - including one pre-ransomware deployment)

• Month 12: 14 findings per month average (10.1 true positives - prevented 3 significant breaches)

The pre-ransomware detection alone justified the entire program. We found staging behavior 18 hours before the ransomware would have deployed. Estimated cost of that prevented ransomware attack: $8-15 million based on similar incidents.

Cost of the hunting program: $280,000 annually.

Metrics That Matter: Measuring Detection Effectiveness

You need to measure detection effectiveness, but most organizations measure the wrong things.

I consulted with a company that proudly reported "99.7% alert response rate" to their board. Sounds impressive until you realize they were responding to alerts by clicking "acknowledge" without investigating. Their actual investigation rate was 12%.

Meanwhile, they were missing breaches that lingered for weeks.

Here are the metrics that actually matter, based on programs I've built and measured:

Table 13: Essential Detection Metrics

I implemented this metrics program at a technology company in 2021. Here's what happened over 12 months:

Starting Metrics (Month 1):

• MTTD: 18.7 hours

• MTTI: 8.3 hours

• False Positive Rate: 84%

• True Positive Rate: 34%

• Detection Coverage: 41% of ATT&CK

Ending Metrics (Month 12):

• MTTD: 2.1 hours

• MTTI: 1.4 hours

• False Positive Rate: 19%

• True Positive Rate: 87%

• Detection Coverage: 89% of ATT&CK

The improvement wasn't magical—it was systematic tuning, training, and continuous optimization.

The Future of Incident Detection

Let me end with where I see detection heading based on what I'm implementing with forward-thinking clients today.

AI/ML-Powered Detection: Everyone talks about AI in security. Most of it is marketing nonsense. But genuine machine learning for behavioral analysis is already proving valuable. I've implemented UEBA solutions that detected insider threats and compromised credentials weeks before traditional rules would have flagged them.

Automated Investigation: SOAR platforms are evolving from simple automation to intelligent investigation orchestration. The best implementations I've seen reduce MTTI by 60-75% for common alert types.

Deception Technology: I've deployed deception at three organizations in the past two years. The results are remarkable—100% true positive rate (if an alert fires, it's definitely bad), near-instant detection of lateral movement, and attackers revealing their TTPs by interacting with decoys.

Cloud-Native Detection: As workloads move to cloud, detection must follow. The most advanced programs I'm building now have cloud-native detection that's equally sophisticated as traditional infrastructure monitoring.

Threat Intelligence Integration: Moving beyond simple IOC matching to understanding adversary campaigns, TTPs, and targeting. The best threat intel integrations I've built feed directly into detection logic and hunting hypotheses.

But here's my prediction for the biggest change: detection will become inseparable from response.

Right now, detection and response are separate phases. In five years, they'll be a single continuous flow. You'll detect, immediately contain at machine speed, investigate while contained, and either remediate or release based on investigation findings. All within minutes, largely automated.

We're not there yet. But we're getting close.

Conclusion: Detection as Strategic Defense

I started this article with a company that detected their breach 73 days late and paid $23 million for that detection failure. Let me tell you how that story actually ended.

After the breach, they rebuilt their entire detection program from scratch. Total investment over 18 months: $2.3 million.

In the three years since, they've detected and stopped:

• 12 ransomware deployment attempts

• 7 data exfiltration campaigns

• 23 lateral movement operations

• 4 insider threat situations

• 89 compromised account incidents

Every single one of these was detected within 4 hours of initial indicators. Every single one was contained before significant damage occurred.

Estimated total cost of those prevented breaches: $47+ million.

ROI on that $2.3 million investment: 2,043%.

But more importantly, the CISO sleeps at night now. So does the board.

"Detection isn't about perfect prevention—it's about seeing threats early enough that you can respond before they become catastrophes. The difference between detection in 4 hours and detection in 4 weeks is literally the difference between an incident and an existential crisis."

After fifteen years building detection programs, here's what I know for certain: organizations with mature detection capabilities don't prevent all breaches, but they prevent breaches from becoming disasters.

The attackers are already inside your network. Right now. The question isn't "will we get breached?" The question is "how quickly will we detect it?"

And that question determines whether you're paying for an incident response or paying for a company-ending catastrophe.

You can build detection capability now, when you have time and budget to do it right. Or you can build it later, during the panicked all-hands meeting after the breach makes headlines.

I've helped organizations in both scenarios. Trust me—the first way is cheaper, faster, and far less painful.

The choice is yours. But choose quickly. Because somewhere, right now, there's activity in your logs that you're not seeing. Activity that's normal. Activity that's just a little bit unusual. Activity that's the early warning of what becomes next month's crisis.

The question is: are you looking?

Need help building your incident detection program? At PentesterWorld, we specialize in practical detection engineering based on real-world breach experience. Subscribe for weekly insights on detecting what matters.