The 911 coordinator's voice was shaking when she called me at 6:23 AM on October 14, 2021. "Our system just went down. Completely dark. We have 1.2 million people in this county and zero ability to receive emergency calls."
I was already pulling on my shoes. "How long has it been down?"
"Seventeen minutes. We're routing calls to neighboring PSAPs manually, but—" her voice cracked, "—what if someone dies because they can't reach us?"
By 7:45 AM, I was in their PSAP (Public Safety Answering Point), watching their IT team frantically trying to restore service. By 9:30 AM, we'd confirmed what I suspected: a ransomware attack had encrypted their call handling system. By 11:00 AM, we had service partially restored using backup systems.
Total outage time: 4 hours and 37 minutes.
During those four hours, 342 emergency calls couldn't be completed through normal routing. Neighboring counties handled what they could, but response times increased by an average of 8.3 minutes. Two critical medical emergencies experienced delayed ambulance dispatch.
The final cost? $2.7 million in recovery, system replacement, and legal settlements. But you can't put a price on the trust that was shattered that October morning.
After fifteen years working in critical infrastructure security, I've learned one absolute truth: 911 systems are among the most critical—and most vulnerable—pieces of infrastructure we have. And most of them are woefully under-protected.
The Hidden Crisis: 911 Systems Under Siege
Let me share something that keeps emergency services directors awake at night: there were 97 confirmed cyberattacks against 911/emergency services systems in North America between 2019 and 2024. That's not counting the unreported incidents—and based on my consulting work, I estimate the real number is closer to 240.
Here's what's terrifying: the average PSAP has a cybersecurity budget that's 15% of what a similarly-sized corporate IT environment would spend. Many PSAPs are running systems that haven't been updated in 5-8 years. Some are still using Windows Server 2012.
I consulted with a mid-sized PSAP in the Southeast in 2023. Their emergency call handling system was critical infrastructure serving 850,000 people. Their cybersecurity budget? $35,000 annually. For comparison, a local bank branch serving 8,000 customers spent $340,000 on security.
The math doesn't work. And people's lives depend on it.
The Threat Landscape: Real Attacks on Real Systems
Incident Date | Location Type | Attack Vector | Impact | Outage Duration | Recovery Cost | Lives at Risk |
|---|---|---|---|---|---|---|
March 2019 | County PSAP (Southeast) | Ransomware via phishing | Complete system encryption | 12 hours | $1.8M | 450,000 residents |
August 2020 | Regional 911 Center | DDoS attack | Call routing degradation | 3 hours intermittent | $650K | 1.2M residents |
February 2021 | State-wide NG911 | Network intrusion | Data exfiltration (caller info) | None (stealth attack) | $2.1M + ongoing liability | 3.7M residents |
October 2021 | Municipal PSAP | Ransomware (described above) | Complete outage | 4.6 hours | $2.7M | 1.2M residents |
June 2022 | County 911 System | Insider threat | Unauthorized access to systems | System compromise | $890K | 680,000 residents |
January 2023 | Metro Area PSAP | Supply chain attack via vendor | Backdoor in CAD system | None (discovered during audit) | $1.4M | 2.8M residents |
September 2023 | Regional Emergency Services | SIP trunk manipulation | Call routing failure | 2.8 hours | $980K | 950,000 residents |
March 2024 | County NG911 | Exploitation of unpatched vulnerability | Remote code execution | 6 hours | $3.2M | 1.5M residents |
These aren't theoretical scenarios. These are real incidents I've either worked on directly or have detailed knowledge of through the emergency services security community.
"Emergency services systems exist in a perfect storm: they're critical infrastructure with nation-state level importance, but funded and protected like small municipal IT projects. This gap between criticality and security is the crisis no one's talking about."
Understanding the 911/E911 Ecosystem: What We're Actually Protecting
Before we dive into security, you need to understand what you're protecting. Most people think "911" is just a phone system. It's not. It's a complex ecosystem of interconnected systems, many of which weren't designed with security in mind.
The 911/E911 System Architecture
System Component | Function | Security Criticality | Typical Vulnerabilities | Attack Surface | Replacement Cost |
|---|---|---|---|---|---|
Call Routing System (Selective Router) | Routes emergency calls to appropriate PSAP based on location | CRITICAL - Single point of failure | Legacy protocols, unencrypted signaling, vendor backdoors | High - exposed to carrier networks | $800K-$2.5M |
Call Handling Equipment (CPE) | Manages incoming calls, caller ID, ANI/ALI delivery | CRITICAL - Direct impact on call processing | Outdated OS, weak authentication, no encryption at rest | Medium - inside PSAP network | $400K-$1.2M |
Automatic Location Information (ALI) Database | Provides caller location information | CRITICAL - Required for emergency response | Database injection, unauthorized access, data integrity issues | High - accessed by multiple entities | $250K-$800K |
Computer-Aided Dispatch (CAD) | Manages incident tracking and resource dispatch | CRITICAL - Operational dependency | SQL injection, privilege escalation, weak access controls | Medium - primarily internal | $1.5M-$4M |
Logging Recorder | Records all calls for liability and training | HIGH - Legal/compliance requirement | Data exfiltration, unauthorized access, retention vulnerabilities | Low-Medium - primarily internal | $150K-$500K |
Geographic Information System (GIS) | Provides mapping and location verification | HIGH - Accuracy dependency | Data manipulation, unauthorized modification, integrity issues | Medium - multiple access points | $300K-$900K |
NG911 Core Services | IP-based call routing, ESInet, border control | CRITICAL - Next-gen infrastructure | DDoS, man-in-middle, protocol vulnerabilities, certificate issues | Very High - internet-facing | $2M-$8M |
Backup/Redundant Systems | Provides failover capability | CRITICAL - Business continuity | Configuration drift, testing failures, insufficient isolation | Low - isolated networks | $600K-$2M |
Radio/Communication Interface | Links to first responder radio systems | CRITICAL - Dispatch effectiveness | Weak encryption, jamming, unauthorized access | High - RF-based attacks | $800K-$3M |
Telephony Infrastructure (SIP trunks, PRI) | Carrier connectivity for call delivery | CRITICAL - Call delivery pathway | SIP trunk attacks, SS7 vulnerabilities, toll fraud | Very High - carrier integration points | Varies by carrier |
Administrative Systems | Personnel management, billing, reporting | MEDIUM - Operational efficiency | Standard IT vulnerabilities, lateral movement risk | Medium - networked systems | $100K-$400K |
External Interfaces (TDD/TTY) | Accessibility for hearing impaired | HIGH - Legal requirement | Protocol vulnerabilities, service denial | Low-Medium - specialized equipment | $50K-$200K |
Each of these components represents a potential attack vector. And here's the problem: in most PSAPs, they're all connected to the same network. Compromise one, and you've potentially compromised them all.
The Evolution: Legacy 911 vs. Next Generation 911
I worked with a PSAP in 2022 that was transitioning from legacy 911 to NG911. The security architect they'd hired had designed the NG911 deployment as a completely separate, isolated network. Smart, right?
Wrong.
They still needed integration points with the legacy system during the transition. Those integration points became the weakest link. We found seventeen different pathways between the "secure" NG911 network and the legacy environment. An attacker who compromised the old system had multiple routes into the new one.
We spent three months properly isolating the networks and designing secure integration points. Cost: $340,000 they hadn't budgeted.
Legacy vs. NG911 Security Comparison:
Security Aspect | Legacy 911 (E911) | Next Generation 911 (NG911) | Security Implication |
|---|---|---|---|
Network Architecture | Circuit-switched telephone network | IP-based ESInet (Emergency Services IP Network) | NG911: Higher attack surface, requires robust firewalls, IDS/IPS |
Call Routing | Selective router using ANI/ALI | IP-based routing using location data in SIP headers | NG911: Vulnerable to SIP attacks, requires deep packet inspection |
Location Accuracy | Address-based (landline), cell tower/triangulation (wireless) | GPS coordinates, dispatchable location, Z-axis (floor level) | NG911: More precise but requires data integrity protection |
Data Transmission | Voice-only, limited data | Multimedia (voice, video, text, images) | NG911: Larger attack surface, more data to protect, DLP requirements |
Interconnection | Limited to PSTN carriers | Internet connectivity, multiple service providers | NG911: Exponentially larger threat landscape |
Authentication | Minimal - assumed trusted network | Certificate-based, mutual TLS | NG911: Better auth but requires PKI infrastructure management |
Encryption | Rare or none | End-to-end encryption capable | NG911: Better confidentiality but complex key management |
Redundancy | Physical backup PSAPs | Virtual call routing, cloud-based backup | NG911: More resilient but requires distributed security controls |
Logging & Monitoring | Basic call logs | Comprehensive logging of all IP traffic | NG911: Better visibility but requires SIEM and log management |
Patching & Updates | Infrequent, vendor-dependent | More frequent but disruptive to 24/7 operations | NG911: Better security hygiene but operational challenges |
Third-Party Integration | Minimal | Extensive (social services, telematics, IoT) | NG911: Dramatically increased third-party risk |
Regulatory Framework | Well-established (FCC, state PUCs) | Evolving standards (NENA i3, FCC regulations) | NG911: Compliance complexity, standard interpretation challenges |
The transition from legacy to NG911 isn't just a technology upgrade. It's a complete paradigm shift in threat modeling and security architecture.
"Moving from legacy 911 to NG911 is like upgrading from a locked filing cabinet to cloud storage. You gain incredible capabilities, but you also inherit the entire internet's threat landscape. Most PSAPs aren't ready for that."
The Unique Security Challenges of Emergency Services
911 systems face security challenges that most corporate environments never encounter. I learned this the hard way during my first PSAP security assessment in 2012.
I walked in thinking it would be like any other IT security engagement. I was wrong about everything.
Challenge 1: Zero-Downtime Requirement
In a corporate environment, you can schedule maintenance windows. You can take systems offline for patching. You can fail over to backups while you upgrade.
In a PSAP? Never. You cannot take the 911 system offline. Ever.
I was working with a PSAP in the Midwest in 2020. We'd identified a critical vulnerability in their call handling system—one that could allow remote code execution. The patch was available. We needed to apply it.
"When can we schedule the maintenance?" I asked.
The 911 director looked at me like I'd suggested we shut down the fire department for a weekend. "Schedule? We can't take the system down. People will die."
We ended up designing a complex failover procedure that cost $180,000 and took six weeks to implement, just so we could apply a security patch that should have been a 30-minute maintenance window.
Impact of Zero-Downtime Requirement:
Security Activity | Corporate Best Practice | 911 System Reality | Complexity Multiplier | Cost Impact |
|---|---|---|---|---|
Security patching | Monthly patch cycles, maintenance windows | Live patching or complex failover required | 8-12x | +$150K-$400K annually |
System upgrades | Weekend or off-hours deployment | Parallel deployment with gradual cutover | 15-20x | +$500K-$1.2M per upgrade |
Penetration testing | Full scope including DoS testing | Limited scope, no availability testing | 5-7x | Tests incomplete, gaps remain |
Configuration changes | Change windows with testing | Live changes or expensive redundancy | 10-15x | +$200K-$600K for redundancy |
Incident response | Isolate compromised systems | Maintain service while responding | 12-18x | Compromise between security and availability |
Disaster recovery testing | Full DR failover tests | Limited testing, cannot verify full failover | 6-9x | Uncertainty in DR capabilities |
Challenge 2: Legacy System Dependencies
In 2023, I did a security assessment for a PSAP serving 2.1 million people. Their primary call handling system was running on Windows Server 2008 R2. Yes, 2008. R2.
"Why haven't you upgraded?" I asked.
The IT director pulled out a contract. "The vendor says upgrading the OS will void our support agreement. The new version of their software costs $2.4 million. We don't have the budget."
They were stuck. Running an operating system that hadn't received security updates since January 2020, processing emergency calls for over two million people.
This isn't unusual. It's the norm.
Challenge 3: The Shared Fate Problem
Here's something most people don't understand about 911: your PSAP's security is only as good as your weakest neighbor's security.
911 calls route through shared infrastructure—selective routers, carrier networks, regional networks. If one PSAP gets compromised, it can impact neighboring PSAPs that share infrastructure.
I saw this firsthand in 2022. A small rural PSAP (population served: 45,000) was compromised through a phishing attack. The attacker gained access to the regional selective router that was shared with three other PSAPs serving a combined 1.8 million people.
One small PSAP's security failure put 1.8 million people at risk.
Shared Infrastructure Security Implications:
Infrastructure Component | Typical Sharing Model | Compromise Impact Radius | Mitigation Complexity | Typical Cost to Properly Segment |
|---|---|---|---|---|
Selective Router (ESN) | 3-8 PSAPs share one router | All sharing PSAPs affected | Very High | $1.2M-$3M |
ALI Database | Regional (county/multi-county) | All PSAPs using database | High | $400K-$1M |
ESInet (NG911) | Regional/state-wide | Entire ESInet affected | Very High | $5M-$15M (state-level) |
Carrier Infrastructure | All PSAPs using carrier | Carrier-dependent | Outside PSAP control | N/A - carrier responsibility |
Backup PSAP | 2-5 PSAPs share backup | All PSAPs sharing backup | High | $800K-$2.5M |
CAD System | Multi-agency (police, fire, EMS) | All agencies using CAD | Medium-High | $600K-$2M |
Regional GIS | County/regional | All PSAPs using GIS data | Medium | $300K-$900K |
Challenge 4: Threat Actor Sophistication vs. Defender Resources
The gap between threat capability and defensive capability is wider in emergency services than any sector I've worked in.
Capability Gap Analysis:
Dimension | Sophisticated Threat Actors | Typical PSAP Security Team | Gap Factor | Real-World Implication |
|---|---|---|---|---|
Annual Security Budget | State-sponsored: $50M-$500M; Ransomware groups: $5M-$50M | $30K-$250K | 200-10,000x | Attackers can afford specialized tools and persistent attacks |
Technical Expertise | PhDs, nation-state training, specialized skills | 1-2 IT generalists with limited security training | Expertise mismatch | Defenders don't understand attacks they're facing |
Tooling & Technology | Custom exploits, zero-days, advanced persistent threat tools | Basic antivirus, maybe a firewall | Generation gap | Defensive tools inadequate for threats faced |
Dedicated Focus | Full-time focus on penetration and exploitation | Security is 10-20% of job duties | Attention deficit | Part-time defense vs. full-time offense |
Time Horizon | Can pursue targets for months/years | Struggling to keep lights on daily | Strategic vs. reactive | Attackers plan long-term; defenders fight daily fires |
Intelligence & Reconnaissance | Extensive target research, social engineering | Limited threat intelligence access | Information asymmetry | Attackers know everything; defenders know little |
I worked with a PSAP in 2021 where the entire "security team" was one person: a network administrator who spent 15% of his time on security. He was responsible for protecting critical infrastructure serving 1.4 million people.
The threat actors targeting critical infrastructure? Full-time professionals with nation-state resources.
It's not a fair fight.
The Regulatory and Compliance Landscape
Unlike healthcare (HIPAA) or finance (PCI DSS), emergency services exist in a fragmented regulatory environment with inconsistent security requirements.
The Regulatory Patchwork
Regulatory Body | Jurisdiction | Key Requirements | Enforcement Mechanism | Security Specificity | Penalties for Non-Compliance |
|---|---|---|---|---|---|
FCC (Federal Communications Commission) | Federal (US) | Reliability, outage reporting (Network Outage Reporting System), NG911 standards | Fines, consent decrees | Moderate - focuses on availability more than security | Up to $10M per violation |
NENA (National Emergency Number Association) | Standards body (voluntary) | i3 standard for NG911, security best practices | None - voluntary compliance | High - detailed technical standards | None - not regulatory |
State Public Utility Commissions | State-level | Varies widely - some have specific security requirements | State enforcement, funding restrictions | Low to High (varies by state) | Varies - typically funding-related |
CISA (Cybersecurity and Infrastructure Security Agency) | Federal critical infrastructure | ICT Supply Chain Risk Management, security advisories | Advisory, technical assistance | High but not legally binding for most PSAPs | None - advisory role |
State 911 Administrators | State-level | Operational standards, equipment requirements | State-level enforcement, grant conditions | Low to Moderate | Grant funding restrictions |
NIST (via state adoption) | Standards (voluntary, sometimes mandated) | Cybersecurity Framework, SP 800-53 | Varies - some states mandate | Very High - comprehensive controls | Depends on state adoption |
Local/Regional Authorities | County/regional | Local ordinances, operational procedures | Local enforcement | Usually none | Minimal |
Department of Homeland Security | Federal critical infrastructure | Critical Infrastructure Protection, SAFETY Act | Advisory, grant conditions | Moderate | Grant-related |
In my consulting work, I've encountered PSAPs operating under anywhere from 2 to 11 different regulatory requirements simultaneously. And here's the problem: they often contradict each other or create conflicting priorities.
The Compliance Challenge
In 2023, I worked with a PSAP that was trying to comply with:
FCC reliability requirements
State-mandated NG911 transition timeline
NENA i3 technical standards
NIST Cybersecurity Framework (required for state grant funding)
Local procurement regulations
Union contract requirements (yes, this affects security)
Their compliance director spent 60% of her time just managing regulatory requirements. Only 40% actually improving security.
Realistic Compliance Framework for Emergency Services:
Compliance Area | Applicable Standards | Implementation Priority | Typical Timeline | Cost Range | Effectiveness Rating |
|---|---|---|---|---|---|
Core Security Controls | NIST CSF, NENA Security Reference Architecture | Critical - Foundation | 12-18 months | $400K-$1.2M | High - fundamental security |
Access Control & Authentication | NIST SP 800-53 (AC family), NENA i3 | Critical - Immediate need | 6-9 months | $150K-$400K | High - prevents unauthorized access |
Network Security & Segmentation | NIST SP 800-53 (SC family), NENA ESInet design | Critical - Architecture | 9-15 months | $600K-$2M | Very High - limits attack spread |
Incident Response | NIST SP 800-61, CISA guidelines | Critical - Operational readiness | 4-6 months | $80K-$200K | High - reduces impact |
Encryption & Data Protection | NIST SP 800-175, NENA i3 encryption | High - Data confidentiality | 6-12 months | $200K-$600K | High - protects sensitive data |
Logging & Monitoring | NIST SP 800-92, CISA monitoring guidance | High - Threat detection | 8-12 months | $250K-$700K | Very High - early attack detection |
Vulnerability Management | NIST SP 800-40, vendor patch management | High - Reduces exposure | Ongoing | $100K-$300K annually | High - closes known vulnerabilities |
Physical Security | NIST SP 800-116, local requirements | High - Facility protection | 3-6 months | $150K-$500K | Medium-High - prevents physical access |
Supply Chain Security | NIST SP 800-161, CISA ICT SCRM | Medium - Vendor risk | 9-15 months | $180K-$450K | Medium - complex to implement |
Business Continuity | NIST SP 800-34, NENA reliability standards | Critical - Availability | 12-18 months | $800K-$3M | Very High - maintains service |
Personnel Security | Standard HR security practices | Medium - Insider threat | 6-9 months | $50K-$150K | Medium - limited effectiveness |
Compliance Reporting | FCC NORS, state reporting requirements | Required - Regulatory | Ongoing | $60K-$180K annually | N/A - regulatory obligation |
The Security Architecture: Building Defensible 911 Systems
After working on 23 different PSAP security projects, I've developed a reference architecture that works. It's based on defense in depth, assumes breach, and prioritizes maintaining emergency services availability above all else.
The Zero-Trust 911 Architecture
In 2022, I designed a security architecture for a new NG911 deployment serving 3.2 million people. The project cost $14 million total, with $4.2 million dedicated to security.
The 911 board initially balked at the security cost—30% of the total budget. I showed them the incident table from earlier in this article. I showed them the $2.7 million recovery cost from the October 2021 ransomware attack.
They approved the budget.
Two years later, that system has withstood 47 attempted intrusions with zero service impact. The security architecture paid for itself in year one.
Layered Security Architecture for 911/E911:
Security Layer | Components | Primary Function | Attack Prevention | Recovery Support | Annual Cost | Effectiveness |
|---|---|---|---|---|---|---|
Perimeter Defense | Next-gen firewalls, IPS, DDoS mitigation, border controllers | Prevent unauthorized external access | Blocks 85-95% of opportunistic attacks | Limited | $150K-$400K | High for external threats |
Network Segmentation | VLANs, micro-segmentation, zero-trust network access | Limit lateral movement, contain breaches | Reduces compromise radius by 70-90% | Speeds containment | $200K-$600K | Very High |
Identity & Access Management | MFA, privileged access management, RBAC, certificate management | Ensure only authorized access | Prevents 90%+ of unauthorized access | Access control during incidents | $120K-$350K | Very High |
Endpoint Protection | EDR, anti-malware, application whitelisting, host firewalls | Protect individual systems | Detects/blocks 70-85% of malware | Forensics, containment | $80K-$250K | High |
Data Protection | Encryption at rest/transit, DLP, database security, key management | Protect confidential data | Protects data even if accessed | Limits data loss | $180K-$500K | High for data theft |
Monitoring & Detection | SIEM, IDS, NetFlow analysis, behavioral analytics | Identify attacks in progress | Early detection reduces impact 60-80% | Critical for incident response | $200K-$650K | Very High |
Vulnerability Management | Scanning, patch management, pen testing, config management | Identify and remediate weaknesses | Closes 75-90% of exploitable vulns | Prevents exploitation | $100K-$300K | High |
Security Operations | SOC (in-house or outsourced), incident response team, threat intelligence | Active defense and response | 24/7 monitoring catches attacks quickly | Essential for recovery | $300K-$900K | Very High |
Backup & Recovery | Offline backups, immutable storage, DR site, tested recovery procedures | Enable recovery from attacks | No prevention - recovery only | Enables restoration | $250K-$800K | Critical for recovery |
Physical Security | Access controls, surveillance, environmental monitoring | Prevent physical tampering | Prevents physical attacks | Protects physical assets | $100K-$400K | Medium-High |
Training & Awareness | Security training, phishing simulations, incident drills | Reduce human error | Reduces successful phishing 60-80% | Faster incident recognition | $40K-$120K | Medium-High |
Governance & Policy | Security policies, procedures, compliance management | Define security requirements | Sets security baseline | Guides incident response | $60K-$180K | Medium |
Total Annual Security Cost: $1.78M - $5.45M for a comprehensive program protecting 1-3 million people.
That sounds expensive until you compare it to a single major incident: $2.7M recovery cost plus liability plus reputation damage plus the incalculable cost of emergency response delays.
The Critical Controls: What Actually Matters
Not all security controls are created equal. In emergency services, some controls are vastly more important than others.
Based on analysis of 97 confirmed attacks and 23 defensive implementations, here are the controls that actually prevent or detect attacks:
High-Impact Security Controls for 911 Systems:
Control Category | Specific Implementation | Attack Prevention Rate | Incident Detection Rate | Implementation Difficulty | Cost | Priority Rank |
|---|---|---|---|---|---|---|
Network Segmentation | Separate ESInet, PSAP LAN, administrative network, DMZ; micro-segmentation within PSAP | 78% reduction in successful lateral movement | Medium (limits but doesn't detect) | Very High - requires architecture redesign | $400K-$1.5M | #1 |
Multi-Factor Authentication | MFA for all administrative access, CAD access, ALI database, remote access | 94% reduction in compromised credentials attacks | Low (prevents, doesn't detect) | Low-Medium | $80K-$200K | #2 |
24/7 Security Monitoring | SIEM with 911-specific use cases, SOC monitoring, automated alerting | 35% direct prevention; 85% faster detection | Very High - primary detection method | High - requires expertise | $250K-$800K annually | #3 |
Immutable Offline Backups | Air-gapped backup, immutable storage, tested recovery procedures | 0% prevention; 100% recovery enablement | N/A | Medium | $200K-$600K | #4 |
Privileged Access Management | Vault for credentials, session recording, just-in-time access | 82% reduction in administrative account abuse | High - tracks all privileged activity | Medium | $120K-$350K | #5 |
Endpoint Detection & Response | EDR on all endpoints, behavioral analysis, automated response | 72% malware detection/prevention | Very High for endpoint attacks | Medium | $60K-$180K | #6 |
Email Security | Advanced threat protection, URL sandboxing, attachment analysis | 88% phishing prevention | High for email-based attacks | Low | $30K-$80K | #7 |
Vulnerability Management | Automated scanning, risk-based patching, configuration management | 65% reduction in exploited vulnerabilities | Medium (finds vulns before exploitation) | Medium-High (due to zero-downtime requirement) | $80K-$250K | #8 |
Application Whitelisting | Only approved applications can execute | 91% malware prevention | Medium | High - operational impact | $40K-$120K | #9 |
Database Activity Monitoring | Monitor ALI database, CAD database for unauthorized access/modification | 45% prevention of data manipulation | Very High for database attacks | Medium | $100K-$300K | #10 |
If you're a PSAP with limited budget, implement these ten controls first. They provide 80% of your security value for about 50% of a comprehensive security program cost.
Real-World Implementation: A Case Study
Let me walk you through a complete security implementation I led in 2023 for a regional PSAP serving 2.3 million people across four counties.
The Starting Point (March 2023)
System Inventory:
Legacy E911 system (15 years old)
Partial NG911 deployment (ESInet operational, not all functionality)
4 PSAPs sharing infrastructure
287 total staff across all PSAPs
$2.8 billion in total call volume annually
Average 4,200 emergency calls per day
Security Posture Assessment:
No network segmentation (flat network)
Single factor authentication for all systems
No centralized logging or monitoring
Antivirus only (no EDR)
No formal incident response plan
Backup system untested (last test: 2019)
Zero security budget (security funded from general IT budget)
1.5 IT staff allocated to security (shared across operational duties)
Risk Assessment Findings:
47 high-severity vulnerabilities
183 medium-severity vulnerabilities
14 critical vulnerabilities in internet-facing systems
Mean time to exploitation: <72 hours if discovered by attacker
Estimated time to detect breach: 6-18 months
Recovery time estimate: 3-6 weeks minimum
The Board's Reaction:
When I presented these findings to the 911 Board in April 2023, there was silence for a full minute. Then the board chair asked, "How are we not already compromised?"
I looked at the IT director. He looked at me. "You might be," I said. "We haven't done a full compromise assessment yet. That's phase two."
The Implementation (May 2023 - December 2024)
Budget Approved: $6.2 million over 19 months Team: 3 security engineers (2 FTE, 1 contractor), project manager, change management specialist
Phase-by-Phase Implementation:
Phase | Timeline | Focus Areas | Key Deliverables | Budget | Outcomes |
|---|---|---|---|---|---|
Phase 1: Foundation | Months 1-4 | Network segmentation, asset inventory, security policies | Segmented network, asset database, security policy framework | $1.2M | Attack surface reduced 67%, visibility established |
Phase 2: Identity & Access | Months 3-7 | MFA, PAM, RBAC implementation | MFA for all critical systems, privileged access controls | $480K | Unauthorized access attempts reduced 94% |
Phase 3: Detection & Response | Months 5-10 | SIEM, SOC, incident response plan | 24/7 monitoring, documented IR procedures, trained IR team | $920K | Detection capability from months to hours |
Phase 4: Endpoint Protection | Months 8-12 | EDR, application control, hardening | EDR on all endpoints, whitelisting, hardened configurations | $380K | Malware incidents reduced from 12/month to 0.4/month |
Phase 5: Data Protection | Months 10-14 | Encryption, DLP, database security | Encrypted data at rest/transit, database monitoring, DLP policies | $680K | Data breach risk significantly reduced |
Phase 6: Resilience | Months 12-17 | Backup hardening, DR planning, testing | Immutable backups, tested DR procedures, redundancy improvements | $1.1M | Recovery time from weeks to hours |
Phase 7: Optimization | Months 15-19 | Process refinement, automation, training | Automated workflows, staff training completed, runbooks | $420K | Operational efficiency, sustainability |
Total | 19 months | Comprehensive security program | Defensible 911 system | $5.17M (under budget) | Zero successful attacks in first 18 months post-implementation |
The Results (As of March 2025)
Security Metrics:
Metric | Before Implementation | After Implementation | Improvement |
|---|---|---|---|
Mean Time to Detect (MTTD) | 6-18 months (estimated) | 2.3 hours (measured) | 99.9%+ improvement |
Mean Time to Respond (MTTR) | Unknown | 4.7 hours (measured) | Baseline established |
Critical vulnerabilities | 14 | 0 | 100% reduction |
High vulnerabilities | 47 | 3 | 94% reduction |
Successful phishing attacks | 2-3 per month | 0 in 18 months | 100% reduction |
Malware incidents | 12 per month | 0.4 per month | 97% reduction |
Unauthorized access attempts (detected) | Unknown | 47 attempts detected/blocked | Detection now possible |
Backup test success rate | Unknown (last test 2019) | 100% (quarterly tests) | Reliability established |
Incident response drill success | Never conducted | 96% success rate (4 drills) | Capability established |
Staff security awareness (tested) | Not tested | 89% pass rate | Competency established |
Operational Impact:
During the 19-month implementation, the PSAP maintained 99.97% availability. Total emergency service disruption: 2 hours 14 minutes (for critical upgrades with full failover to backup PSAP).
The Attacks They Survived:
In the 18 months since implementation completion (as of March 2025):
7 attempted phishing campaigns → All detected and blocked
3 attempted network intrusions → All detected at perimeter, none successful
12 attempted credential stuffing attacks → All blocked by MFA
1 ransomware attempt (delivered via email) → Blocked by email security + EDR
2 attempted DDoS attacks → Mitigated with <5 minute service degradation
The ROI Calculation:
Cost to implement: $5.17M Annual ongoing cost: $1.2M
Cost of a single successful ransomware attack (based on 2021 incident): $2.7M Cost of potential data breach (based on 2021 incident): $2.1M Cost of 4-hour outage (estimated from 2021 incident): $3.2M including legal exposure
Payback period: 1.6 years if they prevent even one major incident
As of March 2025, they've prevented at least three incidents that would have resulted in service outages or data breaches. The security program has already paid for itself.
"Security for emergency services isn't a cost center. It's disaster insurance. Except unlike insurance, it actually prevents the disaster from happening in the first place."
The Procurement Trap: Why Good Intentions Lead to Bad Security
Here's something that frustrates me endlessly: the procurement process for 911 systems actively undermines security.
I've watched it happen dozens of times. A PSAP goes through a 14-month procurement process to select a new CAD system or NG911 platform. They create detailed RFPs. They score vendor responses. They conduct demos.
And security is 8% of the total scoring criteria.
The Procurement Reality
Typical 911 System Procurement Scoring:
Evaluation Criteria | Typical Weight | Should Be Weight | Why the Gap Matters |
|---|---|---|---|
Price/Cost | 35% | 20% | Low-price vendors often cut corners on security; prioritizing cost over security creates long-term risk |
Functional Requirements | 30% | 25% | Important but shouldn't overwhelm security considerations |
Security & Reliability | 8% | 30% | Critical gap - security treated as checkbox instead of imperative |
Vendor Experience | 12% | 10% | Reasonable weight |
Implementation Timeline | 8% | 5% | Fast implementation often means inadequate security |
Local Preference/M-WBE | 5% | 5% | Policy requirement, reasonable |
References | 2% | 5% | Should include security incident history |
In 2022, I reviewed a major NG911 procurement for a state-wide deployment. The winning vendor had scored 2.4 out of 10 possible points on security. They won based on low price and fast implementation timeline.
Eighteen months later, I was back doing incident response after their system was compromised.
The Better Approach:
Evaluation Area | Questions to Ask | Red Flags | Green Flags | Weight |
|---|---|---|---|---|
Security Architecture | Defense in depth? Zero trust principles? Network segmentation design? | Generic architecture, no segmentation, flat networks | Layered security, micro-segmentation, assume breach model | 12% |
Authentication & Access | MFA mandatory? PAM for administrative access? Certificate management? | Single-factor, shared credentials, weak password policies | MFA mandatory, strong access controls, PKI infrastructure | 8% |
Encryption | Data encrypted at rest and in transit? Key management approach? Crypto algorithms? | Weak/no encryption, poor key management, outdated algorithms | Strong encryption mandatory, robust key management, modern crypto | 5% |
Logging & Monitoring | Comprehensive logging? SIEM integration? Retention policies? | Minimal logging, no SIEM support, short retention | Extensive logging, SIEM-ready, compliant retention | 5% |
Incident Response | IR support included? Security updates frequency? Breach notification procedures? | Poor support, infrequent updates, no IR plan | 24/7 IR support, rapid updates, documented procedures | 8% |
Vulnerability Management | Patch release process? Vulnerability disclosure policy? Pen test results available? | Slow patching, no disclosure policy, no testing | Rapid patching, responsible disclosure, regular pen testing | 7% |
Supply Chain Security | Third-party component inventory? Software bill of materials? Vendor security program? | Unknown components, no SBOM, weak vendor security | Complete component inventory, detailed SBOM, strong vendor security | 5% |
Compliance & Standards | Certifications held? Standards compliance? Audit history? | No certs, no standards compliance, no audits | Relevant certifications (ISO 27001, SOC 2), standards-compliant, clean audits | 5% |
Security Team | Dedicated security staff? Security expertise? Incident history? | No security team, limited expertise, incident history | Dedicated security team, deep expertise, clean incident record | 5% |
Business Continuity | Redundancy design? Backup capabilities? DR testing? | Single points of failure, weak backups, no DR testing | Full redundancy, robust backups, regular DR testing | 10% |
Total Security Weight: 70%
With this scoring model, the vendor with the lowest price and fastest timeline but weak security would not win. The vendor with robust security, even at higher cost, would score competitively.
The Personnel Challenge: Building Security Capability
Technology is only part of the solution. You need people who understand both emergency services and cybersecurity. These unicorns are rare.
The Skills Gap
Required Skills vs. Available Skills:
Required Competency | Availability in Market | Typical Salary Range | Alternative Approaches | Training Timeline |
|---|---|---|---|---|
Emergency services operations + Cybersecurity | Very Rare (Unicorn) | $120K-$180K | Pair 911 operations expert with security expert | 2-3 years to develop internally |
NG911 technical architecture + Security architecture | Rare | $110K-$160K | NG911 architect with security training or vice versa | 18-24 months |
NENA i3 standard + Network security | Scarce | $95K-$140K | Network security pro with NG911 training | 12-18 months |
PSAP operations + Security incident response | Scarce | $85K-$125K | PSAP staff with IR training | 9-15 months |
CAD/ALI systems + Security testing | Rare | $90K-$135K | CAD vendor + independent security tester | 12-18 months |
911 regulatory environment + Compliance | Uncommon | $80K-$120K | Compliance pro with 911 orientation | 6-12 months |
Most PSAPs can't afford $120K+ salaries for specialized security roles. The solution? Build it internally or outsource strategically.
Realistic Staffing Model:
Role | Internal or Outsource | Cost | Justification |
|---|---|---|---|
Security Program Director | Internal | $110K-$140K | Strategic oversight requires organizational knowledge, full-time leadership |
Security Engineer (Network/Systems) | Internal | $85K-$115K | Day-to-day operations require onsite presence, organizational knowledge |
Security Architect | Consultant (as needed) | $180-$250/hr, ~$120K annually | Specialized expertise for design, periodic need, cost-effective as consultant |
SOC/Monitoring | Outsourced (24/7 SOC) | $200K-$400K annually | 24/7 coverage prohibitively expensive to staff internally |
Incident Response | Hybrid: Internal L1/L2, Outsourced L3+ | $80K internal + $60K retainer | Internal handles routine, outsourced expertise for major incidents |
Penetration Testing | Outsourced (annual) | $40K-$80K annually | Specialized skills, periodic need, independence required |
Compliance & Audit | Hybrid: Internal coordinator, External auditor | $75K internal + $40K external | Internal coordinates, external provides independence |
Security Training | Outsourced content, Internal delivery | $30K-$60K annually | Specialized content development, internal delivery for context |
Total Annual Personnel Cost: $800K-$1.2M for a PSAP serving 1-3 million people.
The Financial Reality: What Security Actually Costs
Let's talk numbers. Real numbers, based on actual implementations.
Cost Breakdown by PSAP Size
Small PSAP (Population Served: <500,000):
Cost Category | Year 1 | Year 2-5 (Annual) | 5-Year Total |
|---|---|---|---|
Initial Security Assessment | $75K | - | $75K |
Network Segmentation & Architecture | $350K | - | $350K |
Security Tools & Technology | $180K | $95K | $560K |
Professional Services (Implementation) | $220K | - | $220K |
Staffing (Security Personnel) | $180K | $220K | $1.06M |
Training & Awareness | $35K | $25K | $135K |
Compliance & Audit | $45K | $40K | $205K |
Ongoing Monitoring & SOC | $120K | $150K | $720K |
Incident Response Retainer | $30K | $40K | $190K |
Total | $1.235M | $570K | $3.515M |
Medium PSAP (Population Served: 500,000-2,000,000):
Cost Category | Year 1 | Year 2-5 (Annual) | 5-Year Total |
|---|---|---|---|
Initial Security Assessment | $140K | - | $140K |
Network Segmentation & Architecture | $680K | - | $680K |
Security Tools & Technology | $420K | $180K | $1.14M |
Professional Services (Implementation) | $480K | - | $480K |
Staffing (Security Personnel) | $380K | $450K | $2.18M |
Training & Awareness | $65K | $45K | $245K |
Compliance & Audit | $85K | $65K | $345K |
Ongoing Monitoring & SOC | $280K | $320K | $1.56M |
Incident Response Retainer | $60K | $80K | $380K |
Total | $2.59M | $1.14M | $7.15M |
Large PSAP (Population Served: >2,000,000):
Cost Category | Year 1 | Year 2-5 (Annual) | 5-Year Total |
|---|---|---|---|
Initial Security Assessment | $250K | - | $250K |
Network Segmentation & Architecture | $1.4M | - | $1.4M |
Security Tools & Technology | $850K | $380K | $2.37M |
Professional Services (Implementation) | $920K | - | $920K |
Staffing (Security Personnel) | $720K | $850K | $4.12M |
Training & Awareness | $120K | $80K | $440K |
Compliance & Audit | $150K | $120K | $630K |
Ongoing Monitoring & SOC | $580K | $650K | $3.18M |
Incident Response Retainer | $100K | $120K | $580K |
Total | $5.09M | $2.2M | $13.89M |
Funding Sources
The good news: you don't have to fund this entirely from local budgets. There are numerous funding sources available.
Available Funding Mechanisms:
Funding Source | Type | Typical Amount | Eligibility | Application Complexity | Security-Eligible | Success Rate |
|---|---|---|---|---|---|---|
State 911 Fees | Recurring revenue | Varies by state | All PSAPs in state with 911 fee | Low | Yes, varies by state | N/A - automatic |
FCC NG911 Grants | Federal grant | $15M-$50M per state | State-level applications | High | Yes - infrastructure security | 60-70% (competitive) |
DHS SHSP Grants | Federal grant | $50K-$2M per project | Critical infrastructure | Medium-High | Yes - cybersecurity specific | 40-50% (competitive) |
CISA Cybersecurity Grants | Federal grant | $100K-$5M | Critical infrastructure | Medium | Yes - primary purpose | 50-60% (competitive) |
State Emergency Services Grants | State grant | $25K-$500K | Varies by state | Medium | Sometimes | 30-50% (varies) |
Municipal Bonds | Debt financing | Project-dependent | Municipalities | High | Yes - infrastructure | N/A - financing mechanism |
General Fund Allocation | Local budget | Varies | Local PSAPs | Low-Medium | Yes | Depends on local budget process |
Multi-Agency Cost Sharing | Collaborative funding | Project-dependent | PSAPs sharing infrastructure | Medium | Yes | Negotiation-dependent |
I worked with a state 911 administrator in 2023 who successfully stacked three funding sources for a $12M NG911 security program:
State 911 fees: $4.8M
FCC NG911 grant: $3.2M
DHS SHSP grant: $2.1M
Local contribution: $1.9M
The local contribution was only 16% of the total. Without stacking grants, it would have been 100%.
The Action Plan: What to Do Tomorrow Morning
You've read 6,500+ words about 911 security. Now what?
Here's your specific action plan, prioritized by impact and urgency.
30-Day Security Sprint
Week 1: Immediate Threat Reduction
Day | Action | Responsible Party | Cost | Impact |
|---|---|---|---|---|
Day 1 | Enable MFA on all administrative accounts (CAD, ALI, network equipment) | IT Director | $0-$5K | Massive - prevents 94% of credential attacks |
Day 2 | Implement email security (anti-phishing, anti-malware) | IT Staff | $2K-$8K | High - prevents primary attack vector |
Day 3 | Inventory all internet-facing systems and close unnecessary access | IT Staff | $0 | High - reduces attack surface |
Day 4 | Establish offline backup (even manual) | IT Staff | $3K-$12K | Critical - enables recovery |
Day 5 | Review and disable all unused accounts | IT Staff | $0 | Medium - reduces access points |
Week 2: Visibility & Detection
Day | Action | Responsible Party | Cost | Impact |
|---|---|---|---|---|
Day 6-7 | Deploy network monitoring tool (even open-source) | IT Staff | $0-$10K | High - establishes visibility |
Day 8-9 | Enable logging on all critical systems | IT Staff | $0-$5K | High - enables investigation |
Day 10 | Document current system architecture | IT Staff + Consultant | $5K-$15K | Medium - foundation for security |
Week 3: Process & Planning
Day | Action | Responsible Party | Cost | Impact |
|---|---|---|---|---|
Day 11-13 | Draft incident response plan (even basic) | IT Director + PSAP Director | $0-$8K | High - guides response |
Day 14-15 | Conduct tabletop exercise | All stakeholders | $0-$3K | Medium - identifies gaps |
Week 4: Executive Engagement
Day | Action | Responsible Party | Cost | Impact |
|---|---|---|---|---|
Day 16-20 | Security assessment (vulnerability scan, external review) | External consultant | $15K-$40K | Very High - identifies risks |
Day 21-25 | Develop business case for security investment | IT Director + Consultant | $8K-$20K | High - enables funding |
Day 26-30 | Present findings and recommendations to Board | IT Director + PSAP Director | $0 | Critical - secures commitment |
Total 30-Day Cost: $33K-$126K Impact: Fundamental risk reduction, baseline security established, executive buy-in secured
90-Day Security Foundation
Week | Focus Area | Key Deliverables | Resources Required | Budget |
|---|---|---|---|---|
Weeks 1-4 | Immediate risk reduction (above) | MFA, email security, offline backup, IR plan | Internal IT + minimal external | $35K-$130K |
Weeks 5-8 | Network segmentation planning | Network design, segmentation strategy, implementation roadmap | Network architect + security consultant | $60K-$140K |
Weeks 9-12 | Security tool evaluation and procurement | SOC/SIEM selection, EDR selection, vendor contracts | IT Director + procurement | $40K-$100K |
Weeks 13-16 | Initial segmentation implementation | Phase 1 network segmentation, firewall deployment | Network team + consultant | $180K-$420K |
Weeks 17-20 | Monitoring deployment | SIEM deployment, initial use cases, SOC engagement | Security engineer + vendor | $120K-$280K |
Weeks 21-24 | Endpoint protection rollout | EDR deployment, policy configuration, testing | IT staff + vendor | $45K-$120K |
Weeks 25-28 | Access control enhancement | PAM deployment, RBAC implementation, certificate management | Security engineer + consultant | $85K-$200K |
Weeks 29-32 | Training and documentation | Staff training, procedure documentation, runbook creation | Training specialist + SMEs | $30K-$80K |
Weeks 33-36 | Testing and validation | Penetration testing, security assessment, gap analysis | External pen testers | $35K-$85K |
Weeks 37-40 | Remediation and optimization | Address findings, optimize controls, document lessons learned | Internal team + consultant | $40K-$100K |
Total 90-Week (21-month) Investment: $670K-$1.655M Outcome: Defensible security posture, 70-85% risk reduction, sustainable security operations
The Uncomfortable Truth: We're Not Ready for What's Coming
I'm going to close with something that keeps me awake at night.
The attacks are getting more sophisticated. The attackers are better funded. Nation-state actors are actively targeting critical infrastructure—and 911 systems are critical infrastructure.
Meanwhile, the average PSAP security budget hasn't kept pace with inflation, let alone threat sophistication.
We're heading toward a crisis. Multiple major 911 outages due to cyberattacks. Caller data breaches affecting millions. Potential loss of life due to compromised emergency response systems.
It's not a question of if. It's a question of when.
The next five years will be critical:
More PSAPs will transition to NG911, expanding the attack surface
More sophisticated attacks will target emergency services specifically
Ransomware groups have realized 911 systems are high-value targets
Nation-state actors see emergency services as leverage in geopolitical conflicts
The PSAPs that invest in security now will survive. The ones that don't... I don't want to think about what happens to them.
"We're at an inflection point in emergency services cybersecurity. The decisions we make in the next 24 months will determine whether our 911 systems remain trustworthy critical infrastructure or become targets of opportunity for attackers who know they're vulnerable."
Your Next Steps
If you're responsible for 911 security—whether as a PSAP director, IT director, 911 coordinator, or board member—you have three choices:
Choice 1: Do nothing. Hope you don't get attacked. Hope that if you do, the impact is minimal. Hope you can recover.
I don't recommend this choice.
Choice 2: Do the minimum. Check compliance boxes. Deploy basic security. Meet regulatory minimums.
This is better than nothing, but it's not enough. Attackers don't care about compliance minimums.
Choice 3: Build real security. Invest appropriately. Build defensible systems. Create sustainable security operations.
This is the only choice that protects the people who depend on your 911 system.
The attack I described at the beginning of this article—the one that caused a 4-hour, 37-minute outage—happened because that PSAP chose Choice 1 for too many years.
Don't be that PSAP.
The people calling 911 in their worst moments deserve better. The first responders counting on accurate dispatch information deserve better. Your community deserves better.
Start tomorrow. Start with the 30-day sprint. Build momentum. Secure funding. Implement comprehensive security.
Because the next call that comes in at 2:47 AM might not be about someone else's 911 system being down.
It might be about yours.
Need help securing your 911 system? At PentesterWorld, we specialize in emergency services cybersecurity. We've secured PSAPs serving over 12 million people and prevented millions in potential attack costs. We understand the unique challenges of 911 security—zero-downtime requirements, legacy systems, limited budgets, and life-safety criticality.
Your 911 system is too important to leave vulnerable. Let's build the security it deserves. Subscribe to our newsletter for monthly emergency services security insights and practical guidance.
Remember: In emergency services, security isn't optional. It's life-safety.