Maria Delgado had reviewed enough Reports on Compliance to know when a control was real and when it was theater. As the lead QSA on the assessment of TerraPay Solutions — a mid-market payment processor handling roughly 40 million transactions a year for regional grocery chains and fuel retailers — she'd flagged something odd in the walkthrough of Requirement 10. The log review team could describe their process fluently. They just couldn't produce a single ticket showing they'd actually reviewed an alert in the last quarter.
"We do it," the security manager told her. "We just don't always write it down."
That sentence cost TerraPay eleven weeks and roughly $310,000 in remediation, contractor hours, and a delayed go-live with a new acquiring bank. Not because the control was fundamentally broken — the SIEM was tuned reasonably well and alerts were, in fact, mostly getting looked at — but because PCI DSS testing procedures don't accept "we do it" as evidence. They require something an assessor can examine, observe, or verify through interview and technical test. Without that, a control that exists in practice doesn't exist on paper, and on paper is the only place a Report on Compliance lives.
This is the part of PCI DSS most compliance programs get wrong: they build controls to satisfy the requirement and forget to build evidence to satisfy the testing procedure. Those are two different jobs. This article is about the second one.
Who this is for and what you'll walk away with
This is written for compliance managers, security engineers, and internal audit teams preparing for a QSA or ISA-led PCI DSS assessment, and for anyone who has read a Report on Compliance template and wondered what "examine," "observe," and "interview" actually mean in practice. You'll walk away understanding the four validation methods assessors use, how testing procedures differ across requirement families, what sampling really means for systems and facilities, how the five reporting statuses get assigned, and — critically — how testing changes when you adopt the customized approach instead of the defined approach. By the end, you should be able to look at any PCI DSS requirement and know, specifically, what evidence you'd need to hand an assessor.
Why "Testing Procedures" Are the Real PCI DSS
Every organization that reads the PCI DSS standard reads the requirements. Far fewer read the testing procedures that sit underneath each one — and that's a mistake, because the testing procedures are where the standard tells you, in granular language, exactly what "in place" means. Each numbered sub-requirement in PCI DSS v4.0.1 carries one or more testing procedures, each assigned a validation method: examine, observe, interview, or test (technical testing). The requirement tells you what to build. The testing procedure tells you what you'll be asked to prove — and, by extension, what evidence to start collecting today rather than the week before your assessment.
This distinction matters because PCI DSS was rewritten in v4.0 specifically to make testing procedures more rigorous and more explicit than they were under v3.2.1. The Council's stated goal was to reduce subjectivity — to make it less likely that two QSAs would reach different conclusions about the same control. That effort shows up directly in the testing procedure language: verbs like "examine," "observe," and "interview" are used precisely, and each testing procedure specifies not just that evidence must exist, but often the exact document, log, or configuration screen the assessor expects to see.
If you're building or refreshing a PCI DSS compliance program and haven't yet read the PCI DSS Complete Guide covering the full 12-requirement structure, that's a useful primer before this article, since everything here assumes familiarity with the six goals and twelve requirements.
The Four Testing Methods, Defined
PCI DSS testing procedures rely on four validation methods, sometimes combined within a single testing procedure. None of them is inherently "stronger" than another — a technical test isn't automatically better evidence than a well-documented interview — but each answers a different question about a control.
Method | What it answers | Typical evidence produced | Example testing procedure language |
|---|---|---|---|
Examine | Does documentation, configuration, or a record confirm the control exists and operated? | Policies, standards, configuration exports, logs, tickets, diagrams, contracts | "Examine firewall and router configuration standards to verify..." |
Observe | Does the control actually happen, live, the way people say it does? | Assessor's direct observation notes, screen-shares, walkthroughs, physical site tours | "Observe processes and interview personnel to verify..." |
Interview | Do the people who own or execute the control understand and follow it? | Interview notes, corroborating statements across roles | "Interview responsible personnel to verify that..." |
Test (technical testing) | Does the control actually work when exercised — via inspection, sampling, or hands-on verification? | Scan results, penetration test reports, sampled configuration checks, re-performance results | "Test detection mechanisms by generating a script..." |
A few things to notice here. First, "test" in PCI DSS language is narrower than the everyday meaning — it specifically refers to technical or hands-on verification, not the general activity of assessing. Second, most testing procedures combine at least two methods; a QSA rarely relies on documentation alone (examine) without also confirming through interview or observation that the documented process is the one actually followed. Third, the standard is explicit that assessors must use "sufficient" methods to reach a conclusion — meaning if examine and interview together don't resolve ambiguity, the assessor is expected to escalate to observation or technical testing.
"The single biggest misconception I run into is that a policy document is evidence. A policy is evidence that a policy exists. Whether the control operates is a completely separate question, and it's the one my testing procedures are actually built to answer." — Marcus Whitfield, QSA, Ironclad Assurance Group
Examine: Documentary Evidence Deep Dive
Examine-based testing procedures are the backbone of most PCI DSS assessments simply because so much of the standard concerns configuration state and documented process. When a testing procedure says "examine," the assessor is looking at something static: a firewall rule set, a data retention policy, a vendor contract, a change ticket, a training completion log, an inventory of system components.
The trap organizations fall into is treating "examine" evidence as sufficient by itself, particularly for requirements where the standard actually calls for examine and interview, or examine and test. A well-written information security policy that nobody follows still satisfies the "examine the policy" half of a testing procedure — it just fails the interview or observe half that checks whether the policy reflects reality. Evidence quality for examine-based procedures comes down to three things: currency (is this the version in effect during the assessment period, not a stale draft), specificity (does it name the actual systems, roles, and thresholds rather than speaking in generalities), and traceability (can the assessor tie the document to an approval date, an owner, and a review cadence).
Common examine artifacts assessors request repeatedly across requirement families include network diagrams and data flow diagrams, the cardholder data environment inventory, firewall and router configuration standards, system hardening standards, encryption and key management policies, vendor and service provider agreements with the required PCI DSS responsibility language, incident response plans, vulnerability scan reports (both internal and ASV), penetration test reports, access control matrices, and the annual policy review and awareness records for Requirement 12.
Observe: Watching Controls in Action
Observation-based testing procedures require the assessor to watch something happen rather than read about it. This is where PCI DSS testing gets uncomfortable for organizations that have strong documentation but weaker execution, because you cannot fabricate an observation after the fact the way you might, in theory, backfill a document.
Observation shows up heavily in physical security testing under Requirement 9 — the assessor walking the data center floor, watching badge-in procedures at a restricted door, checking whether a visitor log is actually completed at the point of entry rather than reconstructed later. It also shows up in process-heavy requirements: watching a system administrator provision a new user account to confirm the approval workflow is followed step by step, watching a help desk analyst handle a password reset request to confirm identity verification actually happens before the reset, or watching how removable media is handled when it leaves a facility.
Observation is powerful evidence precisely because it's hard to game convincingly across a whole sample. A single staged demonstration might pass, but PCI DSS testing procedures typically pair observation with sampling across multiple instances, multiple personnel, and sometimes multiple facilities — which means the theater has to hold up consistently, and inconsistency is exactly what experienced assessors are trained to notice.
"I've watched a badge reader demo go perfectly and then watched the same door get propped open with a fire extinguisher twenty minutes later during my facility walk-through. Observation isn't a scheduled event people can prepare a scene for — or shouldn't be, if the sample selection is done right." — Priya Ramanathan, Internal Security Assessor (ISA), Coastal Retail Holdings
Interview: Talking to the People Who Run Controls
Interview-based testing procedures exist because documentation and configuration state can't tell you whether the people operating a control understand its purpose, its exceptions, or what to do when something breaks. An interview testing procedure typically requires the assessor to speak with "personnel responsible for" a given control — language you'll see repeated throughout the standard — and the goal is triangulation: does what this person says match what the documentation says, and does it match what a second person in an adjacent role says?
Interviews are also where inconsistency surfaces fastest. If the security policy says access reviews happen quarterly, the access management lead says they happen "roughly every few months, when we get to it," and the sampled evidence shows two reviews in the last eighteen months, the assessor now has three data points in conflict, and the testing procedure isn't satisfied by any one of them alone.
Assessors generally interview across three tiers: control owners (the person accountable for a policy or process, often a manager or director), control operators (the person who actually executes day-to-day tasks — the engineer who reviews the SIEM alerts, the badge administrator, the developer who runs the code review checklist), and, for select requirements, end users or general staff to confirm security awareness training actually landed (Requirement 12's phishing-awareness and incident-reporting testing procedures often pull a small sample of general staff, not just security personnel).
Test (Technical Testing): Hands-On Verification
Technical testing is the validation method organizations most associate with "real" security assessment, and for good reason — it's the method that produces the least ambiguous evidence. Technical testing procedures direct the assessor to inspect actual configuration states, sample actual system components, generate actual test traffic, or review actual scan and penetration test output.
Examples across the standard: for Requirement 1, technically testing a sample of firewall rule bases to confirm that only documented, business-justified services and ports are permitted, and that any risky protocol has a documented business reason and compensating control. For Requirement 3, technically inspecting how the primary account number is rendered in stored files, databases, and logs to confirm it's unreadable — truncated, tokenized, hashed, or encrypted — everywhere it's found. For Requirement 6, technically reviewing code review records and the output of automated or manual application security testing prior to release. For Requirement 10, technically testing that a sampled critical file, when altered, actually generates an alert reviewed by personnel (a live demonstration, not a description). For Requirement 11, reviewing quarterly ASV scan results and confirming rescans clear all high-severity vulnerabilities, and reviewing the annual (or more frequent, for Requirement 6 target systems) penetration test methodology and results.
Technical testing is also the validation method most closely tied to sampling — an assessor rarely tests every firewall or every server; they test a representative, appropriately sized sample and extrapolate a conclusion about the whole population, a practice examined in detail later in this article.
How Testing Procedures Map to Requirement Numbering
PCI DSS v4.0.1 numbers testing procedures to mirror the sub-requirement they validate — sub-requirement 1.2.5 has testing procedure 1.2.5, sub-requirement 3.5.1 has testing procedure 3.5.1, and so on, often broken further into .a, .b, .c sub-parts where a single control needs multiple validation methods to be fully tested. A sub-requirement might read as a single sentence describing the control, while its testing procedure breaks into 1.2.5.a (examine documented standards), 1.2.5.b (examine configurations), and 1.2.5.c (interview personnel) — three distinct evidence asks stacked under one control statement.
This granular mapping is exactly why the sub-requirement structure matters operationally, not just academically. If you've mapped your evidence library at the requirement level ("we have firewall evidence") rather than the sub-requirement level ("we have evidence for 1.2.5.a, 1.2.5.b, and 1.2.5.c specifically"), you'll discover gaps during the assessment rather than before it. Our companion piece on the sub-requirements breakdown goes deeper into how the twelve top-level requirements decompose into these testable units, and it pairs naturally with this article — that piece maps the control structure; this one maps what gets tested against it.
Evidence Expectations by Requirement Family
The table below summarizes the dominant validation methods and characteristic evidence for each of the twelve requirements. No requirement uses only one method — this shows the emphasis.
Requirement | Dominant methods | Characteristic evidence assessors request |
|---|---|---|
Req 1 — Network security controls | Examine, Test | Network/data-flow diagrams, firewall rule-base exports, change tickets, rule review records |
Req 2 — Secure configurations | Examine, Test | Hardening standards, configuration baselines (CIS Benchmarks or vendor equivalents), default-account inventories |
Req 3 — Protect stored account data | Examine, Test | Data retention/disposal policy, PAN storage inspection, key management procedures, encryption/tokenization proof |
Req 4 — Protect data in transit | Examine, Test | TLS configuration exports, cipher suite inventories, certificate management records |
Req 5 — Malware protection | Examine, Observe, Test | Anti-malware deployment records, scan logs, update/definition timestamps |
Req 6 — Secure systems and software | Examine, Interview, Test | SDLC policy, code review records, application security testing output, change management tickets |
Req 7 — Access control (need-to-know) | Examine, Interview | Access control matrix, role definitions, access review records, approval workflow tickets |
Req 8 — Identification and authentication | Examine, Test | Password/MFA policy, account provisioning/deprovisioning tickets, authentication configuration exports |
Req 9 — Physical access | Observe, Examine, Interview | Visitor logs, badge system reports, media destruction certificates, facility walk-through notes |
Req 10 — Logging and monitoring | Examine, Test, Interview | Log configuration, SIEM alert samples, daily review records, file integrity monitoring output |
Req 11 — Security testing | Test, Examine | ASV scan reports, internal vuln scan reports, penetration test reports and scope, remediation tickets |
Req 12 — Policy and program | Examine, Interview | Security policy, risk assessment, incident response plan, awareness training records, third-party agreements |
Evidence expectations don't stop at "have the document." Assessors consistently probe for three additional qualities across every requirement family: that the artifact is dated within the current assessment period (or shows an unbroken chain of prior periods for controls with longer review cycles), that it names the specific in-scope systems rather than referencing a generic enterprise-wide policy with no CDE-specific application, and that it shows evidence of actual review or approval by an accountable owner, not just a system-generated export.
Sampling of System Components
Very few organizations run one firewall, one server, or one point-of-sale terminal — most run dozens, hundreds, or thousands. PCI DSS testing procedures explicitly permit sampling rather than requiring the assessor to examine every single system component, but the sampling methodology itself is subject to scrutiny and documentation requirements.
Sampling consideration | What it means in practice |
|---|---|
Sample must be representative | Sample selection has to reasonably represent all types, locations, and configurations of the system component category, not just the easiest ones to reach |
Different system types are separate populations | Web servers, database servers, POS terminals, and firewalls are each sampled independently — a good sample of servers doesn't cover firewalls |
Standardized vs non-standardized processes | Where processes/configurations are centrally standardized (e.g., a single golden image deployed everywhere), smaller samples are defensible; where configurations vary site-to-site, larger samples are required |
Sample size documentation | The assessor documents the rationale for the sample size and composition in the Report on Compliance — this isn't discretionary or informal |
Independent samples across sub-requirements | A component sampled for one testing procedure doesn't automatically satisfy sampling for a different testing procedure — samples can be, and often are, drawn independently per control |
Sampling doesn't reduce scope | Sampling affects how many components are tested; it does not reduce which components are in scope for the requirement itself |
The practical implication for compliance teams: your configuration management and asset inventory need to be good enough that an assessor can actually stratify your environment into meaningful populations (all POS terminals, all database servers, all jump boxes) before a sample can even be drawn. Organizations with messy or incomplete inventories often end up with the assessor treating the entire population as non-standardized by default — which almost always means larger, more expensive samples.
Sampling of Business Facilities and Locations
Sampling isn't limited to technical system components — Requirement 9's physical security controls and Requirement 12's policy and program controls are frequently assessed across multiple business facilities, and the same representative-sampling logic applies to locations as it does to servers.
Facility sampling factor | Assessor consideration |
|---|---|
Facility type diversity | Corporate HQ, data centers, retail branches, and remote/home-based processing locations are typically treated as distinct populations |
Geographic and regulatory variance | Facilities in different jurisdictions or with different local management may need separate sampling even within the same facility type |
Consistency of physical controls | Centrally managed physical security (single badge system, single vendor for locks/cameras) supports smaller facility samples; locally managed, inconsistent controls push samples larger |
Number of retail/branch locations | For merchants with many storefronts, assessors sample a subset of locations but document why the sample is believed representative of the whole chain |
New or recently changed facilities | Facilities that opened, moved, or changed CDE footprint during the assessment period are more likely to be pulled into the sample regardless of statistical selection, since they represent the least-tested control state |
A retail merchant with 300 branch locations should not expect an assessor to visit all 300 — but should expect the sample selection logic and its rationale to appear in assessor workpapers, and should expect that sample to be redrawn, not reused verbatim, each assessment cycle.
Defined Approach vs Customized Approach: How Testing Differs
PCI DSS v4.0 introduced the customized approach as an alternative to the traditional defined approach, and this is the single biggest shift in how testing procedures actually function in current practice. Under the defined approach, the standard's testing procedures apply exactly as written — the assessor examines, observes, interviews, and tests against the specific control described. Under the customized approach, the organization designs its own control to meet the stated Customized Approach Objective for a requirement, and the assessor must derive and validate testing procedures specific to that custom control, because the standard's pre-written testing procedures don't apply to a control the standard didn't anticipate.
Dimension | Defined approach | Customized approach |
|---|---|---|
Testing procedure source | Pre-written in the PCI DSS standard, applied as-is | Derived by the assessor specifically for the custom control |
Assessor's role | Apply documented procedures consistently | Design, document, and justify a bespoke testing methodology |
Required supporting analysis | None beyond standard evidence | Targeted risk analysis (TRA) documenting the control's ability to meet the objective |
Assessor sign-off burden | Lower — procedures are standardized across all QSAs | Higher — assessor must justify their derived procedures and conclusions in the ROC |
Typical use case | Most organizations, most requirements | Mature security programs with non-standard but objective-equivalent controls |
Report on Compliance documentation | Standard reporting template sections | Additional narrative describing the custom control, the TRA, and the assessor-derived testing |
The customized approach is not a shortcut, and organizations sometimes discover this the hard way. Because the assessor must build testing procedures from scratch — rather than reaching for a template the Council already wrote — the level of assessor effort, and by extension the assessment cost and timeline, often increases for any requirement taken through the customized route, even though the underlying control might be more sophisticated than the defined-approach equivalent.
The Targeted Risk Analysis and Its Role in Customized Testing
A targeted risk analysis is the documentation backbone that makes customized-approach testing possible. Without it, an assessor has nothing to test against, because there's no defined success criteria beyond the (deliberately abstract) Customized Approach Objective written into the standard. A defensible targeted risk analysis typically documents the assets and processes at risk, the threat scenario the custom control addresses, how the custom control meets the requirement's stated objective, the frequency and method by which the organization itself monitors the control's ongoing effectiveness, and management's review and approval of the analysis.
The assessor's derived testing procedures for a customized control almost always include a test of the targeted risk analysis itself — is it current, is it approved by appropriate management, does its logic actually hold up — layered on top of testing the custom control's operation. This means customized-approach requirements effectively double the evidence burden: evidence that the analysis is sound, plus evidence that the control the analysis justifies is operating. Organizations considering the customized approach for even a handful of requirements should expect a meaningfully more document-intensive assessment relative to sticking with the defined approach.
"Customized approach isn't 'do whatever you want and explain it later.' I've had clients bring me a control that's genuinely more effective than the defined-approach version, but because the targeted risk analysis was thin, I couldn't build defensible testing procedures around it, and it ended up costing more time than just meeting the requirement the standard way." — Devon Okafor, QSA, Bastion Compliance Partners
The Five Reporting Statuses
Every testing procedure result resolves into one of five reporting statuses recorded in the Report on Compliance, and understanding these statuses is essential to understanding what an assessment actually concludes.
Status | Meaning | Practical implication |
|---|---|---|
In Place | The control is fully implemented and testing confirms it meets the requirement as tested | No follow-up action required for this control cycle |
In Place with Remediation | The control was not fully in place at some point during the assessment period but was remediated before the ROC was finalized, with evidence of the fix | Requires documented proof of remediation completed prior to report completion; can still result in a compliant overall report if all findings are remediated |
Not Applicable | The requirement does not apply to the environment being assessed (e.g., a wireless-specific requirement where no wireless technology is in use) | Requires documented justification explaining why the requirement doesn't apply — not simply asserted |
Not Tested | The requirement was excluded from the scope of this particular assessment (common for requirements outside a defined engagement scope, such as an SAQ that doesn't cover every requirement) | Must be explicitly justified and is distinct from "Not Applicable" — the control might apply but simply wasn't assessed this cycle |
Not in Place | The control does not meet the requirement as tested and has not been remediated | Results in a non-compliant finding for that requirement; drives the overall compliance determination |
The distinction between "Not Applicable" and "Not Tested" trips up a surprising number of compliance teams. Not Applicable is an affirmative, justified conclusion that a requirement doesn't apply to your environment at all. Not Tested means the requirement could apply but fell outside this specific assessment's boundaries — it's a scoping statement, not a compliance statement, and it should never be used as a workaround to avoid testing something inconvenient.
From Requirement to Status: The Testing Flow
The mermaid diagram below traces how a single requirement moves from applicability determination through testing method selection to final reporting status — the path every control in a Report on Compliance travels.
flowchart TD
A[Requirement identified in scope] --> B{Applicable to this environment?}
B -- No --> C[Status: Not Applicable\nwith documented justification]
B -- Yes --> D{Within this assessment's defined scope?}
D -- No --> E[Status: Not Tested\nwith documented justification]
D -- Yes --> F{Defined approach or customized approach?}
F -- Defined --> G[Apply standard testing procedures:\nExamine / Observe / Interview / Test]
F -- Customized --> H[Assessor derives testing procedures\nfrom targeted risk analysis]
G --> I{Evidence confirms control operating\nas required?}
H --> I
I -- Yes --> J[Status: In Place]
I -- No, but remediated before ROC finalized --> K[Status: In Place with Remediation]
I -- No, unresolved --> L[Status: Not in Place]Case Study: TerraPay's Requirement 10 Remediation
TerraPay Solutions, the payment processor from this article's opening, is a composite drawn from patterns common in mid-market processor assessments. Maria Delgado's team found that TerraPay's SIEM alerting was technically sound — rules were tuned, alert volume was manageable, and the security operations team genuinely was triaging alerts most days. What failed the testing procedure wasn't the control's existence; it was the complete absence of a reviewable record showing daily log review had occurred, a specific examine-and-interview testing procedure under Requirement 10.
The remediation took eleven weeks: implementing a ticketing workflow that auto-generated a daily review record tied to actual SIEM queries run, backfilling three months of retroactive evidence where reconstructible from underlying system logs (accepted by the assessor as partial mitigating evidence, though not a substitute for contemporaneous records going forward), and training the security operations team on evidence hygiene as a discrete responsibility separate from the security work itself. Final cost: approximately $310,000 in contractor hours, delayed onboarding penalties from the new acquiring bank relationship, and one calendar quarter of remediation status on the AOC before the next full annual cycle. The technical control passed on re-test in under a week once the evidence trail existed — the eleven weeks were almost entirely about building a documentation habit, not fixing security.
Case Study: A Grocery Chain's Facility Sampling Surprise
A 220-location regional grocery chain — another composite scenario reflecting common SAQ D and physical-security testing patterns — assumed its facility sampling for Requirement 9 would mirror the prior year's assessment: a handful of "representative" stores near the corporate office. The incoming ISA-led assessment, run under a revised internal audit charter, instead stratified the population by point-of-sale hardware generation, since three different POS refresh waves had left older, legacy terminal configurations in about 40 stores that had not been touched during the two most recent assessment cycles.
The sample pulled six of those legacy-configuration stores specifically because they represented the least-recently-tested population, not because they were geographically convenient. Two of the six had signage and badge procedures that had drifted from the documented standard — a finding that would have gone unnoticed for a third consecutive year under the old "convenient sample" logic. The chain's response was to formalize its own internal facility stratification model, categorizing stores by POS generation, remodel status, and time since last physical security review, which materially improved the defensibility of its own internal audit sampling going into the next external assessment.
Assessor Workpapers and Evidence Retention
Every conclusion an assessor reaches — the validation method used, the sample drawn, the specific artifact examined — is documented in workpapers that support the Report on Compliance narrative. This matters to compliance teams for a practical reason: assessors expect to retain copies of, or detailed references to, the specific evidence reviewed, not just a summary conclusion. If a QSA writes "examined firewall configuration export dated [date], confirmed rule 47 restricts inbound traffic to documented business need," that export needs to exist, be retrievable, and match the date cited.
Organizations preparing for assessment should maintain an evidence library organized by sub-requirement (not just by requirement) with version control, a clear owner per artifact, and a retention period that covers at minimum the current assessment period plus enough historical evidence to demonstrate the control operated continuously, not just at a single point-in-time snapshot. This is directly relevant to how a QSA scopes and runs the engagement — our dedicated piece on the QSA assessment process covers how assessors structure engagements, staff testing, and manage workpapers end-to-end, which pairs well with the evidence-preparation focus here.
Common Evidence Gaps Found During Testing
Gap pattern | Requirement family most affected | Why it fails testing |
|---|---|---|
Policy exists but references outdated system names/owners | Req 12, cross-cutting | Fails "current and accurate" examine criteria |
Access reviews performed but undocumented | Req 7, Req 8 | No artifact for the examine/interview testing procedure to inspect |
Log review happens but isn't recorded | Req 10 | Interview and examine testing procedures both fail without a record |
Vulnerability scans run but remediation not tracked to closure | Req 11 | Testing procedure requires evidence of rescan/closure, not just initial scan |
Physical visitor logs kept inconsistently across locations | Req 9 | Sampling exposes inconsistency that a single-site review would miss |
Vendor agreements missing explicit PCI DSS responsibility language | Req 12.8 | Examine testing procedure specifically requires this language, not implied responsibility |
Change tickets lack security/compliance sign-off field | Req 6 | Testing procedure for change control requires evidence of that specific approval step |
Sample population not stratified (only "easy" systems tested internally) | Cross-cutting | Internal pre-assessment testing misses the same components an external assessor would sample differently |
Testing Cadence: Annual, Quarterly, and Continuous
Not all PCI DSS testing procedures run on the same clock, and confusing an annual assessment cadence with the underlying control's actual required frequency is a common planning mistake.
Cadence | Representative requirements | What must exist between formal assessments |
|---|---|---|
Continuous / ongoing | Req 10 log review, Req 5 anti-malware updates, Req 1 firewall rule justification | Evidence has to be contemporaneous — an assessor testing a full year expects records spanning the full year, not evidence generated just before the engagement |
Quarterly | Req 11 ASV external scans, internal vulnerability scans, some access reviews | Four discrete evidence sets per year minimum; gaps in any quarter typically drive an "In Place with Remediation" or worse |
Semi-annual | Some firewall rule-base reviews, some access recertifications | Two discrete evidence sets; assessors check both actually occurred, not just the more recent one |
Annual | Penetration testing, risk assessment refresh, policy review and approval, security awareness training | One evidence set, but assessors verify it reflects the current environment, not a stale prior-year artifact re-dated |
Assessment-cycle (point-in-time confirmation, continuous underlying control) | Overall Report on Compliance / AOC | The ROC/AOC is the formal output, but the underlying controls it certifies are expected to operate continuously across the period, which is the "business-as-usual" philosophy central to v4.0 |
Preparing Your Evidence Library Before the Assessor Arrives
Preparation step | Purpose |
|---|---|
Map evidence to sub-requirement, not just requirement number | Matches how testing procedures are actually structured and numbered |
Assign a named owner per evidence artifact | Assessor interviews go smoother when the right person is identified in advance |
Pre-stratify system and facility populations | Speeds sampling and demonstrates a mature asset inventory |
Run an internal dry-run of interview questions | Surfaces inconsistent answers before an external assessor does |
Verify document dates fall within the assessment period | Prevents "stale evidence" findings, a frequent source of avoidable delay |
Confirm vendor/service provider contracts include required PCI DSS responsibility language | Directly satisfies a specific examine testing procedure under Req 12 |
Reconcile scan and pentest remediation tickets to closure | Req 11 testing procedures specifically check remediation evidence, not just initial detection |
If using the customized approach anywhere, finalize the targeted risk analysis first | Assessor cannot derive testing procedures without it — this is a blocking dependency |
Preparing an evidence library this way is essentially building the raw material that eventually becomes your Report on Compliance narrative — for organizations validating at Level 1 or opting for a full assessment rather than a self-assessment questionnaire, understanding what that document actually requires helps reverse-engineer the evidence prep. Our companion article on ROC documentation requirements walks through the report's structure section by section.
The Role of ASVs and Penetration Testers in Requirement 11 Testing
Requirement 11's testing procedures rely heavily on third parties whose own work becomes the evidence an assessor examines. An Approved Scanning Vendor conducts the quarterly external vulnerability scans, and the assessor's testing procedure for this control is largely an examine-and-test exercise against the ASV's own report: confirming scans were run at the required cadence, confirming no high-severity vulnerabilities remain unresolved across passing scans, and confirming the scanned scope actually matches the organization's external-facing cardholder data environment footprint.
Penetration testing carries a heavier evidentiary load. The assessor examines the penetration test methodology (is it based on an industry-standard approach), the scope (does it cover the full CDE perimeter and any segmentation controls relied upon to reduce scope), the tester's qualifications and independence from the team that built the environment being tested, and the remediation and retest evidence for any exploitable finding. Segmentation testing specifically requires evidence that the segmentation controls actually prevent access between the CDE and out-of-scope networks — a technical test, not just a network diagram review — and this segmentation testing evidence becomes especially important for organizations relying on scope reduction to limit their overall assessment burden.
Interview Prep: What Assessors Ask, and Whom
Role interviewed | Typical testing-procedure focus | What a weak answer sounds like |
|---|---|---|
CISO / Security Director | Program governance, risk assessment ownership, policy approval | Vague ownership ("the team handles that") without a named accountable individual |
Network/Firewall Engineer | Rule justification, review cadence, change process | Can't explain why a specific rule exists or who approved it |
System Administrators | Hardening standards, patch cadence, account provisioning | Describes a process that doesn't match the documented standard |
SOC / Log Review Analyst | Alert triage, escalation criteria, review frequency | Can describe the process but can't produce a record of having done it |
Application Developers | Secure coding practices, code review, vulnerability remediation | Unfamiliar with the organization's own SDLC policy document |
HR / Onboarding Staff | Background checks, awareness training completion, termination procedures | Can't confirm deprovisioning is triggered automatically or promptly from HR events |
Facilities / Physical Security | Badge administration, visitor procedures, media destruction | Procedure differs from what's documented, or differs across locations |
Third-Party / Vendor Management | Service provider due diligence, contract language, ongoing monitoring | Unable to produce evidence of annual confirmation from service providers |
Interview inconsistency is one of the fastest ways a control moves from "In Place" toward "Not in Place" — not because the interview method is punitive, but because it's specifically designed to cross-check the examine and observe evidence, and mismatches are exactly what it's built to surface.
Common Mistakes Organizations Make During Testing
Compliance teams repeatedly stumble on the same handful of testing-readiness mistakes. The most common is treating "we have a policy" as equivalent to "we have evidence the policy is followed" — a gap that interview and observe testing procedures are specifically designed to expose. A close second is failing to retain evidence with dates that clearly fall inside the assessment period; a beautifully maintained access review process that can't produce a timestamped record for the specific months under review still fails the examine testing procedure. Third is under-preparing the people who'll be interviewed — assuming the CISO can speak for what a help desk analyst actually does day to day, when testing procedures often specifically require talking to the person who performs the task, not their manager.
A fourth mistake is misunderstanding sampling as something you can influence by presenting your "best" systems. Assessors are trained to stratify populations and are increasingly skeptical of samples that conveniently exclude legacy, recently changed, or geographically remote components — precisely the populations most likely to reveal a gap. Fifth, and increasingly common under v4.0, is underestimating the documentation burden of the customized approach; organizations sometimes select it because a control feels more sophisticated, without appreciating that the assessor now has to build testing procedures from scratch, which adds time and cost the defined approach wouldn't have incurred. Sixth is neglecting Requirement 12's less glamorous testing procedures — vendor contract language, annual policy review sign-off, incident response plan currency — because they feel administrative rather than technical, even though they're tested with the same rigor as a firewall configuration.
Testing Procedures as a Competitive Advantage, Not Just an Audit Cost
It's tempting to treat all of this — validation methods, sampling logic, reporting statuses, evidence libraries — as pure compliance overhead: work you do because a QSA demands it, with no upside beyond avoiding a finding. That framing undersells what a mature testing-readiness program actually buys an organization. A company that can hand an assessor a clean, sub-requirement-mapped evidence library, with named owners and contemporaneous records, isn't just passing an assessment faster — it's demonstrating to acquiring banks, payment processors, and enterprise customers doing vendor due diligence that its security program produces reliable, verifiable signal rather than after-the-fact reconstruction. That reputation shortens sales cycles, reduces the friction of onboarding new banking relationships, and lowers the cost of every subsequent assessment cycle, because the evidence habits built for one PCI DSS assessment period carry forward into the next rather than being rebuilt from scratch. Treat testing-procedure readiness as an operating discipline, not an annual scramble, and the assessment stops being the hard part.
If your team is building out that evidence library and wants a structured starting point, PentesterWorld's PCI DSS 12-Requirements Checklist maps control activity to requirement, and the PCI DSS Gap Analysis Tool helps identify where your current evidence would fail a testing procedure before an assessor ever finds it. For teams deciding between the defined and customized approach on specific requirements, the PCI DSS Scoping & CDE Worksheet is a useful companion for documenting the population and rationale sampling will be drawn against. And if you're heading toward a full Level 1 assessment, our ROC/AOC Reader's Guide eBook breaks down exactly how testing-procedure results translate into the final report you'll hand your acquirer.
Payment security programs increasingly sit alongside other assurance frameworks — organizations validating PCI DSS often carry SOC 2 reports for the same payment infrastructure, particularly where a payment processor's SOC 2 covers overlapping access control and monitoring commitments to enterprise customers, and many also maintain ISO 27001 certification; our cross-pillar piece on whether you need both ISO 27001 and PCI DSS is worth a read if you're managing multiple assurance obligations against the same cardholder data environment. The audit-sampling logic in this article also echoes closely with how SOC 2 examinations size their own control samples, which is a useful comparison if your team supports both attestation types.
Ready to see exactly where your evidence would hold up under an assessor's testing procedures — and where it wouldn't? PentesterWorld's PCI DSS advisory team runs pre-assessment readiness reviews structured around the same examine, observe, interview, and test methodology your QSA or ISA will use, so nothing about assessment day is a surprise. Reach out to scope a readiness review before your next assessment cycle begins.