My #BSidesLV slides: "Mobilizing the PCI Resistance: Lessons Learned From Fighting Prior Wars (SOX-404)"

I had a great time presenting the work I'm doing with the PCI Scoping SIG to the #BSidesLV community. It was especially gratifying to me that the practitioners in the room agreed so strongly with me on the importance of correct scoping.

Simply put, the success of any PCI DSS compliance initiative is very dependent on accurate definition and scoping of the Cardholder Data Environment and the scope of assessment.

Thank you Mike Dahn (@MikD) and for allowing me to present in your dojo. And I felt incredibly honored when you said that you're already telling your clients that the scoping tools that we're building will likely be the most significant change to the PCI SSC in years.

There is no doubt in my mind that Mike has been one of the top contributors to the PCI DSS body of knowledge, so that kind of endorsement means a lot to me!

Taking Bets On When The "Mike/Josh Hug" Will Occur!

I am also announce that I am taking up the challenge from Josh Corman (@joshcorman) and Mike Dahn (@MikD) to help them "hug it out."  Given both of their genuine passion for helping organizations be simultaneously secure and compliant, and their obvious talents, I think I'm up to the task!

I'm starting a pool: who can most accurately predict when "Mike/Josh Hug" will finally occur? The winner will get a framed picture of the hug, signed by both Mike and Josh.

Full Slideshare link below!

2010 07 BSidesLV Mobilizing The PCI Resistance 1c

View more presentations from realgenekim.

Talk abstract:

Properly Mobilizing the PCI Resistance: Lessons Learned From Fighting Prior Wars (SOX-404)"

I have noticed that there is a growing wave of discontent and disenchantment from information security and compliance practitioners around the PCI DSS.  Josh Corman has been an effective voice for these concerns, providing an intellectually honest and earnest analysis in his talk “Is PCI The No Child Left Behind Act For Infosec?”

The problem are well-known and significant: too much ambiguity in the PCI DSS, Qualified Security Assessors (QSAs) and consultant using subjective interpretations, existing guidance either too prescriptive or too vague, scope missing critical systems that could risk cardholder data, overly broad scope and excessive testing costs, excessive subjectivity and inconsistency, poor use of scarce resources, no meaningful reduction in risk of data breaches, and so forth.

For years, I have been studying the PCI DSS compliance problem, as well.  I have noticed many similarities to the PCI compliance challenges and the “SOX-404 Is The Biggest IT Time Waster” wars in 2005.  I was part of the leadership team at the Institute of Internal Auditors (IIA) where we did something about the it. We identified inability to accurately scope the IT portions of SOX-404 as the root cause of the billions of dollars of wasted time and effort, while not reducing the risk of financial misstatements.

I propose to present the two-year success story of the IIA GAIT project and how we changed the state of the IT audit practice in support of SOX-404 financial reporting audits.  We defined the four GAIT Principles, which could be used to correctly scope the IT portions of SOX-404.  We mobilized over 100K internal auditors, the SEC and PCAOBregulatory and enforcement bodies, as well as the external auditors from the 8 big CPA firms (e.g, Big Four and other firms doing SOX advisory work).  In short, we made a difference, in a highly political process that involved many constituencies.

I am attempting to do something similar with the PCI Security Standards Council, through my work as part one of the leaders of the PCI Scoping SIG (Special Interest Group).  My personal goal is to find a “third way” to better enable correct scoping of the PCI Cardholder Data Environment, and create a risk-based approach of substantiating the effective controls to ensure that cardholder data breaches can be prevented, and quickly detected and corrected when they do occur.

My desired outcome is to find fellow travelers who also see the pile of dead bodies in PCI compliance efforts, and work with those practitioners to catalyze a similar movement to achieve the spirit and intent of PCI DSS.

Are You As Good At Scoping PCI As You Think? (Mobilizing The PCI Resistance, Part VII)

I'm getting ready for my BSides talk on Wednesday on "Mobilizing the PCI Resistance." And I made a slide that was designed to hint that defining the scope of PCI assessment may not always be as easy one might think.

In today's blog post, I pose some (possibly trick) questions.  But before I do that, to refresh your memory, here's what are the previous postings:

• Part I: "Upset about the subjectivity and ambiguity in the PCI DSS compliance standards? My #BSides submission on the answer..."
• Part II: "The problems that management and auditors faced in 2005 and 2006 for the IT portions of SOX-404."
• Part III: "Quantifying the huge amount of wasted IT audit effort in SOX-404"
• Part IV: "What goes wrong in a bottom-up SOX-404 audit: a cautionary tale..."
• Part V: "The GAIT Vision For Solving The SOX-404 IT Scoping Problem
• Part VI: The Politics Of SOX-404 And GAIT (And Implications With PCI)

PCI Scoping Quiz: Please Show Your Work

At one time, I believed that the Cardholder Data Environment being defined by where cardholder data "enters, is transmitted, processed, stored, displayed or printed" is sufficient to inform most scoping decisions.

Gosh, was I wrong.

Don't worry if the answers don't seem immediately obvious. I'd wager I would have gotten most of these wrong last year.

Question 1:  Is the Cardholder Data Environment (CDE) equivalent to the PCI Scope of Assessment?

Question 2: Is a domain controller (e.g., Windows Active Directory server) that is being relied upon by CDE applications for authentication and security services in the PCI Scope Of Assessment?

Question 3: How about a domain controller (e.g., Windows Active Directory server) that is not relied upon by any CDE applications?

Question 4: Is a network attached stapler that happens to be on the same network segment as a CDE system component always also in the CDE?

Question 5: Does it matter if a workstation that a customer service representative uses a thin- or thick-client?

Question 6: When should it be acceptable that if a virtualization hypervisor hosting  a production application in the CDE be also able to host another VM without it being part of the CDE, as well?

Question 7: If you have a domain controller that is not in the CDE, but in the scope of PCI assessment, is a print server on the same network segment as that domain controller also in the scope of PCI assessment?

Bonus Exercise: For each of the questions where you answered "in scope of the PCI assessment," describe a strategy to contain the scope, such that systems connected to that system are not in scope.  (See Michelle Klinger's great post on the "PCI Contagion Dilemma.")

(Image courtesy: Flickr: Zeligfilm)

The Answers And My Goal

The answers for all of the questions is actually, "It depends."  But just as problematic is the fact that people could arbitrarily disagree with your answer, with little ability to defend its validity.  The trick is being able to state, "Yes, it depends, but on what does it depend upon?"

Also, generating a consensus on scoping conclusions takes a lot of time. Prior to creating a structured method with agreed upon definitions, the Scoping SIG required over 40 hours to come to a scoping conclusion for one scenario.

With the proposed guidance under development, our team was able to generate a consensus on 15 scoping conclusions in less than 2 hours.

We believe there are three things needed to aid the scoping effort, and be able to defend your answers, so that another person can follow your reasoning.  Even if they don't agree with your scoping conclusion, at least they can scrutinize your assumptions.

My goal is to have the PCI Scoping SIG deliver;

• Define and deliver the following, in a manner that clarifies and supports the spirit and intent of protecting cardholder data:
  • Scoping principles and definitions (should be 200 words or less)
  • A structured scoping methodology (should be a decision tree, with fewer than 30 boxes)
  • A library of scoping scenarios demonstrating its usage for educational and clarification purposes (should be about 30 pages)
• Create useful tools and guidance that will assist in the scoping effort for both merchants and QSAs.

Interested?

Mobilizing The PCI Resistance, Part VI: The Politics Of SOX-404 And GAIT (And Exploring PCI As Well)

This is Part 6 of the "Mobilizing PCI Resistance" series.  Briefly, we've covered:

• Part I: "Upset about the subjectivity and ambiguity in the PCI DSS compliance standards? My #BSides submission on the answer..."
• Part II: "The problems that management and auditors faced in 2005 and 2006 for the IT portions of SOX-404."
• Part III: "Quantifying the huge amount of wasted IT audit effort in SOX-404"
• Part IV: "What goes wrong in a bottom-up SOX-404 audit: a cautionary tale..."
• Part V: "The GAIT Vision For Solving The SOX-404 IT Scoping Problem

The Politics Of SOX-404 And GAIT (And Implications With PCI)

The GAIT project was one of the most politically charged projects I've ever been involved with.  The diagram below shows each one of the constituencies, and the relationships between them.

As we go through them, I'll discuss the equivalents to the PCI universe.  I posit that while the PCI ecosystem is also very political, it's less so than for SOX-404.  This is good news.  :-)

My reason for this blog post is to describe how we analyzed the lay of the land, and created a winning strategy that allowed us to change the compliance landscape. By doing this with PCI, we can play to win.

(By the way, this type of diagram is called a "value network," and was taught to me by the famous Eileen Forrester at the Software Engineering Institute at Carnegie Mellon University.  Full slides available on SlideShare)

So, let's go through each of primary constituencies, and then I'll discuss how they relate to each other:

The Federal Regulatory Enforcement Bodies:

The Sarbanes-Oxley Act of 2002 was enacted as a response to accounting scandals at WorldCom, Enron, Tyco International, etc.

• 

  The Securities and Exchange Commission (SEC) created a new set of requirements that public companies (i.e., "SEC registrants") had to comply with.

• The Public Corporation Accounting Oversight Board (PCAOB) was created by SOX-404, charged with overseeing, regulating, inspecting and disciplining accounting firms in their roles as auditors of public companies.
  
  In other words, they audit the auditors.  For instance, on at least an annual basis, they would audit the work papers submitted by the Big Four firms to ensure the quality of their work, and that it's in compliance with the standards of independence. Each year, they would publish a report describing when the external auditors behaved in ways that was "out of line."

The PCI equivalent here is the PCI Security Standards Council and the card brands. The parallels between the PCAOB and the new PCI requirements for QSAs to submit their work papers and Reports on Compliance (ROC) is striking. PCI is now actively auditing the auditors.

The Public Companies:

• 

  CEOs and CFOs: These are the people specifically named in SOX-404 as personally accountable and responsible for the effectiveness of controls that support the accuracy of the company financial statements (e.g., 10-K statements submitted to the SEC).

  When people make jokes about "keep the executives out of orange jumpsuits" or "keep them out of jail," these are the people who would be wearing them.

• 

  CIOs and IT Management: We identified that one of the primary contributors around the wasted time/effort in support of SOX-404 was in the domain of IT controls. Business management owned the financial controls, and IT management owned the IT controls.

• Information security: More specifically, many of the IT controls were owned by information security.

The PCI equivalent here are the merchants, service providers, and anyone else who has custodianship of cardholder data. These are the people who are often complaining that the PCI DSS is too subjective, too arbitrary, that the cost of compliance is too high, etc.

And again, this is the likely the community we must mobilize first.

The Internal Auditors

Internal auditors are independent of Business Management and IT Management. Their goal is to ensure that the risks to the organization are understood and that adequate controls exist to prevent, detect and correct for those risks.

(Interestingly, in my experience, in most organizations, internal auditors are only recently showing up at the table for PCI compliance. But, they have a wealth of experience to bring in terms of managing external auditors. And it's my understanding that internal auditors can now submit PCI Reports On Compliance now, even though they're not technically a QSA.)

The External Auditors:

External auditors generate an opinion on the financial statements. They also have their work papers audited by the PCAOB.

The PCI equivalent here are the Qualified Security Assessors (QSAs). Instead of generating an opinion on the accuracy of financial statements, they file a ROC.

Professional Organizations:

• Institute of Internal Auditors (IIA): 118K+ due paying members: internal auditors
• Information Systems Audit And Control Association (ISACA): 80K+ due paying members: information security and IT auditors
• National Association Of Corporate Directors (NACD): professional organization of corporate board members.

The IIA was clearly at the forefront of this issue, and this is the community we mobilized.  (ISACA was peripherally involved in the beginning, but dropped out due to unbelievably bad relationships between them and IIA.)

(In the PCI world, I'm not sure if there is a professional organization that we can mobilize. Instead, I believe the best lever is the PCI Community of Participating Organizations.)

Our Resulting SOX-404 Strategy: Start With Internal Auditors, Where Collective Outrage Was Highest

The constituency who probably saw the problems the clearest were the internal auditors, who saw that the amount of investment in IT controls were way out of proportion to the risk. While other constituencies also saw the problem, we believed it was the internal auditors who could describe it most clearly, and could mobilize a response.

We then gathered the heads of audit (and sometimes IT audit) from some of the largest publicly traded companies. Our goal was to assemble the auditors from top Fortune 50 companies, and have them uniformly say, "we all see a common problem, and it is causing significant economic harm. And we have a joint proposal for what we should do about it."

As mentioned in a previous blog post, the initial companies involved included General Motors, Wal-Mart, Hewlett Packard, Intel, Microsoft, Marathon Oil, Chevron Phillips Chemical, Business Objects, etc.

When those companies start screaming, people listen.

Next, Rope In Their External Auditors

Now that we had an impressive list of publicly traded companies all saying we had a problem, the next step was to engage their audit firms.

We had each one of the audit executives reach out to their IT audit engagement partners, and invite them to the GAIT summit. We selected and organized the companies to ensure that we had coverage of each of the Big Four firms (i.e., PwC, KPMG, E&Y and D&T). This was framed as a strong request to the Big Four firm, along the lines of, "I believe that it is very important for the relationship between our two companies that you personally attend this summit. And I'd like you to bring someone from your national practice, as well."

That last request was important, because the goal was not to change the IT scoping practices of, say, E&Y and Microsoft. Instead, the goal was to affect all of the engagements that E&Y was involved in. The national practices at the Big Four firms are usually responsible for establishing the standards, practices and procedures for the entire organization. They're typically some of the most senior and experienced partners, as well as promising staff members likely to become partners. Why would they want to attend?

Lastly, How To Mitigate The Response Of "External Auditors Say, 'Up Yours'?"

This was probably one of the biggest obstacles to the GAIT initiative. How could encourage or compel the Big Four firms to participate? A cynical person would say that there was no real reason for them to get involved, as GAIT would do the following:

• Decrease billable hours in support of SOX-404 projects (i.e., "stop the SOX-404 gravy train")
• Increase the standards that they"™d have to adhere to (i.e., "create rope that they could get hung on")

Our solution was to make sure that Bill Powers from the PCAOB attended each of the GAIT Summits. The goal of GAIT transformed then to ensure that IT work performed in support of SOX-404 represented a continuation of the top-down, risk-based approach described in Auditing Standard 2, a publication published by the PCAOB. The implication is that the PCAOB would be very interested in any Big Four firms who did not want to participate in the GAIT process.

Voila: The GAIT Summits Began

Now we had assembled each of the constituencies required, each that could achieve a desired outcome out of a successful GAIT project.

• Internal auditors: accurately scope and substantiate IT controls in support of SOX-404
• Company management: ensure accurate financial statements and contain costs
• External auditors: stay out of trouble with the PCAOB
• Regulators: ensure that IT audit activities matched the spirit and intent of AS-2

Next up, I'll pose some thought-experiments on PCI scoping, and walk through the GAIT Principles.

Mobilizing The PCI Resistance, Part V: The GAIT Vision For Solving The SOX-404 IT Scoping Problem

This is Part 5 of the "Mobilizing PCI Resistance" series.  Briefly, we've covered:

• Part I: "Upset about the subjectivity and ambiguity in the PCI DSS compliance standards? My #BSides submission on the answer..."
• Part II: "The problems that management and auditors faced in 2005 and 2006 for the IT portions of SOX-404."
• Part III: "Quantifying the huge amount of wasted IT audit effort in SOX-404"
• Part IV: "What goes wrong in a bottom-up SOX-404 audit: a cautionary tale..."

Okay, enough on the problem.  Let's talk about the solution....

What We Wanted GAIT To Achieve

So, what was our vision in January 2006?

• Enable auditors and management to appropriately identify and link assertions to IT activities and processes, and then appropriately scope relevant IT controls work
  
  What we wanted to achieve provide was a way for auditors and management to precisely scope what in IT mattered for the achievement of SOX-404 objectives.  Or put more precisely, to link internal control objectives for financial reporting to specific IT functionality.
  
  And then only audit those things.  Instead of carpet-bombing/auditing everything in IT.
  
  
• Provide a common context for management and auditors to support and test management’s assessment that the necessary IT controls exist and are effective
  
  Initial target is internal control objectives for financial reporting, but should extend to operating effectiveness and complying with laws and regulations (as defined by COSO)
  
  What we were suggesting here is that "SOX-404 is only the beginning. The same principles could be applied to the other COSO objectives: security, compliance with laws/regulations/contractual obligations."  
  
  (Look, it's the PCI DSS!!!)

And Stopping The Madness Of "See, This Audit Deficiency Didn't Really Matter!"

Lastly, shown above is what is known as "Chart 3" of the "A Framework For Evaluating IT Control Deficiencies" document, authored by the nine CPA firms that did SOX-404 audits or advisory work, as well as Dr. William F. Messier, Jr.

Basically you would have to dig out this chart for every IT deficiency to try to wiggle out of a material weakness.  You would go through this decision tree to decide whether the deficiency would result in a material weakness, a significant deficiency or just a deficiency.

Just so at the end you could say, "See?  I told you so!  That audit finding isn't really important."

Trouble is, to arrive at that decision took man-weeks of work. Why was the test performed in the first place?

Our observation is that if you were spending lots of time going through Chart 3 for all your IT findings, only to find that they wouldn't result in a material weakness, it was a scoping error.  So, GAIT would enable you to do this thinking up front, during scoping, so that we would only perform those tests that would result in an undetected material weakness.

In my next post, I intend to write about the constituencies  and politics of getting GAIT approved by all the parties:

• internal auditors
• IT management
• security/compliance management
• professional organizations: IIA, ISACA, FASB
• enforcement organizations: PCAOB

I'll talk about how we assembled the constituencies, what was in it for them, and how I learned to use one of the most valuable tools in my career.

And then I'll start talking about the GAIT Principles, and how we're extending it towards application towards PCI DSS.

(Many were fellow committee members with me at the Institute of Internal Auditors.  In the next post, I'll describe why we had assembled the specific players in the room: SEC publicly held companies, their audit engagement partners from the Big Four, as well as their respective national practice leaders, the Institute of Internal Auditors, and the PCAOB.)





Mobilizing The PCI Resistance, Part IV: When Bottom-Up SOX-404 Audits Go Bad. Really Bad.

This is Part 4 of the "Mobilizing PCI Resistance" series.  Briefly, we've covered:

• Part I: "Upset about the subjectivity and ambiguity in the PCI DSS compliance standards? My #BSides submission on the answer..."
• Part II: "The problems that management and auditors faced in 2005 and 2006 for the IT portions of SOX-404."
• Part III: "Quantifying the huge amount of wasted IT audit effort in SOX-404"

In this article, I will share a cautionary tale of how the problems discussed in the previous articles result in such horrible outcomes.  More specifically, how the inability to scope correctly the IT portions of SOX-404 led to tons of firefighting by information security and auditors and wasted effort, all at the expense of more important things that they should have been focusing on.

Like keeping the organization secure.  As opposed to trying to achieve compliance of a misguided and mis-scoped audit.  Which is at the heart of the PCI problem.

How Bottom-Up SOX-404 Auditing Happens: A Cautionary Tale

How do these audit horror stories happen?  How do intelligent people in management and audit end up auditing things that don't matter?

We studied this problem when we wrote the "Visible Ops Security: Achieving Common Security and IT Operations Objectives in 4 Practical Steps" book.

This is an excerpt from the book, in a chapter called "A SOX-404 Cautionary Tale":

When external auditors started testing against SOX-404 in the first year, IT findings represented the largest category of findings, totaling more than the combined findings in the revenue, procure-to-pay, and tax categories.  It’s estimated that as much as $3 billion was spent in the first year of SOX-404 to fix IT controls to remediate these findings. Ultimately, most of these findings were found not to be direct risks to accurate financial reports and did not result in a material weakness.  This is because they followed a bottom up versus a top-down, risk-based approach.

Consider the following scenario: The SOX-404 team asks for an information security review of a WebSphere server that runs the materials management systems. The review shows that it’s a custom WebSphere application running on a cluster of servers that is connected to a clustered Oracle database. We then locate the firewall and determine the segment it’s on.

An information security review of the materials management system uncovers:

• Numerous ghost accounts
• A lack of password aging policies
• Critical vulnerabilities in the Java code, including cross-site scripting issues in the HTML
• Vulnerabilities in the Oracle database configuration
• Firewall rules that are suspect and need further investigation

Our task list keeps expanding and the internal auditors are showing up next week. We decide to focus on the operating system level, and our suspicions prove to be correct: The operating system is not running at the latest patch levels. We add this to our list of corrective actions that need to be taken right away, and start talking with the owners of the operating system, database, and application, and even the firewall team.

When the internal audit team comes in, we are candid and transparent about all the issues. Management is informed about the risks, and soon 50+ people are working on all these issues, dropping other high-priority projects to get these issues fixed in time. After all, the argument is made, these issues should be fixed eventually because they do represent risk.

But there just isn’t enough time. The external auditors come in and find all of these issues. They start preparing a management letter stating that the integrity of the IT general control process (ITGC) environment cannot be substantiated.

As a result, more high-level meetings take place, and the financial people start to argue that the ITGC issues really can’t lead to an undetected financial reporting error. They pull out the “nine firm document” and use something called “Chart Three” to make the case.  Then management and the CPA firms argue back and forth about the linkage, and management starts bringing in all the business experts to show that a failure in the ITGCs for this system could not result in inaccurate financial reports.

Finally, the owner of the materials management business process determines that even if the application, database, operating system, and firewall were compromised by a person trying to perpetrate fraud, the attempt would be caught by a daily financial reconciliation between the materials management inventory report and another report from the ERP system.

Given this new evidence, everyone agrees that reliance is actually placed on the daily financial reconciliation, which would catch both fraud and errors. Furthermore, they agree that reliance is not placed on the IT system and the supporting ITGCs.  So, the IT systems are out of scope, and no further IT testing is required.

Everyone is relieved. As the information security practitioners, however, we struggle with this unsettling question about why we went through all this trouble if our efforts were not required to substantiate the accuracy of financial statements. Furthermore, we wonder if all the “good hygiene habits” are actually important and can be justified.

To be clear, it’s not that the downstream manual financial reconciliation control is the best control possible. The point is that if the scoping of IT controls were done correctly in the first place, the only control weaknesses that we would have tested and found would be those that truly jeopardized accurate financial reporting. Instead, we found control weaknesses on systems that were out of scope, and then kept digging needlessly.

Next up, I'll discuss the GAIT vision that was realized in February 2007, when the GAIT guidance was finally published.