My First Three Weeks With The BSides Board

I’m writing this blog post to explain briefly why I chose to accept the BSides board position, what my goals are, and provide a brief status report.

Why I joined the BSides board

Over the years, I’ve come to respect the work of everyone in the BSides community.  I’m amazed and continually reminded of how many people BSides has positively influenced and the vibrant community they’ve created.  I’ve been to three events, and during my tenure at Tripwire, we became one of the first global sponsors.  I’ve always loved the people who congregate there, and I’m grateful for how it's reconnected me with old colleagues and friends.  I proudly consider myself a part of the BSides community.

In mid-December, I was asked by Mike Dahn and Jack Daniel to join them on the BSides board.  I first admitted to them that I’ve primarily been a beneficiary of everyone’s hard work, and that there are countless people who have contributed far more than me.  But after talking with them, I told them that it was a privilege to be asked and that I would be happy to serve for a one-year term and help in any way I could.  

My goals

My goal is to help ensure that BSides succeeds in its mission: to continue to help more information security practitioners achieve their fullest potential, both now and in the future.

Clearly there have been some growing pains. To paraphrase Bill Brenner, this is really an opportunity to "make a better BSides." Our goal as a board is to help BSides grow, become more effective and accountable, as well as more transparent.

A brief status report

Mike, Jack and I started having nearly daily, now weekly, phone calls. The top issues we're working on are the following:

• Create a timeline to complete all the filings necessary for BSides to become officially a 501c3 not for profit corporation
• Create a timeline to retain an outside bookkeeper and release audited financials, going all the way back to the first events, to show that all account balances and values are exactly as they should be, and that all the money went where it was supposed to
• Create a communication calendar so that we regularly release information on what we’ve promised and how we’re doing on those promises, in order to earn back any lost trust with the community

On the 501c3 front, the team continues to move towards the official filing. BSides remains a California public benefit S corporation. As such, there is no official board, but we’ve started to organize and adopt all the structures and processes required for when we have official 501c3 status.

Part of this is getting regular financial reports released that are audited by an independent third party. The team has spent two weeks interviewing firms to take over the daily bookkeeping operations, as well as a CPA firm who can attest to the accuracy of the financials.   I’m particularly pleased that as soon as BSides completes the transition to a 501c3, the CPA firm that opines on the financials of the widely-revered Electronic Frontier Foundation (EFF) will do the same for BSides.  

While I've studied the SecurityErrata post, based on my analysis, I believe that any financial reporting errors found will be small. My biggest concern is that volunteers sometimes paid event suppliers out of their own pockets, due to BSides cash flow issues -- these transactions may not have been recorded or repaid properly. Of course, we will fix any issues we find.

And finally, on the communications front, this will be the first of many communications you’ll see from the team to make you aware of what we’re focused on, and how we’re doing on the commitments we’ve made.

Some last thoughts

Mike and Jack have been terrific to work with, and I’m confident that we’ll have more positive information to share throughout January and February.  From there, the focus of the board will be to discuss the structure that will best serve the BSides mission and community.  

I want to thank the many people who took the time to give me advice, provide recommendations on trusted bookkeepers and accountants, and much more. I particularly want to acknowledge Branden Williams, Brian Costello, Matt Hixson, Todd Butson and Bob McCarthy and countless others for their help, for which I’m very grateful.

Mobilizing The PCI Resistance, Part VI: The Politics Of SOX-404 And GAIT (And Exploring PCI As Well)

This is Part 6 of the "Mobilizing PCI Resistance" series.  Briefly, we've covered:

• Part I: "Upset about the subjectivity and ambiguity in the PCI DSS compliance standards? My #BSides submission on the answer..."
• Part II: "The problems that management and auditors faced in 2005 and 2006 for the IT portions of SOX-404."
• Part III: "Quantifying the huge amount of wasted IT audit effort in SOX-404"
• Part IV: "What goes wrong in a bottom-up SOX-404 audit: a cautionary tale..."
• Part V: "The GAIT Vision For Solving The SOX-404 IT Scoping Problem

The Politics Of SOX-404 And GAIT (And Implications With PCI)

The GAIT project was one of the most politically charged projects I've ever been involved with.  The diagram below shows each one of the constituencies, and the relationships between them.

As we go through them, I'll discuss the equivalents to the PCI universe.  I posit that while the PCI ecosystem is also very political, it's less so than for SOX-404.  This is good news.  :-)

My reason for this blog post is to describe how we analyzed the lay of the land, and created a winning strategy that allowed us to change the compliance landscape. By doing this with PCI, we can play to win.

(By the way, this type of diagram is called a "value network," and was taught to me by the famous Eileen Forrester at the Software Engineering Institute at Carnegie Mellon University.  Full slides available on SlideShare)

So, let's go through each of primary constituencies, and then I'll discuss how they relate to each other:

The Federal Regulatory Enforcement Bodies:

The Sarbanes-Oxley Act of 2002 was enacted as a response to accounting scandals at WorldCom, Enron, Tyco International, etc.

• 

  The Securities and Exchange Commission (SEC) created a new set of requirements that public companies (i.e., "SEC registrants") had to comply with.

• The Public Corporation Accounting Oversight Board (PCAOB) was created by SOX-404, charged with overseeing, regulating, inspecting and disciplining accounting firms in their roles as auditors of public companies.
  
  In other words, they audit the auditors.  For instance, on at least an annual basis, they would audit the work papers submitted by the Big Four firms to ensure the quality of their work, and that it's in compliance with the standards of independence. Each year, they would publish a report describing when the external auditors behaved in ways that was "out of line."

The PCI equivalent here is the PCI Security Standards Council and the card brands. The parallels between the PCAOB and the new PCI requirements for QSAs to submit their work papers and Reports on Compliance (ROC) is striking. PCI is now actively auditing the auditors.

The Public Companies:

• 

  CEOs and CFOs: These are the people specifically named in SOX-404 as personally accountable and responsible for the effectiveness of controls that support the accuracy of the company financial statements (e.g., 10-K statements submitted to the SEC).

  When people make jokes about "keep the executives out of orange jumpsuits" or "keep them out of jail," these are the people who would be wearing them.

• 

  CIOs and IT Management: We identified that one of the primary contributors around the wasted time/effort in support of SOX-404 was in the domain of IT controls. Business management owned the financial controls, and IT management owned the IT controls.

• Information security: More specifically, many of the IT controls were owned by information security.

The PCI equivalent here are the merchants, service providers, and anyone else who has custodianship of cardholder data. These are the people who are often complaining that the PCI DSS is too subjective, too arbitrary, that the cost of compliance is too high, etc.

And again, this is the likely the community we must mobilize first.

The Internal Auditors

Internal auditors are independent of Business Management and IT Management. Their goal is to ensure that the risks to the organization are understood and that adequate controls exist to prevent, detect and correct for those risks.

(Interestingly, in my experience, in most organizations, internal auditors are only recently showing up at the table for PCI compliance. But, they have a wealth of experience to bring in terms of managing external auditors. And it's my understanding that internal auditors can now submit PCI Reports On Compliance now, even though they're not technically a QSA.)

The External Auditors:

External auditors generate an opinion on the financial statements. They also have their work papers audited by the PCAOB.

The PCI equivalent here are the Qualified Security Assessors (QSAs). Instead of generating an opinion on the accuracy of financial statements, they file a ROC.

Professional Organizations:

• Institute of Internal Auditors (IIA): 118K+ due paying members: internal auditors
• Information Systems Audit And Control Association (ISACA): 80K+ due paying members: information security and IT auditors
• National Association Of Corporate Directors (NACD): professional organization of corporate board members.

The IIA was clearly at the forefront of this issue, and this is the community we mobilized.  (ISACA was peripherally involved in the beginning, but dropped out due to unbelievably bad relationships between them and IIA.)

(In the PCI world, I'm not sure if there is a professional organization that we can mobilize. Instead, I believe the best lever is the PCI Community of Participating Organizations.)

Our Resulting SOX-404 Strategy: Start With Internal Auditors, Where Collective Outrage Was Highest

The constituency who probably saw the problems the clearest were the internal auditors, who saw that the amount of investment in IT controls were way out of proportion to the risk. While other constituencies also saw the problem, we believed it was the internal auditors who could describe it most clearly, and could mobilize a response.

We then gathered the heads of audit (and sometimes IT audit) from some of the largest publicly traded companies. Our goal was to assemble the auditors from top Fortune 50 companies, and have them uniformly say, "we all see a common problem, and it is causing significant economic harm. And we have a joint proposal for what we should do about it."

As mentioned in a previous blog post, the initial companies involved included General Motors, Wal-Mart, Hewlett Packard, Intel, Microsoft, Marathon Oil, Chevron Phillips Chemical, Business Objects, etc.

When those companies start screaming, people listen.

Next, Rope In Their External Auditors

Now that we had an impressive list of publicly traded companies all saying we had a problem, the next step was to engage their audit firms.

We had each one of the audit executives reach out to their IT audit engagement partners, and invite them to the GAIT summit. We selected and organized the companies to ensure that we had coverage of each of the Big Four firms (i.e., PwC, KPMG, E&Y and D&T). This was framed as a strong request to the Big Four firm, along the lines of, "I believe that it is very important for the relationship between our two companies that you personally attend this summit. And I'd like you to bring someone from your national practice, as well."

That last request was important, because the goal was not to change the IT scoping practices of, say, E&Y and Microsoft. Instead, the goal was to affect all of the engagements that E&Y was involved in. The national practices at the Big Four firms are usually responsible for establishing the standards, practices and procedures for the entire organization. They're typically some of the most senior and experienced partners, as well as promising staff members likely to become partners. Why would they want to attend?

Lastly, How To Mitigate The Response Of "External Auditors Say, 'Up Yours'?"

This was probably one of the biggest obstacles to the GAIT initiative. How could encourage or compel the Big Four firms to participate? A cynical person would say that there was no real reason for them to get involved, as GAIT would do the following:

• Decrease billable hours in support of SOX-404 projects (i.e., "stop the SOX-404 gravy train")
• Increase the standards that they"™d have to adhere to (i.e., "create rope that they could get hung on")

Our solution was to make sure that Bill Powers from the PCAOB attended each of the GAIT Summits. The goal of GAIT transformed then to ensure that IT work performed in support of SOX-404 represented a continuation of the top-down, risk-based approach described in Auditing Standard 2, a publication published by the PCAOB. The implication is that the PCAOB would be very interested in any Big Four firms who did not want to participate in the GAIT process.

Voila: The GAIT Summits Began

Now we had assembled each of the constituencies required, each that could achieve a desired outcome out of a successful GAIT project.

• Internal auditors: accurately scope and substantiate IT controls in support of SOX-404
• Company management: ensure accurate financial statements and contain costs
• External auditors: stay out of trouble with the PCAOB
• Regulators: ensure that IT audit activities matched the spirit and intent of AS-2

Next up, I'll pose some thought-experiments on PCI scoping, and walk through the GAIT Principles.

Mobilizing The PCI Resistance, Part V: The GAIT Vision For Solving The SOX-404 IT Scoping Problem

This is Part 5 of the "Mobilizing PCI Resistance" series.  Briefly, we've covered:

• Part I: "Upset about the subjectivity and ambiguity in the PCI DSS compliance standards? My #BSides submission on the answer..."
• Part II: "The problems that management and auditors faced in 2005 and 2006 for the IT portions of SOX-404."
• Part III: "Quantifying the huge amount of wasted IT audit effort in SOX-404"
• Part IV: "What goes wrong in a bottom-up SOX-404 audit: a cautionary tale..."

Okay, enough on the problem.  Let's talk about the solution....

What We Wanted GAIT To Achieve

So, what was our vision in January 2006?

• Enable auditors and management to appropriately identify and link assertions to IT activities and processes, and then appropriately scope relevant IT controls work
  
  What we wanted to achieve provide was a way for auditors and management to precisely scope what in IT mattered for the achievement of SOX-404 objectives.  Or put more precisely, to link internal control objectives for financial reporting to specific IT functionality.
  
  And then only audit those things.  Instead of carpet-bombing/auditing everything in IT.
  
  
• Provide a common context for management and auditors to support and test management’s assessment that the necessary IT controls exist and are effective
  
  Initial target is internal control objectives for financial reporting, but should extend to operating effectiveness and complying with laws and regulations (as defined by COSO)
  
  What we were suggesting here is that "SOX-404 is only the beginning. The same principles could be applied to the other COSO objectives: security, compliance with laws/regulations/contractual obligations."  
  
  (Look, it's the PCI DSS!!!)

And Stopping The Madness Of "See, This Audit Deficiency Didn't Really Matter!"

Lastly, shown above is what is known as "Chart 3" of the "A Framework For Evaluating IT Control Deficiencies" document, authored by the nine CPA firms that did SOX-404 audits or advisory work, as well as Dr. William F. Messier, Jr.

Basically you would have to dig out this chart for every IT deficiency to try to wiggle out of a material weakness.  You would go through this decision tree to decide whether the deficiency would result in a material weakness, a significant deficiency or just a deficiency.

Just so at the end you could say, "See?  I told you so!  That audit finding isn't really important."

Trouble is, to arrive at that decision took man-weeks of work. Why was the test performed in the first place?

Our observation is that if you were spending lots of time going through Chart 3 for all your IT findings, only to find that they wouldn't result in a material weakness, it was a scoping error.  So, GAIT would enable you to do this thinking up front, during scoping, so that we would only perform those tests that would result in an undetected material weakness.

In my next post, I intend to write about the constituencies  and politics of getting GAIT approved by all the parties:

• internal auditors
• IT management
• security/compliance management
• professional organizations: IIA, ISACA, FASB
• enforcement organizations: PCAOB

I'll talk about how we assembled the constituencies, what was in it for them, and how I learned to use one of the most valuable tools in my career.

And then I'll start talking about the GAIT Principles, and how we're extending it towards application towards PCI DSS.

(Many were fellow committee members with me at the Institute of Internal Auditors.  In the next post, I'll describe why we had assembled the specific players in the room: SEC publicly held companies, their audit engagement partners from the Big Four, as well as their respective national practice leaders, the Institute of Internal Auditors, and the PCAOB.)





Mobilizing The PCI Resistance, Part IV: When Bottom-Up SOX-404 Audits Go Bad. Really Bad.

This is Part 4 of the "Mobilizing PCI Resistance" series.  Briefly, we've covered:

• Part I: "Upset about the subjectivity and ambiguity in the PCI DSS compliance standards? My #BSides submission on the answer..."
• Part II: "The problems that management and auditors faced in 2005 and 2006 for the IT portions of SOX-404."
• Part III: "Quantifying the huge amount of wasted IT audit effort in SOX-404"

In this article, I will share a cautionary tale of how the problems discussed in the previous articles result in such horrible outcomes.  More specifically, how the inability to scope correctly the IT portions of SOX-404 led to tons of firefighting by information security and auditors and wasted effort, all at the expense of more important things that they should have been focusing on.

Like keeping the organization secure.  As opposed to trying to achieve compliance of a misguided and mis-scoped audit.  Which is at the heart of the PCI problem.

How Bottom-Up SOX-404 Auditing Happens: A Cautionary Tale

How do these audit horror stories happen?  How do intelligent people in management and audit end up auditing things that don't matter?

We studied this problem when we wrote the "Visible Ops Security: Achieving Common Security and IT Operations Objectives in 4 Practical Steps" book.

This is an excerpt from the book, in a chapter called "A SOX-404 Cautionary Tale":

When external auditors started testing against SOX-404 in the first year, IT findings represented the largest category of findings, totaling more than the combined findings in the revenue, procure-to-pay, and tax categories.  It’s estimated that as much as $3 billion was spent in the first year of SOX-404 to fix IT controls to remediate these findings. Ultimately, most of these findings were found not to be direct risks to accurate financial reports and did not result in a material weakness.  This is because they followed a bottom up versus a top-down, risk-based approach.

Consider the following scenario: The SOX-404 team asks for an information security review of a WebSphere server that runs the materials management systems. The review shows that it’s a custom WebSphere application running on a cluster of servers that is connected to a clustered Oracle database. We then locate the firewall and determine the segment it’s on.

An information security review of the materials management system uncovers:

• Numerous ghost accounts
• A lack of password aging policies
• Critical vulnerabilities in the Java code, including cross-site scripting issues in the HTML
• Vulnerabilities in the Oracle database configuration
• Firewall rules that are suspect and need further investigation

Our task list keeps expanding and the internal auditors are showing up next week. We decide to focus on the operating system level, and our suspicions prove to be correct: The operating system is not running at the latest patch levels. We add this to our list of corrective actions that need to be taken right away, and start talking with the owners of the operating system, database, and application, and even the firewall team.

When the internal audit team comes in, we are candid and transparent about all the issues. Management is informed about the risks, and soon 50+ people are working on all these issues, dropping other high-priority projects to get these issues fixed in time. After all, the argument is made, these issues should be fixed eventually because they do represent risk.

But there just isn’t enough time. The external auditors come in and find all of these issues. They start preparing a management letter stating that the integrity of the IT general control process (ITGC) environment cannot be substantiated.

As a result, more high-level meetings take place, and the financial people start to argue that the ITGC issues really can’t lead to an undetected financial reporting error. They pull out the “nine firm document” and use something called “Chart Three” to make the case.  Then management and the CPA firms argue back and forth about the linkage, and management starts bringing in all the business experts to show that a failure in the ITGCs for this system could not result in inaccurate financial reports.

Finally, the owner of the materials management business process determines that even if the application, database, operating system, and firewall were compromised by a person trying to perpetrate fraud, the attempt would be caught by a daily financial reconciliation between the materials management inventory report and another report from the ERP system.

Given this new evidence, everyone agrees that reliance is actually placed on the daily financial reconciliation, which would catch both fraud and errors. Furthermore, they agree that reliance is not placed on the IT system and the supporting ITGCs.  So, the IT systems are out of scope, and no further IT testing is required.

Everyone is relieved. As the information security practitioners, however, we struggle with this unsettling question about why we went through all this trouble if our efforts were not required to substantiate the accuracy of financial statements. Furthermore, we wonder if all the “good hygiene habits” are actually important and can be justified.

To be clear, it’s not that the downstream manual financial reconciliation control is the best control possible. The point is that if the scoping of IT controls were done correctly in the first place, the only control weaknesses that we would have tested and found would be those that truly jeopardized accurate financial reporting. Instead, we found control weaknesses on systems that were out of scope, and then kept digging needlessly.

Next up, I'll discuss the GAIT vision that was realized in February 2007, when the GAIT guidance was finally published.

Mobilizing The PCI Resistance, Part III: Quantifying The Huge SOX-404 Problem...

Previously, I wrote in Part I about "Upset about the subjectivity and ambiguity in the PCI DSS compliance standards? My #BSides submission on the answer...", and in Part II, I wrote about the problems that management and auditors faced in 2005 and 2006 for the IT portions of SOX-404.

In Part III of this series, I will continue walking through the January 2006 GAIT summit slides, and show you the objective evidence that there was a real problem that needed to be solved, and our vision of what the solution was.

The Damage Of Bottom-Up Auditing

Actually, let me rewind a bit.  I didn't realize it at the time, but in 2005, I heard a great presentation by Patrick Gunderman that hinted at the magnitude and scale of the SOX-404 IT audit problem. Back then a Senior Manager in the KPMG audit practice.  He showed a slide that blew me away.

In the slide above, KPMG found that "The estimated percentage of deficiencies identified show IT controls accounting for the most (34 percent), followed distantly by revenue (13 percent), procure to pay (10 percent), and fixed assets (10 percent)."

What this means is that auditors were spending time digging around IT infrastructure, and finding lots of deficiencies.  Then for each one, management would either have to remediate, or argue with the auditors that it wasn't worth fixing, because an IT control failure would not result in an undetected material error.  Now, if the Enron and Worldcom failures were caused by rogue DBAs, then maybe this level of scrutiny was warranted.  But, something definitely doesn't seem right...

It’s estimated that as much as $3 billion was spent in the first year of SOX-404 to fix IT controls to remediate these findings. Ultimately, most of these findings were found not to be direct risks to accurate financial reports and did not result in a material weakness.  This is because they followed a bottom up versus a top-down, risk-based approach.

At the January 2006 GAIT Summit, we had publicly traded companies present how this problem was affecting them and their need for a better way.  Universally, they talked about the huge IT audit effort and fees associated with SOX-404 that was totally disproportionate to the risk.

These companies included (in no particular order), Goldman Sachs, Marathon Oil, Microsoft, Hewlett Packard, Chevron Phillips Chemical, Business Objects and so forth.

One of the most compelling data points was presented by Fawn Weaver at Intel.

This slide shows how 50% of the SOX-404 compliance effort was IT-related, which was generating almost 80% of the findings.  Yet, none of those findings represented a real risk to an undetected material error.  (So again, why was all that work performed?  It shouldn't have been.)

In my next post, I will write about how bottom-up auditing happens and our vision behind GAIT.  Next, I will write about the politics of GAIT, and how we assembled the constituencies, what was in it for them, and how I learned to use one of the most valuable tools in my career.

All of this helps (at least, in my mind) inform the PCI problem statement, as well as the strategy of how we can solve it.