(This post is significantly “Draft”. I’m just out of time to edit it more and I figured “sooner” was better than “later”)
As many of you know, I attended NIST’s first “working” workshop to develop a National Cyber Security Framework (CSF) last week. This post is more of my own critique/comments on the process than a summary, but I’ve included some structural commentary as well below.
So, first thing first: thank you NIST, Facilitators, Participants, and the White House. This is a huge opportunity for us all to get our “stuff” together on a very short timeline in a process that has more than it’s fair share of detractors. The fact that so many(400+? What was the final total?) came together to work is impressive and encouraging. I’m happy and honored to be able to provide my $0.02 (Ok, I do keep hijacking the #nistcsf hashtag, so maybe $0.04).
NIST: National Institute of Standards and Technology, part of the Department of Commerce; A solid organization that does good work. One experienced with their products can see why the White House tasked them with facilitating the development of a National Cyber Security Framework (CSF). They are good at getting people together, soliciting input, and putting it all together nicely. Unfortunately, what I am not sure they are good at is what is needed most – the ability to break new ground. (Although I’m happy to be wrong here, I haven’t seen it myself. Correct me if I’m mistaken?). It’s built into the name…the word “Standards” implies something proven, known, and defensible. The (collective) cyber security disciplines, however, are still the antithesis of those. We have yet to regularly build secure systems and so I fear that an organization looking for *effective* “standards” in this area at this time may be set up for failure. Still, there is time.
Workshop Mechanics: NIST handled the mechanics of gathering input nicely. We were organized into 8 groups of people who rotated (in the same groups in order to encourage consistent dialogue) through 4 different topical tracks (derived from RFI responses)over the course of three days. The tracks were (and I’m summarizing): Threat Management & Info Sharing, Maturing Cyber Security, the Business of Cyber Security, and Cyber Security Dependencies & Resiliencies. The facilitators (including Bruce Potter from the Shmoo Group) by and large did a good job eliciting input from us, even though they had only been engaged a short time before the workshop and did not all have experience in the national critical infrastructure dialogue. The real limiting factor from a facilitation standpoint was not the facilitators, but NISTs philosophical process approach to meeting the executive order’s requirements.
Workshop Philosophy: Early on, I (and others) were previously critical of a “lack of rails” provided to participants to structure the dialogue (and I still am, but I’ll hit that later), but I spent some time time talking to both facilitators and NIST staff and feel more comfortable that I understand what they are doing and why. Their perspective is that they need to gather all the “pieces” of the puzzle from industry first without applying any inherent structure and then, in later workshops, analyze all of those pieces to find and useful underlying structure or concepts in the data, apply those to the framework, and then fill in the meat. In many situations, this is a good idea and I understand why they are doing it. The biggest takeaway here is that NIST thinks we shouldn’t get too bent out of shape by or read too much into the questions being (or not being) asked at this stage. The insight, creativity, and thinking will come later (according to them). In other words if you think (as Russ Thomas also does): “Clearly, most ppl running #NISTCSF process don’t see innovation as essential. Their focus is curating ‘best practice’”, then you are right at this moment in time…but perhaps also wrong in the long term.
Workshop Participants: Everyone-ish. Heavy on electric, industry association, academia, and vendor perspectives though. Low representation from other sectors. I’m wondering if (really, can someone answer this?) all the other sectors are busy supporting DHS Executive Order work like the CIIDWG (Critical Infrastructure Identification Working Group) instead of the NIST Framework process. I’m not sure why they would – the efforts are complementary – or why electric chose to show up – but there were clear biases in industry. Is NERC handling the other EO work on behalf of electric and so the only way for asset owners to get involved in the process is via the framework? I don’t know.
Suggestion for NIST: Add a “class” ahead of the next one so that everyone participating has the same background in what’s going on. Everyone came into this from wildly different backgrounds on “critical infrastructure protection” and it took a long time to normalize them (which never actually completely happened).
Process Observations: The process went “ok”. The main observations I have are related and center around dialogue maturity. First, the same issues were brought up in every single track – whatever the topic. This denotes the participants’ fundamental lack of a structured, coherent model of the problem space in their minds. The dialogue did *improve* over the three days as people (including myself) got their hot topics “off their chests” and began to really think about what they were being asked. But, in my mind, without appropriate structural/conceptual rails, it would take *months* of this before a group this large got past their pre-conceptions and tribal knowledge and actually got down to truly productive “thinking”. I understand why NIST did what they did, but I think time will show it to have been a mistake.
Observations On & Suggestions For Content: For the most part, responses hit a ton of different topics and I think it’s out of scope for me to comment too much on the actual discussions. Others – and obviously NIST – will do a better job of that than I would. However, I thought three items were of interest:
1. “Lensing”: For those of you who have seen my SOURCE Boston 2013 presentation from a month ago, you’ll be familiar with the concept of “lensing” cyber security into focus for different audiences. I had never heard of or used the term before until Ms. Jen Giroux and I talked about it. What I found interesting was that the term was used by NIST staff to describe how, at the end of the process, there would not be some final product as much as a bunch of information “lensed” for different audiences. Maybe it’s just coincidence, but maybe all my mouthing off has had some positive impacts. :) Hope so.
2. “Reasoning Tools vs Standards”: Is the framework going to just have static standards and controls, or will it provide reasoning tools (or be a reasoning tool) to help “make things better”? There was no clear answer and dialogue seemed to suggest a little of both.
3. “Defend vs Improve”: Few participants or facilitators seemed to talk much about about changing the strategic relationship between defenders and attackers. There was little acknowledgement that simply “defending” against the bad guys will result in ultimate failure – a) they will simply out-resource us and b) As we add complexity, unless we improve processes, we will make more and more mistakes implementing security.
That said, and in support of the lensing idea, I suggest something similar to the following diagram be used in the upcoming feedback evaluation process to help organize answers:
Short Topical Notes:
1.Several participants brought up “Control Systems vs IT” and ask if “ICS will be covered”: Of course the NISTCSF will deal with control systems issues. :) It’s being developed (among other things) in support of the executive order’s outcome-based perspective on cybersecurity (which, also of course, would naturally consider control systems since their failure tends to create significantly bad outcomes). The level in the security stack at which both the EO and the CSF operate at is higher than the level at which the distinctions between IT/ICS happen and so are inclusive of both without the need to distinguish between the two in policy statements.
2. Several people also wanted to limit dialogue to “cyber only” and ignore “business” or other non-core cyber considerations completely. I think these suggestions show an unfortunately poor grasp on the problem space -pThere is no such thing as “cyber security only” and our core failings have been in the broader set of activities beyond core cyber.
3. Lots of people said “lexicon”. Lexicon is only a component of what we need: A class-level Ontology. Think of an ontology in this sense as a high level set of relationships between concepts that fit together to define what we mean by cyber security. A lexicon contains the words and definitions we use to implement that ontology in real life.
Summary: Good Process, Missing several key concepts (from participant input and facilitation), I hope they get included later.
Poor Problem Identification, A Boats vs Airplanes Parable: A community agrees that they need to travel across the ocean better than they have been. They’re experienced in building boats, but they feel they need to do better because many boats sink and there is an overall feeling that they’re too slow. So, they get all the boat builders together, lay out all of the best practices, and then try and figure out how to build the best boat possible. Unfortunately, at the outset, they did not identify “too slow” as being “more than 24 hours” – there’s no way a boat will *ever* cross an ocean that fast. Unfortunately, the best solution – heavier than air flight – is not well known. Few in the community are aware of the concept and all of the questions being asked center around boat-building any way. Their efforts are doomed to fail – no one will ever be happy with a boat – but they don’t know it. If instead, however, the community had realized that the existing models – no matter how well perfected – would never meet their requirements, perhaps they would have gotten the boat builders together and provided some rails: “A boat won’t work, we have to get people safely across the ocean in less than 24 hours, but here’s how flight works – apply the same problem solving skills you’ve developed as boat builders to the problem of building aircraft.” But they didn’t. This is how I feel the NIST Framework process is proceeding. We are strategically approaching cyber security “wrong” and continued use of current models will assure that the bad guys will continue to retain the advantage.
Parable Applied: Conceptually, I like the idea of gathering everyone’s thoughts, laying them out, seeing what the relationships are, and then identifying good framework components and structures. The problem here is that over and over again I’ve seen – while helping to shape government perspectives on cyber security, working with an entire sub-sector at once on cyber risk management as a fed and later as a facilitator, and trying to sort out industry confusion on other cyber topics – that our haphazard models for security are neither well understood nor particularly effective. Everyone knows what a saw does, a hammer does, a screwdriver does, but no one has *ever* defined what a boat is, and even then, what we really need is an airplane. The boats we’re building – even the best of them – keep sinking and are too slow. As NIST goes out and gathers data, I hope they realize that the patterns they’re looking for probably don’t exist yet and that any framework stemming from current best practices will only exacerbate the problem, not make it better. This is because if we spend all of our time, resources, and good will trying to build a better boat, there will be little left later for the airplane we need. Making boats flight-worthy (as opposed to airplanes sea-worthy) is very, very expensive and hard.
Parable Plus Scale: I’ve also noticed very little emphasis on the problem of “sustainability over time”, which is a large issue when dealing with current levels of scale and increasing complexity. What I mean here is that many (most?) folks (not all) seem to think in terms of “fixing things now”. Many seem to believe that we just need to get everyone up to a certain level of security *now* and we can go from there. The philosophy of “fixing it now”, though, is actually one of our most serious vulnerabilities. “Fixing it now”, in our current environments, takes a *huge* amount of time, resources, marketing, commitment, thinking, etc. (for organizations or the nation, depending on scope) to get everyone together to figure out what “the best answers” are. Unfortunately, by the time we finish the process to anyone’s satisfaction, the world has changed and our solutions won’t work. What we need are not answers to how to secure ourselves now, but answers to how to make the process of deriving and implementing effective security concepts fast, accurate, and efficient enough to keep pace with the world in a way that a) won’t blow out all of our resources and b) will have a high level of quality in implementation. We need to create tools and cyber/non-cyber environments that will shorten the overall cyber security lifecycle, simplify the process, and reduce errors. The Framework must aim to aid the derivation of answers, not provide them. Better derivation tools will help us figure out what we really need and maybe we’ll learn to build air planes instead of boats. I really wish I had seen more of this perspective from the organizers and the participants.
Final Words: If we do not change our entire approach to cyber security, if we do not learn to dramatically adjust what we do based on failure, if we do not handle the issues of scale/complexity, the bad guys will continue to win. We will run out of money, time, and will if we keep walking down these same paths. I sincerely hope the next NIST Cyber Security Framework development workshops will take these realities into consideration as part of the process. Please read these two posts for further exploration of related topics:
Quick shout out to the people who made the workshop less dry with good conversation and good company: Lena Smart, Andy Bochman, Mike Dahn, Fowad Muneer, and the CMU REP Team. :) Was good seeing/meeting you all.