Pkviz Packet Visualization Animator Source Code Available

All -

I finally decided to put the Xcode project and associated source for pkviz up for free download and license it under GPL v3.

I’ve created a google code page for it HERE.

You can grab a stand alone zip of the source/project HERE.

(I’ve never used SVN before, so what’s up at the google code page might periodically be fubared, so you might want to start with the zip)

Feel free to download, comment, and please -contribute-. This was my first Objective-C app and first Xcode project, so if it’s a mess…well…deal or help? :)

Just remember the google code page if you want to post some updates or questions.

I’ve also made some haphazard notes to help people understand the code:

—–

The aquireData class handles reading the tcpdump text file. It uses Core Data to store the data. If I had to do it over, I wouldn’t have used Core Data…but it is what it is.  You can find the data model by double-clicking pkviz_DataModel under the Models folder in the project in Xcode.

pkGraphView is a subclass of NSView that I use to handle the layers, which are done in Core Animation (easy enough to understand). The view has a delegate function (drawLayer) which I handle in the layerDelegate class to deal with drawing the paths for each layer.

Everything else is handled by transformData – it’s pretty much my controller.

Rough flow:

the Load button tells aquireData to parse tcpdump and store in a core data context

The launch button kicks off transform data, which pulls in the data from the core data context, sticks it into an array, launches a thread to pop out individual packets, and then tells the view when it’s read to display another packet.  Everything else stops, starts, adjusts the current packet referenced, or aids this animation loop process.

The main array of packets in transformData is bytepakposSet.  It is an array of packet arrays. packet arrays contain arrays of bytes with 2 values in them: bytevalue, and byteposition

so, if you wanted to access the third packet in bytepakposSet and see what the byte value of the first byte stored is, you’d do:

[[[[bytepakposSet objectAtIndex:2] objectAtIndex:0] objectAtIndex:0] intValue];

if you wanted to get the byte value and position returned in an array:

[[bytepakposSet objectAtIndex:2] objectAtIndex:0]

Core Data doesnt return objects in order, so you dont know ahead of time what order the bytes are in the packet, youll have to sort them by position in packet first. You can find position:

[[[[bytepakposSet objectAtIndex:2] objectAtIndex:0] objectAtIndex:1] intValue];

Data Visualization Art Prints Available for Purchase

I guess once I get going, I keep going for awhile. Recently, I put up some T-shirts for sale which use my art for designs. However, after a few years of showing them, I also wanted to get some of my data and security visualization art available as well and, yesterday, I finally did it.  You can click here to go to the store:

Data Visualization Art Prints

Some of these don’t look quite as surreal or “clean” as other data visualization art, but that’s because I’m very interested in the specific cross-section of usability and “prettiness” in the aesthetics of images: The place where what makes them useful is also what makes them attractive. Finding that line, in my mind, is what makes them “art”.  One could make some really cool looking images out of most semi-structured data, but it would cease to be useful. The ones here retain their function to security and data analysts while, at the same time, being attractive pieces.

If you’re interested in other security visualization information, try secviz.org

Packet Visualizer/Animator DONE! (ish) and Tool Posted for Download

Whew. I can relax.

For the past 2-3 months, I’ve been working on my first real Objective-C project (my iphone app is still going, it just took a back seat to this): An application that will read tcpdump output and animate the packets over time using their inherent byte / packet structure

And now…it’s up and in beta-ish quality. (Meaning it works, though some error checking and minor features arent quite where I want them.)

You can download it here for free: http://sintixerr.wordpress.com/pkviz-packet-visualizer-and-animator/

See it in motion here:

This project was important to me and has been a long time coming. I’ve wanted to write a packet visualizer since I first started working with data viz 5 or so years ago at NetSec and was using Advizor. That tool cost thousands of dollars per seat, didnt really animate (at least the way I needed), and only parsed CSV or databases. The free tools – like GnuPlot, just weren’t up to the task at all.

I also wanted something that could plot out data in interesting, pretty ways for some art projects I have in mind.

So, I originally started this time around on a quest to write a short python parser for tcpdump ascii hex output to put into <some generic viz tool> just to get started…but somehow I ended up writing a full-fledged visualizer (my first GUI project ever, I might add!). The learning process was a blast – I feel like I’m a much better coder for it – and I’ll be able to extend/expand on this to use for other art and security projects that are on my plate or are coming up.

I’m pretty excited about it. To see this finished through after years of whining to myself about it, procrastinating, and genuinely not having enough time, is pretty awesome. I’ve even already created a couple of cool shots that I’m happy to call “art” (granted, there is some photoshop processing here, but they’re both true to their originals!):

Anyway, Mac Users, check out the tool and let me know what you think!

Art and Security: Data Visualization’s Lessons for Enterprise Security

If you’ve read some of my recent posts here, you’ll have seen that Im back working on creating data visualization pieces as art.  In the process of making these,  I was reminded again of the relationship between art and security and its practical implications for enterprise security efforts that literally dictate success of failure. Bear with me as I walk through the art piece first and then arrive at the security observations :)

First, to work, art has to have a solid concept. You might accidentally create a piece that’s appealing on some level if you just throw paint at canvas, but you probably won’t repeat that success often and observers will understand this.

Taking that into the realm of data visualization, you can make all the pretty graphs you like, but unless you do some leg-work ahead of time and massage the data into shape, they’ll be of little use and only may accidentally be visually appealing in a way that let’s you intuitively grok it.   (I think this is philosophically similar to some of what Tufte teaches, but I don’t remember for sure.)

For example, if I wanted to (as I did) visually represent the stimulus bill in a meaningful way on screen at once, I could really just use a microscopic font…or turn the whole thing into a jpg and resize it to fit on screen. But what would that accomplish? It would just be mush.  We wouldn’t have identified or accounted for inherent structural properties that we needed to keep to preserve order. We also wouldn’t have separated the wheat from the chaff – useless information would hide useful information. And we wouldn’t have manually added linkages between data points that would help us draw meaningful conclusions visually to account for a loss of resolution in individual words.

What would work, instead, is to turn (as I did) the Stimulus Bill into columns of useful information. You could convert the free form english structure of the Bill into a tabular format and add meta data about the text that I wanted to see in the visuals.  You could add line numbers, position in sentences, group words by sections of the document and add word counts, etc. All this would show up visually and present a much more useful visualization that would also, because of the new more conscious conceptual structure, be more appealing to look at.

So what does this have to do with security? Everything.

Recently, much has been made of the new SANS CAG control list. Basically, this is a list of “best practice” security measures and controls that, if properly done, will make the most impact in securing organizations. Where’s the problem? The problem is that none of these are new (except WiFi). They’ve all been around longer than I’ve worked in the field (7ish years) and probably much longer than that. Everyone who works in security knows them.  Most CTO’s, CIO’s, and CISO’s will probably not be unfamiliar with them. But yet, they’re either not implemented or, more often, they just don’t work.

If these really are best practices (and they are), but yet they’re not working, where’s the disconnect? I think it’s lack of structure. Most organizations do not operate their businesses in a manner that can be secured. There are inherent structural flaws (as in, there isnt any) in the enterprises themselves that conflict with and outright prevent security from happening – just like in art and visualizations.  No matter how much effort or money you throw at the problem, cyber/IT/technical security controls will get you nowhere quickly (if anywhere ever) without a properly run and organized business. What failed cyber or IT security really is, ultimately, is a symptom of failed Operational Risk Management.

If you can’t track assests, if you haven’t identified your key data, if you don’t have clear and measurable business objectives for IT and cyber systems, if you don’t have a clear line of sight between the risk of technical failure to business impact, your security controls -will- fail.

Why? Because an organization run without these things will consistently make poor decisions based on incorrect, out of date, or conflicting information. In other words, you have to build break points into the business to be able to check, measure, and change the the organization at key junctures in order to make good risk-based decisions.  “Risk-Based decision making” get’s bantered about like “moving forward” and “synergies” – but it’s not an empty phrase and it has real, concrete impacts and prerequisites.

Let’s look at a best-case scenario where everyone wants to do the right thing, but there isn’t an enterprise or business architecture in place. Everyone goes through an evaluation of need and risk, pick the right controls, put them in place. Hunky dory, yeah? Well, what happens when a new line of business is added? Nothing to do with security, right? What if the new line is taking critical data that wasn’t exposed by the other systems and making it public inadvertently? Would you know that? If you need to patch critical systems quickly to prevent a flaw, would you know which ones kept your business running? Would you have documented in an easily accessible manner the fact that your manufacturing systems depended on a feature that the new patch – which works just fine on desktops – disables? Etc. Not to mention that your IDS’s depend on this info, your firewalls, your SEMs, everything.  There is relatively little happening on your network that is inherently bad outside of a business context. There are many more (and probably better) examples…but there are two take-home points:

1. Everyone with the authority to make changes to your business needs to be aware of the secondary dependencies of those decisions and how they intersect with security and inform others of changes they make
2. If you try and do this without managed processes and without maintaing and continuously updating the information about the business in an architecture, you’ll fail. It’s too hard, too expensive, and takes to long to keep doing it from scratch. It’ll never be accurate, timely, relevant, etc.

Business leadership at all levels and in many (most?) organizations simply are making bad decisions that affect security.  It’s not that we don’t know, as security professionals, the right things to do. It’s that we can’t express it in terms of business risk and the business leaders typically don’t seem to have the structure built in to affect positive change throughout the organization. Build some good, clean structure with visible break points at critical junctures in your business flow and then security will start to become cheaper, easier, and more effective.

More Stimulus Bill Art: The Stimulus Mountains

Histogram of major words in the US Stimulus Bill. Big Grey blob in the second "mountain peak" from the left represents "Health" in the Bill. You can see it takes a disproportionate place in everything.

The Stimulus Mountains

Originally uploaded by sintixerr

This is a follow-up to a previous post and is philosophically related to this post.

On the subject of these “data visualizations as art”, I’ve been trying to better articulate why I think they’re art and how I’m trying to evolve my process.

What it comes down to is that there seems to be two pieces to developing the visualizations:

1. Choosing the right structure and things to measure about the text or data…what makes sense to compare to what. How do you reduce the noise and non-dependent variables? Each type of text you’re measuring and each circumstance has different relationships. There is a lot of science to this part, but it’s not completely predicatable. There is art.
2. How do you visually best enhance and needle out the important details, contrast between points, etc so that they can be “seen” in the noise that doesnt matter? This is all art. Understanding how color, shape, contrast, etc all work together and how to use all of those to present a dense amount of information without being overwhelming is tricky and depends on the skill of the one creating it…

It’s my belief that playing to what we understand as people’s abilities to process and comprehend aesthetics in art involves exactly the same techniques and takes advantage of the same aspects of peoples brains/senses as good visual data analysis. So, if you’re doing data analysis, you start out figuring out #1, and then move to #2 based on #1.

What I was trying to do with these stimulus images – and the last of my security visualizations – was start out with concepts of what I’d like for #2 (how they would “feel”) and then figure out what I needed to do in #1 (massage the data) to get there…while still remaining true to the underlying information.

Next up (and once I learn more Objective C), I’m going to try and read in the stimulus bill to Quartz Composer and combine my recent interactive/music visualizations with the Bill. We’ll see if that goes anywhere interesting. :)

Also, Artomatic returns to DC this year. I very well may be displaying this stuff there when it comes around. This or the music/webcam visualizations.