Add Post   Gallery
This is a community portal. Sign up on the left and start posting about analytics and visualization of security data.

 


 

screenshot of ip traffic in deep node

screenshot of ip traffic in deep node

Youtube video using Afterglow, twopi and Nginx logs.

I attended Visual Analytics Workshop last year at BlackHat and have gotten endless use from afterglow, neato, etc to make interesting visualizations.

Here is a short youtube video I put together, with attack data taken from Nginx logs:



(Music is by a local San Francisco band: Vetiver)

DNS Mapping

DNS Mapping

Over at dnsdumpster.com I created a DNS recon tool that generates a DNS map on the fly using 80+GB of DNS data from the scans.io project. This map is the secviz.org domain.

MyDoom botnet

MyDoom botnet

This graph visualization shows the propagation of malware through a deliberately infected computer network. Twelve machines in the network were infected to see how the traffic spread to other machines. Over 7800 machines were included in the dataset.
All network in a single chart. Yellow links indicate benign traffic; red links indicate traffic with at least 1 infected packet. Nodes are sized by volume of traffic.
Data taken from the MyDoom-A.tar.gz, available here
Image generated with KeyLines.

Botnet activity

Botnet activity

Visualization showing botnet activity geographically. The time bar at the bottom shows temporal trends and filters traffic shown on the map.
Data from http://www.caida.org/data/passive/sipscan_dataset.xml.
Image generated using KeyLines.

Botnet traffic

Botnet traffic

Using a dataset from http://www.uvic.ca/engineering/ece/isot/datasets/index.php, this graph shows botnet traffic between 5000 computers at the University of San Diego. Different colors were used to indicate different protocols. Nodes represent computers and were sized by degree. Edges represent packets, weighted by packet size. Image generated using KeyLines.

Visual Analytics Needs a Strong Data Backend

Visual Analytics, especially the exploration of data requires a scalable and flexible data backend. It is not uncommon that gigabytes, maybe even terabytes of data need to be queried for a specific analytics tasks. Furthermore, the more context around log data is available, the more expressive the data gets and the deeper the insight that can be discovered in the data. How can we gather all that context and combine it with both network-based, as well as host-based data? What are the data access requirements? How can we run data mining algorithms, such as clustering across all of the data? What kind of data store do we need for that? Do we need a search engine as a backend? Or a columnar data store?

I recently wrote a paper about the topic of a security data lake that is a concept of a data backend enabling a variety of processing and access use-cases. A short introduction to the topic is available as well.

Maybe at a later point in time, I will try to address the topic of data science and techniques, as well as workflows to make all that big data actionable. How do you take a terabyte of data and find actual insights? Just dropping that data into a network graph visualization is not going to help. You need a bit more to make that happen. But again, more on that later.

If you want to learn more about how to visualize and analyze terabytes of data, attend the Visual Analytics Workshop at BlackHat 2015 in Las Vegas.

Again, here is where you download the paper.

Linked Graphs Showing DNS Traffic on the Network

Linked Graphs Showing DNS Traffic on the Network

This is a screenshot from a tool called Mondrian where we show network traffic. DNS traffic in particular. The bar charts show the breakdown of sources, destinations, and ports. The parallel coordinate shows all three variables at the same time. The red parts highlight an interesting visual pattern. What is it?

My Laptop Communicating

My Laptop Communicating

This link graph shows communications of my laptop. The big cluster in the middle is Web traffic. The surroundings are all kinds of processes running under OSX. The output was rendered in WebGL. Copyright 2015 pixlcloud llc.

Network Flows - Linked Views

Network Flows - Linked Views

This image shows a network flow graph of flow events. The output is rendered in WebGL through a proprietary tool. Copyright 2015 pixlcloud llc