Add Post   Gallery
This is a community portal. Sign up on the left and start posting about analytics and visualization of security data.



Applications of Mind Mapping automation in the analysis of information security log files

I have started developping applications based on Mind Mapping to help professionals in the analysis of security log files.

My first example, to start with something easy, has been creating a program to analyze Endpoint Protector log files.

Here is a presentation about this issue.

I plan to create more complex applications in case they can be useful to the information security community.

I would like to get feedback about the first impressions about the possibilities of Mind Mapping in Security Visualization.

Visualisation of Apache log data stored in Logstash with Gephi

Visualisation of Apache log data stored in Logstash with Gephi

Here is an Apache logfile visualisation created in Gephi using data extracted from logstash and elasticsearch.

See blog post at:

To see the animated version check out

Visualizing and Cleaning Traffic Logs - Hands On Guide

I have spent quite a bit of time with the VAST 2013 Mini Challenge 1. The given network traffic log is interesting, but bears some challenges. One of them is the ominous source/destination confusion where the network flow collector didn't correctly record the client side of the connection as the source, but recorded it as the destination. That will create all kinds of problems in your data analysis and you therefore have to fix that first.

I wrote a blog entry on Cleaning Up Network Traffic Logs where I am going step by step through the network logs to determine which records need to be turned around. I am using both SQL and some parallel coordinate visualizations to get the job done. The final outcome is this one-liner Perl hack to actually fix the data:

$ cat nf*.csv | perl -F\,\ -ane 'BEGIN {@ports=(20,21,25,53,80,123,137,138,389,1900,1984,3389,5355);
%hash = map { $_ => 1 } @ports; $c=0} if ($hash{$F[7]} && $F[8}>1024)
{$c++; printf"%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s",
$F[15],$F[14],$F[17],$F[16],$F[18]} else {print $_} END {print "count of revers $c\n";}

Read the full article here: Cleaning Up Network Traffic Logs

If you want to know how to setup a columnar data store to query the network flows, I also wrote a quick step by step guide on loading the network traffic logs into Impala with a Parquet storage engine.

Security Visualization Workshops in Dubai and Seattle Offered by World’s Leading Security Visualization Expert


Big data and security intelligence are the two hot topics in security for 2013. We are collecting more and more information from both the infrastructure, but increasingly also directly from our applications. This vast amount of data gets increasingly hard to understand. Terms like map reduce, hadoop, mongodb, etc. are part of many discussions. But what are those technologies? And what do they have to do with security intelligence? We will see that none of these technologies are sufficient in our quest to defend our networks and information. Data visualization is the only approach that scales to the ever changing threat landscape and infrastructure configurations. Using big data visualization techniques, you can gain a far deeper understanding of what's happening on your network right now. You can uncover hidden patterns of data, identify emerging vulnerabilities and attacks, and respond decisively with countermeasures that are far more likely to succeed than conventional methods. The attendees will learn about log analysis, big data, information visualization, data sources for IT security, and learn how to generate visual representations of IT data. The training is filled with hands-on exercises utilizing the DAVIX live CD.

This class features brand-new material, first presented at BlackHat USA in July 2013. Here is what students said:

"Very good course. The trainer really knows the subject matter and has an incredible delivery of the material."

"Raffy obviously put a lot of time and effort into preparing for this course. Having already read the book, I expected a lot of the material to be a re-hash of what I already saw in the book. I was surprised at how much new material there was to get out of it. Looking forward to applying a lot of these concepts in the real world."

"Raffael did a great job! He knows and understands the subject matter extremely well. I highly recommend this course and instructor."

"One of the best trainings I have ever taken!"

Visual Analytics - Delivering Actionable Security Intelligence

Dates: December 9-10 & 11-12, 2013
Location: Washington State Convention Center
Seattle, Washington, USA
Sign Up Now
Early registration discount ends October 24th!

Network Forensics and Security Visualization

Date: November 3-4, 2013
Location: Dubai, UAW
Sign Up Now

Sample of Tools and Techniques

Tools to gather data:

  • tcpdump and wireshark to analyze packet captures
  • argus, nfdump, nfsen, and silk to process traffic flows
  • snort, bro, suricata as intrusion detection systems
  • p0f, npad for passive network analysis
  • iptables, pf, pix as examples of firewalls

We are also using a number of visualization tools to analyze example data in the labs:

  • graphviz, tulip, cytoscape, and gephi
  • afterglow
  • treemap
  • mondrian, ggobi

Under the log management section, we are going to discuss:

  • rsyslog, syslog-ng, nxlog
  • logstash, graylog
  • commercial log management and SIEM solutions

The section on big data is covering the following:

  • hadoop (HDFS, map-reduce, HBase, Hive, Impala, Zookeper)
  • search engines like: elastic search, Solr
  • key-value stores like MongoDB, Cassandra, etc.
  • OLAP and OLTP

About the Trainer

Raffael Marty is one of the world's most recognized authorities on security data analytics. The author of Applied Security Visualization and creator of the open source DAVIX analytics platform, Raffy is the founder and ceo of PixlCloud, a next-generation data visualization application for big data. With a track record at companies including IBM Research and ArcSight, Raffy is thoroughly familiar with established practices and emerging trends in data analytics. He has served as Chief Security Strategist with Splunk and was a co-founder of Loggly, a cloud-based log management solution. For more than 12 years, Raffy has helped Fortune 500 companies defend themselves against sophisticated adversaries and has trained organizations around the world in the art of data visualization for security. Practicing zen has become an important part of Raffy's life.


Log Analysis

  • Data sources
  • Data Analysis and Visualization Linux (DAVIX)
  • Log data processing

Log Management and SIEM

  • Log management and SIEM overview
  • Application logging guidelines
  • Logging as a service
  • Big data technologies


  • Information visualization history
  • Visualization theory
  • Data visualization tools and libraries
  • Visualization resources

Security Visualization

  • Perimeter threat use-cases
  • Network flow data
  • Firewall data
  • IDS/IPS data
  • Proxy data
  • User activity
  • Host-based data analysis

Juniper FW / Syslog-NG / Afterglow on a Apache

Juniper FW / Syslog-NG / Afterglow on a Apache

Web interface with a syslog_NG connect on a Juniper Fiwerall. A simple tool in order to check the scan, the drop and accept rules on real time (more or less -))


NMAP output plugin with afterglow

NMAP output plugin with afterglow

I have made a nmap output plugin script (perl) with afterglow.
Sometimes, It is better to have a globlal view (IP and tcp open port : pink for port's reserved, blue for tcp high) like my image then a xml report made by nmap.


How Analytics Enables Security Visualization - Or Not

I was greatly honored when I got an invitation from the Conference on Knowledge Discovery and Data Mining (KDD) to give a talk about data mining and cyber security.

Knowing me, you might be able to guess the topic I chose to present: Visual Analytics. I am focussing on not the visualization layer or the data layer, but on the analytics layer. In the presentation I am showing what we have been doing with data analytics and data mining in cyber security. The presentation starts out with an overview of what security is and what our data looks like. While I show a few examples for different areas in cyber security, I am mainly highlighting problems and challenges we have been facing within these areas with regards to analytics and data mining.

The presentation has 5 parts:

  • Cyber Security - Lay of the Land: A quick introduction to the information / cyber security field.
  • Data Mining in Security: For the data scientists out there, how does security data look like and what are some of the challenges you will face when dong data mining on security data (see slide below).
  • Visual Analytics: This section discusses why is visual analytics a promising approach to the security data problem?
  • Security Visualization: In three areas I am showing examples of visualization that we are using in the security field. I also outline the problems we are facing with the approaches.
  • Challenges: This is a summary of some of the challenges we have in security data analytics. See below.

For each of the six areas in data mining, the following slide shows a couple of challenges that one will run into when trying to apply them to cyber security data:

Security Visualization Challenges

At the end, I am presenting a number of challenges to the community; hard problems that we need help with to advance insights into cyber security of infrastructures and applications. The following slide summarizes the challenges I see in data mining for security:

Definitely not a complete list. Please comment and add other challenges! If you have any suggestions on solving the challenges, please contact me or comment on this post as well!

DAVIX Survey - Your Input is Needed

We are preparing for the next DAVIX release and have constructed a survey to get your input on the tools you would like included, the delivery mechanism, and general information on your security visualization needs. Your participation in the survey would be greatly appreciated!

The survey is located at

We would like to collect all responses by July 31, 2013.

AfterGlow Slide Deck

AfterGlow to GephiI recently released a short slide deck on AfterGlow.

AfterGlow is a security 'visualization' tool that simplifies the task of creating network graphs. It reads CSV files and converts them into a graph representation based on a set of configurations that the user defines (colors, edge thickness, node sizes, clustering, etc.). AfterGlow is a pretty powerful tool and filly this slide deck summarizes the features and provides a couple of interesting examples of how to use the tool.

These slides will also be part of my Visual Analytics workshop during BlackHat at the end of the month. There are still a couple of seats available!

VizSec 2013 - Paper Deadline Extended, Poster Deadline Announced

The 10th Visualization for Cyber Security (VizSec) will be held in Atlanta GA, USA on October 14, 2013 in conjunction with IEEE VIS. VizSec brings together researchers and practitioners in information visualization and security to address the specific needs of the cyber security community through new and insightful visualization techniques.

The paper deadline has been extended to July 22, 2013 at 5:00pm PDT. Full papers offering novel contributions in security visualization are solicited. Papers may present techniques, applications, practical experience, theory, analysis, or experiments and evaluations. We encourage papers on technologies and methods that promise to improve cyber security practices, including, but not limited to:

  • Situational awareness / understanding
  • Incident handling including triage, exploration, correlation, and response
  • Computer forensics
  • Recording and reporting results of investigation
  • Reverse engineering and malware analysis
  • Multiple data source analysis
  • Analyzing information requirements for computer network defense
  • Evaluation / User testing of VizSec systems
  • Criteria for assessing the effectiveness of cyber security visualizations (whether from a security goal perspective or a human factors perspective)
  • Modeling system and network behavior
  • Modeling attacker and defender behavior
  • Studying risk and impact of cyber attacks
  • Predicting future attacks or targets
  • Security metrics and education
  • Software security
  • Mobile application security
  • Social networking privacy and security
  • Cyber intelligence
  • Human factors in cyber security

We are also soliciting posters. Poster submissions may showcase late-breaking results, work in progress, preliminary results, or visual representations relevant to the VizSec community. Accepted poster abstracts will be made available on this website. Poster submissions are due August 23, 2013 at 5:00pm PDT.

See for the full Call for Papers and additional details.