Graph Exchange

This small experimental project was done for the Shadowserver Foundation. They are a volunteer, Not for Profit organization who deal in the capture, analysis and dissemination of data and intelligence relating to nefarious activity on the internet. Shadowserver provided us with one day worth of data (which was several gigabytes) for us to apply some known techniques, and experiment with some new ones.

The idea of this project was simply to provide some ideas as to ways to represent their massive datasets visually. There's lot of work to go, however here are few early ideas. My favourite is a light-hearted time series visualization in the theme of an old favourite arcade game originally released in 1972 "Pong".

See all of the samples at http://dataviz.com.au/shadowserver/ideas.html

One of many tests with Afterglow, visualizing SSHD brute force logins (yellow) vs source IP addresses (green).

This one shows quickly the IPs that have the most activity (one IP has the most: the yellow explosion in the middle), along with popularly attempted userids, and the IPs which have been attempting the same userids.

I finally got my spam stats up and running. The results are amazing.
Lightyellow = Subject || Red = Sender || Black = Recipient

It is pretty easy to find the one user that appears to get a significant amount of Spam :). If I had to guess, I would say the single subject, large source and large destination likely originate from Botnets?

The results are from Wed and Thurs of last week.

I extracted this image using PDF malware that I got for analysis purpose. By using perl script I filter out the unneeded content and later put it in sctest(libemu tool). The graph created using dot command in Graphviz package

This graph is automatically generated by the Dorothy framework anytime a new malware is analyzed.
It aggregates three different kind of information : 1) the network activity 2) the dns host resolutions 3) the GET / POST resquest
In this way, we can be able to easily define certain activity related to botnet communications.
A quick legend :
Colors :
Green = Services / hostnames
Red = General target
Purple Red = Known C&C ( in this example there isn't any)
Purple = C&C Web target
Light blue = private network host

Shapes:
Circle = Target
Triangle = Source

The shape's dimension represent the network activity related to that node.

I created the image by using Nepenthes' log, later put the country info by using python-geoip. Finally, use Afterglow and graphviz to illustrate them.

click here for the full picture

The effectiveness of intrusion detection systems, which apply misuse detection, strongly depends on the conciseness and topicality of the applied signatures. Imprecise signatures heavily limit the detection capabilities of the intrusion detection systems and lead to false positives. The reasons for this detection inaccuracy can only to a lesser extent be imputed to qualitative restrictions of the audit functions. Instead, these restrictions must be identified primarily in the signature derivation process itself.

In particular, the derivation of signatures starting from given exploits appears to be a very complex task, which comprises identifying the traces in the audit data that are left behind by an attack and determining characteristic relations of the attack. This procedure requires also a manual audit data analysis. Admittedly, this basic activity is time-consuming, sophisticated, and cumbersome. The main reasons for these difficulties are the flood of very fine-granular information distributed to different sources as well as the non-ergonomic inspection of audit data.

Consequently, abstraction capabilities to extract relevant parts of this data richness are crucial, but common tools for audit data analysis do not tackle this issue. Abstractions, i.e. the goal-oriented accentuation of relevant relations between audit events, while concurrently hiding irrelevant data are a key aspect to support the security officer during audit data analysis. Another key aspect impacting the time requirements of the analysis is the representation of the data to be analyzed. Typically, a textual representation of audit data is used, which only inadequately allows to illustrate relations between audit events and thus is suboptimal for providing a holistic view on system behavior. Unclearly arranged representations are irritating and lead to wrong assessments and conclusions. These drawbacks can be remedied by using a graphical multi-dimensional representation of audit events.

We developed the tool ADO for three-dimensional representation of audit data that can be explored interactively. The user can create arbitrary views on the data and can study and visualize relations or dependencies of the data. Furthermore, the tool ADO is a part of the signature development tool, which supports the knowledge transfer from identified attack relevant relations between audit data and the actually signature modeling.

The current version of ADO supports BSM (Solaris Basic Security Module) audit logs as input data. Our ADO tool consists of the three components sensor, the analysis and transformation component, and the presentation component. The sensor transforms BSM audit events into a common data structure and provides the data to the analysis component. The analysis component allows the user to define metrics and to adjust particular abstraction parameters. These settings control the quantitative analysis which is followed by a space-specific transformation. The resulting three-dimensional virtual audit data world is turned over to visualization component, which offers the user visualization and interactive exploration capabilities.

The picture shows the single stages of an exploration of an attack on a Solaris system by using ADO. Starting from the picture in the upper left part the signature engineer explores a set of audit events and identifies and visualizes attack relevant relations in these events. The picture in the lower right part shows our SEG-Tool with the audit data visualization tool ADO and the other signature modeling components.

You can find more info at Troyak-AS and Peer activity blog entry

I used a perl script to convert syslog Symantec A/V logs to CSV files and loaded the data into Advizor Analyst. This type of graph shows interesting re-infection patterns for individual hosts (horizontal lines), signature updates following malware blooms (vertical patterns with the same colors) as well as others.

Equilibrium Networks' visual network traffic monitoring software (for background information, see http://www.eqnets.com) has successfully passed our internal tests, so we are packaging a Linux-oriented beta distribution that is planned for snail-mailing (no downloads--sorry, but export regulations still apply) on a limited basis before the end of the month. The beta includes premium features that will not be available with our planned free/open-source distribution later this year, but at this early stage we will be happy to provide a special license free of charge to a limited number of qualifying US organizations.

Participants in our beta program will be expected to provide timely and useful feedback on the software, e.g.
• filling perceived gaps in documentation
• proposing and/or implementing improvements
• making feature requests or providing constructive criticism
• providing testimonial blurbs or case studies
• etc.

The software should be able to run in its entirety on a dedicated x86 workstation with four or more cores and a network tap (though you may prefer to try out distributed hardware configurations). If your organization is interested in participating in our beta program, please include a sentence or two describing your anticipated use of this visual network traffic monitoring software along with your organizational background, POC and a physical address in an email to beta [at (same domain name as our website)]. DVDs will only be mailed once you've accepted the EULA. Finally, bear in mind that beta slots are limited.