During the last security incident that I worked on, I needed to grind through 20gb of log files looking for any odd log lines that would indicate the point where the bad guys got in. If I had done it manually, I would still be looking at log data. Instead, I built a tool that converted logs into pattern templates and looked for templates that I had never seen before. This allowed me to zero in on just a few hundred log lines out of all the data.
Templater is a small and fast log processor that provides simple artificial ignorance capabilities. You use the tool to process past log data and store templates that represent normal log line structures. You then run the tool against current or target logs and all normal patterns are automatically ignored. The parser is fast and capable of processing millions of lines per minute. For weblogs and firewall logs, I average 9M lines per minute on a 2GHz x86 machine running *NIX. The template strategy was originally proposed by a friend of mine in 2003 who later built a tool called never before seen (NBS) which also provides artificial ignorance for arbitrary text data as well as text structures.
I am optimizing and testing a new QtWidget I have created for force directed graph layouts. Currently it will only utilize the CPU but I am working on a CUDA implementation as well. When release the widget will allow developers to place force directed graph layouts in their qt applications. Additionally this will be included in a new version of inav hopefully out soon. More images on my flickr page:
I had created two visualization under the scope of GSoC 2011 Honeynet Project. The main aim was to create a 3D mesh with heat map tiles on a world map. The first visualization is a quick proof of concept though the second one is created by using Processing visualization framework.
Please check the documentation site to get detail information and leave me your feedback. The site includes links to demo sites also.
After re-implementing some parser code in difftree with bug fixes and improved bounds checking, I have ported those changes back to quickparser. So, I now present the new and improved quickparser v0.5 with less bugs and more bounds checking. Re-energized after touching the code, I am working on the template based version, which will not be limited to syslog style, key=value formatted logs.