TimeWalker; Product summary
tools for visualising huge amounts of log data
1. Introduction
Many systems produce huge amounts of timestamped data (events) like logs from systemcalls, time-series from network monitoring or transactions from database-applications.
In practice eventdata is often thrown away without any inspection. Some of the main reasons are: waste of resources, poor dataformats, non-scalability of traditional tools, lack of an adequate visual instrument.
However, throwing away eventdata unseen implies losing essential information needed to discover cause-effect relations within (un)wanted or (un)expected systembehaviour. TimeWalker is a tool that makes preservation and disclosure of historical details contained in eventdata attractive and feasible.
The implementation and user-interface are made very flexible and portable by usingwxPythonand C. The first release of TimeWalker will become available(under a GPL-licence) in november 2001 for Win32 and Linux. In this release TimeWalker will work smoothly for about 500000 records in memory that represent individual events or aggregated events collected from much larger (Gb) datasets.
2. Data handling
TimeWalker unifies arbitrary eventformats into a format that enables a much better performance for persistent storage, aggregation and transformation than can be obtained by using a (traditional) database.
Aggregation is the process of compressing arbitrary eventlists into a fixed-time interval sequence containing a single composite record in each interval. With user-specified expressions important correlations can be preserved during aggregation. Aggregated records can be transformed with user-specified expressions into values to plotted.
The clean syntax and semantics of Python is used for all expressions at the user level. Some specific internal techniques are used to improve the performance of the produced byte-code drastically.
3. Visualizing techniques
TimeWalker uses an innovative technique for information-visualisation along the time-axis that enables simultaneous presentation of context and detail of eventdata in a range from 40 years down to 5 minutes. The technique is based on a sliding hierarchical ZoomLens that shows a bundle of multiple beams with predefined (quarter, week, day, hour, 5 min) time-scales. The zoomlens can be shifted by hand or be started as an animation.
The graphical user-interface as a whole is carefully designed for quick pattern-recognition by a regular user. Each part has a fixed place, there is no scrolling, the information density is high, scaling and coloring is automatic, and there is (almost) no static and redundant (textual) information.
Apart from the graphical mainwindow there are also frames for textual browsing and manipulating configuration data, metadata, raw data, documentation, and even (parts of) the reflective running environment. All textual navigation is based on data-driven tree/table-grid combinations.
Graphical and textual visualisations are both interactive and scale up with realtime performance for very large datasets.
4. Usage
There is a general datacollector, with derivations available for some common dataformats like syslog. Experience showed that creating and testing a new collector can be done within one day.
The use of expressions for aggregations and transformations on itself is not complicated, but making the right choices requires domain knowledge as well as some experience with the resulting visual effects.
TimeWalker is supports visual datamining of huge amounts of non-filtered eventdata. It can be considered as a multi-focal looking glass complementary to the limitations of the usual spreadsheet way of (statistical) datareduction.
About the authors
Theo de Ridder has walked around in the software-engineering landscape for more than 30 years. His current interest is painting enduring and aesthetic software patterns using Python as a pencil.
Pim Buurman is an experienced programmer on Unix platforms. He enjoys mostly problems that are hard to solve.