The LogReport Project and the Lire Package

tools for computer/network log file analysis

Revision 1, 2002/04/03.

This paper gives an introduction to Lire, LogReport's tool for performing an integrated analysis of all your Internet and Intranet Services.

Log files contain the traces of computer activity, and by intelligently analyzing these traces you can learn a lot about the behavior of a system and its users. However, log file analysis is tedious because programs generate a lot of data and tools to report on this data are unavailable or incomplete. In cases where such tools do exist, they are quite often specific to one product, which means that you can't compare your qmail and Exim mail servers. As a result, rotate is quite often the only application dealing with the logs.

1. Lire

The LogReport project tries to tackle the problems as outlined above by developing Lire. Lire is a software package to generate useful reports from raw log files of various network programs. Lire is Free Software released under the GNU GPL.

The package is actively being maintained by the LogReport team, which currently consists of five experienced software developers. The development can be followed live on our CVS on SourceForge. A new release gets shipped almost monthly.

Lire runs on four different Unixen, GNU/Linux included. Since it's written in Perl, porting to different platforms is easy. Lire is shipped as a tarball (autoconfiscated), as an RPM and as a Debian package. A FreeBSD port package is available too.

Figure 1. Lire's Architecture

Lire enables you to schedule hardware upgrades, detect anomalities in usage from services. It can be used as a tool in building a traffic-based accounting system for external customers. It gives insight in who's talking to who, which is valuable for marketing and business planners.

>Lire currently supports log files from

• www (apache, IIS, Boa)
• dns (bind v8 and v9 querylogs)
• firewall (cisco IOS, Linux ipchains ipfilter and iptables, BSD IP Filter, WELF from Webtrends)
• email (Exim, Postfix, qmail, sendmail, Netscape Messaging Server, ArGoSoft)
• print (CUPS, LPRng)
• ftp (ProFTPD, WU-FTPD, MS IIS)
• proxy (squid, WELF proxies, MS ISA)
• database (MySQL)

Lire also supports various output formats for the generated reports: HTML, XHTML, XML, PDF and plain ASCII. Some of these formats support graphical representation of the data.

Lire represents the log file in a DLF file (for Distilled Log Format). This is a simple space-separated line-oriented ASCII file. Each logged event is represented by one fixed-fields line. A service coincides with one well-defined raw log file format. Each service has its 2dlf-convertor. A superservice is a class of applications which share the same DLF format, and which will give the same reports. A Lire report consists of several subreports, which can be displayed in graphical form, or as a table. A lot of subreports (144, as of march 2002) come with Lire predefined, but of course you can define your own reports. A report definition is written in the Lire Report Specification Markup Language; it looks like e.g.

[...]
 <lire:report-calc-spec>
   <lire:group sort="-mail_volume" limit="$domain_to_show">
      <lire:field name="to_domain"/>
      <lire:sum name="mail_volume" field="size"/>
   </lire:group>
 </lire:report-calc-spec>
[...] 

We plan to ship Lire version 1.0 in June 2002. This release will enable the merging of reports stored in a report datawarehouse, in order to solve problems one frequently encounters in complex networked environments. Lire 1.0 will offer industry-strength log file analysis.

Figure 2. A subreport from the www superservice.

LogReport offers an Online Responder service. You can send (compressed) logfiles in email messages to dedicated addresses, like <log@postfix.logreport.org> and get a report back as a response. Optionally, you can anonimize the log before submitting it, using a simple script which comes with Lire. Logs can be submitted via an HTTP file upload interface as well.

2. The LogReport Project

If you like Lire, you can help! We need log files to test our code, and to be able to add support for more services. Of course we also welcome code contributions. Contact us if you would like to share your code.

Finally: Fund us: Funding from the NLnet Foundation which currently enables us to spend a lot of time on Lire will run out in the near future. Other ways to support Lire's continued development are possible too, of course. Contact us if you're interested.