News

EC publishes study on Next Generation Internet 2025 2018/10/05

Bob Goudriaan successor of Marc Gauw 2017/10/12

NLnet Labs' Jaap Akkerhuis inducted in Internet Hall of Fame 2017/09/19

NLnet and Gartner to write vision for EC's Next Generation Internet initiative 2017/04/12

Dutch Ministry of Economic Affairs donates 0.5 million to "Internet Hardening Fund" 2016/12/16

Vietsch Foundation and NLnet cooperate in internet R&D for research and education 2016/09/28

 

Proposal for Lire-2.0 related development

This document can serve as the basic of a sponsorship request to NLnet for more Lire development. It is made of 4 parts. The first one discusses the two different audience of Lire. The second part lists the features for which the sponsorship is requested. The benefits of these features is discussed in the light of the two Lire audience. The third section discusses man-power availability for this development. Finally, the fourth section discuss how the schedule and release management could be done.

Audience for 2.0 development

I think that the Lire audience is made of two classes of users. (As a side note, it will be important to keep this in mind while designing the user survey. At least to try to verify this hypothesis) We could also say that there are two Lire.

Lire - The Log Analyser

This first class of users sees Lire as an application which can generate reports for a lot of log formats. What these people are interested in is:

I think that most of current Lire users fall into this group.

Lire - The Log Analysis Toolkit

This second usage patterns is less concerned about the default reports or the supported log formats but more interested in using Lire as a toolkit. That is a set of tools that can be easily integrated in a custom deployment situation. The people are interested in:

We have few users of this kind, but they are the ones making the most contributions to Lire: Sun, MavEtJu, Noël Bardelot (that's the one integrating Lire into Cyber Sentry). This was also the important use case for the NFB project and future consultation gigs.

Complementary Areas

Of course, there are areas of interest to both audiences. One of these would be performance. It could also be argued that the first use case is a specific customisation of the toolkit. I've selected features that appeal to both class of users or that appealed to the toolkit's users. If the Board feels that it would be a better strategy to focus on the "Log Analyser" case, I could modify that feature list in consequence.

Features discussion

Each feature starts by a discussion on what should be the scope of the implementation. When possible, an initial breakdown of the tasks involved in implementing that features are given.

After that comes an estimate of the amount of development days estimated to implement the feature. The estimate are done without dependencies. That is, it is done by estimating the amount of work needed to add that feature to the current source tree without adding nothing else. This will make it easier for release management as the features can be added in any order.

After that comes a discussion of the benefits of adding that feature. The benefits are discussed from the point of view of the two class of users previously identified.

Finally, risks related to that feature are discussed. If possible, a risk-avoidance strategy will be proposed.

SQLite Based Report Generation

This feature is about re-implementing the report generation component to use SQLite as the query engine. I've sent an email to the development list with ideas on how to implement that feature.

Tasks

Estimate: 15 days.

Benefits:

Risks

It will be best if we start implementing that more complex operator to exercise the usefulness of the design.

Store Configuration

This is about exposing Lire::DlfStore into the user interface. This feature is about making it possible to use lr_config to configure a Lire::DlfStore. This means the following possibilities:

Tasks

Estimate: 10 days

Benefits:

Risks:

Internationalisation

It can be argued that a reporting solution which doesn't support the user's native language isn't of much value. (All web reporting tools out there support reports other language than English). Although making Lire supports other language than English would be a big task (because of the translations involved), making it possible to support other languages than English is a lot less work.

Tasks:

Estimate: 8 days

Benefits:

Risks:

Multiple Schemas Reporting

With the new DlfStore in place, it would be possible to generate reports using data coming from multiple log files and thus using multiple schemas.

This feature isn't about storing the report configuration using an XML format nor about configuring report configuration file using lr_config (what Wessel started working on).

Tasks:

Estimate: 4 days

Benefits:

Risks:

Analysers API

Analysers are an important part of Lire, but their API needs to be updated to match the functionalities of the DlfConverter. The analyser should also be run after each log importation step and not every time a report is generated.

Tasks:

Estimate: 20 days

Benefits:

Risks:

Better HTML Output

The HTML reports are really suboptimal. They look awful because we are using DocBook as an intermediary format and the table support in the DocBook stylesheets is deficient. Developing an HTML formatter which generates HTML directly from the XML report would brings many benefits to Lire.

Estimate: 4 days

Benefits:

Risks:

Better PDF Output

The PDF output format could also be improved for about the same reasons than the HTML format: DocBook tables looks really ugly.

Estimate: 5 days

Benefits:

Risks:

Improved Charting Tool

The charts generated by Lire could be improved several ways. It should be possible to generate charts which plots more than one variable on the same chart. The charts should be configurable in other ways than by changing report specifications.

Tasks:

Estimate: 8 days

Benefits:

Risks:

Man-power

For the rest of the summer, I'm available to work part-time on Lire at a rate of about 24 hours a week. (I have another 2 days a week contract with a University research group.) In September, my availability could be increased to 32 hours.

I have an interesting suggestion to make though to increase the work effort to develop Lire. Wolfgang Sourdeau, a friend of mine, is also available for work. Joost already knows him, but for those who don't, I'll introduce him briefly: he's a Belgian who has been living in Montreal for the past 4 or 5 years. He was one of my employee when I worked at iNsu. He's a Debian developer and is also skilled in perl and C.

Besides these qualities, there is another thing that could be interesting for the Lire project: we could code major parts of it using pair programming. Wolfgang and I started doing this a few weeks ago on some other projects. (Since he was also looking for work, I proposed him two months ago to collaborate together on future projects.)

For those who don't know what pair programming is about, it's a practice advocated by the tenants of eXtreme Programming where all production code is written by two programmers at the same desk. What could be seen first as a waste of resource gives really interesting results. The most important benefits of pair programming are:

Schedule and Release Management

In summary, here are the various features along with their estimates:

1. SQLite Based Report Generation 15 days
2. Store Configuration 10 days
3. Internationalisation 8 days
4. Multiple Schemas Reporting 4 days
5. Analysers API 20 days
6. Better HTML Output 4 days
7. Better PDF Output 5 days
8. Improved Charting Tool 8 days +
  Total:      74 days

All of these features are independent of each others (except for the Analysers API item for which the estimate needs to be re-evaluated depending whether the "SQLite Based Report Generation" and "Store Configuration" are implemented or not.

This means that we can reorganise the priority of the features and we can make a release anytime a subset of these features is considered enough for a release. (With the unit tests and functional tests, making a release is a relatively straightforward job: it took about 2 days last time.)

Apart from the feature "SQLite Based Report Generation" which I think should be our top-priority, I don't have a strong opinion as to the order of the other ones. (Although putting "Internationalisation" in early, will make it less work to I18N other features.)

This also means that the feature list could be prioritised according to input from our users. This could be made part of the user survey that was suggested at the last Board's meeting.

As far as release dates goes, I think it would be wise to let pass two weeks of pair programming to measure our velocity (that is the amount of development-days we are able to make by week). Release dates could then be picked up according to the features we want implemented before making a release (and the above estimates).

As a final note, I think we should make more release than less. Since all of these features brings important benefits to the users, I think we should push out these benefits sooner than later.

I'm eagerly waiting for your comments on this proposal. I'll gladly answers any questions regarding this proposal. Also, if the board would like to focus more on the "Log Analyser" audience, you can suggest other features and I'll integrate them to this proposition.

Best regards
Francis J. Lacoste

Calls

Send in your ideas.
Deadline Feb 1st, 2018.

 

Project LogReport

NLnet Projects