Logblog: From Log Apathy to Log Enlightenment

« RSA 2008 Summary | Main | Critical Log Management Questions - Answered! »

From Log Apathy to Log Enlightenment

So, I was talking to this small log management vendor the other day and he confided to me that his product faces fierce competition in his target market (which is, important to note, small to medium companies with 10-100 systems): and this competition is apathy.

More specifically, his prospects either just blow him off by saying "pah, who needs logging!" or they profess their undying love of all things logging - and then still don't buy his product (which is priced, shall we say, "to go")

Admittedly - and somewhat tongue-in-cheek, these are the same companies that form the core of today's botnets (due to various reasons including their scarce resources) and enable RBN to deliver high-quality malicious services to criminal enterprises worldwide. Still, if you happen to have thoughts along the line of "who needs logs?" or "ah, logging? it will come later!", you really deserve a nice fat check from RBN and other malicious "hacking" syndicates since it is extremely likely that your overall attitude towards security is just as misguided...

But how to progress from such ... what was before the Stone Age? ... Sharpened Stick Age? to modernity? Most companies go through the following stages in regards to their logging:

Deep log ignorance: "Logs? What are those?"
Shallow log ignorance: "Later...later...later... #37 on the TODO list."
Log collection: "We gather and store dead log data...cold."
Log searching: "We will dig into the pile when we have to ... hopefully never!"
Log analysis and reporting: "We know our logs - and what they mean"

(also see my post "Natural Flow of Log Management" for some specifics)

Of course, compliance (PCI DSS and others) helped move people from 1. and 2. to 3., but, sadly, people often get stuck at 3. (just collection) or 4. (collection + maybe search) and never progress to Logging Enlightenment of 5.

Yes, PCI DSS and other regulations mandate not just log collection, not just dead cold log storage, but also log review (daily, in case of PCI DSS Requirement 10), but "review" happens to be the item that gets overlooked all too often.

Why is that?

I think the reason is that log analysis is still too hard and still not automated enough for an average organization. Yes, I did see some corporations that built their own log analysis systems that - surprise! - exceeded the best available from the vendors [at the time]. However, a typical company IT department would not have Ph.D. poring through hardcore text mining research papers in order to improve their home-grown log analysis AI. They expect the vendors to eat the logs, chew on them for a bit - and then spit out the answers.

Are we there yet? No, but we will be!!!