February 2008 Archives

« January 2008 | Main | March 2008 »

Top 11 Reasons to Analyze Your Logs

As promised, here is another "Top 11 Reasons" which is about log analysis. Don't just read your logs (definitely don't just collect them); analyze them. Why? Here are the reasons:

Seen an obscure log message lately? Me too - in fact, everybody have. How do you know what it means (and logs usually do mean something) without analysis? At the very least, you might need to bring additional context to know what some logs mean (example: IP address -> hostname -> server owner)
Logs often measure in gigabytes and soon will in terabytes; log volume grows all the time - it definitely passed the limit of what a human can read a long time ago, it then made simple filtering 'what logs to read' impossible as well: automated log analysis is the only choice.
Do you peruse your logs in real time? This is simply absurd! However, automated real-time analysis is entirely possible (and some logs do crave for your attention ASAP - e.g. major system failures, confirmed intrusions, etc)
Can you read multiple logs at the same time? Yes, kind of, if you print them out on multiple pages to correlate (yes, I've seen this done :-)). Is this efficient? God, no! Correlation across logs of different types is one of the most useful approaches to log analysis.
A lot of insight hides in "sparse" logs, logs where a single record barely matters, but a large aggregate does (e.g. from one "connection allowed" firewall log to a scan pattern). Thus, the only way to extract that insight from a pool of data is through algorithms that "condense" that collection of logs into usable knowledge (some say, visualization is the way to go)
Ever did a manual log baselining? This is where you read the logs for a while and learn which ones are normal for your environment. Wonna do it again? Thought so :-) Log baseline learning is a useful and simple log analysis technique, but humans can only do it for so much before burning out.
OK, let's pick the important logs to review. Which ones are those? The right answer is "we don't know, until we see them." Thus, to even figure out which logs to read, you need automated analysis.
Log analysis for compliance? Why, yes! Compliance is NOT only about log storage (e.g. see PCI DSS). How to highlight compliance-relevant messages? How to see which messages will lead to a violation? How do you satisfy those "daily log review" requirements (again, see PCI DSS)? Through automated analysis, of course!
Logs allow you to profile your users, your data and your resources/assets. Really? Yes, really: such profiling can then tell you if those users behave in an unusual manner (in fact, the oldest log analysis systems worked like that). Such techniques may help reach the holy grail of log analysis: have the system automatically tell you what matters for you!
Ever tried to hire a log analysis expert? Those are few and far between. What if your junior analysts can suddenly analyze logs just as well? One log analysis system creator told me that his log data mining system enabled exactly that. Thus, saving a lot of money to his organization.
Finally, can you predict future with your logs? I hope so! Research on predictive analytics is ongoing, but you can only do it with automated analysis tools, not with just your head alone (no matter how big :-)) ...

Past top 11 reasons:

Technorati tags: logging, log management, tips

Posted February 22, 2008 in Innovation , LogMatters | Permalink | Comments (0)

« January 2008 | Main | March 2008 »

Poll: What logs do you actually LOOK at?

This is my 6th logging poll (vote here now!)- links to the previous five polls below.

This one is deceptively similar to the #1 below, but it is not. This poll is What logs do you actually LOOK at? and not Which Logs Do You Collect? In other words, are you a log packrat? Are you collecting and never using the log data? You are making a mistake, if you don't.

Past polls:

Poll #5 "What are your top challenges with logs?" (analysis)

Poll #4 "Who looks at logs in your organization?" (analysis)

Poll #3 "What do you do with Logs?" (analysis)

Poll #2 "Why collect logs?" (analysis)

Poll #1 "Which logs do you collect?" (analysis )

Technorati tags: logging, logs, polls, log management

Posted February 14, 2008 in Log Management & Intelligence , LogMatters | Permalink | Comments (0)

« January 2008 | Main | March 2008 »

Logging Poll #5 "Top Logging Challenges" Analysis

OK, this poll was insightful! The raw results are here and below:

What can we learn from this?

First, what are the top challenges? It is with great regret :-) that I report that the #1 challenge is exactly the one I thought it would be: We collect logs but don't have time/resources to look at them. Yes, automated "analysis challenge" has only become more of a challenge as people deploy more tools that enable log collection on a massive scale (e.g. 75,000 logs/second). I dare to predict that we will finally have to tackle this one in the next year or two. In fact, this challenge rears it ugly head via another popular response, Lack of log analysis tools, which made Top 5 responses as well.

Second, even though I didn't make any predictions about the #2 entry, but I was surprised: No way to effectively search all logs is a very close #2 (obviously, 1 vote is not statistically significant here). Indeed, log searching is an elusive little problem, especially when we want to do it fast and on a large pool of logs. Even though I think we need to search less and discover more, the need to search logs will be with us forever (and, no, I don't think you need a special product just to do that ...)

Third, I am happy to report that this poll indicates that we finally broke the back of "the beast" of not having logs. Responses that point at not having logs (e.g. Logging is not enabled, We don't know what logging we must enable, etc) are not terribly popular (then again, maybe it is due to self-selection of the blog readers ...)

Fourth, infrastructure! Specifically, No infrastructure to manage the log volume we have is very popular as well (#4). This proves the point that I used to not take very seriously in the past (by mistake): when megabytes become gigabytes and those grow into terabytes, many things that used to trivial (e.g. moving logs from A to B, saving logs to disk, etc) become grand engineering challenges... Indeed, to manage high volume of logs you need a scalable log management solution (example)

Sixth, as I lamented, few care about log security (this counts as laments, I guess). Secure storage of logs is only bothering a few people. One word: yet! As of today, stored log hashing + (sometimes!) log transport encryption + (rarely!) encrypted archives are the state of the art.

Next poll is coming up!

Technorati tags: logging, log management, polls

Posted February 08, 2008 in Innovation , Log Management & Intelligence , LogMatters | Permalink | Comments (0)

Visit loglogic.com