Logblog: Logging Poll #5 "Top Logging Challenges" Analysis

« SANS Security Laboratory Thought Leadership Interview With Dr Anton Chuvakin | Main | Poll: What logs do you actually LOOK at? »

Logging Poll #5 "Top Logging Challenges" Analysis

OK, this poll was insightful! The raw results are here and below:

What can we learn from this?

First, what are the top challenges? It is with great regret :-) that I report that the #1 challenge is exactly the one I thought it would be: We collect logs but don't have time/resources to look at them. Yes, automated "analysis challenge" has only become more of a challenge as people deploy more tools that enable log collection on a massive scale (e.g. 75,000 logs/second). I dare to predict that we will finally have to tackle this one in the next year or two. In fact, this challenge rears it ugly head via another popular response, Lack of log analysis tools, which made Top 5 responses as well.

Second, even though I didn't make any predictions about the #2 entry, but I was surprised: No way to effectively search all logs is a very close #2 (obviously, 1 vote is not statistically significant here). Indeed, log searching is an elusive little problem, especially when we want to do it fast and on a large pool of logs. Even though I think we need to search less and discover more, the need to search logs will be with us forever (and, no, I don't think you need a special product just to do that ...)

Third, I am happy to report that this poll indicates that we finally broke the back of "the beast" of not having logs. Responses that point at not having logs (e.g. Logging is not enabled, We don't know what logging we must enable, etc) are not terribly popular (then again, maybe it is due to self-selection of the blog readers ...)

Fourth, infrastructure! Specifically, No infrastructure to manage the log volume we have is very popular as well (#4). This proves the point that I used to not take very seriously in the past (by mistake): when megabytes become gigabytes and those grow into terabytes, many things that used to trivial (e.g. moving logs from A to B, saving logs to disk, etc) become grand engineering challenges... Indeed, to manage high volume of logs you need a scalable log management solution (example)

Sixth, as I lamented, few care about log security (this counts as laments, I guess). Secure storage of logs is only bothering a few people. One word: yet! As of today, stored log hashing + (sometimes!) log transport encryption + (rarely!) encrypted archives are the state of the art.

Next poll is coming up!