« March 2008 | Main | May 2008 »

Critical Log Management Questions - Answered!

Here are some interesting log management questions I got asked some time ago; reposting here for our blog readers.

Q1: For those companies that have successfully implemented enterprise-wide logging, what were the big nasty surprises that they encountered?

A1: Here are a few:

• political boundaries within the organization: "these are our logs, and you are not getting them"
• privacy laws: some logs cannot be collected in some countries; some cannot cross the border, some cannot be seen by some people, etc. This is true mostly in EU, less in US.
• legal blocks: work with legal before deploying any org-wide log management; legal might try to prevent certain data from ever being created (for fear of being legally discovered later)
• log volume: underestimating log volume is common and pretty nasty
• related to the last one: vendors being "optimistic" about their tool scalability
• time synchronization (of course!), specifically, lack thereof.

Q2: For those companies that have successfully implemented enterprise-wide logging, what was their implementation approach?

A2: Typically, 2-3 vendor PoC or pilot first. Then with the chosen vendor: phased approach based on location + type of log source (e.g. firewalls, then routers, then OS, then proxies, etc) + network topology (e.g. DMZ, then internal) + log source criticality (e.g. critical servers first; the rest next). This might be handy to look at.

Q3: What kind of storage requirements have been experienced by those organizations who have successfully implemented enterprise-wide logging?

A3: Massive? :-)

Here is a simple example: PCI DSS is a bit more aggressive than NERC since it mandates 1 year of log retention vs NERC 90 days, so: 1 year worth of logs is = 365 days x 24 hours x 3600 seconds x 1 (one!!!) busy firewall with 100 log messages each second x 200 bytes per message average (e.g. valid for PIX and ASA devices) = 588 gigabytes / year of raw log data uncompressed (assuming 10x compression you'd get about 60GB of compressed log data per year)

Store it in RDBMS? Multiple it by 2-3. Have an index? Add about 30%.

The bottom line is: terabyte is the unit to measure logs.

Q4: At the organizations that have successfully implemented enterprise-wide logging, how logging impacted network and system performance?

A4: Too broad a question, so here are a few pointers:

• logging affects performance much more on some types of systems compared to other types: most painful examples are databases where some people (can't find a link...sorry) report performance loss of up to 40% if logging all SELECT statements and other data retrieval commands (you need to log selectively on these); in other cases (e.g. web servers) there is no performance loss and logging is "always on"
• log collection: agents impact system performance (long post on this subject): a little when they run (everybody knows this) and A LOT when they crash (few people think about it - agent software memory leaks are not uncommon); unlike agents, remote agentless log collection barely affects system performance (unless you have one of the few esoteric cases)
• log transfer and network performance: look for compressed (logs compress really well), TCP-based transfers; syslogging over UDP uncompressed has a chance of doing a pipe saturation DoS on your network. Yes, people say "use a dedicated LAN," but this is definitely wishful thinking for many. Also, raw UDP syslog in large quantities over WAN = insanity :-)

Q5: What were some successful strategies for obtaining buy-in from system owners and operators in regards to turning logging on?

A5: OK, also too broad a question, but here are some pointers:

• provide them a useful service based on their logs (e.g. performance measurement, availability monitoring, compromise detection :-), or other security metrics, etc)
• help them with their compliance mandates (e.g. create reports that they can show to the auditors that "bug" them)
• give them tools to better solve their problems (e.g. allow access to a log management tool so that can investigate issues better, search the logs, check on their users, etc)

Q6: How the organizations that have successfully implemented enterprise-wide logging dealt with unusual devices (=log sources) that have no log management vendor support?

A6: They were in massive pain - if they choose a log management vendor wrong. You need to look for vendors that have "universal log source support" with NO requirement for a custom rules or custom collector/connector/agent development. LogLogic have generic text log collectors that can grab and analyze unknown logs. Typically this is done via some form of text indexing that works across all logs, including those from unknown, vertical, esoteric or custom-developed log sources

Hope it was useful!

Posted April 24, 2008 in Compliance , Innovation , Log Management & Intelligence , LogEd | Permalink | TrackBack (0)

« March 2008 | Main | May 2008 »

From Log Apathy to Log Enlightenment

So, I was talking to this small log management vendor the other day and he confided to me that his product faces fierce competition in his target market (which is, important to note, small to medium companies with 10-100 systems): and this competition is apathy.

More specifically, his prospects either just blow him off by saying "pah, who needs logging!" or they profess their undying love of all things logging - and then still don't buy his product (which is priced, shall we say, "to go")

Admittedly - and somewhat tongue-in-cheek, these are the same companies that form the core of today's botnets (due to various reasons including their scarce resources) and enable RBN to deliver high-quality malicious services to criminal enterprises worldwide. Still, if you happen to have thoughts along the line of "who needs logs?" or "ah, logging? it will come later!", you really deserve a nice fat check from RBN and other malicious "hacking" syndicates since it is extremely likely that your overall attitude towards security is just as misguided...

But how to progress from such ... what was before the Stone Age? ... Sharpened Stick Age? to modernity? Most companies go through the following stages in regards  to their logging:

1. Deep log ignorance: "Logs? What are those?"
2. Shallow log ignorance: "Later...later...later... #37 on the TODO list."
3. Log collection: "We gather and store dead log data...cold."
4. Log searching: "We will dig into the pile when we have to ... hopefully never!"
5. Log analysis and reporting: "We know our logs - and what they mean"

(also see my post "Natural Flow of Log Management" for some specifics)

Of course, compliance (PCI DSS and others) helped move people from 1. and 2. to 3., but, sadly,  people often get stuck at 3. (just collection) or 4. (collection  + maybe search) and never progress to Logging Enlightenment of 5.

Yes, PCI DSS and other regulations mandate not just log collection, not just dead cold log storage, but also log review (daily, in case of PCI DSS Requirement 10), but "review" happens to be the item that gets overlooked  all too often.

Why is that?

I think the reason is that log analysis is still too hard and still not automated enough for an average organization. Yes, I did see some corporations that built their own log analysis systems that - surprise! - exceeded the best available from the vendors [at the time]. However, a typical company IT department would not have Ph.D. poring through hardcore text mining research papers in order to improve their home-grown log analysis AI. They expect the vendors to  eat the logs, chew on them for a bit - and then spit out the answers.

Are we there yet? No, but we will be!!!

Posted April 22, 2008 in Compliance , Innovation , Log Management & Intelligence , Security | Permalink | TrackBack (0)

« March 2008 | Main | May 2008 »

RSA 2008 Summary

So, RSA 2008 has come and gone.  Now that everybody has recovered from the information overflow, it is time for summaries and reflections. Before we begin, go read my RSA Impressions (Part 1,2,3,4). Next, read what others said about RSA 2008 (via my del.icio.us feed). Then reflect on past RSA shows (2006, 2007).

Ready now?

First, what was the theme? Unlike in the past, I personally couldn't pick any. The candidates were GRC (and compliance + risk management) , DLP (and other data-leak/-theft technologies), IAM (and identity enablement), but none quite measured up to being a show "theme."

What I saw too much off? Even though their numbers have shrunk, I still saw too many NAC vendors there (Lockdown, here we come!). One of my friends joked that there were more "GRC vendors" than NAC vendors, but both were in low enough numbers to make it a trend. As far as loud noises back from RSA 2007, "identity-driven this or that for security" was still very visible. What is also bizarre is that people still start log management companies. I saw a couple of new ones - amazing, isn't it? Yes, it is a hot space, but come on! Accept that LogLogic is the leader and be done with it :-)

Overblown messages? "Information-centricity." It was cool and hot :-) and new when Hoff said it.  But when it trickled down to keynotes of some "trailing edge" vendor executive, it became boring and stale. And, while we talk about "information centricity" people must still worry about "A" (availability) first (see this discussion).

What I didn't see enough of? VOIP security. Somehow this previously hot trend is quieted down. Also, I saw a lot of web application security vendors, but I think that is still not enough as this is an area with a raging fire, not just "some hotness." Also, I expected to see more vendors messaging around (and, actually helping with!) fraud. Dan Geer's Verdasys kinda mentioned that, but pretty much in passing. Is fraud handled outside of security (and thus out of RSA)? I am not sure.

What I didn't see at all? I didn't see much "market consolidation" - no huge deals, no vendors of note "taken out", etc. Still a huge number of security companies around ... some with products that seem deeply out of touch with today's threat landscape.  One of the speakers said that nowadays "no single security pure-play expected to change the world", but it sure seems like many will try!   Along the same line, Mike R said that such shows are 18-24 months ahead of what "normal" people deploy. This might explain the VOIP and other missing items.

As I said before, "consumerization" of IT - i.e. IT infrastructure, servers, laptops, storage, services, computing resources, applications, etc provisioned outside of IT departments was an elephant in the room. It is not simply "unmanaged IT" or "consumer-grade IT for business", it is the whole "not-IT-department IT" phenomenon. Yes, via mashups it even includes "non-IT application development" (read this fun 451Group report on that). Security implications of this are nothing short of ginormous.

In light of this, I liked how one presenter said this: "we lost the desktop" - meaning "1/3 is managed by users [not IT], 1/3 is unmanaged and 1/3 is compromised/ "managed" by hackers."  Sad but true... Dave Aitel used to joke how in the future banks will have to "re-compromise" your PC so that some temporary security can be established for you to transact business with them. Are such horrifying times upon us already? Maybe desktop virtualization will help with this ...

Overall, RSA 2008 show was an enjoyable, enlightening and thought-provoking experience, as usual. I've heard it was the same even for people who didn't attend a single presentation - indeed, RSA is about people, not slides...

See you next year at RSA 2009!

Posted April 16, 2008 in Compliance , Innovation , Log Management & Intelligence , Risk Management , Security | Permalink | TrackBack (0)

« March 2008 | Main | May 2008 »

Security Mash-Ups in the Cloud

In the past 90 days, Savvis, Verizon Business Services and SecureWorks launched hosted or cloud-based log management services. Customers do not pay for owning any log management infrastructure, but rather for using it and typically pay monthly on a per-device basis. Security in general and log management specifically lend itself well for a cloud-based model since compliance mandates and new threats continue to evolve and require continuous updating of reports and alerts, in addition to a high level of expertise, often not found in-house.  Stephen Northcutt from SANS has made the case for outsourcing Log Analysis here.

But there is one more significant advantage to Security in the Cloud … Security service providers can be (and are) light-years ahead of product vendors when it comes to putting together integrated compliance offerings. Customers typically access cloud-based offerings through a web portal. Using web services, service providers can quickly and easily create compliance mash-ups, for example combining reports from vulnerability scanning services with log management-based user activity reports. Very cool!

Posted April 15, 2008 in | Permalink | TrackBack (0)

« March 2008 | Main | May 2008 »

On Trusting Log Timestamps

It is about time: specifically, timing logs. As I said in my Log Trust and Protecting Logs from Admins posts, the issue of trust is critical in the logging world. After all, logs = accountability; and the latter in unthinkable without trust. If we are to at least pretend that logs objectively record events and user actions, we need to unambiguously establish WHAT happened and WHEN. This post deals with the 'WHEN'  issue.

So, can we trust that the time stamp in the log file or the one added by the log management system correctly describes when the recorded event actually happened?

We will start from locating the timestamps in logs. Most of the log formats, such as file-based logs (web servers and proxies, various application, some security gear, etc) and syslog, Windows event logs, database audit tables, etc contain a timestamp. Here are a few examples:

1. Mar 20 10:03:47
2. 19/Mar/2006:04:27:33 -0800
3. Sun Mar 19 04:45:19 2006

In fact, once I saw somebody use a timestamp to define logs as "timed records of IT activity." So, time is critical for logs being, well, logs :-) At this point it is worthwhile to note that file-based logs will contain a timestamp IN the file, while syslog records arriving over the UDP or TCP port 514 connection are usually timestamped upon arrival BY the syslog daemon (using its own "knowledge" of time) - and then it shows up in the syslog files in /var/log.

Let's assess whether this "in-log timestamp" provides an adequate way of timing the actual events that are being logged. Answering this question is important for investigations and troubleshooting, but becomes  nearly a matter of life and death in case of log forensics. Here are some fun cases and issues to consider:

First, what are the chances of a  completely false timestamp in logs (BTW, today is Jan 1, 1970!) When might that happen? Typically when a logging system own clock is reset or not set correctly. This timestamp clearly should NOT be trusted. How do you then time this event? This is a separate story that will be discussed some other time...

Second, it’s always 5PM somewhere: in other words, what time zone are your logs in? EST? PDT? GMT? UTC? Or any of more than 24 other possibilities. If you have no idea, you should not trust the timestamp.

Third, are you in drift? Is your system clock? Those pesky drift seconds turn into minutes that then work to undermine the accuracy of timing the records (and thus your certainly and trust in evidence quality)

Fourth, syslog forwarder mysteries are plenty: some of the syslog messages will be delayed in transit and then  be timestamped by the final recipient daemon, thus completely losing when the event was originally logged. Admittedly, this delayed syslog messages are rare, but as more people employ buffering syslog daemons (e.g. syslog-ng), it might happen more often.

Fifth, more esoteric, but still real (and really annoying!): some system logs will contain two timestamps. If you don't possess in-depth knowledge of this specific log, confusion has a chance to undermine the trust as well (so, which timestamp should I use?)

Sixth, most people will not think that they will fall to something that stupid: 24 vs 12 hour time. However, when facing an unknown (and poorly designed!) log format, beware that 5:17 might well be 17:17...

Finally, if you know that something got logged at 5:17AM, then when did it happen? Beware of "Log lag!" This issue is actually too tricky to give it justice here... The simplest example is when the process leaves a log record when it exits, not when it starts, which might happen days earlier (thus creating a log lag).

As we dive into more issues with timing logs, we also need to think about sequence timing and absolute timing. Sequence of logged events is a critical fact! Miss the sequence and the whole “house of cards” goes …  But! Absolute time is also important! Can we be assured of both being correct all the time? (hint: no)

So,  when you look at logs next time and you see a timestamp there - start thinking about all of this :-)

Posted April 04, 2008 in Innovation , Log Management & Intelligence , Risk Management , Security | Permalink | TrackBack (0)

« March 2008 | Main | May 2008 »

Is your digital shadow protected?

Many UK retailers handling credit card transactions have no visibility of who has been accessing data within the company network, as reported by Computing Magazine based on research by Vanson Bourne on our behalf. Almost half (45 per cent) of the 60 medium-to-large UK retailers polled – which include supermarket chains, department stores and clothing retailers – are unable to track and trace data access. Even at those retailers who do track data access, security breaches can go unnoticed for an entire day in most implementations.

The responsibility for protecting and maintaining 85 per cent of your ‘digital shadow’ including financial records, web surfing histories, historical credit card transactions and such lies with businesses such as retailers, according to a study commissioned by EMC.

Is it surprising anybody that fraud is at an all-time high? This too has been reported on by our UK friends at Computing Magazine, based on research by KPMG. It sure seems like across the pond they have all the numbers to backup why user tracking should be a priority.  So why isn't it? Problems cited by the IT directors in our survey include budget restrictions (24 per cent), time (14 per cent) and other priorities (41 per cent).  

Posted April 02, 2008 in | Permalink | TrackBack (0)

« March 2008 | Main | May 2008 »

What Log Data can (not) do for Hannaford

Indications are that the recent, massive, breach at Hannaford Bros. Co.'s supermarkets was an inside job leveraging a sophisticated malware deployment.  We were asked today whether malware will generate logs that we could have picked up in the Hannaford case and the answer in general is "no". These types of software are very evasive and try not to leave any type of trail while running.

However, what could help investigation in this case is logs generated by the systems on a day-to-day basis. These logs can be used for forensic analysis, e.g., potentially identifying who did the installation and where they came from. In this case, logs from the POS systems, servers that manage the data, databases that store the data, firewalls that protect these systems, can all be used for forensic analysis.

Of course this case is getting national attention because the number of customers affected (4.2 million) and the large number of fraud cases (1,800) already linked to the data breach. However, there are a ton of breaches that happen on a daily basis. A good place to track them is The Breach Blog.

Posted April 01, 2008 in | Permalink | TrackBack (0)

Visit loglogic.com

Subscribe to this blog’s feed

• May 2008
• April 2008
• March 2008
• February 2008
• January 2008
• December 2007
• November 2007
• October 2007
• September 2007
• August 2007
• July 2007
• June 2007
• May 2007
• April 2007
• March 2007
• February 2007
• January 2007
• December 2006
• November 2006
• October 2006
• September 2006
• August 2006
• July 2006
• June 2006
• May 2006
• April 2006
• March 2006
• February 2006
• January 2006
• December 2005
• November 2005
• October 2005
• September 2005

Blogroll

• CynergisTek
• Bruce Schneier
• InfoSecDaily

Compliance

• Continuity Central
• Compliance Pipeline
• SOX Compliance Journal
• Sarbanes-Oxley 101

Good Reading

• Log Management ROI
• Log Management Best Practices
• SANS Institute Log Management White Paper

LogLogic

• O’Reilly
• Anton Chuvakin
• LogLogic.Com
• Jian Zhen

LogLogic Partners

• Juniper Networks
• ONStor
• Counterpane
• Bluecoat

Sites We Watch

• The Merc
• CRN
• IT Architect
• The Reg
• Security Focus
• Slashdot
• TechWeb
• SANS
• C/Net
• CIO Insight
• InformationWeek
• Jamie Lewis - Burton Group
• eWeek
• IT Manager's Journal
• MIT Magazine
• VNU