Instant Poll: What works best when justifying security spending in your organization?

You are only ONE CLICK away from seeing what your peers are saying. 

Click on one of the choices below to see INSTANT results.

Posted by Dominique Levin on June 30, 2009 in Log Management & Intelligence , Security | Permalink | TrackBack (0)

LogLogic Buys Exaprotect: 3 Reasons Why Customers Win.

By Dominique Levin
VP Marketing & Strategy

Last week LogLogic announced its intend to acquire Exaprotect. In February we had already announced a partnership with Exaprotect to deliver the LogLogic Security Event Manager. In February we also announced LogLogic Compliance Manager, which has since shipped to the general public, and LogLogic Database Security Manager, generally available later this quarter. Now we have added the Exaprotect Change Manager product line. In a mere couple of months LogLogic went from a singularly focused company with leading log management platforms to having five product lines working together to form the most complete security management suite.

So how does this all benefit customers?  The combined product portfolio answers 3 simple questions for customers:

What is happening?

What is important?

What to do about it?

1. What is happening? Log Management and Database Activity Monitoring.

It all starts and ends with log data. You cannot secure or manage what you cannot see. Therefore, first focus on building a central repository of user and system activity. You do this through aggregating, summarizing and archiving log data. Log data can tell you who are accessing your network, systems and even who are seeing, changing or moving individual information objects. Per a recent SANS survey, 99 percent of customers are collecting (or planning to collect in the next year) some log data but for many it is work in progress. Virtually all collect network data (“who is accessing my network?”) and most collect system-level data (“who is accessing my systems?”). For most companies even collecting a complete activity record remains a work in progress. Leading-edge organizations are now turning their attention to understanding activities around business applications, transactions and monitoring access to specific sensitive information objects. This is particularly true for structured information in databases. Databases are a one-stop shop for valuable data. Organized criminals are targeting sensitive data in databases to sell for $300 per record. Since the data is structured, you know where it resides and you can monitor access to these specific records. LogLogic expanded into database activity monitoring with a specialized database sensor. The sensor sees more than you would through native logs, including activities that are triggered by stored procedures, obfuscated queries and such. This is great as a stand alone product, but at the end of the day, database activity should be analyzed in context with all other activity data – hence the convergence of log management and database activity monitoring.

2. What is important? Compliance management and security event management.

Just having the data on a pile is of course not enough. Once you have a central record of activity, you need look at this information. Few organizations are proactive about this. LogLogic compliance management and security event management applications can help. LogLogic Compliance Manager is about deciding who should be looking at what log data when and then enforcing such log review process through software. Compliance is a collaborative process and Compliance Manager facilitates collaboration on pro-active security. It productizes best practices, presents reviewers with an easy in-box of log review tasks and the ability to annotate and score activities. Ultimately the log review scores roll up into a dashboard that presents executives with the overall timeliness of review and a compliance score. It is still human beings who do the bulk of the actual analysis. LogLogic Security Event Manager goes one step further and uses cross-device correlation and contextual analysis with vulnerability and asset data to prioritize suspicious activities automatically. For example, access to a HR database followed by a large e-mail sent, could be suspicious and needs to be investigated immediately.

3. What to do about it? Change management and database security.

Contextual analysis of log data is cool and it can go a long way turning raw log data into actionable information and even into recommendations. However, security Nirvana would be self healing. Increasingly software could make automated recommendations and predictions about unusual and suspicious activities and could prevent bad things from happening in the first place. LogLogic Change Manager and the LogLogic Database Security agent both have the ability to enforce security policies. Most customers aren’t quite ready to automatically re-configure a firewall policy based on a security alert, but at some point in the future as predictions become more accurate, automatic remediation will become a reality. One area where automated prevention is a reality is in database security. About 20% of database security customers also turn on active blocking. It makes sense that blocking would be more prevalent with systems that can do fine-grain monitoring. It is tricky to kick somebody off the network wholesale based on a security alert. There are still too many false positives. If you get it wrong you seriously hurt productivity. That is not a good thing ever, but especially not in an economic downturn. Most organizations prioritize productivity over security. It is much more acceptable however, to block access to a specific piece of information based on suspicious activity.

In summary, with the addition of Exaprotect, LogLogic can better protection information at a lower cost. This is good news at a time that few customers can afford to maintain the staff and budgets to integrate many disparate point products.  Unified security management also leads to better information protection. Pro-active security monitoring (LogLogic Security Event Manager and LogLogic Compliance Manager), combined with fine grain monitoring (LogLogic Database Security Manager) leads to more accurate prevention (LogLogic Change Manager and LogLogic Database Security Manager) and better information protection.

Posted by Dominique Levin on April 27, 2009 in Compliance , Log Management & Intelligence , LogLogic News , Security | Permalink | TrackBack (0)

Gartner Says We’re Cool!

by Lex van den Berghe
LogLogic Customer Evangelist

For years, we've been drinking our own Kool-Aid, saying that log management is cool. It's more than just a check box on your list of regulatory compliance initiatives – at LogLogic we turn water into wine by collecting all sorts of information from every machine in your business and we make sense of it. Once you know what's in your logs, you can get a better idea of your security and operational posture. Just as business intelligence (BI) systems help enterprises make sense of endless streams of data, we take the Tower of Babel and translate it into one language.

If you don't believe me, check out Gartner's new report, "Cool Vendors in Storage Technology and Systems, 2009," published last Monday by a variety of Gartner analysts and researchers. The report says, "“Bandwidth reduction and greater safeguards against internal and external threats via log management will result in cost savings to the customer in time saved and threats avoided.”

We are the only log management and security company included. Why? Without log management, 30% of an enterprise's data would be lost. It would literally slip through the cracks. While other companies have focused on starting with the security perimeter to protect information assets, we have always focused on the "lowly log" as the root of the challenge, and that means collecting every single piece of information possible. How else would you get a comprehensive picture of what's happening to your data?

So, how do we store and help you search all of that information, which is often more than a terabyte of new data each day?

First, we've started with a Linux appliance that can be deployed on premise, as a managed service or even as a virtual appliance. From there, customers have a wide range of options and flexibility to use our compliance suites or build their own applications on top of our platform to gain back value from their logs. But, we didn't stop there. By building a solid logging foundation, you can monitor and manage security events, database activities and regulatory compliance for any industry. You can search for events just like you would search for information using Google, and you can do it in a fraction of the time it would take you to manually process this information or to deploy multiple solutions from multiple vendors.

The truth is, logging is cool. It's time to jump on the bandwagon if you haven't already – check out our latest article on the convergence of security information and event management (SIEM) and log management in Network World.

Posted by Lex Van den Berghe on April 01, 2009 in Innovation , Log Management & Intelligence , LogLogic News | Permalink | TrackBack (0)

From Native to Instrumented

According to Wikipedia:

To log is a verb derivative of the noun logbook; the verb form means to record in a logbook, and may have been coined in the 1820s. The term logbook itself stems from the practice of floating a stationary "log" (actually a wooden block attached to a reel via rope) to provide a fixed point of reference for the purpose of measuring a ship's speed (see Knot (speed)). Computer scientists adopted the verb to log circa 1963 to describe the systematic recording of specific types of data processing events.

Logs have been around since the dawn of computers. It's been used by computer scientists, IT administrators, security analysts and network operators to perform analysis, troubleshooting and forensics for over 45 years. In most cases, however, users refer to logs as pieces of information that devices and applications generate on their own. For example, routers and switches generate logs that detail their status and what they are doing. Firewalls generate logs showing the various connections passing through, or not. Operating systems and utilities generate logs to communicate accesses to different parts of the system. Web proxies generate logs to describe user surfing activities. These logs are what one would consider to be "native logs."

One type of logs that users don't generally talk about, or event consider, are logs that are generated by agents or systems monitoring network or applications. This type of logs is called "instrumented logs." The most well-known type of instrumented logs is probably IDS logs. IDS monitors the network and reports any attacks that are happening. These reported events are usually considered as logs. However, there are other types of "instrumented logs" that users don't normally consider as logs. For example, most of the application performance management (APM) tools use agents or other means to retrieve information or statistics from various applications. These information are then sent to a central server for processing. In most cases, these information are not considered to be logs, but they do fit the definition of "systematic recording of specific types of data processing events."

Another example of instrumented logs are what Adrian Lane discussed in his blog post "Database Activity Monitoring & Event Collection Options." In this post, he mentioned several methods that monitor monitor database activities via sensors that either monitor the OS stack or the database memory. All of these methods generate "instrumented logs" that are sent to a central server for archival and analysis.

At the end of the day, whether it's "native" or "instrumented" logs, they are still pieces of valuable information that must be collected, archived, and analyzed. Also, the way these logs are analyzed are the same regardless whether it's "native" or "instrumented." As Jon Oltsik said,

In today's dangerous security landscape, no data is considered "noise" anymore. Rather, security analysts now want access to terabytes of historical data for analysis. Furthermore, this underlying data has become more complex.

.
.
.

It means collecting, normalizing, and storing a ton of data. It means sophisticated algorithms and processor-intensive query engines.

As sophisticated enterprises move up the stack (Network to OS to Applications) in their log management projects, we will likely see more and more of the log data come from sensors instrumenting the applications. This type of "instrumented" logs provide another rich set of information, sometimes richer in information compare to their "native" counterparts. Existing log management and security event management solutions can then take advantage of this set of information for compliance management, threat management and fraud detection.

Posted by on January 23, 2009 in Innovation , Log Management & Intelligence | Permalink | TrackBack (0)

IRS Should Audit Itself - or at least its cybersecurity logs

Slashdot is one of the places you can read about a recent report from the inspector general's office at the US Internal Revenue Service, the agency's IT staff hasn't been routinely checking its cybersecurity audit logs. Gasp! What?!? In short, the IRS is not in compliance with the Federal Information Security Management Act, called FISMA for short.

A quote from the report issued Monday and covered by PC World today states: "These weaknesses increase the likelihood that intruders from the Internet could gain access to sensitive taxpayer data residing on the IRS network without being detected."

We can't argue. The report says the IRS has effectively deployed intrusion detection systems (IDS) at its Internet gateways, but didn't have a process in place for vetting the logs. In addition, the IRS gave privileged users access to audit logs, leaving room for internal foul play. The report recommends the IRS institute a policy for saving audit logs and putting them through independent review by non-privileged administrators.

IRS CIO, Arthur Gonzalez, says the agency is working aggressively to protect its Internet gateways and to improve its overall security posture. Mind you, the report covered the period from February 2007 to March 2008, meaning that in the last 20 months or so, taxpayers have been vulnerable to identity theft.

FierceCIO.com writer, Judi Hasson, reported, "The action was like baking half a loaf when a full loaf was essential."

The IRS report was released the same day as Cisco's Annual Security Report, which found that Internet-based cyberattacks are becoming increasingly sophisticated and specialized as profit-driven criminals continue to hone their approach to stealing data from businesses, employees and consumers.

Highlights from the report as covered by Network World reveal:

• The overall number of disclosed vulnerabilities grew by 11.5 percent this year, compared to 2007.
• 
  
• Vulnerabilities in virtualization nearly tripled from 35 to 103 year over year.


• A 90% growth in threats originating from legitimate domains is double what was seen in 2007.

In addition, the Cisco report predicts insider threats to grow in 2009 as the global economic downturn entices employees to steal corporate data. On the upside, the Cisco report also foresees companies continuing to adopt well-enforced data security policies to make compliance easier and to reduce incidents of data loss. You can download the free report and watch an overview of the report on YouTube for more information.

Posted by on December 20, 2008 in Log Management & Intelligence | Permalink | TrackBack (0)

Podcast: IT systems analytics become more crucial as cloud and SaaS adoption raises complexity bar

Read a full transcript of the discussion. Find it on iTunes/iPod.

Software-as-a-service (SaaS) and cloud computing are changing the nature of IT systems’ performance requirements and heightening expectations for end users from online applications and services.

Increasingly, an extended level of visibility, management, and performance will apply to those serving up applications as services, regardless of their hosting origins or models. The more the apps and services fulfill a need, the more the users will expect even better results and performance.

In other words, the more these organizations succeed, the more they need to scale, leverage virtualization and cloud infrastructure methods, embark of service oriented architecture (SOA) and then keep all the trains running fast and on time. Using the latest tools and analytics — the equivalent of business intelligence (BI) for IT — on the systems and across the gathering complexity becomes essential.

To learn more about how systems log tools and analysis are aiding providers of cloud and SaaS, I recently spoke with fellow blogger Phil Wainewright, an independent analyst and director at Procullux Ventures, and SaaS blogger at ZDNet and ebizQ, as well as with Jian Zhen, senior director of product management at LogLogic.

Posted by on December 17, 2008 in Cloud Computing , Log Management & Intelligence , SaaS | Permalink | TrackBack (0)

Enable visibility and transparency through application logging

LogLogic has been advocating comprehensive logging for all IT components (or configuration items if you are in the ITIL camp) including applications for a long time now. We have worked with many of our customers to ensure that there's 100% collection and analysis of their IT log data. In the last several months there's been a huge uptick in the area of application logging, specifically for the application developers. This is partially due to the general interest in cloud computing and SaaS applications.

To quote a few blogs, Amrit Williams said in his blog "Amazon AWS, Google App Engine, Microsoft Azure, and More - Part 1: Can We Secure The Cloud?" (emphasis mine):

The one suggestion that elicited the greatest interest and most questions was a simple one; develop your applications so that they can be easily audited by the security and IT teams once they are in production, enable auditing that can capture access attempts (successful or not), date/time, source IP address, etc…the folks I talked to afterwards told me it was probably the single most important concept for them during the summit - enable visibility.

Todd Hoff said in "Log Everything All the Time":

you need to log everything all the time so you can solve problems that have already happened across a potentially huge range of servers.

What you need to be able to do is trace though all relevant logs, pull together a time line of all relevant operations, and see what happened. And this is where trace/info etc is useless. You don't need function/method traces. You need a log of all the interesting things that happened in the system.

Todd also gave a fairly extensive list of suggestions to application developers on how they should be logging in his article.

By logging, capturing and analyzing everything, IT organizations can enable visibility and transparency into their applications. This not only helps with troubleshooting and forensics as Todd suggested, but it will help IT organizations achieve and enhance accountability. It will help IT do more with less.

Bottom line:

• Log everything all the time
• Capture and centralize archive all logs
• Analyze (search, report, correlate, alert, trend) all logs to understand what's happening in your IT infrastructure
• Continuously monitor the trend (automatically or manually) to ensure no anomalies will occur without notice

Posted by on December 03, 2008 in Log Management & Intelligence | Permalink | TrackBack (0)

BriefingsDirect Transcripts: Systems Log Analytics Offers Operators Valued Performance Insights While Setting Stage for IT Transformation Benefits

Despite growing complexity, IT organizations need to reduce operations costs, increase security and provide more insight, clarity, and transparency across multiple IT systems -- even virtualized systems. A number of new tools and approaches are available for gaining contextual information and visibility into what goes on within IT infrastructure.

IT systems information gushes forth from an increasing variety of devices, as well as networks, databases, and lots of physical and virtual servers and blades. Putting this information all in one place, to be analyzed and exploited, far outweighs manual, often paper-based examination. The automated log forensics solutions that capture all the available systems information and aggregate and centralize that information are becoming essential to efficient IT management.

To learn more about systems logs analytics, Dana Gardner recently moderated a sponsored BriefingsDirect panel discussion podcast with Pat Sueltz, the CEO at LogLogic; Jian Zhen, senior director of product management at LogLogic, and Pete Boergermann, technical support manager at Citizens & Northern Bank.

Listen to the podcast. Download the podcast. Find it on iTunes/iPod.

Read a full transcript of the discussion.

Posted by on September 14, 2008 in Log Management & Intelligence , Project Lasso | Permalink | TrackBack (0)

Logging Poll #9 Analysis: Log Protection and Security

This is the analysis of my last poll; the responses are here and also below.

First, the most obvious conclusion: people still don't care much about log security; I am saying that since this was BY FAR the least popular of my polls. Only 24 people responded, so everything below is pretty unscientific.  A good way to explain it: look at the recent media? Do these people care about their key business data and their customer data security? Nope. So, how do we make them care about securing their log data?

Second,  it is entirely unsurprising that 83% of respondents want "Authenticated access to log server." In fact, I'd opine that 100% of people want authenticated access to any of their servers :-) But, this was my "red herring" to set the baseline for the rest of the questions... 

However, this is where the buck stops: other security measures are notably less popular.

Third, "Logging all access to logs" is my favorite and I am happy to see it reported as popular. But do people really do it?  Do you log access to log server OR log access to actual logs? Think about it... I suspect a lot of people who do the latter still answered "yes" to this question. So, does your system allow you to log all access to log data? If not, get LogLogic! :-)

Fourth,  "Reliable / acknowledged network transfer of log data" and "Encryption of log data in transit " are two true "no-brainer" security features; they took the next spot at 45% and 50% of those who answered. They are simple, they are easy, they make  sense - and, obviously, they don't make logs entirely secure so you need to do more. Why only 50%? Where is THE OTHER 50%?!

Fifth, "all things crypto" are below 40%. "Cryptographic hashing of stored logs", "Cryptographic signing of stored log data" and "Encryption of stored log data" all hover at around 30%. I attribute them to general disregard of log security AND reliance on "system security" (separate server, etc) over "data security" measures for log protection.

Finally, I am embarrassed to admit that I missed  the obvious security measure: "Separate server for logging, not accessible from the Internet." Happily, one of my readers added this using "Other security measures" choice. Indeed, this is a good point - and a good idea to do it. Another option mentioned there was "Destroy old logs."  Indeed, it helps log security by preventing unauthorized access to logs.

Possibly related posts:

• Top 11 Reasons to Secure and Protect Logs
• All other polls and their analysis

Posted by on September 05, 2008 in Innovation , Log Management & Intelligence , Risk Management , Security | Permalink | TrackBack (0)

Anton Logging Tip of the Day #16: Virtually There - Journey Into VMWare ESX Log Analysis

Following the new "tradition" of posting tips of the week, I decided to follow along and join the initiative.

So, after a long delay, Anton Logging Tip of the Day #16: Virtually There - Journey Into VMWare ESX Log Analysis

CISecurity guide for VMWare (here) and DISA STIG for virtual machines (here) both mandate collection and analysis of VM platform logs; none goes into enough details on what to look for in logs. Let's try to shed some light on security-focused log analysis of VMWare ESX v. 3.x logs.

First, at least until ESXi becomes the default choice, one needs to keep in mind that ESX has "Linux-inside" and thus diving into /var/log will not reveal any "alien technology" (well, not much of it  :-)). However, one of the most useful logs is /var/log/hostd.N which is not a descendant of Linux standard logs. Extensive VM event records are written into this file.

Let's focus on various types of logins to the ESX platform and identify logs that indicate a successful and failed attempts to log in. Here are a few useful examples to analyze:

Successful logins:

• May 30 09:20:42 esx2 su(pam_unix)[9405]: session opened for user root by jhonny(uid=1626)

This is a classic Linux root login message; you can watch for these by searching VMWare ESX logs for "session AND opened AND user AND root."  Notice the user name of the user who switched to root.

• May 30 09:20:34 esx2 sshd(pam_unix)[9364]: session opened for user jhonny by (uid=0)

This is also a classic Linux message for a normal (non-root) user login.

• [2008-05-25 06:57:48.774 'ha-eventmgr' 111639472 info] Event 40645 : User jhonny@1.1.1.1 logged in

This is a VMWare -specific application login to ESX. You can track such events by username, by event ID or by keywords "event AND logged AND user" (if you are using search)

Failed logins:

• May 30 09:20:31 esx2 sshd[9356]: Failed password for jhonny from 1.1.1.1 port 54773 ssh2

Another classic Linux message from the ESX system; a failure to login due to incorrect password.

• May 27 12:06:59 esx2 sshd[4756]: Failed password for illegal user jonny from 1.1.1.1 port 30594 ssh2

A message indicating a failure to login due to incorrect username (note a typo).

• May 25 07:03:48 esx1 sudo:     jhonny : 3 incorrect password attempts ; TTY=pts/0 ; PWD=/var/log ; USER=root ; COMMAND=/bin/bash

This ESX Linux platform message should also be familiar to Linux/Unix admins: it indicates multiple sudo password failures; look for such messages in the logs.

BTW, do you need to be reminded to track NOT only failed, but also successful login events?! This applies to virtual as well as physical environments.

Overall, you must prepare for the future by learning to analyze  VMWare logs, just like you handled "legacy OS", such as Linux/Unix and Windows.

As I said before, I am tagging all the tips on my del.icio.us feed; here is the link: All Security Tips of the Day.

Posted by on August 27, 2008 in Innovation , Log Management & Intelligence , Security | Permalink | TrackBack (0)

Logging Stories from the Field

Our brilliant field engineer, Dimitri McKay (his blog) brings another fun and insightful story from the field:  "I recently went on-site for a proof of concept. I’ve always loved these exercises, as it gives me a chance to help a customer see that which was invisible in the past, whether it be virus-outbreaks, users abusing bandwidth via bit-torrent and file sharing, or VOIP phones assaulting DHCP servers for IP addresses. This particular customer had an interesting configuration.  They had been sending their critical/alert and emergency firewall logs to a 3rd party security operations center. That SOC was supposed to monitor the firewall data for any risky traffic, identify any anomalies, and report the instant there’s an issue.

Because of this, we wanted to forward those specific messages on to the 3rd party security operations center. We configured the LogLogic appliance to collect ALL firewall messages, and send off all Emergency, Critical and Alert messages to that 3rd party vendor.  Over the next couple hours, we turned routed stream after stream of syslog data to the LogLogic appliance. Slowly raising our MPS rates, collecting data for alerting, collecting data for reporting; overall collecting data to sift and sort through. We cranked the logging levels way up. Our goal was to "abuse" the box, and get as much out of this POC as possible. The firewalls, routers, switches, and Unix/Linux boxes were logging; we then added several dozen Windows hosts logging via Lasso.

Now came the fun part. We then started drilling down into the data. I illustrated to our customer the agile reports, we ran searches across a mountain of data, and I showed the onlookers what interesting information we could mine from that mass of daily log data.  At one point, I ran an FTP report.

Suddenly there were questions: "Why are there like 450 active FTP connections from Germany?"
Which led to more questions: who is that? what are they attempting to do?

Within several minutes, we were able to see that an FTP server had been left wide open in the DMZ with 'anonymous' logins allowed. We were also able to see the file names being uploaded/downloaded via the firewall port 21 traffic logs.  Next, we began looking at the logs from the compromised server itself.  We were able to see that the server that had been compromised had been actually compromised back in March.

The logs also revealed that they attempted a dictionary attack (over nearly 3 months) hoping to get access to the box, and the fear was: if they accessed to the box, and ran something like l0ftcrack on it, the box had once been part of the domain, and so user accounts and passwords could be revealed.  The box was a virtual machine "test box" that a developer had fired up, added to the domain, didn't harden, and had then transferred it to the DMZ violating a half dozen or more security protocols.  In that same DMZ was their email system, which likely had the same login/pass combo, and if that was compromised, who knows.

At the end of the day we were able to identify a compromised machine, make the security officer look like a superstar, and illustrate just how fast and agile our reports and search capabilities are. If we didn't prove that LogLogic is the clear best fit, we certainly created a perfect use case for Log Management. "

Enjoy more of Dimitri's writing on his blog!

Posted by on August 26, 2008 in Log Management & Intelligence | Permalink | TrackBack (0)

Even More Critical Logging Questions - Answered

I recently did this webcast on logging for accountability (slides and recording here) and people asked a lot of good questions. Here are some of the answers for them as well as our blog readers.

Q1: How do you handle variety of log sources? There are so many, almost beyond my capability.

A1: Sorry to ponder the meaning of "is" here, but what is meant by "handle"? It is really not that hard to collect logs from a large number of diverse sources, given the right tools (as long as the logs can be delivered via syslog or grabbed as files). Now, there will certainly be challenges  when the volume of logs gets large, but if by "handle" you mean "collect + store", it is really not that hard, again, given the right tools. Now, if "handle" means "make sense of what all those logs are trying to tell you," it is a different story altogether. It is indeed pretty hard to extract the meaning of all those logs automatically.

Q2: You talked about the importance of logging; however for an intermediate or novice admin what are the starting steps .. what are the minimal logs they should start at once?

A2: Answered in "Log Management - Day 1" If you want a simple list of logging things to "enable today,"  I cannot really answer it since I know neither your needs, nor your environment. Remember, "requirements first - tools second!"

Q3: What regulations, rules or guidance exist regarding sharing or visibility of logs to users?

A3: PCI DSS says in Requirement 10.5:  "Secure audit trails so they cannot be altered.
10.5.1 Limit viewing of audit trails to those with a job-related need
10.5.2 Protect audit trail files from unauthorized modifications"

NIST guidance for FISMA also says something similar (for example, look in NIST 800-92 doc). Overall, log protection and security are mentioned in many other regulations as well, all the way to ISO and COBIT.

Q4: How I can learn what exactly I need to log?

A4: Let me answer "how can I learn" part and not the "what exactly I need to log part," as it is actually answerable (also see discussion on "MUST-DO Logging for PCI?") . To learn what you need to log, first ask "Why?" (and then see this) - basically establish what you want to accomplish with logs, then catalogue your systems, then figure how to tweak the logging knobs - and then actually go and tweak them.

Q5: What is "more control" and what is "less control" that you mention in the webcast? Can you give an example?

A5: OK, I did say that "sometimes when you implement more controls, you actually have less control." What do I mean? If you buy a firewall (a network security control) and then - over time, of course - configure it with 7800 rules (!) that are supposed to give you control over who can and cannot access your network, you will not gain control over your environment. You will actually be less in control of who is touching your network, compared to, say, having only 20 rules.

Q6: What about mandated NIST controls for government systems? Auditing is a specific control for Moderate and High risk systems. What list of events do you recommend for auditing?

A6: This is too long to answer here, but NIST 800-92 Guide is a really good source of such info ("Guide to Computer Security Log Management [PDF]") Also, see my presentation on NIST 800-92 Guide in the Real World.

Q7: The issue that many organizations get stuck on is the monitoring process, and defining what exceptions to monitor for? Is there guidance for this? How much of it is system specific and how much is applicable generally to all systems?

A7: I outlined some general ideas back in 2004 via this presentation; it is mostly general, but also has pointers to specific system. Keep in mind that it is focused on security, not operational monitoring (which is often no less important - in fact, often MORE important).

Enjoy! Sorry for being brief with some of the answers.

Other questions that I answered in the past:

• More Log Management Questions - Answered!
• Some Burning Logging Questions - Answered!

Posted by on August 07, 2008 in Innovation , Log Management & Intelligence | Permalink | TrackBack (0)

Tomorrow's Logging Problems - Part II

I would like to continue the discussion I started in my previous post called "Today's Logging Problems - Then Future Problems - Part I." Specifically, upon outlining some problems with logging, I will now forecast what will happen with them in 18-24 months.

• Which problems will be solved and forgotten?
• Which ones will simply go away?
• Which ones will persist and in fact increase?
• Finally, which new ones might emerge?

First, I'll predict that "Not knowing what to log" problem will be mostly solved in 18-24 months; at least as far as major regulations go, people will have a pretty good idea a) what the auditors want them to log (and review!) b) what they need to log for solving their problems. Now, esoteric log sources and custom application might still present a challenge from that point of view, but for basic "staples" (firewall, network gear, major OS) the mystery will be over (again, see "Tell me EXACTLY what to log for PCI?" for some reference)

Next, the problem of "Log volume" will definitely get worse, much worse. One might think that 100,000 each second is a lot of log - but there WILL BE more at many companies! Big application log explosion is coming, fueled by the need to address logging in areas where such motivation was lacking before (basically, custom and vertical applications) as well as harness the power of "uncommon" logs for such tasks as fraud analysis or SOA monitoring. Keep in mind that even though in some areas logging is not a preferred way of monitoring and auditing activities (see this discussion on database logs here), application logging will still explode on us ...

The problem of "Log diversity" (the fact that most logs all look different in format and meaning) will get worse before it will get better - and better it WILL get since standards are being developed. We will see people struggling with all sorts of bizarre log data in the coming years. Virtualization (on logs and VM), web services and SOA, various ERP applications and even cloud services will increase the diversity of logging in the coming years.

Similar to the above, a problem of "Bad logs" (ones that are subjective, miss key information, require groping for a crystal ball to understand, turn log analysis into a painful experience or are useless in some other way) will also follow the pattern of the above log diversity problems - it will get worse before it gets better (via the CEE standard effort that now covers the OpenXDAS effort as well!) I noticed that people started asked me questions about "how to do application logging right?" and "what to tell application developers about logging?" which almost never happened in the past. More on this in the future!

"Getting the logs" has gotten much easier in recent years; agentless collectors like Project Lasso (which, BTW, just got updated) and grabbing files remotely via secure protocols made application log collection easier (TCP syslog and buffering also helped). Next, Windows 2008 will make it MUCH easier for the whole Windows kingdom due to their use of web services. However, in the future it might resurface as we try to collect logs from unusual places, again, clouds come to mind as well as virtual environments (e.g. how do you get logs off a dormant VM?). What's the next frontier in this area? Log discovery - automatic finding and identifying log files on systems in order to analyze and retain them.

All this, however, pales in comparison with my favorite "uber-challenge", "Making sense of logs in an automated fashion" - this baby is definitely not going away in 2-3 years. Much more research is needed to make that "log->conclusion" jump automatically without head-scratching, invoking ancient deities and making wild guesses. Once there, we can attempt to reliably handle "proactive logging" (i.e. analyzing various failure or compromise precursors in logs and then predicting the future based on them), another Holy Grail of logging domain.

Anything new will emerge? Yes, I think awareness of the "Logging Gap" problem will grow. "Logging gap" happens when you combine "a need to log" with utter "inability to do so." For example, this will happen when people will know that they HAVE TO log, say, for compliance, but will have no way of doing it due to application or platform limitations. This will become one of the challenges and special "logging add-ons" will appear to close the logging gap and create additional logs where activity audit is desperately needed, but native logging is not helping to achieve it.

Also, I think people will finally wake up to "Log security" challenges - i.e. producing for use as evidence, compliance attestations, etc. Log security is not getting the attention it deserves, but I think this challenge will finally emerge in full force in the next 2-3 years. BTW, my next poll addresses that (vote)

Anything else I missed? Share away!

Related posts:

• Ideal Tool to Solve Real Problems ... of the Near Future?

• Ideal Log Management Tool?

Posted by on August 06, 2008 in Innovation , Log Management & Intelligence | Permalink | TrackBack (0)

Logging Poll #8 - Log Security and Protections

My next logging poll is out - with it I set out to figure out the old mystery of mine, why people don't protect their log data (e.g. see this lamentation "Top 11 Reasons to Secure and Protect Your Logs")

Vote away! As always, results will be posted.

Past polls and analysis are all here.

Posted by on August 05, 2008 in Log Management & Intelligence , LogMatters | Permalink | TrackBack (0)

Today's Logging Problems - Then Future Problems - Part I

Remember my write-up about an ideal log management tool?  Somebody asked me: "That's great that you have such a clear  vision of a future log management technology - but tell me first what future business problems will such 'ideal tool of the future' solve?"  First, I pointed at the fact that there are plenty of log-related problems today which we are not even close to solving. We need to solve the problems of today first, before we can get to solving the future problems.

So, what I consider to be the biggest log-related problems of today?

1. Not knowing what to log - whether  for compliance, tracking attackers or troubleshooting system problems. Remember all the comedy about "Tell me EXACTLY what to log for PCI?" If not, reread it!
2. Log volume  - there are too many log messages (seriously, 100,000 each second is a lot of log - but there is more at large companies!), and, which is worse, a lot of them are of unknown value to the users (might be useful, might not - but you never know in advance); thus, log clutter networks, systems and brains of security/system analysts.
3. Log diversity - logs all look different (at least while standards are being developed) and no single person have the skill set to understand  more than a few types. PIX admin groking SAP logs? No way!
4. In light of the above, just pure bad logs are also a major challenge - logs that miss a key piece of info (like the infamous "login failed" without the username of the user failing to login) or are useless in some other way are sadly common.
5. How about getting the logs from all the places where they are located (think application logs here) - it is a problem if you want to expand your operational awareness to applications.
6. Finally (not really, the list can go on and on), making sense of logs in  an automated fashion is still a #1 challenge  (IMHO) - we are getting better creating tools for humans to go through logs (via reports and search), but log->conclusion process still requires a human, and a darn smart one.

Now, when you read the above think "end user", not "log management  vendor" challenges. Along the same line, this picture from 4th SANS Log Management Survey shows how people perceive the logging challenges:

as well as my logging challenges poll (analysis here):

Now, let's think of logging problems of the near future, say in 2 years.  But you'd have to wait for the next post for this :-)

Posted by on July 31, 2008 in Innovation , Log Management & Intelligence | Permalink | TrackBack (0)

Log Management Project - Day One

Inspired by this and this here (and this too). It started from this example, coming from another domain:

“You’re hired on at a new company placed in charge of securing their online business. You know next to nothing about the technical details of the infrastructure other than they have no existing web/software security program and a significant portion of the organizations revenues are generated through their websites.

What is the very first thing do on day 1?”

At about the same time, I saw a message posted to one of the mailing lists where the poster wondered: "I’ve been asked to look into finding a replacement to our current log management/auditing system.  This is a field I haven’t even come close to touching before, and really don’t know the ideal things to look for (or ignore), etc. I’ve been searching through SANS site as well as googling, and I’m not coming up with a lot of great starter information. " And then he asks "Where should I start?"

This is indeed a really good question!  Let's rephrase the above for the case of logging:

"You’re hired on at a new company placed in charge of TAKING CONTROL OVER THE LOGS. You know next to nothing about the technical details of the infrastructure other than they have no existing LOG MANAGEMENT process and tools... What is the very first thing do on day 1?”

So the "Day 1" of a log management project. What's up?!

The very first thought that should cross you mind before you even do whatever first thing you wanted to do is "WHY?"

"Log management" is a solution, not a problem. What is your problem that you now have a mandate to solve? In other words "Why log management for you?" Logs server way too many different purposes so that proceeding without asking "Why?" is dangerous.

What is it that motivated your boss (or his boss, or whoever) to decide to "address this", to "take control over logs?" Was it a new compliance mandate, PCI perhaps? Was it a recent incident where investigation hit the wall due to utter lack of logs? Was it a new corporation-wide IT efficiency improvement project? Was it a lawsuit where an e-discovery request was not satisfied and thus fine was levied? Was it a hot IT project that is impossible to complete without having a tool to analyze logs?

This "need" is very important since logging is a huge realm and not focusing on the need is akin to starting a journey into a hostile wilderness without  a map - in other words, it might be fun for a while, but it will probably end badly.

Next, what do you actually do first? Figure out what logs are needed for this effort and what systems produce them (and who is responsible for them!) Analyzing SAP logs for J-SOX is a VERY different effort from analyzing Cisco ASA logs for network troubleshooting.

Only at this point you can start thinking about "tools:" parsers, logs, databases, reports, alerts, indexing and other technical things as well as capacity planning, scalability, etc. This is the stage where you learn the lingo and learn to cut through marketing messaging to get to the actual tool capabilities.

So, remember: given mandate to "tame the logging monster", think "WHY?" first!

Posted by on July 29, 2008 in Innovation , Log Management & Intelligence | Permalink | TrackBack (0)

Logging Poll #8 Analysis: Essential Log Context

In my poll #8, I  asked a question: what information is most important when analyzing a particular log record. Live results are here and final count is also below:

What can we conclude?

First, good documentation never hurts  - indeed, the most popular information to look for when facing a new log record is documentation on what it means. While some software vendors are great in this regard, many other don't bother documenting their logs or document them only when customers complain.

Second, I was not sure that the second popular choice would be "Other logs from about the same time (this and other systems)."  This strongly points at huge value of cross-device log analysis (see this recent log entry on that),  where all the logs are consolidated and analyzed together (it goes without saying that time is synchronized OR at least corrected across those logs). Indeed, if you are confused about a log and documentation is not available, reviewing "what else was/is going on?" is smart. Trusting log time stamps across many systems is also key for that.

Third, having IP addresses in logs is great, but human-readable names are better: IPs in logs needs to be mapped to DNS or Netbios names. Indeed, given that often such names reveal where the system is, who might own it, what its function is, etc this information is not just a mapping, but true log information enrichment.

Fourth, so, what's next? The above 3 top responses are indeed universally useful, but the next choice digs deeper: flows, packets, connections and other network information does complement logs and is often studied in combination with logs (e.g. see a strange log entry then go see who connected to the system at that time or where the system itself connected to).

Fifth, next comes a group of pretty much everything else: other logs from the same system, logs about the same system as well as loosely defined 'similar' log entries. These come handy, but are not top choices. In fact,  from this I conclude that people think that A LOT of additional context information is needed to make sense of a confusing log entry.

Sixth, what was surprising? I thought that identity lookups (e.g. IP to real name or other user identity information) would score higher.  I also suspect that people were confused by "logs ABOUT the same systems" (what I meant is, for example, use firewall logs that mention the system which log we are now analyzing) and this should score higher.

Seventh, anything fun in the "Other" category? Yes, there were a few insightful ones: first, results of a Google search (such as for the info from the log entry in question)! Very true indeed. Also named were logs from the same daemon/program (how can I miss it?),  logs from previous incidents and information on the logging system owner.  All very useful indeed. Thanks for good ideas!

Next poll coming up soon!

Past logging polls and their analysis:

Posted by on June 09, 2008 in Innovation , Log Management & Intelligence | Permalink | TrackBack (0)

Cross-Device-Type Log Management vs Device-Specific Log Management

Now, I have to first admit that, in general, dealing with logs on a device-specific basis is a utterly inefficient. What I mean here is when you gather Windows logs in one place, Linux logs in another place, database logs in yet another place; all in different formats, all in different systems not connected to each others, all managed by different people who don't talk to each other (and sometimes hate each other). Basically, this situation is "logs at their worst": all different, all disjointed and, as a result, all next to useless.

However, there are rare situations where you can choose device-specific log management approach without wasting time and money. For example, you might be motivated by the fact that tools that can handle one specific type of log data (e.g. Windows-only, web server-only or Cisco PIX-only) are usually many times less expensive than cross-device log management tools. The table below clarifies it:

Also, while looking at logging tools, one needs to make a distinction between tools that can collect all sorts of logs but only allow you to analyze one log type at a time (e.g. sawmill) vs tools that can collect all sorts of logs AND allow you to analyze all of them together (e.g. LogLogic). The former tools still fall under "device-specific log management" despite their ability to gather hundreds of different log types.

The bottom line: in most cases, cross-device, uniform log management provides huge value, especially if your motivation for log management is compliance or incident response.

Posted by on June 02, 2008 in Innovation , Log Management & Intelligence , LogMatters | Permalink | TrackBack (0)

More Log Management Questions Answered

I did this VERY fun webcast with WhiteHatWorld this week and a lot of good questions about log management came up. I am answering them here for my readers.

Q1: Is a preferred log management program to consolidate the log data and then allow us to review them?

A1: The answer is "Yes!" for a vast majority of use cases consolidating logs work better than the silo'ed approach. Also, this will be answered in a  longer dedicated post within a few days.

Q2: Is it feasible to use a log management tool to try to determine whether application events / failures are being caused by infrastructure issues?

A2:Wow, fantastic question! The answer  to this is "Yes, if you have the right logs collected." In most cases,  to get to the bottom of such issues requires having BOTH application (e.g. PeopleSoft or Oracle) and infrastructure logs (e.g. Windows or Solaris).

Q3: What the typical retention schedule for logs which might be required logs for compliance issues?

A3: I wish I can give a simple answer for this, but there is none. Well, PCI DSS makes it simple: 1 year for logs from in-scope systems. Other regulations are not as clear and the numbers, or - more often! - guesses at such number range from 90 days to 7 years and more.  90 days to 1 year is a common retention policy for security (on the longer side of this range) and operationally (on the shorted side of this time range) useful logs. Check this out for a few ideas for long you might need the logs.

Q4: Once you have logged the events, what do you do with them?

A4: This question truly opens up a whole Universe of questions, issues, challenges, etc. But here is my attempt at a short answer: a) you collect the logs and now you can search through them in case you need (incident, system failure, etc) to b) you summarize them and notice the trends - overall know what is going in your environment c) you analyze them in real time to trigger alerts on "critical" log messages - failures, attacks, etc.  See this slide deck for some useful pointers.

Q5: Why do I create a log policy? 

A5: Log policy is a clear and simple document that show what you log on each system (and why): it helps you to configure logging across all the systems as well as helps to know what information you have in your environment (should an auditor ask, for example). A log policy also defines log retention, log review practices, etc. NIST 800-92 Guide to Security Log Management  [PDF] is a good source of info on this subject.

Enjoy!

Posted by on May 23, 2008 in Innovation , Log Management & Intelligence | Permalink | TrackBack (0)

Anton Logging Tip of the Day #15: Fear and Loathing in Event 560 (and 562 and 567)

Following the new "tradition" of posting a security tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it "pay it forward" to the community.

So, Anton Logging Tip of the Day #15: Fear and Loathing in Event 560 (and 562 and 567)

This tip digs into a seemingly simple, but really VERY esoteric subject: monitoring file access and modification via a Windows event log. Now, some people - who never studied this subject - tend to have a very simplistic view of this: just enable Object Access auditing, then right-click on a file or directory, click Security->Advanced->Auditing and then pick what types of events will be logged and by what accessing entities (i.e. users or computers). OK, so this will produce some logs, that is for sure. But are they useful?

First, why are we doing this? We typically need to know the following when we audit file access in Windows (or any other OS for that matter) for security (monitoring and investigation) or compliance:

• Time/date
• Computer where it happened
• User who touched the file
• Application he used to access the file
• File name + location (directory, share, etc)
• Type of access (read, write, create, delete, etc)
• Status (i.e. success or failure)

Can we get this from the above logs? No.

What? No!?! Really?

Yes, really. We can get some of the above, some of the time, not all of the above, all of the time. Here is an example, we are looking at event ID 560 (picture) and then at an extract from its description field.

Event:

Description (selected field):

Object Server: Security

Object Type: File

Object Name: C:\0\TestBed\simple_text_file.txt

Image File Name: C:\WINDOWS\system32\notepad.exe

Primary User Name: Anton

Primary Domain: XXXXXX

Accesses: READ_CONTROL

SYNCHRONIZE

ReadData (or ListDirectory)

WriteData (or AddFile)

AppendData (or AddSubdirectory or CreatePipeInstance)

ReadEA

WriteEA

ReadAttributes

WriteAttributes

WTH is that? Well, we know that the user  'Anton' has successfully read? wrote? changed attributes? did something? with a file named "C:\0\TestBed\simple_text_file.txt" using a program named "C:\WINDOWS\system32\notepad.exe." That's the best we can get, in this case! We may try to look at event IDs 562 and 567, but this missing information (i.e. the exact action performed) will not be added.

BTW, there will be  a few more dozen (sometime hundreds!) of the 560s, 562s and 567s  produced - all from just opening the text file in a notepad. The above event is notable for having BOTH "notepad" and "simple_text_file.txt" in the same event; others will have either of the two.

Anything else gets in the way? Yes, lots! MS Office will write to all files, even just opened for reading (with no user modifications to the content whatsoever), which will screw up your log monitoring efforts. If the file is on a share, more information will be missing (e.g. username might be).

So, how to use Windows event logs for file access tracking?

1. Enable logging (as described above)
2. Pick events 560 (most useful) and 562, 567 (useful too)
3. Look for fun filenames that might be touched by the users (have a list of files and users handy)
4. Figure out what programs were used to access them (this is called "Image File Name" in "WinLogSpeak")
5. Ponder the 'Accesses' section of each event until your brain turns blue :-) or until you decide whether such access is authorized or not...

Overall, this is still very useful for file access monitoring, but the process is somewhat painful.

BTW, I am tagging all the tips on my del.icio.us feed. Here is the link: All Security Tips of the Day.

Posted by on May 08, 2008 in Compliance , Innovation , Log Management & Intelligence , Security | Permalink | TrackBack (0)

Logging Poll #8 Context for Log Analysis

So, my next poll is up - and it is fun (but more technical): what information is most useful when trying to make sense of a log entry?
Vote here! Analysis will be posted here in a few weeks.

Past polls:

Posted by on May 05, 2008 in Innovation , Log Management & Intelligence , LogEd | Permalink | TrackBack (0)

Critical Log Management Questions - Answered!

Here are some interesting log management questions I got asked some time ago; reposting here for our blog readers.

Q1: For those companies that have successfully implemented enterprise-wide logging, what were the big nasty surprises that they encountered?

A1: Here are a few:

• political boundaries within the organization: "these are our logs, and you are not getting them"
• privacy laws: some logs cannot be collected in some countries; some cannot cross the border, some cannot be seen by some people, etc. This is true mostly in EU, less in US.
• legal blocks: work with legal before deploying any org-wide log management; legal might try to prevent certain data from ever being created (for fear of being legally discovered later)
• log volume: underestimating log volume is common and pretty nasty
• related to the last one: vendors being "optimistic" about their tool scalability
• time synchronization (of course!), specifically, lack thereof.

Q2: For those companies that have successfully implemented enterprise-wide logging, what was their implementation approach?

A2: Typically, 2-3 vendor PoC or pilot first. Then with the chosen vendor: phased approach based on location + type of log source (e.g. firewalls, then routers, then OS, then proxies, etc) + network topology (e.g. DMZ, then internal) + log source criticality (e.g. critical servers first; the rest next). This might be handy to look at.

Q3: What kind of storage requirements have been experienced by those organizations who have successfully implemented enterprise-wide logging?

A3: Massive? :-)

Here is a simple example: PCI DSS is a bit more aggressive than NERC since it mandates 1 year of log retention vs NERC 90 days, so: 1 year worth of logs is = 365 days x 24 hours x 3600 seconds x 1 (one!!!) busy firewall with 100 log messages each second x 200 bytes per message average (e.g. valid for PIX and ASA devices) = 588 gigabytes / year of raw log data uncompressed (assuming 10x compression you'd get about 60GB of compressed log data per year)

Store it in RDBMS? Multiple it by 2-3. Have an index? Add about 30%.

The bottom line is: terabyte is the unit to measure logs.

Q4: At the organizations that have successfully implemented enterprise-wide logging, how logging impacted network and system performance?

A4: Too broad a question, so here are a few pointers:

• logging affects performance much more on some types of systems compared to other types: most painful examples are databases where some people (can't find a link...sorry) report performance loss of up to 40% if logging all SELECT statements and other data retrieval commands (you need to log selectively on these); in other cases (e.g. web servers) there is no performance loss and logging is "always on"
• log collection: agents impact system performance (long post on this subject): a little when they run (everybody knows this) and A LOT when they crash (few people think about it - agent software memory leaks are not uncommon); unlike agents, remote agentless log collection barely affects system performance (unless you have one of the few esoteric cases)
• log transfer and network performance: look for compressed (logs compress really well), TCP-based transfers; syslogging over UDP uncompressed has a chance of doing a pipe saturation DoS on your network. Yes, people say "use a dedicated LAN," but this is definitely wishful thinking for many. Also, raw UDP syslog in large quantities over WAN = insanity :-)

Q5: What were some successful strategies for obtaining buy-in from system owners and operators in regards to turning logging on?

A5: OK, also too broad a question, but here are some pointers:

• provide them a useful service based on their logs (e.g. performance measurement, availability monitoring, compromise detection :-), or other security metrics, etc)
• help them with their compliance mandates (e.g. create reports that they can show to the auditors that "bug" them)
• give them tools to better solve their problems (e.g. allow access to a log management tool so that can investigate issues better, search the logs, check on their users, etc)

Q6: How the organizations that have successfully implemented enterprise-wide logging dealt with unusual devices (=log sources) that have no log management vendor support?

A6: They were in massive pain - if they choose a log management vendor wrong. You need to look for vendors that have "universal log source support" with NO requirement for a custom rules or custom collector/connector/agent development. LogLogic have generic text log collectors that can grab and analyze unknown logs. Typically this is done via some form of text indexing that works across all logs, including those from unknown, vertical, esoteric or custom-developed log sources

Hope it was useful!

Posted by on April 24, 2008 in Compliance , Innovation , Log Management & Intelligence , LogEd | Permalink | TrackBack (0)

From Log Apathy to Log Enlightenment

So, I was talking to this small log management vendor the other day and he confided to me that his product faces fierce competition in his target market (which is, important to note, small to medium companies with 10-100 systems): and this competition is apathy.

More specifically, his prospects either just blow him off by saying "pah, who needs logging!" or they profess their undying love of all things logging - and then still don't buy his product (which is priced, shall we say, "to go")

Admittedly - and somewhat tongue-in-cheek, these are the same companies that form the core of today's botnets (due to various reasons including their scarce resources) and enable RBN to deliver high-quality malicious services to criminal enterprises worldwide. Still, if you happen to have thoughts along the line of "who needs logs?" or "ah, logging? it will come later!", you really deserve a nice fat check from RBN and other malicious "hacking" syndicates since it is extremely likely that your overall attitude towards security is just as misguided...

But how to progress from such ... what was before the Stone Age? ... Sharpened Stick Age? to modernity? Most companies go through the following stages in regards  to their logging:

1. Deep log ignorance: "Logs? What are those?"
2. Shallow log ignorance: "Later...later...later... #37 on the TODO list."
3. Log collection: "We gather and store dead log data...cold."
4. Log searching: "We will dig into the pile when we have to ... hopefully never!"
5. Log analysis and reporting: "We know our logs - and what they mean"

(also see my post "Natural Flow of Log Management" for some specifics)

Of course, compliance (PCI DSS and others) helped move people from 1. and 2. to 3., but, sadly,  people often get stuck at 3. (just collection) or 4. (collection  + maybe search) and never progress to Logging Enlightenment of 5.

Yes, PCI DSS and other regulations mandate not just log collection, not just dead cold log storage, but also log review (daily, in case of PCI DSS Requirement 10), but "review" happens to be the item that gets overlooked  all too often.

Why is that?

I think the reason is that log analysis is still too hard and still not automated enough for an average organization. Yes, I did see some corporations that built their own log analysis systems that - surprise! - exceeded the best available from the vendors [at the time]. However, a typical company IT department would not have Ph.D. poring through hardcore text mining research papers in order to improve their home-grown log analysis AI. They expect the vendors to  eat the logs, chew on them for a bit - and then spit out the answers.

Are we there yet? No, but we will be!!!

Posted by on April 22, 2008 in Compliance , Innovation , Log Management & Intelligence , Security | Permalink | TrackBack (0)

RSA 2008 Summary

So, RSA 2008 has come and gone.  Now that everybody has recovered from the information overflow, it is time for summaries and reflections. Before we begin, go read my RSA Impressions (Part 1,2,3,4). Next, read what others said about RSA 2008 (via my del.icio.us feed). Then reflect on past RSA shows (2006, 2007).

Ready now?

First, what was the theme? Unlike in the past, I personally couldn't pick any. The candidates were GRC (and compliance + risk management) , DLP (and other data-leak/-theft technologies), IAM (and identity enablement), but none quite measured up to being a show "theme."

What I saw too much off? Even though their numbers have shrunk, I still saw too many NAC vendors there (Lockdown, here we come!). One of my friends joked that there were more "GRC vendors" than NAC vendors, but both were in low enough numbers to make it a trend. As far as loud noises back from RSA 2007, "identity-driven this or that for security" was still very visible. What is also bizarre is that people still start log management companies. I saw a couple of new ones - amazing, isn't it? Yes, it is a hot space, but come on! Accept that LogLogic is the leader and be done with it :-)

Overblown messages? "Information-centricity." It was cool and hot :-) and new when Hoff said it.  But when it trickled down to keynotes of some "trailing edge" vendor executive, it became boring and stale. And, while we talk about "information centricity" people must still worry about "A" (availability) first (see this discussion).

What I didn't see enough of? VOIP security. Somehow this previously hot trend is quieted down. Also, I saw a lot of web application security vendors, but I think that is still not enough as this is an area with a raging fire, not just "some hotness." Also, I expected to see more vendors messaging around (and, actually helping with!) fraud. Dan Geer's Verdasys kinda mentioned that, but pretty much in passing. Is fraud handled outside of security (and thus out of RSA)? I am not sure.

What I didn't see at all? I didn't see much "market consolidation" - no huge deals, no vendors of note "taken out", etc. Still a huge number of security companies around ... some with products that seem deeply out of touch with today's threat landscape.  One of the speakers said that nowadays "no single security pure-play expected to change the world", but it sure seems like many will try!   Along the same line, Mike R said that such shows are 18-24 months ahead of what "normal" people deploy. This might explain the VOIP and other missing items.

As I said before, "consumerization" of IT - i.e. IT infrastructure, servers, laptops, storage, services, computing resources, applications, etc provisioned outside of IT departments was an elephant in the room. It is not simply "unmanaged IT" or "consumer-grade IT for business", it is the whole "not-IT-department IT" phenomenon. Yes, via mashups it even includes "non-IT application development" (read this fun 451Group report on that). Security implications of this are nothing short of ginormous.

In light of this, I liked how one presenter said this: "we lost the desktop" - meaning "1/3 is managed by users [not IT], 1/3 is unmanaged and 1/3 is compromised/ "managed" by hackers."  Sad but true... Dave Aitel used to joke how in the future banks will have to "re-compromise" your PC so that some temporary security can be established for you to transact business with them. Are such horrifying times upon us already? Maybe desktop virtualization will help with this ...

Overall, RSA 2008 show was an enjoyable, enlightening and thought-provoking experience, as usual. I've heard it was the same even for people who didn't attend a single presentation - indeed, RSA is about people, not slides...

See you next year at RSA 2009!

Posted by on April 16, 2008 in Compliance , Innovation , Log Management & Intelligence , Risk Management , Security | Permalink | TrackBack (0)

On Trusting Log Timestamps

It is about time: specifically, timing logs. As I said in my Log Trust and Protecting Logs from Admins posts, the issue of trust is critical in the logging world. After all, logs = accountability; and the latter in unthinkable without trust. If we are to at least pretend that logs objectively record events and user actions, we need to unambiguously establish WHAT happened and WHEN. This post deals with the 'WHEN'  issue.

So, can we trust that the time stamp in the log file or the one added by the log management system correctly describes when the recorded event actually happened?

We will start from locating the timestamps in logs. Most of the log formats, such as file-based logs (web servers and proxies, various application, some security gear, etc) and syslog, Windows event logs, database audit tables, etc contain a timestamp. Here are a few examples:

1. Mar 20 10:03:47
2. 19/Mar/2006:04:27:33 -0800
3. Sun Mar 19 04:45:19 2006

In fact, once I saw somebody use a timestamp to define logs as "timed records of IT activity." So, time is critical for logs being, well, logs :-) At this point it is worthwhile to note that file-based logs will contain a timestamp IN the file, while syslog records arriving over the UDP or TCP port 514 connection are usually timestamped upon arrival BY the syslog daemon (using its own "knowledge" of time) - and then it shows up in the syslog files in /var/log.

Let's assess whether this "in-log timestamp" provides an adequate way of timing the actual events that are being logged. Answering this question is important for investigations and troubleshooting, but becomes  nearly a matter of life and death in case of log forensics. Here are some fun cases and issues to consider:

First, what are the chances of a  completely false timestamp in logs (BTW, today is Jan 1, 1970!) When might that happen? Typically when a logging system own clock is reset or not set correctly. This timestamp clearly should NOT be trusted. How do you then time this event? This is a separate story that will be discussed some other time...

Second, it’s always 5PM somewhere: in other words, what time zone are your logs in? EST? PDT? GMT? UTC? Or any of more than 24 other possibilities. If you have no idea, you should not trust the timestamp.

Third, are you in drift? Is your system clock? Those pesky drift seconds turn into minutes that then work to undermine the accuracy of timing the records (and thus your certainly and trust in evidence quality)

Fourth, syslog forwarder mysteries are plenty: some of the syslog messages will be delayed in transit and then  be timestamped by the final recipient daemon, thus completely losing when the event was originally logged. Admittedly, this delayed syslog messages are rare, but as more people employ buffering syslog daemons (e.g. syslog-ng), it might happen more often.

Fifth, more esoteric, but still real (and really annoying!): some system logs will contain two timestamps. If you don't possess in-depth knowledge of this specific log, confusion has a chance to undermine the trust as well (so, which timestamp should I use?)

Sixth, most people will not think that they will fall to something that stupid: 24 vs 12 hour time. However, when facing an unknown (and poorly designed!) log format, beware that 5:17 might well be 17:17...

Finally, if you know that something got logged at 5:17AM, then when did it happen? Beware of "Log lag!" This issue is actually too tricky to give it justice here... The simplest example is when the process leaves a log record when it exits, not when it starts, which might happen days earlier (thus creating a log lag).

As we dive into more issues with timing logs, we also need to think about sequence timing and absolute timing. Sequence of logged events is a critical fact! Miss the sequence and the whole “house of cards” goes …  But! Absolute time is also important! Can we be assured of both being correct all the time? (hint: no)

So,  when you look at logs next time and you see a timestamp there - start thinking about all of this :-)

Posted by on April 04, 2008 in Innovation , Log Management & Intelligence , Risk Management , Security | Permalink | TrackBack (0)

Quick and Fun Poll on Behavior Change

While I am analyzing my previous poll, here is a quick and fun one: do we change our behavior when monitored? Vote away!

Posted by on March 27, 2008 in Compliance , Log Management & Intelligence , LogEd , LogMatters , Security | Permalink | TrackBack (0)

Anton Logging Tip of the Day #14: More access_log Fun: What Are You Not GETting?

Following the tradition of posting a tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it "pay it forward" to the community.

So, Anton Logging Tip of the Week #14: More access_log Fun: What Are You Not GETting?

In this tip, we will look at some bizarre artifacts that show up in web server access logs today. Here we have a production log from an Apache web server that is full of interesting (and sometimes ominous!) little mysteries that we will investigate in order to determine their impact on security and operational health of the site.

Logs do contain more mysteries than we have time, so we will focus on a few of them: specifically, unusual web request methods.  Let's see who is trying to POST or use some other method (OPTIONS, HEAD, PUT or something - see a list here) on our site, instead of just GET'ting the content (GET command is used by web browsers to retrieve the pages, while POST is used to upload content, press buttons, etc  - at least in "web 1.0" land  - see earlier tip #12 where POST request was found in proxy logs)

Here is one little artifact that attracted my attention due to a POST request vs a web forum as well as a battery of slashes (which actually increases in subsequent request - of which there were many)

10.10.102.250 - - [12/Feb/2008:16:10:50 -0500] "POST /phpBB3////ucp.php?mode=register&sid=e5efaa77a777066c61f71808e9e57b19 HTTP/1.0" 200 14397 http://www.example.com/phpBB3///ucp.php?mode=confirm&id=7640df05c7e24b7acf7a68800fe6dc59&type=1&sid=e5efaa77a777066c61f71808e9e57b19 "Mozilla/5.0 (Windows; U; WinNT4.0; en-US; rv:1.2) Gecko/20021126"

... more...

10.10.102.250 - - [12/Feb/2008:16:12:29 -0500] "POST /phpBB3///////////////ucp.php?mode=login&sid=e5efaa77a777066c61f71808e9e57b19 HTTP/1.0" 200 9355 "http://www.example.com/phpBB3//////////////ucp.php?mode=login&sid=e5efaa77a777066c61f71808e9e57b19" "Mozilla/5.0 (Windows; U; WinNT4.0; en-US; rv:1.2) Gecko/20021126"

This one really is a mystery; what do we know about it? The server responded to the request OK (code 200), so the POST actually happened. The first request was a request to register with a web discussion board and the second was a request to login. Multiple slashes are  actually ignored  by the web server, so why put them in the request (no answer)? Also, I think that the User-Agent is spoofed ... do you know why? Finally, if I see something like that in my logs, I will definitely investigate it, primarily due to the fact that Apache responded with 200 OK code.

The next one is so classic it it dumb (and so dumb, it's a classic :-))

10.10.123.226 - - [12/Feb/2008:03:46:54 -0800] "POST /_vti_bin/shtml.exe/_vti_rpc HTTP/1.1" 404 - "-" "MSFrontPage/6.0"

10.10.123.226 - - [12/Feb/2008:03:46:55 -0800] "OPTIONS / HTTP/1.1" 200 20210 "-" "Microsoft Data Access Internet Publishing Provider Protocol Discovery"

It is probably one of the ancient IIS attacks (check out this fun BlackHat preso on that, circa 2003) - why would someone probe for it now is beyond me. In any case, Apache on Linux and "*.exe" don't mix :-)

The final log record is also fun:

10.10.101.222 - - [12/Feb/2008:15:33:22 -0800] "PUT /zk.txt HTTP/1.0" 405 223 "-" "Microsoft Data Access Internet Publishing Provider DAV 1.1"

The above uses a PUT request which is pretty much deprecated now; the purpose of the above is clearly malicious. In fact, modern Apache shouldn't even allow it, thus it responds with code 405 "Method Not Allowed." Nothing to worry about (even though some poor critter got owned with that! BTW, if you follow that link, check out HTTP response code 201 - if you see it in your logs, run! :-))

Overall, if you see too many POSTs or too many "GET then POST" sequences from the same IP in rapid succession, investigate it since no legitimate access should produce such a pattern...

As further reading, I heartily recommend this paper: "Detecting Attacks on Web Applications from Log Files"

Also, I am tagging all the tips on my del.icio.us feed. Here is the link: All Security Tips of the Day.

Posted by on March 12, 2008 in Innovation , Log Management & Intelligence | Permalink | TrackBack (0)

Poll #7: What tools do you use for Windows Event Log collection?

My next fun logging poll is here - please vote! It is about tools for centralized collection of Windows Event Log from servers and other systems. One of the somewhat surprising discoveries from my previous poll was that few people look at Windows logs; this poll drills down into it.

And, don't forget that ProjectLASSO Windows event collector allows people to grab Windows event logs remotely without those hated agents...

Past logging polls and their analysis:

Posted by on March 07, 2008 in Innovation , Log Management & Intelligence , Project Lasso | Permalink | TrackBack (0)

Logging Poll #6 "Which Logs Do You LOOK At?" Analysis

This poll on looking at logs  poll was relatively popular; lets see what we can learn (live results are also here).

First, what are the top 3 log types that people look at? They are:

1. Unix/Linux server syslog
2. Web server logs
3. Firewall logs

How does that compare with the top 3 log types that people collect (see picture showing results from my previous poll below)?

These are:

1. Unix/Linux server syslog
2. Firewall logs
3. Web server logs

Huh? They are the same - doesn't it just make sense? What are the possibilities here?

a. People only collect the logs they plan to look at, OR

b. People only look at logs they collect (duh!).

Strangely, I find a) unlikely; I think most people collect more than they can review and that the incident/issue response and compliance needs drive collection more than review or analysis.

Another observation is that all of the "big 3" log types are useful for security, operations and compliance and not just for security (like NIDS/NIPS logs). Is that why they are so popular?

Second, I was fearful that "I only look at whatever logs needed for the incident/issue investigation" will win. It didn't!!! This to me indicates that proactive log review is not as unpopular as I feared. Good! It is working.

Third, obviously, nobody (well, 4%...) looks at all logs they collect.

Fourth, much more people look at Unix/Linux logs than Windows server logs (factor of 3x); this is not entirely unexpected and my next poll will drill down into this.

Finally, I am SHOCKED that people don't look at NIDS/NIPS logs (only 11% do). Why have you deployed those "beasts" if you don't look at what they produce? Then again, maybe you haven't...

Next poll coming up!

Posted by on March 07, 2008 in Innovation , Log Management & Intelligence | Permalink | TrackBack (0)

Audit/Monitor Controls or Audit/Monitor BEFORE Control?

Back in 2004, Forrester paper called "The Natural Order Of Security Yields The Greatest Benefits" proclaimed that "the adoption of security has a natural order: 1) authentication; 2) authorization; 3) administration and 4) audit." Note that audit which, in this case, broadly includes audit, monitoring and detection, comes last. It seems to be fairly in line with common sense: you audit the controls after you put them in place; you monitor after you have authentication and authorization taken care of and you detect the violations after you organized your administration.

The paper clarifies: "With people and contexts defined, protective controls in place, and policies outlined, the fourth set of questions [i.e. regarding the audit] includes: “What happened?”, “What is happening?” or, especially, “Is it working?”

However, is this really so? Or, is this always so? First, when reality collided with plans,  many of the organizations that followed that wisdom got mired in one phase (e.g. in trying to get authentication under control) and ended up having no audit whatsoever: in other words, they are flying blind while implementing controls!  Second, in some cases controls (authentication, authorization, administration) will actually be impossible to implement, while audit will be possible. Imagine retrofitting a legacy application for granular authorization? Third, in some cases implementing prevention/control will be much more complicated, compared to implementing audit: thus, people will face a choice between a half-baked control to a full-blown audit capability? An example will be managing which file each user can access vs monitoring/auditing which file each user have accessed. The latter is doable, while the former is next to impossible. Another way to phrase it is "reactive but possible" vs "proactive but impossible" (hint: pick the former :-))

I think the idea of putting audit first in some cases is the way we'd need to progress. "Wow, what a blasphemy!", some might say ...  After all, if you have not defined controls, what are you going to audit? But remember that audit is meant broadly in this context and thus the opposite question is very relevant: what are you going to control if you have no idea what is going on? People sometimes define a security policy based on how things should be (and then WSJ happens - refresh your memory of the "WSJ saga" here), but then spent years trying to bring policy and reality together (and end up with an environment which is "half-controlled") Won't it be better to audit first, then control?  Or, at the very least, to run BOTH project at the same time so that improvements in control are audited and audit discoveries lead to control improvements!

Obviously, "do IDS before IPS" falls under the same principle: monitor first, [try to] control second. Here is another example: implement log management before identity management. Looking at logs will tell you what privileges the users actually use for doing their daily jobs. Then you can mix it up with what the idea access policy will be.

So, think about it! Questioning the common wisdom does often bring interesting insights.

Posted by on March 06, 2008 in Innovation , Log Management & Intelligence , Security | Permalink | TrackBack (0)

Audit/Monitor Controls or Audit/Monitor BEFORE Control?

Back in 2004, Forrester paper called "The Natural Order Of Security Yields The Greatest Benefits" proclaimed that "the adoption of security has a natural order: 1) authentication; 2) authorization; 3) administration and 4) audit." Note that audit which, in this case, broadly includes audit, monitoring and detection, comes last. It seems to be fairly in line with common sense: you audit the controls after you put them in place; you monitor after you have authentication and authorization taken care of and you detect the violations after you organized your administration.

The paper clarifies: "With people and contexts defined, protective controls in place, and policies outlined, the fourth set of questions [i.e. regarding the audit] includes: “What happened?”, “What is happening?” or, especially, “Is it working?”

However, is this really so? Or, is this always so? First, when reality collided with plans,  many of the organizations that followed that wisdom got mired in one phase (e.g. in trying to get authentication under control) and ended up having no audit whatsoever: in other words, they are flying blind while implementing controls!  Second, in some cases controls (authentication, authorization, administration) will actually be impossible to implement, while audit will be possible. Imagine retrofitting a legacy application for granular authorization? Third, in some cases implementing prevention/control will be much more complicated, compared to implementing audit: thus, people will face a choice between a half-baked control to a full-blown audit capability? An example will be managing which file each user can access vs monitoring/auditing which file each user have accessed. The latter is doable, while the former is next to impossible. Another way to phrase it is "reactive but possible" vs "proactive but impossible" (hint: pick the former :-))

I think the idea of putting audit first in some cases is the way we'd need to progress. "Wow, what a blasphemy!", some might say ...  After all, if you have not defined controls, what are you going to audit? But remember that audit is meant broadly in this context and thus the opposite question is very relevant: what are you going to control if you have no idea what is going on? People sometimes define a security policy based on how things should be (and then WSJ happens - refresh your memory of the "WSJ saga" here), but then spent years trying to bring policy and reality together (and end up with an environment which is "half-controlled") Won't it be better to audit first, then control?  Or, at the very least, to run BOTH project at the same time so that improvements in control are audited and audit discoveries lead to control improvements!

Obviously, "do IDS before IPS" falls under the same principle: monitor first, [try to] control second. Here is another example: implement log management before identity management. Looking at logs will tell you what privileges the users actually use for doing their daily jobs. Then you can mix it up with what the idea access policy will be.

So, think about it! Questioning the common wisdom does often bring interesting insights.

Posted by on March 06, 2008 in Innovation , Log Management & Intelligence , Security | Permalink | TrackBack (0)

Poll: What logs do you actually LOOK at?

This is my 6th logging poll (vote here now!)- links to the previous five polls below.

This one is deceptively similar to the #1 below, but it is not. This poll is What logs do you actually LOOK at? and not Which Logs Do You Collect? In other words, are you a log packrat? Are you collecting and never using the log data? You are making a mistake, if you don't.

Past polls:

Posted by on February 14, 2008 in Log Management & Intelligence , LogMatters | Permalink | TrackBack (0)

Logging Poll #5 "Top Logging Challenges" Analysis

OK, this poll was insightful! The raw results are here and below:

What can we learn from this?

First, what are the top challenges? It is with great regret :-) that I report that the #1 challenge is exactly the one I thought it would be: We collect logs but don't have time/resources to look at them. Yes, automated "analysis challenge" has only become more of a challenge as people deploy more tools that enable log collection on a massive scale (e.g. 75,000 logs/second). I dare to predict that we will finally have to tackle this one in the next year or two. In fact, this challenge rears it ugly head via another popular response, Lack of log analysis tools, which made Top 5 responses as well.

Second, even though I didn't make any predictions about the #2 entry, but I was surprised: No way to effectively search all logs is a  very close #2 (obviously, 1 vote is not statistically significant here). Indeed, log searching is an elusive little problem, especially when we want to do it fast and on a large pool of logs. Even though I think we need to search less and discover more, the need to search logs will be with us forever (and, no, I don't think you need a special product just to do that ...)

Third, I am happy to report that this poll indicates that we finally broke the back of "the beast" of  not having logs. Responses that point at not having logs (e.g. Logging is not enabled, We don't know what logging we must enable,  etc) are not terribly popular (then again, maybe it is due to self-selection of the blog readers ...)

Fourth, infrastructure! Specifically, No infrastructure to manage the log volume we have is very popular as well (#4). This proves the point that I used to not take very seriously in the past (by mistake): when megabytes become gigabytes and those grow into terabytes, many things that used to trivial (e.g. moving logs from A to B, saving logs to disk, etc) become grand engineering challenges... Indeed, to manage high volume of logs you need a scalable log management solution (example)

Sixth, as I lamented, few care about log security (this counts as laments, I guess).  Secure storage of logs is only bothering a few people. One word: yet! As of today, stored log hashing + (sometimes!) log transport encryption + (rarely!) encrypted archives are the state of the art.

Next poll is coming up!

Posted by on February 08, 2008 in Innovation , Log Management & Intelligence , LogMatters | Permalink | TrackBack (0)

SANS Security Laboratory Thought Leadership Interview With Dr Anton Chuvakin

Here is an insightful interview with me done by Stephen Northcutt at SANS. I share a bunch of thoughts on logging and log management. For example, what is my #1 logging pet peeve, what's the #1 logging mistake, will we ever see log standards, why are we looking at an increase in the number of log types we need to look at, etc.

It starts like this: "Dr. Anton Chuvakin from LogLogic has agreed to be interviewed by the Security Laboratory and we certainly thank him for his time! He is probably the number one authority on system logging in the world, and his employer is probably the leading vendor for logging, so we appreciate this opportunity to share in his insights."

Posted by on January 31, 2008 in Innovation , Log Management & Intelligence | Permalink | TrackBack (0)

Fight the Log Silos!

While the world of logging is full of inconsistencies and troubles (e.g. ugly logs!), there is one that beats many others: siloed approach to logs!

There is little that I hate more than  siloed approach to logs. A situation when you have your security team "owning" network IDS logs, network team having firewall and router logs (as well as all SNMP traps) and, say, a system admins possessing  the logs from servers and desktop is not only sad, counterproductive, inefficient and wasteful, but also dangerous.

Where does such approach to logs (when they are divided by both technical and political chasms!) breaks down most painfully? In case of an incident response, of course. This is where instead of one query across all logs and all log sources (or whatever needed subset of logs or log sources), you'd end up with having run around, beg, connect, wait, swear, wait, download logs, dig in many places at once, wait, grep, suffer with many UIs, swear more, etc. All of the above instead of connecting to your shiny new log management system and running a few reports, drilldowns and searches across the relevant logs!

Where else does it break down? Compliance, of course! Most regulations and mandates don't call out logs by the type of the log source,  but apply to all logs across. Thus one system to verify the compliance status is much more productive compared to digging in many systems.

Ideally, you'd break down the silo walls by deploying a log management platform across the entire organization and then letting every team that needs logs to get them from the system in a controlled fashion, via the interface or a web API (BTW, LogLogic platform has a web API to get logs!). Apart from being a trend (e.g. see recent ESG report on that), it will make your IT and security operations that much more efficient - and pleasant!

On the other hand, what is bizarre is that some newer vendors,  who claim to do log management, actually work to propagate, not combat, the siloed approach. For example, selling the tool for $5000 to each of the many separate teams within the organization IMHO must be made illegal :-) as it builds walls, not bridges; digs holes and overall "silo-izes" your IT operation...

Posted by on January 25, 2008 in Log Management & Intelligence | Permalink | TrackBack (0)

NERC CIP Rules Out - Logs In!

NERC security rules [PDF], that were updated and became mandatory last week, might well become "a new PCI DSS" and trigger "a golden age" of security in the energy industry: the rules are mandatory, they are specific (more specific than a lot of other regulatory security guidance) and there is an enforcement body (NERC) that can make life miserable for those not complying. 

Here are some log-related examples from the guidance:

"R5.1.2. The Responsible Entity shall establish methods, processes, and procedures that generate logs of sufficient detail to create historical audit trails of individual user account access activity for a minimum of ninety days. "

"R6.4. The Responsible Entity shall retain all logs specified in Requirement R6 for ninety calendar days."

"R6.5. The Responsible Entity shall review logs of system events related to cyber security
and maintain records documenting review of logs. "

So, again: have logs, retain them ("Top 11 Reasons to Collect and Preserve Computer Logs") and review them ("Top 11 Reasons to Look at Your Logs"). Or, better, have a log management tool do it for you!

Posted by on January 24, 2008 in Compliance , Log Management & Intelligence , LogMatters | Permalink | TrackBack (0)

Poll: What are your top challenges with logs and logging?

This poll is especially fun: What are your top challenges with logs and logging? Vote here.

Past polls were:

Posted by on January 21, 2008 in Innovation , Log Management & Intelligence | Permalink | TrackBack (0)

Logging Poll #4 "Who Looks at Logs?" Analysis

Time to analyze my final 2007 poll on logs. In it, I asked who actually looks at logs at the organization. Here is what came up: results are here and also included below.

What can we conclude from this?

First, an obvious conclusion is in order! No matter how many times one can utter the word "compliance," logs are still most useful for mundane (one would hope!) system administration. Yes, indeed, sysadmins are the primary consumers of logs - yesterday, today, and - likely! - tomorrow as well.

Second, I am saddened by the fact that application developers have not warmed up to logs, at least not en masse (and not according to this limited poll...). I am guessing when they start thinking of logging when creating their applications, they will be more aware of the fact that you can troubleshoot the applications using logs ...

Third, incident response team showing that low is some kind of fluke, I am sure. Everybody knows that logs are indispensable during incident response. Yes, even if only a little logging was enabled or even logging defaults left in place, logs often reveal answers unobtainable via any other mechanisms.

Next poll coming soon!

Posted by on January 08, 2008 in Innovation , Log Management & Intelligence , LogEd , LogLogic News | Permalink | TrackBack (0)

Logging Glossary: Operational Log Message

As I mentioned here,  I started publishing the LogLogic Logging Glossary. So, here is the twelfth term (first second third fourth fifth sixth seventh eighth ninth tenth eleventh):

Operational Log Message

A log message about the state of a product, the product's operation or support of its application, or the interaction of the product with its environment.

Operational messages are one of the three basic message types. They can be summarized as:

• Administrative – Changes to the product
• Operational – The product itself and the job it is doing.
• Application – The product doing its job.

Example of operational messages are: backup succeeded, update applied, memory is running low, system load high, etc.

Future Glossary entries will cover Administrative Log Messages and Operational Log Messages

Posted by on December 20, 2007 in Log Management & Intelligence , LogEd | Permalink | TrackBack (0)

Poll: Who looks at logs at your organization?

Here is my next poll about logs: Who looks at logs at your organization? Vote here! Also, my past polls and analysis are here.

Past polls were:

Posted by on December 19, 2007 in Log Management & Intelligence , LogMatters | Permalink | TrackBack (0)

On Approaches to Database Monitoring

So, people sometimes ask me about how to do database logging/auditing/monitoring and log analysis right. The key choice many seem to struggle with for database auditing and monitoring is reviewing database logs vs sniffing SQL traffic off the wire.  Before proceeding, please look for more background on database log management, auditing and monitoring in my database log management papers (longer, more detailed - shorter)  The table below summarizes the situation with database monitoring and auditing - now you can make your choice more intelligently (items in bold are the ones I consider key):

Choose logs if you care for the relevant Pros (esp key ones) associated with them; choose sniffing if you care for the Pros and are NOT undermined by their Cons (e.g. difficulties of supporting encrypted traffic)

Of course, one can also opt for a combined approach which follows the ideas of "double the benefits - for double the cost"...

(*) Nobody really knows what it will be in each particular situation: 0-40% were observed under various conditions by various people ...

Posted by on December 17, 2007 in Innovation , Log Management & Intelligence , LogMatters | Permalink | TrackBack (0)

Again, On Criticality of Logs

I just wanted to highlight two pieces that, again, speak - or, better, scream! - about the importance of logs. I suspect that LogBlog readers don't need additional motivation to take logs seriously, but these are just too useful to skip.

First is the interview with some convicted criminal hacker, who said that '... it would have been easy for IT and security managers to detect him in their companies' systems ... if they'd been looking. The problem was that, generally, no one was paying attention.

"If they were just monitoring their boxes and keeping logs, they could easily have seen us logged in there," he said, adding that IT could have run its own scans, checking to see logged-in users."'

Amen to that! Indeed, many of the successful-and-then-undetected attacks are due to incompetence. Why? 'Cause lacking logs or ignoring logs is indeed negligent and incompetent!

Second, is my comment on the TJX case, which kinda follows the same idea: 'Dr. Anton Chuvakin, a security expert with LogLogic, said TJX [probably] didn't have decent logs. "What took TJX months was looking at all their systems and determining who took what data, from where, where it was sent, etc. The investigation took them months. They likely didn't have any logs, because they had to do system forensics rather than log analysis to arrive at their conclusions about who stole the data and how. If they had collected and analyzed log data centrally, the investigation would have been a piece of cake," he said in an e-mailed comment to InternetNews.com.'

Indeed, doing disk forensics to know who did what is much more painful than checking reliable logs. Save yourself by logging, then saving and reviewing the logs!

So, one more time (not the last, mind you!):

• Make sure you have and collect logs (here are 11 reasons for that)
• Make sure you look at logs (here are the 11 reasons)

Posted by on December 11, 2007 in Log Management & Intelligence , LogLogic News , LogMatters | Permalink | TrackBack (0)

Logging Poll #3 "What Do You Do With Logs?" Analysis

So, the results of my 3rd poll are ready: live results are here, picture is also in this post.

First, this poll way more popular than my previous "why" poll. It seems like people do dislike to wonder "why."

Second, what are  the two choices, that are by far the most popular? They are:

• Store raw logs on a server (23%)
• Search raw logs (grep) when needed (24%)

Yes, this is the "state of the art" of logging:   collection of raw logs and "as needed" grep aka "slow and painful" search. In fact, the above answers might not even be given by the same people: some might be grepping logs on the individual servers, while others collect them on syslog servers and never touch them. That is why being in log management business is such a great thing: you have nearly the whole world to evangelize about the value of logs and log management tools.

Third, what's the next most popular idea of analyzing logs? It is "Run my own log analysis tool" at 10% of the respondents. Indeed, this movement still lives and thrives: people choose the Build->Suffer approach to log management often enough ...

Fourth, next come my somewhat self-inflicted surprise: apart from commercial log management (at 4%) and rolling one's own (discussed above at 10%), I added the option of "Use other log analysis tools"   which captured 7% of the vote. But what does that mean? I have no idea!

Fifth, I am NOT surprised by the lack of popularity of the rule-based correlation tools, such as SIEM (at 2%). When I made my decision to join LogLogic, I had to ponder this one really, really hard. My conclusion at the time (which is also valid now, even more than back in 2006) was that "SIEM is for some, log management is for everybody." This poll confirms this further.

Finally, all my logging polls and analysis are here. Next one is coming up!

Posted by on December 08, 2007 in Log Management & Intelligence , LogEd , LogMatters | Permalink | TrackBack (0)

Anton Logging Tip of the Day #13: Into the Darkness ... or The Ominous World of Unix Binary Audit Logs

Following the new "tradition" of posting a security tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative.

So, Anton Security Tip of the Day #13: Into the Darkness ... or The Ominous World of Unix Binary Audit Logs

In this tip, we will take a peek at one of the most esoteric areas of logging: Unix binary audit logs. Solaris BSM and Trusted Solaris auditing is the least unknown :-) example of it, even though other Unix vendors have similar auditing capabilities - see this for HP-UX Audit and this for IBM AIX audit. Linux kernel audit is also pretty much the same thing. If you look for information on 'Solaris BSM audit logs' , you'd find plenty of tips on how to enable such logging, a little on how to manage/rotate the log files, a bit on how to survive the resulting data deluge and ALMOST NOTHING on what to do with the log data, which is kinda sad :-) After looking at BSM logs for a while, I developed an opinion that nobody has ever looked at them on a regular basis :-)

So, let's assume you enabled Solaris BSM kernel audit for user "root" and few other "interesting" users (there is no per-object logging in Solaris; other Unix variants do have it) via the following commonly recommended per-user configuration in /etc/security/audit_user:

root:lo,ad,fw:no

anton:all,-all:no

jsmith:all,-all:no

This configuration file pretty much leads to logging of everything that the above listed users do. Now, you have audit files growing like mushrooms in your /var/audit. What good does it give us?  First, we need to convert the binary audit files into text - something along the lines of

# auditreduce -A  /var/audit/20071127193515.not_terminated.SunUltra10 | praudit -l > /tmp/sol_box_11272007

will do. Now what? In this tip we will pick one thing to use these logs for: how use them to see who is trying to copy sensitive files off the system.

First, who is connecting out - lets's search the logs for 'connect' calls (if you are using LogLogic for it, use Index Search for this task; if not, grep will have to do, but be prepared to wait, possibly a looooooooooooong time :-)). A few recommended searches:

• "connect AND 172.16.10.*"   or "connect AND NOT 172.16.10.*"   (to look for connection to specific IPs or only to outside networks) or simply 'connect AND username' to look for connections by a specific user (if you log many users)

Here is an example found (with connect, IP and user in bold):

header,103,2,connect(2),,Tue Nov 27 11:36:46 PST 2007, + 193 msec,argument,1,0x4,so,socket,0x0002,0x0002,0x80d6,SunUltra10,0x0016,10.1.1.41,subject,root,anton,other,anton,other,29902,29720,0 1611 172.16.0.173,return,success,0

At this point we already know the user name of the user who run that connecting process since it will be in the results (you can also the user to search as I showed above).

Next, what are those connections - let's try to uncover which programs actually connected (BSM logs don't make that easy!). Let's search for process starts in the same time frame (within a few seconds):

• "execve AND NOT ls AND NOT <whatever other commands you don't care to see>" will give you a list of started programs.

Example found:

header,124,2,execve(2),,Tue Nov 27 11:36:46 PST 2007, + 115 msec,path,/usr/bin/scp,attribute,100555,root,bin,136,1573,0,subject,root,anton,other,anton,other,29901,29720,0 1611 172.16.0.173,return,success,0

OK, so somebody is trying to connect via SSH/SCP. Notice that both records - connect and execve -  have the same timestamp, up to a second. Sadly, time and parent process ID ( which is in our case 29720) is all that correlates them together.

Finally, what file was affected (i.e. copied off the system via scp in this case) - more digging is in order; we again use the process ID and time. The easiest is to search for a file name or browse all records around the same time frame (might be A LOT!):

• "*secret.zip* AND anton" will work; we can add the above process ID and look for "anton AND 29720" (but expect a lot of data since this is a shell process ID)

For example:

header,135,2,open(2) - read,,Tue Nov 27 11:36:47 PST 2007, + 900 msec,path,/tmp/not-so-secret.zip.gz,attribute,100600,anton,other,0,32743959,18446744073709551615,subject,root,anton,other,anton,other,29901,29720,0 1611 172.16.0.173,return,success,4

What do we know now? This user connected to this system and MAYBE copied this file via, MAYBE via scp. How cool is that? (A: not cool at all, since we are not sure, but that is all we can do with such logs)

To conclude, if you can avoid dealing with Solaris BSM logs, please do so :-) On a more serious note, you now know why these logs were called "the ugliest logs ever."  Even more seriously (but still pretty humorously), these logs are a classic example of trees that make every effort to obscure the forest, because these logs record system calls and not processes or user actions (and connect, execve and read are all logged separately). There are also many, many more idiosyncrasies where these come from :-)

Also, I am tagging all the tips on my del.icio.us feed. Here is the link: All Tips of the Day.

Posted by on November 30, 2007 in Compliance , Log Management & Intelligence , LogEd , Security | Permalink | TrackBack (0)

Log Poll #3: What Do You Do With Collected Logs?

Time for another fun logging poll: What Do You Do With Collected Logs?

Vote here! This is my Logging Poll #3, below are the links to past polls with my analysis:

• Poll #1 "Which Logs Do You Collect?" (vote here, results so far here, analysis here)
• Poll #2 "Why Collect Logs?" (vote here, results so far, my analysis)

Posted by on November 27, 2007 in Log Management & Intelligence | Permalink | TrackBack (0)

Protecting Logs from Admins: A Lost Battle?

One of the truly major challenges of log management is obtaining trusted logs of administrator activities, sometimes broadened to cover so-called "privileged users" (see my log trust pyramid post for more discussion). Many consider it to be the "lost battle" of logging. However, logging administrator access and actions is more important than ever today; it is one of the few workable ways of dealing with insider attacks.

So, I created this little table where I try to summarize some ways of protecting the C-I-A (confidentiality, integrity, availability) of various logs from administrators and privileged users (some successful and some very hard to pull off). Note that in the case of databases and applications, we need to protect their logs from DBAs and application admins and not from the underlying server platform admins.

UPDATE: table below might be corrupted in some blog readers; see the full table here.

UPDATE: table above might be corrupted in some blog readers; see the full table here.

Comments?

Posted by on November 26, 2007 in Innovation , Log Management & Intelligence , Risk Management , Security | Permalink | TrackBack (0)

Imagining an Ideal Log Management Tool?

The idea came from Jeremiah Grossman (here) when he described "The Best Web Application Vulnerability Scanner in the World" thus: "Within a few moments of pressing the scan button it’ll find every vulnerability, with zero false positives, generate a pretty looking report, and voila you’re compliant with GLBA, HIPAA, and PCI-DSS. Of course, we all know such a web application scanner is simply not possible to create for a variety of reasons."

So, let's imagine the ideal log management application.

1. Logging configuration: the ideal log app will go and find all possible log sources (systems, devices, applications, etc) and then enable the right kind of logging on them according to a high level policy given to it
2. Log collection: it will collect all the above logs securely (and without using any risky super-user access ) and with little to no impact to networks and systems
3. Log storage: it can security store the above logs in the original format for as long as needed and in a manner allowing quick access to them - in both raw and summarized/enriched form
4. Log analysis: this ideal application will be able to look at all kinds of logs, known to it and previously unseen, from standard and custom log sources, and tell the user what they need to know about their environment and based on their needs: what is broken? what is hacked? where? what is in violation of regulations/policies? what will break soon? who is doing this stuff? The analysis will power all of the following: automated actions, real-time notifications, long-term historical analysis as well as compliance relevance analysis
5. Information presentation: this tool will distill the above data, information and conclusions generated by the analytic components and present then in a manner consistent with the user's role: from operator to analyst to engineer to executive. Interactive visual and drillable text-based data presentation across all log sources. The users can also customize the data presentation based on their wishes and job needs, as well as information perception styles
6. Automation: the ideal log management tool will be able to take limited automated actions to resolve discovered and confirmed issues as well as generate guidance to users so that they know what actions to take, when full-auto mode is not appropriate. The responses will range from full-auto actions to assisted actions ('click here to fix it') to issuing detailed remediation guidance. The output will include a TODO-list of discovered items complete with actions suggested, ordered by priority
7. Compliance: this tool can also be used directly by auditors to validate or prove compliance with relevant regulations by using regulation-specific content and all the collected data. The tool will also point at gaps in data collection as it applies to specific regulations that the user is interested in complying

In other words, this magic black box will have raw log data shoveled from one side and will have answers to key questions that concern the IT users and their managers on the other side.

Am I nuts? Well, can I dream for a second? :-)

Posted by on November 06, 2007 in Innovation , Log Management & Intelligence , LogMatters | Permalink | TrackBack (0)

Top 11 Reasons to Secure and Protect Your Logs

Following my now-famous Top 11 Reasons to Collect and Preserve Computer Logs and Top 11 Reasons to Look at Your Logs, here is the promised "Top 11 Reasons to Secure and Protect Your Logs"

1. Let's review why you are reviewing logs. Will logs that might have been changed by somebody, somewhere, somehow still be useful for items 1-11 from here? No? Secure them!
2. Logs in court? Challenges abound! To respond to them, one needs to protect the logs so you can claim that they are both authentic and reliable.
3. A human error still beats an evil hacker as the main cause of IT problems. Are your logs safe from it? Available when needed? Protect them from crashes and other faults!
4. PCI DSS just says so: "Secure audit trails so they cannot be altered." Want to do it- or pay the fines?
5. Do you protect financial records? Identity info? Passwords? Some of it ends up in logs (even if by mistake) - thus making them sensitive, protected information. Secure the C-I-A of logs!
6. Do you look at logs during incident investigation? Do you want them to be "true" or full of random (if creative...) "stuff", inserted by the guilty party? Secure the logs!
7. Think that "attacks vs logging" are theoretical? Think again. Are your logs safe or vulnerable? Is your logging tool 0wned?
8. Syslog + UDP = log injection + log loss. Are you protected (reliable TCP, confirmed delivery, encryption - SSH, SSL, VPN)?
9. Why modify logs? No, really, why change logs? If you never change logs - and you never should, unless it is appending to them - hash them right away after collection to make them immutable.
10. Logs are backed up on tape - who will see them? Well, whoever restores the tape, that's who! Encrypt them to protect them from accidental and malicious disclosure if tape is lost.
11. Why log access to logs? Same reason why you had the logs in the first place - to review who did what. Who broke through and stole the logs? Who browsed them without permission? Only logs will tell - if you have them!

Overall, one need to strive for having no holes in log safeguards from log birth to analyst conclusion based on log information...

Possibly related posts:

• Top 11 Reasons to Look at Your Logs

• Top 11 Reasons to Collect and Preserve Computer Logs

Posted by on November 02, 2007 in Innovation , Log Management & Intelligence , LogMatters | Permalink | TrackBack (0)

Just What is Enterprise Class? Part V

Here is the final enlightening post from Dimitri McKay, our super-brilliant network and systems engineer from the East Coast, where he continues to discuss the meaning of the phrase "enterprise-class," which is certainly MUCH more than a marketing buzzword! Part I is here, Part II is here, Part III is here.  Part IV is here.

Dimitri McKay says: "This is the final piece in the five part series entitled “Enterprise Class”. The term “Enterprise Class” for most of us has appeared on Bullshit-Bingo cards for years, but when we actually explore WHAT “Enterprise Class” is and how it pertains to a certain product, you can learn quite a bit about that product, and how it stands up to both the competition and how it stacks up against what it is and what it should be.

As of this writing, LogLogic has set free it’s next code-base, LogLogic4 Release 2. With added features and fixes, LogLogic fits the Enterprise all the more with each release.

13. Usability: Because of the large number of customers using LogLogic, the mission critical nature of the business problems being addressed, and the system complexity that is introduced due to the previous topics, usability is a critical factor for LogLogic. Our goal is to strike an appropriate balance between ease of use and complex functionality. Let’s face it. Using LogLogic to handle forensics, or compliance, or general reporting and alerting is an easy solution to a very complex problem. We are taking the chaos of a massive amount of log-traffic and we are taming it via the use of log forensics tools, summary reports, and real-time alerts.

When I was on the support team one of the biggest questions customers would call and ask after purchasing the LogLogic appliance was “Now what? What should I be looking for?” - and the answer LogLogic came up with was the compliance suites. The compliance suites are a phenomenal way of helping the customer hit the ground running. It’s simple. “Install this suite of search filters, alerts and custom reports. Then tailor it to your network. Now you’ve solved your compliance problems and/or government mandates.”

14. Compatibility/Maintainability: To minimize downtime as part of providing high availability, LogLogic must provide compatibility from one release to the next. Backward compatibility means that the when a new version of LogLogic  is installed, it continues to work with the previously installed customizations and other systems witch which it interacts such as source devices and remote authentication services.

 Forward compatibility means that when systems around the LogLogic are upgraded, the LogLogic must continue to function as it did before, even though the new software may support new log formats. When fixes are provided for LogLogic, downtime for installing the fix must be minimized by providing support for incremented updates such as patches and hot-fixes. LogLogic must also offer several solutions to handle updates such as automatic update via the web, or manual file updates due to the appliances being offline.

Thank you for reading my five piece foray into the Enterprise. Please feel free to email me your thoughts, comments, or post them below. Stay tuned to LogLogic as we journey where no other LMI vendor has gone before."

Posted by on October 31, 2007 in Innovation , Log Management & Intelligence , LogMatters | Permalink | TrackBack (0)

Poll: Why Do You Collect Logs?

The previous poll (vote here, live results here, analysis here) proved to be a success so the next one is here.

This time the question is: "Assuming that you centrally COLLECT system, network or security logs from their originating sources, what is THE MAIN reason for doing it?"

Vote on!

Posted by on October 31, 2007 in Log Management & Intelligence , LogMatters | Permalink | TrackBack (0)

Log Collection Poll Results

My poll on log collection have been a great success and now the results are in. Here is the link to the running totals as of now (the graphic snapshot from yesterday can be seen here). Let's review and discuss the findings after running it for slightly over a week.

• First, which of my expectations were NOT met? Well, I did expect that firewalls will be #1, not Linux/Unix servers. Admittedly, the difference is not so big, but I am impressed: Unix syslog still rocks the logging world :-)
• Second, the top source of collected logs is also the hardest to analyze due to its lack of structure. Nowadays I treat syslog from Unix/Linux as "broken English" and not as "data." It is a dog to parse (that is why we are trying to find something novel)
• Third, I was amazed that database logs were THAT high on the list. Wow! All the evangelizing seems to have worked out :-)
• Fourth, Windows server log collection is still in the dumps - but we need it! Go grab LASSO and dump those event logs into syslog without pesky agents. Easy!
• Firth, other Unix logs - what are those? We might never know what the respondents meant: still, I think that these are binary audit logs and other fine-grained audit logging. Indeed, many people starting to look at BSM audits and other "ugly ducklings" of logging.
• Sixth, web server logs are gold - everybody knows it. The poll confirms this as well: they are #2. Some fun analysis tips from me are coming soon.

Next poll coming soon! Thanks a lot for responding!

Possibly related posts:

• Top 11 Reasons to Look at Your Logs

• Top 11 Reasons to Collect and Preserve Computer Logs

• Database log management

Posted by on October 26, 2007 in Innovation , Log Management & Intelligence , LogMatters | Permalink | TrackBack (0)

Poll: Which Logs Do You Collect?

I figured I'd do a poll a week since people really like it. So, my first poll-a-week: Which Logs Do You Collect?
Vote away! I will post and comment on results here after a few weeks.

Posted by on October 17, 2007 in Log Management & Intelligence , LogEd | Permalink | TrackBack (0)

Simple Log-based User Profiling for Activity Monitoring

Recently I've been looking into detecting stolen/shared access account credentials. Now, how can you detect that? No NIDS will trigger, NIPS will let is pass, no unusual types of log records might ever by  produced (especially if only limited logging is enabled). And what raises the stakes is that this type of activity is not  only about "hacked" accounts, but also about insider abuse of accounts.

However, there will likely be changes in how normal log records are produced.

Let's summarize some known methods for using a simple user "profile" to detect account theft aka account sharing aka user impersonation aka access with stolen/shared credentials. It implies that we've been collecting the logs before the incident and have a solid trail of normal users and legitimate account owners.

1. Unusual login source IP (e.g. normal user always comes from 10.0.X.Y or 10.1.X.Y, but now we see a login from 172.16.0.Z) - will work in some cases, but not for the free-for-all servers such as at a University
2. Unusual login time (e.g. normal user always comes from 9AM to 5PM, now we see 3AM) - will work in most cases, but will fail if the attacker happens to be in the same time zone
3. Unusual login session length  (e.g. normal user always stays logged in for 5-20 minutes, now we see a 10 hour-long session) - works only if logout is logged; might not catch a lot of realistic but malicious connections
4. Unusual login frequency (e.g. legitimate user logs in once a day in the morning, now we see dozens of connections) - will work for some cases, but others will be missed
5. Unusual login failure/success ratio before a successful login (e.g. normal user always types the password right the first time, now we see failures than successes) - not too reliable, obviously
6. Unusual list of user actions performed (normal user only reads these files, but now we see writes to a very different set of files) - will work most frequently, but needs more granular logging of file access, object changes, etc (audit logging) [more on this in the near future!]

So, if you have logs of user activities, at the very least, logins and logouts ( but having records of more user activities is always better!),  for the last few weeks or months, one can compute the above profiles using historical data and then compare them with current numbers (very similar to some of the methods from my classic log mining presentation).

The final missing bit is for how long to collect your normal user behaviors: I discovered that 1 week to 1 month works pretty well. Less time yields unstable results and more time necessitates much more data crunching without much gain.

Posted by on October 12, 2007 in Innovation , Log Management & Intelligence , LogEd , LogMatters , Security | Permalink | TrackBack (0)

Just What is Enterprise Class? Part IV

Here is the next enlightening post from Dimitri McKay, our super-brilliant network and systems engineer from the East Coast, where he continues to discuss the meaning of the phrase "enterprise-class," which is certainly MUCH more than a marketing buzzword! Part I is here, Part II is here, Part III is here. 

"10. Serviceability/Manageability: The scale of deployment of a LogLogic solution, whether it be managed remotely or locally, requires features which allow the solution to be managed effectively. FCAPS is an acronym which is often used to remember the following guidelines of  security log appliance management:

Fault, Configuration, Accounting, Performance, and Security

The users of LogLogic must be able to monitor what is happening on the system itself, as well as the network around it and applications on it, and to have the ability to diagnose faults when they occur.

Fault diagnosis requires the ability to collect sufficient information about the fault when it occurs, preferably without having to reproduce the fault. We log our own appliance, and anything else on the network, and then alerts can be handled via SNMP traps sent to an SNMP trap receiver such as HP Open View or IBM Tivoli. The other option is an alert sent to a remote pager or mobile device, or even an email to the NOC/SOC personnel.

11.  Customizability/Flexibility/Integrability: LogLogic is used to solve complex business problems on a large scale. For some it’s used for various compliance needs, such as PCI or SOX. To others, it’s an alerting or filtering tool, while forwarding a few of the log records on to a SEIM (typically, as much as it can handle, which is usually not as much as needed ...) or a specific other security tool. To others, LogLogic is used for general reporting, incident investigations or forensics. Regardless of how it’s used, LogLogic a different tool to different people.

LogLogic appliances are rarely rolled out as a single unit in an environment. Generally they are rolled out by location, by message per second (MPS) requirements... or by long term storage requirements, but any way you shake it, LogLogic architecture is designed to meet the needs of the Enterprise (and this is not to say that SMBs won't benefit from log management!).

Usually you’ll see several LX reporting/alerting appliances feeding back to a single long term storage ST appliance, but that’s not written in stone. Sometimes customers send all of their log data direct to the ST storage appliance (which handles a massive 75,000 messages per second) in order to take advantage of a single IP address destination. This, in my own humble opinion, is a good data center solution.

My point is, the architecture of the LogLogic appliances is a variable (but still easy!) which gives you total control. There is no firm “config” that is required for boxes to be placed in. Instead, the architecture is flexible, and allows for multiple configurations depending on the environment they reside in. This is the ability to remain agile.

Living in an enterprise world, we must adopt to new technologies as they become standards. Often some of the features on the LogLogic appliances overlap with other technologies already in use. Because of that, we have engaged the ability to utilize those technologies in those situations. One example is that LogLogic has but doesn’t always provide its own identity management functions. Typically there is another authentication management that the enterprise is already using, such as TACACS+ or RADIUS. The enterprise may have adopted particular standards to manage user databases and access control. For this LogLogic needs to be flexible in it’s architecture, allow customizability within the enterprise, and integrate well within the framework of the already existing environment.           

12. Support: With any mission critical software deployment, the enterprise must have a reliable way of getting support for diagnosing problems and getting fixes and advanced replacements. This includes 24x7 availability for support personnel and an organization with the experience and expertise to understand how the appliances are used in enterprises. Our support is not only top-notch, but is also praised by our customers!

Thank you for tuning in to part lV of “Enterprise Class” where I’ve laid down WHAT enterprise-class really is, and how LogLogic has tackled it. I’d also like to thank our competitors who have been visiting my blog. You can imitate, but you’ll never duplicate us, boys!

Next piece will conclude the series ... stand by!"

Posted by on October 11, 2007 in Innovation , Log Management & Intelligence , LogEd , LogMatters | Permalink | TrackBack (0)

PoS Logs out of PCI Scope? Surely You Are Joking...

... well, turns out they were dead serious. As I expressed my puzzlement to our resident PCI auditor, he explained that PoS logs and overall security of PoS devices are often "in-scope for PCI, but out of scope for a typical PCI audit."  How bizarre is that?  But let's start from the beginning.

First, what on Earth is a PoS? PoS, or Point-of-Sale terminal, is a machine that stores (or whoever else who takes credit cards) use to process credit card transactions: scan cards, communicate with verification server, print receipts, etc. It might be standalone or combined with a cash register. It might very very simple - just card reader + transaction unit in a single hardware unit - or as complex as a Windows PC with a cash drawer and no software updates (a scary thing indeed!)

So, in the latter case, there are certainly logs involved. In fact, there are also PoS-specific application logs, such as this example below, coming from an IBM SurePoS device:

--------------------------------------------------------------------------------------------------------------------------------

01/11 09:11 CC     5 W518 PROGRAM ADDDDDXUXDL HAS ENDED                           

                   B3/S111/E007 REASON=2 TYPE=3 RC=00000000            

SOURCE: OCF                                                                     

REASON: Application ended            PROGRAM TYPE: Background               

RC: No error      

--------------------------------------------------------------------------------

PoS devices might be configured to store credit card numbers locally (for backup) and also to offload them to a "branch server" (a store server or both a store server and a regional server). Are there logs of who accessed them on the local PoS system? Unlikely. Are they looked at? Probably not.  Maybe the logging is done better on the branch or store central server, but even this is not a certainty.

Overall, I am willing to bet a bottle of decent champagne that very few people, if anybody, in the whole world are regularly looking at PoS logs. At some happy point in the future, I predict they will start since the Beast of PCI will make them :-) When this happens, we will talk about PoS log analysis.

As of today, you would do comparatively well if you will collect and save them and thus will have a chance to review them for incident response for your next data theft case (or show them to an unusually nosey PCI auditor...)

More fun PoS security reading is here [PDF].

Posted by on October 05, 2007 in Compliance , Log Management & Intelligence , LogEd , LogMatters | Permalink | TrackBack (0)

Just What is Enterprise Class? Part III

Here is the next enlightening post from Dimitri McKay, our brilliant network and systems engineer from the East Coast, where he continues to discuss the meaning of the phrase "enterprise-class," which is certainly MUCH more than a marketing buzzword! Part I is here, go read it first. Part II is here, go read it next.

"This is the third piece in the five part series titled “Enterprise Class”. Here we’ve gone through what IS enterprise class and how that corresponds to the LogLogic solution. Feel free to post your comments.

 7.Performance: Performance is a broad topic. We’re not just talking local log collection performance, but also the ability to perform efficiently while in a distributed environment is critical to enterprise class hardware and software.

While the ability to add more LogLogic appliances allows more devices to be logged and application data to be collected, if the software does not perform with a high level of efficiency, the cost to deploy a system for the required number of devices will be prohibitive.

LogLogic’s top end reporting appliances handle 4000 messages per second sustained. That is a HUGE amount of traffic. if you’ve ever had to sift through it, you’d agree. Think about it. That’s 240,000 messages per minute. 14.4 million messages per hour. That’s 345.6 million messages per day. Grep that with your syslog server. Parse that with your SIEM. It’s not going to happen (not without an ENORMOUS price tag). And our log collection appliance do even more, way more: 75,000 messages  per second sustained. Wow!

Performance on a LogLogic appliance is measured based on several factors: CPU usage, messages per second, and disk usage which are important not only for the main functions of the system such as receiving and parsing logs, but also for the ability to run reports and searches on those messages.

8.Security: Because LogLogic is used for mission critical functions, security is essential. LogLogic must ensure that users who are authenticated to the appliances only have access to the functions they need and access to the log sources that are determined by their job role. LogLogic must also offer the ability to encrypt data as it moves between appliances (which we do via LogLogic TCP) and to also offer the ability to use WORM or data at rest encryption (which is handled by the EMC Centera or NetApp for WORM and Decru for data" at-rest" encryption).

LogLogic also covers security with the ability to verify that the system is secure through audit trails, general protection of the appliance through a secure linux kernel locked down in the default install, perform self logging, and then hashing the raw logs which are kept in an immutable format. At LogLogic we take security serious, and the box is put through a variety of vulnerability assessments on a regular basis.

9.Documentation: Because LogLogic handles device support through a specialized group of folks we call “LogLabs” we are able to add devices at an expeditious rate. With each of those new devices comes the required measures to configure each of those devices for logging, and with that knowledge transfer that to documentation. Over time our Documentation Group has continued to grow. Knowledge management is essential to managing the enterprise class solution while alleviating the strain on any of the support or field engineering staff.

Check back soon, to see more about how LogLogic does Enterprise right. In the next episode I will run off on a tangent about usability and flexibility/interoperability/customizability and support! We are boldly going where no other LMI vendor has gone before!"

Posted by on October 04, 2007 in Innovation , Log Management & Intelligence , LogEd , LogMatters | Permalink | TrackBack (0)

Log Trustworthiness Hierarchy

This post is inspired by the old TaoSecurity post Enterprise Trust Pyramid where he explains that he trusts some security, system and network evidence data more than other. For example, dedicated NSM sensor records are trusted more than vanilla server logs. With this post, I am looking to establish a log trustworthiness hierarchy so that people start thinking about trusting log data as a kind of a spectrum, from "probably trash" to "guaranteed to be an accurate record of activities."

So, do you trust your logs to accurately depict what happened on the system or network? Which logs do you trust the most? How do we increase this trust?

My first draft of such trust hierarchy follows below (from low trust to high trust):

1. Compromised system logs (mostly trash, but might contain bits that attacker missed/ignored)
2. Desktop / laptop OS and application logs (possibly changed by users, legitimate systems owners, etc)
3. All logs from others systems where 'root'/Administrator access is not controlled (e.g. test servers, etc)
4. Unix application logs  (file-based)
5. Local Windows application logs
6. Local Unix OS syslogs
7. Unix kernel audit logs, process accounting records
8. Local Windows server OS (a little harder to change)
9. Database logs (more trusted since DBA cannot touch them, while 'root' can)
10. Other security appliance logs (located on security appliances)
11. Various systems logs centralized to a syslog server
12. Network device and firewall logs (centralized to syslog server)
13. Logs centralized to a log management system via a real-time feed (obviously, transport encryption adds even more trust)

Admittedly, the differences between some of them are minor or even non-existent ...

To conclude, some logs DO in fact provide reliable evidence in case of an incident; you just need to know which ones to trust and which ones to only consider to be "hints" (or possibly even a misdirection). But of course, you need to first have logs and then look at them.

Posted by on September 27, 2007 in Log Management & Intelligence , LogEd , LogMatters , Security | Permalink | TrackBack (0)

Just What is Enterprise Class? Part II

Here is the next enlightening post from Dimitri McKay, our network and systems engineer from the East Coast, where he continues to discuss the meaning of the phrase "enterprise-class," which is certainly MUCH more than a marketing buzzword! Part I is here, go read it first.

"4.Scalability: To support a large number of devices in growing enterprises, SIEM and log management products differ a great deal in how they scale. Some of the SIEM products require you to add as many as three new appliances, (which all carry a hefty price tag) just for a slight increase in throughput. Others require you to forklift out your existing hardware to bring in new, bigger (and more expensive) equipment. In my own humble opinion these options are completely ridiculous.

The LogLogic scalability approach is uniquely simple. “Add log-traffic to your LogLogic appliances until you reach capacity on that box. Then add another box. Rinse and repeat. Each box does it’s own collection, does its own parsing, reporting and searching, and does its own storage. No special other boxes required. All encompassing solution.

Of course, with “Scalability” comes other requirements of how to add capacity not only vertically, but what of horizontal scalability? Such features would include the ability to report across all LogLogic appliances, manage users and their access rights across a global environment, and use some form of remote authentication such as TACACS+ or RADIUS. This distribution should be an asset, and not a limitation of the overall system scalability. All of these are addressed and in the current LogLogic 4.x.

5.High Availability: Due to the number of corporations and government agencies depending on LogLogic for regulatory compliance, system security and forensics, as well as heartbeat monitoring and troubleshooting, it’s safe to say that LogLogic is “Mission Critical”.

Because LogLogic is a “Mission Critical” system, it must support the internal and external capacity for zero downtime via high availability. This is accomplished via bonded NIC’s, hot-swap drives in various RAID arrays and Mirrored sets, hot swap redundant power supplies and most importantly... the ability to link the boxes together in an active pair. Should there be a failure in a LogLogic pair, both appliances are exact clones of each other, and the passive appliance jumps right into gear as the primary. In milliseconds. It’s genius.

In a perfect world, there would be zero downtime on “Mission Critical” systems. Here at LogLogic, we try to minimize that to as little as possible through the use of various forms of HA, and it shows. Have you seen our new hardware?

6.Reliability: Although high availability plays a part here, I prefer to think of reliability as the quality of the code-base, the product design itself, and the forward motion of feature sets and additional device support in a timely fashion.

LogLogic is built on a stripped down lean, mean hardened Linux platform. From there we didn’t go the route of a proprietary datastore which can start as a good idea and then turn into a huge hulking mess with nothing but frustrations stemming from using, maintaining and increasing the capabilities of, but rather we went with a solid database solution for storing meta-data. Industry standard. Well known. Easy to use. Robust.

Then we built in some brilliant parsers and a very straight forward GUI. Sprinkle that with some great features and a little bit of magic and you have a LogLogic appliance. Our upgrade plans and implementation are fast and furious. We release new device support monthly. Monthly additions. Monthly updates.

At the end of the day, however, extenuating circumstances happen. Should there be a hardware or software problem, LogLogic support is there to help. With 24/7 support and advanced replacement available to our platinum level enterprise class customers, all the bases are covered.

Build the appliance on a solid kernel, with a well known and well tuned database. Add HA, some of the finest engineering talent in the world, and a great support team, and LogLogic does enterprise right.

Stay tuned for the 3rd installment of “Enterprise Class”. LogLogic is going where no LMI vendor has gone before."

Posted by on September 27, 2007 in Innovation , Log Management & Intelligence , LogEd , LogMatters | Permalink | TrackBack (0)

More Web Proxy Log Tips

After publishing my proxy log tip (here), I figured I'd post a few more mini-tips on web proxy logging as well as a link to my full presentation (webcast with voice, slides only) on web proxy log management.

First, why look at proxy logs? Apart from my overall answer that applies to all logs, proxy-specific reasons are the following:

1. Review users’ activities on the web (not just surfing!)
2. Monitor applications' HTTP activity
3. Detect Web-enabled malware traffic
4. Study proxy performance metrics

Most people just focus on #1 above and kind of forget #2-#4. Also, the focus of #1 is often narrow - what do they surf at work?- and not broad - what do they do on the web? - which is much more useful. While there is no direct mention of proxy logs in recent regulations, monitoring what users do with YOUR data is clearly part of the compliance mandates (and, obviously, a good idea in general!) Indirect references to proxy logging can be seen in the following:

• Proxy monitoring is part of the overall control and governance (thus SOX, HIPAA, GLBA, etc)
• Legal requirements to have audit trails (thus HIPAA, PCI)
• Breach disclosure laws also impact (SB1386 and others, soon US-wide)
• On the flip side, privacy laws might mandate the opposite i.e. NOT having logs.

So, please treat proxy logs with the respect they deserve!  Here is my full presentation (webcast with voice, slides only), on analyzing and managing web proxy logs.

Related posts:

• Anton Logging Tip of the Day #12: Proxy Log Fun - Proxy Logs vs Information Leakage

Posted by on September 24, 2007 in Log Management & Intelligence , LogEd , LogMatters , Security | Permalink | TrackBack (0)

Just What is Enterprise Class? Part I

Here is another enlightening post from Dimitri McKay, our network and systems engineer from the East Coast, where he discusses the meaning of the phrase "enterprise-class" (which is sometimes known to be abused in marketing literature...)

"The Enterprise is generally a huge task, and this post itself is quite large. Due to the mass of both this post and the average Enterprise roll-out, I will break this down into 5 parts. Here is part 1 in the 5 part series.

I hear the term “Enterprise Class” as a buzz word to define hardware and software. Sometimes it’s used to describe a feature set, and to some it’s a scalability reference. Here’s what “Enterprise Class” means to me, and how it pertains to LogLogic solutions as a whole.

1. Multi-user: LogLogic handles the Multi-user process in a two prong attack. First, we offer the basic granular local account system which gives users the rights needed to do the job on only the devices they require. Now, on the enterprise side, we have added the ability to connect to TACACS+, RADIUS and soon LDAP/AD servers in order to alleviate the task of managing users on the LogLogic Appliances. Multi-user is a must for anything that is to be considered “Enterprise Class”. I’m hoping to see some additional authentication methods in the future, however, as for now the TACACS+ and RADIUS option certainly covers the majority of our enterprise customers.

2. Accessibility: In 1998, Congress amended the Rehabilitation Act to require Federal agencies to make their electronic and information technology accessible to people with disabilities. Inaccessible technology interferes with an individual's ability to obtain and use information quickly and easily.  Section 508 was enacted to eliminate barriers in information technology, to make available new opportunities for people with disabilities, and to encourage development of technologies that will help achieve these goals. Most of the specifications for software pertain to usability for people with vision impairments. For example, one provision requires alternative keyboard navigation, which is essential for people with vision impairments who cannot rely on pointing devices, such as a mouse. Other provisions address animated displays, color and contrast settings, flash rate, and electronic forms, among others (see more at Section508.gov).   Now, at LogLogic, we try to make our GUI as straight forward and easy as possible. We adhere our GUI to the standards and framework put forth by the web browser itself, which thereby adheres itself to the accessibility options within both that browser and also the Operating System.

3. Localization: As with any business solution, our world isn’t set in one language or specific dialect. As a whole, LogLogic offers language options not only in English but also in languages relevant to foreign countries that are bound by the same requirements that LogLogic can address such as JCobit and J-Sox, which are the Japanese versions of COBIT and Sarbanes-Oxley (SOX) compliance. As we move forward LogLogic continues to add alternative language support such as Korean and Chinese.

Stay tuned for the next episode of “Enterprise Class” where I will run off on a tangent about HA, reliability and scalability."

Posted by on September 24, 2007 in Log Management & Intelligence , LogEd , LogMatters | Permalink | TrackBack (0)

More Project LASSO Updates

Dimitri McKay, our network and systems engineer from the East Coast contributes this fun story: "When I first started at Loglogic, the only option for retrieving Windows logs was via an open source product called Snare. Snare was a simple concept. It would basically tail a Windows event viewer and whenever a new event was written, Snare would convert that to syslog, and send it over the wire (UDP or TCP) to its destination. Basic concept, basic execution. 

    Now, all of this was well and good, however, I couldn’t help but feel like an agent was not the greatest of solutions. As a former Windows Engineer, I was sort of put off with the idea of agents as a whole. It seemed like everyone was offering agents, and those agents were never happy on my systems in some form or another. So began Project Lasso. 

    Over some Thai food, Matt Foley, Andrew Morris and myself had a conversation about an agent-less Windows solution, and later that day I found myself writing a PRD for the “Windows Remote Event Collector” which was code named Project Lasso. 

    The name stuck, and with our 4.0 release I’m really happy with the ongoing development of the product. The concept is simple... Lasso is installed either as an agent to monitor itself, or on a “Project Lasso Server” where it uses WMI connections to connect to the other hosts it will be pulling logs from. There it pulls the string dll’s to a local repository, and then pulls the windows events themselves. Both the string dll’s and the events are then sent to the receiving syslog or Loglogic appliance where they are parsed and indexed. [Anton: they can also be sent to any other syslog receiver, such as a syslog-ng server]

    In 4.0 we added a few features I’m very excited about. 

The ability to use custom shares. In version 3.x we needed a Domain Admin account to pull the full Windows events. The Domain Admin account was the only account which had access to:

• The Event Viewer (for the header and events themselves)
• The Registry (to point to the string dll’s.) 
• The file system (via c$ to get the string dll’s.)

In Project Lasso 3 if you set some Active Directory policies, you could eliminate the need for a Domain Admin for the registry and event viewer, but not the C$ admin share. The only other account that had access to that was the Backup Administrator. Now, in Project Lasso 4 we don’t have that issue any longer because we can configure just a standard Lasso User account to have access to the registry, the event viewer, and a local drive via a custom share.

The ability to change the Hostlist.ini on the fly: In Project Lasso version 3.x when you added hosts to be monitored, you had to restart the Lasso service. Well, this process became somewhat onerous in that every time an Administrator needed to add new servers/clients to be monitored by Project Lasso he had to re-start the service which would take quite a bit of time to check those string .dll’s for changes or additions and then start pushing logs to the destination server. This could sometimes take more than a few minutes depending on the number of hosts. So much for real-time alerting or reporting. None of this is a problem anymore as the Project Lasso service will now grab the new hosts on the next pass.

Shared DLL Repository: Prior to Project Lasso version 4, when a dll Repository was created on the Lasso Server, it downloaded all .dll’s for all monitored hosts. This means there was a ton of the same .dll’s in the repository as each machine would most likely have the same .dll’s. These folders were about 120MB large in Lasso 3 times the number of hosts you were monitoring. That can be a big storage problem in a short amount of time. In Lasso 4, the dll’s are shared among hosts so it has a much smaller disk space requirement.

In closing, if you’re interested in routing Windows Events to a syslog or Loglogic appliance, you can do so via Project Lasso either in agent mode or in agent-less / remote collector mode.

Feel free to check out the always free Lasso 4 on Logforge"

Indeed, Project Lasso  is in wide use among LogLogic customers as well as in the broader world. Deal with Windows logs? Grab Project Lasso!

Posted by on September 17, 2007 in Log Management & Intelligence , LogEd , LogMatters , Project Lasso | Permalink | TrackBack (0)

New LogLogic Partner Program Unveiled

Today, LogLogic announced a NEW tiered channel program that provides partners unparalleled resources, support and margin incentives for the express purpose of shortening partner sales cycles and increasing close rates.

As a result of compliance mandates (SOX, HIPAA, COBIT, PCI, FISMA, etc.), which often require companies to carefully track, manage, and report on their log data - enterprises are scrambling to institute log management and intelligence solutions to become compliant or face financial ramifications and / or a public relations nightmare. LogLogic's solutions play well here, providing proactive enforcement and remediation through Log Management and Intelligence as well as a real-time view of adherence to multiple regulations and standards.

To lead the channel charge, LogLogic has brought on Richard Marquez as vice president of Partner Sales. Richard is a 20-year channel veteran superstar who most recently led Partner Sales at Computer Associates after spending 10 years at Symantec building one of the industry's most successful channel business. Welcome Richard! Click here to learn more about LogLogic's Partner Program. You can also register your interest in becoming a partner.

Posted by LogLogic on September 13, 2007 in Log Management & Intelligence | Permalink | TrackBack (0)

Read "PCI Compliance" book chapter on logging!

As you know, the "PCI Compliance" book is out. Syngress/Elsevier publisher released one chapter from the book for free: and it is my chapter on log management and logging in PCI! Enjoy it here! [PDF] The book (and, in fact my logging chapter!) already got some glowing reviews.

While people still argue whether PCI is simple or overly complex, PCI DSS is certainly motivating a lot of people to take a long hard look at their IT security ... For example, more people are starting to look at database logs as a result of PCI. While some vendors are still lagging behind, you can get your database logs into LogLogic today!

Posted by on August 24, 2007 in Compliance , Log Management & Intelligence , LogEd , LogMatters , Security | Permalink | TrackBack (0)

Logging Glossary: Log Preprocessing

As I mentioned here,  I started publishing the LogLogic Logging Glossary. So, here is the eleventh term (first second third fourth fifth sixth seventh eighth ninth tenth):

Log Preprocessing

1. The process of modifying or transforming a message as preparation for later stage processing.

Preprocessing can include multi-line message handling, encoding conversion, and special character substitution.

2. The prerequisite analysis of headers or other information in a log that define the formats or values present in the log messages.

The preprocessing assures correct parsing and identification of values. 

Posted by on August 20, 2007 in Log Management & Intelligence , LogEd , LogMatters | Permalink | TrackBack (0)

Just What is "Scalability"?

Remember that discussion about syslog servers? While some people might try to sell you a mere syslog server for big money, others fall in the opposite end of the spectrum: they think that building a high-performance scalable log management system is as easy as installing a syslog on a machine ...

This fun story was related to me by Dimitri McKay, our network and systems engineer from the East Coast. He said: "Our biggest competitor in the field seems to be home-grown syslog servers. Although tons can be saved creating a generic "syslog server," at the end of the day, the resources and man-hours needed to maintain the thing offset the savings to total cost of ownership. I've seen several cases where a customer has a number of these servers running, 'grepping' their way through mountains of log data, and setting up script after script after script. Usually such a thing is started by someone very clever on that team, who gets bored, and moves on to something else, or some other job, and the syslog servers fall into disarray because nobody knows how and what to do with them.

When comparing an open source syslog server to a high-performance (75,000 log messages/second!) Loglogic ST, you do get what you pay for.

Specifically, one of our government customers hired a new employee who looked at the Loglogic ST3010 and expressed his disdain for commercial "out of the box" solutions. His exact words? "I can totally build that for like nothin'." So off our new friend goes, putting together a home-grown syslog server. 

First, he starts with a Linux kernel, installs a syslog collector, and then opens the flood gates direct from the Loglogic ST3010 via log message routing. The syslog server suddenly becomes unreachable, non-responsive to pings, SSH attempts, the whole nine. The flood gate of log traffic was then turned off.  Back to the drawing board.

Our new employee then comes back with two machines taped together in a cluster. The flood gates are opened, forwarding messages direct from the ST3010 to the home-grown solution, and the same result ensues. No response from pings, SSH, or even console access.

This continues on... three servers clustered, four servers clustered, five servers clustered... till the sixth server was added, and that's when our customer pulled the plug on his new hire. Too much time and hardware resources were being used in a project that was being done quickly and efficiently by one Loglogic ST3010 appliance. Why try to fix it, if it “ain’t broken!”

At the end of the day, you get what you pay for, and you pay for what you get."

Posted by on August 10, 2007 in Innovation , Log Management & Intelligence , LogMatters , Security | Permalink | TrackBack (0)

Anton Logging Tip of the Day #12: Proxy Log Fun - Proxy Log Analysis for Possible Information Leakage Detection

Following the new tradition of posting a tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it "pay it forward" to the community.

So, Anton Logging Tip of the Day #12: Proxy Log Fun - Proxy Log Analysis for Possible Information Leakage Detection

You probably know that web proxies (such as Squid, BlueCoat SG and others) produce a lot of detailed logs, that record all web traffic flowing through the proxy as well as pass/block decisions made by the proxy's content filters and possibly embedded anti-malware tools. Proxy logs can be used for a whole range of things, from routine monitoring for Acceptable Use Policy (AUP) compliance to malware detection as well as possibly looking for security scourge of 2007 - web browser attacks by malicious or compromised web servers.

Specifically, in this tip we will learn how proxy logs can be used for detection of file uploads and other outbound information transfers vie the web. First, think what is the legitimate use of file upload functionality in your environment. For example, if using web-based mail services is allowed, then sending an attachment will include an upload. What else? The rest will be considered at least suspicious...

In addition to file uploads, some malicious or commonly unauthorized applications will use similar methods to steal or transfer data, that will be reflected in proxy logs. Looking for HTTP methods (such as POST) and content-type in combination with either known suspicious URL  or user-agent (i.e. web client type) can often reveal spyware infections, actively collecting data. Admittedly, a well-written spyware can certainly fake the user-agent field so it is clearly not reliable, but still useful to add to our query above. 

So, here are some of the criteria we will use to look for information uploads in Squid and BlueCoat SG proxy logs:

• HTTP method (logged as "cs-method" by BlueCoat) = POST  (as opposed to the usual GET, used to retrieve web content).
• For information uploads: content type (logged as "RS(content-type)" by BlueCoat) = pretty much anything but "html/text" (which is the type used for uploading  web form contents) - especially try content types  "application/octet-stream", "application/msword", "application/powerpoint", "application/vnd.ms-excel", "application/pdf" and a few others to look for common file uploads.
• For spyware and application data transfers: user-agent set to anything but the common ones (i.e. not Mozilla, iTunes, LiveUpdate, etc) or even to "unknown." One can also try user-agent containing your favorite messaging app (e.g. "MSN Messenger", etc)  and see which such applications are in use.

(if you feel adventurous, other interesting content-types to try are "application/x-javascript" and "text/javascript")

Here are the examples found in proxy logs using the above query, including some "classics" (while spyware specimen are a bit dated, this method of  detecting them via logs is still relevant and useful):

1. 1124376766.026 RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/hotbar/4.0/HbRpt.dll
2. 1124392388.975 RELEASE -1 FFFFFFFF 810FFBF233584C330353CF0A8C31F5D2 503 -1 -1 -1 unknown -1/813 POST http://log.cc.cometsystems.com/dss/cc.2_0_0.report_u
3. 2007-05-19 03:55:12 160 10.1.1.3 - - - OBSERVED "Spyware/Malware Sources;Spyware Effects;Web Advertisements" - 200 TCP_NC_MISS POST text/html;%20charset=utf-8 http bis.180solutions.com 80 /versionconfig.aspx ?did=5342&ver=1.0 aspx - 10.1.1.2 273 175 - - none - -
4. 2007-05-21 03:10:40 4 10.1.1.3 Joanna- authentication_redirect_to_virtual_host PROXIED "Search Engines/Portals" - 307 TCP_AUTH_REDIRECT POST - http storage.msn.com 80 /storageservice/schematizedstore.asmx - asmx "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; MSN Messenger 7.5.0324)" 10.1.1.2 791 2566 - - none - -
5. 2007-05-22 21:35:09 215 10.1.2.237 200 TCP_NC_MISS 217 8122 POST http kenobi.example.com /exchange/john.smith/Drafts1/RE:%2520CustomerList.xls-2.EML - - DIRECT kenobi.example.com - "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)" PROXIED none - 10.1.170.42 SG-HTTP-Service - none –

The first three are traces of spyware (one was even identified by a BlueCoat content filter as "Spyware/Malware", the fourth is MSN Messenger-based activity while the fifth is emailing the Excel file via web mail.

Here are some other signs that will make the above log entry extra-suspicious is:

• A dead giveaway: upload happens to a "known bad" URL (e.g containing "gator" and others above) 
• Upload happens to an unresolved IP address (do a "whois" on it!)
• Uploads happens to a port not equal to 80 (i.e. the URL contains a port such as http://10.1.10.10:31337)
• Upload has confidential file name in the log entry (e.g. somebody dumb emailing a sensitive file to himself - as discussed here)

Overall, this log analysis method is good for casting a broad net to catch not just spyware-infected systems, but also unauthorized applications (e.g. method=POST and user-agent=iTunes), instant messaging (e.g. method=POST  and then by user-agent, content or URL), simple forms of data theft and document handling policy violations (emailing files to self via web mail: method=POST and sensitive file name present in the entry; also content-type set to popular Office file types) as well as other abuses of web access. As a result, proxy logs provide an extremely rich AND readily available source of data about threats that users face!

To top it off, one promising direction of future research is using web proxy logs to detect client-side exploits by malicious web servers (more on this in the near future!)

Possibly related posts:

• Anton Security Tip of the Day #6: The Other Web Log
• All my past security and logging tips

Also, I am tagging all the tips on my del.icio.us feed. Here is the link: All Tips of the Day.

Posted by on August 07, 2007 in Innovation , Log Management & Intelligence , LogEd , LogMatters , Security | Permalink | TrackBack (0)

Musings on 100% Log Collection

One of the most exciting, complicated and, at the same time, very common questions from the field of log management is the "what logs to collect?" question. This comes up during compliance-driven log management projects (in the form of "what to collect for PCI DSS compliance?") as well as operationally-driven (in the form of "what logs from this application do I need to detect faults and errors?") or security-driven log management projects (in the form of "which logs will help me during the incident response?")

What are the answers that one sometimes hear? Otherwise awesome log management guidance NIST 800-92 "Guide to Computer Security Log Management" confuses the reader with this fascinating blurb: "generally, organizations should only require logging and analyzing the data that is of greatest importance." And how do people to know which logs are of importance? (I did have a bit of a debate with NIST folks on that...)

Other answers are situation-specific and thus limited in their usefulness ("need IDS alerts + server logs to detect intrusions via correlation", "need all logs that show access to PHI").  I spoke about the pitfalls of "prioritizing before collection" in my presentation "Six Mistakes of Log Management" and its companion paper.  In some cases, such as the incident response scenario, you might be naturally leaning towards grabbing as much as possible, since you never know which bit will help you answer that dreaded "WHAT happened?!" question ...

On the other hand, there is a simple answer that doesn't suffer from the above issues: collect everything. However, many folks go into a state of shock upon hearing it :-) "Everything!?! HOW can you collect 'everything''? What about storage, bandwidth, hardware, etc?"

But you know what? It really isn't as bad as you think! Just think that:

1. Logs compress really well (1:10 to 1:15 compression ratios are not unheard of), so bandwidth and storage are less of an issue that initially estimated
2. Disk storage is cheap (and tape is cheaper still); holding a billion of log records might well cost  under $200 (!) in disk drive cost
3. Figuring out what you need, might need, can be "told to need", will need, etc is genuinely hard. You will never get it right!
4. Many logs don't need to get collected the next second after generation, thus allowing you to save some bandwidth by moving them when network is less busy.
5. Technologies exist to make sense of "everything", not just "hand-picked" and parsed logs. More on the "making sense" part later

Convinced yet? So, if you are pondering "what logs to collect?", try to switch your mindset into thinking "what will it take for me to collect everything?" You probably won't regret this decision!  At the same time, one can try to reverse this question and ask "why collect everything?" - in this case, the answer will be "because any other collection strategy is worse."

Related posts:

• Top 11 Reasons to Collect and Preserve Computer Logs

• Six Mistakes of Log Management

Posted by on July 19, 2007 in Compliance , Log Management & Intelligence , LogEd , Security | Permalink | TrackBack (0)

Logging Glossary: Log Header

As I mentioned here,  I started publishing the LogLogic Logging Glossary. So, here is the tenth term (first second third fourth fifth sixth seventh eighth ninth):

Log Header

A set of information at the beginning of a log file or the start of a log message.

The information usually consists of context information, or meta data.

It may have a fixed or variable format. Fixed formats and some variable formats are usually well defined in vendor documentation. Some variable formats, such as with Syslog are subject to vendor discretion.

Posted by on July 17, 2007 in Log Management & Intelligence , LogEd | Permalink | TrackBack (0)

A 'Syslog Server' for $X0,000? No!

I hate it when people call what we provide (i.e. log management) a "syslog server." I really do. Why will someone pay $X0,000 for just a box to "collect syslog?" No, really, why? I won't! It does indeed sound silly and wasteful.
By now, many people understand that log management is not about collecting syslog in one big trash can. You can do that much easier and cheaper if that is indeed your goal. Why would someone collect syslog in a trash can is a separate story :-), even though collecting logs is pretty useful at times. But using the collected log data is much more valuable!
So, please get it! Log management is about scalable (meaning you can deal with a lot of data) collection (yes, collection too) + retention (meaning storage and then destruction) + analysis (real-time and historical methods of making sense of data) of all types of log data (not just syslog!!!), and about making such data available for all organizational needs (security, compliance, operations, fun bed-time reading :-), etc).

Posted by on July 09, 2007 in Log Management & Intelligence , LogMatters | Permalink | TrackBack (0)

Top 11 Reasons to Look at Your Logs

As promised, I am following my Top 11 Reasons to Collect and Preserve Computer Logs with just as humorous and hopefully no less insightful "Top 11 Reasons to Look at Your Logs." 

1. The first reason is again disarmingly simple (is it, really? :-)). Read PCI DSS lately? Glanced at HIPAA? Suffer under FISMA? Yes, all of the above regulations say that you must not only have, but also review logs periodically.
2. Are your servers compromised now? How do you know if all your logs are stashed on a tape in a closet? Look at them! Now!
3. An incident happens. Really, who needs extra motivation to look at logs in such case? Using logs for incident response is a true "no-brainer" (however, you need to be pretty "brainy" to actually analyze them in case of an incident)
4. Users - from a CEO to a janitor. You do have to know what they do on your IT systems! How? Read the logs! Everybody leaves tracks.
5. Systems log plenty of errors. Sometimes they are silly, sometimes - benign. However, often they mean that "stuff" is about to hit the fan. Periodic review of logs reveals them and saves the day.
6. Network slowed to a crawl?  Applications are slooow? Server is not ... well, serving? :-) Where is the answer? In the logs, but you need to read them and understand them.
7. That policy you wrote a few months ago. Anybody following that? Anybody remembers that? Halloooo! Check the logs and you'd know.
8. By now you know that your auditor might ask for your logs. But do you know they might also check whether you looked at them? Do you? Review the logs and leave the record of this activity in the logs.
9. Change can be good. But then again, it may be the sign that your controls are lacking. Who changes what and when? From what and to what? Just review the logs.
10. Now, you hate looking at logs. You have too many (as if everybody else doesn't...)! In this case, look at a specific subset of logs that you never saw before- NBS. Or just deploy log management that can do it for you.
11. Logs can help you predict the future (if you review, know and love them :-)). Don't believe it? If you read them for long enough, you develop an ability to predict the future, albeit mostly future problems :-)

See also: Top 11 Reasons to Collect and Preserve Computer Logs. 

Coming soon: "Top 11 Reasons to Secure and Protect Your Logs"!

Posted by on July 06, 2007 in Log Management & Intelligence , LogMatters , Security | Permalink | TrackBack (0)

Our Very First Customer

Recently our first customer -- a financial company -- participated in a case study on Log Management as part of the SANS Institute's What Works series. The customer (sorry, we can't share their name) deployed our very first evaluation unit and was our initial beta site. It was facinating to hear Victor Hsiang reminisce about the early days of LogLogic from the customer's perspective -- and how we've evolved as a company, technology -- and now industry since then.

He talked about those first products. What we do well. What we could do better. He says we are mature (shh, don't tell anyone) and responsive. But most of all he says that we have evolved into a product that "helped us move forward in standardizing on one log management solution for "this large financial firm" globally." (Yes, we are blushing.)

Victor also talks about how he uses log management for Compliance (SOX) use case and discusses how the product is being used in live troubleshooting with the company's own customers. Staff's time is being spent on custom reporting, but he also explains that once they had a template and process, is simple. Our software team is particularly pleased he said the interface was "intuitive" and did not require training. (They liked it so much the GUI team is even are trying to use it as an excuse to get an extra day off next week :-)

Our favorite quote?

"It was literally bring the box in, or the appliance, install it in the rack, provide power, IP address it, give it a DNS and a gateway. Then exit the data center and go back to your desk and start to configure your devices to send logs to it. "

Oh and yes Victor, we'll look into that feature request. Check out the case study and 40 minute replay at SANS.

Technorati : Log Management, LogLogic, SOX

Posted by on June 28, 2007 in Compliance , Log Management & Intelligence , Risk Management | Permalink | TrackBack (0)

Defend In Depth

Earlier in June, Mark Ford, who lead's Deloitte and Touche's Security and Privacy Services practice, was interviewed by IT Security.

According to Ford, IT security is one of the top issues for CIOs. However, he also noted that a corporate focus on IT security is quicker to take hold in information-centric organizations (banks and other financial service organizations, for example) than in industries like manufacturing or retail that are more geared towards consumers and whose focus is on business operations. With the increase in regulatory focus of such mandates as PCI-DSS and SOX, this has changed over the past few years as corporations in a variety of industries need to have strong IT security in order to be compliant.

What does this mean? Ford commented that historically, companies have put emphasis on the perimeter threat as the key component of IT security. But now, security emphasis is shifting towards a layered defense of the IT infrastructure. The perimeter is still important, but can no longer be considered the main component.

A shift away from the traditional security approaches and towards log management and intelligence, which allows for just the layered sort of approach Ford approves.

Technorati : Compliance, LMI, Log Management, PCI Compliance, Sarbanes-Oxley

Posted by on June 25, 2007 in Log Management & Intelligence , Risk Management , Security | Permalink | TrackBack (0)

National Data Security Laws Evolving

Only about two-thirds of existing state data breach notification laws apply to state agencies according to Bruce Brody, vice president of information assurance at CACI in an article at Federal Computing Week.

FCW asks the question -- Would a federal breach notification law bring greater security and sanity to those who find their personal data has been lost or stolen?

The impact of emerging state laws targeting data security and how government agencies factor into being governed by those laws is quite an interesting read. And joining these local initiatives are a host of proposed national bills. The implications of these laws as well as the impact they could have on non-government organizations is going to be a hot debate. Even we got into the discussion, as our resident expert Anton Chuvakin (also LogLogic's Director of Product Management ) weighs in on the topic in the FCW piece.

Chuvakin notes that while existing state laws are already working to protect consumers, he cautions on the realities a national law could bring with it:

"Because many existing state laws are effectively working to protect consumers affected by data breaches, federal legislators must be careful not to pass a national law that is less rigorous than the laws many states have passed, said Anton Chuvakin, director of product management at LogLogic, a risk mitigation company. Were that to happen, he added, "some citizens could lose the protections they enjoy now."

As we noted earlier this year, ignoring data security mandates could cost plenty. Thanks to very high profile breaches like TJMaxx are not only making headlines, but some consumers are nervous about their data and privacy. And US Congress to the European Commission (EC), along with state initiatives in Minnesota, Texas, and California are popping up to deal with the issue.

Technorati : Compliance, Log Management, PCI Compliance, Sarbanes-Oxley, chuvakin

Posted by on June 12, 2007 in Compliance , Log Management & Intelligence | Permalink | TrackBack (0)

2007 Log Management Survey Detailed

Yesterday, we held a webcast with The SANS Institute to announce the complete findings of the 2007 Log Management Survey. The survey, sponsored by LogLogic for the third year in a row, polled more than 650 IT professionals in government, financial services, banking, manufacturing, healthcare, telecommunications, and education sectors from the North American Global 2000 (G2000) - Forbes's comprehensive list of the world's biggest companies.

The verdict? The G2000 continues to adopt log management and intelligence to end the 'logjam.' Turns out that despite its importance, security is not the prime motivation for log management. More than half of those surveyed reported operations management and monitoring the health of the network as the prime motivation for using log data. And, 43% indicated compliance with SOX, PCI and other mandates as the top priority. Download the Executive summary here or check out the Webcast findings here.

Technorati : Event Management, LMI, PCI, SOX, forensics

Posted by on June 07, 2007 in Compliance , Log Management & Intelligence , Risk Management | Permalink | TrackBack (0)

California Considers Making PCI DSS Law

This week California is considering a bill that would require organizations that accept credit and debit cards to follow the Payment Card Industry (PCI) Data Security Standard. Noncompliance could mean banks would have to cover the costs associated with notifying customers that their credit card numbers may have been stolen and the cost of replacing credit cards, at a cost that could run upwards of $1 million per breach, according to estimates in a California State Senate report in May that provided details on the bill. The California law would apply to anyone who wanted to do business with a California resident, according to this article at Government Executive blog.

The public backlash after the January disclosure of a major security breach by Massachusetts based retailer TJX has acted as a stimulus for attention and consumer protection mandates. Just last week Minnesota enacted the Plastic Card Security Act, based on the PCI Standard. And other states like Massachusetts and Texas are also considering laws. The Lone Star state's House voted unianimously to approve the PCI-related bill, but the state Senate closed its session before it could vote on the issue.

Log management can help out with complying with the PCI DSS regulations quickly, plugging into your existing IT infrastructure. For some tips, check out 7 Habits of Highly Effective PCI Compliance- a Forrester Webcast with analyst Khalid Kark, sponsored by LogLogic. A PCI book is on the way from LogLogic's Anton Chuvakin later this year.

Technorati : Compliance, Log Management, PCI

Posted by on June 07, 2007 in Compliance , Log Management & Intelligence , Risk Management , Security | Permalink | TrackBack (0)

ITIL Gets An Update

ITIL is getting an update after 7 long years. The UK's Office of Government Commerce (OGC) is updating their comprehensive documentation of best practices for IT Service Management. As part of the launch of the new ITIL V3, the organization is hosting a series of roadshows throughout the world beginning today in London -- with stops in San Jose, CA and Chicago, IL here in the US in June.

New ITIL V3 is to consist of 5 central books and an official introduction book, according to the OGC. These incorporate the best of ITIL V1, V2 and tested current best practice for ITSM. The 5 central books are made up of Service Strategy, Service Design, Service Transition, Service Operation and Continual Service Improvement.

CIO.com is featuring a good overview of ITIL with some pros and cons of using the framework. The short version is that ITIL is the industry's most widely accepted approach to IT service management, provides a cohesive set of best practices, drawn from the public and private sectors globally and is supported by field-tested implementation methodologies and assessment tools, certification and accredited training organizations.

LogLogic offers an ITIL Pocket Guide a pragmatic approach to ITIL implementations. There are 50 reports and 45 alerts in the LogLogic ITIL package to get you started with ITIL and Log Management and Intelligence. To obtain a copy of the guide, go here.

Technorati : Compliance, ITIL, LMI

Posted by on June 05, 2007 in Compliance , Log Management & Intelligence | Permalink | TrackBack (0)

Why Banks Are Moving to Log Management

Banking Information Security magazine is covering the emerging trend in log management as it makes it way through the banking sector. They profile LogLogic cusomer, Citizens & Northern Bank, a $1.2B bank out of Pennsylvania, that has made log management a requirement for meeting compliance mandates with Gramm-Leach-Bliley and Sarbanes-Oxley. Using log management, the bank's auditors now have a way to easily track and monitor log data and get compliant fast.

Citizen's Bank learned early on what other G2000 companies are now realizing -- log management is inceasingly becoming the weapon of choice in the quest for compliance. Banking, an industry known for security and scrutiny of IT products is joining a growing trend of deploying log management for security, forensics, loss prevention and compliance. Why? As the article explains, the Industry's Federal Financial Institutions Examination Council (FFIEC) says that "without real log management, organizations are out of compliance and at risk" and calls on companies to monitor their log data. From the article,

"As administrators responsible for various network devices and operating systems, we need to know what typical behavior is," says Pete Boergermann, head of MIS at Citizens & Northern. "When we look at events, we are more apt to know what we are looking at and respond."

Read more about Citizen and Northern Bank's log management deployment in this SANS What Works case study from last year. Log Management and Intelligence is on the IT and business agenda. Industry trends are available in the just-released market survey on log management adoption. The LogLogic-sponsored research study with the SANS Institute is available for preview here or join us with SANS for a presentation of the trend at a joint webcast.

The complete article is at Banking Information Security, please note that registration may be required.

Posted by on June 04, 2007 in Compliance , Log Management & Intelligence , Risk Management , Security | Permalink | TrackBack (0)

G2000 Logjam Continues To Spur Log Management Says SANS Survey

We teamed up with the SANS Institute again this year to survey the G2000 on the trends driving log management and intelligence. You can dowload a copy of the preliminary findings of the 2007 Log Management Survey or sign up to attend a webcast presentation of the results with SANS on June 6th.

Based on last year's findings SANS Institute CEO Stephen Northcutt said:

"In our analysis, we found that G2000 IT leaders are today not only validating the use case for log intelligence, but are signaling the increased need for the market to evolve even further with log services that cost-effectively allow them to exchange information with other solutions."

The 2007 survey saw that trend continue to evolve. Polling more than 650 IT professionals in government, financial services, banking, manufacturing, healthcare, telecommunications, and education sectors from the North American Global 2000 (G2000) - Forbes's comprehensive list of the world's biggest companies, key findings:

· Log data monitoring continues to grow exponentially. Of those surveyed over 61% report using log data to assess IT incidents and minimize downtime (an increase of 24% over 2006 survey results).

· Log data retention is up significantly, but most of the G200 and G2000 are still failing to meet compliance regulations. Despite regulatory recommendations or requirements that logs be maintained for three to five years, 11% say they keep log files between 30 and 90 days, 10% retain data for six months and 5% less than 30 days. Remarkably, a full 14% say are not sure how long they keep log data, relying on the system default as defined by their operating system.

· Security while important is not the prime motivation for log management. More than half of those surveyed reported operations management and monitoring the health of the network as the prime motivation for using log data. 43% indicated compliance with SOX, PCI and other mandates as the top priority.

· The quantity of stored log data is rising. 57% percent of survey respondents store logs from as many as 500 sources.

Who:     Dr. Anton Chuvakin, Director of Product Management LogLogic
         Mike Rothman, President and Principal Analyst, SecurityIncite,
         and author of The Pragmatic CSO

What:    Online webinar roundtable discussing log management trends and
         best practices.

When:    May 31, 2007, 2 PM ET / 11 AM PT

Registration: www.loglogic.com