« July 2007 | Main | September 2007 »
Read "PCI Compliance" book chapter on logging!
As you know, the "PCI Compliance" book is out. Syngress/Elsevier publisher released one chapter from the book for free: and it is my chapter on log management and logging in PCI! Enjoy it here! [PDF] The book (and, in fact my logging chapter!) already got some glowing reviews.
While people still argue whether PCI is simple or overly complex, PCI DSS is certainly motivating a lot of people to take a long hard look at their IT security ... For example, more people are starting to look at database logs as a result of PCI. While some vendors are still lagging behind, you can get your database logs into LogLogic today!
Posted August 24, 2007 in Compliance , Log Management & Intelligence , LogEd , LogMatters , Security | Permalink | TrackBack (0)
« July 2007 | Main | September 2007 »
Create Controls Once. Comply Many Times.
As we've been saying, it's better to establish controls once and then map those (single) controls to multiple compliance mandates than it is to comply with every compliance mandate individually. Richard has some more thoughts on this in what is a pretty useful guide:
Control and governance frameworks like COBIT and ISO 17799 can help organizations in three ways: understanding the dimensions of security and governance requirements, illustrating the many options there are to meet requirements and structuring an ongoing compliance program.
And:
Compliance with any regulation, contract or standard requires a structured cyclical approach to accomplish its goals.
Look out for our Webcast on this very topic coming-up soon.
Posted August 22, 2007 in | Permalink | TrackBack (0)
« July 2007 | Main | September 2007 »
From Compelling Idea to Market Success
We’ve always prided ourselves on our openness and transparency and with this in mind I wanted to make you aware of changes taking place at LogLogic. This will be one of my last posts as CEO as I turn my focus to other priorities. I won’t be going far away and remain an advisor to, and major shareholder in LogLogic.
Since joining the Company three years ago we’ve taken a compelling idea – creating the Google of Log Data – and expanded it to create the most powerful Log Management and Intelligence Platform in the market for collecting, storing, reporting, alerting and sharing log data from virtually any source. We do it faster, more easily, with more reliability, and, more openness than anyone else out there. Fortune 100 companies use us as a cornerstone of their compliance efforts. Government agencies use as a key component in their security frameworks. And, leading Internet businesses use us to keep their data centers running 24x7x365.
With over 350 customers and growing, there’s no question LogLogic is the market share and customer satisfaction leader. Last quarter Gartner recognized us as a leader in the SIEM Magic Quadrant. We’ve maintained a 100% commitment to the channel and validated out technology strength with the broadest set of OEM relationships in the industry. And, we’ve grown from a focus on the Nth American market to having a global sales and support footprint with customers in nearly every major market.
This success is directly attributable to the incredible team we’ve been able to build. A team spanning employees, customers and partners. I want to thank you all for your support and friendship – long may it continue. We have always focused on succession planning and as a result we have a rock-solid foundation in place to ensure this momentum continues. Dominique Levin will be stepping in as interim-CEO while we complete the CEO search started some 30 days ago.
As I said, I’m still here and looking forward to contributing to LogLogic’s incredible success. Keep on logging!
- Chris
Posted August 22, 2007 in | Permalink | TrackBack (0)
« July 2007 | Main | September 2007 »
Logging Glossary: Log Preprocessing
As I mentioned here, I started publishing the LogLogic Logging Glossary. So, here is the eleventh term (first second third fourth fifth sixth seventh eighth ninth tenth):
Log Preprocessing
1. The process of modifying or transforming a message as preparation for later stage processing.
Preprocessing can include multi-line message handling, encoding conversion, and special character substitution.
2. The prerequisite analysis of headers or other information in a log that define the formats or values present in the log messages.
The preprocessing assures correct parsing and identification of values.
Posted August 20, 2007 in Log Management & Intelligence , LogEd , LogMatters | Permalink | TrackBack (0)
« July 2007 | Main | September 2007 »
Just What is "Scalability"?
Remember that discussion about syslog servers? While some people might try to sell you a mere syslog server for big money, others fall in the opposite end of the spectrum: they think that building a high-performance scalable log management system is as easy as installing a syslog on a machine ...
This fun story was related to me by Dimitri McKay, our network and systems engineer from the East Coast. He said: "Our biggest competitor in the field seems to be home-grown syslog servers. Although tons can be saved creating a generic "syslog server," at the end of the day, the resources and man-hours needed to maintain the thing offset the savings to total cost of ownership. I've seen several cases where a customer has a number of these servers running, 'grepping' their way through mountains of log data, and setting up script after script after script. Usually such a thing is started by someone very clever on that team, who gets bored, and moves on to something else, or some other job, and the syslog servers fall into disarray because nobody knows how and what to do with them.
When comparing an open source syslog server to a high-performance (75,000 log messages/second!) Loglogic ST, you do get what you pay for.
Specifically, one of our government customers hired a new employee who looked at the Loglogic ST3010 and expressed his disdain for commercial "out of the box" solutions. His exact words? "I can totally build that for like nothin'." So off our new friend goes, putting together a home-grown syslog server.
First, he starts with a Linux kernel, installs a syslog collector, and then opens the flood gates direct from the Loglogic ST3010 via log message routing. The syslog server suddenly becomes unreachable, non-responsive to pings, SSH attempts, the whole nine. The flood gate of log traffic was then turned off. Back to the drawing board.
Our new employee then comes back with two machines taped together in a cluster. The flood gates are opened, forwarding messages direct from the ST3010 to the home-grown solution, and the same result ensues. No response from pings, SSH, or even console access.
This continues on... three servers clustered, four servers clustered, five servers clustered... till the sixth server was added, and that's when our customer pulled the plug on his new hire. Too much time and hardware resources were being used in a project that was being done quickly and efficiently by one Loglogic ST3010 appliance. Why try to fix it, if it “ain’t broken!”
At the end of the day, you get what you pay for, and you pay for what you get."
Posted August 10, 2007 in Innovation , Log Management & Intelligence , LogMatters , Security | Permalink | TrackBack (0)
« July 2007 | Main | September 2007 »
Anton Logging Tip of the Day #12: Proxy Log Fun - Proxy Log Analysis for Possible Information Leakage Detection
Following the new tradition of posting a tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it "pay it forward" to the community.
So, Anton Logging Tip of the Day #12: Proxy Log Fun - Proxy Log Analysis for Possible Information Leakage Detection
You probably know that web proxies (such as Squid, BlueCoat SG and others) produce a lot of detailed logs, that record all web traffic flowing through the proxy as well as pass/block decisions made by the proxy's content filters and possibly embedded anti-malware tools. Proxy logs can be used for a whole range of things, from routine monitoring for Acceptable Use Policy (AUP) compliance to malware detection as well as possibly looking for security scourge of 2007 - web browser attacks by malicious or compromised web servers.
Specifically, in this tip we will learn how proxy logs can be used for detection of file uploads and other outbound information transfers vie the web. First, think what is the legitimate use of file upload functionality in your environment. For example, if using web-based mail services is allowed, then sending an attachment will include an upload. What else? The rest will be considered at least suspicious...
In addition to file uploads, some malicious or commonly unauthorized applications will use similar methods to steal or transfer data, that will be reflected in proxy logs. Looking for HTTP methods (such as POST) and content-type in combination with either known suspicious URL or user-agent (i.e. web client type) can often reveal spyware infections, actively collecting data. Admittedly, a well-written spyware can certainly fake the user-agent field so it is clearly not reliable, but still useful to add to our query above.
So, here are some of the criteria we will use to look for information uploads in Squid and BlueCoat SG proxy logs:
- HTTP method (logged as "cs-method" by BlueCoat) = POST (as opposed to the usual GET, used to retrieve web content).
- For information uploads: content type (logged as "RS(content-type)" by BlueCoat) = pretty much anything but "html/text" (which is the type used for uploading web form contents) - especially try content types "application/octet-stream", "application/msword", "application/powerpoint", "application/vnd.ms-excel", "application/pdf" and a few others to look for common file uploads.
- For spyware and application data transfers: user-agent set to anything but the common ones (i.e. not Mozilla, iTunes, LiveUpdate, etc) or even to "unknown." One can also try user-agent containing your favorite messaging app (e.g. "MSN Messenger", etc) and see which such applications are in use.
(if you feel adventurous, other interesting content-types to try are "application/x-javascript" and "text/javascript")
Here are the examples found in proxy logs using the above query, including some "classics" (while spyware specimen are a bit dated, this method of detecting them via logs is still relevant and useful):
- 1124376766.026 RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/hotbar/4.0/HbRpt.dll
- 1124392388.975 RELEASE -1 FFFFFFFF 810FFBF233584C330353CF0A8C31F5D2 503 -1 -1 -1 unknown -1/813 POST http://log.cc.cometsystems.com/dss/cc.2_0_0.report_u
- 2007-05-19 03:55:12 160 10.1.1.3 - - - OBSERVED "Spyware/Malware Sources;Spyware Effects;Web Advertisements" - 200 TCP_NC_MISS POST text/html;%20charset=utf-8 http bis.180solutions.com 80 /versionconfig.aspx ?did=5342&ver=1.0 aspx - 10.1.1.2 273 175 - - none - -
- 2007-05-21 03:10:40 4 10.1.1.3 Joanna- authentication_redirect_to_virtual_host PROXIED "Search Engines/Portals" - 307 TCP_AUTH_REDIRECT POST - http storage.msn.com 80 /storageservice/schematizedstore.asmx - asmx "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; MSN Messenger 7.5.0324)" 10.1.1.2 791 2566 - - none - -
- 2007-05-22 21:35:09 215 10.1.2.237 200 TCP_NC_MISS 217 8122 POST http kenobi.example.com /exchange/john.smith/Drafts1/RE:%2520CustomerList.xls-2.EML - - DIRECT kenobi.example.com - "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)" PROXIED none - 10.1.170.42 SG-HTTP-Service - none –
The first three are traces of spyware (one was even identified by a BlueCoat content filter as "Spyware/Malware", the fourth is MSN Messenger-based activity while the fifth is emailing the Excel file via web mail.
Here are some other signs that will make the above log entry extra-suspicious is:
- A dead giveaway: upload happens to a "known bad" URL (e.g containing "gator" and others above)
- Upload happens to an unresolved IP address (do a "whois" on it!)
- Uploads happens to a port not equal to 80 (i.e. the URL contains a port such as http://10.1.10.10:31337)
- Upload has confidential file name in the log entry (e.g. somebody dumb emailing a sensitive file to himself - as discussed here)
Overall, this log analysis method is good for casting a broad net to catch not just spyware-infected systems, but also unauthorized applications (e.g. method=POST and user-agent=iTunes), instant messaging (e.g. method=POST and then by user-agent, content or URL), simple forms of data theft and document handling policy violations (emailing files to self via web mail: method=POST and sensitive file name present in the entry; also content-type set to popular Office file types) as well as other abuses of web access. As a result, proxy logs provide an extremely rich AND readily available source of data about threats that users face!
To top it off, one promising direction of future research is using web proxy logs to detect client-side exploits by malicious web servers (more on this in the near future!)
Possibly related posts:
Also, I am tagging all the tips on my del.icio.us feed. Here is the link: All Tips of the Day.
Posted August 07, 2007 in Innovation , Log Management & Intelligence , LogEd , LogMatters , Security | Permalink | TrackBack (0)
« July 2007 | Main | September 2007 »
Sustained Performance
A note from the field... The word “sustained” makes all the difference when talking about log management. I hear people, especially log management vendors, talking about messages per second peak numbers, but often when the pedal reaches the floor, those peek numbers are completely unachievable.
A couple weeks ago I was on-site at a customer of ours in the Caribbean. They are a local ISP using our appliance to monitor their Netscreen Firewalls. Now, the LX1000 has a sustained rate of 1500 event messages per second. This you’ll find on our marketing material, you’ll find it on our website, you’ll find it all over the web. The LX1000 handles 1500 messages per second. Sustained. This customer firewall was sitting at a steady 1400 messages per second at two in the afternoon.
The customer was concerned at first. This company is an ISP, and they didn’t know what sort of messages per second they would see during peak times of 4:30 to 9:30pm. We let the machine run over-night collecting data for the following days configuration of alerts and reports.
Day two, I sat down and took a look at the previous nights performance. The LogLogic appliance during the middle of the night had dropped to receiving a paltry 400 messages per second, however, we saw the average raise up to roughly 2300 messages per second, peaking at nearly 3200 messages per second.. and it did so without a single dropped message. That's 3200+ MPS flood of data without a single dropped message.
The word “sustained” is key when looking at a log management solution. If you are going to compare apples to apples, you really have to compare the apples to apples. Some log management solutions will show you an orange, and claim it to be an apple. When the rubber hits the road, make sure your log management solution can handle the rates they claim. - Dimitri
Posted August 07, 2007 in | Permalink | TrackBack (0)
« July 2007 | Main | September 2007 »
Project Lasso 4 Released
Project Lasso 4.0 is out! Project Lasso has been downloaded more than 10,000 times since its launch in 2006. Project Lasso can be used for collecting Windows events and sending them to Syslog servers. When used with LogLogic 4, Windows events can be alerted and reported on in real-time, securely stored, and easily shared with other applications and dashboards.
Project Lasso collects all log data from Windows hosts without the need for any agents or code installed on the remote system - this speeds up deployment and reduces administration, leading to a much higher ROI. Windows DLL files contain critical information relating to the log messages themselves.
LogLogic has cracked the code on remote collection by combining the log data and the DLL information to produce actionable information in a format that allows it to be more rapidly searched and reported against.
LogLogic customers using Project Lasso in conjunction with LogLogic's Log Management Data Warehouse can combine Windows, Active Directory, Microsoft SQL, Exchange, IIS and ISA information with all the other platforms and applications (including custom or homegrown) within their enterprises.
For the first time large enterprises have an ability to track a user or IP address (on a global basis) from the time a connection is made (internally or externally) to every system and application that is then accessed. This end-to-end user activity monitoring and reporting from a single interface is proving invaluable to large enterprises needing to meet governance, risk and compliance requirements.
Project Lasso is available for free as a download from http://www.loglogic.com/logforge/
Posted August 02, 2007 in LogLogic News | Permalink | TrackBack (0)
« July 2007 | Main | September 2007 »
EU Steps-Up Logging Directives
To comply with EC telecommunications logging directives (as other EU nations recently have), the UK has passed a law that starting October 1 telecommunications firms must generate and retain logs of landline and mobile communications for one year. We'd be happy to assist!
Posted August 01, 2007 in | Permalink | TrackBack (0)
November 2007
| Sun | Mon | Tue | Wed | Thu | Fri | Sat |
|---|---|---|---|---|---|---|
| 1 | 2 | 3 | ||||
| 4 | 5 | 6 | 7 | 8 | 9 | 10 |
| 11 | 12 | 13 | 14 | 15 | 16 | 17 |
| 18 | 19 | 20 | 21 | 22 | 23 | 24 |
| 25 | 26 | 27 | 28 | 29 | 30 |
Categories
Archives
- November 2007
- October 2007
- September 2007
- August 2007
- July 2007
- June 2007
- May 2007
- April 2007
- March 2007
- February 2007
- January 2007
- December 2006
- November 2006
- October 2006
- September 2006
- August 2006
- July 2006
- June 2006
- May 2006
- April 2006
- March 2006
- February 2006
- January 2006
- December 2005
- November 2005
- October 2005
- September 2005