« October 2007 | Main | December 2007 »
Anton Logging Tip of the Day #13: Into the Darkness ... or The Ominous World of Unix Binary Audit Logs
Following the new "tradition" of posting a security tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative.
So, Anton Security Tip of the Day #13: Into the Darkness ... or The Ominous World of Unix Binary Audit Logs
In this tip, we will take a peek at one of the most esoteric areas of logging: Unix binary audit logs. Solaris BSM and Trusted Solaris auditing is the least unknown :-) example of it, even though other Unix vendors have similar auditing capabilities - see this for HP-UX Audit and this for IBM AIX audit. Linux kernel audit is also pretty much the same thing. If you look for information on 'Solaris BSM audit logs' , you'd find plenty of tips on how to enable such logging, a little on how to manage/rotate the log files, a bit on how to survive the resulting data deluge and ALMOST NOTHING on what to do with the log data, which is kinda sad :-) After looking at BSM logs for a while, I developed an opinion that nobody has ever looked at them on a regular basis :-)
So, let's assume you enabled Solaris BSM kernel audit for user "root" and few other "interesting" users (there is no per-object logging in Solaris; other Unix variants do have it) via the following commonly recommended per-user configuration in /etc/security/audit_user:
root:lo,ad,fw:no
anton:all,-all:no
jsmith:all,-all:no
This configuration file pretty much leads to logging of everything that the above listed users do. Now, you have audit files growing like mushrooms in your /var/audit. What good does it give us? First, we need to convert the binary audit files into text - something along the lines of
# auditreduce -A /var/audit/20071127193515.not_terminated.SunUltra10 | praudit -l > /tmp/sol_box_11272007
will do. Now what? In this tip we will pick one thing to use these logs for: how use them to see who is trying to copy sensitive files off the system.
First, who is connecting out - lets's search the logs for 'connect' calls (if you are using LogLogic for it, use Index Search for this task; if not, grep will have to do, but be prepared to wait, possibly a looooooooooooong time :-)). A few recommended searches:
- "connect AND 172.16.10.*" or "connect AND NOT 172.16.10.*" (to look for connection to specific IPs or only to outside networks) or simply 'connect AND username' to look for connections by a specific user (if you log many users)
Here is an example found (with connect, IP and user in bold):
header,103,2,connect(2),,Tue Nov 27 11:36:46 PST 2007, + 193 msec,argument,1,0x4,so,socket,0x0002,0x0002,0x80d6,SunUltra10,0x0016,10.1.1.41,subject,root,anton,other,anton,other,29902,29720,0 1611 172.16.0.173,return,success,0
At this point we already know the user name of the user who run that connecting process since it will be in the results (you can also the user to search as I showed above).
Next, what are those connections - let's try to uncover which programs actually connected (BSM logs don't make that easy!). Let's search for process starts in the same time frame (within a few seconds):
- "execve AND NOT ls AND NOT <whatever other commands you don't care to see>" will give you a list of started programs.
Example found:
header,124,2,execve(2),,Tue Nov 27 11:36:46 PST 2007, + 115 msec,path,/usr/bin/scp,attribute,100555,root,bin,136,1573,0,subject,root,anton,other,anton,other,29901,29720,0 1611 172.16.0.173,return,success,0
OK, so somebody is trying to connect via SSH/SCP. Notice that both records - connect and execve - have the same timestamp, up to a second. Sadly, time and parent process ID ( which is in our case 29720) is all that correlates them together.
Finally, what file was affected (i.e. copied off the system via scp in this case) - more digging is in order; we again use the process ID and time. The easiest is to search for a file name or browse all records around the same time frame (might be A LOT!):
- "*secret.zip* AND anton" will work; we can add the above process ID and look for "anton AND 29720" (but expect a lot of data since this is a shell process ID)
For example:
header,135,2,open(2) - read,,Tue Nov 27 11:36:47 PST 2007, + 900 msec,path,/tmp/not-so-secret.zip.gz,attribute,100600,anton,other,0,32743959,18446744073709551615,subject,root,anton,other,anton,other,29901,29720,0 1611 172.16.0.173,return,success,4
What do we know now? This user connected to this system and MAYBE copied this file via, MAYBE via scp. How cool is that? (A: not cool at all, since we are not sure, but that is all we can do with such logs)
To conclude, if you can avoid dealing with Solaris BSM logs, please do so :-) On a more serious note, you now know why these logs were called "the ugliest logs ever." Even more seriously (but still pretty humorously), these logs are a classic example of trees that make every effort to obscure the forest, because these logs record system calls and not processes or user actions (and connect, execve and read are all logged separately). There are also many, many more idiosyncrasies where these come from :-)
Also, I am tagging all the tips on my del.icio.us feed. Here is the link: All Tips of the Day.
Posted November 30, 2007 in Compliance , Log Management & Intelligence , LogEd , Security | Permalink | TrackBack (0)
« October 2007 | Main | December 2007 »
Log Poll #3: What Do You Do With Collected Logs?
Time for another fun logging poll: What Do You Do With Collected Logs?
Vote here! This is my Logging Poll #3, below are the links to past polls with my analysis:
- Poll #1 "Which Logs Do You Collect?" (vote here, results so far here, analysis here)
- Poll #2 "Why Collect Logs?" (vote here, results so far, my analysis)
Posted November 27, 2007 in Log Management & Intelligence | Permalink | TrackBack (0)
« October 2007 | Main | December 2007 »
Protecting Logs from Admins: A Lost Battle?
One of the truly major challenges of log management is obtaining trusted logs of administrator activities, sometimes broadened to cover so-called "privileged users" (see my log trust pyramid post for more discussion). Many consider it to be the "lost battle" of logging. However, logging administrator access and actions is more important than ever today; it is one of the few workable ways of dealing with insider attacks.
So, I created this little table where I try to summarize some ways of protecting the C-I-A (confidentiality, integrity, availability) of various logs from administrators and privileged users (some successful and some very hard to pull off). Note that in the case of databases and applications, we need to protect their logs from DBAs and application admins and not from the underlying server platform admins.
UPDATE: table below might be corrupted in some blog readers; see the full table here.
| "C" - prevent admins from reading logs | "I" - prevent admins from changing logs | "A" - prevent admin from disabling logging | |
| Standard Unix | Very hard! Maybe stealth logging (sebek) | Remote logging via syslog to another server, append-only log files (via RBAC) | Very hard! But this is logged and thus can be detected (also: stealth logging) |
| Windows server | Very hard! Maybe stealth logging (sebek for Windows) | Pull the logs ASAP to a central server | Very hard! But this is logged and can be detected (also stealth logging) |
| Databases | DBA activity log stored outside the database (append-only access) | DBA activity log stored outside the database (append-only access) | DBA activity log stored outside the database |
| Firewalls and network gear | Remote logging via syslog to another server - no local logging | Remote logging via | Very hard! But this is logged and can be detected |
| IDS/IPS boxes | Remote logging to another server - no local logging | Remote logging to another server, inaccessible to admin | Very hard! But this is logged and can be detected |
| Misc enterprise applications | App admin log outside the app (not readable to application user) | App admin log outside the app (only appendable by the application user) | Very hard! But this is logged and can be detected |
UPDATE: table above might be corrupted in some blog readers; see the full table here.
Comments?
Posted November 26, 2007 in Innovation , Log Management & Intelligence , Risk Management , Security | Permalink | TrackBack (0)
« October 2007 | Main | December 2007 »
Imagining an Ideal Log Management Tool?
The idea came from Jeremiah Grossman (here) when he described "The Best Web Application Vulnerability Scanner in the World" thus: "Within a few moments of pressing the scan button it’ll find every vulnerability, with zero false positives, generate a pretty looking report, and voila you’re compliant with GLBA, HIPAA, and PCI-DSS. Of course, we all know such a web application scanner is simply not possible to create for a variety of reasons."
So, let's imagine the ideal log management application.
- Logging configuration: the ideal log app will go and find all possible log sources (systems, devices, applications, etc) and then enable the right kind of logging on them according to a high level policy given to it
- Log collection: it will collect all the above logs securely (and without using any risky super-user access ) and with little to no impact to networks and systems
- Log storage: it can security store the above logs in the original format for as long as needed and in a manner allowing quick access to them - in both raw and summarized/enriched form
- Log analysis: this ideal application will be able to look at all kinds of logs, known to it and previously unseen, from standard and custom log sources, and tell the user what they need to know about their environment and based on their needs: what is broken? what is hacked? where? what is in violation of regulations/policies? what will break soon? who is doing this stuff? The analysis will power all of the following: automated actions, real-time notifications, long-term historical analysis as well as compliance relevance analysis
- Information presentation: this tool will distill the above data, information and conclusions generated by the analytic components and present then in a manner consistent with the user's role: from operator to analyst to engineer to executive. Interactive visual and drillable text-based data presentation across all log sources. The users can also customize the data presentation based on their wishes and job needs, as well as information perception styles
- Automation: the ideal log management tool will be able to take limited automated actions to resolve discovered and confirmed issues as well as generate guidance to users so that they know what actions to take, when full-auto mode is not appropriate. The responses will range from full-auto actions to assisted actions ('click here to fix it') to issuing detailed remediation guidance. The output will include a TODO-list of discovered items complete with actions suggested, ordered by priority
- Compliance: this tool can also be used directly by auditors to validate or prove compliance with relevant regulations by using regulation-specific content and all the collected data. The tool will also point at gaps in data collection as it applies to specific regulations that the user is interested in complying
In other words, this magic black box will have raw log data shoveled from one side and will have answers to key questions that concern the IT users and their managers on the other side.
Am I nuts? Well, can I dream for a second? :-)
Posted November 06, 2007 in Innovation , Log Management & Intelligence , LogMatters | Permalink | TrackBack (0)
« October 2007 | Main | December 2007 »
LogLogic Ranked #9 in Fast 50 Rising Star
LogLogic is now officially seeking additional office space to house the recent flurry of awards and accolades they have received. The latest? The Deloitte Silicon Valley Technology Fast 50 Rising Star award is a special designation for fast-growth companies that have been in business a minimum of three years, but less than five and is part of the Silicon Valley Technology Fast 50 program, which ranks the 50 fastest growing technology, media, telecommunications, and life sciences companies headquartered in Silicon Valley. Rankings are based on percentage revenue growth over three years (2004-2006). In these years, LogLogic exhibited a revenue grown percentage of 1383% -- good enough to be ranked #9 -- not too shabby.
Posted November 05, 2007 in | Permalink | TrackBack (0)
« October 2007 | Main | December 2007 »
Data Domain CEO Frank Slootman Joins LogLogic Advisory Board
LogLogic welcomes Data Domain CEO Frank Slootman to its advisory board! Frank's company, Data Domain (a data center storage company), went public in June of 2007. Prior to Data Domain, Slootman served as an executive at Borland Software and Compuware Corporation. In his capacity on the LogLogic advisory board, Slootman will advise LogLogic on strategy and scaling operations as LogLogic continues its leadership in the fast growing Log Management and Intelligence market.
We're all excited because Frank’s experience leading a fast growing, appliance company through an IPO is of tremendous value to LogLogic at this point in our history -- his insights are like having a crystal ball into the future -- and a vote of confidence from a top executive in the industry. Welcome, Frank!
Posted November 05, 2007 in | Permalink | TrackBack (0)
« October 2007 | Main | December 2007 »
Top 11 Reasons to Secure and Protect Your Logs
Following my now-famous Top 11 Reasons to Collect and Preserve Computer Logs and Top 11 Reasons to Look at Your Logs, here is the promised "Top 11 Reasons to Secure and Protect Your Logs"
- Let's review why you are reviewing logs. Will logs that might have been changed by somebody, somewhere, somehow still be useful for items 1-11 from here? No? Secure them!
- Logs in court? Challenges abound! To respond to them, one needs to protect the logs so you can claim that they are both authentic and reliable.
- A human error still beats an evil hacker as the main cause of IT problems. Are your logs safe from it? Available when needed? Protect them from crashes and other faults!
- PCI DSS just says so: "Secure audit trails so they cannot be altered." Want to do it- or pay the fines?
- Do you protect financial records? Identity info? Passwords? Some of it ends up in logs (even if by mistake) - thus making them sensitive, protected information. Secure the C-I-A of logs!
- Do you look at logs during incident investigation? Do you want them to be "true" or full of random (if creative...) "stuff", inserted by the guilty party? Secure the logs!
- Think that "attacks vs logging" are theoretical? Think again. Are your logs safe or vulnerable? Is your logging tool 0wned?
- Syslog + UDP = log injection + log loss. Are you protected (reliable TCP, confirmed delivery, encryption - SSH, SSL, VPN)?
- Why modify logs? No, really, why change logs? If you never change logs - and you never should, unless it is appending to them - hash them right away after collection to make them immutable.
- Logs are backed up on tape - who will see them? Well, whoever restores the tape, that's who! Encrypt them to protect them from accidental and malicious disclosure if tape is lost.
- Why log access to logs? Same reason why you had the logs in the first place - to review who did what. Who broke through and stole the logs? Who browsed them without permission? Only logs will tell - if you have them!
Overall, one need to strive for having no holes in log safeguards from log birth to analyst conclusion based on log information...
Possibly related posts:
Posted November 02, 2007 in Innovation , Log Management & Intelligence , LogMatters | Permalink | TrackBack (0)
December 2007
| Sun | Mon | Tue | Wed | Thu | Fri | Sat |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| 9 | 10 | 11 | 12 | 13 | 14 | 15 |
| 16 | 17 | 18 | 19 | 20 | 21 | 22 |
| 23 | 24 | 25 | 26 | 27 | 28 | 29 |
| 30 | 31 |
Categories
Archives
- December 2007
- November 2007
- October 2007
- September 2007
- August 2007
- July 2007
- June 2007
- May 2007
- April 2007
- March 2007
- February 2007
- January 2007
- December 2006
- November 2006
- October 2006
- September 2006
- August 2006
- July 2006
- June 2006
- May 2006
- April 2006
- March 2006
- February 2006
- January 2006
- December 2005
- November 2005
- October 2005
- September 2005