LogBlog

« June 2008 | Main | August 2008 »

Today's Logging Problems - Then Future Problems - Part I

Remember my write-up about an ideal log management tool?  Somebody asked me: "That's great that you have such a clear  vision of a future log management technology - but tell me first what future business problems will such 'ideal tool of the future' solve?"  First, I pointed at the fact that there are plenty of log-related problems today which we are not even close to solving. We need to solve the problems of today first, before we can get to solving the future problems.

So, what I consider to be the biggest log-related problems of today?

  1. Not knowing what to log - whether  for compliance, tracking attackers or troubleshooting system problems. Remember all the comedy about "Tell me EXACTLY what to log for PCI?" If not, reread it!
  2. Log volume  - there are too many log messages (seriously, 100,000 each second is a lot of log - but there is more at large companies!), and, which is worse, a lot of them are of unknown value to the users (might be useful, might not - but you never know in advance); thus, log clutter networks, systems and brains of security/system analysts.
  3. Log diversity - logs all look different (at least while standards are being developed) and no single person have the skill set to understand  more than a few types. PIX admin groking SAP logs? No way!
  4. In light of the above, just pure bad logs are also a major challenge - logs that miss a key piece of info (like the infamous "login failed" without the username of the user failing to login) or are useless in some other way are sadly common.
  5. How about getting the logs from all the places where they are located (think application logs here) - it is a problem if you want to expand your operational awareness to applications.
  6. Finally (not really, the list can go on and on), making sense of logs in  an automated fashion is still a #1 challenge  (IMHO) - we are getting better creating tools for humans to go through logs (via reports and search), but log->conclusion process still requires a human, and a darn smart one.

Now, when you read the above think "end user", not "log management  vendor" challenges. Along the same line, this picture from 4th SANS Log Management Survey shows how people perceive the logging challenges:

image_thumb

as well as my logging challenges poll (analysis here):

image_thumb1

Now, let's think of logging problems of the near future, say in 2 years.  But you'd have to wait for the next post for this :-)

Posted July 31, 2008 in Innovation , Log Management & Intelligence | Permalink | TrackBack (0)

« June 2008 | Main | August 2008 »

Log Management Project - Day One

Inspired by this and this here (and this too). It started from this example, coming from another domain:

“You’re hired on at a new company placed in charge of securing their online business. You know next to nothing about the technical details of the infrastructure other than they have no existing web/software security program and a significant portion of the organizations revenues are generated through their websites.

What is the very first thing do on day 1?”

At about the same time, I saw a message posted to one of the mailing lists where the poster wondered: "I’ve been asked to look into finding a replacement to our current log management/auditing system.  This is a field I haven’t even come close to touching before, and really don’t know the ideal things to look for (or ignore), etc. I’ve been searching through SANS site as well as googling, and I’m not coming up with a lot of great starter information. " And then he asks "Where should I start?"

This is indeed a really good question!  Let's rephrase the above for the case of logging:

"You’re hired on at a new company placed in charge of TAKING CONTROL OVER THE LOGS. You know next to nothing about the technical details of the infrastructure other than they have no existing LOG MANAGEMENT process and tools... What is the very first thing do on day 1?”

So the "Day 1" of a log management project. What's up?!

The very first thought that should cross you mind before you even do whatever first thing you wanted to do is "WHY?"

"Log management" is a solution, not a problem. What is your problem that you now have a mandate to solve? In other words "Why log management for you?" Logs server way too many different purposes so that proceeding without asking "Why?" is dangerous.

What is it that motivated your boss (or his boss, or whoever) to decide to "address this", to "take control over logs?" Was it a new compliance mandate, PCI perhaps? Was it a recent incident where investigation hit the wall due to utter lack of logs? Was it a new corporation-wide IT efficiency improvement project? Was it a lawsuit where an e-discovery request was not satisfied and thus fine was levied? Was it a hot IT project that is impossible to complete without having a tool to analyze logs?

This "need" is very important since logging is a huge realm and not focusing on the need is akin to starting a journey into a hostile wilderness without  a map - in other words, it might be fun for a while, but it will probably end badly.

Next, what do you actually do first? Figure out what logs are needed for this effort and what systems produce them (and who is responsible for them!) Analyzing SAP logs for J-SOX is a VERY different effort from analyzing Cisco ASA logs for network troubleshooting.

Only at this point you can start thinking about "tools:" parsers, logs, databases, reports, alerts, indexing and other technical things as well as capacity planning, scalability, etc. This is the stage where you learn the lingo and learn to cut through marketing messaging to get to the actual tool capabilities.

So, remember: given mandate to "tame the logging monster", think "WHY?" first!

Posted July 29, 2008 in Innovation , Log Management & Intelligence | Permalink | TrackBack (0)

« June 2008 | Main | August 2008 »

The many faces of privileged users

Privileged users are users who have legitimate access or administration rights to sensitive information or mission-critical systems.  87% of insider incidents are caused by privileged users.  Most are inadvertent violations of change management process or acceptable use policy.  Others are deliberate, mostly by disgruntled employees.

There are more “privileged users” in your organization than you might think:

Database administrators: Responsibilities include keeping the system available as well as performance optimized.  Continuously configure and re-configure the system as well as adding, removing, or updating user account information and privileges.

Identity and access management system administrators: May need powers to create new user profiles, add or change the privileges and access rights of existing users.

Server administrators:  Sysadmins are installing, supporting, and maintaining new servers and new software, including adding, removing, or updating user account information, resetting passwords, etc.

Storage administrators:  Access to granular information on file systems.  Responsible for backup and recovery schedules.  Allocating storage resources to applications.

Help desk employees: Many people in the help desk area have admin privileges on thousands of PCs for legitimate support purposes.  This gives them access to executive PCs and sensitive information.

Privileged users are really inevitable. Operating system and applications are limited in the granularity of providing specific administrator access rights. For example, in order to perform password resets on Windows the user requires administrator rights that provide much more than just reset password. Any time users have broad access rights, they can potentially abuse the trust we give in them and potentially abuse those access rights. This gives rise to an enterprise-wide methodology and system to manage and monitor access rights and privileges, rather than leaving this function to the individual systems and departments.

Privileged user management combines role management and privileged user monitoring. Role management and user provisioning systems can help to audit and design user privileges, and to centrally assign control user privileges. Privileged user monitoring is the “surveillance camera” logging all privileged user activity and allow alerting on suspicious actions and on-going review of all actions. In many cases just the fact that the activity is monitored is a good way to motivate people not to abuse their access rights. A log management system can help to establish a baseline of user activity, from which you can evaluate whether privileges are assigned on a “least privilege basis” and to perform enterprise-wide monitoring of privileged user activity.

Ultimately, privileged user management answers 3 critical questions about your privileged users:

1. Do I really need that many privileged users?

Get a clear view of who was doing what to the network and what information they were accessing.  Are there privileges which can be taken away without hurting productivity?  Log management systems can give you a baseline of user activity.

2. Do my privileged users really need that many privileges?

Are there users who have privileges that go beyond normal privileges for their job function?  Compared this to their job functions and then began paring down access.  Role management systems can give you a baseline of user roles and privileges.

3. Do I know what privileged users are doing with their access rights?

Do you check for abuse of privileges on a regular basis?  Do you know who are accessing critical information or making critical changes?  Log management systems can monitor and audit user activity.

Posted July 09, 2008 in | Permalink | TrackBack (0)

« June 2008 | Main | August 2008 »

Tough Security Questions for SaaS Providers - Part 2

[ Originally posted at OnSaaS ]

This is part 2 of the tough security questions for SaaS providers. In part 1 of the series, we asked the following questions:

1. Data Locality - Where's my data?
2. Data Segregation - How is my data segregated with other customers, potentially my competitors?
3. Data Access - Who can access my data in your company?
4. Access Audit - Who has accessed my data and where's my access logs?

We are continuing this discussion with the following questions in part 2.

5. How are the users authenticated and authorized?
6. Web Application Security - How secure is the SaaS provider's web application?
7. Data Breaches - How do you protect my data from insider breaches?
8. PCI DSS - Are you compliant with PCI DSS?

5. How are the users authenticated and authorized?

Companies have spent hundreds of man years and millions of dollars trying to setup single-sign-on systems inside the corporate firewalls. Most companies, if not all, are storing their employee information in some type of LDAP servers. In the case of SMB companies, a segment that has the highest SaaS adoption rate, Active Directory seems to be the most popular tool for managing users. In many cases, companies have designed their IT infrastructure so that all authentication, including VPN, web proxy, file server, and others will go through this single infrastructure. The process of employee onboarding and termination is much easier this way.

Just as companies start to have some success, the advent of the SaaS model changes the scenario again. With SaaS, the software is hosted outside of the corporate firewall. Many times user credentials are stored in the SaaS providers' databases and not part of the corporate IT infrastructure. This means SaaS customers must remember to remove/disable accounts as employees leave the company and create/enable accounts as come onboard. In essence, having multiple SaaS products will increase IT management overhead.

SaaS customers will start asking questions on identity and access integration and providers would be wise to design such features in early on. For example, SaaS providers can provide delegate the authentication process to the customer's internal LDAP/AD server so that companies can retain control over the management of users.

6. Web Application Security - How secure is the SaaS provider's web application?

One of the "must-have" requirements for a SaaS application is that it has to be used and managed over the web (in a browser.) This creates an interesting scenario. In the on-premise scenario, when a vulnerability is found, at least you have your firewall protecting the application so you may get a bit more time to patch it (assuming the application vendor provides the patch in a timely fashion.) However, in the SaaS world, there is no such luxury. Any vulnerability identified can potentially have detrimental impact on all of the customers. Even leading security companies aren't immune to security holes in their web applications.

Web application security is quite a hot topic these days and it's discussed by many security researchers such as rmogull and RSnake. Here's an interesting article on "What web application security really is".

Verizon Business recently released their Verizon Business 2008 Data Breach Investigations Report. Of all the breaches, 59% of the breaches involve hacking, with the following breakdown:

  • Application/Service layer -39%
  • OS/Platform layer - 23%
  • Exploit known vulnerability -18%
  • Exploit unknown vulnerability - 5%
  • Use of back door -15%

Attacks targeting applications, software, and services were by far the most common technique, representing 39 percent of all hacking activity leading to data compromise. This follows a trend in recent years of attacks moving up the stack. Far from passé, operating system, platform, and server-level attacks accounted for a sizable portion of breaches. Eighteen percent of hacks exploited a specific known vulnerability while 5 percent exploited unknown vulnerabilities for which a patch was not available at the time of the attack. Evidence of re-entry via backdoors, which enable prolonged access to and control of compromised systems, was found in 15 percent of hacking-related breaches. The attractiveness of this to criminals desiring large quantities of information is obvious.

Currently there's really no mandate or requirement for SaaS providers to provide detailed security analysis of the SaaS application. However, it would be wise for the SaaS providers to start considering something similar to what PCI DSS has required of the merchants:

  • 6.5 Develop all web applications based on secure coding guidelines such as the Open Web Application Security Project guidelines. Review custom application code to identify coding vulnerabilities. Cover prevention of common coding vulnerabilities in software development processes, to include the following:
    • 6.5.1 Unvalidated input
    • 6.5.2 Broken access control (for example, malicious use of user IDs)
    • 6.5.3 Broken authentication and session management (use of account credentials and session cookies)
    • 6.5.4 Cross-site scripting (XSS) attacks
    • 6.5.5 Buffer overflows
    • 6.5.6 Injection flaws (for example, structured query language (SQL) injection)
    • 6.5.7 Improper error handling
    • 6.5.8 Insecure storage
    • 6.5.9 Denial of service
    • 6.5.10 Insecure configuration management
  • 6.6 Ensure that all web-facing applications are protected against known attacks by applying either of the following methods:
    • Having all custom application code reviewed for common vulnerabilities by an organization that specializes in application security
    • Installing an application layer firewall in front of web-facing applications.

Additional sources of information provided as a starting point for more information on web application security would include


Trey Ford of Security Spin Control has a fairly good explanation of the recently released PCI information supplement on requirement 6.6.

SC Magazine also has an article on Deconstructing PCI 6.6 for the management folks.

7. Data Breaches - How do you protect my data from insider breaches?

In the Verizon Business breach report blog, Verizon Business stated that

While criminals more often came from external sources, and insider attacks result in the greatest losses, criminals at, or via partner connections actually represent the greatest risk. This is due to our risk equation: Threat X Impact = Risk
  • External criminals pose the greatest threat (73%), but achieve the least impact (30,000 compromised records), resulting in a Psuedo Risk Score of 21,900
  • Insiders pose the least threat (18%), and achieve the greatest impact (375,000 compromised records), resulting in a Pseudo Risk Score of 67,500
  • Partners are middle in both (73 39% and 187,500), resulting in a Pseudo Risk Score of 73,125

Many SaaS advocates claim that SaaS providers can do a better job at protecting the customers' data. Unfortunately, just because the data is now in the cloud, it does not reduce the risk of insider breaches. Insiders still have access to the data, they are just accessing it a different way. Just because the data is in the cloud, the responsibility of segregation of duties and access authorization still fall on the customers, not the SaaS or cloud computing providers. So yes, it may reduce the chance of insiders getting direct access to, say, a database, it does not in any way reduce the risk of insider breaches. In fact, it may even increase the possibility as you now have to take into consideration of the cloud or SaaS providers’ employees. They have access to a lot more information and a single incident could expose information from many customers.

SaaS providers should be prepared to answer questions on what tools and processes are utilized to ensure segregation of duties and protect from insider breaches. Remember, in the case of the mult-billion dollar insider incident at Société Générale, IT management had implemented all of the controls recommended by auditors, but nobody was monitoring them. So it's extremely critical to be able to show the processes around these security controls.

8. PCI DSS - Are you compliant with PCI DSS?

PCI DSS has a specific section for hosting providers (including SaaS providers):

Requirement A.1: Hosting providers protect cardholder data environment

As referenced in Requirement 12.8, all service providers with access to cardholder data (including hosting providers) must adhere to the PCI DSS. In addition, Requirement 2.4 states that hosting providers must protect each entity’s hosted environment and data. Therefore, hosting providers must give special consideration to the following:

A.1 Protect each entity’s (that is merchant, service provider, or other entity) hosted environment and data, as in A.1.1 through A.1.4:


  • A.1.1 Ensure that each entity only has access to own cardholder data environment
  • A.1.2 Restrict each entity’s access and privileges to own cardholder data environment only
  • A.1.3 Ensure logging and audit trails are enabled and unique to each entity’s cardholder data environment and consistent with PCI DSS Requirement 10
  • A.1.4 Enable processes to provide for timely forensic investigation in the event of a compromise to any hosted merchant or service provider.

A hosting provider must fulfill these requirements as well as all other relevant sections of the PCI DSS. Note: Even though a hosting provider may meet these requirements, the compliance of the entity that uses the hosting provider is not necessarily guaranteed. Each entity must comply with the PCI DSS and validate compliance as applicable.

Simply put, SaaS providers must be compliant with PCI DSS in order to host merchants that must comply with PCI DSS.

We will continue with our tough security questions in part 3 of this series.

Posted July 07, 2008 in Cloud Computing , Compliance , SaaS , Security | Permalink | TrackBack (0)

« June 2008 | Main | August 2008 »

Tough security questions for SaaS providers - Part 1

[ Originally posted at OnSaaS ]

We will be writing a series of blog posts on the tough questions that SaaS providers can expect to get from customers or they should ask themselves. The questions will span many different areas including security, compliance, sales, marketing and operations. This is Part 1 of the security questions.

As we mentioned previously here, one of the biggest obstacle to enterprise SaaS adoption is the issue of trust. Customers are asking SaaS providers "Can I Trust You?!" The security analysts and warriors are asking similar questions.

Some SaaS advocates have argued that concerns for SaaS security are just red herring. It is true that, to date, there hasn't been any major breaches amongst SaaS providers. However, we have already seen some activities such as the breach at Salesforce.com. In addition, we have seen many anecdote evidence that multi-tenant architectures in the B2C (e.g., Flickr, YouTube) world are prone to data leakage.

One may argue that these are consumer sites and are not relevant for the SaaS providers. However, the same technologies and architectures are being used in both the consumer and enterprise world. In fact, as the trend of IT consumerization continues, we will see more and more of the consumer technologies being used in enterprise applications. Think about it this way, what if this Salesforce.com and your customer list popped up in your competitor's screen?

SaaS providers should be prepared to answer security questions from customers and enterprises. Here are a list of questions that SaaS providers will likely get asked during customer trials/evaulations.

1. Data Locality - Where's my data?

Due to compliance and data privacy laws in various countries, locality of data is of utmost importance in many enterprise architecture. For example, in many EU and South America countries, certain types of data cannot leave the country because of potentially sensitive information. In addition to the issue of local laws, there's also the question of whose jurisdiction the data falls under when an investigation occurs. In most cases, the government where the data is housed will likely win. A good example of this type of concern is when the French cabinet banned the use of Blackberry devices.

Many enterprises have architected around these issues for the on-premise software they install. However, with cloud computing and SaaS, this issue is even more exasperated. In a cloud computing environment, sometimes you don't know where your data is stored or where your application is being run; and some proponents of cloud computing are also saying that you shouldn't have to worry about where the computing resources are as long as your application is running and behaving as it should. However, other leaders in the cloud computing space are taking note of the data privacy and locality issues. For example, Amazon recently announced the availability of an European S3 cloud, and Salesforce.com is also planning Singapore data center.  

Given the regulatory compliance and data privacy concerns, SaaS providers should be ready to answer tough questions about where their computing resources are and will customer data be ever transferred outside to another jurisdiction with different laws.

2. ata Segregation - How is my data segregated with other customers, potentially my competitors?

Everyone's talking about the benefits of multi-tenancy in the SaaS world, but many seem to ignore one of the biggest security concerns, mixing customer data together, that came along with multi-tenancy.   

One of the reasons that hampered SaaS adoption initially was trust. End users must trust that the SaaS providers have the best security in place to protect their data and never expose their data to anyone outside of the authorized domain. Therefore, the ability to segregate data by end customer is a critical requirement for the SaaS providers. There are many architectural methods in segregating the end customer data. At the end, the requirements come down to that users must never see the data that they are not authorized to see, and that end customer's data should never be exposed to other end customers. 

Saas Providers would be wise to consider data segregation early on in the architectural design. For most ISVs turning into SaaS providers, this is an unfamiliar territory and should seek guidance if possible. SaaS providers should also understand the customer concerns and address them early on.

3. Data Access - Who can access my data in your company?

Enterprises have spent hundreds of thousands of dollars on identity and access management systems, log management systems and other software to ensure that employees access only information they are allowed. Within the confines of their firewalls, IT organizations may feel that they have the situation somewhat under control. The advent of SaaS have changed that. With the company data outside of the firewall and in a "cloud," IT organizations no longer can control who and when their data-in-the-cloud will be accessed. Without visibility into the cloud, IT organizations are accepting a much bigger risk compare to when everything's inside the firewall. Even though many SaaS providers have offered various capabilities such as authentication integration with customers' own LDAP servers, this perception of lost control is a difficult hurdle to get over.

SaaS providers offering cloud services, whether it's infrastructure, platform or application, should accept the responsibility of protecting customer data as a single breach could affect all of the customers. SaaS providers must be prepared to help customers understand their security policies on user access, activity monitoring as well as segregation of duties.

4. Access Audit - Who has accessed my data and where's my access logs?

The last few years we have seen a rise of log management and SIEM solutions aimed at compliance-aware organizations. These products is responsible for collecting, analyzing, correlating, archiving and reporting on all activities happening inside an IT infrastructure. Part of the reason these products became such a success is because of the need to track and monitor user activities in the world of regulatory compliance. In addition to compliance, IT organizations use logs to help them identify security issues, perform troubleshooting and forensics analysis, and analyze traffic and user patterns.

With software in the cloud, network, system and application logs are no longer easily accessible by IT organizations. They either have to negotiate access to these logs during contract time, or they have find new ways of monitoring user activities. Given that the IT organizations don't "own" the software, it makes it even more difficult to "hack" around the system. Without access logs, IT organizations may not be able to answer simple questions from auditors, such as "who have accessed the financial information in the past quarter?"

Knowing how critical access logs are to compliance, operations and security matters for IT organizations, SaaS providers should consider providing access logs as a part of their normal service or have it as an option for customers. As an example, Amazon's S3 service offers options to enable and download access logs.

We will continue with our tough security questions in part 2 of this series.

Posted July 07, 2008 in Cloud Computing , Compliance , SaaS , Security | Permalink | TrackBack (0)

« June 2008 | Main | August 2008 »

Security First, Software-as-a-Service Second

In a recent blog post, we advocated the adoption of the PCI Data Security Standard by SaaS service providers.  From recent CIO surveys, it is clear that enterprises are placing high priority on Security. In fact, Security features much more prominently than implementing SaaS, which bodes well for careful consideration of privacy and security concerns in the cloud-computing world (including the potential adoption of a security standard for SaaS providers):

1. In a Morgan Stanley CIO Survey (June 2008) SaaS/OnDemand is the lowest spending priority listed with 3% of votes as compared to 23% voting for Security as a top spending priority. A similar survey by Merrill Lynch (June 2008) found that Security is the Top 1 spending priority (36% of votes). The Top 2 spending priority – integration – received 16% of votes. SaaS did not make the Top 10.

2. While 50% of CIO’s in a Merrill Lynch survey (June 2008) cite some use of SaaS technology (growing to 72% in 3 years), only 4% of software functionality is delivered as SaaS, projected to be 12% in three years. Compare this to virtualization, another hot technology: 73% of CIO’s use some virtualization, adding up to 17% of total capacity today, growing to 50% in 3 years.

3. Co-incidentally, SaaS is one of the first technologies to be cut according to the Merrill Lynch June 2008 survey. When asked “if you had to defer spending due to the soft economy, how do you prioritize spending”: virtualization would still receive an INCREASE of 71% in spending, whereas SaaS/OnDemand will see a DECREASE of 20%.

Posted July 07, 2008 in | Permalink | TrackBack (0)

Visit loglogic.com

I ♥ Logs

Subscribe to this blog’s feed RSS

August 2008
Sun Mon Tue Wed Thu Fri Sat
          1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31            
Categories
Archives
Blogroll
Blogroll
Compliance
Good Reading
LogLogic
LogLogic Partners
Sites We Watch