Logblog: From Native to Instrumented

« Protect Your Data | Main | A little known fact on log management and SIEM »

From Native to Instrumented

According to Wikipedia:

To log is a verb derivative of the noun logbook; the verb form means to record in a logbook, and may have been coined in the 1820s. The term logbook itself stems from the practice of floating a stationary "log" (actually a wooden block attached to a reel via rope) to provide a fixed point of reference for the purpose of measuring a ship's speed (see Knot (speed)). Computer scientists adopted the verb to log circa 1963 to describe the systematic recording of specific types of data processing events.

Logs have been around since the dawn of computers. It's been used by computer scientists, IT administrators, security analysts and network operators to perform analysis, troubleshooting and forensics for over 45 years. In most cases, however, users refer to logs as pieces of information that devices and applications generate on their own. For example, routers and switches generate logs that detail their status and what they are doing. Firewalls generate logs showing the various connections passing through, or not. Operating systems and utilities generate logs to communicate accesses to different parts of the system. Web proxies generate logs to describe user surfing activities. These logs are what one would consider to be "native logs."

One type of logs that users don't generally talk about, or event consider, are logs that are generated by agents or systems monitoring network or applications. This type of logs is called "instrumented logs." The most well-known type of instrumented logs is probably IDS logs. IDS monitors the network and reports any attacks that are happening. These reported events are usually considered as logs. However, there are other types of "instrumented logs" that users don't normally consider as logs. For example, most of the application performance management (APM) tools use agents or other means to retrieve information or statistics from various applications. These information are then sent to a central server for processing. In most cases, these information are not considered to be logs, but they do fit the definition of "systematic recording of specific types of data processing events."

Another example of instrumented logs are what Adrian Lane discussed in his blog post "Database Activity Monitoring & Event Collection Options." In this post, he mentioned several methods that monitor monitor database activities via sensors that either monitor the OS stack or the database memory. All of these methods generate "instrumented logs" that are sent to a central server for archival and analysis.

At the end of the day, whether it's "native" or "instrumented" logs, they are still pieces of valuable information that must be collected, archived, and analyzed. Also, the way these logs are analyzed are the same regardless whether it's "native" or "instrumented." As Jon Oltsik said,

In today's dangerous security landscape, no data is considered "noise" anymore. Rather, security analysts now want access to terabytes of historical data for analysis. Furthermore, this underlying data has become more complex.
.
.
.

It means collecting, normalizing, and storing a ton of data. It means sophisticated algorithms and processor-intensive query engines.

As sophisticated enterprises move up the stack (Network to OS to Applications) in their log management projects, we will likely see more and more of the log data come from sensors instrumenting the applications. This type of "instrumented" logs provide another rich set of information, sometimes richer in information compare to their "native" counterparts. Existing log management and security event management solutions can then take advantage of this set of information for compliance management, threat management and fraud detection.

Posted January 23, 2009 in Innovation , Log Management & Intelligence | Permalink