Logs and Metrics

Data Science and Machine Learning both require data and more accurately a bunch of information. Logs and Metrics are two great types of data collecting techniques. These ways don’t leave single info without being unnoticed. Both these have different use cases and different way of representation. Over the internet, you can find different Logs and Metrics collecting as well as representing tools. Some of them will be suggested below in the article. You will also come to know about what Google and Microsoft do with this enormous amount of data.

What are the Logs? 

Log files are required to store different events that occurred or occurring. The event can be like a login, visit a website, download of any kind of media, installing an application etc… Windows basically has three main types of event logs are listed below and the job is the same as the name represents application, system and security event logging. Logs can be basically understood as the work is done or performed with the help of a keyboard, mouse or other input devices to perform any task.

Windows events logs have different logs like

• Application Event Logs: The Application log records events related to different Windows system components, like different drivers and built-in interface elements present in the system.
• System Event Logs: The System log records events related to programs installed on the system.
• Security Event Logs: This records the logs related to logging in attempts and resource access.

windows event viewer

What are Metrics?

Metrics are basically evolved from the word measurement. Measurement of different hardware and system required to perform any task. For example, when you play a movie on your laptop different requirements occur like a monitor for display, speaker for audio, RAM to run the movie, Graphics for better performance. The measurement or amount of all these requirements is stored in a file which is called metrics. Metrics are collected and stored at a fixed or specific time.

Metrics of Hardware required

See firewall log file in Linux

# vi /var/log/firewalld

log command

logs

Representation of Logs and Metrics:

All the logs and metrics files have a way of representation or storing the data. This data is stored in a unique fashion so that it could be used further to get information. Each data unit is separated by some kind of delimiter like a comma, semicolon, tab, quotes etc… This representation is resolved by using general expressions.

Different delimiter used in metrics file

Google and Microsoft Use Case: 

Microsoft best use of logs is to troubleshoot the computer. These logs help Microsoft to diagnose the problem and resolve it. On the other hand, Google uses history to make things like advertisement and content more personalize and useful. Different events files are maintained for different tasks to be performed. Like if you face any problem with the system application, then the logs of system generated files will be checked. Both Google and Microsoft have different tools to monitor logs and metrics over the cloud platform GCP and AZURE.

Analyzing and Monitoring Tools