Logs analysis and understanding, and, more generally, business systems and applications data-in-motion, are increasingly hot topics for any organization.
These digital streams are literally exploding in all IT departments around the world, and are emerging as one of the most difficult but interesting challenges to face.
Today, who is in charge of log management, how can achieve at the same time a technological (the log is a Big Data asset), regulatory (in compliance with GDPR compliance) and Business evolution? Are the software currently on the market able to give an effective answer to all these issues? And do Vendors and System Integrators have the transversal knowledge to address the new log lifecycle paradigm?
A Useful Resource For All Corporate Actors
We consider that any system and business application write every day huge amounts of data for every single digital action performed by internal or external users, their lack of interpretation could make lose important opportunities to the different Corporate actors.
IT Operations, for example, thanks to the real-time analysis of large and heterogeneous log streams, could understand more quickly the origin of certain problems occurred to the infrastructure.
CyberSecurity could correlate more information, providing the SOC with greater investigative capacity to resolve security breaches or prevent potential threats.
Business could obtain more detailed reports on customer behavior and synthetic dashboards on clustering, because Corporate logs hide data that no type of Analytics (including Google) can provide with such breadth and depth.
Audit – perhaps at the request of the Judiciary – may have to verify a certain operation of a user that occurred many months before (which would require a retention policy of this data, often, on the contrary, often only trashed a few hours after their generation).
Data-In-Motion Governance
These are just some of the various examples of use cases related to data-in-motion, which make us understand how much is increasingly necessary an extended log content governance.
From the choice of information sources, to the configuration of the related connectors, from real-time ingestion of huge volumes of raw data from heterogeneous sources, to the widest and most correlated content analysis possible, from the publication of customized dashboards according to the requesting user, to the retention management, up to the recovery of specific temporal portions of information, perhaps archived months before, with the aim of defining a single point of log collection and storage, whose access is adequately profiled and secured.
Market Solutions not for All Need
There are interesting solutions that only partially meet these requirements, such as ELK stack, which requires in-depth knowledge for value recovery in the case of massive volumes, and a significant amount of development to tailor such a broad and diversified framework to the specific needs of a company.
Solutions such as Graylog, which do not include some key features, including log retention management.
Solutions such as Splunk, which have considerable pricing, and not designed for all needs, given the scope mainly configured on security, infrastructure, but not applications.
The Lifecycle Log Paradigm
In any case, log lifecycle governance does not only need a good software solution, but also a new paradigm.
The growing need to recover and understand huge amounts of raw information passing through increasingly complex architectures, together with privacy regulations and the dictates imposed by Data Protection, in a historical period in which the number of digital information grows exponentially and the diversification of use cases is potentially unlimited, a clear and innovative vision of the exploitation of the “log” resource is essential.
In the era of Big Data, if we were to think of information systems as an iceberg, the log would represent the submerged part and, as repository of an enormous and often unknown information heritage, it can no longer be considered only a technical asset, but must be recognized as one of the most interesting digital resources for an evolved company that wants to bring new value to its business.
The “log lifecycle” paradigm cannot therefore ignore the adoption of an Enterprise-class software platform that integrates both a cutting-edge technology and specialized expertise…