To have full visibility into modern cloud environments, businesses need to collect an ever-growing avalanche of log data from a range of highly complex data sources. Indexing logs is key for real-time monitoring and troubleshooting, but it can quickly become expensive at high volumes, meaning that organizations often must choose which logs to index and which to archive. But there are many situations where businesses require complete access to long-term historical logs, without needing sub-second query response times. For example, security investigations and compliance audits may require querying logs from the past year or more. Or, organizations may want to analyze trends across high-cardinality data sets over long time periods. Additionally, DevOps teams creating postmortems or troubleshooting support issues may need to look back at log data from many months prior to the incident itself. But current logging solutions don’t offer a cost-effective way to store and query your complete log data over a long time window, forcing customers to make tradeoffs and lose critical visibility.
That’s why Datadog developed Online Archives, an always-on log warehousing solution that allows you to retain and search all of your log data for 15 months or more for the same amount it costs to index data for one month. Online Archives is an alternative to indexing, meaning teams will be able to continue using indexes for real-time log streaming and alerting, and use Online Archives for situations requiring historical investigation and analysis.
Perform historical analysis and investigations with ease
Having the ability to search and analyze huge amounts of historical log data is vital in a number of different situations that don’t necessarily need immediate query responses. These include things like running security investigations across large environments, performing audits to adhere to strict compliance frameworks, and running long-term analytics on high cardinality datasets.
For example, when you experience a security breach or receive a report of an insider threat, your security team will need to comb through weeks, if not months, of log events to identify malicious activity. An investigation of all the activity from a particular, suspicious IP address may require scanning petabytes of data, assessing the timeline of activity from that IP, and generating reports for other teams (e.g., legal and executive).
Similarly, businesses operating in regulated industries—such as financial services, insurance, healthcare, and aviation—have stringent requirements around servicing audit requests. Keeping vast amounts of log data in a queryable state for any requests from auditors or government agencies gets expensive. And any follow up requests from auditors will require you to constantly provide up-to-date reports.
E-commerce providers, digital content makers, sports and entertainment companies, and businesses using IoT devices frequently need to perform long-term analytics on high cardinality datasets, such as users, IP addresses, device IDs, or items purchased, among others. As an example, a gaming company might want to compare the long-term trends in signups across different platforms for millions of users. Creating custom metrics to track signups requires foresight and will be very expensive. Instead, you can now use Online Archives to run data analysis queries on huge cardinality datasets for a fraction of the cost.
Online Archives addresses all these use cases by keeping logs queryable for 15 months or more, so teams don’t spend time spinning up new solutions, moving data between tiers, or worrying about query capacity and associated costs.
Effortless configuration and data exploration
Online Archives can warehouse logs regardless of whether they are indexed or not. For indexed logs, you can simply attach a new Online Archive to an existing log index.
Online Archives are a superset of all your logs routed to an index. Even when you have set exclusion filters on your indexes, logs filtered out will still be available in Online Archives.
You can query and analyze your logs stored in Online Archives directly from the Log Explorer the same way as you would with other indexed logs.
Start using Online Archives
Online Archives is currently in limited availability. Go here to request further information, or reach out to a representative at Datadog to discuss your use case and retention requirements. If you’re new to Datadog and want to monitor your logs, metrics, distributed request traces, and more in one fully unified platform, you can start a 14-day free trial.
