Drowning In Security Data Costs? You Get A Data Lake

A common client request I’ve gotten over the past several years is how to best manage growing data costs in the security information and event management (SIEM) system. For most, it requires a strategic approach to storing and accessing the data; either use cold/frozen storage, separate analytics, and ingest using a data cloud like Snowflake; or use a data pipeline management tool to reduce data volumes and potentially route it to a lower cost storage option. Since Amazon Security Lake popped onto the scene in 2023, many have used it as a low-cost option to store long-term data in the Open Cybersecurity Schema Framework for easy access. Other vendors have also introduced storage solutions for low-cost, long-term data storage (e.g., Cribl Lake), which can be especially useful if you are already using the tool for data routing.

Data, Data Everywhere, And No Perfect Solution

Still, security data management issues have persisted. In The Forrester Wave™: Security Analytics Platforms, Q4 2022, one piece of customer feedback Microsoft Sentinel customers gave was that the offering is costly because its pricing model is based on the volume of data ingested and predicting costs can be difficult. Similar concerns came up across vendors in the recently-released update of that report, The Forrester Wave™: Security Analytics Platforms, Q2 2025. Although it’s not the only SIEM system in which customers have had this challenge, it’s the one we are talking about today, as Microsoft just announced the Microsoft Sentinel Data Lake.

Microsoft Takes The Data Lake Plunge

Microsoft Sentinel Data Lake is now a feature of Microsoft Sentinel, providing a low-cost data storage option that is still accessible in the platform. In a major architectural change, it shifts the platform to having two data tiers: the analytics tier (more expensive, used for detections, investigation, etc.) and the data lake tier for long-term storage.

According to Microsoft, data retention in the data lake tier is priced at less than 15% of its traditional analytics logs. You can still access the data in the data tier using KQL and create retrohunts (scheduled or otherwise) across the data that promote the data into the analytics tier (for a fee, of course). Users can also interact with the data using the Microsoft Sentinel Visual Studio Code extension and PySpark. This can aid better data exploration through Jupyter notebooks, a pivotal change that speaks to users’ growing need to have better control and understanding of their data for detection engineering.

Carry Your Own Water To Learn The Value Of Every Drop

An African proverb says, “Once you carry your own water, you will learn the value of every drop.” This also applies to security data. Even with a security data lake like Microsoft Sentinel Data Lake, you still need to be strategic with the data you bring into the platform. Before this, we saw some customers make sacrifices with the data they ingested into Sentinel versus the data they put into Azure Log Analytics so they could have that long-term storage accessible in some form. This simplifies the equation by giving an option in which long-term data is made to be used and potentially promoted in Sentinel directly. It’s still critical to decide what data you need immediately for detection and response versus what data should be stored long term for access for compliance and threat hunting.

But Wait, There’s More

Another part of the Microsoft announcement that may have slipped under the radar is that Microsoft Defender Threat Intelligence will be converged into Defender XDR and Sentinel at no additional cost, starting in October 2025. This is in line with changes from Cisco Splunk, which now integrates Cisco Talos threat intelligence into the enterprise security license for free. It’s also in line with much of the security industry’s evolution to a platform approach.