“Ingesting millions of HTTP 200 OK load balancer health probes into a high-cost SIEM workspace isn’t observability—it’s burning cash. You don’t need to store every ping at $2.30 per gigabyte to stay secure.”

In early 2025, in my current role as Associate Director, Cloud Architecture & AI, I was brought in to address a cloud financial emergency.

An enterprise retail client received an unexpected invoice from Microsoft Azure. Their monthly Log Analytics and Microsoft Sentinel ingestion bill had skyrocketed from $15,000 to over $68,000 in less than 60 days.

The client’s Finance Director issued an ultimatum: Cut the logging bill by 40% within two weeks, or all new cloud feature deployments will be frozen.

---

The Default-Diagnostic Trap

When teams deploy resources in Azure, the easiest configuration path is clicking “Select All Diagnostics -> Send to Log Analytics Workspace.”

That default setting works fine for small lab environments. In a multi-subscription enterprise production environment, it is a financial time bomb.

The client’s development teams had enabled full diagnostic logging across 20 Azure Application Gateways, 15 Azure Kubernetes Service (AKS) clusters, and hundreds of storage accounts. Everything was streaming raw, unparsed telemetry straight into a primary Sentinel SIEM workspace.

---

The Mess: Paying $2.30/GB to Store “HTTP 200 OK”

To find out what was eating the budget, we ran a Kusto Query Language (KQL) audit across the workspace.

// KQL Query to identify top log ingestion tables by volume
Usage
| where TimeGenerated > ago(30d)
| summarize TotalGB = sum(Quantity) / 1000 by DataType
| sort by TotalGB desc

The output was startling:

Over 72% of total ingested log volume came from a single table: AGWAccessLog (Azure Application Gateway Access Logs).

When we analyzed the individual log records, over 80% of those entries were repetitive HTTP 200 OK synthetic health check probes fired every 5 seconds by the load balancers to backend container pods.

// Millions of these records were ingested daily into Sentinel at $2.30/GB
{
  "resourceId": "/SUBSCRIPTIONS/.../APPLICATIONGATEWAYS/AGW-PROD",
  "operationName": "ApplicationGatewayAccess",
  "properties": {
    "httpStatus": 200,
    "requestUri": "/healthz",
    "userAgent": "AppGateway-HealthReachable"
  }
}

We were paying over $38,000 a month to store millions of useless synthetic 200 OK health check logs inside an expensive security analytics workspace!

An overzealous junior engineer attempted to fix it by disabling diagnostic logging on the Application Gateways entirely. Three days later, a minor security alert triggered on a public endpoint, and the incident response team had zero HTTP traffic logs to perform forensic analysis.

We couldn’t just turn off logging. We had to log smarter.

---

The Solution: Telemetry Tiering & DCR Edge Filtering

We instituted a 3-Tier Telemetry Architecture using Azure Data Collection Rules (DCRs) and Azure Monitor Transformation Engine.

# 3-Tier Observability Architecture

| Telemetry Tier                 | Data Types                                    | Destination                    | Cost per GB |
| ------------------------------ | --------------------------------------------- | ------------------------------ | ----------- |
| **Tier 1: High-Priority SIEM** | Security Audits, WAF Blocks, `4xx/5xx` Errors | Log Analytics / Sentinel       | $2.30 / GB  |
| **Tier 2: Basic Logs**         | Performance Counters, Operational Traces      | Basic Log Workspace Tier       | $0.50 / GB  |
| **Tier 3: Raw Archive**        | `200 OK` Access Logs, Raw Packet Traces       | Azure Blob Storage (Cold Tier) | $0.02 / GB  |

1. DCR Transformation Filtering at Ingestion

We configured Azure Data Collection Rules (DCR) to inspect incoming Application Gateway logs before they entered the workspace. Using KQL transformation statements inside the DCR, we dropped synthetic /healthz probe requests dynamically.

// # Data Collection Rule (DCR) KQL Transformation Step
{
  "transformKql": "source | where requestUri !contains '/healthz' and userAgent !contains 'AppGateway-HealthReachable'"
}

2. Log Ingestion Tiering to Basic and Cold Storage

We re-routed high-volume operational logs (such as container stdout logs and successful network traces) away from the Analytics Tier into Azure Log Analytics Basic Logs ($0.50/GB) and archived raw logs to Azure Blob Storage Cold Tier ($0.02/GB) for long-term compliance retention.

# Azure CLI command to set table plan to Basic for high-volume operational tables
az monitor log-analytics workspace table update --resource-group RG-OPS --workspace-name WS-LOGS --name ContainerLogV2 --plan Basic

---

The Impact

• Cost Reduction: Reduced monthly cloud logging spend from $68,000 to $26,000—a 61.7% cost reduction exceeding the client’s 40% goal.
• Zero Security Loss: Retained 100% of security-relevant audit logs (4xx/5xx errors, WAF blocks, authentication failures) inside Microsoft Sentinel.
• Query Performance Improvement: Reduced KQL query execution times across Sentinel dashboards by 70% due to the removal of noise data.

Key Takeaway

Filter Noisy Telemetry at Ingestion.

Do not stream unparsed, raw diagnostic logs into high-cost SIEM workspaces. Use Data Collection Rules (DCR) to drop synthetic health checks at the edge, tier operational logs to lower-cost workspace plans, and reserve high-cost security analytics for actionable audit events.

---

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin