Cloud Logging: Log Routing, Retention, and Cost Control on GCP

Logging is the GCP service most likely to become an unpleasant line-item surprise on a bill, and the reason is structural rather than a pricing gotcha: it’s genuinely easy to log everything by default and only notice the cost once volume has grown for months. Cloud Logging automatically captures logs from nearly every GCP service without any setup, which is convenient in exactly the way that makes cost discipline easy to defer — until a debug-level log line, left enabled in production since a launch six months ago, turns out to be the single largest contributor to a monthly logging bill nobody was watching closely.

Understanding Cloud Logging well means understanding not just how to search and view logs, but how to route, filter, and retain them deliberately — because the default behavior (keep everything, index everything, retain it all) is rarely the behavior you actually want once volume gets real.

How Logs Actually Flow

Application / GCP service emits log entry
         │
         ▼
   _Default log bucket (30-day retention, out of the box)
         │
         ├──► Log Router evaluates sinks
         │
    ┌────┼────────────┬─────────────────┐
    ▼    ▼             ▼                 ▼
 _Required  Custom     BigQuery         Cloud Storage
 bucket     bucket     dataset          bucket
(audit logs, (custom    (for SQL         (for cheap,
 immutable)  retention)  analysis)        long-term archive)

Every project has two default buckets you don’t create yourself: _Required (audit logs, retained by policy, can’t be deleted) and _Default (everything else, 30-day retention by default). The Log Router is what determines whether a given log entry stays in the default bucket, gets excluded entirely, or gets routed to an additional destination — and this routing decision, made deliberately per log type, is the actual lever for both cost control and making logs useful for their intended purpose (compliance retention, SQL analysis, cheap long-term archive).

Log Sinks: Routing Logs Where They’re Actually Useful

# Route audit logs to BigQuery for long-term SQL-queryable analysis
gcloud logging sinks create audit-logs-to-bq \
  bigquery.googleapis.com/projects/my-project/datasets/audit_logs \
  --log-filter='logName:"cloudaudit.googleapis.com"'

# Route everything to a cheap Cloud Storage bucket for compliance archive
gcloud logging sinks create archive-to-gcs \
  storage.googleapis.com/my-log-archive-bucket \
  --log-filter='severity>=DEFAULT'

Sending audit logs to BigQuery specifically is a genuinely common and valuable pattern — Cloud Logging’s own search interface is fine for recent, ad-hoc investigation, but “which users accessed this dataset over the last six months” is a query you actually want to run in SQL against structured data, not piece together through the logging console’s search interface. Cloud Storage archival, meanwhile, is the answer for “we need to retain this for regulatory reasons but almost never actually need to search it” — storage costs a fraction of keeping everything in an actively-indexed logging bucket indefinitely.

Exclusion Filters: The Single Highest-Leverage Cost Control

# Stop ingesting noisy health-check logs entirely
gcloud logging sinks update _Default \
  --add-exclusion=name=exclude-health-checks,filter='httpRequest.requestUrl="/healthz"'

# Exclude debug-level logs from a specific noisy service in production
gcloud logging sinks update _Default \
  --add-exclusion=name=exclude-debug-noisy-service,filter='resource.labels.service_name="noisy-service" AND severity="DEBUG"'

This is genuinely the highest-leverage cost control available, and it’s frequently skipped because it requires someone to actually look at what’s being logged and decide it’s not worth paying to ingest — a health check endpoint being hit every few seconds by a load balancer, generating a log line every single time, is a classic example of enormous volume with essentially zero investigative value. Excluding it doesn’t lose anything useful; it just stops paying to store and index something nobody will ever search for.

A practical way to find these candidates without guessing is sorting log volume by resource and log name in the Logs Explorer’s own summary view — the highest-volume entries are almost always a small handful of sources, and reviewing just the top five or ten by volume typically surfaces most of the meaningful savings available, rather than needing to audit every log source in the project individually.

Log-Based Metrics: Turning Logs Into Alertable Signals

Sometimes the thing you want to alert on doesn’t exist as a native metric — it’s a pattern in log content. Log-based metrics bridge that gap:

gcloud logging metrics create payment-failures \
  --description="Count of payment processing failures" \
  --log-filter='jsonPayload.event="payment_failed"'

This turns any log line matching the filter into a genuine Cloud Monitoring metric — countable, graphable, and usable in an alerting policy exactly like any built-in metric. This is the practical bridge between “we log this event but have no way to know if it’s spiking” and “we get paged when payment failures exceed a threshold,” without needing to build separate custom-metric instrumentation for something that’s already being logged anyway.

Structured Logging: Making Logs Actually Queryable

import logging
import json

# Unstructured — hard to filter/query precisely
logging.info(f"Order {order_id} failed: {error_message}")

# Structured — every field is independently filterable
logging.info(json.dumps({
    "message": "Order processing failed",
    "order_id": order_id,
    "error_code": error_code,
    "customer_tier": customer_tier,
    "severity": "ERROR"
}))

Structured (JSON) logging is what makes a query like “show me every failed order for premium-tier customers with error code E402 in the last hour” a precise, fast filter rather than a fragile regex-based text search across unstructured log lines. The upfront cost is minor — logging a JSON object instead of an interpolated string — and the payoff compounds every single time someone needs to actually investigate something in production, which is a trade most teams don’t regret making once they’ve experienced trying to grep through unstructured log text during an actual incident.

Tracing a Log Entry’s Full Journey

It’s useful to see the complete decision path a single log entry travels, since “where did this log go, and why is it costing what it’s costing” is a genuinely common question once a team starts taking log routing seriously.

The branch worth internalizing is that exclusion happens before ingestion — an excluded log never costs anything and is never searchable, full stop — while routing to a sink happens after ingestion into a bucket and is a copy operation, not a move. This means a log entry can simultaneously exist in the default bucket (with its own 30-day retention) and in a BigQuery dataset (with whatever retention that dataset is configured for), each governed independently — understanding this is what makes “why can I query this in BigQuery but not in Logs Explorer anymore” make sense rather than feel like a bug.

Real-World Use Case: Cutting a Logging Bill Without Losing Visibility

A company noticing logging costs growing faster than actual traffic is a common, fixable scenario. The practical audit: identify the highest-volume log sources (usually a small number of noisy services or endpoints account for a disproportionate share of total volume), determine which of those actually get searched or referenced during incidents (often close to zero for health checks and routine polling), and add targeted exclusion filters for the ones with genuinely low investigative value while leaving error-level and business-critical logs fully intact. This surgical approach — cutting specifically identified low-value volume rather than uniformly reducing retention or verbosity everywhere — typically cuts logging costs substantially without meaningfully reducing the team’s actual ability to debug real problems, because the logs that get cut were never the ones anyone was searching anyway.

Best Practices

Add exclusion filters for known-noisy, low-value log sources proactively, not reactively after a surprising bill — health checks, readiness probes, and routine polling are the classic first targets.
Route audit and compliance logs to BigQuery or Cloud Storage explicitly, rather than relying on default bucket retention, which is neither optimized for long-term SQL query nor for cheap indefinite archive.
Adopt structured (JSON) logging as a default coding standard, not an occasional nice-to-have — the query precision it enables pays for itself the first time it meaningfully speeds up an incident investigation.
Review log volume by source periodically, treating it like any other cost center worth actively managing rather than a fixed, unavoidable expense.

Common Mistakes

Leaving debug-level logging enabled in production indefinitely. Debug logging is genuinely useful during active development or incident investigation and genuinely wasteful left on by default for months at a time in steady-state production.

Never reviewing what’s actually driving logging costs. Without periodically checking log volume by source, a single noisy service or misconfigured verbose logger can silently dominate the bill for months before anyone notices.

Treating the default 30-day retention as sufficient for compliance needs without checking actual requirements. Some compliance frameworks require retention well beyond 30 days — assuming the default is fine without verifying against actual regulatory requirements is a gap that surfaces badly during an audit, not before.

Frequently Asked Questions

Does excluding logs from ingestion mean they’re lost entirely? For excluded entries, yes — they’re not ingested into Cloud Logging at all (this is what saves the cost), so exclusion filters should be applied deliberately to genuinely low-value log sources, not broadly, since excluded logs can’t be retroactively recovered.

What’s the difference between a log sink’s exclusion filter and simply not logging something in application code? Exclusion filters operate on logs already being emitted, letting you control ingestion centrally without redeploying application code — useful for a fast, immediate cost fix, whereas reducing verbosity in application code is a more deliberate, code-level change with its own deploy cycle.

Can I search logs older than the retention period if I need them for an investigation? Only if they were routed to a longer-retention destination (BigQuery, Cloud Storage) before the default bucket’s retention window expired — logs that age out of the default bucket without being routed elsewhere are genuinely gone, which is exactly why deliberate sink configuration for anything with a longer retention need matters.

Is structured logging required, or just recommended? It’s not required — GCP accepts and stores unstructured text logs fine — but the query and filtering precision structured logging enables is significant enough that most teams treat it as a de facto standard rather than optional polish.

Does routing a log to a sink remove it from the original bucket? No — sink routing is a copy, not a move. The log entry remains in its original bucket (subject to that bucket’s own retention) while also appearing wherever the sink routed it, and each destination’s retention is managed independently.

Summary

Cloud Logging’s default behavior — capture everything, index everything, retain it for 30 days — is a reasonable starting point and a genuinely expensive steady state if left unexamined as traffic and log volume grow. The actual skill is routing deliberately: exclusion filters for high-volume, low-value noise; BigQuery or Cloud Storage sinks for anything needing longer retention or SQL-queryable analysis; log-based metrics for turning meaningful log patterns into alertable signals; and structured logging as a default habit that makes every future investigation faster. None of this is complicated individually — it’s simply work that’s easy to defer indefinitely until a bill or an incident forces the conversation, and cheaper by far to build in from the start than to retrofit under pressure.