Cloud Monitoring: Metrics, Alerting, and SLOs That Actually Catch Problems

A monitoring setup that generates fifty alerts a week, most of which nobody investigates because they’re either false positives or genuinely low-priority, is functionally worse than no monitoring at all — it trains the team to ignore alerts, which means the one alert that actually matters gets ignored right along with the noise. Cloud Monitoring (the service most people still call by its former name, Stackdriver) gives GCP teams the raw capability to build genuinely useful monitoring — metrics collection, dashboards, alerting policies, uptime checks, SLO tracking — but capability isn’t the same as a monitoring setup that actually works, and the gap between the two is almost entirely about alert design discipline, not tooling limitations.

This is worth stating upfront because most Cloud Monitoring content covers the mechanics (how to create a dashboard, how to write an alerting policy) without addressing the actual skill that determines whether the resulting system is useful: deciding what deserves an alert in the first place.

Metrics: What’s Actually Being Measured

GCP services automatically export metrics to Cloud Monitoring without any setup — CPU utilization, request latency, error rates are collected by default for most managed services. Custom application metrics require explicit instrumentation:

from google.cloud import monitoring_v3
import time

client = monitoring_v3.MetricServiceClient()
project_name = f"projects/my-project"

series = monitoring_v3.TimeSeries()
series.metric.type = "custom.googleapis.com/orders/processing_time"
series.resource.type = "global"

point = monitoring_v3.Point()
point.value.double_value = 245.3  # milliseconds
now = time.time()
point.interval.end_time.seconds = int(now)
series.points = [point]

client.create_time_series(name=project_name, time_series=[series])

Custom metrics are what let monitoring reflect business-relevant signals, not just infrastructure health — order processing time, queue depth, cache hit ratio, checkout completion rate. Infrastructure metrics alone can look perfectly healthy (CPU fine, memory fine, no errors) while the actual business-critical thing users care about (orders completing successfully within an acceptable time) is degraded, which is why mature monitoring setups always include custom metrics tied to what actually matters to the business, not just what’s collected automatically for free.

Alerting Policies: The Part That Requires Real Judgment

displayName: "High Error Rate - Checkout Service"
conditions:
  - displayName: "Error rate above 5%"
    conditionThreshold:
      filter: >
        resource.type="cloud_run_revision"
        AND resource.labels.service_name="checkout-service"
        AND metric.type="run.googleapis.com/request_count"
        AND metric.labels.response_code_class="5xx"
      comparison: COMPARISON_GT
      thresholdValue: 0.05
      duration: 300s
      aggregations:
        - alignmentPeriod: 60s
          perSeriesAligner: ALIGN_RATE
notificationChannels:
  - projects/my-project/notificationChannels/12345

The duration: 300s setting is doing real work here — it requires the condition to hold for a sustained 5-minute window before firing, not a single momentary spike. This single setting is the difference between an alert that fires on every brief, self-resolving blip (training the team to ignore it) and one that only fires when something is genuinely, sustainedly wrong. Getting duration and threshold values right for each specific alert — not applying one generic template everywhere — is the actual skill, and it’s learned by tuning against real traffic patterns, not by guessing at reasonable-sounding defaults upfront.

The aggregations block matters just as much as duration, and it’s easier to get subtly wrong — ALIGN_RATE computes a rate over each alignment period, which behaves very differently from a raw count aggregator when traffic volume itself fluctuates throughout the day. An alert tuned against low-traffic evening data and left unchanged can either fire spuriously during a normal daytime traffic peak, or fail to fire during a genuine daytime incident because the threshold was calibrated for a much quieter baseline — another reason tuning against real, representative traffic data matters more than picking numbers that merely sound reasonable in isolation.

SLO-Based Alerting: A Fundamentally Different Approach

Rather than alerting on a raw metric threshold, SLO-based alerting asks a more directly useful question: “are we burning through our error budget faster than we can sustain?”

Service Level Objective: 99.9% of requests succeed over 30 days
Error budget: 0.1% of requests can fail = ~43 minutes of full downtime
             equivalent over 30 days

Burn rate alerting:
  Fast burn:  consuming 2%+ of the 30-day budget in 1 hour
              → page immediately, this is a real incident
  Slow burn:  consuming 5%+ of the budget over 6 hours
              → ticket, investigate during business hours,
                not urgent enough to page someone at 2 AM

gcloud alpha monitoring slo create \
  --service=checkout-service \
  --slo-id=availability-slo \
  --goal=0.999 \
  --rolling-period=30d \
  --request-based-good-total-ratio-good-service-filter='metric.labels.response_code_class!="5xx"'

This multi-window, multi-burn-rate pattern (fast burn pages immediately, slow burn creates a lower-urgency ticket) is genuinely the most mature approach to alerting design available, and it directly solves the alert-fatigue problem: instead of every metric threshold crossing generating an alert regardless of actual user impact, alerts fire proportional to how much of your actual reliability commitment is at risk, which correlates far better with “does a human need to act on this right now” than a raw threshold does.

Uptime Checks: External, Black-Box Verification

Everything covered so far assumes your services are reporting metrics correctly — but the scariest failure mode is one where the service is down enough that it can’t even report its own health. Uptime checks solve this by probing from outside, independent of the service’s own instrumentation:

gcloud monitoring uptime create checkout-uptime-check \
  --resource-type=uptime-url \
  --hostname=checkout.example.com \
  --path=/healthz \
  --port=443 \
  --check-interval=60s \
  --timeout=10s

The multi-location probing is what makes uptime checks trustworthy rather than noisy — a single probe location failing might just mean a transient network issue between that specific probe and your service, but multiple geographically distributed locations failing simultaneously is a much stronger signal of a genuine outage. This external, black-box verification is a meaningfully different and complementary signal to internal metrics — a service can be internally reporting healthy metrics while genuinely unreachable from the outside due to a load balancer or DNS misconfiguration, which is exactly the gap uptime checks are designed to catch.

Dashboards: Built for Different Audiences

┌────────────────────────┬────────────────────────────────────────┐
│ Dashboard type            │ Audience and purpose                        │
├────────────────────────┼────────────────────────────────────────┤
│ Service-level overview     │ On-call engineer during an incident —      │
│                             │ error rate, latency, saturation at a       │
│                             │ glance, no digging required                 │
│ Business metrics            │ Product/leadership — orders per hour,       │
│                             │ conversion rate, signups, revenue-adjacent  │
│                             │ signals                                      │
│ Deep infrastructure          │ Platform/SRE team — resource utilization,   │
│                             │ capacity trends, cost-relevant metrics       │
└────────────────────────┴────────────────────────────────────────┘

A common mistake is building one dashboard trying to serve every audience — it ends up too cluttered for an on-call engineer trying to quickly assess an incident and too infrastructure-focused to be meaningful for a product stakeholder. Purpose-built dashboards per audience, even with some metric overlap between them, serve each use case far better than one comprehensive dashboard trying to be everything to everyone.

Real-World Use Case: Reducing Alert Fatigue on an On-Call Rotation

A team with an on-call rotation drowning in low-value pages — CPU spikes that self-resolve, brief latency blips, transient errors that retry successfully — is a common and genuinely fixable problem. The fix, in practice: audit every existing alert for whether it fired on something that actually required human intervention in the last quarter; convert threshold-based alerts on noisy metrics to SLO burn-rate alerts tied to genuine user-facing impact; and set meaningful duration windows on the remaining threshold alerts to filter out momentary blips. Teams that go through this exercise seriously typically cut their alert volume substantially while, counterintuitively, catching genuine incidents faster — because the team actually trusts and responds immediately to the alerts that remain, instead of having learned to triage everything with suspicion first.

Best Practices

• Alert on symptoms that affect users (error rate, latency, SLO burn), not every possible cause (a single VM’s CPU spike that auto-scaling already handles) — cause-level alerts on self-healing conditions are a primary source of alert fatigue.
• Set alert duration and thresholds based on actual historical traffic patterns, not guessed defaults — pull real data before deciding what “abnormal” looks like for a specific service.
• Use SLO-based, multi-burn-rate alerting for anything with a genuine reliability commitment, reserving simple threshold alerts for lower-stakes, more mechanical conditions.
• Review and prune alerting policies on a recurring schedule. An alert that made sense a year ago, for a system that’s since changed significantly, can easily be stale and noisy without anyone noticing until it’s causing real fatigue.
• Combine internal metrics with external uptime checks for anything genuinely production-critical — each catches failure modes the other structurally cannot.

Common Mistakes

Alerting on every metric threshold crossing without a sustained-duration requirement. This is the single most common cause of alert fatigue — a momentary, self-resolving spike shouldn’t page anyone, and requiring the condition to hold for a meaningful window filters this out cleanly.

Building one dashboard for every audience instead of purpose-built views. This produces a dashboard nobody finds genuinely useful, cluttered for incident response and too technical for business stakeholders simultaneously.

Treating custom metric instrumentation as optional. Relying purely on default infrastructure metrics misses the business-relevant signals (checkout success rate, queue processing time) that often degrade well before infrastructure metrics show any problem at all.

Frequently Asked Questions

What happened to the Stackdriver name? Google rebranded Stackdriver into Cloud Monitoring (and Cloud Logging as a separate but related service) — the underlying capability is continuous with what was previously called Stackdriver, just reorganized and renamed as part of GCP’s broader operations suite.

Does Cloud Monitoring work for resources outside GCP, like on-prem servers or other clouds? Yes, via the Ops Agent installed on external hosts, or through custom metric ingestion from any source capable of calling the Monitoring API — useful for teams wanting unified monitoring across a hybrid or multi-cloud environment rather than a separate tool per platform.

How is SLO-based alerting different from just setting a lower threshold on error rate? SLO burn-rate alerting factors in both the rate of errors and the time window, tying the alert directly to “how much of our reliability budget is actually at risk,” which correlates with real user impact far better than an arbitrary raw threshold that doesn’t account for how much budget is already spent or how quickly it’s being consumed.

Can Cloud Monitoring alerts trigger automated remediation, not just notify a human? Yes, alerting policies can trigger webhooks or Pub/Sub notifications that feed into automated remediation workflows (a Cloud Function restarting a service, for instance), though this requires deliberate additional integration beyond the basic alert configuration.

Why do uptime checks matter if I already have detailed internal metrics? Internal metrics depend on the service being healthy enough to report them — a DNS misconfiguration, a load balancer issue, or a complete service outage can all leave internal metrics looking fine (or simply absent) while the service is genuinely unreachable from outside. Uptime checks are an independent, external verification that specifically catches this gap.

Summary

The mechanics of Cloud Monitoring — creating metrics, building dashboards, writing alerting policy YAML — are the easy, well-documented part. The actual differentiator between a monitoring setup that catches real problems and one that trains a team to ignore its own alerts is alert design discipline: sustained-duration thresholds instead of momentary spikes, SLO burn-rate alerting tied to genuine user impact instead of arbitrary raw thresholds, and purpose-built dashboards instead of one dashboard trying to serve every audience. Getting this right is less about learning new Cloud Monitoring features and more about being genuinely disciplined about what deserves to interrupt a human being, and when.