Secret Manager: Storing API Keys and Credentials the Right Way on GCP

Every codebase, no matter how disciplined the team, has a moment where a database password or third-party API key ends up somewhere it shouldn’t — a .env file accidentally committed to git, a hardcoded string in a config that gets copied between environments and never cleaned up, a Slack message with “here’s the prod key, don’t share this.” None of these are malicious; they’re the natural result of not having a purpose-built place to put secrets that’s easier to use correctly than to use wrong. Secret Manager is Google’s answer: a centralized, versioned, access-controlled store for exactly this class of sensitive string — API keys, database credentials, TLS certificates, service account keys, tokens.

The pitch isn’t complicated, but the value compounds over time in ways that aren’t obvious on day one: every secret access is logged, every secret has a version history, and access is controlled through the same IAM system you already use for everything else in GCP — no separate secrets-management access model to learn or maintain.

Core Concepts

Secret (a named container, e.g. "prod-db-password")
   │
   ▼
Secret Versions (each update creates a new immutable version)
   │
   ├── Version 1 (DISABLED)
   ├── Version 2 (DISABLED)
   └── Version 3 (ENABLED — the current active value)

A secret is the logical name your application references — prod-db-password, stripe-api-key. Updating a secret’s value doesn’t overwrite anything; it creates a new immutable version. This is deliberate: rotating a credential doesn’t destroy the ability to reference the previous value, which matters enormously during an incident where you need to quickly understand what changed or roll back to a previous version while investigating.

Creating and Accessing a Secret

# Create a secret and its first version in one step
echo -n "super-secret-db-password" | \
  gcloud secrets create prod-db-password \
  --data-file=- \
  --replication-policy=automatic

# Add a new version (rotating the value)
echo -n "new-rotated-password" | \
  gcloud secrets versions add prod-db-password \
  --data-file=-

# Access the latest version
gcloud secrets versions access latest --secret=prod-db-password

From application code, the pattern is nearly identical across languages — fetch the secret at startup or on demand, never bake it into a config file or Docker image:

from google.cloud import secretmanager

client = secretmanager.SecretManagerServiceClient()
name = "projects/my-project/secrets/prod-db-password/versions/latest"
response = client.access_secret_version(request={"name": name})
db_password = response.payload.data.decode("UTF-8")

Referencing latest in application code is convenient but worth a second thought — it means a secret rotation takes effect the moment the value changes, without a deploy, which is exactly the behavior you want for some secrets (an emergency credential rotation should propagate immediately) and exactly the behavior you don’t want for others (a change you want to test in staging before it silently affects production).

Pinning Versions vs Always Using Latest

┌─────────────────────┬────────────────────────────────────────────┐
│ Reference pattern     │ Behavior                                      │
├─────────────────────┼────────────────────────────────────────────┤
│ `latest`              │ Always returns the newest enabled version —   │
│                       │ rotation is instant and implicit               │
│ Specific version #    │ Always returns that exact version — rotation   │
│                       │ requires an explicit config/deploy change      │
└─────────────────────┴────────────────────────────────────────────┘

For most application secrets, latest is the right default — it’s what makes rotation actually low-friction rather than a deploy-coordination exercise. But for anything where an unexpected value change mid-request could cause hard-to-debug failures (a signing key used to validate incoming webhooks, for instance, where the sender and receiver must agree on exactly which key version is active), pinning to a specific version and rotating deliberately through a controlled deploy is the safer pattern.

Secrets as Environment Variables — Without the Risk

A common integration pattern injects secrets directly into Cloud Run or GKE as environment variables, sourced from Secret Manager rather than a plaintext value in the deployment config:

gcloud run deploy my-service \
  --image=gcr.io/my-project/my-service \
  --set-secrets="DB_PASSWORD=prod-db-password:latest,API_KEY=stripe-key:latest"

This gets you the ergonomics of a plain environment variable inside your running container — no code changes needed to read os.environ["DB_PASSWORD"] — while the actual secret value never touches your deployment YAML, your CI/CD logs, or your source control. The Cloud Run runtime resolves the reference and injects the real value only at container startup.

IAM: Who Can Access What

Secret Manager access control follows the same resource-scoped IAM model as the rest of GCP, and getting this right is most of what “using Secret Manager securely” actually means in practice.

# Grant a specific service account access to a specific secret only
gcloud secrets add-iam-policy-binding prod-db-password \
  --member="serviceAccount:api-service@my-project.iam.gserviceaccount.com" \
  --role="roles/secretmanager.secretAccessor"

roles/secretmanager.secretAccessor grants read access to secret values but not the ability to see or modify the secret’s metadata or add new versions — a genuinely useful separation between “this workload needs to read this credential” and “this person manages the credential’s lifecycle.” Granting project-wide roles/secretmanager.admin to every service account that needs any secret is the single most common way teams undermine the isolation Secret Manager is designed to provide — a compromised service account with broad secret access can read every credential in the project, not just the one it actually needs.

What Rotation Actually Looks Like End to End

It’s worth walking through the full rotation sequence once, because the value of versioning is easy to state abstractly and much clearer when you see the actual sequence of events during a live rotation.

Notice that disabling and destroying are two separate, deliberately sequenced steps. Disabling a version blocks access but keeps it recoverable — if the new version turns out to be wrong (a typo in a rotated password, for instance), you can re-enable the previous version immediately without regenerating anything. Destruction is the point of no return, and treating it as a separate, later step — only after you’ve confirmed the new version works in production — is what makes rotation a safe, reversible operation rather than a one-way gamble.

Real-World Use Case: Rotating a Compromised API Key Under Pressure

This is the scenario that makes the investment in Secret Manager obviously worth it in hindsight: a third-party API key is accidentally exposed (in a log line, a support ticket, a public repo). Without centralized secret management, remediation means finding every place that key is hardcoded or copied across config files, environments, and possibly other engineers’ local setups — a slow, error-prone scramble under real time pressure. With Secret Manager, remediation is: rotate the key with the third-party provider, add the new value as a new secret version, and every service referencing latest picks up the new value automatically on next access, no code deploy required. The difference between these two remediation timelines, during an actual incident, is the entire argument for adopting this pattern before you need it.

Best Practices

• One secret per credential, not one secret holding a JSON blob of everything. Granular secrets let you scope IAM access precisely — a service that only needs the database password shouldn’t also be able to read the Stripe key just because they were bundled together.
• Enable automatic replication unless you have a specific regional data-residency requirement, which needs user-managed replication instead — automatic replication handles multi-region durability without you managing it.
• Set up secret rotation reminders or automation, especially for credentials tied to third-party systems where Secret Manager itself can’t force the upstream provider to rotate — a stale, never-rotated secret undermines a lot of the security value of the system around it.
• Audit access logs periodically, not just when investigating an incident — Cloud Audit Logs captures every secret access, and reviewing who’s actually accessing production secrets is a genuinely useful, underused check.
• Never log secret values, even at debug level. It sounds obvious, but a debug log line that accidentally prints a fetched secret is one of the most common ways a properly-stored credential still ends up leaked — through the logging pipeline rather than through Secret Manager itself.

Common Mistakes

Treating Secret Manager as a place to “set and forget” rather than actively rotate. Storing a credential securely doesn’t make it immune to leaking eventually — regular rotation limits the blast radius when it does.

Granting broad secretmanager.admin access instead of scoped secretAccessor per secret. This defeats much of the isolation value Secret Manager provides over a shared config file.

Not disabling old versions after rotation. Old, disabled-but-not-destroyed versions remain recoverable (useful for rollback), but leaving every historical version enabled indefinitely means a compromised credential from months ago might still technically be usable if an old version is ever mistakenly referenced.

Frequently Asked Questions

Is Secret Manager the same as Cloud KMS? No — KMS manages cryptographic keys; Secret Manager stores actual secret values (which, under the hood, are encrypted using KMS-managed keys). Use Secret Manager for credentials and API keys, KMS for encrypting your own data directly.

Does Secret Manager have a size limit? Yes, secret payloads are capped (64 KiB per version), which is more than sufficient for credentials and keys but not intended for storing larger files or certificates bundles beyond that size — those typically belong in Cloud Storage with restricted access instead.

Can I get notified when a secret is accessed or changed? Yes, via Cloud Audit Logs and Pub/Sub notifications configured on secret events, which is commonly wired into alerting for high-sensitivity secrets.

What happens if I destroy a secret version that’s still referenced by a running service? Access attempts against a destroyed version fail immediately and irreversibly — destruction, unlike disabling, cannot be undone. Disable a version first and confirm nothing depends on it before destroying.

Can Secret Manager automatically rotate credentials for me? Secret Manager itself only manages versions and can notify you on a schedule that rotation is due — it doesn’t generate new credential values for arbitrary third-party systems on its own. For GCP-native credentials, pairing it with a Cloud Function triggered on a schedule that generates a new value and calls the versions.add API is a common pattern for genuine automated rotation.

Should secrets shared across multiple environments (dev, staging, prod) live in one project or be separated? Separate projects per environment, each with its own Secret Manager secrets, is the safer default — it means a developer with access to staging secrets never has an implicit path to production credentials, which a shared project with only IAM-level separation can’t guarantee as cleanly.

Summary

Secret Manager’s value isn’t a new capability so much as removing every excuse for the bad habits that lead to leaked credentials — hardcoded values, shared plaintext files, no audit trail. The pattern that actually delivers on that promise is disciplined: one secret per credential, IAM access scoped to exactly the workload that needs it, and rotation treated as a routine operational task rather than something that only happens reactively after an incident. Once that discipline is in place, the payoff shows up exactly when you need it most — during a real credential rotation under time pressure, where the difference between minutes and hours of remediation work is whether you built this correctly beforehand, long before the incident that actually tests it.