Google Cloud Datastore: The Predecessor to Firestore and When It Still Makes Sense

Cloud Datastore was Google’s managed NoSQL document store for App Engine applications before Firestore existed. In 2017, Google merged Datastore’s capabilities into Firestore and Firestore became the recommended choice for new projects. Datastore mode in Firestore is how existing Datastore applications continue to run.

Understanding Datastore matters if you inherit a project built on it, are migrating an App Engine application, or need to decide whether to stay on Datastore or upgrade to Firestore Native mode.

The Data Model: Kinds, Entities, and Keys

Datastore uses different terminology from Firestore, but the concepts are similar:

Firestore term    ←→    Datastore term
Collection              Kind
Document                Entity
Document ID             Key name / Key ID
Nested collection       Entity group (ancestor path)

An entity in Datastore:

Kind: Product

Entity key: /Product, 1234567
Entity properties:
  name: "Wireless Keyboard"          (String, indexed)
  price: 49.99                       (Float, indexed)
  in_stock: True                     (Boolean, indexed)
  description: "..."                 (String, unindexed - too long)
  tags: ["electronics", "input"]     (Array of Strings)
  created_at: 2025-01-10T09:00:00Z   (DateTime)

Keys are either auto-assigned numeric IDs or user-specified string names. Keys can include an ancestor path, which defines the entity’s position in a hierarchical entity group.

Entity Groups and Ancestor Queries

Datastore’s ancestor relationship groups related entities in a logical hierarchy. This is important for two reasons: ancestor queries return results with strong consistency, and Datastore transactions are limited to entities within the same entity group.

Entity group structure:
  /Order,order_001
    └── /Order,order_001/OrderItem,1
    └── /Order,order_001/OrderItem,2
    └── /Order,order_001/OrderItem,3

Ancestor query: "give me all OrderItems where ancestor = /Order,order_001"
  → strongly consistent result
  → all items returned are current as of query time

Non-ancestor query: "give me all OrderItems where total > 50.00"
  → eventually consistent (may miss very recent writes)

This distinction drove a lot of Datastore application design — developers structured entity groups to ensure strong consistency for related data within a transaction scope.

Querying with GQL and the Python Client

Datastore has a query language called GQL (Google Query Language) that resembles SQL but has important limitations:

GQL examples:

SELECT * FROM Product WHERE in_stock = TRUE AND price < 100.0
  -- Works: equality and inequality on indexed properties

SELECT * FROM Product WHERE tags = 'electronics'
  -- Works: multi-valued property filter

SELECT * FROM Product WHERE price < 100.0 ORDER BY name
  -- Fails without a composite index on (price, name)

SELECT * FROM Product WHERE description CONTAINS 'keyboard'
  -- Fails: Datastore does not support text search

Using the Python client library:

from google.cloud import datastore

client = datastore.Client(project="my-project")

# Create an entity
product_key = client.key("Product", "keyboard-001")
product = datastore.Entity(key=product_key)
product.update({
    "name": "Wireless Keyboard",
    "price": 49.99,
    "in_stock": True,
    "tags": ["electronics", "input"],
})
client.put(product)

# Read an entity by key
result = client.get(product_key)
if result:
    print(f"{result['name']}: ${result['price']}")

# Query with filters
query = client.query(kind="Product")
query.add_filter("in_stock", "=", True)
query.add_filter("price", "<", 100.0)
query.order = ["price"]

for entity in query.fetch(limit=10):
    print(entity["name"], entity["price"])

Indexes in Datastore

Datastore maintains two types of indexes:

Built-in indexes: Automatically created for every property of every entity, enabling single-property filters and orders.

Composite indexes: Required for queries that filter on multiple properties or combine a filter with an order on a different property. Defined in index.yaml:

indexes:
  - kind: Product
    properties:
      - name: in_stock
        direction: asc
      - name: price
        direction: asc
      - name: name
        direction: asc

Without the composite index, queries that require it return an error rather than a result. This is a common source of frustration for developers new to Datastore.

Transactions

Datastore transactions are serializable but limited to entities in the same entity group (by default). Cross-group transactions support up to 25 entity groups per transaction but are slower.

with client.transaction():
    order_key = client.key("Order", "order_001")
    order = client.get(order_key)

    if order["status"] != "pending":
        raise ValueError("Order already processed")

    order["status"] = "confirmed"
    order["confirmed_at"] = datetime.utcnow()
    client.put(order)

    # This will succeed or the whole transaction rolls back
    confirmation_key = client.key("OrderConfirmation", "order_001")
    confirmation = datastore.Entity(key=confirmation_key)
    confirmation["order_id"] = "order_001"
    confirmation["sent_at"] = datetime.utcnow()
    client.put(confirmation)

Datastore Mode vs Firestore Native Mode

When you create a new Firestore database, you choose Native mode or Datastore mode. The underlying storage is the same (Firestore), but the API and capabilities differ:

┌─────────────────────────────────────────────────────────────────────────────┐
│ Feature                    │ Datastore Mode         │ Native Mode           │
├────────────────────────────┼────────────────────────┼───────────────────────┤
│ API                        │ Datastore API          │ Firestore API         │
│ Real-time listeners        │ No                     │ Yes                   │
│ Offline client sync        │ No                     │ Yes (mobile/web SDK)  │
│ Security rules             │ No (IAM only)          │ Yes                   │
│ Consistency model          │ Eventual (non-ancestor)│ Strong consistency    │
│ Mobile/web SDK             │ No                     │ Yes (Firebase SDK)    │
│ Maximum entity size        │ 1 MB                   │ 1 MB (document)       │
│ Collection group queries   │ Equivalent (kind query)│ Yes                   │
└────────────────────────────┴────────────────────────┴───────────────────────┘

Migrating from Datastore to Firestore Native Mode

Migration cannot happen in-place — you cannot upgrade a Datastore Mode database to Native mode. The migration path is:

Step 1: Export Datastore data
  gcloud datastore export gs://my-bucket/export

Step 2: Create a new Firestore database in Native mode (different project or region)

Step 3: Import data into Firestore (Datastore export format is importable)
  gcloud firestore import gs://my-bucket/export

Step 4: Update application code
  - Replace Datastore API calls with Firestore API calls
  - Terminology changes: kinds → collections, entities → documents
  - Query changes: ancestor queries replaced by collection group queries
  - Remove entity group constraints
  - Add Security Rules if replacing IAM-based access

Step 5: Test thoroughly, then cut over traffic

When to Stay on Datastore Mode

Despite Firestore Native mode being the recommended choice for new projects, some situations favor keeping Datastore mode:

Existing App Engine Standard applications that use the built-in Datastore API — these work without any changes in Datastore mode
Server-side-only applications where real-time listeners and mobile SDKs provide no benefit
Applications with heavy ancestor query usage where Datastore’s strongly-consistent ancestor queries are already the central design pattern
Migration cost vs benefit — if the application works well and real-time features are not needed, the migration effort may not be justified

The practical answer: if you’re building something new, use Firestore Native. If you’re maintaining something old on Datastore that works, evaluate the migration cost before committing.

Summary

Cloud Datastore is the server-side NoSQL document store that App Engine applications have relied on since 2008. Its key concepts — kinds, entities, keys, ancestor paths, and composite indexes — remain relevant for the millions of applications built on it. Firestore in Datastore mode maintains backward compatibility while Firestore Native mode adds real-time listeners, offline sync, client-side security rules, and strong consistency for all queries. New development should use Firestore Native mode. Existing Datastore applications can continue running in Datastore mode indefinitely, migrating only when the new capabilities of Native mode justify the effort.