Cloud Data Fusion: Visual ETL Pipelines on GCP Without Writing Spark Code

Most data engineering teams have a backlog of integration work that has nothing to do with sophisticated stream processing — pull data from a legacy SQL Server database, clean up a few columns, land it in BigQuery. Writing and maintaining a custom Spark or Beam job for every one of these feels like overkill, but hand-rolled Python scripts scattered across cron jobs age badly too. Cloud Data Fusion exists for exactly this gap: a visual, drag-and-drop pipeline builder for ETL/ELT work that generates and runs real Spark jobs underneath, without requiring every analyst-adjacent engineer to write Scala.

It’s built on CDAP (Cask Data Application Platform), an open-source project Google acquired and productized, which matters practically because pipelines you build aren’t locked into a proprietary format — the underlying application model is portable.

What Data Fusion Actually Is

Data Fusion is not a lightweight SaaS tool — it’s a full application platform that provisions a dedicated Kubernetes-backed instance per environment. When you create a Data Fusion instance, GCP spins up a GKE cluster running the CDAP control plane, and your pipelines execute as Spark jobs on ephemeral Dataproc clusters that Data Fusion provisions and tears down automatically per pipeline run.

┌─────────────────────────────────────────────────────────┐
│                 Cloud Data Fusion Instance                │
│  (persistent GKE-backed control plane + UI)               │
├─────────────────────────────────────────────────────────┤
│  Pipeline Studio (drag-and-drop canvas)                   │
│  Wrangler (interactive data cleaning)                     │
│  Plugin Hub (170+ connectors and transforms)               │
└──────────────────┬──────────────────────────────────────┘
                    │ pipeline execution triggers
                    ▼
        Ephemeral Dataproc cluster (per run)
                    │
                    ▼
        Source ──► Transform ──► Sink
   (Cloud SQL, SAP, Salesforce,   (BigQuery, GCS,
    Kafka, on-prem JDBC, etc.)     Bigtable, etc.)

This architecture explains two things people are often surprised by: instances are billed hourly even when idle (because the control plane is always running), and pipeline execution has real startup latency (because a Dataproc cluster has to spin up before the job runs) — Data Fusion is not the tool for sub-minute latency requirements.

Core Concepts

Plugins are the building blocks of a pipeline — sources (read from a system), transforms (clean/reshape data), and sinks (write to a destination). Data Fusion ships with 170+ plugins covering databases, SaaS APIs (Salesforce, SAP, ServiceNow), file formats, and message queues, and you can write custom plugins in Java when the built-in library doesn’t cover a system you need.

Wrangler is the interactive data-cleaning tool, and it’s the feature that most differentiates Data Fusion from generic ETL tools. You upload or connect to a sample of your data, and Wrangler shows you the actual values column by column, letting you build transformation logic (parse a date format, split a column, mask a field, remove nulls) by clicking on the data itself rather than writing transformation code blind. Every click generates a “directive” — a recorded transformation step — that becomes part of the pipeline when you’re done.

Pipelines are the visual DAG connecting sources through transforms to sinks. Each node is configured through a UI panel rather than code, and the whole pipeline compiles down to a Spark job that Data Fusion submits to the ephemeral Dataproc cluster.

Building a Pipeline: A Concrete Example

Say you need to pull customer records from an on-prem SQL Server database, standardize phone number formats, drop rows missing an email, and load the result into BigQuery daily.

Source: Add a Database plugin, configure the JDBC connection string, and specify the query or table.
Wrangler transform: Open a sample of the source data in Wrangler. Click on the phone column, apply a “regex replace” directive to standardize formatting, then filter out rows where email is null — each of these becomes a recorded directive.
Sink: Add a BigQuery plugin, point it at the target dataset and table, and set write disposition to “truncate and reload” or “append” depending on your use case.
Schedule: Attach a time-based trigger (daily at 2 AM) or a pipeline-completion trigger if this pipeline should run after an upstream pipeline finishes.

The resulting pipeline, once deployed, runs as an unattended scheduled job — no engineer touches it again unless the source schema changes or a new transformation is needed.

What a Wrangler Directive Actually Looks Like

Every click in the Wrangler UI is silently recorded as a line in a directive script, which you can view and edit directly. This is worth knowing because it means Wrangler transformations aren’t a black box — they’re readable, diffable text:

parse-as-csv :body ',' true
drop body
rename body_1 customer_id
rename body_2 phone_raw
send-to-error !dq:isnull(phone_raw)
set-column phone_clean regex-replace(phone_raw, '[^0-9]', '')
filter-row-if-matched email '^$'

Reading this top to bottom: parse a raw CSV column, drop the original, rename the split columns, route null-phone rows to an error collector instead of failing the whole pipeline, strip non-numeric characters from phone numbers, and filter out rows with empty email addresses. Because this is plain text, it can be reviewed in a pull request the same way you’d review a SQL transformation — which matters for teams trying to keep even their “no-code” pipelines under some form of change control.

Batch vs Real-Time Pipelines

Data Fusion supports two distinct pipeline types with different execution engines underneath.

┌───────────────┬─────────────────────────┬─────────────────────────┐
│ Pipeline type │ Engine                   │ Best for                 │
├───────────────┼─────────────────────────┼─────────────────────────┤
│ Batch         │ Spark on Dataproc        │ Scheduled ETL, backfills │
│ Realtime      │ Spark Streaming          │ CDC, near-real-time sync │
└───────────────┴─────────────────────────┴─────────────────────────┘

Realtime pipelines are less commonly used than batch in practice — for genuinely low-latency streaming, most teams reach for Dataflow directly instead, because Data Fusion’s realtime mode still carries more operational overhead than a purpose-built streaming pipeline. Batch is where Data Fusion earns its keep.

Real-World Use Case: Legacy System Migration

A common scenario: a company migrating off a legacy on-prem data warehouse to BigQuery has dozens of source tables from SAP, Oracle, and flat file exports, each needing slightly different cleaning logic, and a data engineering team too small to hand-write and maintain dozens of custom Spark jobs. Data Fusion lets a smaller team build these pipelines visually, review them in a UI a non-Spark-expert stakeholder can actually read, and hand off maintenance to analysts who understand the business logic even if they can’t write Scala. The visual pipeline also becomes documentation — anyone can open the pipeline and see exactly what transformations happen, which is a real advantage over a 400-line PySpark script during an audit or handoff.

Pricing Reality Check

Data Fusion pricing has a structural quirk that catches teams off guard: you pay for the instance itself (an hourly rate for the always-on control plane) regardless of whether pipelines are running, on top of the Dataproc compute consumed per pipeline execution. A Basic edition instance left running 24/7 for a team that only runs pipelines during business hours is paying for idle control-plane time. Teams that use Data Fusion heavily but only during working hours often script instance start/stop via the API to avoid burning budget on nights and weekends — the instance supports this, but it’s not the default behavior and you have to build it yourself.

Best Practices

Use macros for environment-specific config (dataset names, connection strings) so the same pipeline can be promoted from dev to prod without manual edits.
Version pipelines through the CDAP application lifecycle rather than treating the UI as the source of truth — export pipeline JSON and check it into source control.
Keep Wrangler directives simple and inspectable. A pipeline with 40 chained directives on one column is harder to debug than splitting complex transformations into a few clearly-named custom transform nodes.
Right-size the Dataproc profile per pipeline. Data Fusion lets you configure cluster size per pipeline run — a small lookup-table pipeline doesn’t need the same cluster size as a multi-terabyte nightly load.

Common Mistakes

Treating Data Fusion as a low-latency tool. Between control-plane overhead and ephemeral cluster spin-up, expect pipeline start latency measured in minutes, not seconds. This is fine for hourly or daily batch jobs and wrong for anything expecting near-instant execution.

Under-provisioning the instance edition. The free Developer edition instance is genuinely useful for prototyping but has real limitations on pipeline concurrency and isn’t meant for production workloads — teams sometimes build out a production dependency on a Developer instance and then hit a wall.

Skipping schema validation before scheduling. A source schema change (a renamed or dropped column) breaks a scheduled pipeline silently until the next run fails. Enabling schema drift handling, or at minimum alerting on pipeline failure, avoids discovering this days later.

Data Fusion vs Dataflow vs Dataproc

These three GCP data-processing services get confused constantly because they overlap in capability. Dataflow is the right choice for custom, code-first stream or batch processing where you need fine control over the transformation logic — you write Apache Beam pipelines in Java or Python. Dataproc is for teams that already have Hadoop/Spark/Hive jobs and want a managed cluster to run them on, largely unchanged. Data Fusion sits above both, targeting the case where the transformation logic is standard ETL (extract, clean, load) and a visual interface is more valuable than code-level control — and under the hood, it’s actually generating and running jobs on Dataproc.

Frequently Asked Questions

Do I need to know Spark to use Data Fusion? No — that’s the point of the tool. You configure plugins and Wrangler directives through the UI, and Data Fusion compiles the pipeline to Spark for you.

Can Data Fusion connect to on-premises databases? Yes, via the Cloud Data Fusion private instance option with VPC peering, or through a JDBC driver plugin configured with appropriate network connectivity (VPN or Interconnect) to reach on-prem systems.

What happens if a pipeline fails partway through? Behavior depends on the sink’s write disposition and whether the pipeline was configured with checkpointing. For BigQuery sinks with truncate-and-reload, a failed run typically leaves the previous data intact until a successful run completes — but this should be verified per pipeline rather than assumed.

Is Data Fusion suitable for real-time analytics dashboards? Generally no — the realtime pipeline mode exists but for genuinely latency-sensitive dashboards, a purpose-built Dataflow streaming pipeline feeding Pub/Sub and BigQuery streaming inserts is the more common and better-supported pattern.

Can non-engineers actually build production pipelines in Data Fusion? Partially. Wrangler genuinely lowers the bar for data cleaning logic, and an analyst comfortable with spreadsheet-level transformations can go a long way. But connection configuration, error handling design, and performance tuning of the underlying Dataproc cluster still benefit from engineering involvement, especially for pipelines processing meaningful data volume.

A Note on Team Fit

Data Fusion works best when the organization already has a mix of skill levels touching the same pipelines — a data engineer setting up connections and cluster profiles, an analyst refining Wrangler directives against real sample data, and a platform team managing instance lifecycle and cost. If your team is small and fully Spark-literate, hand-written Beam or PySpark jobs under source control will likely be faster to build and easier to unit test than the equivalent visual pipeline. Data Fusion earns its cost when the visual, reviewable nature of the pipeline is itself valuable — for audits, for cross-functional handoffs, or for reducing the bus factor on integration logic that would otherwise live only in one engineer’s head.

Summary

Data Fusion’s real value is turning ETL work that would otherwise require a Spark-literate engineer into something a broader team can build and maintain visually, without sacrificing the ability to run at genuine data-warehouse scale underneath. It’s not the right tool for low-latency streaming or for teams that already have mature, code-first data pipelines they’re happy maintaining — but for the very common case of “move data from A to B with some cleaning in between, on a schedule,” it removes a surprising amount of engineering overhead. Budget for the always-on instance cost from day one, and treat pipeline export-to-source-control as a non-negotiable practice rather than an afterthought.