Cloud/ AWS / AWS Step Functions: Visual Workflow Orchestration for Distributed Applications

AWS Amazon Web Services 106 guides · updated 2026

Hands-on guides to compute, storage, databases, networking, and serverless on the world's most widely adopted cloud platform.

AWS Step Functions: Visual Workflow Orchestration for Distributed Applications

When you chain several Lambda functions together, the connection logic — error handling, retries, timeouts, passing output from one step to the next — ends up scattered across Lambda code that is hard to read and harder to test. AWS Step Functions moves that coordination logic out of your application code and into a declarative state machine definition. The application code handles the business logic; Step Functions handles the flow control.

This article covers what Step Functions does well, where it falls short, and the practical details that come up in real projects.

What a State Machine Is

A Step Functions state machine is a JSON document (written in Amazon States Language, ASL) that defines a sequence of states and the transitions between them. Each state represents one unit of work — calling a Lambda function, inserting into DynamoDB, waiting for human approval, running a parallel set of tasks, or branching based on output.

State Machine: order-processing


[Start]
   |
   v
[ValidateOrder]  -- Task: Lambda function
   |
   |-- Success --> [CheckInventory]
   |-- Failure --> [NotifyCustomer: order invalid]
   |
   v
[CheckInventory]  -- Task: DynamoDB GetItem
   |
   |-- Item in stock  --> [ChargePayment]
   |-- Out of stock   --> [BackorderItem]
   |
   v
[ChargePayment]  -- Task: Lambda function (Stripe API)
   |
   |-- Success --> [FulfillOrder]
   |-- Failure --> [RefundAndNotify]
   |
   v
[FulfillOrder]  -- Task: Lambda function (warehouse system)
   |
   v
[End]

Each box is a state. The arrows are transitions. The conditions on the arrows are choice rules or catch blocks. All of this is defined in ASL, not in application code.

Standard vs Express Workflows

Step Functions has two workflow types with different pricing models and reliability guarantees.

Standard Workflows:

Express Workflows:

Standard Workflows
  Use when:
  - Audit trail matters (finance, compliance)
  - Workflow may take hours or days
  - Human approval steps are involved
  - Each execution must happen exactly once


Express Workflows
  Use when:
  - High throughput (>1,000 executions/second)
  - Short duration (seconds to a few minutes)
  - Cost is a concern for large volumes
  - At-least-once is acceptable (idempotent operations)

Express Workflows come in two sub-types:

Amazon States Language (ASL)

ASL is the JSON-based language for defining state machines. A minimal working example:

{
  "Comment": "Order validation and payment flow",
  "StartAt": "ValidateOrder",
  "States": {
    "ValidateOrder": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789:function:ValidateOrder",
      "Next": "CheckInventory",
      "Catch": [
        {
          "ErrorEquals": ["ValidationError"],
          "Next": "NotifyCustomer"
        }
      ],
      "Retry": [
        {
          "ErrorEquals": ["Lambda.ServiceException"],
          "IntervalSeconds": 2,
          "MaxAttempts": 3,
          "BackoffRate": 2
        }
      ]
    },
    "CheckInventory": {
      "Type": "Task",
      "Resource": "arn:aws:states:::dynamodb:getItem",
      "Parameters": {
        "TableName": "inventory",
        "Key": {
          "product_id": {"S.$": "$.product_id"}
        }
      },
      "Next": "OrderComplete"
    },
    "NotifyCustomer": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789:function:NotifyCustomer",
      "End": true
    },
    "OrderComplete": {
      "Type": "Succeed"
    }
  }
}

Key ASL state types:

Error Handling and Retry Logic

Step Functions has two mechanisms for handling errors: Retry and Catch.

Retry automatically re-runs the failed state with configurable backoff:

"Retry": [
  {
    "ErrorEquals": ["States.TaskFailed"],
    "IntervalSeconds": 5,
    "MaxAttempts": 3,
    "BackoffRate": 2,
    "JitterStrategy": "FULL"
  }
]

With BackoffRate: 2 and IntervalSeconds: 5, the delays between attempts are 5s, 10s, 20s. JitterStrategy: FULL adds randomness to prevent thundering herd problems when multiple executions fail simultaneously.

Catch redirects the execution to a different state when retries are exhausted or a specific error type is thrown:

"Catch": [
  {
    "ErrorEquals": ["PaymentDeclined"],
    "Next": "RefundAndNotify",
    "ResultPath": "$.error"
  },
  {
    "ErrorEquals": ["States.ALL"],
    "Next": "HandleGenericError"
  }
]

ResultPath controls where the error information is placed in the state input/output — $.error adds it as a field rather than overwriting the entire input.

Integrations: Direct SDK Calls

Step Functions has two integration patterns for calling AWS services:

Lambda invocations: Pass data to a Lambda function and use the return value as the next state’s input. Lambda handles the actual service call. This is flexible but adds a Lambda hop.

Optimised integrations (SDK integrations): Step Functions calls AWS services directly without Lambda as middleware. Supported services include DynamoDB, S3, SNS, SQS, ECS, Batch, Glue, SageMaker, EventBridge, and many others.

"PutOrderInDynamoDB": {
  "Type": "Task",
  "Resource": "arn:aws:states:::dynamodb:putItem",
  "Parameters": {
    "TableName": "orders",
    "Item": {
      "order_id": {"S.$": "$.order_id"},
      "status":   {"S": "PENDING"},
      "amount":   {"N.$": "States.JsonToString($.amount)"}
    }
  },
  "Next": "SendConfirmationEmail"
}

Direct integrations save cost (no Lambda execution charges), reduce latency, and simplify debugging — there are fewer moving parts in the execution graph.

Input and Output Processing

One of the trickier parts of Step Functions is controlling what data flows between states. Four fields control this:

State Input
    |
    v (InputPath filters)
Selected Input
    |
    v (Parameters transforms)
Task Input
    |
[Task Runs]
    |
    v
Task Result
    |
    v (ResultSelector filters)
Filtered Result
    |
    v (ResultPath merges with original input)
Combined State Result
    |
    v (OutputPath filters)
State Output --> becomes next state's input

Getting this right takes practice. The Step Functions console includes a built-in data flow simulator that lets you trace data through each step without running an actual execution.

Real-World Scenario: Document Processing Pipeline

A legal firm receives contracts in PDF format via an S3 upload. Each contract must be extracted, classified, reviewed for compliance clauses, and stored in their case management system. Human review is required if the compliance check flags any issues.

[Trigger: S3 upload event]
         |
         v
[ExtractText]  -- Textract async job
         |
         v
[ClassifyDocument]  -- Comprehend entity detection (Lambda)
         |
         v
[ComplianceCheck]  -- Lambda + internal rules engine
         |
         |-- No flags --> [StoreInCaseSystem]
         |-- Flags found --> [WaitForHumanReview]  (Wait state)
                                   |
                              [Human reviews in UI]
                                   |
                              [API call sends task token back to Step Functions]
                                   |
                             [Resume: StoreInCaseSystem]

The human review step uses a callback pattern with a task token. Step Functions pauses the workflow and sends a token to the review application. When the reviewer approves, the application calls SendTaskSuccess with the token, and the workflow resumes. This can pause for days without incurring per-minute charges (Standard Workflow).

Interview Notes

Q: What is the difference between Standard and Express Workflows? Standard Workflows guarantee exactly-once execution, store full history for 90 days, support 1-year maximum duration, and are billed per state transition. Express Workflows are at-least-once, log to CloudWatch, have a 5-minute maximum, and are billed per execution and duration. Standard is for auditable, long-running business processes. Express is for high-volume, short-duration event processing.

Q: How do you pass data between states in Step Functions? Each state receives the previous state’s output as its input. You control this using InputPath, Parameters, ResultSelector, ResultPath, and OutputPath. The full input document is available throughout using the $ reference.

Q: What is the callback pattern in Step Functions? A task sends a task token to an external system and pauses (waitForTaskToken). The external system does its work (which may take seconds or days) and then calls SendTaskSuccess or SendTaskFailure with the token. Step Functions resumes the execution from that point. Used for human approval steps and integration with systems that do not support synchronous callbacks.

Q: Can Step Functions call services outside AWS? Yes, through Lambda. Write a Lambda function that calls the external API, then use that Lambda as a task in the state machine. There is no direct HTTP task type for arbitrary external URLs, though you can use EventBridge API Destinations or Lambda for this.