How to Get the Status of an AWS Glue Workflow Using boto3

Checking a Glue workflow’s status looks like a one-liner. It isn’t, for two reasons:

The boto3 Glue client has no waiters. You write the polling loop yourself.
A workflow run’s Status can be COMPLETED even when jobs inside it failed. COMPLETED means the workflow finished, not that it succeeded.

This post shows how to get a reliable answer.

Start a run and keep the RunId

import boto3

glue = boto3.client("glue")

run_id = glue.start_workflow_run(Name="ingest-events-wf")["RunId"]

If something else started the run (a schedule or an EventBridge event), get the latest one:

wf = glue.get_workflow(Name="ingest-events-wf")["Workflow"]
run_id = wf["LastRun"]["WorkflowRunId"]

Read the run

run = glue.get_workflow_run(
    Name="ingest-events-wf",
    RunId=run_id,
    IncludeGraph=True,
)["Run"]

print(run["Status"])      # RUNNING | COMPLETED | STOPPING | STOPPED | ERROR
print(run["Statistics"])

Statistics is where the real outcome is:

{
    "TotalActions": 4,
    "SucceededActions": 3,
    "FailedActions": 1,
    "TimeoutActions": 0,
    "StoppedActions": 0,
    "ErroredActions": 0,
    "RunningActions": 0,
    "WaitingActions": 0,
}

A run with Status == "COMPLETED" and FailedActions == 1 is a failed pipeline.

Per-node status from the graph

With IncludeGraph=True, run["Graph"]["Nodes"] contains every trigger, job, and crawler, with their runs for this workflow run:

def node_states(run: dict) -> list[dict]:
    out = []
    for node in run.get("Graph", {}).get("Nodes", []):
        if node["Type"] == "JOB":
            for jr in node.get("JobDetails", {}).get("JobRuns", []):
                out.append({
                    "type": "JOB",
                    "name": node["Name"],
                    "state": jr["JobRunState"],
                    "error": jr.get("ErrorMessage"),
                })
        elif node["Type"] == "CRAWLER":
            for c in node.get("CrawlerDetails", {}).get("Crawls", []):
                out.append({
                    "type": "CRAWLER",
                    "name": node["Name"],
                    "state": c["State"],
                    "error": c.get("ErrorMessage"),
                })
    return out

Nodes that never ran (downstream of a failure) have no job runs or crawls, so they don’t appear in the output. That’s how you tell “failed” from “never reached”.

A polling loop that tells the truth

import time

TERMINAL = {"COMPLETED", "STOPPED", "ERROR"}
# JobRunState + CrawlState values that mean "did not succeed"
BAD_STATES = {"FAILED", "TIMEOUT", "ERROR", "STOPPED", "EXPIRED", "CANCELLED"}


def wait_for_workflow(name: str, run_id: str, poll: int = 30, timeout: int = 3600) -> dict:
    deadline = time.monotonic() + timeout
    delay = poll
    while True:
        run = glue.get_workflow_run(Name=name, RunId=run_id, IncludeGraph=True)["Run"]
        if run["Status"] in TERMINAL:
            return run
        if time.monotonic() > deadline:
            raise TimeoutError(f"{name}/{run_id} still {run['Status']}")
        time.sleep(delay)
        delay = min(delay * 2, 300)  # back off, cap at 5 min


def succeeded(run: dict) -> bool:
    stats = run["Statistics"]
    failed = (
        stats["FailedActions"]
        + stats["TimeoutActions"]
        + stats["ErroredActions"]
        + stats["StoppedActions"]
    )
    return run["Status"] == "COMPLETED" and failed == 0


run = wait_for_workflow("ingest-events-wf", run_id)
if not succeeded(run):
    for n in node_states(run):
        if n["state"] in BAD_STATES:
            print(f"{n['type']} {n['name']}: {n['state']} {n['error'] or ''}")
    raise SystemExit(1)

IncludeGraph=True makes the response larger. In a tight polling loop you can poll without the graph and fetch it once at the end.

Listing recent runs

get_workflow_runs supports a paginator:

paginator = glue.get_paginator("get_workflow_runs")
for page in paginator.paginate(Name="ingest-events-wf", IncludeGraph=False):
    for r in page["Runs"]:
        print(r["WorkflowRunId"], r["Status"], r["StartedOn"], r["Statistics"]["FailedActions"])

Useful for a quick health check across the last N runs.

IAM

The caller needs glue:GetWorkflow, glue:GetWorkflowRun, glue:GetWorkflowRuns, and glue:StartWorkflowRun (if it starts runs) on the workflow ARN.

When to stop polling from Python

If this loop runs in Lambda or a cron box just to wait on Glue, consider moving orchestration to Step Functions, which waits on Glue natively. See Triggering AWS Glue Jobs from EventBridge.