Why I Built My Own Docker Pipeline Orchestrator Instead of Reaching for Airflow¶

Here's the entire abstraction:

steps:
  - name: fetch-data
    image: my-registry/fetch-data:latest
    retry: 3
    on_failure_step:
      image: my-registry/notify:latest
      cmd: ["--message", "fetch-data failed"]

A step is a Docker image. It runs, it succeeds or it fails, and if it fails enough times something else runs to deal with it. That's it. That's the whole requirement. No DAG editor, no scheduler UI, no metadata database to babysit. Just containers and YAML.

I built this — docker-pipeline, MIT licensed, up on GitHub — because I got tired of every orchestration tool being ten times bigger than the problem I had. Run some business logic, packaged as a container, on a schedule or on demand, with retries and a way to react when it dies. That's the entire spec. Most tools that solve it also want to solve fifty other things I don't have.

What I needed¶

I run a personal project doing AI-based stock market analysis, with a local AI Agent on top that writes me a daily analysis. The modeling stays mine — this article is about what runs underneath it: a handful of steps that run in sequence, on a schedule, sometimes triggered by hand, each one doing one job, each one occasionally failing for reasons that have nothing to do with my code (or maybe).

The first version of this ran on Streamlit and Airflow. It worked, technically. But every time I touched the Airflow side I was reminded that I was running a scheduler, a webserver, and a metadata database to babysit what amounted to five sequential steps. That's a lot of surface area for "run this, then run that."

So when I started rebuilding the whole thing — proper frontend, proper backend and architecture this time — I didn't want to bring Airflow along for the ride. I looked at the alternatives first, because building your own anything should be the last resort, not the first instinct.

Why not the obvious tools¶

Airflow, Prefect, Dagster — all three are built around the assumption that you want to define your workflow in their framework: DAGs as Python objects, tasks as decorated functions, a whole vocabulary of ops and assets. Dagster in particular is Python-heavy in a way that fights against what I wanted: business logic that stays completely opaque inside a Docker image. I don't want my orchestrator to know or care what's inside the container. I want it to run the image, watch the exit code, and get out of the way.

Kestra — closer to what I wanted structurally: YAML-based, less Python lock-in. But it's not fully free either — the core is open, and the features I'd eventually want sit behind a paid plan. For a personal project I'd rather not build on something where growing past a hobby use case means hitting a paywall.

n8n — genuinely nice for workflow automation, but it wants to be a whole hosted application: its own database, its own UI, its own persistent service you maintain. I wanted something closer to a library — deploy it, point it at a YAML file, done.

GitHub Actions — tempting because it's already there, but it's built around git events and CI semantics, not long-running business pipelines. Trying to bend it into a general-purpose scheduler for infrastructure that isn't about testing and deploying code always feels like fighting the tool.

None of these are bad tools. They're just solving a bigger problem than the one I had. Strip my requirement down and it was: run a container, on a schedule or an API call, with retries and a failure path. Everything else — the DAG visualizer, the asset lineage, the plugin ecosystem — is orchestration overhead I don't need for five containers.

Where the idea came from¶

The real trigger wasn't a side project itch, though. At my day job, I work on an event-driven ML pipeline built on AWS Lambda and Docker — business logic packaged as container images, triggered by events, each unit of work isolated and stateless. It's a pattern I like a lot: the orchestration layer knows nothing about what's inside the container, and the container knows nothing about what triggered it. Clean boundary, easy to reason about, easy to test each piece independently.

I wanted that same boundary for my own infrastructure, without paying for Lambda invocations or building against AWS-specific primitives. Docker gives you the isolation for free — a container is already a self-contained unit of business logic. What was missing was the thin layer on top: something that runs the container on a schedule or a REST call, retries it if it fails, and does something intelligent when it fails for good. So I wrote that thin layer myself.

How it actually works¶

A pipeline is a YAML file. metadata holds the pipeline-level config — name, a cron schedule if you want one, whether it's allow_api_trigger-able. steps is a flat list, each one a Docker image plus its run configuration:

metadata:
  name: daily-newsletter-pipeline
  schedule: "0 6 * * *"
  allow_api_trigger: true

steps:
  - name: fetch-api-data
    image: registry/fetch:latest
    retry: 3
    timeout: 300
    pull_policy: if-not-present
    on_failure: abort

  - name: run-models
    image: registry/predict:latest
    retry: 1
    on_failure_step:
      image: registry/alert:latest
      cmd: ["--channel", "pipeline-failures"]

Under the hood, each step is executed through the Docker SDK — client.containers.run(image, command=cmd, environment=env, volumes=volumes, detach=True) — a real container, not a subprocess pretending to be one. Logs stream live off the container into the tool's own logger, and a polling loop checks container.status every half second, either until it exits or until the step's timeout kills it.

It doesn't have to run next to the daemon it's driving, either — DOCKER_BASE_URL (or --docker-url) takes a tcp://host:port just as easily as a local socket, with TLS applied automatically if the standard Docker CLI env vars are set.

Retries use exponential backoff — 2 ** (attempt - 1) seconds between tries, nothing fancy. on_retry_step and on_failure_step are just full step definitions run through that same function recursively — a failure hook is another container, and it can have its own retries and its own failure hook.

What broke before I added any of this¶

The first version of this tool had none of that. No retries, no failure hooks, nothing. When a step failed — my own code, or Docker itself, didn't matter — there was nothing in place to handle it: no retry, no notification, no cleanup of whatever the failed run left behind. It just died, silently.

Some of it really was infrastructure: an image pull timing out, a container failing to start because of a transient Docker daemon issue. Some of it was probably my own code. The point was never that the code was innocent — it's that it didn't matter whose fault it was, because nothing existed to do anything about it either way. I found out by noticing the newsletter hadn't shown up, not because anything told me.

retry and on_failure_step exist for that reason, not because I sat down and designed a resilient system from first principles. The tool kept failing in ways nothing could recover from, and I got tired of finding out after the fact.

What it deliberately doesn't do¶

I want to be honest about the scope here, because it would be easy to oversell this as "I built my own Airflow." I didn't. The step execution loop is a flat, sequential list — for i, step in enumerate(pipeline.steps):. There's no DAG. No parallel fan-out, no conditional branching beyond a per-step on_failure: abort or continue. If you need twelve steps with complex interdependencies running in parallel across a cluster, this is the wrong tool, and Airflow or Dagster will serve you better.

It's not an oversight — it's the point. My pipelines are short, linear sequences: fetch data, run models, generate output, maybe notify. A DAG engine would solve a problem I don't have, at a cost — operational complexity — I'd be paying for nothing. The lesson from skipping Airflow in the first place was "don't adopt more machinery than your problem requires." It'd be a little embarrassing to learn that lesson and then rebuild all of Airflow's complexity myself under a different name.

State is also just in-memory right now — job status lives in a thread-locked dict, not a database. Fine for a single-node personal setup where the process doesn't restart mid-run. Not fine for anything that needs to survive a crash and pick up where it left off. That's a real limitation, and if this ever needs to run something more critical than a personal stock newsletter, it's the first thing I'd change.

What's next¶

A few things on the list, roughly in the order I actually want them:

• Persistent logs. Right now logs only exist as long as the stream is running — restart the process and they're gone. SQLite is the obvious fix: dump them somewhere I can actually go back and check what happened on a run from three days ago, not just the one currently in front of me.
• Kubernetes as a target, not just a local Docker daemon. Same execution model, just pointed at a cluster instead of one machine — the point where a personal tool actually needs to scale past a single box.
• Maybe a web UI, if the CLI/API combination ever stops being enough — something to see every pipeline runner I have going, browse their logs, check status, without SSHing in or curling the API by hand. Not committed to this one; the CLI covers what I actually need right now.

None of this is urgent. The tool does what I built it for. But it's the obvious next set of gaps if I ever lean on it harder than a personal stock pipeline and whatever else I point it at.

Where it's running now¶

This is the orchestration layer under my stock market analysis project — pulling data, running the models, running analysis, etc., all as isolated steps triggered on a schedule, with the ability to kick off a run by hand through the API when I want to test a change without waiting for the cron. I'm keeping the modeling and strategy side to myself, but the infrastructure underneath it is exactly what's described here, and it's public: pipeline_scheduler is the package name, ghcr.io/threadr-r/docker-pipeline is the image, and the repo is MIT licensed if anyone wants to point their own containers at it.

The bigger lesson, for me, wasn't really about orchestration at all. It was that the pattern I like at work — isolate business logic in a container, keep the orchestration layer dumb and generic — is worth carrying into personal projects even when nobody's making you do it. The tool is small. The idea underneath it is the part I'd defend.