Creating TaskFlow - Universal Workflow Engine

2025-07-19

TaskFlow: Sometimes the best abstraction is the one that makes a dozen different projects collapse into one.

See Project here

1. The Pattern Recognition Moment

I was knee-deep in my third "quick automation project" of the last few months when it hit me. The one use case where AI could really help were all different flavors of the same thing:

  1. Input arrives (email, document, API call, whatever)
  2. Some workflow processes it (extract, transform, classify, route)
  3. Results go somewhere (database, notification, another system)
  4. Someone needs to see what happened (dashboard, reports, alerts)

I was essentially rebuilding the same plumbing every time (and reusing the same code), just with different pipes. However building a new UI and database schema for each automation was a PITA.

2. The Abstraction That Actually Worked

TaskFlow emerged from a simple observation: most business automation problems are just workflows operating on jobs. Strip away the domain-specific terminology, and you're left with:

  • Jobs: The things being processed (emails, documents, requests)
  • Workflows: The steps to process them
  • Views: Ways to see what's happening
  • I/O: Getting stuff in and out

Here's the tech stack that made it real:

Component Technology Why I Picked It
Backend Python/FastAPI Simple and works
Database MySQL Because I already know how to use it
Frontend React/TypeScript Type safety saves debugging time
Container Docker/K8s Team deployment was a hard requirement

3. The Universal Automation Engine Hidden in Plain Sight

Here's the thing that blew my mind once I built it: by abstracting away the labels, TaskFlow becomes a solution for almost any repetitive automation task.

Watch what happens when you change a few field names in the UI:

What Users See What TaskFlow Sees What Actually Happens
"Support Tickets" Jobs Process text, classify, route
"Invoice Processing" Jobs Extract data, validate, approve
"Resume Screening" Jobs Parse, score, rank
"Content Moderation" Jobs Analyze, flag, escalate

It's all the same engine underneath. Change the labels, swap the workflow, and suddenly your invoice processor is a content moderator.

If you can format your inputs as a list and get an LLM to give reliable results, TaskFlow might be your solution.

4. Workflows in the UI: The Game Changer

The biggest unlockw was moving workflow definitions out of code and into the UI.

Instead of this deployment dance:

  1. PM requests change
  2. Write code
  3. Test locally
  4. PR review
  5. Deploy
  6. Wait for results
  7. Repeat when it doesn't work

We now have:

  1. PM opens TaskFlow UI
  2. Tweaks workflow parameters
  3. Tests on sample data
  4. Monitors results in real-time

The product team can now test hypotheses without writing a single line of code, or waiting for me to get around to it.

5. The future I'm building toward? Direct integration with fine-tuning pipelines

Imagine:

TaskFlow collects examples → Automatically fine-tunes model → Deploys better classifier → Monitors performance → Collects more examples

It's a self-improving system. Every job processed makes the next one more accurate. flow configurations.

6. When NOT to Use TaskFlow

TaskFlow isn't for everything:

  • Real-time processing: If you need sub-second latency, look elsewhere
  • Complex DAGs: It handles linear workflows well, complex branching is off the table for now

7. What's Next

TaskFlow is handling production workloads, but there's more to do:

  • Workflow versioning — Currently a pain when updating live workflows
  • Multi-tenant isolation — Current you need to do a deployment for each user group, wouldn't it be better to just have one deployment with multiple versions?

The Bottom Line

Sometimes the best code is the code you don't write repeatedly. TaskFlow let me collapse a variety of similar projects into one configurable system. It's not revolutionary — it's just the right abstraction for a common pattern.