A distributed job scheduling system I built to understand what happens when queues, concurrent workers, retries and failures collide.
Built with
A full-stack job scheduling system built around a FastAPI backend, priority queue, concurrent worker pool, persistent state and recovery mechanisms. The project focuses on the engineering problems that appear when jobs can fail, workers can disappear and multiple processes access shared state.
A job scheduler sounds simple until something goes wrong.
A job enters a queue. A worker picks it up. The worker completes it.
But what if the worker crashes halfway through?
What if two workers pick up the same job?
What if a low-priority job never gets a chance to run?
What if the server restarts while jobs are running?
I built this project specifically to explore those problems.
The system combines a React dashboard, FastAPI API, priority queue, worker pool, retry system, SQLite persistence and recovery logic into one application.
I had used APIs and databases before, but I wanted to understand the systems thinking behind asynchronous workloads.
The problem I set myself was deliberately simple:
**If something fails at any point in the pipeline, can the scheduler recover without silently losing work?**
That question led me into concurrency, state machines, persistence, idempotency and failure recovery.
A lot of software engineering tutorials show the happy path.
Submit a job. Run the job. Return the result.
Real systems are interesting because the happy path is not the difficult part.
I wanted to build something where failures were part of the design rather than an edge case. The project became my way of learning how backend systems behave when multiple things are happening at once.
The system is split into separate components so that the API, scheduler, workers and persistence layer each have a clear responsibility.
The API accepts and validates jobs. The scheduler decides what should run. Workers execute the work. Persistence records state. Recovery deals with work left behind after failures.
That separation also made it possible to test individual components without running the entire application.
React dashboard
Submit jobs and monitor their state
FastAPI service
REST API, validation and job lifecycle endpoints
Priority queue
Determines which waiting job should run next
Worker pool
Executes multiple jobs concurrently
Persistence layer
Stores jobs, states, attempts and history
Recovery system
Detects orphaned work and requeues jobs
Jobs move through explicit states such as queued, running, succeeded, failed and retrying.
Making those transitions explicit meant I could reason about the system in terms of state transitions rather than scattered boolean flags.
Jobs carry a priority which determines their position in the queue.
I also explored ageing so that a continuous stream of high-priority work could not cause low-priority jobs to wait indefinitely.
Multiple workers can execute jobs simultaneously.
This introduced a completely different class of bugs compared with single-threaded code because operations that looked atomic were not necessarily atomic once multiple workers were involved.
Failed jobs can be retried with increasing delays and a maximum number of attempts.
This prevents a permanently broken job from continuously consuming worker capacity.
The scheduler checks persistent state during startup and identifies jobs that were marked as running but no longer have an active worker.
Those jobs can then be requeued instead of disappearing permanently.
The system is split into containers so individual components can be stopped or restarted independently.
This made failure testing much more realistic than simply calling a function that raises an exception.
A priority-based scheduling system where waiting time can influence effective priority, reducing the risk of starvation.
Retry delays increase with each failed attempt so repeatedly failing jobs do not overwhelm the system.
Persistent job state is used to reconstruct what happened after a restart and identify work that needs to be requeued.
17
Automated tests passing
My first implementation separated "find the next job" from "mark it as claimed".
That seems harmless until two workers perform those operations at almost exactly the same time.
The bug taught me that concurrency requires thinking about operations as atomic units rather than individual lines of code.
SQLite behaved very differently once several workers began writing at the same time.
I had to rethink transaction boundaries and keep database operations short enough that workers were not unnecessarily blocking one another.
From the scheduler's perspective, a slow worker and a dead worker can initially look identical.
Adding heartbeats gave the system information it could use to distinguish between the two.
A system that works perfectly with one worker can fail immediately when several workers operate simultaneously.
That changed how I approach testing concurrent software.
Saving only completed results is not enough.
If a system needs to recover after a crash, it needs to know what was happening before the crash as well.
Retrying a failed operation sounds simple until the operation has side effects.
That led me to think much more seriously about idempotency and designing jobs that can safely be executed again.
The biggest shift in my thinking was realising that failure handling should not be bolted onto a system afterwards.
It has to influence the architecture from the beginning.