Systems project · Feb 2026 – present
TaskForge
A durable distributed job platform: at-least-once execution, fenced worker leases, and crash recovery built directly on PostgreSQL.
- Role
- Sole author
- Status
- In active development
- Stack
- Go, PostgreSQL, Docker, OpenTelemetry, S3-compatible storage
- jobs succeeded while 24 workers were SIGKILLed
- 10,000 / 10,000
- recorded local fault-injection run, 12 workers
- sustained throughput
- 1,000/min
- 12 workers, shipped defaults
- p95 dispatch latency
- 209 ms
- tuned profile; 964 ms with shipped defaults
- worst recovery after a kill
- 11 s
- tuned profile; 50 s with shipped defaults
The problem
Background jobs fail in boring, dangerous ways: a worker dies holding work, a message is lost after the database commit, a retry runs twice and charges a card twice.
TaskForge implements the hard parts of a job system directly, rather than wrapping a queue framework, so each failure mode has an explicit mechanism and a test.
Architecture
How a job moves through TaskForge
Clients submit jobs to the API with an idempotency key. The API writes the job and an outbox row in one PostgreSQL transaction. An outbox publisher sends small advisory notifications to the broker, which wakes workers. Workers pull: one SQL transaction claims the highest-priority eligible job and creates a lease. Workers heartbeat and report outcomes under a fence. A scheduler promotes delayed work and re-notifies lost notifications, and a reconciler expires stale leases and timeouts.
- Client / CLI to API (submit)
- API to PostgreSQL (job + outbox)
- PostgreSQL to Outbox publisher (poll)
- Outbox publisher to Broker (notify)
- Broker to Workers (wake)
- Workers to PostgreSQL (claim)
- Scheduler to PostgreSQL
- Reconciler to PostgreSQL (repair)
- PostgreSQL is the only authoritative store. The broker carries advisory notifications and never job state, so a lost message costs reachability, not correctness.
- Every attempt runs under a lease and a fencing token. A worker that wakes up after its lease expired cannot commit anything.
- A dead-lettered job can be listed and replayed as a new job, leaving the original history intact.
Decisions and trade-offs
At-least-once, stated plainly
TaskForge guarantees durable at-least-once execution and documents that handlers with side effects must be idempotent.
Trade-off: Exactly-once is not on offer. The contract is weaker on paper, but it is one the system can actually keep under crashes.
Transactional outbox
A state change and its notification are written in the same transaction; a separate publisher delivers it.
Trade-off: Adds a publisher process and a table to poll, and removes the gap where a commit succeeds but the message is lost.
Workers pull instead of being pushed
A worker with a free slot asks for work, and one SQL transaction enforces capacity, capabilities and priority.
Trade-off: Claims cost a database round trip, but capacity limits and priority are enforced in one place.
How it is tested
- 1,000+ Go test functions, including integration suites against real PostgreSQL and failure-recovery suites.
- Benchmarks are recorded in the repository with the commit, machine, settings and fault-schedule seed, and every figure is measured on PostgreSQL's clock.
- In the fault-injection run, 10,000 jobs were submitted while 24 workers holding attempts were SIGKILLed and restarted; all 10,000 succeeded.
- Continuous integration runs on every push to main.
Outcomes
- Crash recovery, retries with bounded exponential backoff, cancellation over heartbeats, timeouts and the dead-letter queue all work end to end.
- A command-line client with a tested exit-code contract drives the public API.
Scope: TaskForge is a personal project in active development. Benchmarks ran on one Apple M3 laptop, not production traffic. With shipped defaults it misses two of its own targets (p95 dispatch under 500 ms, recovery under 30 s); the tuned profile meets both.