Distributed deployment¶
PostgreSQL
|
Distributed leases
|
+-----------------+-----------------+
v v v
Goatbeat #1 Goatbeat #2 Goatbeat #3
LEADER STANDBY STANDBY
|
v
Task backend
|
+----------+----------+
v v v
Worker Worker Worker
Server A Server B Server C
The same architecture covers one scheduler and one worker on a single machine, or three schedulers and a hundred workers: only the process counts change.
Checklist¶
- One PostgreSQL database reachable by every scheduler (see PostgreSQL setup).
goatbeat migrate(ormanage.py migrate) once per release.- Two or three scheduler processes on different hosts, each with the same code version.
- Workers for your task backend, consuming every routed queue.
- Metrics and logs collected from every scheduler (see observability).
Rolling deployments¶
Deploy schedulers one at a time. A stopping leader releases its lease, a standby takes over, and
the new version publishes its schedules when it starts. Removed code schedules stay enabled
until schedule prune runs with the new code.
Network partitions¶
A scheduler cut off from PostgreSQL stops dispatching when its local deadline passes, before
another instance can acquire the lease. With the standalone command, connections use TCP keepalives so
a half-open connection is noticed in about a minute; in Django, set keepalives in OPTIONS.