Track unhealthy heartbeat + pause job fetching when unhealthy

[brandur]

This one's driven by another issue flagged out of the active rescue
system: as currently implemented, the producer's heartbeat produces an
error if it fails, but there's no additional feedback. So if a producer
were to chronically start failing its heartbeats, it'd be possible for
the JobRescuer (were its rescue queries to keep working in this case,
which is not certain) to start incorrectly rescuing jobs as producers
started to look inactive. This could then result in duplicate jobs.

This change doesn't fully address that problem, but partly remediates it
by having each producer track whether its heartbeat is unhealthy. If
found to be unhealthy, the producer stops fetching new jobs because if a
simple producer update is failing, presumably the rest of the database
is also experiencing significant trouble. In this case jobs running in
an unhealthy producer could still be rescued, but at least they wouldn't
start work again as their producer is unhealthy, thus avoiding a
potentially runaway feedback loop.

[sashu2310]

The fetch-pause half of this seems right whatever the topology. It's the rescuer pause I wanted to ask about.

ProducersHealthyFunc reads the leader's own producersByQueueName, but the producer whose jobs are about to get rescued is usually on some other client, so most of the time the check passes and the rescue goes ahead anyway. You already note other clients aren't visible, so not news. It's more that the two cases split cleanly. If the leader itself is the sick one, its rescue queries are on the same bad connection and probably fail on their own, so the guard isn't doing much there. If a peer is the sick one, the leader is healthy, the guard passes, and that's the case that produces the duplicate. The guard fires in the first and the problem lives in the second.

Is the cross-client version meant to live in the rescue query, joining each stuck candidate to its producer's last heartbeat? That works from any leader since it reads the table. Might already be how the Pro path does it, I can't see that side, in which case this guard is fine as belt-and-braces.

Also, OSS StandardPilot.ProducerKeepAlive is return nil so heartbeatUnhealthy can't flip outside Pro. Might be worth a line in the description so nobody on OSS goes looking for it.