After some time we know either the client or CloudFront will give up on
pending compilations. As such, if we continue to process compilations
after the client's timed out we're just clogging up the compilation
queue with pointless work.
As such, this change now supports the notion of "stale" work which will
be abandoned once it's made it to the front of the queue. Only compiles
coming from the user will be abandoned, so discovery and health checks
are unaffected.
Hopefully this will mitigate the number of nodes marked unhealthy due to
being overloaded: work they were doing was "pointless" anyway, and them
being killed by going unhealthy is equivalent to abandoning all the work
in flight anyway, but this means there's a fighting chance the node will
recover quickly enough to return a "healthy" status.
6c476bbccc