sphnx and spool condor pool worker nodes are being drained and rebooted to remediate a critical linux vulnerability.

sphnx worker nodes will be fully drained en mass, running jobs will be allowed to run to completion (unless killed by their owner), but no new jobs will be able to run until some nodes fully drain, reboot, and start accepting new jobs again.

spool worker nodes will get a rolling partial drain + reboot to try to best balance remediation speed and minimization of disruption. Some jobs will be killed*, but unless they include their own hold expression specifying otherwise, they'll get automatically restarted on another node.

* full drain with no jobs killed would take way too long for a rolling remediation. When a node is remediated, jobs running for <2hrs will be killed, as will remaining jobs once a node reaches a 90% drain level (90% of it's cpu's unallocated), but as stated, killed jobs should automatically be restarted.

The above plans were quickly developed with the largest stakeholders. We recognize this may be problematic for some spool experiments/users, my apologies. Thank you for your patience and understanding.