WorkerSet: consolidate the worker start delay code