fix: wait for worker exit to clean the worker set