Remove matching worker set element when a worker exit.