refactor: unflag worker startup at the right time on worker set