fix: ensure event listeners are removed at worker set stop