Add proper worker set elements reference counting.