When viewing the Users tab within the job, use the "X" action button on the right for each user to remove them from this job.