Put the lease attention as possible on co workers talk. The more attention you give toward the situation the more the you will empower office chat and rumor mills. What somebody thinks or heard about you is just not important ,it is just an opinion which you cannot control . An opinion is just an opinion until you confirm or deny it, do neither. Be proud of yourself and your contribution as a person and an employee. I will stand with you. Yes there are laws and rules against this type of situation, but running off to HR will do nothing but bring more attention to a private matter. You cannot legislate other peoples ethics.