When we've been the subject of those kinds of stupid rumors, my husband has always said "if they're talking about me, at least they're leaving someone else alone."  Pretty good way to look at it I think!  

Don't let it bug you.....some folks are just too small-minded to matter.