Isn't the issue one of responsibility? I'm not sure whether it's societal conditioning, genetic inclination, or a little bit of both, but I'm inclined to believe that men in this society are on average somewhat more distracted by/pulled by an attractive member of the appropriate sex than women are (and yes, this is the most shallow description of the pattern--there are all sorts of mainstream interactions initiated from both sides that come from this that I'm not digressing into). But even assuming this asymmetry, who has primary responsibility for someone's feeling of attraction; that person, or the person to whom they are attracted? The answer strikes me as obvious, but our society doesn't seem to agree.