If take virginity away from a virgin, I will feel guilty because :
1. Society still by and large believe virginity of a woman is important, so if the girl who lost her virginity to me also believes in this archaic societal perceptions, then it makes me feel indebted to her. I don't like that feeling as the relationship will become an obligation on me rather than real love.
2. There is a feeling of having taking something from her, then worry about the future of the relationship and not being the "right" partner for her (the virgin's) first time.