No need for the simplistic one-sided thinking really. Changing sex partners even after marriage like you change your socks gets tiring fast too. There's not much meaning in it actually even if most of us would want to lie to ourselves about it.
The Western idea of free sex has in a way kinda destroyed the true meaning of love and marriage. Do you not notice that couples during our parents' generation upwards were much more loyal and loving to each other and you wonder why?