I must confess that occasionally it helps me to go through in my mind what chazal teach us: "kol hanosein eyno b'mah she'eino shelo - afilu mashe yesh lo, notlin mimenu -whoever puts his eyes on that which is not his, even what is his is taken away from him". In other words, the more I reserve my eyes for my wife (and for goodness in general) the greater my appreciation of her (and of goodness in general) will become. Every time I let myself enjoy the figure or face of another woman it will automatically cause me to lose a bit of the pleasure I take in my wife (and in the goodness of people in general). It is not a punishment at all! It is just the way we are. It's the way I work, at least.
There just is no escape from the fact that fantasy takes me out of reality, period. "Motzee'in es ha'odom min ha'olam - It takes a man out of this world".