In my professional life, I sometimes deal with women. Most of them dress conservatively, so I never thought there was a problem associated with looking at them. How can I function in the business world if I can't even look at a beautiful woman?
Here's a simple question. How would you feel if another man entertained immodest thoughts about your wife? Even if he were your best friend or relative, wouldn't you be offended? Then again, your wife is an attractive woman. Isn't your friend just having a normal human reaction?
The answer is obvious. Marital relationships are exclusively private. The Sages' word for marriage is Kiddushin, which means, "consecrated." Once married by Torah law, husband and wife are dedicated specifically to each other and set apart from everyone else. Their relationship is exclusive on every level. A Jewish wife is forbidden to every other man, except her own husband.
It stands to reason that if a man looks with desire at a married woman, even if she is modestly dressed, he is breaching the privacy of her relationship with her husband.
The same respect must be shown to a modestly dressed woman who is single. Her beauty is reserved for the man she will eventually marry. Looking at her with desire will undoubtedly lead to impure thoughts. This is why the law is so clear: "It is forbidden to derive pleasure from looking at the beauty of a woman, even if she is modestly dressed." (Shulchan Aruch, Even HaEzer, Chapter 21, Law 1)