As a man I always use ma'am, both in real life and when I worked on the phone for a clothing company a few years back. I always use it as a sign of respect, miss (from a stranger) can seem diminutive. A few have corrected me though and said they prefer miss, their mom/grandmom is ma'am, but always in a friendly way. Nobody has ever been genuinely offended by me saying ma'am, but someone might be if I use miss. Heck, if it's someone whose full name I know but I'm not well acquainted with I'll use Ms along with their surname. Teachers, for example, Ms Anderson. I won't call her Rebecca unless she gives me permission - call me Rebecca. (There is no Rebecca Anderson, just made the name up as an example).
