People who use that saying to defend mass surveillance are invariably morons. It's not about what you have to hide. Your doctor, your congressman, your sheriff, the security at your workplace, the workers at your local nuclear plant, any of them could have something to hide. Even being a perfect obedient little NPC who never said anything controversial in an email, doesn't make you an island.