Actually i got audited and used a similar excuse. The IRS went on to fine me 40,000$ for being a smug asshole about it
The IRS makes it almost impossible to fight it. You can appeal, but if you lose they tack on interest and late fees. If they lose, you get some money back BUT you have to then take that money they give you and add it as taxable income as interest proceeds. Both ways, you will lose the fight
(post is archived)