Apparently the saying refers to a yankee doodle song in which in my area everyone says 'hat' in the sing instead of 'cap' because no one sticks feathers in caps.