Seems to be more a mark of fear/insecurity. I never view it as rude, more a sign of shyness/weakness. They aren't harming me by looking away, only themselves