DNA. I think physiognomy is the precursor to DNA. Kinda like, old school scientists trying to figure out what bloodlines cause crime or somesuch. Though environment has its part to play as well. Shitty people have shitty decision making skills, are more likely to do drugs and alcohol to excess, and are more likely to fuck up their body in other ways. This all adds up over the years and produces a certain "look" that someone who isn't in that "culture" can spot from a mile away.