I think unrestrained expansion is their concern. Blacks will just increase their population without regard to anything else and Jews will try to parasitize any societies they come in to contact with. Beings undergoing a permanent population boom and beings who refuse to only live off their own production are probably viewed as bad neighbors.