It is a basic principle for all creatures to fear that which is the same size or bigger than them, but different.
I have some pretty large rats in my backyard, and a cat. If the cat sees a rat, he will just let it pass by without even flinching, because they are about the same size. While the cat is made to attack creatures, they are only those that are smaller than them.
But then my cat saw an advantage. The rat was focused on something else for a while, ignoring the cat. So the cat saw that he had the upper hand and attacked the rat, wounding it, playing with it and then killing it.
Difference and equal power equates fear.
Fear leads to anxiety
Anxiety seeks advantage over fear.
Anxiety attacts the object of fear.
Those of other race not only look different, but more importantly, they hold to different mores and cultural presuppositions than one. Because those of other race have a population that could potentially harm one’s own race, and it is different, the other race is an object of fear. "They might insist that we follow their ways."-- this translates into core racist language "They have taken over our cities"; "They have conspired to control us economically."; "They will blemish our civilization." These are all statements of anxiety.
Those in fear/anxiety will then seek an advantage-- economically, militarily, politically, etc-- to make themselves "bigger". Once they feel safe, they will attack that which caused them fear. This is primarily to give them and their society freedom from the anxiety they have been feeling. Once the object of anxiety is completely subdued or destroyed, then the original society can feel safe.
This is the pattern you can find in Nazi Germany, in Israel/Palestine and in Rwanda.
If we want to be rid of racial/cultural/social prejudice, we need to create opportunities for open relationship between the opposing groups. In familiarity and relationship, fear disappears. Once both sides recognize the other as humans with needs and desires like them, there is the possibility to compromise for the sake of peace for both sides.