The answer to that varies from person to person, and it has varied in aggregate over time, but the effect on those who remain has been the same. It doesn't matter whether those who left were/are fleeing actual danger, race-based perception of danger, or pure economic rationality. If you tip a marble on to a sloped ramp, it will continue rolling down even without further impulse. The fact that at one time race and racism were significant drivers of this effect is sufficient to justify their consideration in any present discussion of causes and/or remedies.
I think a lot of white Americans feared retribution and competition from minorities. Before integration, they had treated minorities very poorly to say the LEAST. I think it's pretty normal to be afraid when someone you beat with a horse whip last tuesday suddenly gets to own guns and businesses in your neighborhood.
In addition, propaganda from racist organizations was running wild. Stuff like "White Genocide" and "Race Riots" were predicted by racist organizations who still profit from this fear today.
But naturally the answer is that they were fleeing minorities, violence, AND landlords. In most people's minds all three were mixed up. In the end, the makeup of the neighborhoods they fled to were built around isolation enabled by cars.
Were they fleeing minorities or were they fleeing violence and landlords?