I don't think this is an even remotely plausible generalization.
Every city has racist fans, and racially progressive fans, and fans who don't particularly have political viewpoints on race. I see no evidence that Philadelphia, today, is different from Boston or Atlanta or Baltimore or LA or many other MLB cities in that respect.
And I seriously doubt that the "hate has no home here" folks have any overlap with the "run every athlete of color out of town" folks. The second group is, I think, extremely small.
The most popular athlete in Philly right now is probably Joel Embiid, closely followed by Ben Simmons and Carson Wentz. Fletcher Cox is much beloved, too. And the Sixers fans have recently shown how much they are supporting a struggling Marquelle Fultz. Wayne Simmonds has been one of the most popular Flyers over his career here for his gritty, hard-nosed play. Outside of Wentz, those are all athletes of color.
Historically, the most popular Eagles are probably Reggie White and Brian Dawkins, and the most popular Sixers are probably Dr. J, Wilt, and Allen Iverson. There are more white guys on the Phillies' and Flyers' lists, but that has a lot to do with the demographics of the sports. Today, there is very widespread support in Philly for Dick Allen being a hall-of-fame player, and for Jimmy Rollins' historical greatness given his key role in the 2008 World Champion Phillies and outstanding career in red pinstripes. We now mostly love Mike Schmidt and Steve Carlton, but, even given their historic greatness, there were many who wanted to run both of them out of town when they played.
I think those who see Philly as particularly racist are suffering from "availability bias" in their perceptions.