I assure you, "they" are small in number. Most people don't care what a character's race is, and fandom culture is generally more progressive than the mainstream. I agree that a character's physical attributes don't change my ability to relate to a character, or more importantly, to find them engaging.

It's funny how you mention how Hermione being canonically black would change the context of the blood purity themes, because that reminds me of the outcry when J.K. Rowling said that werewolf status was a metaphor for being LGBT+, which would mean that she accidentally wrote a "Depraved Homosexual" in the form of Fenrir Greyback.