This is me to a degree. White Scottish mum, brown Indian dad. Never knew him. Never knew any of his family. Never had anything to do with India and feel no affiliation to that country or its people, probably because I've had no real life connection to it as such. Note I'm not anti-India, just stating the fact about how 'Indian' I feel or project to other people. There will be white UK born folk on this forum that know more, perhaps a lot more, about India than I do.
However, because of my skin colour, some people will and do make certain judgements. Having said that, making judgements about people based on how they look isn't an uncommon thing. The danger is when people then do something that crosses the line simply based on how someone looks.