The assertion that you possess privilege, and that said privilege affects your perspective and therefore your choices, IS NOT A SLIGHT ON YOUR CHARACTER. It's almost never intended to be, certainly it never should be. The possession of privilege is not a thing with objective moral valence. It is neither a good thing, nor a bad thing. The valence depends on what you do with it, as with the possession of any power or advantage.
So when someone says, "It appears you are privileged on this axis, and that may be affecting your assessment of this situation," I beg you to receive it as feedback rather than criticism (a careful distinction, I am aware). If that someone *is* disrespecting you in their tone, that's not okay, I'm not saying otherwise. I'm saying, if that assertion itself strikes you in an uncomfortable place, try to sit with it for a second without projecting your discomfort into a) your consideration of and b) your response to this thought.
(In general, I tend to phrase such things as questions, because I genuinely hope you will ask them of yourself rather than hear accusation coming from me and plead not guilty.)
Because your privilege has no correlation with your character. That's its nature: it is a thing you have by virtue of external factors and/or phenotype in which you played no role growing up and little to no role since.
None of that means I should be defensive about *having* the privilege. None of that means I need to apologize for having it, and I guess I'm fortunate not to know anyone who'd ask me to.
I'll never ask you to. So if you hear otherwise in discussions with me, let me know and I'll back the truck up and restate myself. If you think someone's demanding your apology for being who you are or coming from whence you came? Well, they aren't entitled to it - but don't *not* consider how those things are affecting your perspective and whether or not the perspective of someone without your privilege may be better informed than yours.