rub someone the wrong way
rub (one) the wrong way
To irritate one due to someone or something's presence, nature, or habitual behavior (as opposed to directly and intentionally). Primarily heard in US. The way she smiles all the time really rubs me the wrong way. It seems so fake. He's always talking about how those commercials rub him the wrong way, but I think they're cute.
rub someone (up) the wrong wayirritate or repel someone.
The image here is of stroking an animal against the lie of its fur.