rub someone the wrong way

rub someone (up) the wrong way

irritate or repel someone.
The image here is of stroking an animal against the lie of its fur.
See also: rub, way, wrong