a burr under your saddle

a burr under (or in) your saddle

a persistent source of irritation. North American informal
See also: burr, saddle