prey on mind

prey on somebody's mind

if something preys on someone's mind, they worry about it for a long time I lost my temper with her the other day and it's been preying on my mind ever since.
See also: mind, on, prey