bear a millstone (around (one's) neck)

To be forced to deal with some particularly difficult burden or hindrance. I never should have bought that apartment. Having to pay a second mortgage in this economy has been like bearing a millstone around my neck. You've borne this millstone for too long. It's time you forgave yourself for what happened. The candidate is bearing quite a millstone about his neck now that his chief adviser's laundry list of scandals has been exposed.