the ghost at the feast

Someone or something that acts as a reminder of something negative and thus ruins the enjoyment of something. Primarily heard in UK. I think I'll stay home. I'm afraid that since everyone knows about my recent diagnosis, I will be the ghost at the feast.

a ghost (or spectre) at the feast

someone or something that brings gloom or sadness to an otherwise pleasant or celebratory occasion.

The ghost or spectre of Banquo at the feast in Shakespeare's Macbeth is the most famous literary instance of this. There are other versions of the expression. A skeleton at the feast dates from the mid 19th century and probably refers to the ancient Egyptian practice of having the coffin of a dead person, adorned with a painted portrait of the deceased, present at a funeral banquet. A death's head at the feast alludes to the use of a death's head or skull as a memento mori (an object which serves as a reminder of death).