go down in the world

To lose one's wealth or social status. In the early 20th century, a respected woman in high society dramatically went down in the world if she got divorced. Mr. Pemberton's business went bankrupt? My, he's certainly gone down in the world then. I don't care if I go down in the world because of it—I'd rather marry a pauper for love than marry a prince for money.

go (or come) down in the world

drop in status, especially by becoming poorer.

come/go ˈdown/ˈup in the world

become less/more successful; become poorer/richer: Since she left Harvard she’s gone down in the world.