pick up marbles

pick up your marbles (and go home/leave)

to suddenly leave an activity you have been involved in with other people, because you do not like what is happening If you don't like the way we do things around here, well, you can pick up your marbles and leave.
See also: marble, pick, up