What you just said is indeed true. A: "Why are you passing on the promotion? Are you reluctant to take on more responsibility?" B: "There's that, but I'm also not sure I want to stay with the company any longer."
See also: that