that's just it

that’s just ˈit

(also that’s just the ˈtrouble) (informal) that is exactly the problem: ‘You only need to spend £5 more and you can get a really good dictionary.’ ‘That’s just it, I’m afraid — I haven’t got another £5.’
See also: just