in (all) fairness (to one)

In justification of one or one's actions; in defense of one. I know his remarks are controversial, but, in all fairness to him, they're grounded in pretty concrete scientific evidence. In fairness, I did try to warn her about the implications of a deal like that, but I still should have done more to stop it from happening.

in (all) ˈfairness (to somebody)

used to introduce a statement that defends somebody who has just been criticized, or that explains another statement that may seem unreasonable: In all fairness to him, he did try to stop her leaving.