This young Jim — who is and is not Spock's Jim — is breathing hard in the wake of their mind meld. Spock knows his control has faltered, weakened by age, and fatigue, and familiarity. He has let too much slip over with the information he meant to share.
It's understandable, logical even, that here in this strange place he would seek anchor in Jim. But this is not his Jim. And this Jim is not for him to have. This Jim belongs to the Spock of this universe.
Jim steps close, cradling Spock's face in his hands, and Spock permits it. He has grown compassionate toward his moments of humanity, as he has aged, and he's always been loath to deny Jim. It's nearly perfect.
But he cannot permit it. This is not his Jim, and Jim is not for him to have. "That is not meant for me, old friend. We must go," he says. He turns his back, knowing that in doing so he helps this universe's destiny slip into place.