It hurts Sam to think that letting Dean go would be the best thing to do. To just part ways, and Dean could finally be happy. To finally get up without the weight that has always been on his shoulders. Hurts to think that their life together had become so miserable, so messed up, that they had lost this -- these simple moments when they could just be themselves, content and safe and at peace with each other. Happy.
They don't trust each other anymore. Not like they used to. Those times are over, and they have messed up too bad to ever be what they used to.
After everything that has happened, everything they have done to each other, they really cannot trust the other. They both want to, that's for sure. They want to, need to... It's what keeps them going, keeps them human but... It's not that easy. Not anymore.
Ruby, the demon blood, all the secrets and lies. Lucifer...
Soulless him, letting Dean turn into a vampire. Sam's so-called happiest memories... Which, to be honest, are happy, but not the best. Not really. As if he could be at his happiest without his brother by his side. And the amulet...
Letting Dean rot in the Purgatory, giving up on trying to find out where he is. Settling down with Amelia.
Dean tricking him into letting an angel possess him, keeping it a secret. His words to Dean afterwards... The Mark.
Dean turning into a demon, keeping the Book of the Damned... Charlie...
And no matter how much they hope that they could be like they used to, that they could trust each other... They cannot. Maybe not ever. Because the things they have done are going to weight on them forever, no matter what.