Xander and Jesse had bonded in third grade about a mutual fascination of girls washrooms. Why must they be seperate? Are they the same colour? Do the toilets look the same? Are there flowers? Jesse had no one to ask, and Willow always lied, she said they were exactly the same.
It wasn't as if Xander grew up to be into toilet-play. It was just that he had been taught at an impressionable age that anything involving bathroom had to be lied about.
So when Angelus came into the Bronze and pushed his back into a dryer, large palms concentrated on his shoulders, he didn't speak. When pants were pulled down and one of the hands wrapped around him, and he squirmed and hit the silver disk and warm air blew on his ass, he didn't comment. When he came, no words- possibly babble, but no comprehensible words. And when Angelus walked off, still hard in his leather pants, Xander didn't mention that either.
Because if Willow wasn't going to talk about her bathroom, Xander wouldn't talk about his. Especially because she would think it was cruel he didn't reciprocate. He highly doubted she would focus on the evil homosexual sex, just on the 'you didn't help him?' part. But he didn't feel guilty about Angel not getting off. It wasn't as if Xander was the one to initiate, why should he have joined in? Besides, he was dating Cordelia. He was busy.