Why do some ships become so much more popular than others?
My one-sentence theory is: some ships fit into neat binary grooves in our minds. (Derrida-style.) Those ships are the most popular.
To elaborate, each person in the pairing is an archetypal balancing factor for the other. Take the most popular ship of all, Kirk/Spock. Kirk = Instinct. Spock = Logic. That's a binary pairing, right there. It's what Lacan would call a "dual-entry matrix". Basically, it's a psychological impetus, not only for the characters but for the shippers; there's gotta be binary action going on, there. Something to deconstruct.
When it comes to fannish shipping, deconstruction means making two seemingly "opposite" personalities discover compatibility. (Harmony.) Or, in lay terms, some kinda yin-yang love-fu. For example: Fraser (Discipline)/Kowalski (Brashness). Or Stiles (Empathy;Randomness)/Derek (Menace;Focus). Or Eames (Fluidity)/Arthur (Solidity). Or Loki (Deceit)/Thor (Honor). Or Steve (Responsibility;Prudishness)/Tony (Irreverence;Promiscuity).
See, it's about chemistry. What does chemistry mean? Incompleteness. True chemistry has to be about give-and-take. Both characters have to lack something that is present in the other. And those qualities must react to each other. Like, uh, covalent bonds. Or elective affinities. #NERDMOMENT But seriously. There has to be a mutual spark.
In case my chemgasm about bonds and affinities didn't make any sense, here are two brief quotes from Wikipedia that I hope will clarify matters:
A covalent bond is a form of chemical bonding that is characterized by the sharing of pairs of electrons between atoms. The stable balance of attractive and repulsive forces between atoms when they share electrons is known as covalent bonding.
Above: A covalent bond forming H2 (right), where two hydrogen atoms share the two electrons. (Source.)
See what I mean? It's a pattern that's prevalent everywhere in nature, and that includes our minds. We need to share atoms, physically and metaphysically, literally and metaphorically. And we need our fictional characters to do the same, to reflect our needs (and fulfill them). The ideal pairing is one that answers to those needs. The most popular pairing in any fandom is the one that most closely resembles a covalent bond.
In chemical physics and physical chemistry, chemical affinity is the electronic property by which dissimilar chemical species are capable of forming chemical compounds. Chemical affinity can also refer to the tendency of an atom or compound to combine by chemical reaction with atoms or compounds of unlike composition. (Source.)
In human terms, if you wanna go all Goethe about it (and trust me, you do), real chemistry between any two people (fictional or otherwise) is based on the ability of dissimilar personalities to form compounds. I mean relationships. Uh. Relationship-compounds?
The more intense the give-and-take, the stronger the reaction. The better the chemistry, the higher the shippyness. Couples that can learn from each other and grow together make for more interesting relationships. That's the essence of pairing popularity.
If two characters are too similar to each other, they won't have enough differences to make for a compelling binary matrix; they simply won't have enough chemistry to make a fascinating couple. At most, it'll be a bromance, not a romance. The lack of exigent passion will eventually relegate this relationship to the realm of "always a bridesmaid, never a bride". It'll be like approaching some sort of psychosexual asymptote - or, uh, in normal-speak, like being friend-zoned.
On the other hand, if the characters are too different, there won't be any common ground to build on, no affinity, no possibility for a covalent bond. Covalency requires sharing, and you can't share with someone that isn't even on your plane of existence. At most, such a relationship will only end in mutually assured destruction, which doesn't do much for the long-term viability of the pairing. Such a pairing cannot, therefore, be the most popular.
This dilemma (too close? Too far?) leads to the Goldilocks Process of shipping; we seek out pairings that are just right. The characters aren't similar enough to be boring or different enough to be impossible; they're in that perfect overlap of affinity and covalency, of dissimilarity and compatibility.
Of course, we don't think about all this when we first start shipping two characters. It's a gut thing. A Kirk thing. A fly-by-the-seat-of-your-Y-fronts thing. You know. It's gotta feel right.
But what goes into the FEELS is all of what we've talked about - chemistry, duality, balance, need. Covalency. Affinity.
It's all at a subconscious level; most of us start shipping by instinct, and then start thinking about it, about how we got there, about what it means. It's not an unusual phenomenon; we often find something beautiful by instinct, only to have that instinct turn out to be Darwinian in origin. What appears to be instantaneous is, in fact, eons in the making.
And so is your preference for your favorite fictional pairing! Something about it speaks to you, beyond the mere socially acceptable forms of fannish liturgy; something about it is exactly what you need, the call of Self to Other, a binary opposition resolving itself. Something about it is basal and basic, a primordial chemistry, an ancient balancing-act in the process of... well, balancing itself.
So, that's it! That's my theory on the popularity of certain pairings. Will ye or nill ye, as ye please!
Additional reading: An excellent example of this theory is Elective Affinities by Caecilia, a novel-length work of fanfiction that textually incorporates many of the concepts I've explored in this essay. That, and it's a wonderful story. Just saying.