The Art of Internal Balance
The journey to diversity in this game is a long one and difficult to undertake. There are two ways to make common armylists more diverse, neither of them easy. The first is to add a lot more units to each codex which is already taking place with great success. The previous Imperial Guard codex had about 30 unit choices in it if you included special characters - the current one boasts about 50. I think it's safe to say the current codex is far more versatile offering many different build options to the IG player especially as it features no 'mandatory' units like the old version of the Command Squad.
The second way to add diversity is to the list of competitive lists is to make unit entries and upgrade options better with respect to each other. What does this mean? Well when was the last time you heard of anyone using a Leman Russ Exterminator? Poor internal balance means choosing between units in the same category is really no choice at all. There's no reason to choose the Exterminator over the Hydra beyond personal emotional attachments so people don't. That's unbalanced.
On the other hand have you ever wondered if you should take a Zoanthrope or a Hive Guard? Now I see some of you nodding your heads sagely but you're wrong - that's not good internal balance either because you haven't priced either choice low enough to account for the fact that the armylist as a whole lacks the ranged anti-tank it needs to deal with full mechanization.
So what is good internal balance? There are many but my favorite comparison and, by far the simplest, is the Attack Bike Squadron versus the Land Speeder. Both units fulfill the same role at roughly the same cost in points. They overlap in their uses but trade off certain strengths and weaknesses between them. The Attack Bike is vulnerable to instant death but can procure a far better cover save. The Land Speeder is more resistant to small arms fire but the Attack Bike can't be stunned or immobilized.
Both choices feature a low cost in comparison to their effectiveness but are limited by the FOC slot they share. Any reason to use one above the other is so minuscule so as to be completely negligible. Thus the choice becomes less about what is more effective and more about what the player feels more comfortable using.
Let's take another example of good internal balance. How the Imperial Guard Veterans versus the Infantry Platoon? Again we see overlapping roles and similar points costs in the same FOC category. Excellent pricing means both are a good choice but this time there's an additional dimension to be considered: that of squad options and loadouts. Arguably any weapons you care to give these squads, save perhaps grenade launchers and heavy bolters (and even those are situationally effective), will make them a formidable addition to your list. This means that there is a virtually unending combination of options you could take and none would be grossly overpriced compared to your opponent's choices
It's important to note that the study of the game as a whole will reveal the meltagun and autocannon/missile launchers are more popular but this is simply a fad that has come into vogue due to incessant internet nitpicking. Crunching numbers may reveal fractional differences in efficiency against units common in other areas but that does not negate ones personal choice in what to field.
Diversity is a difficult thing to acquire especially when the nerds that play the game are often far too good at math. However, GW is slowly putting one foot in front of the other and offering its playerbase what it has always wanted: the ability to field exactly the army you like without fear of losing every single game. It's responding to player feedback altering subtly those choices that we shunned in earlier codices. Vanguard Veterans (they work much better with Descent of Angels and slight points decrease) are an excellent example of this and here's hoping the Dark Eldar codex will be the ultimate triumph of internal balancing as a codex that features a mere 5 or 6 viable units suddenly gets the treatment that 5th ed. codices have been enjoying.