The concept of “higher taxa”, and how such taxa arise, is a very complicated subject, and one that deserves a better presentation that it currently is receiving. Here is my correction of Point 1 above. Please keep in mind my previous statement: Anything less than about 500 pages is necessarily incomplete, and that includes this. And to be honest, I wrote this while downloading geologic maps, so there are any number of nuances that I’ve missed; however, I believe this to be a decent summary of several summaries of summaries of where taxonomy currently stands, with sufficient personal bias (though I tried to call out where that is and why I think what I do).
First, what is a “higher taxa?” To answer that we have to ask “What is taxonomy?” This is far from a simple question—taxonomy is an evolving system, and therefore a moving target for definitions. However, we can break taxonomy into roughly two groups: the practical and the theoretical.
Practical taxonomy is best seen in the Linnaean system: Domain, Kingdom, Phylum, Class, Order, Family, Genus, Species. This sort of taxonomy also includes “common names”—informal classifications based on observation. Examples of this include pretty much every non-scientific classification of organisms, and are found all over the world. The thing about these systems is that they do not assume evolutionary relationships. Even Linnaeus didn’t (he couldn’t—there was no theory of evolution at the time he was writing). These are merely ways to organize living things in a way that makes sense—or, as a professor of mine put it, “figuring out how to put things in boxes”. Like went with like not because they were assumed to have evolved from one another, but merely because that’s a natural human behavior. Groups of like things that were similar were grouped into larger groups—which we now call higher taxa.
Another example of practical taxonomy, which clearly illustrates the lack of evolutionary assumptions in such taxonomies, is the Christian Chain of Being, which stretched from dirt to God, each organism in its place, which was chosen by God. This is totally different from modern biology’s assessment of taxonomy, and built on a drastically different theoretical framework. Yet it worked quite well as a taxonomic system. There’s also the Hindu ranking of organisms according to past lives, which utilizes a third independent system.
Theoretical taxonomy came around long after Darwin died, and include such systems as cladistics and phylogeny. These specifically attempt to reconstruct evolutionary histories, rather than organizing life. They usually take the results of evolutionary history, including genetic data, morphological data, and sometimes stratigraphic data and plot organisms according to which species gave rise to which daughter species. I say “usually” because there’s a rather infamous report which used completely random data (computer punch cards) to successfully reconstruct a known phylogenetic tree (specifically constructed to test which method works best, cladistics or phylogenics). These are typically statistical methods. To learn more about them, I highly recommend looking up PAUP in Google, finding their home page, and reading everything on that web page. It’s become more or less standard for paleontology, and is a great resource for getting your feet wet in statistical methods in paleontology (which, remember, is the science which studies life through time).
A significant difference between theoretical and practical taxonomy is that theoretical taxonomy generally do not discuss “taxa”, they discuss phyletic groups or clades. While this may sound like an attempt to increase the importance of one’s discipline by creating new jargon the differences are actually significant. The classification of something as a taxa within the Linnaean system DOES NOT imply evolutionary relationships; the organisms may be related, or they may not be. This is particularly well illustrated by the debate over whether or not Phylum Arthropoda should be a phylum or not. One popular argument is that arthropods show an enormous amount of variation, and phyla should contain approximately equal amounts of such variation—hardly an evolutionary argument! In contrast, within theoretical taxonomies grouping things together DOES imply an evolutionary relationship. Also, theoretical taxonomy give us such concepts as polyphyletic and paraphyletic groups, for which there is no equivalent in practical classification (they’re just not valid names, as per the International Code of Zoological Nomenclature).
It’s a perfectly reasonable question to ask why the higher taxa in the Linnaean system, and indeed all of the taxa in that system, so closely match what we find in the theoretical taxonomies. The answer is complicated. To some extent, the practical/theoretical dichotomy is false—you have to have SOME theoretical framework even for practical taxonomy, and similar criteria are used for both practical and theoretical taxonomy. However, to some extent this similarity is a support of the theory of evolution as well. Obviously organisms can be grouped based on completely different theoretical frameworks—as previously mentioned, the Chain of Being specifically argues against any hierarchical structure, with everything is in its place, as each rung on a ladder is in its place. The fact that these frameworks tend to agree with evolutionary predictions indicates (but, I must emphasize, does not prove) that there is a real connection between these groups.
A major consequence of this dichotomy in taxonomic systems is that this entire thread is built on a flawed premise. The “higher taxa” DID NOT EVOLVE. Higher taxa are not an evolutionary concept—they are an organizational concept, which frequently (but not always) is correlated with certain clades. Correlation, I’m sure you’re aware, does not equal causation, and asking evolutionary questions about a taxonomy which is not concerned with evolutionary relationships is a misapplication of that taxonomy. To be fair, modern systematic is a combined (and, in my opinion, flawed) approach, which attempts to balance the utility of practical taxonomy with the biological realities involved in theoretical taxonomy, and it utilizes the nomenclature of the Linnaean system; therefore, some confusion is understandable. That said, if you want to talk evolutionary history you want to talk clades, not higher taxa, no matter the scale. A clade is all of the descendants of a particular species (though personally I prefer populations, as the real unit of evolution is the population, not the species).
Why do we consider higher taxa to be significant? Frankly, extinctions. Without extinctions (and frequently mass extinctions) there would appear to be a much finer gradation between higher taxa than we currently have. Take, for example, birds. Birds are dinosaurs, which in turn are lizards, meaning that birds are lizards. The reason we don’t see birds as lizards is because all of the intermediate species (the “transitional forms” Creationists insist we have none of) are extinct. We know what those forms are—we’ve found a surprising number in China, for example, showing a very good gradation between therapod dinosaurs and birds. If those lines hadn’t gone extinct at the K/Pg boundary we would see a smooth gradation between dinosaurs and birds, and we would never have considered Avis to be such a high taxa. Arthropods also demonstrate this—had the majority of arthropod bodyplans not died out in the Paleozoic (the exact timing is a big fuzzy, given the paucity of good lagerstatens in the Cambrian—I’ve heard of Burgees Shale Fauna in the Middle Ordovician) we would not consider certain forms to be as radically different as we do today. If a few ancestral humanoid species were around we’d see a smooth transition between humans and apes. In contrast, there is a huge difference between an ant and a bombarder beetle, but we put them both in the box labeled “Insect” because we have living representatives of the gradation between ants and bombarder beetles. Circle species clearly illustrate this: The ONLY reason we consider them to be one species is because we have an unbroken line of transitional forms. If those transitional populations were to go extinct (or a significant number, in some cases as low as one) we would consider them separate taxa. So our classification of things into higher taxa can clearly be seen to be at least in part because of the extinction of the intermediate forms.
(Yes, I know this is the Cone of Increasing Diversity—I’m hoping that my mention of the roll mass extinctions play in this process shows shows that I acknowledge the validity of punctuated equilibrium.)
The rest of it is due to the difficulty in thinking in geologic time scales when it comes to animals, and a lack of understanding of how the largest clades first split. Some split very, very early in metazoan evolution, and it’s difficult to identify the exact nature of that split, particularly considering they don’t fossilize and biologists are constrained to a pathetically small sample of living things. The split between clams and squid likely happened well before either evolved a shell (before the ocean chemistry allowed for such shells), for example, and we will likely never find the fossils of the transitional forms. As geologists know, if we don’t have the rock we don’t have the data—so sadly, the precise nature of the difference between larger clades may never be known. We can hypothesize, however, that larger clades, being functionally identical to their smaller counterparts, formed in the exact same manner—populations split, and the daughter clades of each original population form what we consider large clades today. And given how often that hypothesis has been shown to be right, it’s a robust one.
The real question is why various bauplans have become channelized, but that is beyond the scope of this discussion.