Perhaps I am not thinking of this in the right way but I can think of some contrary examples. One has to ask, which mean?
Take pathogenic microorganisms, if one is looking at the range of virulence a highly virulent influenza virus will drift toward less virulent because even though the virulent organism will be highly successful at first, eventually the less virulent strain will overtake the virulent ones because the more virulent organisms kill their hosts and that limits spread. Likewise a less virulent strain isn't likely to overtake the more virulent strains that don't kill their hosts.
The population size of a species will equilibrate with the food supply, so if one is taling about mean population size, too large of a population won't be sustained and too small of one cannot be sustained either if the gene pool is not varied enough.
Species become stable with much slower genetic drift when the population is in equilibrium with the environment. At some point a stronger lion being able to control a pride does not result in significant drift toward stronger and stronger lion cubs. Or if it does, the evolutionary drift is very very slow. Instead the gene pool gains an advantage because it contains a lot of variability. If evolutionary drift only moved toward stronger and stronger lions, eventually one would expect that gene pool to be very limited.