If you don't believe there are limits perhaps you should go and breed a completely new and viable life-form from a "previous" life-form. Maybe start with a kind of plant, and end up with a worm or a stick insect of your own creation. The Nobel Prizes are awaiting you.
In the TOE, there is supposed to be enormous plasticity, so the above should be possible.
Of course nothing anything like this is ever achieved. Selective breeding pushes the limits of variation but stays within them.
As to the "mechanism" for these limits.. well you would have to ask the geneticists about that. But I don't think they really have any idea yet.
The answer, I reckon, will be due to informational constraints.
Say I have a Shakespeare sonnet. No random changing of its letters is going to turn that sonnet into a 3 hour long play, or a 400 page novel, or a telephone directory, or a haiku poem. Also, the likelihood of random changes to the letters over an extended period improving the effectiveness of the Shakespeare sonnet, as meaningful and beautiful poetry, would be so vanishingly improbable that a man would need to be irrational to believe that such a process is essentially what lies at the heart of the enormous diversity and complexity of life.
In biology if you try to randomly change the global organisation of the genetic information by, for example, changing the number of chromosomes ,you get a deleterious, often fatal, consequence to the organism.
http://en.wikipedia.org/wiki/Aneuploidy
In the analogy, this could be one example of trying to turn a sonnet into a play. It just doesn't work, and is typically damaging.