Peruvian slave traders, French general-purpose bastards, and diseased Catholic priests.
When the Dutch arrived in the 18th century, the population of Easter Island had declined from its 16th century peak, but the society was still stable and relatively prosperous. The real disaster came in the 19th century, between 1860 and 1880, and it was entirely due to external factors.
The Peruvians enslaved one third of the population of Easter Island in a single year, killed 99% of the slaves through overwork and disease, and returned the handful of survivors infected with smallpox.
A Catholic priest with tuberculosis passed it on along with his religion and killed a quarter of the survivors of the Peruvian slaughter and the smallpox epidemic.
The French outright killed half of the survivors of that. Missionaries evacuated whoever they could; by 1880 only about 100 natives remained on the island.