Genetic modification is being done on hybrid plants that already lack genetic diversity. Living things need genetic diversity to adapt to a changing environment. Widespread planting of relatively few GM crops in place of different types of crops decreases diversity, and increases the chance of crop failure.
The Irish potato famine is a lesson in monoculture. One strain of potato—resistant to drought—was planted to the exclusion of other varieties. Virtual clones, since they were grown from cuttings rather than seeds, the plants were susceptible to something new: a fungus that wiped them out (study here).
The eminent Indian rice researcher, R.H. Richharia, collected 24,000 species of rice in one region of India, and estimated there were 200,000 altogether in India. Today there are only about 6,000, due to the planting of the same types of hybrids all over the country.