You need to maintain enough diversity to ensure all non deleterious genes are passed on. Diversity of genes is required to increase chances of survival in rapidly changing environmental or social stressors, aka changing natural selection.