throw the baby out with the bathwater

Advanced (C1) neutral

Lose something good while trying to get rid of something bad.

"In reforming the education system, we must not throw the baby out with the bathwater."