And its not like the last ice age just neatly stopped - there was a significant cooling period during the Younger Dryas which wasn't quite a bad as a full Ice Age but was still pretty severe:

https://en.wikipedia.org/wiki/Younger_Dryas