The issue isn't just the total number, it's the demography. The population is getting very old at a breakneck speed. I'm not sure how healthcare and pensions are supposed to be provided for old people when the tax base gets smaller and smaller every year. The end result is pretty terrible elder poverty.
Not to mention the changing political dynamics of having a lot of old people outnumber younger people by 2 or 3 times in the voting booth. The gerontocracy is bad enough now. Imagine it 3x worse.
Looks fine to me. What's your definition of "old" here? Are you specifically worried about Korea like you mentioned in another post?
I'd worry about the implications of a global 1.6 birthrate if they were actually going to happen. I don't think that's very likely, though. I don't know if we're even going to get down to 2.
Since the mid 1980s, the median age in Italy has gone from 34 to 46 and the trend is accelerating. This is despite massive immigration. Expect this to happen everywhere in the next 40 years. India and China are already below replacement. All of LatAm, Europe, and South East Asia as well.
Unless someone thinks of a democratic, scalable solution it's going to be a lot of pain for a lot of people for a long time.
The population only went over 2 billion in 1927. Dropping from current numbers is fine, and less competition over space would be nice.