It raises the question - begging the question is something else.
As for the question, it probably has to do with the gender of the noun. I bet 'deer' derives from a neuter-gendered word in Anglo-Saxon, while 'bird' does not.
Noun gender is the system used by many languages to categorize words that have different declension rules. It's atrophied in English, but is implicitly still present in the various "inconsistencies" that pop up.
Note that those are animal we generally hunt/eat. I'd bet this is tied to the language of the ruling/hunting classes of England, back when they spoke French more than English.
There are also some middle-ground words like "Shark". One goes fishing for "shark" like they would "fish" but it is more common to say "several sharks" using a plural as opposed to "several fish" using the singular. But "fishes" is still a word, which likely goes back to ruling classes who ate fish but generally did not hunt them as they would have deer.
"Fishes" is a plural of a plural. You wouldn't likely say "two fishes", but you might say "all the fishes in the sea", referring to many groups of fish (much as you might refer to the "peoples of the world" referring to many cultures). Aside from that, I bet you're onto something.
I think it was done just to make it harder for those languages that do not have a concept of plural. Of course I'm kidding, but it has to be super frustrating trying to learn it as ESL.
It does beg the question though, how’d we come up with the list of animals to pluralize like that? Why “five birds” but not “five deers?”