It's not arbitrary at all, at least in my dialect. "He went to hospital" and "he went to the hospital" mean different things. The first implies that he was a patient, he was there for the purpose for which the hospital exists, the second implies he went to that location for some other reason (such as bringing flowers, or working there).
Likewise "went to school" (as a scoolchild) vs "went to the school" (to collect someone, or to teach). "At university" (a student) vs "at the university" (physical location, for any reason). "Went to prison" (as a criminal) vs. "went to the prison" (to visit his brother, or to fix their telephones). "Does she go to church?" (to church services) vs. "did you go to the church?" (to see the stained glass).
"Go to the doctor" is a bit different, the doctor isn't an institution, he's a person. Although I think that implies you went for medical reasons, if you stopped by to pick up some paperwork, you might say "I have to stop at the doctor's" i.e. his rooms. But maybe that's also because you would probably talk to the receptionist anyway. But "went to the barber" implies you had a shave, while you might "stop at the barber's" to pick someone up.
Likewise "went to school" (as a scoolchild) vs "went to the school" (to collect someone, or to teach). "At university" (a student) vs "at the university" (physical location, for any reason). "Went to prison" (as a criminal) vs. "went to the prison" (to visit his brother, or to fix their telephones). "Does she go to church?" (to church services) vs. "did you go to the church?" (to see the stained glass).
"Go to the doctor" is a bit different, the doctor isn't an institution, he's a person. Although I think that implies you went for medical reasons, if you stopped by to pick up some paperwork, you might say "I have to stop at the doctor's" i.e. his rooms. But maybe that's also because you would probably talk to the receptionist anyway. But "went to the barber" implies you had a shave, while you might "stop at the barber's" to pick someone up.