What I picked up from a lifelong typographer is that kerning should be about the area enclosed by the two letters. The aim is to make that consistent. I think that might help in this kind of case.