I've been using an equivalent method, after reading about imdb's method for comparing the best movies of all time.
I extend it, hackily, to allow categories to be declared to apply to objects with arbitrary confidence at any time, and to declare the same categorization multiple times.
I then consider both the confidence amounts and number of declarations in comparing the overall confidence in different categorizations.
I use a 0-1.0 scale for confidence, then product adjusted confidence for each potential categorization as (sum of confidences) / (num confidence declarations + 3).
This is equivalent to assuming a prior of three declarations of zero confidence; this effectively rewards higher binders of votes, such that a single declaration for category A of confidence 1.0 will tie three declarations for category B of confidence 5.0.
I extend it, hackily, to allow categories to be declared to apply to objects with arbitrary confidence at any time, and to declare the same categorization multiple times.
I then consider both the confidence amounts and number of declarations in comparing the overall confidence in different categorizations.
I use a 0-1.0 scale for confidence, then product adjusted confidence for each potential categorization as (sum of confidences) / (num confidence declarations + 3).
This is equivalent to assuming a prior of three declarations of zero confidence; this effectively rewards higher binders of votes, such that a single declaration for category A of confidence 1.0 will tie three declarations for category B of confidence 5.0.