ModernBERT may be a better base model if finetuning a model for a specific use c...

diwank · 2025-08-15T15:10:02 1755270602

also ettin is a new favorite and a solid alternative: https://huggingface.co/jhu-clsp/ettin-encoder-1b

I'd encourage you to give setfit a try, along with aggressively deduplicating your training set, finding top ~2500 clusters per label, and using setfit to train multilabel classifier on that.

Either way- would love to know what worked for you! :)