If I understand you problem clearly, you can use TFIDF to reduce the weight of m... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		nestorD on Jan 13, 2020 \| parent \| context \| favorite \| on: Show HN: HuggingFace – Fast tokenization library f... If I understand you problem clearly, you can use TFIDF to reduce the weight of meaningless words.

orestis on Jan 14, 2020 [–]

It’s not meaningless words - it’s common English words that are overloaded and I think considering their position in sentences instead would give better results.

I haven’t yet tried TFIDF though so I’ll see what that will do.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact