OpenAI's Tokenizer Page for OSS Models | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		OpenAI's Tokenizer Page for OSS Models (github.com/1rgs)
		3 points by rgs224 on Nov 12, 2023 \| hide \| past \| favorite \| 1 comment

rgs224 on Nov 12, 2023 [–]

I kept having to debug prompt issues with open models

So I built OpenAI's Tokenizer page for all tokenizers on HuggingFace: Llama, Mistral, GPT2, MPT, Persimmon, T5 etc

http://tokenwiz.rahul.gs

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact