Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I use my https://github.com/simonw/ttok command for that - you can pipe stuff into it for a token count.

Unfortunately it only uses the OpenAI tokenizers at the moment (via tiktoken), so counts for other models may be inaccurate. I find they tend to be close enough though.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: