tokenizers v1 (rust)

Hey all!
I am Aritra from Hugging Face. I wanted to share an update on the \tokenizers\ library that we have at Hugging Face. It has gone under major changes and we have finally released version 1 of it.

Here are what we are most excited about:

\> multiple language support
\> multi-thread scaling
\> minimal package size

Read: https://huggingface.co/blog/tokenizers-v1