Search results

do we have plan support java ，we need you

Status: Open.

#1918 In huggingface/tokenizers;

· mullerhai opened on Dec 25, 2025

<code>spm_precompiled</code> dependency bricks rust-analyzer in certain build setups

Status: Open.

#1917 In huggingface/tokenizers;

· gvilums opened on Dec 21, 2025

Decoding of AddedTokens containing Latin-Extended A characters are wrong

Status: Open.

#1915 In huggingface/tokenizers;

· andrewivan123 opened on Dec 19, 2025

Wrong and unsuppressable print when instantiating BPE

Status: Open.

#1913 In huggingface/tokenizers;

· bauwenst opened on Dec 15, 2025

[Docs] <code>Visualizer</code> dead links

Status: Open.

#1910 In huggingface/tokenizers;

· dudeperf3ct opened on Dec 13, 2025

[Docs] <code>Encode Inputs</code> rendering issues

Status: Open.

#1909 In huggingface/tokenizers;

· ariG23498 opened on Dec 12, 2025

<code>TemplateProcessing</code> does not apply <code>type_id</code> to overflow encodings

Status: Open.

#1908 In huggingface/tokenizers;

· antoine-agthe-unity opened on Dec 9, 2025

Windows arm64 wheel release

Status: Open.

#1906 In huggingface/tokenizers;

· finnagin opened on Dec 8, 2025

python release workflow failing

Status: Open.

#1904 In huggingface/tokenizers;

· finnagin opened on Dec 8, 2025

Guide: Compiling <code>tokenizers</code> on Android/Termux

Status: Open.

#1902 In huggingface/tokenizers;

· Manamama-Gemini-Cloud-AI-01 opened on Dec 1, 2025

Performance: batch_encode scales poorly on high-core Server CPUs compared to sharded tokenizer instances

Status: Open.

#1900 In huggingface/tokenizers;

· stargazerZJ opened on Nov 30, 2025

Support splitting to chunks during encode

Feature Request

Status: Open.

#1897 In huggingface/tokenizers;

· xrl1 opened on Nov 28, 2025