[COPY] --- T2-COPYRIGHT-BEGIN ---
[COPY] t2/package/*/tokenizers/tokenizers.desc
[COPY] Copyright (C) 2022 - 2025 The T2 SDE Project
[COPY] SPDX-License-Identifier: GPL-2.0
[COPY] --- T2-COPYRIGHT-END ---

[I] Fast State-of-the-Art Tokenizers optimized for Research and Production

[T] Provides an implementation of today's most used tokenizers, with a focus on
[T] performance and versatility.

[U] https://github.com/huggingface/tokenizers

[A] Anthony MOI <m.anthony.moi@gmail.com>
[M] The T2 Project <t2@t2-project.org>

[C] extra/development

[E] add pyrequests

[L] APL
[S] Stable
[V] 0.21.0
[P] X -----5---9 110.300

[O] hook_add preconf 5 "cd bindings/python"

[D] 211665ff69c51a0aa66dd004926d7a51167d320709c10a540a997723 tokenizers-0.21.0.tar.gz https://github.com/huggingface/tokenizers/archive/v0.21.0/