_            _    _        _         _
      /\ \         /\ \ /\ \     /\_\      / /\
      \_\ \       /  \ \\ \ \   / / /     / /  \
      /\__ \     / /\ \ \\ \ \_/ / /     / / /\ \__
     / /_ \ \   / / /\ \ \\ \___/ /     / / /\ \___\
    / / /\ \ \ / / /  \ \_\\ \ \_/      \ \ \ \/___/
   / / /  \/_// / /   / / / \ \ \        \ \ \
  / / /      / / /   / / /   \ \ \   _    \ \ \
 / / /      / / /___/ / /     \ \ \ /_/\__/ / /
/_/ /      / / /____\/ /       \ \_\\ \/___/ /
\_\/       \/_________/         \/_/ \_____\/
r-tok 0.2.3
Propagated dependencies: r-r6@2.6.1 r-cli@3.6.6
Channel: guix-cran
Location: guix-cran/packages/t.scm (guix-cran packages t)
Home page: https://github.com/mlverse/tok
Licenses: Expat
Build system: r
Synopsis: Fast Text Tokenization
Description:

Interfaces with the Hugging Face tokenizers library to provide implementations of today's most used tokenizers such as the Byte-Pair Encoding algorithm <https://huggingface.co/docs/tokenizers/index>. It's extremely fast for both training new vocabularies and tokenizing texts.

Total packages: 1