Tokenizer · emploidai Marketplace
emploidai Marketplace
Add-onsAppletsPlugins
Search tools, teams, and capabilitiesPublish
MarketplacePluginsTokenizer
Plugin
Limited listing

Tokenizer

by jingfelix · v0.0.1

Tokenizing text using various tokenizers from transformers.AutoTokenizer. Returns the token count.

366 installsUpdated Jun 27, 2025
Publisher information is incomplete

This community listing does not yet include every recommended support, privacy, pricing, and permission disclosure. Review the available package permissions before installing.

Capabilities

Tools

Available inside your emploidai workspace after installation.

Data sources

Available inside your emploidai workspace after installation.

Category

tool

Version

0.0.1jingfelix

Requirements

Maximum memory 256MB

Pricing

Not disclosed by publisher

Security & access

Review before installing

CompatibleRequires emploidai 1.0.0+

Permissions

No permissions were declared by the publisher.

Dependencies

No additional dependencies

Resources

Privacy policy
emploidai Marketplace

Discover capabilities. Review access. Install inside your workspace.

DocumentationSecuritySupportPrivacyTerms

dify-plugin-tokenize

Author: jingfelix

Version: 0.0.1

Type: tool

Description

Tokenizing text using various tokenizers from transformers.AutoTokenizer. Returns the token count.

Usage

[!NOTE] Access to huggingface.co is required to download the tokenizer.

[!WARNING] Due to the plugin execution environment limitations of cloud.dify.ai, this plugin is only available for use in self-hosted environments.

You can choose the tokenizer by specifying the tokenizer parameter. The default is gpt2. This plugin utilizes the transformers library to tokenize text. So please check https://huggingface.co/docs/transformers/v4.52.3/en/model_doc/auto#transformers.AutoTokenizer if the tokenizer you want to use is supported.

PRIVACY

Please refer to the privacy policy for more information on how your data is handled.

Last updated: 2025-06-26