Tokenizer
Service for model-specific token counting and prompt truncation. Tokenization depends on the target provider, model, and encoding rules, so this module leaves the actual tokenization function to the service implementation.
The Tokenizer service can count tokens for raw prompt input and shorten a prompt to a token limit by keeping the newest messages that fit. This module defines the service tag, the service interface, and a make constructor that builds a full tokenizer service from a token-counting function.
Constructors
Models
Tokenizer service interface providing text tokenization and truncation operations.
Details
This interface defines the core operations for converting text to tokens and managing content length within token limits for AI model compatibility.
Signature
interface Service {
readonly tokenize: (input: RawInput) => Effect<Array<number>, AiError>;
readonly truncate: (input: RawInput, tokens: number) => Effect<Prompt, AiError>;
}Services
Service tag for model tokenization services.
When to use
Use to access or provide model-specific token counting and prompt truncation operations.
Details
This tag provides access to tokenization functionality throughout your application, enabling token counting and prompt truncation capabilities.
Signature
declare class Tokenizer extends Shape<"effect/ai/Tokenizer", Service, this> {
constructor(_: never);
}
Creates a Tokenizer service implementation from tokenization options.
Details
This function constructs a complete Tokenizer service by providing a tokenization function. The service handles both tokenization and truncation operations using the provided tokenizer.