Skip to main content

​
Chunking

Defined in the chunking module.
A unified class for text chunking with various chunking strategies.

​
Constructor

​
chunker_type
str
default:"'token'"
No description available.
​
chunk_size
int
default:"512"
No description available.
​
chunk_overlap
int
default:"128"
No description available.
​
tokenizer_or_token_counter
str
default:"'gpt2'"
No description available.
​
embedding_model
Optional
No description available.

​
Methods

SUPPORTED_CHUNKERS()

Lazy load chunker classes.

embedding_model()

Lazy load the embedding model.

chunker()

Lazy load the chunker instance.

chunk()

Chunk text using the configured chunking strategy.