Chunking Strategies

PraisonAI integrates chonkie for high-performance document chunking.

Quick Start

from praisonaiagents import Agent

# Default chunking (token-based)
agent = Agent(
    instructions="Answer questions from documents.",
    knowledge=["research.pdf", "docs/"]
)

response = agent.start("What are the key findings?")

Available Strategies

Strategy	Best For	Speed
`token`	Fixed-size chunks	⚡ Fastest
`sentence`	Natural boundaries	⚡ Fast
`recursive`	Structured docs (markdown)	⚡ Fast
`semantic`	Topic segmentation	🔄 Medium
`sdpm`	Research papers	🔄 Medium
`late`	Best embeddings	🔄 Medium

Token Chunking

Fixed-size token chunks. Fast and predictable.

Sentence Chunking

Split at sentence boundaries. Natural flow.

Recursive Chunking

Hierarchical splitting. Great for markdown.

Semantic Chunking

Similarity-based splits. Topic coherence.

Chunker Configuration

All Parameters

Parameter	Type	Default	Applies To
`type`	str	`"token"`	All
`chunk_size`	int	512	All
`chunk_overlap`	int	128	token, sentence
`tokenizer_or_token_counter`	str	`"gpt2"`	token, sentence, recursive
`embedding_model`	str	auto	semantic, sdpm, late

Strategy Examples

Token
Sentence
Recursive
Semantic

agent = Agent(
    instructions="Process documents.",
    knowledge={
        "sources": ["docs/"],
        "chunker": {
            "type": "token",
            "chunk_size": 256,
            "chunk_overlap": 50
        }
    }
)

agent = Agent(
    instructions="Process articles.",
    knowledge={
        "sources": ["articles/"],
        "chunker": {
            "type": "sentence",
            "chunk_size": 512
        }
    }
)

agent = Agent(
    instructions="Process markdown docs.",
    knowledge={
        "sources": ["README.md", "docs/"],
        "chunker": {
            "type": "recursive",
            "chunk_size": 512
        }
    }
)

agent = Agent(
    instructions="Process research papers.",
    knowledge={
        "sources": ["papers/"],
        "chunker": {
            "type": "semantic",
            "chunk_size": 512,
            "embedding_model": "all-MiniLM-L6-v2"
        }
    }
)

Choosing a Strategy

Installation

pip install "praisonaiagents[knowledge]"

This installs the chonkie library automatically.

Knowledge Base

Configure knowledge sources and retrieval

RAG Agents

Build retrieval-augmented agents

Getting Started

Core Concepts

Guides

Features

Models

Databases

Observability

Memory

Knowledge

RAG

Persistence

Tools

Other Features

Developers

Configuration

Best Practices

Getting Started (No Code)

Chunking Strategies

Quick Start

Available Strategies

Token Chunking

Sentence Chunking

Recursive Chunking

Semantic Chunking

Chunker Configuration

All Parameters

Strategy Examples

Choosing a Strategy

Installation

Knowledge Base

RAG Agents

Getting Started

Core Concepts

Guides

Features

Models

Databases

Observability

Memory

Knowledge

RAG

Persistence

Tools

Other Features

Developers

Configuration

Best Practices

Getting Started (No Code)

Documentation Index

​Quick Start

​Available Strategies

Token Chunking

Sentence Chunking

Recursive Chunking

Semantic Chunking

​Chunker Configuration

​All Parameters

​Strategy Examples

​Choosing a Strategy

​Installation

​Related

Knowledge Base

RAG Agents

Quick Start

Available Strategies

Chunker Configuration

All Parameters

Strategy Examples

Choosing a Strategy

Installation

Related