Skip to main content

​
estimate_tokens

Method
This is a method of the ContextManager class in the manager module.
Estimate tokens with optional validation.

​
Signature

def estimate_tokens(text: str, validate: bool) -> Tuple[int, Optional[EstimationMetrics]]

​
Parameters

​
text
str
required
Text to estimate
​
validate
bool
default:"False"
Whether to validate against accurate count

​
Returns

​
Returns
Tuple[int, Optional[EstimationMetrics]]
Tuple of (token_count, metrics)

​
Uses

  • hexdigest
  • hashlib.md5
  • text.encode
  • estimate_tokens_heuristic
  • estimate_tokens_accurate
  • EstimationMetrics
  • warning
  • logging.getLogger

​
Used By

​
Source

View on GitHub

praisonaiagents/context/manager.py at line 707