evaluate
Method
This is a method of the Evaluator class in the eval module.Run the evaluation.
Signature
Returns
The result of the operation.
Guide
Reference
- description
- Get Agent Card
- Get Status
- new
- prefix
- version
- new
- skill
- version
- With Streaming
- description
- Get Status
- new
- prefix
- Run Error
- Run Finished
- Run Started
- Text Delta
- Evaluate Simple
- new
- actual
- build
- expected
- input
- threshold
- failure
- success
- Add Tool
- chat
- Clear Memory
- history
- ID
- instructions
- model
- name
- new
- run
- simple
- start
- Tool Count
- new
- name
- start
- stop
- version
- API Key
- Base URL
- build
- instructions
- LLM
- Max Iterations
- Max Tokens
- memory
- Memory Config
- model
- name
- new
- stream
- temperature
- tool
- tools
- verbose
- agent
- new
- run
- step
- custom
- new
- Agent ID
- API Key
- new
- No Telemetry
- config
- Get Config
- heartbeat
- register
- Report Metrics
- On Agent Created
- achat
- chat
- name
- Is Empty
- Is Verbose
- len
- new
- run
- start
- agent
- Agent Arc
- build
- new
- process
- verbose
- Average Duration
- new
- Record Error
- Record Success
- Success Rate
- Request Approval
- On Display
- name
- validate
- On Event
- model
- name
- new
- speech
- transcribe
- build
- config
- model
- name
- speed
- verbose
- voice
- language
- new
- Response Format
- speed
- timeout
- voice
- Base URL
- Is Available
- Mark Error
- Mark Rate Limited
- metadata
- model
- new
- priority
- Rate Limit Rpm
- Rate Limit Tpm
- reset
- agents
- generate
- new
- Add Source
- Add Sources
- index
- Is Indexed
- new
- query
- sources
- Allow Tool
- Block Tool
- Full Auto
- level
- Max Actions
- new
- No Approval
- Add Keyword
- Case Sensitive
- new
- Channel Type
- metadata
- name
- new
- Command Prefix
- extra
- new
- Polling Interval
- webhooks
- attachment
- channel
- command
- Command Args
- Is Command
- Message Type
- metadata
- new
- Reply To
- sender
- text
- Text Content
- Thread ID
- Bot User
- Delete Message
- Edit Message
- Get Agent
- Get Channel
- Get User
- Is Running
- name
- On Command
- On Message
- platform
- Send Message
- Send Typing
- Set Agent
- start
- stop
- action
- attachment
- Reply To
- text
- Display Name
- Is Bot
- metadata
- new
- username
- new
- With Ratios
- tokens
- disabled
- new
- ttl
- With Prompt Caching
- chunk
- new
- metadata
- new
- page
- score
- execute
- generate
- name
- new
- review
- build
- config
- instructions
- model
- name
- Allowed Languages
- new
- sandbox
- timeout
- new
- With Output
- evaluate
- add
- clear
- Get Context
- Is Empty
- len
- new
- allocate
- Allocate Custom
- available
- new
- With Limits
- new
- Max Tokens
- model
- new
- Output Reserve Pct
- strategy
- With Monitoring
- agent
- Duration Ms
- error
- input
- metadata
- new
- output
- tool
- add
- Is Over Budget
- new
- remaining
- utilization
- new
- Add Segment
- Allocate Budget
- Default For Model
- Is Over Budget
- new
- remaining
- reset
- Track Messages
- Track System
- Track Tools
- utilization
- content
- new
- priority
- Agent End
- Agent Start
- clear
- emit
- events
- new
- noop
- Tool End
- Tool Start
- With List Sink
- clear
- events
- new
- clear
- events
- record
- add
- clear
- Is Empty
- len
- messages
- new
- evaluate
- new
- build
- criterion
- threshold
- new
- Weighted Score
- With Feedback
- With Weight
- delete
- get
- store
- name
- new
- research
- build
- config
- instructions
- model
- name
- Include Citations
- Max Depth
- Max Sources
- new
- timeout
- Get All Sources
- Get Citation Text
- new
- Default Targets
- new
- when
- On Display
- agent
- args
- content
- data
- error
- meta
- new
- result
- tool
- filename
- metadata
- new
- source
- embed
- Embed Batch
- Find Most Similar
- new
- similarity
- simple
- API Base
- API Key
- build
- config
- model
- name
- new
- verbose
- API Base
- API Key
- dimensions
- Encoding Format
- new
- timeout
- dimension
- first
- Is Empty
- len
- agent
- config
- handoff
- IO
- LLM
- memory
- tool
- With Context
- workflow
- new
- With Context
- As Percentage
- Is Passing
- new
- With Confidence
- With Reasoning
- evaluate
- Agent End
- Agent Error
- Agent Start
- Correlation ID
- custom
- data
- Handoff End
- Handoff Start
- LLM Request
- LLM Response
- new
- Tool End
- Tool Error
- Tool Start
- Workflow End
- Workflow Start
- Clear History
- Events By Source
- Events By Type
- history
- new
- publish
- Subscription Count
- Total Subscriptions
- unsubscribe
- With History
- Max Iterations
- new
- No Stream
- timeout
- get
- Expansion Ratio
- new
- With Metadata
- all
- From Str
- new
- Cooldown On Error
- Cooldown On Rate Limit
- Exponential Backoff
- Max Retries
- Max Retry Delay
- new
- Retry Delay
- Add Profile
- Default Config
- Get Next Profile
- Get Profile
- Get Profile Mut
- Get Retry Delay
- Is Empty
- len
- List Profiles
- Mark Failure
- Mark Success
- new
- On Failover
- Remove Profile
- Reset All
- status
- add
- As String
- clear
- get
- Is Empty
- new
- size
- Cache Enabled
- Cache Ttl
- Max Parallel
- Max Turns
- model
- new
- timeout
- Workspace Path
- Add File
- Is Empty
- new
- To Context String
- Total Files
- Line Range
- new
- Relevance Score
- new
- Max Messages
- new
- With Dir
- Complete Step
- Is Complete
- new
- progress
- Set Status
- Average Duration
- Calls Per Second
- new
- record
- Client ID
- close
- Connected At
- Is Connected
- receive
- send
- auth
- host
- Max Connections
- new
- port
- Session Timeout
- tls
- data
- new
- source
- target
- metadata
- new
- Reply To
- text
- Text Content
- broadcast
- Close Session
- Create Session
- emit
- Get Agent
- Get Session
- health
- host
- Is Running
- List Agents
- List Sessions
- port
- Register Agent
- start
- stop
- Unregister Agent
- Add Message
- Agent ID
- Client ID
- close
- Created At
- Get Messages
- Get State
- Is Active
- Last Activity
- Session ID
- Set State
- name
- validate
- add
- config
- Is Empty
- len
- new
- validate
- enabled
- Error Template
- Fallback Response
- LLM Validator
- Log Results
- Max Retries
- new
- On Fail
- On Failure
- policy
- fail
- failure
- From Tuple
- Get Result Or
- Is Failure
- Is Success
- pass
- success
- With Metadata
- With Modification
- Check Safety
- config
- Get Tool Description
- Get Tool Name
- new
- Prepare Context
- Tool Description
- Tool Name
- clear
- contains
- depth
- get
- new
- pop
- push
- Async Mode
- Context Policy
- Detect Cycles
- Include History
- Max Concurrent
- Max Context Messages
- Max Context Tokens
- Max Depth
- message
- metadata
- new
- Preserve System
- Timeout Seconds
- To Map
- Allow All
- Allow Only
- deny
- Deny All
- Is Allowed
- Keep Last N
- new
- Remove All Tools
- Remove System Messages
- context
- messages
- new
- Source Agent
- failure
- success
- With Depth
- With Duration
- With Source
- With Target
- execute
- matches
- new
- With Matcher
- With Name
- new
- With Agent
- With Error
- With Extra
- With Message
- With Tool
- Before Agent
- Add Definition
- Add Hook
- Add Hook With Matcher
- execute
- Execute Async
- Has Hooks
- Hook Count
- new
- Remove Hook
- allow
- Allow With Reason
- ask
- block
- deny
- Is Allowed
- Is Denied
- suppress
- With Context
- With Modified Input
- After Agent
- After Tool
- Before Agent
- Before Tool
- new
- On Error
- enabled
- new
- expr
- generate
- name
- new
- build
- config
- model
- name
- verbose
- new
- quality
- size
- style
- new
- new
- new
- new
- judge
- new
- With Config
- With Threshold
- new
- add
- Add Document
- chunk
- clear
- delete
- get
- Is Empty
- len
- new
- search
- build
- chunking
- config
- Retrieval Strategy
- Agent ID
- Chunk Size
- chunking
- Default Limit
- embedder
- Enable Reranking
- new
- Rerank Model
- Retrieval K
- Retrieval Strategy
- source
- sources
- User ID
- With Rerank
- add
- delete
- Delete All
- get
- Get All
- search
- update
- Block On Failure
- new
- With Model
- With Prompt
- Before LLM Call
- max
- min
- new
- content
- Line Count
- new
- Relevance Score
- API Key
- Base URL
- Max Tokens
- new
- temperature
- assistant
- system
- user
- chat
- Chat With Tools
- model
- chat
- Chat Stream
- model
- Call Tool
- connect
- disconnect
- Get Prompt
- Is Connected
- List Prompts
- List Resources
- List Tools
- new
- Read Resource
- status
- build
- config
- HTTP
- name
- security
- server
- tool
- websocket
- new
- debug
- new
- security
- transport
- image
- text
- new
- Is Running
- new
- Register Resource
- Register Tool
- resources
- start
- stop
- tools
- Input Schema
- new
- clear
- config
- Default Memory
- history
- In Memory
- new
- search
- store
- Clear Short Term
- Get Short Term
- Search Short Term
- Store Long Term
- Store Short Term
- Max Messages
- new
- provider
- With Long Term
- new
- new
- option
- clear
- history
- search
- store
- assistant
- system
- tool
- user
- cleanup
- disable
- enable
- flush
- Get Property
- Is Enabled
- new
- Session ID
- Set Property
- Set User ID
- Track Agent Complete
- Track Agent Start
- Track Error
- Track Event
- Track LLM Call
- Track Tool Execution
- User ID
- With Session ID
- Add Response
- Add Tool Calls
- new
- With Response
- Add Shared
- Agent Ids
- get
- Get Mut
- new
- Register Agent
- extract
- name
- new
- build
- config
- model
- name
- Image Limit
- new
- pages
- flush
- record
- Default Model
- new
- file
- new
- silent
- verbose
- Must Match
- Must Not Match
- new
- evaluate
- new
- build
- Max Duration
- Max Ttft
- threshold
- new
- With Tokens
- With Tokens Per Second
- With Ttft
- All API Stats
- All Function Stats
- clear
- disable
- elapsed
- enable
- Get API Stats
- Get Function Stats
- Get Report
- Is Enabled
- new
- Slowest Apis
- Slowest Functions
- Track API
- Track Function
- Add Step
- Completed Steps
- description
- Get Step
- Get Step Mut
- Has Failed
- Is Complete
- new
- Next Step
- progress
- Step Count
- complete
- Depends On
- estimated
- fail
- Is Ready
- new
- skip
- start
- count
- delete
- list
- load
- Load Mut
- new
- persist
- restore
- save
- With Path
- Add Step
- Complete Step
- Create Plan
- current
- Fail Step
- Is Complete
- new
- progress
- steps
- Auto Approve
- enabled
- LLM
- new
- Read Only
- With Reasoning
- info
- name
- all
- description
- hook
- hooks
- new
- Plugin Type
- version
- disable
- enable
- Enabled Count
- Execute Hook
- get
- Get Hook Plugins
- Is Empty
- Is Enabled
- len
- List Plugins
- new
- register
- initialize
- metadata
- disable
- enable
- get
- has
- Is Empty
- Is Enabled
- len
- list
- List Enabled
- new
- register
- Get List
- Is Enabled
- Add Rule
- check
- Check And Redact
- clear
- disable
- enable
- Get Rule
- Get Rule Mut
- List Rules
- new
- Remove Rule
- Rule Count
- block
- pass
- redact
- warn
- action
- apply
- description
- disable
- enable
- keyword
- matches
- new
- pattern
- priority
- replacement
- Detect Strategy
- Expand Sync
- model
- name
- new
- build
- instructions
- Max Tokens
- model
- name
- new
- temperature
- verbose
- Detect Strategy
- Expand Abbreviations
- model
- name
- new
- Rewrite Sync
- abbreviation
- abbreviations
- build
- instructions
- Max Queries
- Max Tokens
- model
- name
- new
- temperature
- verbose
- Add Source
- Build Context
- new
- query
- Truncate Context
- build
- config
- model
- source
- Citations Mode
- compress
- Max Context Tokens
- new
- rerank
- Score Threshold
- strategy
- Top K
- Add Citation
- Citation Count
- new
- name
- new
- Send Text
- build
- config
- model
- name
- voice
- instructions
- Max Response Output Tokens
- modalities
- new
- temperature
- Turn Detection
- voice
- new
- enabled
- LLM
- Max Iterations
- Min Iterations
- new
- prompt
- new
- With Improvement
- evaluate
- new
- build
- Expect Tool
- threshold
- rerank
- As Bool
- As List
- As Str
- Is None
- Is Some
- Cpu Percent
- Disk Write Mb
- generous
- Max Open Files
- Max Processes
- Memory Mb
- minimal
- Network Enabled
- new
- standard
- Timeout Seconds
- context
- enable
- new
- RAG
- source
- Add Chunk
- new
- Top Chunks
- retrieve
- strategy
- get
- All Queries
- new
- Primary Query
- With Hypothetical Document
- With Metadata
- With Step Back Question
- With Sub Queries
- all
- From Str
- Get Target
- run
- Auto Cleanup
- docker
- env
- limits
- new
- subprocess
- Working Dir
- cleanup
- execute
- Execute File
- Get Status
- Is Available
- List Files
- Read File
- reset
- Run Command
- Sandbox Type
- start
- stop
- Write File
- complete
- fail
- new
- output
- start
- success
- timeout
- Is Empty
- len
- new
- new
- Allow Fs
- Allow Network
- Allowed Host
- Allowed Path
- permissive
- restrictive
- permissive
- restrictive
- Add Assistant Message
- Add Message
- Add User Message
- clear
- delete
- exists
- Get History
- ID
- load
- Message Count
- messages
- new
- save
- Set Agent Name
- Set User ID
- With Store
- Add Message
- clear
- Get Chat History
- new
- assistant
- new
- system
- To Message
- user
- With Metadata
- delete
- exists
- list
- load
- save
- new
- Load Full
- Load Metadata
- Metadata Count
- new
- Skill Count
- discover
- Get Default Dirs
- List Skills
- load
- new
- Skill Count
- To Prompt
- From Properties
- Get Allowed Tools
- new
- validate
- Auto Discover
- dir
- new
- path
- Add Event
- end
- Is Ended
- new
- Set Attribute
- Set Error
- With Parent
- new
- With Attribute
- failure
- success
- On Event
- Event Count
- Get Content
- Get Error
- Has Error
- Is Complete
- new
- process
- Agent ID
- content
- Delta Text
- error
- Error Event
- First Token
- metadata
- new
- reasoning
- Request Start
- Run ID
- Session ID
- Stream End
- Tool Call
- Add Callback
- Callback Count
- emit
- new
- Increment Tokens
- Mark First Token
- Mark Last Token
- Mark Request Start
- Mark Stream End
- new
- Stream Duration Ms
- Tokens Per Second
- Total Time Ms
- Ttft Ms
- Update From Event
- Can Retry
- Display Name
- ID
- Increment Retry
- Is Completed
- Is Failed
- new
- Result Str
- Set Failed
- Set Result
- Substitute Variables
- build
- decision
- Depends On
- Expected Output
- Is Start
- Loop Task
- Max Retries
- name
- new
- Next Task
- On Error
- Output File
- Output Variable
- Task Type
- variable
- As Str
- new
- Parse Json
- With Agent
- With Duration
- With Json
- With Metadata
- With Tokens
- clear
- disable
- enable
- Event Count
- events
- Events By Type
- new
- record
- With Max Events
- complete
- Elapsed Ms
- new
- property
- new
- With Data
- With Duration
- new
- No System Prompt
- prompt
- response
- system
- From Level
- Get Tokens For Complexity
- high
- low
- maximum
- medium
- minimal
- new
- To Map
- adaptive
- build
- Complexity Multiplier
- level
- Max Time Seconds
- Max Tokens
- Min Tokens
- enabled
- With Level
- Average Time Per Session
- Average Tokens Per Session
- Average Utilization
- clear
- End Session
- Get Summary
- new
- Over Budget Count
- Session Count
- Start Session
- Is Over Budget
- Is Over Time
- Time Remaining
- To Map
- Token Utilization
- Tokens Remaining
- complete
- due
- Is Overdue
- new
- priority
- tag
- add
- By Priority
- By Tag
- completed
- get
- Get Mut
- Is Empty
- len
- new
- overdue
- pending
- progress
- remove
- Available Context
- Can Add Context
- new
- definition
- description
- execute
- name
- Parameters Schema
- arguments
- name
- new
- arguments
- ID
- new
- Is Correct
- new
- Execute Tool
- Get Tools
- description
- execute
- name
- Parameters Schema
- definitions
- execute
- get
- has
- Is Empty
- len
- list
- new
- register
- failure
- success
- Add Event
- Current Span ID
- elapsed
- End Span
- Get Span
- Get Span Mut
- new
- Set Attribute
- Span Count
- spans
- Start Span
- new
- data
- new
- Span ID
- Trace ID
- export
- close
- flush
- write
- Add Exporter
- End Span
- End Trace
- new
- Start Span
- Start Trace
- Trace Count
- env
- header
- HTTP
- stdio
- timeout
- websocket
- new
- add
- clear
- delete
- get
- Get All
- Is Empty
- len
- search
- generate
- name
- new
- build
- config
- model
- name
- new
- seconds
- size
- analyze
- compare
- describe
- name
- new
- build
- config
- model
- name
- detail
- Max Tokens
- new
- Max Results
- new
- No Fetch
- No Search
- provider
- new
- Add Result
- get
- Last Result
- new
- set
- Add Approval Callback
- Add Display Callback
- Add Error Log
- aembed
- aembedding
- aembeddings
- Apply Config Defaults
- Async Display Callbacks
- Build Context
- Callback Count
- Clean Display Content
- Clean Triple Backticks
- Clean Whitespace
- Cleanup Telemetry Resources
- Clear All Callbacks
- Clear Display Callbacks
- Clear Error Logs
- Cosine Similarity
- Create Context Agent
- Create Context Agent With Config
- Credit Card Rule
- Deduplicate Chunks
- Detect Memory Backend
- Detect URL Scheme
- Disable Performance Mode
- Disable Plugins
- Disable Telemetry
- Discover And Load Plugins
- Discover Plugins
- Display Error
- Display Generating
- Display Instruction
- Display Interaction
- Display Reasoning Steps
- Display Self Reflection
- Display Tool Call
- Display Working Status
- Email Rule
- embed
- embedding
- embeddings
- Enable Performance Mode
- Enable Plugins
- Enable Telemetry
- Ensure Plugin Dir
- Error Logs
- Estimate Messages Tokens
- Estimate Tokens
- Estimate Tokens Heuristic
- Estimate Tool Schema Tokens
- Evaluate Condition
- Execute Async Callback
- Execute Async Callbacks
- Execute Callback
- Execute Callbacks
- Execute Sync Callbacks
- Extract Json
- Format Skill For Prompt
- Generate Skills Xml
- Get Collector
- Get Config
- Get Config Path
- Get Default
- Get Default Plugin Dirs
- Get Defaults Config
- Get Dimensions
- Get Enabled Plugins
- Get Error Logs
- Get Event Bus
- Get Model Context Window
- Get Model Limit
- Get Monitor
- Get Output Reserve
- Get Performance Report
- Get Plugin Manager
- Get Plugin Template
- Get Plugins Config
- Get Telemetry
- handoff
- Handoff Filters
- Has Callbacks
- Is Numeric String
- Is Path Like
- Is Performance Mode
- Is Plugin Enabled
- Is Plugins Enabled
- Is Policy String
- Is Read Only Tool
- Is Research Tool
- Is Restricted Tool
- Is Telemetry Enabled
- List Plugins
- Load Plugin
- Log Error
- Loop Step
- Make Array Error
- Make Preset Error
- parallel
- Parse Plugin Header
- Parse Plugin Header From File
- Parse Policy String
- Phone Rule
- Profanity Rule
- Prompt With Handoff Instructions
- publish
- Record Event
- Register Approval Callback
- Register Async Display Callback
- Register Display Callback
- Remove Approval Callback
- Remove Display Callback
- repeat
- Request Approval
- resolve
- Resolve Autonomy
- Resolve Autonomy Preset
- Resolve Caching
- Resolve Caching Preset
- Resolve Context
- Resolve Context Preset
- Resolve Execution
- Resolve Execution Preset
- Resolve Guardrail Policies
- Resolve Guardrail Preset
- Resolve Guardrails
- Resolve Hooks
- Resolve Knowledge
- Resolve Knowledge Preset
- Resolve Memory
- Resolve Memory Preset
- Resolve Multi Agent Execution Preset
- Resolve Multi Agent Output Preset
- Resolve Output
- Resolve Output Preset
- Resolve Planning
- Resolve Planning Preset
- Resolve Reflection
- Resolve Reflection Preset
- Resolve Routing
- Resolve Skills
- Resolve Web
- Resolve Web Preset
- Resolve Workflow Step Execution Preset
- route
- run
- Ssn Rule
- subscribe
- Subscribe All
- Suggest Similar
- Sync Display Callbacks
- tool
- Trace Context
- Track API
- Track Function
- Track Workflow
- Truncate Context
- Validate Config
- Validate Keys
- when
Functions
evaluate • Rust AI Agent SDK
evaluate: Run the evaluation.
⌘I