> ## Documentation Index > Fetch the complete documentation index at: https://docs.praison.ai/llms.txt > Use this file to discover all available pages before exploring further. # Benchmarks CLI > Run performance benchmarks from the command line # Benchmarks CLI Run performance benchmarks to compare AI SDK vs Native backends. ## Commands ### Run All Benchmarks ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} praisonai-ts benchmark run ``` Output: ``` ✓ Running All Benchmarks Iterations: 5 Benchmark Results ──────────────────────────────────────────────────────────────────────────────── Name Mean Min Max P95 Unit ──────────────────────────────────────────────────────────────────────────────── Full Import 58.30 0.01 174.86 174.86 ms Memory (Agent creation) 2.45 1.89 3.21 3.21 MB Backend Resolution 4.23 3.12 5.67 5.67 ms ──────────────────────────────────────────────────────────────────────────────── ℹ Use --real flag to run benchmarks with real API calls ``` ### Import Time Benchmark ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} praisonai-ts benchmark import --iterations 10 ``` Output: ``` ✓ Import Time Benchmark Iterations: 10 Benchmark Results ──────────────────────────────────────────────────────────────────────────────── Name Mean Min Max P95 Unit ──────────────────────────────────────────────────────────────────────────────── Core Import (Agent) 27.53 0.33 81.86 81.86 ms AI SDK Import 34.04 13.81 69.46 69.46 ms Full Import 58.30 0.01 174.86 174.86 ms ──────────────────────────────────────────────────────────────────────────────── AI SDK import overhead: 34.04ms Core import (no AI SDK): 27.53ms ``` ### Memory Benchmark ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} praisonai-ts benchmark memory --iterations 5 ``` ### Latency Benchmark ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} # Mock mode (no API calls) praisonai-ts benchmark latency # Real API calls praisonai-ts benchmark latency --real ``` ### Streaming Benchmark ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} # Requires --real flag praisonai-ts benchmark streaming --real --iterations 3 ``` ### Embedding Benchmark ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} # Requires --real flag praisonai-ts benchmark embedding --real --iterations 3 ``` ## Options | Option | Description | | -------------- | -------------------------------------- | | `--iterations` | Number of iterations (default: 5) | | `--backend` | Backend to test (ai-sdk, native, both) | | `--real` | Use real API calls (requires keys) | | `--json` | Output as JSON | | `--verbose` | Verbose output | ## Examples ### Quick Performance Check ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} $ praisonai-ts benchmark import --iterations 3 ✓ Import Time Benchmark Iterations: 3 AI SDK import overhead: 34.04ms Core import (no AI SDK): 27.53ms ``` ### Full Benchmark Suite ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} $ praisonai-ts benchmark run --real --iterations 5 ✓ Running All Benchmarks Iterations: 5 Benchmark Results ──────────────────────────────────────────────────────────────────────────────── Name Mean Min Max P95 Unit ──────────────────────────────────────────────────────────────────────────────── Full Import 58.30 45.21 74.86 74.86 ms Memory (Agent creation) 2.45 1.89 3.21 3.21 MB First-Call Latency (Real) 523.45 412.33 634.56 634.56 ms ──────────────────────────────────────────────────────────────────────────────── ``` ### JSON Output ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} $ praisonai-ts benchmark import --json ``` ```json theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} { "success": true, "data": { "results": [ { "name": "Core Import (Agent)", "iterations": 5, "mean": 27.53, "min": 0.33, "max": 81.86, "p95": 81.86, "stdDev": 32.45, "unit": "ms" }, { "name": "AI SDK Import", "iterations": 5, "mean": 34.04, "min": 13.81, "max": 69.46, "p95": 69.46, "stdDev": 21.23, "unit": "ms" } ] } } ``` ### Compare Backends ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} # Test with AI SDK PRAISONAI_BACKEND=ai-sdk praisonai-ts benchmark latency --real # Test with native PRAISONAI_BACKEND=native praisonai-ts benchmark latency --real ``` ## CI/CD Integration ### GitHub Actions ```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} name: Performance Tests on: [push, pull_request] jobs: benchmark: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - uses: actions/setup-node@v3 with: node-version: '18' - run: npm install - run: npm run build - name: Run Benchmarks run: npx praisonai-ts benchmark run --json > benchmark.json - name: Check Regression run: | import_time=$(jq '.data.results[0].mean' benchmark.json) if (( $(echo "$import_time > 100" | bc -l) )); then echo "Import time regression: ${import_time}ms > 100ms" exit 1 fi ``` ### Script Integration ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} #!/bin/bash # benchmark-check.sh # Run benchmarks praisonai-ts benchmark import --json > /tmp/bench.json # Extract metrics import_time=$(jq '.data.results[0].mean' /tmp/bench.json) ai_sdk_overhead=$(jq '.data.results[1].mean' /tmp/bench.json) echo "Import time: ${import_time}ms" echo "AI SDK overhead: ${ai_sdk_overhead}ms" # Check thresholds if (( $(echo "$import_time > 100" | bc -l) )); then echo "FAIL: Import time exceeds 100ms" exit 1 fi echo "PASS: All benchmarks within limits" ``` ## Interpreting Results ### Import Time * **\< 50ms**: Excellent * **50-100ms**: Good * **> 100ms**: May need optimization ### AI SDK Overhead * **\~35ms**: Expected overhead for AI SDK import * **0ms**: AI SDK not loaded (lazy loading working) ### Memory * **\< 5MB**: Normal for Agent creation * **> 10MB**: May indicate memory leak ### Latency * **\< 500ms**: Good first-call latency * **> 1000ms**: Network or API issues ## Troubleshooting ### High Import Time ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} # Check what's being loaded DEBUG=* praisonai-ts benchmark import ``` ### Memory Issues ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} # Run with memory profiling node --expose-gc ./dist/cli/index.js benchmark memory ``` ### Inconsistent Results ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} # Increase iterations for more stable results praisonai-ts benchmark run --iterations 20 ```