Documentation Index
Fetch the complete documentation index at: https://docs.praison.ai/llms.txt
Use this file to discover all available pages before exploring further.
PraisonAI Testing Guide
This guide explains how to run tests, understand test tiers, and work with provider-specific tests in PraisonAI.
Quick Start
# Run smoke tests (fastest, ~2 min)
praisonai test --tier smoke
# Run main tests (default, ~5 min)
praisonai test
# Run with parallelization
praisonai test --parallel auto
# Run OpenAI live tests (requires API key)
praisonai test --provider openai --live
Test Tiers
PraisonAI uses a tiered testing approach to balance speed and coverage:
| Tier | Target Runtime | Description |
|---|
smoke | ≤ 2 min | Fast unit tests, no network, no slow tests |
main | ≤ 5 min | Unit + integration, OpenAI gated only |
extended | ≤ 15 min | All providers, gated by API keys |
nightly | ≤ 30 min | Full matrix + stress tests |
Running Specific Tiers
# Smoke tier - fastest, for quick validation
praisonai test --tier smoke
# Main tier - default for development
praisonai test --tier main
# Extended tier - all providers
praisonai test --tier extended --live
# Nightly tier - comprehensive
praisonai test --tier nightly --live
Provider Tests
Tests that require specific LLM providers are automatically detected and gated:
| Provider | Environment Variable | Marker |
|---|
| OpenAI | OPENAI_API_KEY | provider_openai |
| Anthropic | ANTHROPIC_API_KEY | provider_anthropic |
| Google/Gemini | GOOGLE_API_KEY | provider_google |
| Ollama | (local service) | provider_ollama |
| Grok/xAI | XAI_API_KEY | provider_grok_xai |
| Groq | GROQ_API_KEY | provider_groq |
| Cohere | COHERE_API_KEY | provider_cohere |
Running Provider-Specific Tests
# Run only OpenAI tests
praisonai test --provider openai --live
# Run only Anthropic tests
praisonai test --provider anthropic --live
# Run all provider tests
praisonai test --provider all --live
Environment Variables
| Variable | Default | Description |
|---|
PRAISONAI_TEST_TIER | main | Test tier to run |
PRAISONAI_ALLOW_NETWORK | 0 | Allow network calls in tests |
PRAISONAI_LIVE_TESTS | 0 | Enable live API tests |
PRAISONAI_TEST_PROVIDERS | openai | Comma-separated provider list |
PRAISONAI_LOCAL_SERVICES | 0 | Enable local service tests |
OPENAI_API_KEY | 'test-key' | Placeholder injected only if unset/empty for non-real tests |
ANTHROPIC_API_KEY | 'test-key' | Placeholder injected only if unset/empty for non-real tests |
GOOGLE_API_KEY | 'test-key' | Placeholder injected only if unset/empty for non-real tests |
XAI_API_KEY | 'test-key' | Placeholder injected only if unset/empty for non-real tests |
GROQ_API_KEY | 'test-key' | Placeholder injected only if unset/empty for non-real tests |
COHERE_API_KEY | 'test-key' | Placeholder injected only if unset/empty for non-real tests |
Placeholder API Keys in Mock Tests
The autouse setup_test_environment fixture in src/praisonai/tests/conftest.py injects a 'test-key' placeholder for OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_API_KEY, XAI_API_KEY, GROQ_API_KEY, and COHERE_API_KEY so unit tests can construct LLM clients without real credentials.
Since May 2026 (PR #1663), the placeholder is only injected when the variable is unset or empty. If you already exported a real key, it is left untouched.
Implication: if you export a real key and run mock unit tests, any test that isn’t fully mocked will make real, billable API calls. To force the placeholder path, unset the keys first (or run in a clean shell). The fixture skips placeholder injection entirely for tests marked real, network, e2e, or provider_*, and for tests under tests/integration/, tests/live/, or tests/e2e/.
Provider API keys in unit/mock tests
The setup_test_environment autouse fixture preserves any provider API key already present in the shell environment.
Placeholder "test-key" values are injected only for keys that are unset, so SDK imports and guardrails never crash on missing keys. On teardown, originally-set keys keep their values; keys that were missing and got placeholders are removed (not left behind as "test-key").
Tests marked @pytest.mark.real, @pytest.mark.network, @pytest.mark.e2e, or any @pytest.mark.provider_* skip the placeholder injection entirely, so they always see the real environment. Tests located under tests/integration/, tests/live/, or tests/e2e/ are also treated as real and skip placeholder injection.
| Scenario | Behavior |
|---|
| Key set in shell, running unit/mock test | Real key is preserved (used as-is) |
| Key unset, running unit/mock test | Placeholder "test-key" injected, removed on teardown |
Test marked provider_openai / real / network / e2e | Placeholder injection skipped entirely |
Test located under integration/, live/, or e2e/ | Placeholder injection skipped entirely |
OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_API_KEY, XAI_API_KEY, GROQ_API_KEY, COHERE_API_KEY.
Export your real keys in ~/.bashrc / ~/.zshrc if you want local unit tests to call real provider APIs without using the --live flag — the autouse fixture will not overwrite them.
An explicit empty string (export OPENAI_API_KEY="") is also preserved — the fixture only injects placeholders when the variable is fully absent from os.environ.
CLI Options
praisonai test [OPTIONS]
Options:
--tier, -t TEXT Test tier: smoke, main, extended, nightly
--provider, -p TEXT Provider: openai, anthropic, google, ollama, etc.
--live, -l Enable live API tests
--allow-network Allow network access in tests
--parallel, -n TEXT Parallel workers: auto or number
--verbose, -v Verbose output
--coverage, -c Generate coverage report
--fail-fast, -x Stop on first failure
Test Markers
Tests are automatically marked based on their location and content:
Type Markers
unit - Pure unit tests, no networkintegration - Local integration testse2e - End-to-end tests with external calls
Provider Markers
provider_openai - Requires OpenAI APIprovider_anthropic - Requires Anthropic APIprovider_google - Requires Google/Gemini APIprovider_ollama - Requires Ollama running locallyprovider_grok_xai - Requires xAI/Grok APIprovider_groq - Requires Groq APIprovider_cohere - Requires Cohere API
Infrastructure Markers
network - Makes outbound network callslocal_service - Requires Docker or local serviceslow - Takes more than 5 secondsflaky - Known intermittent failuresallow_sleep - Opt out of fast_sleep fixture
Direct pytest Usage
You can also use pytest directly:
# Smoke tests
cd src/praisonai
pytest tests/unit/ -m "not slow and not network" -q --timeout=30
# Main tests with parallelization
pytest tests/unit/ tests/integration/ \
-m "not provider_anthropic and not provider_google" \
-n 2 --timeout=60
# OpenAI live tests
PRAISONAI_LIVE_TESTS=1 pytest tests/ -m "provider_openai" --timeout=120
Network Guard
By default, unit tests block outbound network calls. This prevents:
- Accidental API calls during development
- Flaky tests due to network issues
- Unexpected costs from live API calls
To allow network access:
# Via CLI
praisonai test --live
# Via environment
PRAISONAI_ALLOW_NETWORK=1 pytest tests/
CI/CD Integration
The test suite is designed for CI with separate jobs:
- smoke - Runs on every push, fast validation
- main - Runs on PR/main, comprehensive unit + integration
- openai-live - Runs when
OPENAI_API_KEY secret is set
- extended-* - Provider-specific jobs, nightly schedule
See .github/workflows/test-optimized.yml for the full CI configuration.
Writing Tests
Test Location
tests/unit/ - Unit tests (no network)tests/integration/ - Integration teststests/e2e/ - End-to-end teststests/live/ - Live API tests
Marking Provider Tests
Tests are auto-detected, but you can explicitly mark:
import pytest
@pytest.mark.provider_anthropic
def test_anthropic_feature():
"""This test requires Anthropic API."""
pass
@pytest.mark.network
def test_external_api():
"""This test makes network calls."""
pass
@pytest.mark.slow
def test_long_running():
"""This test takes a while."""
pass
Using Fixtures
Common fixtures are available in conftest.py:
setup_test_environment - autouse fixture that fills missing provider API keys with placeholders; preserves real keys from your shellmock_llm_response - provides mock LLM responses for testingtemp_directory - provides temporary directory for file operations
def test_with_mock_llm(mock_llm_response):
"""Use mock LLM response."""
assert mock_llm_response['choices'][0]['message']['content']
def test_with_temp_dir(temp_directory):
"""Use temporary directory."""
file_path = temp_directory / "test.txt"
file_path.write_text("test")
Troubleshooting
Tests Skipped Unexpectedly
Check if the test is being auto-marked as a provider test:
pytest tests/path/to/test.py --collect-only -q
Network Blocked Error
If you see NetworkBlockedError, the test needs network access:
PRAISONAI_ALLOW_NETWORK=1 pytest tests/path/to/test.py
Timeout Errors
Increase the timeout for slow tests:
pytest tests/ --timeout=120
@pytest.mark.slow
def test_slow_operation():
pass
Pytest Failure Categories
For common test failures and how to distinguish real regressions from stale mocks, see the Pytest Failure Categories Guide.
