Documentation

Quick Start

pip install benchd-harness

# Or from source:
git clone https://github.com/benchdai/harness.git
cd harness && pip install -e .

benchd keys generate --out ./keys

# Creates:
#   keys/private.key  (keep secret)
#   keys/public.key   (share freely)

# Set your LLM API key (for the judge)
export OPENROUTER_API_KEY=sk-or-...

# Run LongMemEval against your system
benchd run \
  -a mcp \
  -b longmemeval-v1 \
  --judge \
  --key ./keys/private.key \
  --adapter-config '{"endpoint": "http://localhost:3000/mcp"}'

# Results saved to: ./runs/run_xxx/manifest.signed.json

benchd submit ./runs/run_xxx/manifest.signed.json

# Or upload at: https://benchd.ai/submit

MCP Systems: Zero-Code Testing

If your memory system exposes an MCP server, you don't need to write any adapter code. The generic MCP adapter auto-discovers your tools and maps them to Bench'd's ingest/recall/reset interface.

# Auto-discover tools
benchd run -a mcp -b longmemeval-v1 --judge \
  --adapter-config '{"endpoint": "http://localhost:3000/mcp"}'

# Explicit tool names
benchd run -a mcp -b longmemeval-v1 --judge \
  --adapter-config '{
    "endpoint": "http://localhost:3000/mcp",
    "ingest_tool": "memory_ingest",
    "query_tool": "memory_query",
    "reset_tool": "memory_delete"
  }'

Writing a Custom Adapter

If your system doesn't support MCP, write a Python adapter. It's ~50 lines:

from benchd_harness.adapters.base import BaseAdapter
from typing import Any, Dict, List, Optional


class MyMemoryAdapter(BaseAdapter):
    """Adapter for My Memory System."""

    @property
    def name(self) -> str:
        return "my-memory-system"

    @property
    def version(self) -> Optional[str]:
        return "1.0.0"

    def setup(self) -> None:
        """Initialize your memory system client."""
        self.client = MyMemoryClient()

    def reset(self) -> None:
        """Clear memory between benchmark questions."""
        self.client.clear()

    def teardown(self) -> None:
        """Clean up resources."""
        self.client.close()

    def ingest(self, turns: List[Dict[str, Any]]) -> None:
        """
        Feed conversation turns into your memory system.

        Each turn has: role, content, timestamp (optional)
        """
        for turn in turns:
            self.client.add_message(
                role=turn["role"],
                content=turn["content"],
            )

    def recall(self, query: str) -> str:
        """
        Query your memory system and return a plain string.
        """
        results = self.client.search(query)
        return results.text

Register your adapter in benchd_harness/adapters/__init__.py and run with benchd run -a my-memory-system.

System Categories

Systems are grouped by what they do. Each category has its own leaderboard and question set.

Available Benchmarks

-b longmemeval-v1

-b locomo-v1

-b smoke-memory-v0

Submitting Results

After running a benchmark, submit your signed manifest to appear on the Bench'd leaderboard. All submissions are verified before publishing.

benchd submit ./runs/run_xxx/manifest.signed.json

Trust Tiers

Results on the leaderboard are categorized by how they were verified:

How Bench'd Uses VerifiedState & ProofMeter

Bench'd is a real-world example of how VerifiedState memory verification and ProofMeter spend attestation work together in production. Here's exactly how we use them.

# VerifiedState is also benchmarked AS a memory system:
benchd run -a verifiedstate -b knowledge-retrieval-v0
# This tests VS's own memory_ingest + memory_query capabilities

# Run reliability benchmark with $5 budget and spend tracking:
benchd run -a graphiti -b reliability-v1 --budget 5.00

# Manifest includes proofmeter section:
# {
#   "proofmeter": {
#     "total_spend_usd": "3.81",
#     "receipt_count": 294,
#     "by_model": {
#       "openai/gpt-4o-mini": { "calls": 294, "cost_usd": "3.81" }
#     },
#     "settlement_merkle_root": "sha256:...",
#     "settlement_status": "settled"
#   }
# }

CI Integration (GitHub Action)

Run Bench'd on every PR to catch memory regressions before they ship.

# .github/workflows/benchd.yml
name: Bench'd Memory Benchmark
on: [pull_request]

jobs:
  benchmark:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Start your memory server
        run: docker-compose up -d memory-server

      - name: Run Bench'd
        uses: benchdai/benchmark-action@v1
        with:
          adapter: mcp
          benchmark: smoke-memory-v0
          endpoint: http://localhost:3000/mcp
          openrouter-key: ${{ secrets.OPENROUTER_API_KEY }}

      - name: Upload results
        if: github.event_name == 'push' && github.ref == 'refs/heads/main'
        run: benchd submit ./runs/*/manifest.signed.json

GitHub Action coming soon. Star benchdai/harness to get notified.