Experiments & Contracts

Base URL http://localhost:3333/api/ · Authentication · Conventions

Experiment Routes

Base path: /api/experiments

Manages A/B testing experiments, evaluation datasets, and benchmarks for the evolution system.

Create Eval Dataset

POST /datasets

• Authentication: Required
• Description: Create an evaluation dataset (manual or synthetic)
• Body:

{
  "name": "Dataset Name",
  "skillId": "skill-id",
  "category": "general",
  "source": "manual|synthetic|history|golden",
  "items": [
    {
      "input": "Test input",
      "expectedOutput": "Expected output",
      "metadata": {}
    }
  ],
  "splitConfig": {
    "train": 0.7,
    "test": 0.2,
    "validation": 0.1
  }
}

• Response:

{
  "success": true,
  "datasetId": "dataset-uuid"
}

List Datasets

GET /datasets

• Authentication: Required
• Parameters:
  • skillId (query, optional): Filter by skill
  • category (query, optional): Filter by category
• Response:

{
  "success": true,
  "datasets": [
    {
      "id": "dataset-id",
      "name": "Dataset Name",
      "skillId": "skill-id",
      "category": "general",
      "source": "manual",
      "itemCount": 100,
      "createdAt": "2024-01-01T00:00:00Z"
    }
  ]
}

Generate Dataset

POST /datasets/generate

• Authentication: Required
• Description: Auto-generate a dataset from goal history, golden standards, or synthetically
• Body:

{
  "skillId": "skill-id",
  "source": "history|golden|synthetic",
  "category": "general",
  "provider": "openai",
  "model": "gpt-4"
}

• Response:

{
  "success": true,
  "datasetId": "dataset-uuid"
}

Get Dataset with Splits

GET /datasets/:id

• Authentication: Required
• Parameters:
  • id (path): Dataset ID
• Response:

{
  "success": true,
  "dataset": {
    "id": "dataset-id",
    "name": "Dataset Name",
    "items": []
  },
  "splits": {
    "train": [],
    "test": [],
    "validation": []
  }
}

Delete Dataset

DELETE /datasets/:id

• Authentication: Required
• Parameters:
  • id (path): Dataset ID
• Response:

{
  "success": true
}

Get Benchmarks

GET /benchmarks

• Authentication: Required
• Description: List golden standard benchmarks available for experiments
• Response:

{
  "success": true,
  "benchmarks": [
    {
      "id": "benchmark-id",
      "name": "Benchmark Name",
      "sourceGoalId": "goal-id",
      "createdAt": "2024-01-01T00:00:00Z"
    }
  ]
}

Create Experiment

POST /

• Authentication: Required
• Body:

{
  "name": "Experiment Name",
  "hypothesis": "Skill v2 will outperform v1 on accuracy",
  "type": "ab_test|benchmark|regression",
  "sourceGoalId": "goal-id",
  "benchmarkId": "benchmark-id",
  "skillId": "skill-id",
  "evalDatasetId": "dataset-id",
  "config": {}
}

• Response:

{
  "success": true,
  "experiment": {
    "id": "experiment-id",
    "name": "Experiment Name",
    "status": "created",
    "createdAt": "2024-01-01T00:00:00Z"
  }
}

List Experiments

GET /

• Authentication: Required
• Parameters:
  • status (query, optional): Filter by status
  • limit (query, optional): Max results (default: 50)
• Response:

{
  "success": true,
  "experiments": [
    {
      "id": "experiment-id",
      "name": "Experiment Name",
      "status": "created|running|completed|failed",
      "type": "ab_test",
      "createdAt": "2024-01-01T00:00:00Z"
    }
  ]
}

Get Experiment with Results

GET /:id

• Authentication: Required
• Parameters:
  • id (path): Experiment ID
• Response:

{
  "success": true,
  "experiment": {
    "id": "experiment-id",
    "name": "Experiment Name",
    "status": "completed",
    "results": {},
    "createdAt": "2024-01-01T00:00:00Z"
  }
}

Run Experiment

POST /:id/run

• Authentication: Required
• Parameters:
  • id (path): Experiment ID
• Body (optional):

{
  "provider": "openai",
  "model": "gpt-4"
}

• Description: Fire-and-forget experiment execution. Returns immediately while the experiment runs in the background.
• Response:

{
  "success": true,
  "message": "Experiment run started"
}

Delete Experiment

DELETE /:id

• Authentication: Required
• Parameters:
  • id (path): Experiment ID
• Response:

{
  "success": true
}

Get Experiment Runs

GET /:id/runs

• Authentication: Required
• Parameters:
  • id (path): Experiment ID
• Description: Get all run results for an experiment
• Response:

{
  "success": true,
  "runs": [
    {
      "id": "run-id",
      "experimentId": "experiment-id",
      "status": "completed",
      "results": {},
      "startedAt": "2024-01-01T00:00:00Z",
      "completedAt": "2024-01-01T00:05:00Z"
    }
  ]
}

---

Contract Routes

Base path: /api/contracts

Contracts are refinement-type runtime invariants (PRD-091 Layer 5). They are either authored by the user/agent or mined by InsightEngine from successful executions ("output must be JSON", "step count ≤ 5", "response includes citation block"). At runtime, ContractsService.check evaluates whether evidence satisfies the contract; violations are counted on the contract row and feed into FitnessScoreService (contract cleanliness component).

Contract shape

{
  "id": "uuid",
  "user_id": "user-uuid",
  "target_type": "tool|workflow|skill|agent",
  "target_id": "target-uuid",
  "name": "Output must be valid JSON",
  "predicate": { "type": "json_valid", "field": "output" },
  "source": "authored|mined",
  "status": "active|disabled|deprecated",
  "confidence": 0.92,
  "evidence_count": 47,
  "violation_count": 2,
  "created_at": "...",
  "updated_at": "..."
}

List Contracts

GET /

• Authentication: Required
• Parameters:
  • status (query, optional): active, disabled, deprecated
  • targetType (query, optional): tool, workflow, skill, agent
• Response:

{ "success": true, "contracts": [ { ... } ] }

Create Contract

POST /

• Authentication: Required
• Body:

{
  "targetType": "tool",
  "targetId": "web-search",
  "name": "Output must include source URLs",
  "predicate": { "type": "regex", "field": "output", "pattern": "https?://" },
  "confidence": 0.9
}

• Description: Authors a new contract. targetType and targetId scope the contract to a specific asset; predicate is a JSON shape interpreted by ContractsService.check. Errors return 400 if required fields are missing.
• Response (201):

{ "success": true, "contract": { ... } }

Get Single Contract

GET /:id

• Authentication: Required
• Response:

{ "success": true, "contract": { ... } }

• Errors: 404 not found, 403 forbidden

Get Contract Violations

GET /:id/violations

• Authentication: Required
• Description: Returns the violation history for a contract — useful when surfacing why a canary-check flagged regression.
• Response:

{
  "success": true,
  "violations": [
    {
      "id": "v-uuid",
      "contract_id": "c-uuid",
      "source_execution_id": "exec-uuid",
      "details": { "expected": "...", "actual": "..." },
      "created_at": "..."
    }
  ]
}

Update Contract Status

PATCH /:id

• Authentication: Required
• Body:

{ "status": "disabled" }

• Description: Currently only status is mutable. Use this to disable a noisy contract without deleting it.
• Response:

{ "success": true, "contract": { ... } }

Check Evidence Against Active Contracts

POST /check

• Authentication: Required
• Body:

{
  "targetType": "tool",
  "targetId": "web-search",
  "runtimeState": { "output": "..." },
  "sourceExecutionId": "exec-uuid"
}

• Description: Evaluates runtimeState against every active contract for (targetType, targetId). Records evidence on each contract; persists a violation row if a predicate fails. Returns per-contract verdicts.
• Response:

{
  "success": true,
  "checked": 3,
  "passed": 2,
  "failed": 1,
  "verdicts": [
    { "contractId": "...", "name": "...", "passed": true },
    { "contractId": "...", "name": "...", "passed": false, "violationId": "..." }
  ]
}

Delete Contract

DELETE /:id

• Authentication: Required
• Response:

{ "success": true, "deleted": true }

---

Mutation History Routes

Base path: /api/mutations

Every router-applied change is recorded here (PRD-091 Layer 7). Each row captures the before-state snapshot and the fitness_before baseline so a regression can trigger non-lossy revert. This is the audit trail the agent should consult when the user asks "what did AGNT change?" or "undo that change."

Mutation shape

{
  "id": "uuid",
  "user_id": "user-uuid",
  "insight_id": "insight-uuid|null",
  "target_type": "agent|skill|workflow|tool|memory",
  "target_id": "target-uuid",
  "operation": "apply|revert",
  "status": "applied|reverted|failed",
  "before_snapshot": { "...": "...": "the asset as it was before the change" },
  "after_snapshot": { "...": "...": "the asset as written" },
  "fitness_before": 0.82,
  "fitness_after": 0.78,
  "fitness_delta": -0.04,
  "reverted_reason": "manual|canary_regression|null",
  "created_at": "...",
  "reverted_at": "..."
}

List Mutations

GET /

• Authentication: Required
• Parameters:
  • status (query, optional): applied, reverted, failed
  • targetType (query, optional): agent, skill, workflow, tool, memory
  • limit (query, optional): default 200
• Description: Returns the mutation history for the authenticated user, newest first.
• Response:

{ "success": true, "history": [ { ... } ] }

Get Single Mutation

GET /:id

• Authentication: Required
• Description: Full row including before_snapshot and after_snapshot. Use this to render a diff or to surface what changed.
• Response:

{ "success": true, "mutation": { ... } }

• Errors: 404 not found, 403 forbidden

Canary Check (Detect Regression)

POST /:id/canary-check

• Authentication: Required
• Description: Re-scores fitness for the mutated asset right now (using FitnessScoreService.forTool / forWorkflow) and compares against the stored fitness_before baseline. Persists fitness_after and fitness_delta on the row. The agent should call this before suggesting revert.
• Response:

{
  "success": true,
  "verdict": {
    "regression": true,
    "delta": -0.12,
    "fitnessAfter": 0.70
  }
}

• Verdict shape:
  • regression: true when delta < -0.05 (default minDelta)
  • regression: false, reason: 'not_applicable' when mutation isn't in applied status
  • regression: false, reason: 'no_baseline' when fitness_before was never captured
  • regression: false, reason: 'no_after_score' when there's no recent execution data to score against

Revert Mutation

POST /:id/revert

• Authentication: Required
• Body (optional):

{ "reason": "manual" }

• Description: Marks the mutation row as reverted with reverted_reason. The actual rollback uses the before_snapshot to restore the asset (handled by the model). Safe to call only when canary-check confirms regression OR the user explicitly requests it.
• Response:

{ "success": true }

• Default reason: "manual" when the user triggers revert; the periodic canary sweep uses "canary_regression".

---