Scorers API

Scorers are created at the project level and can be reused across multiple eval cases.

List scorers


GET /v1/projects/:slug/scorers?page=1&pageSize=20

Response


{
  "data": [
    {
      "id": "uuid",
      "project_id": "uuid",
      "name": "Exact Match (case-insensitive)",
      "type": "exact_match",
      "config": { "case_sensitive": false },
      "dual_sided": false,
      "precision_threshold": null,
      "recall_threshold": null,
      "created_at": "2026-04-07T12:00:00.000Z",
      "updated_at": "2026-04-07T12:00:00.000Z"
    }
  ],
  "pagination": { "page": 1, "pageSize": 20, "total": 1, "hasMore": false }
}

Create scorer


POST /v1/projects/:slug/scorers

Request body

Field	Type	Required	Default	Description
`name`	string	Yes	—	Scorer name
`type`	string	Yes	—	`exact_match`, `regex`, `json_schema`, `contains`, `llm_judge`
`config`	object	Yes	—	Type-specific configuration
`dual_sided`	boolean	No	`false`	Enable precision + recall scoring
`precision_threshold`	number	No	`null`	0–1; precision threshold (dual-sided only)
`recall_threshold`	number	No	`null`	0–1; recall threshold (dual-sided only)

Config by type

exact_match


{ "case_sensitive": true }

regex


{ "pattern": "\\d{4}", "flags": "i", "should_match": true }

json_schema


{ "schema": { "type": "object", "required": ["answer"] } }

contains


{ "values": ["hello", "world"], "mode": "all" }

llm_judge


{
  "rubric": "Evaluate faithfulness to the source context...",
  "model": "gpt-4o",
  "_provider": "openai"
}

Get scorer


GET /v1/scorers/:scorerId

Update scorer


PATCH /v1/scorers/:scorerId

Delete scorer


DELETE /v1/scorers/:scorerId

Soft-deletes the scorer. Returns 204 No Content.