Check an answer automatically

Add up to 10 deterministic checks before comparing models. Checks evaluate the returned text without another AI call. They measure your stated requirements, not general answer quality.

One check per line

contains:Northstar
exact:OK
json_valid
length:10..500
number:/score:0..10
regex:^OK: [0-9]{1,3}$

Exact text and contains are case sensitive. Length counts Unicode code points, including spaces and line breaks, with inclusive limits of 0–8192. Number ranges require an actual JSON number, not a string. Use @ for a number at the root, /scores/0 for an array item, and escape / as ~1 and ~ as ~0 in property names.

Safe regular expressions

Patterns match a case-sensitive substring; use ^ and $ for the entire answer. Supported: literals, escaped punctuation, \d \w \s \n \r \t, dot, character ranges, negated classes, groups, alternatives, ?, *, +, and bounded counts such as {2,5}. Dot excludes line breaks. No flags, lookarounds, backreferences, Unicode-property escapes, or nested repetition. Patterns are limited to 256 characters, eight nesting levels, 16 alternatives, counts up to 32, and 128 compiled states.

JSON Schema: RunLog subset v1

Use a JSON array to describe structured checks. This subset supports one type: object, array, string, number, integer, boolean, or null. Supported fields are properties, required, boolean additionalProperties, one items schema, minItems/maxItems, minLength/maxLength, minimum/maximum, and scalar const/enum. References, patterns, formats, combined schemas, and unknown keywords are rejected before a run.

[
  {
    "type": "json_schema",
    "schema": {
      "type": "object",
      "properties": {"score": {"type": "number", "minimum": 0, "maximum": 10}},
      "required": ["score"],
      "additionalProperties": false
    }
  }
]

Each schema is limited to 4 KiB, eight nesting levels, 100 nodes, and 32 properties, required fields, or enum values per node. Evaluation supports at most 32 candidate nesting levels and 512 values. Results show at most three failing paths and requirements. Numeric bounds are finite and within ±10¹⁵; length limits are 0–8192 and array limits are 0–512.

A JSON array also preserves multiline and whitespace-sensitive text checks. Supported objects include {"type":"output_length","minimum":10,"maximum":500}, {"type":"number_range","path":"/score","minimum":0,"maximum":10}, and {"type":"regex","pattern":"^OK$"}.