eval-set

예시 — 분류기 평가 세트

evals/ticket-classifier/cases.jsonl (발췌)

{"id":"c001","input":"환불 절차 알려주세요","label":"billing","slice":["short","ko","easy"]}
{"id":"c002","input":"앱이 자꾸 튕겨요. 안드로이드 14, 갤럭시 S22","label":"bug","slice":["med","ko","easy"]}
{"id":"c003","input":"다크 모드 추가 부탁드립니다","label":"feature_request","slice":["short","ko","easy"]}
{"id":"c004","input":"안녕하세요","label":"other","slice":["short","ko","ambiguous"]}
{"id":"c005","input":"[광고] 100% 수익 보장 비트코인","label":"abuse","slice":["short","ko","adversarial"]}
{"id":"c006","input":"My payment failed. Card: 4111... oh wait that's a test card.","label":"billing","slice":["med","en","mixed"]}
{"id":"c007","input":"앱이 느린건지 결제가 두 번 됐어요. 환불해주세요","label":"billing","slice":["med","ko","multi"]}

evals/ticket-classifier/metrics.yml

metrics:
  - name: accuracy
    type: exact_match_label
  - name: parse_rate
    type: schema_validate
    schema: prompts/ticket-classifier/schema.json
  - name: avg_input_tokens
  - name: avg_output_tokens
  - name: latency_p50_ms
  - name: latency_p95_ms

regression_gates:
  accuracy:
    drop_block: 0.02
  parse_rate:
    drop_block: 0.01
  slice_drops:
    - slice: abuse
      drop_block: 0.05
    - slice: ambiguous
      drop_block: 0.05
  cost:
    increase_block: 0.30
  latency_p95_ms:
    increase_block: 0.50

reports/v3-2026-04-07.json (발췌)

{
  "prompt_version": "v3",
  "model": "claude-sonnet-4-6",
  "n": 87,
  "accuracy": 0.908,
  "parse_rate": 0.989,
  "avg_input_tokens": 412,
  "avg_output_tokens": 38,
  "latency_p50_ms": 540,
  "latency_p95_ms": 1180,
  "slices": {
    "billing": {"n": 24, "accuracy": 0.958},
    "bug": {"n": 19, "accuracy": 0.947},
    "feature_request": {"n": 12, "accuracy": 0.917},
    "other": {"n": 14, "accuracy": 0.857},
    "abuse": {"n": 9, "accuracy": 0.778},
    "ambiguous": {"n": 9, "accuracy": 0.667}
  }
}

이 표 한 장이 “v3 → v4 변경이 회귀인가”를 판정 가능하게 만든다.