Evaluations

An evaluation is one agentic run over a submission's ready documents: it executes the template's checks, extracts its data fields, and produces a summarized verdict. Evaluations are explicit — you start one when you consider the document set complete, and you can start another after documents change.

Starting an evaluation

curl -X POST https://api.aircredit.de/partner/v1/submissions/$SUBMISSION_ID/evaluations \
  -H "Authorization: Bearer $AIRCREDIT_API_KEY"

const evaluation = await client.startEvaluation({ submissionId: submission.id });
console.log(evaluation.id, evaluation.status); // -> "…", "queued"

evaluation = client.start_evaluation(submission_id=submission.id)
print(evaluation.id, evaluation.status)  # -> "…", "queued"

let evaluation = client
    .start_evaluation()
    .submission_id(&submission.id)
    .send()
    .await?
    .into_inner();
println!("{} {:?}", evaluation.id, evaluation.status);

The evaluation returns in status queued and runs asynchronously. At most one evaluation is active per submission — starting while one is queued or running fails with 409 evaluation_in_progress. The submission needs at least one upload in status ready.

Starting a new evaluation after a previous one completed or failed re-evaluates the current set of ready documents — that is the intended loop after replacing a rejected document.

Lifecycle and progress

An evaluation moves through queued → running → completed (or failed; canceled when the submission is canceled or deleted mid-run). While running, progress_percent gives a coarse position and stage_label a human-readable description of what is currently happening — display-only text whose wording can change at any time; never branch on it.

You can follow progress three ways, cheapest first:

• Webhooks — evaluation.started, throttled evaluation.progress, and finally evaluation.completed / evaluation.failed / evaluation.canceled.
• get_evaluation — poll the single evaluation resource.
• get_submission — the latest_evaluation field carries the same projection inside the full submission detail.

Reading the result

Once status is completed, result carries the full typed payload:

• summary — the aggregate verdict: outcome (v1 emits likely_approved, likely_rejected, or outcome_uncertain), recommendation (approve, review, needs_more_documents, reject), and risk_band (low → critical).
• checks — one entry per template check, each with a verdict status (fulfilled / open / critical / not_applicable / failed), a confidence score, citations into your uploaded documents, and a short reasoning_summary.
• fields — one entry per template data field, each with an extraction status (succeeded / not_found / failed), the extracted value, confidence, and citations.
• documents — per-document classification (detected document types) and readability.

curl https://api.aircredit.de/partner/v1/submissions/$SUBMISSION_ID/evaluations/$EVALUATION_ID \
  -H "Authorization: Bearer $AIRCREDIT_API_KEY"

const evaluation = await client.getEvaluation({
  submissionId: submission.id,
  evaluationId: evaluation.id,
});

if (evaluation.status === "completed" && evaluation.result) {
  const { summary, checks, fields } = evaluation.result;
  console.log(summary.outcome, summary.recommendation, summary.riskBand);

  for (const check of checks.filter(c => c.status === "critical" || c.status === "open")) {
    console.log(check.title, check.status, check.reasoningSummary);
    for (const cite of check.citations) {
      console.log("  see", cite.externalFileId, "pages", cite.pages);
    }
  }
  for (const field of fields.filter(f => f.status === "succeeded")) {
    console.log(field.label, "=", field.value, `(${field.confidence})`);
  }
}

evaluation = client.get_evaluation(
    submission_id=submission.id,
    evaluation_id=evaluation.id,
)

if evaluation.status == "completed" and evaluation.result:
    result = evaluation.result
    print(result.summary.outcome, result.summary.recommendation, result.summary.risk_band)

    for check in result.checks:
        if check.status in ("critical", "open"):
            print(check.title, check.status, check.reasoning_summary)
            for cite in check.citations:
                print("  see", cite.external_file_id, "pages", cite.pages)

    for field in result.fields:
        if field.status == "succeeded":
            print(field.label, "=", field.value, f"({field.confidence})")

let evaluation = client
    .get_evaluation()
    .submission_id(&submission.id)
    .evaluation_id(&evaluation.id)
    .send()
    .await?
    .into_inner();

if let (EvaluationStatus::Completed, Some(result)) = (evaluation.status, &evaluation.result) {
    let summary = &result.summary;
    println!("{:?} {:?} {:?}", summary.outcome, summary.recommendation, summary.risk_band);

    for check in result.checks.iter()
        .filter(|check| matches!(check.status, CheckResultStatus::Critical | CheckResultStatus::Open))
    {
        println!("{} {:?} {:?}", check.title, check.status, check.reasoning_summary);
    }
    for field in result.fields.iter()
        .filter(|field| matches!(field.status, FieldResultStatus::Succeeded))
    {
        println!("{} = {:?} ({:?})", field.label, field.value, field.confidence);
    }
}

Citations and confidence

Every check verdict and extracted field cites the evidence it is based on: the upload_id (and your external_file_id) of the document, page numbers when the document is page-addressable, and often a short excerpt. Use citations to jump a human reviewer straight to the relevant passage instead of re-reading the whole file.

confidence is a score in [0, 1]. It is calibrated for ranking and thresholds — route low-confidence verdicts to manual review rather than treating the number as a probability. For data fields, the template's per-field confidence_threshold controls what the evaluation itself auto-accepts.

When results go stale

Results describe the document set they ran on (input_file_ids). If you upload, replace, or delete documents afterwards, start a new evaluation; reading an old run's detail may fail with 409 analysis_outdated once it has been superseded. The submission's status always reflects the latest evaluation.