• 9 min read
  • 1 code sample

Calibrating OCR confidence thresholds for HS digits

Optical character recognition assigns a per-character confidence to every digit it reads off a scanned invoice or manifest, but choosing the cutoff below which an HS or HTS digit must go to human review is a calibration problem, not a guess: set it too low and confusable pairs like 0/8 and 1/7 slip through and mis-classify an entry, set it too high and reviewers drown in digits the engine actually read correctly. This page treats threshold selection as a precision-recall trade-off measured against a labelled ground-truth set, sweeps the cutoff to find the operating point that meets a target error rate on the classifying digits, and routes only genuinely low-confidence characters to review. The implementation uses Decimal for rate arithmetic and slots into the correction pipeline described in OCR Drift Correction & Validation.

Prerequisites

This calibration assumes a specific evaluation setup and toolchain. Confirm each before applying it:

  • Python 3.10+ with dataclass, Optional, and Decimal. Error and pass rates are held as Decimal so a target like a 0.2% escape rate is represented exactly, not as a binary-float approximation. No heavy ML dependency is required; the sweep is plain arithmetic over a scored sample.
  • A labelled ground-truth set of digit reads. You need a representative sample where each OCR digit read is paired with its true value and the engine’s reported confidence. Without ground truth you cannot measure precision or recall — you can only pick a cutoff blindly. A few thousand digit reads spanning your real scanners and fonts is enough to start.
  • Per-character confidence, not per-field. The engine must expose confidence at the character level. A single field-level score hides the one weak digit inside an otherwise strong 10-digit code, which is exactly the digit you need to catch.
  • A defined cost asymmetry. Decide what a missed error (an accepted wrong digit that reaches classification) costs relative to a false alarm (a correct digit sent to review). Customs misclassification carries duty and penalty exposure, so the missed-error cost usually dominates — that asymmetry drives the operating point.
  • A review queue and structured logging already provisioned. This page routes low-confidence digits to review; it does not build the queue. Configure stdlib logging with the %(asctime)s | %(levelname)s | %(name)s | %(message)s format so every routing decision is auditable.

Framing the trade-off

Treat “accept the OCR digit” as the positive action. At a given confidence threshold, every scored digit falls into one of four buckets: a correct digit accepted (true positive), a wrong digit accepted (false positive — the dangerous escape), a correct digit sent to review (false negative — wasted effort), and a wrong digit sent to review (true negative — a catch). Raising the threshold moves digits from accepted to reviewed: it lowers false positives (fewer escapes) but raises false negatives (more review load). That is the entire tension.

Two rates summarize it. Precision of acceptance is the fraction of accepted digits that were actually correct — high precision means few bad digits escape. Recall of acceptance is the fraction of all correct digits that you accepted automatically — high recall means little needless review. You cannot maximize both; the sweep finds the threshold where precision meets your escape-rate target while recall stays as high as that target allows.

Confusable digit pairs concentrate the risk. The engine is most likely to be wrong, yet still moderately confident, on 0 versus 8, 1 versus 7, 5 versus 6, and 3 versus 8 under smudged or low-DPI scans. Those are precisely the reads that cluster just above a carelessly low threshold, so the operating point must be chosen against a sample that includes them.

Implementation

The calibrator sweeps candidate thresholds over the labelled sample, computes precision and recall at each, and returns the lowest threshold that holds the accepted-error rate at or below the target — minimizing review load subject to the compliance constraint.

import logging
from dataclasses import dataclass
from decimal import Decimal, ROUND_HALF_UP
from typing import Optional

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s | %(levelname)s | %(name)s | %(message)s",
    handlers=[logging.StreamHandler()],
)
logger = logging.getLogger("ocr_calibrator")


@dataclass(frozen=True)
class DigitRead:
    predicted: str        # digit the OCR engine reported
    truth: str            # ground-truth digit from the labelled set
    confidence: Decimal   # engine confidence in [0, 1]


@dataclass(frozen=True)
class OperatingPoint:
    threshold: Decimal
    precision: Decimal    # of accepted digits, fraction correct
    recall: Decimal       # of correct digits, fraction accepted
    accepted_error: Decimal   # of accepted digits, fraction wrong (escape rate)
    review_load: Decimal      # fraction of all digits routed to review


def _q(x: Decimal) -> Decimal:
    return x.quantize(Decimal("0.0001"), rounding=ROUND_HALF_UP)


def evaluate(sample: list[DigitRead], threshold: Decimal) -> OperatingPoint:
    tp = fp = fn_ = tn = 0
    for r in sample:
        accepted = r.confidence >= threshold
        correct = r.predicted == r.truth
        if accepted and correct:
            tp += 1
        elif accepted and not correct:
            fp += 1        # escape: a wrong digit was accepted
        elif not accepted and correct:
            fn_ += 1       # wasted review of a correct digit
        else:
            tn += 1        # good catch: wrong digit routed to review

    accepted = tp + fp
    correct_total = tp + fn_
    total = len(sample)
    precision = _q(Decimal(tp) / accepted) if accepted else Decimal("1")
    recall = _q(Decimal(tp) / correct_total) if correct_total else Decimal("1")
    accepted_error = _q(Decimal(fp) / accepted) if accepted else Decimal("0")
    review_load = _q(Decimal(fn_ + tn) / total) if total else Decimal("0")
    return OperatingPoint(threshold, precision, recall, accepted_error, review_load)


def calibrate(
    sample: list[DigitRead],
    target_escape_rate: Decimal = Decimal("0.0020"),  # <= 0.2% wrong accepted
    step: Decimal = Decimal("0.01"),
) -> Optional[OperatingPoint]:
    """Sweep thresholds; return the LOWEST that meets the escape-rate target,
    which maximizes recall (minimizes review load) under the constraint."""
    best: Optional[OperatingPoint] = None
    t = Decimal("0.50")
    while t <= Decimal("1.00"):
        op = evaluate(sample, t)
        logger.info(
            "t=%s precision=%s recall=%s escape=%s review=%s",
            t, op.precision, op.recall, op.accepted_error, op.review_load,
        )
        if op.accepted_error <= target_escape_rate:
            best = op   # first (lowest) qualifying threshold — best recall
            break
        t += step
    if best is None:
        logger.warning("No threshold met escape target %s; route ALL to review.",
                       target_escape_rate)
    return best


def route_digit(read: DigitRead, op: OperatingPoint) -> str:
    """Accept or route a live digit under the calibrated operating point."""
    if read.confidence >= op.threshold:
        return "ACCEPT"
    logger.info("Digit %r conf=%s below %s; route to review.",
                read.predicted, read.confidence, op.threshold)
    return "REVIEW"

The sweep returns the lowest qualifying threshold on purpose: among all cutoffs that satisfy the escape-rate constraint, the lowest one accepts the most digits automatically, so it minimizes review load while still meeting the compliance target. Raising the threshold beyond that point only adds review work without buying any escape-rate headroom you asked for.

Precision-recall trade-off across an OCR confidence threshold sweep A chart with the confidence threshold on the horizontal axis, rising from 0.5 on the left to 1.0 on the right, and rate on the vertical axis from 0 to 1. As the threshold rises, the precision-of-acceptance curve climbs toward 1 because fewer wrong digits are accepted, while the recall-of-acceptance curve falls because more correct digits are pushed to review. A horizontal target line marks the minimum acceptable precision implied by the escape-rate target. A vertical marker shows the chosen operating point: the lowest threshold at which the precision curve crosses the target line, which keeps recall as high as the constraint allows. To the left of the marker digits are accepted automatically; to the right of the marker low-confidence digits, including confusable pairs such as 0 versus 8 and 1 versus 7, are routed to review. 1.0 rate 0 0.50 confidence threshold 1.00 target precision precision recall operating point ← accept automatically route to review → confusable pairs 0/8 · 1/7 · 5/6 cluster near the cutoff
Raising the threshold lifts precision (fewer wrong digits accepted) but drops recall (more correct digits reviewed); the operating point is the lowest threshold whose precision meets the escape-rate target.

Verification steps

Run these checks against your labelled sample before trusting the calibrated threshold in production:

  1. Escape-rate constraint holds. Take the OperatingPoint returned by calibrate and assert accepted_error <= target_escape_rate. If calibrate returned None, confirm your pipeline falls back to routing every digit to review rather than accepting on a threshold that never qualified.
  2. Lowest-threshold optimality. Evaluate the threshold one step below the chosen point and assert it exceeds the escape target — proving the calibrator picked the lowest qualifying cutoff and did not needlessly inflate review load.
  3. Confusable-pair coverage. Filter the sample to reads where predicted and truth form a known confusable pair (0/8, 1/7, 5/6, 3/8). Confirm this subset is present and non-trivial; a calibration blind to confusable pairs will pick a falsely low threshold.
  4. Precision-recall monotonicity. Sweep the full range and assert precision is non-decreasing and recall is non-increasing as the threshold rises. A violation means your confidences are miscalibrated or the sample is too small to be stable.
  5. Decimal exactness. Confirm every rate in OperatingPoint is a Decimal, not a float. A target escape rate of 0.0020 must compare exactly; a float intermediate can push a borderline point to the wrong side of the constraint.
  6. Live routing parity. Run route_digit over a held-out slice at the chosen operating point and confirm the realized accept/review split matches the sweep’s predicted review_load within sampling error. A large gap signals distribution drift between calibration and production scans, which loops back into correcting OCR drift in scanned customs forms.
  7. Per-position sanity. Spot-check that low-confidence digits in the classifying positions of the code (the subheading digits that change the duty rate) are never auto-accepted at a lower bar than trailing statistical digits — misreads there are the costliest.

Edge cases & gotchas

  • Miscalibrated confidences. Some engines report confidences that are not true probabilities — a reported 0.95 may be wrong far more than 5% of the time. Always validate the confidence-to-error relationship on ground truth before trusting the sweep; if it is non-monotonic, apply a calibration map (isotonic or Platt scaling) first, or the threshold is meaningless.
  • Per-field versus per-digit thresholds. A field-level accept gate hides a single weak digit in a strong code. Always calibrate and route at the digit level, then reject the field if any classifying digit routes to review — one bad digit changes the entire HS heading.
  • Position-dependent risk. The first six digits determine the international subheading and the duty consequence; the trailing statistical suffix is lower stakes. Consider a stricter threshold on the classifying positions than on the tail, rather than one flat cutoff across all ten.
  • Sample drift. A threshold calibrated on last quarter’s scanner and paper stock decays as fonts, DPI, and fax artifacts change. Re-run calibrate on fresh ground truth periodically; a stale operating point silently raises the escape rate without any error surfacing.
  • Zero-acceptance degenerate case. If no threshold meets the target, calibrate returns None and you must route everything to review — never quietly relax the target to force a non-null result. A relaxed escape rate is a compliance decision, not a calibration default.
  • Ground-truth leakage. If the labelled set overlaps the data you calibrated the engine on, precision is optimistically biased. Hold out a genuinely independent sample for the sweep, or the chosen threshold will under-review in production.

Up: OCR Drift Correction & Validation

Authoritative references: WCO HS 2022 six-digit subheading structure; HTSUS ten-digit statistical suffix (USITC); 19 CFR 141.86 (reasonable care in entry data).