• 13 min read
• 3 code samples

Fallback Routing for Unmapped Codes

Within an automated customs brokerage and HS code classification workflow, deterministic resolution of tariff codes is a foundational requirement for regulatory compliance and duty assessment. Production pipelines routinely ingest product descriptions, commercial invoice line items, and supplier-provided classifications that lack a direct mapping to an active Harmonized Tariff Schedule entry. Part of the Core Architecture & Tariff Mapping reference architecture, this workflow handles that operational reality through structured fallback routing: pipeline execution continues without halting while a conservative compliance posture and a complete audit trail are preserved. Fallback routing is not a bypass mechanism — it is a controlled escalation path that moves unmapped or ambiguous codes through deterministic hierarchies, provisional classification buckets, and human-in-the-loop validation queues before any downstream system consumes the data.

# Problem Framing: Silent Misclassification and the Halt-or-Guess Dilemma

The failure mode this workflow exists to prevent is silent misclassification — a line item that fails a primary tariff lookup and is nonetheless assigned a plausible-looking code that flows straight into duty assessment and entry filing. A naive pipeline faces a halt-or-guess dilemma. If it halts on the first unresolved code, a single malformed line on a 4,000-line consolidation stops an entire release window and misses the CBP ACE cargo-release SLA. If it guesses — defaulting to a parent heading or the nearest semantic neighbour without recording that it did so — it manufactures an entry that looks defensible but collapses under a Focused Assessment.

Three concrete conditions trigger fallback in production:

1. Unmapped declared code — the supplier-declared HTS is well-formed but absent from the active schedule (a stale code retired in the last USITC revision, or a foreign CN-code pasted into a US entry).
2. Structurally invalid code — the declared value fails HS digit-length or check constraints (an 8-digit value where the jurisdiction requires a 10-digit statistical suffix).
3. Missing classification entirely — the line arrives with only a free-text product description, deferring resolution to the classification engine.

Fallback routing converts all three into an explicit, typed control path. Every diverted record carries a machine-readable failure code and a cryptographic hash of its source payload, so a reviewer can reconstruct exactly why the record was diverted and what the original document said. The upstream detection of unmapped codes is covered in depth in Handling missing HTS codes in ETL pipelines; this page focuses on what happens after detection — how a diverted record is resolved, validated, and either promoted or quarantined.

# Schema / Data Contract

The contract between the ingestion layer and the fallback engine is a pair of typed records. InvoiceLineItem is the validated input; FallbackRecord is the immutable envelope that captures a diversion. Every field on the fallback record exists to satisfy a later audit question — what failed, why, against which document, and in what state the record currently sits.

import hashlib
import json
import logging
from datetime import datetime, timezone
from enum import Enum
from typing import Any, Optional

from pydantic import BaseModel, Field, ValidationError

logger = logging.getLogger(__name__)


class FailureCode(str, Enum):
    SCHEMA_VIOLATION = "SCHEMA_VIOLATION"     # payload failed structural validation
    UNMAPPED_HTS = "UNMAPPED_HTS"             # well-formed code absent from active schedule
    MALFORMED_HTS = "MALFORMED_HTS"           # digit-length / check-constraint failure
    MISSING_CLASSIFICATION = "MISSING_CLASSIFICATION"  # no code, description only


class ResolutionStatus(str, Enum):
    PENDING_REVIEW = "PENDING_REVIEW"
    RESOLVED_PROVISIONAL = "RESOLVED_PROVISIONAL"
    QUARANTINED = "QUARANTINED"
    RESOLVED = "RESOLVED"


class InvoiceLineItem(BaseModel):
    line_id: str
    product_description: str
    declared_hts: Optional[str] = None
    quantity: float = Field(gt=0)
    unit_value: float = Field(ge=0)
    jurisdiction: str = "US"


class FallbackRecord(BaseModel):
    original_payload: dict[str, Any]
    failure_code: FailureCode
    failure_reason: str
    source_hash: str
    trade_lane: str = "UNKNOWN"
    timestamp: datetime = Field(default_factory=lambda: datetime.now(timezone.utc))
    resolution_status: ResolutionStatus = ResolutionStatus.PENDING_REVIEW


def compute_document_hash(payload: dict[str, Any]) -> str:
    """Order-independent SHA-256 hash for immutable audit trails."""
    serialized = json.dumps(payload, sort_keys=True, default=str).encode("utf-8")
    return hashlib.sha256(serialized).hexdigest()

Two contract rules matter for determinism. First, the source hash is computed over a canonical JSON serialization with sort_keys=True, so two runs over the same payload produce the same hash regardless of dict ordering — the hash is a stable audit key, not an incidental artefact. Second, failure_code and resolution_status are closed enumerations rather than free strings; a diverted record can only occupy one of a finite, reviewable set of states, which keeps the quarantine queue queryable and prevents status typos from hiding records from review.

# Step-by-Step Implementation

Resolution proceeds as a fixed sequence of stages. Each stage has a defined purpose, inputs, outputs, and error condition; a record either resolves at a stage or falls through to the next, and any fall-through is logged.

# Stage 1 — Validate and route at ingestion

Purpose: separate clean records from divertible ones at the point of entry. Inputs: a raw line-item dict and the set of active HTS codes. Outputs: either a validated InvoiceLineItem or a FallbackRecord. Error condition: a schema violation or an unmapped/malformed code branches the record into the dead-letter path with its original payload preserved.

def validate_and_route_line(
    item: dict[str, Any],
    active_hts_index: set[str],
) -> tuple[Optional[InvoiceLineItem], Optional[FallbackRecord]]:
    """Validate schema, confirm HTS existence, divert to fallback on failure."""
    try:
        parsed = InvoiceLineItem(**item)
    except ValidationError as exc:
        return None, FallbackRecord(
            original_payload=item,
            failure_code=FailureCode.SCHEMA_VIOLATION,
            failure_reason=str(exc),
            source_hash=compute_document_hash(item),
        )

    if parsed.declared_hts and parsed.declared_hts not in active_hts_index:
        code = (
            FailureCode.MALFORMED_HTS
            if not parsed.declared_hts.replace(".", "").isdigit()
            else FailureCode.UNMAPPED_HTS
        )
        logger.warning(
            "line %s diverted to fallback: %s (%s)",
            parsed.line_id, code.value, parsed.declared_hts,
        )
        return None, FallbackRecord(
            original_payload=item,
            failure_code=code,
            failure_reason=f"HTS {parsed.declared_hts} absent from active schedule",
            source_hash=compute_document_hash(item),
        )

    return parsed, None

# Stage 2 — Tiered resolution with exclusionary enforcement

Purpose: attempt to resolve a diverted record deterministically before it reaches a human. Inputs: the diverted line item plus the active index, historical classification store, and (optionally) a semantic model. Outputs: a TariffResolutionResult naming the tier that resolved it and a confidence score. Error condition: exclusionary notes block a hierarchical fallback, or no tier clears its threshold and the record is quarantined.

The engine walks a strict precedence order. The initial tier attempts hierarchical parent-code resolution, traversing upward from the 10-digit statistical suffix to the 8-, 6-, or 4-digit level. Crucially, it cannot simply widen to a broader category when legal notes forbid that aggregation: before assigning a provisional parent code, it queries General Rules of Interpretation constraints, Section and Chapter Notes, and exclusionary rulings sourced from the HTS Schedule Database Design schema. Only when no note blocks the candidate does the parent become an eligible provisional match. The heuristic tier — matching against historical shipment patterns, supplier-specific classification histories, and semantic similarity over product-description embeddings — is developed further in Building fallback logic for ambiguous tariff classifications.

from dataclasses import dataclass, field


class ResolutionTier(str, Enum):
    EXACT_MATCH = "EXACT_MATCH"
    PARENT_TRAVERSAL = "PARENT_TRAVERSAL"
    SEMANTIC_FALLBACK = "SEMANTIC_FALLBACK"
    PROVISIONAL_BUCKET = "PROVISIONAL_BUCKET"


@dataclass
class TariffResolutionResult:
    hts_code: str
    tier: ResolutionTier
    confidence_score: float
    compliance_notes: list[str] = field(default_factory=list)
    requires_human_review: bool = False


class FallbackRouter:
    PROVISIONAL_CODE = "9999.99.99"
    SEMANTIC_THRESHOLD = 0.78

    def __init__(self, hts_index: set[str], historical_db: dict, semantic_model=None):
        self.hts_index = hts_index
        self.historical_db = historical_db
        self.semantic_model = semantic_model

    def _traverse_parent(self, hts: str) -> Optional[str]:
        """Climb the fixed-width hierarchy: 10 → 8 → 6 → 4 digits."""
        digits = hts.replace(".", "")
        for length in (8, 6, 4):
            candidate = digits[:length]
            if candidate in self.hts_index:
                return candidate
        return None

    def _blocking_notes(self, candidate: str) -> list[str]:
        """Return GRI / Section / Chapter notes that forbid this candidate.

        In production this queries the normalized notes table; a non-empty
        list means the candidate is legally ineligible as a fallback.
        """
        return []

    def resolve(self, line: InvoiceLineItem) -> TariffResolutionResult:
        declared = line.declared_hts
        if not declared:
            # No code at all — hand straight to the classification engine.
            return TariffResolutionResult(
                self.PROVISIONAL_CODE, ResolutionTier.PROVISIONAL_BUCKET, 0.0,
                compliance_notes=["No declared code; description-only line"],
                requires_human_review=True,
            )

        # Tier 1 — exact match
        if declared in self.hts_index:
            return TariffResolutionResult(declared, ResolutionTier.EXACT_MATCH, 1.0)

        # Tier 2 — parent traversal, gated by exclusionary notes
        parent = self._traverse_parent(declared)
        if parent:
            notes = self._blocking_notes(parent)
            if notes:
                logger.warning("parent fallback for %s blocked: %s", declared, notes)
            else:
                return TariffResolutionResult(
                    parent, ResolutionTier.PARENT_TRAVERSAL, 0.85,
                    requires_human_review=True,
                )

        # Tier 3 — semantic / historical fallback
        if self.semantic_model:
            score = self.semantic_model.score(line.product_description)
            if score >= self.SEMANTIC_THRESHOLD:
                return TariffResolutionResult(
                    self.semantic_model.best_code, ResolutionTier.SEMANTIC_FALLBACK,
                    score, requires_human_review=True,
                )

        # Tier 4 — provisional bucket, always human-gated
        return TariffResolutionResult(
            self.PROVISIONAL_CODE, ResolutionTier.PROVISIONAL_BUCKET, 0.0,
            compliance_notes=["Assigned provisional code pending broker review"],
            requires_human_review=True,
        )

# Stage 3 — Persist state and gate promotion

Purpose: record the resolution outcome and decide whether the record may leave the fallback path. Inputs: the FallbackRecord and its TariffResolutionResult. Outputs: an updated record whose resolution_status is RESOLVED_PROVISIONAL (parent/semantic hit awaiting sign-off) or QUARANTINED (bucket assignment). Error condition: any result with requires_human_review=True is forbidden from promotion until a licensed broker signs off — no automated path may set RESOLVED.

# Validation & Determinism

Determinism is what makes fallback routing auditable rather than a source of drift. The workflow enforces it at three checkpoints:

• Digit-length and structure checks. A candidate is only eligible if its digit form matches the jurisdiction’s required width — 10 digits for HTSUS statistical suffixes, 8 for EU CN-codes. The _traverse_parent climb only ever shortens a valid numeric prefix, so it can never invent a code that was not already a legal ancestor in the schedule.
• Exclusionary cross-checks. Every parent candidate is passed through _blocking_notes before it is offered as a resolution. WCO nomenclature and HTSUS legal notes routinely forbid rolling a specific subheading up into its parent; honouring those notes is the difference between a defensible provisional code and a fabricated one.
• Confidence thresholds and quarantine routing. Each tier carries a fixed confidence score, and the semantic tier applies an explicit SEMANTIC_THRESHOLD. Anything below threshold — and every provisional-bucket assignment — routes to quarantine with requires_human_review=True. The confidence value is stored on the record so a reviewer can sort the queue by risk and so a later reconciliation can measure how often each tier fired.

Because the source hash is computed from a canonically serialized payload, re-running the same batch is idempotent at the audit layer: identical inputs produce identical hashes and identical failure codes, so a replay never generates a second, divergent trail for the same document.

# Downstream Integration

Fallback states must never feed downstream duty or origin logic until they are explicitly resolved. The architecture enforces a hard isolation boundary so provisional codes cannot contaminate financial reporting. Until a record transitions to RESOLVED, it is withheld from the Duty Formula Calculation Frameworks that compute ad valorem and compound rates, and from the Rule of Origin Logic Engines that evaluate preferential-treatment claims — a provisional 9999.99.99 bucket has no defensible duty rate and no origin determination, so exposing it downstream would produce a numerically plausible but legally void entry.

Provisional records are held behind the Security Boundary & Data Isolation layer, which restricts read and write access on the quarantine queue to compliance officers and licensed brokers. Resolution is not a one-way door: when the Tariff Update Ingestion Pipelines deploy a new schedule version, the fallback engine re-evaluates every stalled record against the refreshed index and automatically promotes any that now resolve exactly — a code that was unmapped last week may be a clean Tier 1 hit after a USITC revision adds it. Only records that pass that re-evaluation, or that a broker signs off manually, cross the boundary into the active processing stream.

# Scaling & Resilience

Fallback queues in a high-volume brokerage pipeline must survive multi-thousand-line consolidations without exhausting memory or stalling clean traffic. The controls below keep the divert path bounded and non-blocking.

• Partitioned, bounded queues. The quarantine store is partitioned by trade_lane and commodity class so a flood of unmapped codes on one lane cannot starve review capacity on another. Each partition carries a bounded depth; crossing it trips a circuit breaker that pages the compliance team rather than silently accumulating unresolved liability.
• Streaming over materialization. Large dead-letter batches are consumed through streaming iterators, never loaded into a single in-memory list. This keeps the resident footprint flat during a bulk import and avoids the garbage-collection pauses that would otherwise create pipeline backpressure on the clean path.
• Async resolution with a semaphore. When a resolution tier calls out to an embedding model or the notes table, those calls run under an asyncio.Semaphore that caps concurrent lookups, so a burst of diverted lines cannot exhaust the database connection pool shared with primary ingestion.
• Retry with dead-letter escalation. Transient failures (a notes-table timeout) retry with capped exponential backoff; a record that exhausts its retries is escalated to quarantine rather than dropped, so no line is ever lost between the divert and the review queue.

# Compliance Obligations

Fallback routing is an auditable control surface, not an automated classification shortcut, and its output must satisfy the record-keeping obligations that govern entry filing.

• Retention. Every FallbackRecord, including its source_hash and original payload, is retained for the full record-keeping window a CBP Focused Assessment can reach back over — the divert trail is part of the entry’s defensible history, not scratch state.
• Audit fields. Each record persists the failure code, the resolving tier, the confidence score, the resolving actor (system tier or named broker), and the timestamp of every state transition, so a reviewer can reconstruct the point-in-time decision that produced the filed code.
• Regulatory-notice handling. Federal Register notices and tariff bulletins that retire or add codes flow in through the ingestion pipelines and trigger the re-evaluation pass above; a diverted record’s status is always relative to the schedule version in force on the import date.
• Human-in-the-loop escalation. Every provisional or quarantined assignment requires a digital signature from a licensed customs broker before it may be promoted to RESOLVED and released to duty assessment. No confidence score, however high, substitutes for that sign-off — the gate is a compliance control, not a performance tuning knob.

Together these obligations make fallback routing resilient without loosening the conservative classification posture that international trade regulation demands: the pipeline keeps moving, but nothing provisional ever masquerades as final.

# Related

• Handling missing HTS codes in ETL pipelines — upstream detection and dead-letter routing of unresolved codes.
• Building fallback logic for ambiguous tariff classifications — the heuristic and semantic tier in detail.
• HTS Schedule Database Design — the versioned schema and legal-notes tables the engine queries.
• Tariff Update Ingestion Pipelines — schedule revisions that trigger re-evaluation of stalled records.
• Security Boundary & Data Isolation — the access boundary guarding the quarantine queue.

Up: Core Architecture & Tariff Mapping

Authoritative references: World Customs Organization HS Nomenclature · USITC Harmonized Tariff Schedule · Pydantic validation