• 9 min read
  • 1 code sample

Normalizing CJK invoice fields with Unicode NFC

Chinese, Japanese, and Korean commercial invoices arrive carrying the same product name, quantity, or HS description in several visually identical but bytewise distinct Unicode forms, and unless every field is folded to one canonical shape before matching or hashing, a customs pipeline will treat ８５１７ (fullwidth) and 8517 (ASCII) as different codes, double-count the same consignee, and produce audit hashes that never reconcile. The fix is a deterministic normalization pass built on Python’s unicodedata module: apply NFC or NFKC per field, fold fullwidth digits and compatibility ideographs to their canonical equivalents, then hash the normalized value for the audit trail. This page gives you a runnable normalizer that survives mixed codepages, a verification checklist, and the mojibake-detection and round-trip gotchas that break naive .strip() implementations. It slots into the broader Multi-language Invoice Parsing area.

Prerequisites

This normalizer assumes a specific decode state and toolchain. Confirm each before applying it:

  • Python 3.10+ with a UTF-8 runtime. The code uses dataclass(frozen=True), Optional, and the stdlib unicodedata module. No third-party dependency is required; unicodedata ships the Unicode Character Database that all the folding logic relies on.
  • Bytes already decoded to str, and decoded correctly. Normalization operates on Unicode code points, not bytes. If an upstream reader decoded a Shift-JIS, GB18030, or EUC-KR file as Latin-1 or CP1252, the string is already corrupted (mojibake) and no amount of NFC will repair it. Decode at the true source codepage with errors="strict" and quarantine on UnicodeDecodeError — never errors="replace", which silently injects U+FFFD replacement characters.
  • A field-level normalization policy. NFC and NFKC are not interchangeable. Free-text fields that must survive a byte-for-byte round trip (a legal party name reproduced on a CBP 7501) use NFC; matching keys and numeric fields (HS digits, quantities, weights) use NFKC because compatibility folding is exactly what makes ８５１７ equal 8517. Decide per field, not per document.
  • A quarantine table and structured logging already provisioned. This page routes suspected mojibake to quarantine; it does not create the table. Configure stdlib logging with the %(asctime)s | %(levelname)s | %(name)s | %(message)s format so every normalization event is greppable for audit.

Why NFC and NFKC differ

Unicode defines four normalization forms; two matter here. NFC (Canonical Composition) collapses combining sequences into their precomposed equivalents — a base letter plus a combining diacritic becomes the single precomposed code point — without changing the meaning of any character. It is lossless and reversible in the sense that canonically equivalent strings become bit-identical.

NFKC (Compatibility Composition) goes further: it also folds compatibility characters that are semantically the same but visually or typographically distinct. Fullwidth Latin digits ０１２３, the fullwidth ASCII block used throughout East Asian typesetting, circled and parenthesized numbers, the fraction slash, the ohm sign versus Greek omega, and superscript digits all fold to their plain ASCII or canonical forms under NFKC. That folding is destructive — you cannot recover the original fullwidth glyph — which is why it belongs on matching keys, never on a legal name you must reproduce verbatim.

The customs consequence is concrete. An HS or HTS code keyed off an un-normalized fullwidth string will miss every lookup in a snapshot loaded from ASCII sources, silently over-routing valid line items to fallback. A consignee name normalized with NFKC when the entry document demands NFC will drop the typographic distinction a downstream party-screening match depends on. Choose the form per field and record which form you applied.

Implementation

A single normalizer owns the whole decision: choose the form per field, fold width and compatibility variants, strip zero-width and bidi control characters that East Asian editors inject, and emit a stable audit hash of the normalized value.

import logging
import hashlib
import unicodedata
from dataclasses import dataclass
from enum import Enum
from typing import Optional

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s | %(levelname)s | %(name)s | %(message)s",
    handlers=[logging.StreamHandler()],
)
logger = logging.getLogger("cjk_normalizer")

# Zero-width and bidi controls that CJK editors inject and that break hashing.
_INVISIBLES = {
    "​",  # zero-width space
    "‌",  # zero-width non-joiner
    "‍",  # zero-width joiner
    "",  # zero-width no-break space / BOM
    "‎",  # left-to-right mark
    "‏",  # right-to-left mark
}


class Form(Enum):
    NFC = "NFC"    # lossless — legal free text, party names, descriptions
    NFKC = "NFKC"  # compatibility fold — match keys, HS digits, quantities


@dataclass(frozen=True)
class NormalizedField:
    raw: str
    value: str
    form: str
    audit_hash: str
    had_mojibake: bool


def _strip_invisibles(text: str) -> str:
    return "".join(ch for ch in text if ch not in _INVISIBLES)


def _looks_like_mojibake(text: str) -> bool:
    """Heuristic: U+FFFD replacement chars, or a run of isolated CP1252-range
    high Latin-1 punctuation, signal a wrong-codepage decode upstream."""
    if "�" in text:
        return True
    suspects = sum(1 for ch in text if "€" <= ch <= "Ÿ")
    return suspects >= 2


def normalize_field(raw: Optional[str], form: Form = Form.NFC) -> NormalizedField:
    if raw is None:
        raw = ""
    mojibake = _looks_like_mojibake(raw)
    if mojibake:
        # Do NOT silently patch — a wrong-codepage decode is a data-quality
        # defect. Flag it so ingestion can re-decode at the true source encoding.
        logger.warning("Suspected mojibake in field (raw=%r); flag for re-decode.", raw)

    # 1. Canonicalize, then fold width/compatibility variants if NFKC.
    text = unicodedata.normalize(form.value, raw)
    # 2. Remove invisibles AFTER normalization so BOM-as-ZWNBSP is caught.
    text = _strip_invisibles(text)
    # 3. Collapse ideographic and ASCII whitespace runs to single ASCII spaces.
    text = " ".join(text.split())

    audit = hashlib.sha256(text.encode("utf-8")).hexdigest()
    logger.info("Normalized field via %s (mojibake=%s)", form.value, mojibake)
    return NormalizedField(raw, text, form.value, audit, mojibake)


def normalize_hs_digits(raw: Optional[str]) -> str:
    """HS/HTS codes: NFKC folds fullwidth digits, then keep digits only."""
    folded = normalize_field(raw, Form.NFKC).value
    digits = "".join(ch for ch in folded if ch.isascii() and ch.isdigit())
    return digits

The contract is that every persisted or matched field passes through normalize_field, and the form is recorded alongside the value. A fullwidth HS string ８５１７．６２ becomes 851762 through normalize_hs_digits because NFKC folds each fullwidth digit to ASCII and the ideographic full stop is dropped by the digit filter — so it now keys cleanly against an ASCII-loaded tariff snapshot.

The per-field CJK normalization pipeline and codepoint folding A raw decoded CJK field enters the pipeline. First a mojibake check scans for U+FFFD replacement characters and stray CP1252-range code points; if found the field is flagged for re-decode at the true source codepage. The field then splits by policy: legal free text such as party names takes the NFC path, which composes combining sequences losslessly; match keys and HS digits take the NFKC path, which additionally folds fullwidth digits like fullwidth 8517 to ASCII 8517 and drops compatibility forms. Both paths strip zero-width and bidi control characters, collapse whitespace, then emit a SHA-256 audit hash of the normalized value so the same logical field always hashes identically across documents. raw field decoded str mojibake? U+FFFD scan flag · re-decode true codepage ok NFC path lossless · party names NFKC path fold ８５１７ to 8517 strip zero-width · bidi collapse whitespace SHA-256 audit hash stable across documents same logical field to one canonical value to one hash
Each field is routed by policy — NFC for lossless legal text, NFKC to fold fullwidth digits and compatibility forms — then stripped of invisibles and hashed so the same logical value reconciles across every document.

Verification steps

Run these checks against a representative multi-language batch before trusting the normalizer in production:

  1. Fullwidth-to-ASCII digit fold. Pass normalize_hs_digits("８５１７．６２") and assert the result is exactly "851762". If you get an empty string, your runtime decoded the fullwidth digits as something other than the U+FF10–U+FF19 block — check the source codepage, not the normalizer.
  2. NFC idempotence. Normalize any field twice and assert the second pass is a no-op: normalize_field(x, Form.NFC).value == normalize_field(normalize_field(x, Form.NFC).value, Form.NFC).value. NFC is idempotent by definition; a mismatch means an invisible character survived stripping.
  3. Hash reconciliation. Take the same consignee name expressed with a precomposed character in one document and a decomposed base-plus-combining sequence in another. After NFC, both must produce the identical audit_hash. This is the whole point — canonically equivalent inputs must hash to one value.
  4. Mojibake detection. Feed a Shift-JIS byte string deliberately decoded as CP1252. Assert had_mojibake is True and confirm the record is flagged, not silently normalized. Then re-decode at the correct codepage and confirm the flag clears.
  5. NFKC key parity. Confirm that a party name normalized with NFKC for matching does not overwrite the NFC value persisted for document reproduction. Both forms must coexist; you match on one and reproduce from the other.
  6. Round-trip guard for NFC fields. For any field flowing to a filed document, assert that NFKC was not applied — a fullwidth-to-halfwidth fold on a legal name is a silent data alteration that a downstream reviewer cannot see.

Edge cases & gotchas

  • BOM as a zero-width no-break space. A UTF-8 BOM (U+FEFF) at the start of a field decodes to a zero-width no-break space, which is invisible but hashes. Strip it after normalization — _strip_invisibles runs post-NFC precisely so a leading BOM cannot poison the audit hash.
  • NFKC on quantities and money. Fullwidth digits in quantity or value fields must fold before you parse them into a Decimal; Decimal("１２３") raises InvalidOperation. Route numeric fields through normalize_hs_digits-style ASCII extraction, then parse. This mirrors the weight-field discipline in Reconciling net vs gross weight discrepancies, where a stray fullwidth digit silently breaks unit conversion.
  • Han unification and visually identical CJK glyphs. NFKC does not unify a simplified Chinese character with its traditional counterpart, nor does it merge a Japanese kanji with the Han-unified code point that shares a glyph. Never assume normalization deduplicates product names across a simplified/traditional corpus — that requires a transliteration or mapping layer, not unicodedata.
  • Combining-character order. Two decomposed sequences with the same combining marks in different orders are canonically equivalent only after normalization reorders them by canonical combining class. If you hash before NFC, you will see two hashes for one logical name. Always normalize, then hash.
  • str.isdigit() is broader than ASCII. Fullwidth digits, superscripts, and circled numbers all return True from isdigit(). The digit filter in normalize_hs_digits guards with ch.isascii() first for exactly this reason — drop that guard and a superscript ² leaks into an HS code.
  • Locale-independent folding. unicodedata.normalize is not affected by the process locale, which is the desired property: normalization must be deterministic regardless of the host’s LANG. Do not reach for locale-aware case folding on customs identifiers.

Up: Multi-language Invoice Parsing

Authoritative references: Unicode Standard Annex #15 (Unicode Normalization Forms); WCO HS 2022 nomenclature; ISO 4217 currency minor units.