# Data Tagger Architecture **Status:** Schema landed (migration 142, ADR-038). UI (BB-2) + chat tools (BB-3) pending. **Owner:** Platform pillars (peer to SO / PO / WO / Bid Center). **First use case:** Driscoll Foods PO PDF → NetSuite SalesOrd. --- ## 1. Why a Pillar The platform turns business documents into NetSuite records. Until now this was either (a) an operator typing into the NS UI, or (b) a generic LLM-extraction prompt (the older `document_understanding` path) that stayed at ~70% accuracy because there was no place to record per-sender knowledge. Data Tagger is the layer that learns: one template per (customer, doc_type, ns_record_type) that improves every time the operator corrects it. It is a peer pillar to: | Pillar | What it owns | Lifecycle | | -------------- | --------------------------------------------- | --------------- | | Sales Order | The forward order to a customer | Transactional | | Purchase Order | The forward order to a vendor | Transactional | | Work Order | The build of an assembly | Transactional | | Bid Center | The price commitment to a customer | Contractual | | **Data Tagger**| **The translation: inbound doc → NS-ready JSON** | **Operational** | Folding it into Bid Center conflates contract with operation. Folding it into SO/PO/WO duplicates extraction logic per pillar. One engine, one taxonomy, many templates. --- ## 2. Schema (migration 142) Five tables: 1. **`data_tagger_templates`** — One row per (customer_id, doc_type, ns_record_type, version). `status` ∈ {draft, active, retired}. `customer_match_pattern` lets a template auto-bind to inbound docs (e.g. `from:*@driscollfoods.com`). `hit_count`, `success_count`, `failure_count` track template health. 2. **`data_tagger_tags`** — One row per NS field the template extracts. Each row picks one of the 9 strategies (§3) and carries its `strategy_params_json`. `sample_value`, `sample_md_span`, `sample_region_json` preserve the canonical example so the UI can show "here's where this came from on the original PDF." 3. **`data_tagger_extractions`** — Every application of a template to a document. `status` starts `pending`; operator review moves it to `approved | corrected | rejected | failed`. `applied_to_action_id` links to `proposed_actions` once approved (HITL boundary). 4. **`data_tagger_corrections`** — Reflexion training data. When the operator edits a tag's value, we record (was, corrected, reason). `will_update_template=1` flags the correction for nightly retraining. 5. **`ns_field_taxonomy`** — Curated NS field reference per record type. Seeded with ~70 high-use fields across SalesOrd, PurchOrd, CustInvc, VendBill, WorkOrd, AsmBuild. Drives the tagger UI's field picker and validates `ns_field_path` values resolve to real fields. Indexes cover all FK columns + the common lookup patterns (template selection by customer+doc_type+ns_record_type+status, extraction queue by status, correction backlog by `will_update_template` ∧ `applied_to_template`). --- ## 3. The Nine Extraction Strategies Each strategy has a fixed `strategy_params_json` shape so the engine can dispatch without per-tag code. | # | Strategy | When to use | `strategy_params_json` keys | | - | --------------------- | ----------------------------------------------- | ------------------------------------------------------------------------ | | 1 | `regex_after_label` | Value follows a fixed label (`P.O. # 12345`) | `label`, `label_regex`, `value_regex`, `max_chars`, `case_sensitive` | | 2 | `regex_before_label` | Value precedes a label (`12345 INV`) | Mirror of 1. | | 3 | `fixed_region` | Same PDF coords every time (logos, scanned) | `page`, `x0`, `y0`, `x1`, `y1` | | 4 | `table_with_headers` | Line items in a table — header names known | `header`, `header_aliases`, `column_dtype`, `min`, `max`, `strip_currency` | | 5 | `multi_line_span` | Multi-line block bounded by markers (addresses) | `start_label`, `end_markers`, `max_lines`, `trim` | | 6 | `whole_section` | Free-text section (notes, memos) | `section_header`, `section_terminators`, `max_chars` | | 7 | `formula` | Derived from other extracted fields | `expression`, `inputs[]`, `dtype` | | 8 | `llm_with_schema` | Fuzzy / novel — fallback for unknown senders | `model`, `prompt_template`, `output_schema_json`, `max_tokens` | | 9 | `literal_constant` | Value known once sender is identified | `value` OR `resolve_from` (SQL lookup expression) | Strategies 1–7 are deterministic and free. Strategy 8 is the LLM fallback (used sparingly because it costs ~$0.001/doc and adds latency). Strategy 9 is the trick that makes per-customer templates pay off: once the sender is matched, `entity`, `subsidiary`, `terms`, `customform`, `location` are all `literal_constant` — no extraction needed. --- ## 4. NS Field Taxonomy `ns_field_taxonomy` is the single source of truth for "what NS fields exist per record type." Seeded with ~70 fields: | Record Type | Seeded Fields | Coverage | | ----------- | ------------- | ------------------------------------------------------------------- | | SalesOrd | 20 | tranid, otherrefnum, entity, trandate, shipdate, memo, shipaddress, terms, customform, subsidiary, location, department, line item/qty/rate/location/taxcode/custcol_brand, createdfrom | | PurchOrd | 17 | tranid, entity, memo, trandate, expectedreceiptdate, shipaddress, terms, customform, subsidiary, location, department, createdfrom, line item/qty/rate/location/expectedreceiptdate | | CustInvc | 11 | tranid, entity, otherrefnum, trandate, duedate, memo, terms, createdfrom, line item/qty/rate | | VendBill | 12 | tranid, entity, memo, trandate, duedate, terms, initialtranid, createdfrom, line item/qty/rate/account | | WorkOrd | 9 | tranid, assemblyitem, quantity, location, trandate, startdate, enddate, memo, createdfrom | | AsmBuild | 8 | tranid, createdfrom, item, quantity, location, trandate, memo, custbody_waste | Fields flagged `GUESS:` in `notes_md` need confirmation against the live NS instance before templates depending on them go to `status='active'`: - `SalesOrd.bodyFields.customform` — form ID per subsidiary - `SalesOrd.bodyFields.subsidiary` — default subsidiary ID - `SalesOrd.bodyFields.location` — default location ID (likely 131 Heartland) - `SalesOrd.lineFields.custcol_brand` — exact custcol_ ID on prod - `VendBill.bodyFields.initialtranid` — may be `transactionnumber` on some configs - `AsmBuild.bodyFields.custbody_waste` — exact custbody_ ID on prod --- ## 5. Tagger UI Sketch (BB-2) ``` +---------------------------------------------------------------------+ | Driscoll Foods · PO PDF → SalesOrd · v1 · ACTIVE · 47 hits | +---------------------------------------------------------------------+ | PDF Preview (page 1) | Tags (8) | | | | | +-----------------------+ | [✓] otherrefnum (regex_after_label) | | | DRISCOLL FOODS PO # | | label: "P.O. #" | | | P.O. # PO-987654 ----|---|----> sample: "PO-987654" conf: 0.99 | | | Ship To: ----------- |---|----> shipaddress (multi_line_span) | | | 125 Stevens Ave | | sample: "125 Stevens Ave\\n..." | | | Mt Vernon, NY 10550 | | [?] shipdate (regex_after_label) | | | | | label: "Delivery Date" | | | Item # | Qty | Price | | sample: "06/03/2026" conf: 0.94 | | | 10024 | 24 | $12.85 |---|----> lineFields.item, .qty, .rate | | +-----------------------+ | (table_with_headers) | | | | | [ Re-extract ] | [ + Add tag ] [ Train from doc ] | +---------------------------------------------------------------------+ | Approve → SalesOrd [Approve] [Edit] [Reject] | +---------------------------------------------------------------------+ ``` Highlighted text in the PDF pane corresponds to the selected tag's `sample_md_span_start / sample_md_span_end`. Operator can drag-select a new span to retrain a tag. --- ## 6. Reflexion Loop ``` inbound doc nightly retrain job │ ▲ ▼ │ match template ──► extract ──► extraction (pending) │ │ │ │ │ ▼ │ │ operator review ───────┤ │ │ │ │ ├─ approve ─────┴──► proposed_actions │ │ │ │ ├─ correct ───────────► correction row │ │ │ │ will_update_template=1 │ │ │ ▼ │ │ └────► (nightly) update │ │ strategy_params_json, │ │ bump version, mark │ │ applied_to_template=1 │ │ │ └─ reject ─────► extraction.status='rejected' │ (lower template confidence) ▼ hit_count++ (success_count++ on approve, failure_count++ on reject) ``` A correction with `will_update_template=1` doesn't immediately mutate the live template — it queues for nightly retraining (BB-4). This batches corrections, lets the operator review the proposed change before it goes live, and creates a clean audit trail of "v1 → v2 because of corrections #147, #163, #181." --- ## 7. Runbook — First Use Case (Driscoll PO → SO) 1. **Email arrives** at `orders@gfs` from a Driscoll address. Inbound email handler logs to `inbound_email_log`, stores PDF in R2. 2. **Match template:** `customer_match_pattern='from:*@driscollfoods.com'` resolves to `template_id=1`. 3. **PDF → markdown** via the existing doc renderer; markdown URL stored in `data_tagger_extractions.source_doc_md_r2_url`. 4. **Extract 8 fields** per the seeded tags: - `otherrefnum` ← `regex_after_label` on "P.O. #" - `entity` ← `literal_constant` (Driscoll Foods NS customer ID, resolved from `bid_customers.ns_customer_id where folder='Driscoll_Foods'`) - `shipaddress` ← `multi_line_span` from "Ship To:" to next blank line - `shipdate` ← `regex_after_label` on "Delivery Date" - `memo` ← `whole_section` "Notes" - `lineFields.item/qty/rate` ← `table_with_headers` 5. **Land in `data_tagger_extractions(status='pending')`** with `extracted_fields_json` shaped as the SalesOrd payload. 6. **Operator reviews in BB-2 UI.** Approve → `proposed_actions` → Worker calls `/api/ns/so/create` → `downstream_record_id` populated. 7. **On correction:** correction row written; operator opts in to "teach the template"; nightly job rolls v1 → v2. Success metric for v1: 80% of Driscoll POs approved as-is (no corrections) within 30 days of going live. --- ## 8. Open TBDs - **BB-2 tagger UI** — wireframe above is ASCII; needs Figma + build. - **BB-3 chat tools** — `/api/tagger/templates`, `/api/tagger/extract`, `/api/tagger/approve` not yet implemented. - **BB-4 nightly retrain job** — strategy_params_json update logic per strategy (regex tightening, header alias expansion, llm prompt refinement) not yet specified. - **Default NS IDs** — subsidiary, location, custom forms, custcol/custbody IDs flagged `GUESS:` in seed need confirmation against prod NS. - **LLM strategy budget** — when `llm_with_schema` is used, what's the per-doc cost cap before we force a human review? (Probably $0.01.) - **Confidence threshold** — what `confidence_overall` triggers auto-approve vs. mandatory review? Starting position: never auto-approve (HITL invariant from ADR-031); confidence drives UI sort order only. - **PDF coord strategy** — `fixed_region` requires a PDF coord extractor in the doc pipeline; today we only render to markdown. Coord extraction is a separate dependency. - **Multi-page PDFs** — schema supports page-level coords but the markdown pipeline currently flattens; need to preserve page anchors. - **Template versioning** — v1 → v2 is captured by `version` column but there is no "diff two versions" view yet. Mirror the `quote_version_diffs` pattern from migration 137 when needed.