GFS Platform

Runbook

Procedures, checklists, onboarding, secret rotation, incident response

Procedures
11
Incident playbooks
7
Checklists
7
Onboarding steps
9
NS nav entries
34

Quick links

Onboarding a new admin

End-to-end provisioning from "this person exists" to "they can run sync, query D1, and deploy the Worker". Update the Decisions log (ADR) with the new admin's name and starting scope.

Provisioning a new admin from scratch

  1. Confirm the new admin's identity and need-to-know scope. Decide whether they should have NS Administrator or a scoped custom role (Controller / Sales & Ops Mgr etc).
  2. In NetSuite: Setup → Users/Roles → Manage Roles. Pick or create the role. Assign permissions per least-privilege — never copy Administrator wholesale.
  3. In NetSuite: Setup → Users/Roles → Manage Users → New. Set name, email, password reset link. Assign roles. Set Subsidiary = GFS, Default Role = the one above.
  4. In Cloudflare: dash.cloudflare.com → Manage Account → Members → Invite. Send invite with the minimum role needed (Reader for view-only, otherwise narrower built-in roles).
  5. Local tools (only if the new admin will run sync): install wrangler (npm i -g wrangler), run wrangler login on their machine, confirm they appear in wrangler whoami.
  6. Chartstone Pro: hand off the localhost API key. Set CHARTSTONE_SECRET in their ~/.zshenv. Verify with curl http://127.0.0.1:56411/health on their machine.
  7. Bookmarks: send this admin guide URL, GAPS_TO_CLOSE, and the Cloudflare dashboard link. Tag them in the Decisions log (decisions.html) so they know where decisions are recorded.
  8. Knowledge handoff: 30-min walk-through of admin-dashboard, runbook (this page), and the daily/weekly checklists below. Confirm they can run a D1 query via wrangler.
  9. Update the Personnel section of this runbook with the new admin's name and scope. Record the date and starting permissions in the Decisions log (ADR).

Secret rotation

Cadenced rotation of every Bearer token and API key. Record each rotation in the Decisions log so an audit trail exists.

API_KEY (Worker) — rotate Quarterly + on suspected leak

  1. Generate new key: openssl rand -hex 32
  2. wrangler secret put API_KEY (paste new value)
  3. Update any consuming clients (admin-dashboard checks reload automatically)
  4. Test: curl -H 'Authorization: Bearer $NEW' https://api.ai-globalfoodsolutions.co/api/kpis
  5. Record rotation in Decisions log with date and operator

CHARTSTONE_SECRET (local) — rotate Quarterly + on machine change

  1. Generate in Chartstone Pro UI → API → New Token
  2. Edit ~/.zshenv: replace the CHARTSTONE_SECRET= line
  3. Reload shell: source ~/.zshenv
  4. Test: ~/Desktop/gfs-platform/sync.sh manual run; tail -20 sync.log
  5. Record rotation in Decisions log

NetSuite TBA tokens — rotate Annually + on integration removal

  1. In NS: Setup → Users/Roles → Manage Authentication Tokens
  2. Identify active tokens (SuiteAttach id 11, Analytics Warehouse id 2)
  3. Create new token under the SAME integration record
  4. Update the consuming system with the new TOKEN_ID + TOKEN_SECRET
  5. Test the integration
  6. Revoke old token only after the new one is confirmed working
  7. Record rotation in Decisions log

Cloudflare API token (if used) — rotate Quarterly

  1. dash.cloudflare.com → My Profile → API Tokens
  2. Create new token with minimum scope (zone:read, worker:edit, pages:edit, d1:edit, etc — narrow as possible)
  3. Update local ~/.cloudflare.toml or env
  4. Revoke old token

Incident response playbooks

"X is broken — do these steps." Each playbook is independent and short enough to execute under pressure.

Sync stopped (D1 stale)

  1. Check sync.log: tail -50 ~/Desktop/gfs-platform/sync.log
  2. If localhost connection refused: open Chartstone Pro and verify it's running
  3. If 'Operation not permitted': run xattr -d com.apple.provenance sync.sh; re-launch via launchd
  4. If Bearer rejected: rotate CHARTSTONE_SECRET (see secret rotation)
  5. If wrangler errors: wrangler login; wrangler whoami
  6. Manual catch-up: run sync.sh from terminal; observe stdout
  7. Verify in D1: wrangler d1 execute gfs-netsuite --remote --command 'SELECT MAX(lastmodifieddate) FROM transactions'

D1 quota or write errors

  1. Check D1 size: Cloudflare dashboard → Workers & Pages → D1 → gfs-netsuite → Metrics
  2. If over 10GB: identify largest tables — usually transactions or invoice_lines
  3. Archive: DELETE FROM transactions WHERE year < 2020 (after exporting to R2 via wrangler r2 object put)
  4. If write rate-limited: pause sync.sh temporarily (sudo launchctl unload ~/Library/LaunchAgents/gfs-sync.plist)
  5. Resume: sudo launchctl load ~/Library/LaunchAgents/gfs-sync.plist

API_KEY leaked or suspected leak

  1. Immediately rotate: wrangler secret put API_KEY (new value)
  2. wrangler tail — watch for requests with the old key (will start 401-ing)
  3. Audit access logs in Cloudflare dashboard → Workers → gfs-platform → Logs
  4. Notify anyone who legitimately had the old key with the new one
  5. Record incident in Decisions log with timeline

Cloudflare Pages site down

  1. Check status: https://www.cloudflarestatus.com
  2. If global Pages issue: wait, no action
  3. If specific to gfs-netsuite project: dash.cloudflare.com → Workers & Pages → gfs-netsuite → Deployments → Roll back to previous
  4. If build failed: check deploy logs, fix locally, re-run wrangler pages deploy

NetSuite unavailable

  1. Check status.netsuite.com
  2. If NS down: sync.sh will fail silently (Chartstone returns empty results). D1 keeps the last-known data.
  3. Admin dashboard data quality grades remain valid; live KPIs (when implemented) will show 'stale'
  4. No action until NS is restored. After restoration, sync.sh catches up on its next 15-min run.

Period close blocked

  1. Run the period-close SuiteQL admin query (see data-model → SuiteQL admin library → Period close)
  2. Identify unposted transactions in the period
  3. Either post them OR move them to a different period after verifying impact
  4. Re-run close from NS UI: Setup → Accounting → Manage Accounting Periods → Lock the period

Suitelet 500 error (public broker portal)

  1. Check the Suitelet's script execution log in NS: Customization → Scripting → Script Execution Log
  2. Identify the error — usually a query that timed out or a NULL where a value was expected
  3. If a downstream consumer broke (a broker missing data), notify the broker contact
  4. Fix in the Suitelet code (File Cabinet folder 717413), redeploy
  5. If urgent: take the Suitelet offline (uncheck isonline=T) until fixed

Pricing pipeline procedures

Operating the customer pricing derive → review → push workflow. Reference: ADR-012, pricing roadmap, proposed schema.

Run the pricing derive job (rebuild proposed price list)

  1. Confirm last derive: GET /api/pricing/audit/summary → check last_derived_at
  2. Trigger fresh derive: GET /api/pricing/derive (Bearer auth, ~5–10 sec)
  3. Or via SQL direct: wrangler d1 execute gfs-netsuite --remote --file=migrations/002_run_derive.sql
  4. Verify: GET /api/pricing/audit/summary → derived_count should match expected (~600–800 for current data)
  5. Spot-check: GET /api/pricing/customer/539 (Driscoll Foods) → confirm proposed_count, sample row deltas

Review the three-bucket pricing diff

  1. Pull summary: GET /api/pricing/diff (all customers) or /api/pricing/diff?customer=:id (single)
  2. Listed+Recent: rows that exist in NS + were recently invoiced — drift check (sort by ABS(delta_pct) DESC)
  3. Listed+Stale: rows in NS but no recent invoice — candidates to deprecate (customer hasn't bought in 18+ months)
  4. Not-Listed+Recent: customer is buying but not on NS price list — biggest gap to close, captures negotiated SO-time prices
  5. Cross-reference with open SOs: any in-flight SOs for these customers may be affected at cutover

Identify open SOs that would be affected by a pricing cutover

  1. wrangler d1 execute gfs-netsuite --remote --command "SELECT entity_name, COUNT(*) AS sos, SUM(foreigntotal) AS total FROM transactions WHERE type='SalesOrd' AND status IN ('A','B','D','E','F') GROUP BY entity_name ORDER BY sos DESC"
  2. Currently 90 open SOs across 33 customers — these would need to either re-price or invoice on old prices
  3. Phase 4 will surface this list per customer before pushing to NS
  4. Decision: per-batch, either re-price open SOs (customer notification + admin update) or let them invoice on existing prices (cutover applies to SOs created after timestamp)

Push approved prices back to NS (Phase 4 — gated)

  1. Phase 4 not yet implemented. Steps below describe planned procedure.
  2. Pre-flight: confirm SuiteAPI RESTlet endpoint live + Bearer token rotated
  3. Pilot on 3 customers first: POST /api/pricing/push-to-ns with {customer_ids: [...]}
  4. Verify in NS UI: Customer → Item Prices → confirm rows replaced
  5. Create next-day SO and confirm pricing pulls from the new entries
  6. Bulk: push remaining approved rows in batches of 10–20 customers
  7. Each push writes a row to customer_pricing_history for audit
  8. Rollback if needed: customer_pricing_history holds the prior price; restore via SuiteAPI

Flag and resolve pricing anomalies (Phase 6 — gated)

  1. Phase 6 not yet implemented. Steps below describe planned procedure.
  2. AI review pass populates pricing_anomalies with severity (low/medium/high)
  3. Anomaly types: drift (proposed differs from current > threshold), peer_outlier (customer paying outside their program band), one_off_suspicious (last_paid below median by > 15%), stale (no order in 12+ months)
  4. Review queue: GET /api/pricing/anomalies?resolved=0
  5. Resolve: mark resolved=1 with note (accepted, investigated, corrected, deferred)

Day-to-day procedures

Recovered from the original ops dashboard. Each is a stepwise procedure for a recurring admin task.

Deploy Worker Changes

  1. Edit source:
  2. ~/Desktop/gfs-platform/src/index.ts
  3. Test locally:
  4. cd ~/Desktop/gfs-platform && npm run dev
  5. Opens local dev server at localhost:8787. Test endpoints.
  6. Deploy:
  7. npm run deploy
  8. Runs
  9. wrangler deploy
  10. . Deploys to gfs-platform.mikelevine.workers.dev
  11. Verify:
  12. Hit /api/health and one auth endpoint to confirm

Deploy Dashboard to Pages

  1. Prepare deploy folder:
  2. rm -rf /tmp/gfs-deploy && mkdir -p /tmp/gfs-deploy/docs /tmp/gfs-deploy/diagrams
  3. Copy files:
  4. cp guide/*.html guide/*.css /tmp/gfs-deploy/ && cp guide/diagrams/*.html /tmp/gfs-deploy/diagrams/ && cp dashboard/index.html /tmp/gfs-deploy/command-center.html && cp docs/04-Power-Tools.html /tmp/gfs-deploy/docs/
  5. Deploy:
  6. wrangler pages deploy /tmp/gfs-deploy --project-name gfs-system-guide
  7. Verify:
  8. Check gfs-system-guide.pages.dev

Sync Troubleshooting

  1. Sync not running
  2. Check launchd:
  3. launchctl list | grep gfs
  4. Should show two entries (sync every 900s, report at 7pm).
  5. Check Chartstone:
  6. Open Chartstone Pro, verify it's running on port 56411
  7. curl -s http://127.0.0.1:56411/health
  8. — should return 200.
  9. Check Full Disk Access:
  10. System Settings → Privacy → Full Disk Access → Terminal.app
  11. macOS Sequoia blocks launchd scripts without this.
  12. Check wrangler auth:
  13. wrangler whoami
  14. Should show mikelevine@gfs account. Re-login:
  15. wrangler login
  16. Check sync log:
  17. tail -20 ~/Desktop/gfs-platform/sync.log
  18. Sync running but no data
  19. Check Chartstone secret:
  20. grep CHARTSTONE_SECRET ~/.zshenv
  21. Test query manually:
  22. Run the SuiteQL from sync.sh against Chartstone directly
  23. Check D1 write:
  24. wrangler d1 execute gfs-netsuite --remote --command="SELECT COUNT(*) FROM sync_log"

D1 Database Operations

  1. Query D1 directly
  2. wrangler d1 execute gfs-netsuite --remote --command="YOUR SQL HERE"
  3. Backup D1 (manual export)
  4. wrangler d1 execute gfs-netsuite --remote --command="SELECT * FROM customers" --json > backup_customers.json
  5. Repeat for each table. No native D1 full-export yet. Automate with a shell script.
  6. Reload a table from SQL loader
  7. wrangler d1 execute gfs-netsuite --remote --file=sql/01_customers.sql
  8. Loader files use INSERT OR REPLACE. Safe to re-run.
  9. Check database size
  10. Current:
  11. ~28.3 MB of 10 GB limit (0.3% used). At current growth, years of runway.
  12. Monitor quarterly. D1 limit is 10 GB per database.

Secrets & Credentials

  1. Worker API_KEY
  2. Set via: wrangler secret put API_KEY
  3. Chartstone secret
  4. In ~/.zshenv as CHARTSTONE_SECRET
  5. Wrangler auth
  6. OAuth token — wrangler login to refresh
  7. NS credentials
  8. In Chartstone Pro config
  9. CF account
  10. mikelevine — dashboard.cloudflare.com
  11. NS account
  12. 4656898 — system.netsuite.com
  13. Rotation schedule
  14. API_KEY:
  15. Rotate quarterly. Update Worker secret + dashboard sessionStorage clears automatically.
  16. Chartstone:
  17. Rotate if laptop changes. Update ~/.zshenv.
  18. NS password:
  19. Per company policy. Update Chartstone config after change.

Emergency Recovery

  1. Worker down
  2. Check CF status:
  3. cloudflarestatus.com — verify no platform outage
  4. Check Worker logs:
  5. CF Dashboard → Workers → gfs-platform → Logs
  6. Redeploy:
  7. cd ~/Desktop/gfs-platform && npm run deploy
  8. D1 data corruption
  9. Reload from SQL loaders:
  10. Files in sql/ directory are the original full dataset
  11. Incremental data:
  12. entity backfill, GL accounts, vb_lines added post-load would need re-sync
  13. Laptop failure (total loss)
  14. Worker + D1 + KV + R2 are in Cloudflare
  15. — survive laptop loss
  16. Source code:
  17. once git pushed to remote, recoverable
  18. Until then: this is the single copy. git init + remote push is critical.
  19. Sync will stop
  20. — sync.sh runs on laptop. Server-side sync (Phase 3) eliminates this risk.

Cadenced checklists

Daily / weekly / monthly / quarterly / annual reviews. Recovered from the original ops dashboard.

Daily — Morning (7:00 – 9:00 AM) (20 items)
  • Review KPIs on Command Center
  • Open this dashboard → Dashboard tab. Check: Revenue YTD pace, Open AR total (flag if > $2.5M), Open SO count. Takes 2 min.
  • Check sync health
  • System tab → Sync Log. Verify last sync was
  • 1 hour, check launchd:
  • launchctl list | grep gfs
  • Review new orders overnight
  • NS → Transactions → Sales → Sales Order List. Filter: Date = Today, Status = Pending Fulfillment. Route to warehouse.
  • Check overdue AR
  • Collections tab → Aging table. Any new entries in 61-90 or 90+ columns = escalation needed. Flag for afternoon call.
  • Process pending POs
  • NS → Transactions → Purchases → Purchase Order List. Filter: Status = Pending Receipt. Verify with warehouse on expected deliveries.
  • Check email for vendor confirmations
  • Match PO confirmations to open POs. Update expected receipt dates if changed.
Daily — End of Day (4:00 – 5:00 PM) (15 items)
  • Confirm all shipments processed
  • NS → Transactions → Sales → Item Fulfillment List. Filter: Date = Today. Verify tracking numbers entered.
  • Invoice shipped orders
  • NS → Transactions → Sales → Invoice List. Create invoices for all fulfilled SOs. Verify terms match customer record.
  • Apply received payments
  • NS → Transactions → Customers → Accept Payment. Match checks/ACH/wire to open invoices.
  • Bank deposit if applicable
  • NS → Transactions → Bank → Make Deposits. Group payments received today.
  • Review daily report at 7pm
  • Auto-generated by daily-report.sh (email delivery coming in Phase 3). Compare to morning KPIs.
Weekly — Monday (18 items)
  • Full AR aging review
  • Collections tab or NS → Reports → Financial → A/R Aging. Identify accounts for escalation. Send reminder emails for 30+ days. Owner: Amanda S.
  • Vendor bill reconciliation
  • NS → Reports → Payables → Open Vendor Bills. Match against POs and receipts. Flag discrepancies. Owner: Amanda S.
  • Inventory spot check
  • NS → Reports → Inventory → Inventory Valuation. Check negative items list. Review pending WOs (currently 0). Owner: Elena M.
  • CME pricing review
  • Check CME block/barrel prices. Update Bongards formula calc (trailing week avg + 35% moisture). Notify affected customers of changes. Owner: Michael L.
  • Sales pipeline review
  • Revenue tab → Customer Ranking. Compare to prior week. Review open SOs > $10K. Update forecast. Owner: Sales team.
  • Open PO follow-up
  • NS → Transactions → Purchases → PO List. Filter: overdue POs. Contact vendors for ETA. Owner: Buyer.
Weekly — Friday (9 items)
  • Week-end revenue snapshot
  • Dashboard → Revenue YTD. Compare to last week's number. Calculate weekly run rate vs $29M annual pace.
  • Prepare next week's ship schedule
  • NS → Sales Order List. Filter: Ship Date = next week. Ensure warehouse capacity. Flag large orders.
  • Vendor payment run
  • NS → Transactions → Payables → Pay Bills. Select bills due this week. Process payment batch.
Monthly Close — By 10th of Following Month (69 items)
  • Phase 1: Days 1-3 — Transaction Cleanup
  • Final invoicing
  • — invoice ALL fulfilled SOs from prior month
  • Run saved search: Fulfilled SOs with no invoice. Create invoices. Zero should remain.
  • Apply all payments
  • — zero unapplied payments
  • NS → Customers → Unapplied Payments. Match to invoices or create on-account credit.
  • Bank reconciliation
  • — match NS to bank statement
  • NS → Transactions → Bank → Reconcile. All cleared items should match. Investigate discrepancies.
  • Credit card reconciliation
  • — verify all charges posted
  • Phase 2: Days 3-5 — Accruals & Adjustments
  • AP accruals
  • — accrue for received-not-billed items
  • Create JE: Debit expense, Credit accrued AP. Reverse in new period.
  • Prepaid expense amortization
  • — recognize monthly portions
  • Inventory adjustments
  • — resolve negative items, write-offs
  • NS → Transactions → Inventory → Adjust Inventory. Document reason for each adjustment.
  • Rebate accruals
  • — estimate earned rebates by customer
  • Calculate based on volume tiers. JE: Debit rebate expense, Credit accrued rebate liability.
  • Phase 3: Days 5-7 — Revenue & Review
  • Revenue recognition review
  • — verify all revenue properly recorded
  • Match invoice totals to revenue GL accounts. Investigate variances.
  • Intercompany/related party entries
  • — if applicable
  • Review all journal entries
  • — verify proper documentation
  • NS → Reports → Financial → General Ledger. Filter: JE type, current period.
  • Phase 4: Days 7-9 — Reports & Verification
  • Trial balance review
  • — all accounts balanced
  • NS → Reports → Financial → Trial Balance. Compare to prior month. Investigate swings > 10%.
  • P&L review
  • — revenue, COGS, gross margin, operating expenses
  • Compare to budget. Flag variances. Prepare management commentary.
  • Balance sheet review
  • — assets, liabilities, equity tie-out
  • Cash flow statement
  • — verify operating, investing, financing
  • Phase 5: Day 10 — Close
  • Close accounting period
  • Setup → Accounting → Manage Accounting Periods → select month → Close All. Prevents further posting.
  • Distribute management reports
  • — P&L, Balance Sheet, AR Aging
  • Update forecast
  • — adjust annual projection based on actuals
Quarterly (16 items)
  • Customer tier review
  • — reclassify by volume
  • Compare YTD volume to tier thresholds. Update pricing levels. Notify sales of changes.
  • Vendor performance review
  • — on-time delivery, quality, pricing
  • Procurement tab → Vendor Spend. Compare to prior quarter. Flag underperformers.
  • 1099 vendor review
  • — verify eligibility and tax IDs
  • NS → Lists → Relationships → Vendors. Filter: Is 1099 Eligible = Yes. Verify W-9 on file.
  • Saved search cleanup
  • — archive unused, consolidate duplicates
  • 968 searches exist. ~300 are TAF/SII/Intrastat from unused bundles. ~20 are dead Zapier/Airtable.
Annual (16 items)
  • 1099 filing
  • — generate and file by January 31
  • NS → Reports → Financial → 1099 Report. Verify all eligible vendors have tax ID.
  • Customer contract renewal review
  • Identify all contracts expiring in next 90 days. Schedule renewal discussions. Update pricing.
  • NS role and access audit
  • 97 roles, 116 employees. Verify terminated employees disabled. Review admin access (currently concentrated on 1 user).
  • Budget preparation
  • — build next year's budget in NS
  • Use Revenue tab data + vendor spend analysis for informed projections.
  • Data Quality Dashboard
  • — Every entity type graded A through F based on field completeness. Each F-grade item has an assigned owner and remediation plan. Data quality directly impacts every workflow — bad data = broken processes.

NetSuite navigation map

Common paths into the NetSuite UI for the records and configuration areas this platform touches.

Transactions

Record / ActionPath
Sales OrdersTransactions → Sales → Enter Sales Orders
InvoicesTransactions → Sales → Create Invoices
Item FulfillmentTransactions → Sales → Fulfill Orders
Customer PaymentsTransactions → Customers → Accept Payment
Credit MemosTransactions → Customers → Issue Credit Memo
Purchase OrdersTransactions → Purchases → Enter POs
Item ReceiptsTransactions → Purchases → Receive Orders
Vendor BillsTransactions → Payables → Enter Bills
Bill PaymentsTransactions → Payables → Pay Bills
Journal EntriesTransactions → Financial → Make JE
DepositsTransactions → Bank → Make Deposits
Inventory AdjustTransactions → Inventory → Adjust
Work OrdersTransactions → Manufacturing → Work Orders
Assembly BuildsTransactions → Manufacturing → Build

Records & Lists

Record / ActionPath
CustomersLists → Relationships → Customers
VendorsLists → Relationships → Vendors
ContactsLists → Relationships → Contacts
ItemsLists → Accounting → Items
EmployeesLists → Employees → Employees
GL AccountsLists → Accounting → Chart of Accounts
LocationsLists → Accounting → Locations
DepartmentsLists → Accounting → Departments
ClassesLists → Accounting → Classes
Payment TermsLists → Accounting → Terms

Admin & Setup

Record / ActionPath
Accounting PeriodsSetup → Accounting → Manage Periods
RolesSetup → Users/Roles → Manage Roles
UsersSetup → Users/Roles → Manage Users
ScriptsCustomization → Scripting → Scripts
Script DeploysCustomization → Scripting → Deployments
Saved SearchesReports → Saved Searches → All
WorkflowsCustomization → Workflow → Workflows
Custom FieldsCustomization → Lists/Records/Fields
SuiteQL ToolCustomization → Scripting → Suitelets → SuiteQL #2947
Import CSVSetup → Import/Export → Import CSV

Source

  • data/runbook-source.json — procedures + checklists extracted from the original ops dashboard
  • data/ns-navigation.json — NetSuite navigation map
  • Onboarding, secret rotation, incident response playbooks authored in this build pass

Related surfaces

Admin dashboard

Live health, sync status, risk register, known issues, pricing pipeline status.

Infrastructure

Worker endpoints (per-endpoint detail), cron, bindings, secrets map, DNS records.

Decision log

11 ADRs covering platform architecture, AI stack, pricing pipeline.

Gaps to close

Open-items tracker across 4 tiers — Tier 1 critical includes Access on Pages.

Data model

D1 schema, ER diagram, FK relationships, example queries, SuiteQL admin library.