GFS Platform

Runbook

Procedures, checklists, onboarding, secret rotation, incident response

Procedures
11
Incident playbooks
7
Checklists
7
Onboarding steps
9
NS nav entries
34

Quick links

Onboarding a new admin

End-to-end provisioning from "this person exists" to "they can run sync, query D1, and deploy the Worker". Update the Decisions log (ADR) with the new admin's name and starting scope.

Provisioning a new admin from scratch

  1. Confirm the new admin's identity and need-to-know scope. Decide whether they should have NS Administrator or a scoped custom role (Controller / Sales & Ops Mgr etc).
  2. In NetSuite: Setup → Users/Roles → Manage Roles. Pick or create the role. Assign permissions per least-privilege — never copy Administrator wholesale.
  3. In NetSuite: Setup → Users/Roles → Manage Users → New. Set name, email, password reset link. Assign roles. Set Subsidiary = GFS, Default Role = the one above.
  4. In Cloudflare: dash.cloudflare.com → Manage Account → Members → Invite. Send invite with the minimum role needed (Reader for view-only, otherwise narrower built-in roles).
  5. Local tools (only if the new admin will run sync): install wrangler (npm i -g wrangler), run wrangler login on their machine, confirm they appear in wrangler whoami.
  6. Chartstone Pro: hand off the localhost API key. Set CHARTSTONE_SECRET in their ~/.zshenv. Verify with curl http://127.0.0.1:56411/health on their machine.
  7. Bookmarks: send this admin guide URL, GAPS_TO_CLOSE, and the Cloudflare dashboard link. Tag them in the Decisions log (decisions.html) so they know where decisions are recorded.
  8. Knowledge handoff: 30-min walk-through of admin-dashboard, runbook (this page), and the daily/weekly checklists below. Confirm they can run a D1 query via wrangler.
  9. Update the Personnel section of this runbook with the new admin's name and scope. Record the date and starting permissions in the Decisions log (ADR).

Secret rotation

Cadenced rotation of every Bearer token and API key. Record each rotation in the Decisions log so an audit trail exists.

API_KEY (Worker) — rotate Quarterly + on suspected leak

  1. Generate new key: openssl rand -hex 32
  2. wrangler secret put API_KEY (paste new value)
  3. Update any consuming clients (admin-dashboard checks reload automatically)
  4. Test: curl -H 'Authorization: Bearer $NEW' https://api.ai-globalfoodsolutions.co/api/kpis
  5. Record rotation in Decisions log with date and operator

CHARTSTONE_SECRET (local) — rotate Quarterly + on machine change

  1. Generate in Chartstone Pro UI → API → New Token
  2. Edit ~/.zshenv: replace the CHARTSTONE_SECRET= line
  3. Reload shell: source ~/.zshenv
  4. Test: ~/Desktop/gfs-platform/sync.sh manual run; tail -20 sync.log
  5. Record rotation in Decisions log

NetSuite TBA tokens — rotate Annually + on integration removal

  1. In NS: Setup → Users/Roles → Manage Authentication Tokens
  2. Identify active tokens (SuiteAttach id 11, Analytics Warehouse id 2)
  3. Create new token under the SAME integration record
  4. Update the consuming system with the new TOKEN_ID + TOKEN_SECRET
  5. Test the integration
  6. Revoke old token only after the new one is confirmed working
  7. Record rotation in Decisions log

Cloudflare API token (if used) — rotate Quarterly

  1. dash.cloudflare.com → My Profile → API Tokens
  2. Create new token with minimum scope (zone:read, worker:edit, pages:edit, d1:edit, etc — narrow as possible)
  3. Update local ~/.cloudflare.toml or env
  4. Revoke old token

Incident response playbooks

"X is broken — do these steps." Each playbook is independent and short enough to execute under pressure.

Sync stopped (D1 stale)

  1. Check sync.log: tail -50 ~/Desktop/gfs-platform/sync.log
  2. If localhost connection refused: open Chartstone Pro and verify it's running
  3. If 'Operation not permitted': run xattr -d com.apple.provenance sync.sh; re-launch via launchd
  4. If Bearer rejected: rotate CHARTSTONE_SECRET (see secret rotation)
  5. If wrangler errors: wrangler login; wrangler whoami
  6. Manual catch-up: run sync.sh from terminal; observe stdout
  7. Verify in D1: wrangler d1 execute gfs-netsuite --remote --command 'SELECT MAX(lastmodifieddate) FROM transactions'

D1 quota or write errors

  1. Check D1 size: Cloudflare dashboard → Workers & Pages → D1 → gfs-netsuite → Metrics
  2. If over 10GB: identify largest tables — usually transactions or invoice_lines
  3. Archive: DELETE FROM transactions WHERE year < 2020 (after exporting to R2 via wrangler r2 object put)
  4. If write rate-limited: pause sync.sh temporarily (sudo launchctl unload ~/Library/LaunchAgents/gfs-sync.plist)
  5. Resume: sudo launchctl load ~/Library/LaunchAgents/gfs-sync.plist

API_KEY leaked or suspected leak

  1. Immediately rotate: wrangler secret put API_KEY (new value)
  2. wrangler tail — watch for requests with the old key (will start 401-ing)
  3. Audit access logs in Cloudflare dashboard → Workers → gfs-platform → Logs
  4. Notify anyone who legitimately had the old key with the new one
  5. Record incident in Decisions log with timeline

Cloudflare Pages site down

  1. Check status: https://www.cloudflarestatus.com
  2. If global Pages issue: wait, no action
  3. If specific to gfs-netsuite project: dash.cloudflare.com → Workers & Pages → gfs-netsuite → Deployments → Roll back to previous
  4. If build failed: check deploy logs, fix locally, re-run wrangler pages deploy

NetSuite unavailable

  1. Check status.netsuite.com
  2. If NS down: sync.sh will fail silently (Chartstone returns empty results). D1 keeps the last-known data.
  3. Admin dashboard data quality grades remain valid; live KPIs (when implemented) will show 'stale'
  4. No action until NS is restored. After restoration, sync.sh catches up on its next 15-min run.

Period close blocked

  1. Run the period-close SuiteQL admin query (see data-model → SuiteQL admin library → Period close)
  2. Identify unposted transactions in the period
  3. Either post them OR move them to a different period after verifying impact
  4. Re-run close from NS UI: Setup → Accounting → Manage Accounting Periods → Lock the period

Suitelet 500 error (public broker portal)

  1. Check the Suitelet's script execution log in NS: Customization → Scripting → Script Execution Log
  2. Identify the error — usually a query that timed out or a NULL where a value was expected
  3. If a downstream consumer broke (a broker missing data), notify the broker contact
  4. Fix in the Suitelet code (File Cabinet folder 717413), redeploy
  5. If urgent: take the Suitelet offline (uncheck isonline=T) until fixed

Day-to-day procedures

Recovered from the original ops dashboard. Each is a stepwise procedure for a recurring admin task.

Deploy Worker Changes

  1. Edit source:
  2. ~/Desktop/gfs-platform/src/index.ts
  3. Test locally:
  4. cd ~/Desktop/gfs-platform && npm run dev
  5. Opens local dev server at localhost:8787. Test endpoints.
  6. Deploy:
  7. npm run deploy
  8. Runs
  9. wrangler deploy
  10. . Deploys to gfs-platform.mikelevine.workers.dev
  11. Verify:
  12. Hit /api/health and one auth endpoint to confirm

Deploy Dashboard to Pages

  1. Prepare deploy folder:
  2. rm -rf /tmp/gfs-deploy && mkdir -p /tmp/gfs-deploy/docs /tmp/gfs-deploy/diagrams
  3. Copy files:
  4. cp guide/*.html guide/*.css /tmp/gfs-deploy/ && cp guide/diagrams/*.html /tmp/gfs-deploy/diagrams/ && cp dashboard/index.html /tmp/gfs-deploy/command-center.html && cp docs/04-Power-Tools.html /tmp/gfs-deploy/docs/
  5. Deploy:
  6. wrangler pages deploy /tmp/gfs-deploy --project-name gfs-system-guide
  7. Verify:
  8. Check gfs-system-guide.pages.dev

Sync Troubleshooting

  1. Sync not running
  2. Check launchd:
  3. launchctl list | grep gfs
  4. Should show two entries (sync every 900s, report at 7pm).
  5. Check Chartstone:
  6. Open Chartstone Pro, verify it's running on port 56411
  7. curl -s http://127.0.0.1:56411/health
  8. — should return 200.
  9. Check Full Disk Access:
  10. System Settings → Privacy → Full Disk Access → Terminal.app
  11. macOS Sequoia blocks launchd scripts without this.
  12. Check wrangler auth:
  13. wrangler whoami
  14. Should show mikelevine@gfs account. Re-login:
  15. wrangler login
  16. Check sync log:
  17. tail -20 ~/Desktop/gfs-platform/sync.log
  18. Sync running but no data
  19. Check Chartstone secret:
  20. grep CHARTSTONE_SECRET ~/.zshenv
  21. Test query manually:
  22. Run the SuiteQL from sync.sh against Chartstone directly
  23. Check D1 write:
  24. wrangler d1 execute gfs-netsuite --remote --command="SELECT COUNT(*) FROM sync_log"

D1 Database Operations

  1. Query D1 directly
  2. wrangler d1 execute gfs-netsuite --remote --command="YOUR SQL HERE"
  3. Backup D1 (manual export)
  4. wrangler d1 execute gfs-netsuite --remote --command="SELECT * FROM customers" --json > backup_customers.json
  5. Repeat for each table. No native D1 full-export yet. Automate with a shell script.
  6. Reload a table from SQL loader
  7. wrangler d1 execute gfs-netsuite --remote --file=sql/01_customers.sql
  8. Loader files use INSERT OR REPLACE. Safe to re-run.
  9. Check database size
  10. Current:
  11. ~28.3 MB of 10 GB limit (0.3% used). At current growth, years of runway.
  12. Monitor quarterly. D1 limit is 10 GB per database.

Secrets & Credentials

  1. Worker API_KEY
  2. Set via: wrangler secret put API_KEY
  3. Chartstone secret
  4. In ~/.zshenv as CHARTSTONE_SECRET
  5. Wrangler auth
  6. OAuth token — wrangler login to refresh
  7. NS credentials
  8. In Chartstone Pro config
  9. CF account
  10. mikelevine — dashboard.cloudflare.com
  11. NS account
  12. 4656898 — system.netsuite.com
  13. Rotation schedule
  14. API_KEY:
  15. Rotate quarterly. Update Worker secret + dashboard sessionStorage clears automatically.
  16. Chartstone:
  17. Rotate if laptop changes. Update ~/.zshenv.
  18. NS password:
  19. Per company policy. Update Chartstone config after change.

Emergency Recovery

  1. Worker down
  2. Check CF status:
  3. cloudflarestatus.com — verify no platform outage
  4. Check Worker logs:
  5. CF Dashboard → Workers → gfs-platform → Logs
  6. Redeploy:
  7. cd ~/Desktop/gfs-platform && npm run deploy
  8. D1 data corruption
  9. Reload from SQL loaders:
  10. Files in sql/ directory are the original full dataset
  11. Incremental data:
  12. entity backfill, GL accounts, vb_lines added post-load would need re-sync
  13. Laptop failure (total loss)
  14. Worker + D1 + KV + R2 are in Cloudflare
  15. — survive laptop loss
  16. Source code:
  17. once git pushed to remote, recoverable
  18. Until then: this is the single copy. git init + remote push is critical.
  19. Sync will stop
  20. — sync.sh runs on laptop. Server-side sync (Phase 3) eliminates this risk.

Cadenced checklists

Daily / weekly / monthly / quarterly / annual reviews. Recovered from the original ops dashboard.

Daily — Morning (7:00 – 9:00 AM) (20 items)
  • Review KPIs on Command Center
  • Open this dashboard → Dashboard tab. Check: Revenue YTD pace, Open AR total (flag if > $2.5M), Open SO count. Takes 2 min.
  • Check sync health
  • System tab → Sync Log. Verify last sync was
  • 1 hour, check launchd:
  • launchctl list | grep gfs
  • Review new orders overnight
  • NS → Transactions → Sales → Sales Order List. Filter: Date = Today, Status = Pending Fulfillment. Route to warehouse.
  • Check overdue AR
  • Collections tab → Aging table. Any new entries in 61-90 or 90+ columns = escalation needed. Flag for afternoon call.
  • Process pending POs
  • NS → Transactions → Purchases → Purchase Order List. Filter: Status = Pending Receipt. Verify with warehouse on expected deliveries.
  • Check email for vendor confirmations
  • Match PO confirmations to open POs. Update expected receipt dates if changed.
Daily — End of Day (4:00 – 5:00 PM) (15 items)
  • Confirm all shipments processed
  • NS → Transactions → Sales → Item Fulfillment List. Filter: Date = Today. Verify tracking numbers entered.
  • Invoice shipped orders
  • NS → Transactions → Sales → Invoice List. Create invoices for all fulfilled SOs. Verify terms match customer record.
  • Apply received payments
  • NS → Transactions → Customers → Accept Payment. Match checks/ACH/wire to open invoices.
  • Bank deposit if applicable
  • NS → Transactions → Bank → Make Deposits. Group payments received today.
  • Review daily report at 7pm
  • Auto-generated by daily-report.sh (email delivery coming in Phase 3). Compare to morning KPIs.
Weekly — Monday (18 items)
  • Full AR aging review
  • Collections tab or NS → Reports → Financial → A/R Aging. Identify accounts for escalation. Send reminder emails for 30+ days. Owner: Amanda S.
  • Vendor bill reconciliation
  • NS → Reports → Payables → Open Vendor Bills. Match against POs and receipts. Flag discrepancies. Owner: Amanda S.
  • Inventory spot check
  • NS → Reports → Inventory → Inventory Valuation. Check negative items list. Review pending WOs (currently 0). Owner: Elena M.
  • CME pricing review
  • Check CME block/barrel prices. Update Bongards formula calc (trailing week avg + 35% moisture). Notify affected customers of changes. Owner: Michael L.
  • Sales pipeline review
  • Revenue tab → Customer Ranking. Compare to prior week. Review open SOs > $10K. Update forecast. Owner: Sales team.
  • Open PO follow-up
  • NS → Transactions → Purchases → PO List. Filter: overdue POs. Contact vendors for ETA. Owner: Buyer.
Weekly — Friday (9 items)
  • Week-end revenue snapshot
  • Dashboard → Revenue YTD. Compare to last week's number. Calculate weekly run rate vs $29M annual pace.
  • Prepare next week's ship schedule
  • NS → Sales Order List. Filter: Ship Date = next week. Ensure warehouse capacity. Flag large orders.
  • Vendor payment run
  • NS → Transactions → Payables → Pay Bills. Select bills due this week. Process payment batch.
Monthly Close — By 10th of Following Month (69 items)
  • Phase 1: Days 1-3 — Transaction Cleanup
  • Final invoicing
  • — invoice ALL fulfilled SOs from prior month
  • Run saved search: Fulfilled SOs with no invoice. Create invoices. Zero should remain.
  • Apply all payments
  • — zero unapplied payments
  • NS → Customers → Unapplied Payments. Match to invoices or create on-account credit.
  • Bank reconciliation
  • — match NS to bank statement
  • NS → Transactions → Bank → Reconcile. All cleared items should match. Investigate discrepancies.
  • Credit card reconciliation
  • — verify all charges posted
  • Phase 2: Days 3-5 — Accruals & Adjustments
  • AP accruals
  • — accrue for received-not-billed items
  • Create JE: Debit expense, Credit accrued AP. Reverse in new period.
  • Prepaid expense amortization
  • — recognize monthly portions
  • Inventory adjustments
  • — resolve negative items, write-offs
  • NS → Transactions → Inventory → Adjust Inventory. Document reason for each adjustment.
  • Rebate accruals
  • — estimate earned rebates by customer
  • Calculate based on volume tiers. JE: Debit rebate expense, Credit accrued rebate liability.
  • Phase 3: Days 5-7 — Revenue & Review
  • Revenue recognition review
  • — verify all revenue properly recorded
  • Match invoice totals to revenue GL accounts. Investigate variances.
  • Intercompany/related party entries
  • — if applicable
  • Review all journal entries
  • — verify proper documentation
  • NS → Reports → Financial → General Ledger. Filter: JE type, current period.
  • Phase 4: Days 7-9 — Reports & Verification
  • Trial balance review
  • — all accounts balanced
  • NS → Reports → Financial → Trial Balance. Compare to prior month. Investigate swings > 10%.
  • P&L review
  • — revenue, COGS, gross margin, operating expenses
  • Compare to budget. Flag variances. Prepare management commentary.
  • Balance sheet review
  • — assets, liabilities, equity tie-out
  • Cash flow statement
  • — verify operating, investing, financing
  • Phase 5: Day 10 — Close
  • Close accounting period
  • Setup → Accounting → Manage Accounting Periods → select month → Close All. Prevents further posting.
  • Distribute management reports
  • — P&L, Balance Sheet, AR Aging
  • Update forecast
  • — adjust annual projection based on actuals
Quarterly (16 items)
  • Customer tier review
  • — reclassify by volume
  • Compare YTD volume to tier thresholds. Update pricing levels. Notify sales of changes.
  • Vendor performance review
  • — on-time delivery, quality, pricing
  • Procurement tab → Vendor Spend. Compare to prior quarter. Flag underperformers.
  • 1099 vendor review
  • — verify eligibility and tax IDs
  • NS → Lists → Relationships → Vendors. Filter: Is 1099 Eligible = Yes. Verify W-9 on file.
  • Saved search cleanup
  • — archive unused, consolidate duplicates
  • 968 searches exist. ~300 are TAF/SII/Intrastat from unused bundles. ~20 are dead Zapier/Airtable.
Annual (16 items)
  • 1099 filing
  • — generate and file by January 31
  • NS → Reports → Financial → 1099 Report. Verify all eligible vendors have tax ID.
  • Customer contract renewal review
  • Identify all contracts expiring in next 90 days. Schedule renewal discussions. Update pricing.
  • NS role and access audit
  • 97 roles, 116 employees. Verify terminated employees disabled. Review admin access (currently concentrated on 1 user).
  • Budget preparation
  • — build next year's budget in NS
  • Use Revenue tab data + vendor spend analysis for informed projections.
  • Data Quality Dashboard
  • — Every entity type graded A through F based on field completeness. Each F-grade item has an assigned owner and remediation plan. Data quality directly impacts every workflow — bad data = broken processes.

NetSuite navigation map

Common paths into the NetSuite UI for the records and configuration areas this platform touches.

Transactions

Record / ActionPath
Sales OrdersTransactions → Sales → Enter Sales Orders
InvoicesTransactions → Sales → Create Invoices
Item FulfillmentTransactions → Sales → Fulfill Orders
Customer PaymentsTransactions → Customers → Accept Payment
Credit MemosTransactions → Customers → Issue Credit Memo
Purchase OrdersTransactions → Purchases → Enter POs
Item ReceiptsTransactions → Purchases → Receive Orders
Vendor BillsTransactions → Payables → Enter Bills
Bill PaymentsTransactions → Payables → Pay Bills
Journal EntriesTransactions → Financial → Make JE
DepositsTransactions → Bank → Make Deposits
Inventory AdjustTransactions → Inventory → Adjust
Work OrdersTransactions → Manufacturing → Work Orders
Assembly BuildsTransactions → Manufacturing → Build

Records & Lists

Record / ActionPath
CustomersLists → Relationships → Customers
VendorsLists → Relationships → Vendors
ContactsLists → Relationships → Contacts
ItemsLists → Accounting → Items
EmployeesLists → Employees → Employees
GL AccountsLists → Accounting → Chart of Accounts
LocationsLists → Accounting → Locations
DepartmentsLists → Accounting → Departments
ClassesLists → Accounting → Classes
Payment TermsLists → Accounting → Terms

Admin & Setup

Record / ActionPath
Accounting PeriodsSetup → Accounting → Manage Periods
RolesSetup → Users/Roles → Manage Roles
UsersSetup → Users/Roles → Manage Users
ScriptsCustomization → Scripting → Scripts
Script DeploysCustomization → Scripting → Deployments
Saved SearchesReports → Saved Searches → All
WorkflowsCustomization → Workflow → Workflows
Custom FieldsCustomization → Lists/Records/Fields
SuiteQL ToolCustomization → Scripting → Suitelets → SuiteQL #2947
Import CSVSetup → Import/Export → Import CSV

Source

  • data/runbook-source.json — procedures + checklists extracted from the original ops dashboard
  • data/ns-navigation.json — NetSuite navigation map
  • Onboarding, secret rotation, incident response playbooks authored in this build pass