diff --git a/clients/spice-nvybe/MEETING_PREP_KAY.md b/clients/spice-nvybe/MEETING_PREP_KAY.md new file mode 100644 index 0000000..abd1449 --- /dev/null +++ b/clients/spice-nvybe/MEETING_PREP_KAY.md @@ -0,0 +1,140 @@ +# Kay Meeting Prep β€” Spice N'Vybe Alignment +**Date:** July 4, 2026 +**Purpose:** Align on scope, priorities, and data access β€” not a sales pitch +**Reference:** SNV_AI_Scope_of_Work_v2.pdf (client's signed SOW) + +--- + +## 🎯 Meeting Approach + +**Tone: Discovery + Alignment, not a demo or sales pitch.** + +We've reviewed their SOW in full. We didn't build it, so we're not showing them a dashboard they haven't seen yet. Instead, we're figuring out what they actually need so we can start building what matters. + +> *"Kay, we've reviewed your SOW in detail. A few things we want to nail down so we can start building immediately."* + +--- + +## ❓ Key Questions for Kay (in Priority Order) + +### 1. What's the ONE thing you need working first? + +The SOW sequences the Dashboard (M01) last in Weeks 11-14, after everything else. Is that still your priority order, or has that changed? + +**Probe options if he's unsure:** +- Dashboard with real financial data? +- Chatbot for the Cloud Kitchen team to look up SOPs/menus? +- Training system you mentioned already has a foundation? +- CRE listings coming in automatically? + +**Our recommendation (if he asks):** +> *"The fastest path to value is getting the Dashboard live with just one data source β€” SpotOn sales data. That gives you real numbers in 2-3 weeks. We layer on Plaid, QBO, and everything else after."* + +### 2. MarginEdge β€” do you use it? + +It's referenced in M01, M02, and M03. If they use it, we need API access to scope integration. If they don't, we drop it. + +### 3. Gmail + Google Calendar in the Dashboard β€” critical or nice-to-have? + +The SOW mentions unanswered email queue and a calendar widget. Find out: is this MVP, v1.1, or not pressing? + +### 4. Existing Training System β€” what's there? + +SOW says M06 *"extends the existing kitchen training system already in operation"*. We need to see what they have before we scope. + +### 5. CRE Scraper β€” how do you want to handle it? + +Loopnet/Crexi scraping is legally grey. Do they accept a browser-automation approach, or do they prefer a paid data feed? + +--- + +## πŸ”‘ Critical Unlocks (Need These to Start) + +| # | Item | Who's responsible | Status ⬜ | +|---|---|---|---| +| 1 | **SpotOn API keys** β€” Oakland Park + Ft. Lauderdale | Kay | ⬜ | +| 2 | **QBO OAuth access** β€” realm ID, Intuit dev account | Kay | ⬜ | +| 3 | **Plaid credentials** β€” client_id + secret, or Link flow | Kay | ⬜ | +| 4 | MarginEdge β€” yes/no? If yes, account access | Kay | ⬜ | +| 5 | Gmail API β€” needed for M01? | Kay | ⬜ | +| 6 | Google Calendar API β€” needed for M01? | Kay | ⬜ | +| 7 | M06 existing training system β€” can we see it? | Kay | ⬜ | + +> **Pitch if hesitating:** *"The single biggest unlock is data access. If we can get SpotOn, QBO, and Plaid credentials this week, we can have a live prototype running in 2-3 weeks."* + +--- + +## πŸ—ΊοΈ Our Real Delivery Priority (Not the SOW's Sequence) + +| Phase | What | Why | +|---|---|---| +| **Week 1** | Get API keys | Everything blocks on this | +| **Weeks 2-4** | M01 core dashboard (SpotOn sales first) | Fastest visible win β€” real numbers | +| **Weeks 4-6** | M01 + Plaid + QBO | Full financial picture | +| **Weeks 6-10** | M05 Chatbot (RAG) | Quick win β€” we have the inference infra | +| **After** | M02, M03, M04, M06, M07 | Based on what Kay prioritizes | + +**Key truth:** Don't build M01 last. A live prototype with real data in 3 weeks builds more trust than a 16-week timeline. + +--- + +## πŸ“‹ Conversation Flow + +### 1. Opening β€” Set the Frame (2 min) +> *"Hey Kay, thanks for the time. We've gone through your SOW thoroughly. I want to make sure we're building what actually moves the needle for you, so I've got a few questions to align on before we start."* + +### 2. What's Priority #1? (5 min) +> *"Your SOW sequences the Dashboard last. Is that still the plan, or has the urgency shifted?"* + +Let him talk. Listen for what he *actually* cares about. + +### 3. Data Source Check (5 min) +> *"To build anything, we need access to your data. Let me run through what we'll need..."* + +Go through the **Critical Unlocks** checklist above. + +Key framing: +> *"For QuickBooks β€” read-only access to invoices, P&L, and payables. For banking β€” Plaid is view-only, we can never move money. For SpotOn β€” sales and labor data only."* + +### 4. Scope Clarifiers (5 min) +> *"A few quick clarifiers from the SOW..."* + +- MarginEdge? +- Gmail/Calendar in Dashboard β€” critical or nice-to-have? +- Training system β€” can I see the current version? +- CRE scraper β€” how do you want us to handle the legal side? + +### 5. Close β€” Next Steps (3 min) +> *"Once we have those API keys, I can have a working prototype with your real data in 2-3 weeks. How does that timeline feel to you?"* + +**Confirm:** +- Who's the point of contact for each data source +- Tentative ETA for credentials +- When to follow up + +--- + +## 🚨 Potential Objections & Responses + +| Objection | Response | +|---|---| +| *"The SOW already says all this β€” why are we rehashing it?"* | "We're not rehashing β€” we're confirming. The SOW sequences things one way. I want to make sure that order still works for you before we invest time building." | +| *"I'm not comfortable giving bank access."* | "Plaid is read-only β€” can see balances and transactions, never touch money. Same tech QuickBooks and Venmo use." | +| *"I need everything at once, not phased."* | "I can build faster by starting with what matters most and adding layers. A live dashboard with sales data in 3 weeks is more useful than a complete plan in 3 months." | +| *"Who sees our data?"* | "Stays on our private infrastructure. No third parties, no cloud sharing." | + +--- + +## πŸ“ After the Meeting β€” Capture + +Come back and tell me: + +1. **Kay's priority #1** β€” what does he want first? +2. **API keys** β€” which did he commit to? When? +3. **MarginEdge** β€” yes/no? +4. **Gmail/Calendar** β€” must-have or optional? +5. **M06 training system** β€” what exists? +6. **CRE scraping** β€” his stance? +7. **Timeline** β€” what did you both agree on? + +I'll update the project plan and start building as soon as credentials land. πŸ¦… diff --git a/clients/spice-nvybe/README.md b/clients/spice-nvybe/README.md new file mode 100644 index 0000000..40fc34a --- /dev/null +++ b/clients/spice-nvybe/README.md @@ -0,0 +1,37 @@ +# Spice N'Vybe β€” Gonzague Foods LLC + +**Client:** Spice N'Vybe (Gonzague Foods LLC, Oakland Park, FL) +**Contact:** Kay (Co-owner) +**Engagement:** AI Technology Transformation β€” 7 Modules +**Status:** Discovery Phase β†’ Pending alignment meeting (July 4, 2026) + +## Project Structure + +| File | Description | +|------|-------------| +| `SNV_AI_Scope_of_Work_v2.pdf` | Client's signed SOW β€” 7 AI modules, 22 pages | +| `MEETING_PREP_KAY.md` | Pre-meeting plan for July 4 alignment meeting | +| `architecture/m01-bi-dashboard.md` | M01 architecture spec (SpotOn, Plaid, QBO, PostgreSQL) | +| `research/api-data-sources.md` | API research for QBO, Plaid, SpotOn integrations | + +## The 7 Modules + +1. **M01** β€” GF Unified BI Dashboard (6-8 wks) +2. **M02** β€” 24-Month Financial Forecasting (4-6 wks) +3. **M03** β€” AI Schedule Optimizer (4-5 wks) +4. **M04** β€” CRE Intelligence Tool (3-4 wks) +5. **M05** β€” Operations Chatbot (4-5 wks) +6. **M06** β€” Food Prep Training System (3-4 wks) +7. **M07** β€” Investor Research Engine (3-4 wks) + +## Key Documents + +- SOW v2.0 β€” full client scope +- Meeting prep for Kay alignment (July 4, 2026) +- M01 architecture spec (our deliverable design) +- API data source research + +## Delivery Approach + +Syslog-led. Kay sets the *what*, we own the *how* and *when*. +Fastest path: M01 Dashboard (SpotOn-first) in 2-3 weeks β†’ iterate from there. diff --git a/clients/spice-nvybe/SNV_AI_Scope_of_Work_v2.pdf b/clients/spice-nvybe/SNV_AI_Scope_of_Work_v2.pdf new file mode 100644 index 0000000..9f80cbf Binary files /dev/null and b/clients/spice-nvybe/SNV_AI_Scope_of_Work_v2.pdf differ diff --git a/clients/spice-nvybe/architecture/m01-bi-dashboard.md b/clients/spice-nvybe/architecture/m01-bi-dashboard.md new file mode 100644 index 0000000..670f9b0 --- /dev/null +++ b/clients/spice-nvybe/architecture/m01-bi-dashboard.md @@ -0,0 +1,1876 @@ +# Spice N'Vybe M01 Executive BI Dashboard β€” Architectural Specification + +> **Version:** v1.0 | **Date:** 2026-07-01 +> **Target:** Homelab LXC with future AWS migration +> **Document Type:** Architectural Blueprint (production) + +--- + +## Table of Contents + +1. [System Architecture Overview](#1-system-architecture-overview) +2. [Database Schema Design](#2-database-schema-design) +3. [Data Ingestion Pipeline](#3-data-ingestion-pipeline) +4. [API Endpoint Design](#4-api-endpoint-design) +5. [Frontend Component Architecture](#5-frontend-component-architecture) +6. [Alerting System](#6-alerting-system) +7. [Security Model](#7-security-model) +8. [Deployment Architecture (LXC)](#8-deployment-architecture-lxc) +9. [AWS Migration & Scaling Strategy](#9-aws-migration--scaling-strategy) +10. [Data Reconciliation & Consistency Strategy](#10-data-reconciliation--consistency-strategy) + +--- + +## 1. System Architecture Overview + +### 1.1 High-Level Architecture + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ FRONTEND (React/Next.js) β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ KPI Dashboard β”‚ β”‚ Location Tabsβ”‚ β”‚Alerts Panelβ”‚ β”‚Settings β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ REST API (HTTPS) β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ API GATEWAY / REVERSE PROXY (nginx) β”‚ +β”‚ (auth validation, rate limiting) β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ BACKEND (Node.js/Express) β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ API Router β”‚ β”‚ Auth Service β”‚ β”‚ Alert Engine β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ Ingestion Workersβ”‚ β”‚ Reconciliation β”‚ β”‚ Rate Limiter β”‚ β”‚ +β”‚ β”‚ (QBO/Plaid/SON) β”‚ β”‚ Engine β”‚ β”‚ (bottleneck) β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ Queue Manager β”‚ β”‚ Job Scheduler β”‚ β”‚ +β”‚ β”‚ (Bull/BullMQ) β”‚ β”‚ (node-cron) β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ PERSISTENCE LAYER β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ PostgreSQL 16 β”‚ β”‚ +β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ +β”‚ β”‚ β”‚ Raw Ingestionβ”‚ β”‚ Normalized β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ Tables β”‚ β”‚ Views β”‚ β”‚ β”‚ +β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ +β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ +β”‚ β”‚ β”‚ Aggregationsβ”‚ β”‚ Metadata β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ (Materialized)β”‚ β”‚ (Sync State)β”‚ β”‚ β”‚ +β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ Redis 7 β”‚ β”‚ +β”‚ β”‚ (Queue broker, cache, rate limiter, β”‚ β”‚ +β”‚ β”‚ token store for QBO OAuth) β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ β”‚ β”‚ +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ QuickBooks β”‚ β”‚ Plaid β”‚ β”‚ SpotOn POS β”‚ +β”‚ Online (QBO) β”‚ β”‚ (Banking) β”‚ β”‚ (2 locations) β”‚ +β”‚ OAuth 2.0 β”‚ β”‚ API Key β”‚ β”‚ API Key per loc β”‚ +β”‚ 500 req/min β”‚ β”‚ Cursor sync β”‚ β”‚ 26h window β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +### 1.2 Data Flow Summary + +``` +SpotOn ──► 5-min polls (business hours) ──► Raw Tables ──► Normalization ──► Aggregation + 26h deep sync daily +Plaid ──► /transactions/sync (cursor) ───► Raw Tables ──► Normalization ──► Aggregation + SYNC_UPDATES_AVAILABLE webhook +QBO ──► CDC polling (30min) + webhooks ─► Raw Tables ──► Normalization ──► Aggregation + Re-auth required every 100 days +``` + +### 1.3 Technology Stack (Precise Versions) + +| Layer | Technology | Version | Purpose | +|-------|-----------|---------|---------| +| Frontend | Next.js | 14.x | React framework with SSR/SSG | +| UI | React | 18.x | Component library | +| Charts | Recharts | 2.x | React-native charting | +| Styling | Tailwind CSS | 3.x | Utility-first CSS | +| Tables | TanStack Table | 8.x | Data grid/table | +| Backend | Node.js | 20.x LTS | Runtime | +| Framework | Express | 4.18+ | HTTP server | +| Queue | BullMQ | 5.x | Redis-backed job queue | +| Cache | Redis | 7.x | Queue, cache, rate limiter | +| Database | PostgreSQL | 16.x | Primary data store | +| ORM | Prisma | 5.x | Type-safe DB access | +| Scheduler | node-cron | 3.x | Cron-based sync triggers | +| Auth | JWT + Passport | latest | API auth | +| OAuth | node-quickbooks | latest | QBO OAuth 2.0 | + +--- + +## 2. Database Schema Design + +### 2.1 Naming Conventions + +- `raw_*` β€” Tables that mirror API responses 1:1 (immutable append log) +- `normalized_*` β€” Cleaned, typed, deduplicated records +- `agg_*` β€” Pre-computed aggregates (materialized views) +- `meta_*` β€” Configuration, sync state, credentials + +### 2.2 Location Reference Table + +```sql +-- Core location catalog +CREATE TABLE meta_locations ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + slug VARCHAR(32) UNIQUE NOT NULL, -- 'oakland-park', 'ft-lauderdale' + name VARCHAR(128) NOT NULL, -- 'Oakland Park Flagship', 'Fort Lauderdale Cloud Kitchen' + spoton_site_id VARCHAR(64), -- SpotOn site identifier + spoton_api_key TEXT, -- Encrypted at application layer + qbo_class_id VARCHAR(64), -- QBO Class for cost center tracking + plaid_account_ids JSONB, -- Array of Plaid account IDs for this location + is_active BOOLEAN DEFAULT true, + metadata JSONB DEFAULT '{}', + created_at TIMESTAMPTZ DEFAULT now(), + updated_at TIMESTAMPTZ DEFAULT now() +); + +-- Seed data +INSERT INTO meta_locations (slug, name) VALUES + ('oakland-park', 'Oakland Park Flagship'), + ('ft-lauderdale', 'Fort Lauderdale Cloud Kitchen'); +``` + +### 2.3 SpotOn Raw & Normalized Tables + +```sql +-- ── RAW: Mirror of SpotOn Orders API response ── +CREATE TABLE raw_spoton_orders ( + id BIGSERIAL, + spoton_order_id VARCHAR(64) NOT NULL, + location_id UUID NOT NULL REFERENCES meta_locations(id), + raw_payload JSONB NOT NULL, -- Full API response + ingested_at TIMESTAMPTZ DEFAULT now(), + PRIMARY KEY (id), + UNIQUE (spoton_order_id, location_id) +); +CREATE INDEX idx_raw_so_ingested ON raw_spoton_orders(ingested_at); +CREATE INDEX idx_raw_so_location ON raw_spoton_orders(location_id); + +-- ── RAW: SpotOn Labor Reports ── +CREATE TABLE raw_spoton_labor ( + id BIGSERIAL, + location_id UUID NOT NULL REFERENCES meta_locations(id), + report_date DATE NOT NULL, + raw_payload JSONB NOT NULL, + ingested_at TIMESTAMPTZ DEFAULT now(), + PRIMARY KEY (id), + UNIQUE (location_id, report_date) +); + +-- ── NORMALIZED: Parsed order line items ── +CREATE TABLE normalized_orders ( + id BIGSERIAL, + spoton_order_id VARCHAR(64) NOT NULL, + location_id UUID NOT NULL REFERENCES meta_locations(id), + order_date TIMESTAMPTZ NOT NULL, + order_type VARCHAR(32), -- 'dine-in', 'takeout', 'delivery', 'online' + subtotal DECIMAL(12,2), + tax DECIMAL(12,2), + tip DECIMAL(12,2), + total DECIMAL(12,2) NOT NULL, + payment_method VARCHAR(32), -- 'credit', 'cash', 'gift_card', etc. + item_count INTEGER, + status VARCHAR(32), -- 'completed', 'refunded', 'voided' + raw_id BIGINT REFERENCES raw_spoton_orders(id), + created_at TIMESTAMPTZ DEFAULT now(), + PRIMARY KEY (id) +); +CREATE INDEX idx_norm_orders_date ON normalized_orders(order_date); +CREATE INDEX idx_norm_orders_location ON normalized_orders(location_id); +CREATE INDEX idx_norm_orders_status ON normalized_orders(status); + +-- ── NORMALIZED: Labor costs from SpotOn ── +CREATE TABLE normalized_labor_costs ( + id BIGSERIAL, + location_id UUID NOT NULL REFERENCES meta_locations(id), + report_date DATE NOT NULL, + total_hours DECIMAL(10,2), + total_wages DECIMAL(12,2), + total_labor_cost DECIMAL(12,2), -- wages + taxes + benefits + employee_count INTEGER, + labor_percent DECIMAL(5,2), -- labor cost as % of sales + raw_id BIGINT REFERENCES raw_spoton_labor(id), + created_at TIMESTAMPTZ DEFAULT now(), + PRIMARY KEY (id), + UNIQUE (location_id, report_date) +); +``` + +### 2.4 Plaid Raw & Normalized Tables + +```sql +-- ── RAW: Plaid transactions sync state ── +CREATE TABLE meta_plaid_cursors ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + plaid_item_id VARCHAR(64) UNIQUE NOT NULL, + access_token TEXT NOT NULL, -- Encrypted at application layer + next_cursor TEXT, -- Cursor for incremental sync + initial_cursor TEXT, -- Saved for recovery + last_sync_at TIMESTAMPTZ, + status VARCHAR(32) DEFAULT 'active', -- 'active', 'error', 'requires_update' + error_detail TEXT, + created_at TIMESTAMPTZ DEFAULT now() +); + +-- ── RAW: Plaid transactions (append-only) ── +CREATE TABLE raw_plaid_transactions ( + id BIGSERIAL, + plaid_txn_id VARCHAR(64) NOT NULL UNIQUE, + plaid_item_id VARCHAR(64) NOT NULL REFERENCES meta_plaid_cursors(plaid_item_id), + account_id VARCHAR(64), + raw_payload JSONB NOT NULL, + sync_type VARCHAR(16), -- 'initial', 'added', 'modified', 'removed' + ingested_at TIMESTAMPTZ DEFAULT now(), + PRIMARY KEY (id) +); +CREATE INDEX idx_raw_plaid_item ON raw_plaid_transactions(plaid_item_id); +CREATE INDEX idx_raw_plaid_ingested ON raw_plaid_transactions(ingested_at); + +-- ── NORMALIZED: Cleaned Plaid transactions ── +CREATE TABLE normalized_bank_transactions ( + id BIGSERIAL, + plaid_txn_id VARCHAR(64) UNIQUE NOT NULL, + location_id UUID REFERENCES meta_locations(id), -- Mapped via Plaid account + account_id VARCHAR(64), + transaction_date DATE NOT NULL, + amount DECIMAL(12,2) NOT NULL, -- Positive = debit, negative = credit + description TEXT, + merchant_name VARCHAR(256), + category VARCHAR(128), + personal_finance_category VARCHAR(128), + is_expense BOOLEAN GENERATED ALWAYS AS (amount > 0) STORED, + is_reconciled BOOLEAN DEFAULT false, -- Matched to SpotOn deposit? + reconciled_txn_id BIGINT REFERENCES normalized_orders(id), + created_at TIMESTAMPTZ DEFAULT now(), + PRIMARY KEY (id) +); +CREATE INDEX idx_bank_txn_date ON normalized_bank_transactions(transaction_date); +CREATE INDEX idx_bank_txn_location ON normalized_bank_transactions(location_id); +CREATE INDEX idx_bank_txn_category ON normalized_bank_transactions(category); +CREATE INDEX idx_bank_txn_reconciled ON normalized_bank_transactions(is_reconciled); +``` + +### 2.5 QuickBooks Raw & Normalized Tables + +```sql +-- ── META: QBO OAuth state ── +CREATE TABLE meta_qbo_credentials ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + realm_id VARCHAR(64) UNIQUE NOT NULL, + access_token TEXT NOT NULL, -- Encrypted; rotated every 60 min + refresh_token TEXT NOT NULL, -- Encrypted; rotated every 100 days + token_expires_at TIMESTAMPTZ NOT NULL, + refresh_expires_at TIMESTAMPTZ NOT NULL, + is_active BOOLEAN DEFAULT true, + last_refreshed_at TIMESTAMPTZ, + created_at TIMESTAMPTZ DEFAULT now() +); + +-- ── META: QBO CDC sync cursor ── +CREATE TABLE meta_qbo_cdc_cursors ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + realm_id VARCHAR(64) NOT NULL REFERENCES meta_qbo_credentials(realm_id), + entity_name VARCHAR(64) NOT NULL, -- 'Invoice', 'Bill', 'Payment', etc. + last_cdc_timestamp TIMESTAMPTZ NOT NULL, + last_sync_at TIMESTAMPTZ, + UNIQUE (realm_id, entity_name) +); + +-- ── RAW: QBO CDC events (append log) ── +CREATE TABLE raw_qbo_cdc_events ( + id BIGSERIAL, + realm_id VARCHAR(64) NOT NULL, + entity_name VARCHAR(64) NOT NULL, + entity_id VARCHAR(64) NOT NULL, + operation VARCHAR(16) NOT NULL, -- 'Create', 'Update', 'Delete' + raw_payload JSONB NOT NULL, -- Full entity snapshot + event_timestamp TIMESTAMPTZ NOT NULL, + ingested_at TIMESTAMPTZ DEFAULT now(), + PRIMARY KEY (id) +); +CREATE INDEX idx_raw_qbo_event ON raw_qbo_cdc_events(entity_name, entity_id); +CREATE INDEX idx_raw_qbo_time ON raw_qbo_cdc_events(event_timestamp); + +-- ── RAW: QBO P&L report snapshots ── +CREATE TABLE raw_qbo_profit_loss ( + id BIGSERIAL, + realm_id VARCHAR(64) NOT NULL, + report_date DATE NOT NULL, + report_type VARCHAR(32) NOT NULL, -- 'monthly', 'quarterly', 'ytd' + raw_payload JSONB NOT NULL, + ingested_at TIMESTAMPTZ DEFAULT now(), + UNIQUE (realm_id, report_date, report_type) +); + +-- ── NORMALIZED: QBO Costs (COGS + Expenses) ── +CREATE TABLE normalized_qbo_costs ( + id BIGSERIAL, + location_id UUID REFERENCES meta_locations(id), + cost_date DATE NOT NULL, + cost_category VARCHAR(64) NOT NULL, -- 'cogs', 'rent', 'utilities', 'payroll', 'marketing', etc. + account_name VARCHAR(128), + account_id VARCHAR(64), + amount DECIMAL(12,2) NOT NULL, + source VARCHAR(32) DEFAULT 'qbo', -- 'qbo', 'plaid', 'manual' + source_ref VARCHAR(128), -- Entity ID from source + created_at TIMESTAMPTZ DEFAULT now(), + PRIMARY KEY (id) +); +CREATE INDEX idx_costs_date ON normalized_qbo_costs(cost_date); +CREATE INDEX idx_costs_location ON normalized_qbo_costs(location_id); +CREATE INDEX idx_costs_category ON normalized_qbo_costs(cost_category); +``` + +### 2.6 Aggregation Tables (Materialized Views) + +```sql +-- ── Daily Sales Aggregation ── +CREATE MATERIALIZED VIEW agg_daily_sales AS +SELECT + location_id, + order_date::DATE AS sale_date, + COUNT(*) AS order_count, + SUM(total) AS gross_sales, + SUM(CASE WHEN status != 'refunded' THEN total ELSE 0 END) AS net_sales, + SUM(CASE WHEN status = 'refunded' THEN ABS(total) ELSE 0 END) AS refunds, + SUM(tax) AS tax_collected, + SUM(tip) AS tips, + SUM(item_count) AS total_items, + AVG(total) AS avg_order_value, + COUNT(DISTINCT payment_method) AS payment_methods_used +FROM normalized_orders +GROUP BY location_id, order_date::DATE; + +CREATE UNIQUE INDEX ON agg_daily_sales(location_id, sale_date); + +-- ── Daily Sales Γ— Plaid Deposit Reconciliation ── +CREATE MATERIALIZED VIEW agg_daily_reconciliation AS +SELECT + COALESCE(s.location_id, b.location_id) AS location_id, + COALESCE(s.sale_date, b.transaction_date) AS report_date, + COALESCE(s.net_sales, 0) AS spoton_sales, + COALESCE(SUM(ABS(b.amount)), 0) AS plaid_deposits, -- Credits only + CASE + WHEN s.net_sales IS NULL THEN 'no_sales_data' + WHEN SUM(ABS(b.amount)) IS NULL THEN 'no_bank_data' + WHEN ABS(s.net_sales - SUM(ABS(b.amount))) < 0.01 THEN 'matched' + WHEN ABS(s.net_sales - SUM(ABS(b.amount))) / NULLIF(s.net_sales, 0) < 0.05 THEN 'minor_variance' + ELSE 'variance_detected' + END AS reconciliation_status, + (s.net_sales - COALESCE(SUM(ABS(b.amount)), 0)) AS variance +FROM agg_daily_sales s +FULL OUTER JOIN normalized_bank_transactions b + ON s.location_id = b.location_id + AND s.sale_date = b.transaction_date + AND b.amount < 0 -- Credits (deposits) +GROUP BY COALESCE(s.location_id, b.location_id), + COALESCE(s.sale_date, b.transaction_date), + s.net_sales; + +CREATE UNIQUE INDEX ON agg_daily_reconciliation(location_id, report_date); + +-- ── Daily P&L Summary ── +CREATE MATERIALIZED VIEW agg_daily_profit_loss AS +SELECT + COALESCE(s.location_id, c.location_id) AS location_id, + COALESCE(s.sale_date, c.cost_date) AS report_date, + COALESCE(s.net_sales, 0) AS total_revenue, + lc.total_labor_cost AS labor_cost, + lc.labor_percent, + COALESCE(SUM(c.amount) FILTER (WHERE c.cost_category = 'cogs'), 0) AS cogs, + COALESCE(SUM(c.amount) FILTER (WHERE c.cost_category != 'cogs'), 0) AS operating_expenses, + COALESCE(lc.total_labor_cost, 0) + COALESCE(SUM(c.amount), 0) AS total_costs, + COALESCE(s.net_sales, 0) - (COALESCE(lc.total_labor_cost, 0) + COALESCE(SUM(c.amount), 0)) AS net_profit +FROM agg_daily_sales s +FULL OUTER JOIN normalized_qbo_costs c + ON s.location_id = c.location_id AND s.sale_date = c.cost_date +LEFT JOIN normalized_labor_costs lc + ON s.location_id = lc.location_id AND s.sale_date = lc.report_date +GROUP BY COALESCE(s.location_id, c.location_id), + COALESCE(s.sale_date, c.cost_date), + s.net_sales, + lc.total_labor_cost, + lc.labor_percent; + +CREATE UNIQUE INDEX ON agg_daily_profit_loss(location_id, report_date); + +-- ── Monthly Location Comparison ── +CREATE MATERIALIZED VIEW agg_monthly_comparison AS +SELECT + location_id, + date_trunc('month', report_date) AS month, + SUM(total_revenue) AS revenue, + SUM(total_costs) AS costs, + SUM(net_profit) AS profit, + AVG(labor_percent) AS avg_labor_pct, + SUM(labor_cost) AS total_labor, + SUM(cogs) AS total_cogs, + SUM(operating_expenses) AS total_opex, + COUNT(*) AS days_with_data +FROM agg_daily_profit_loss +GROUP BY location_id, date_trunc('month', report_date); + +CREATE UNIQUE INDEX ON agg_monthly_comparison(location_id, month); +``` + +### 2.7 Alerting & Audit Tables + +```sql +-- ── Alert Log ── +CREATE TABLE meta_alerts ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + severity VARCHAR(16) NOT NULL CHECK (severity IN ( + 'CRITICAL', 'FAILURE', 'ERROR', 'SECURITY', 'WARNING', 'INFO' + )), + category VARCHAR(64) NOT NULL, -- 'ingestion', 'reconciliation', 'auth', 'system', 'kpi' + title VARCHAR(256) NOT NULL, + message TEXT NOT NULL, + source VARCHAR(32), -- 'qbo', 'plaid', 'spoton', 'system' + location_id UUID REFERENCES meta_locations(id), + kpi_name VARCHAR(64), -- 'sales', 'costs', 'net_profit', 'labor_pct' + kpi_value DECIMAL(12,2), + email_sent BOOLEAN DEFAULT false, + email_sent_at TIMESTAMPTZ, + acknowledged BOOLEAN DEFAULT false, + acknowledged_by VARCHAR(128), + acknowledged_at TIMESTAMPTZ, + created_at TIMESTAMPTZ DEFAULT now() +); +CREATE INDEX idx_alerts_severity ON meta_alerts(severity); +CREATE INDEX idx_alerts_created ON meta_alerts(created_at); +CREATE INDEX idx_alerts_category ON meta_alerts(category); + +-- ── Ingestion Job Log ── +CREATE TABLE meta_ingestion_jobs ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + source VARCHAR(32) NOT NULL, -- 'qbo', 'plaid', 'spoton' + job_type VARCHAR(32) NOT NULL, -- 'full_sync', 'incremental', 'webhook', 'reconciliation' + status VARCHAR(16) NOT NULL CHECK (status IN ( + 'queued', 'running', 'completed', 'failed', 'retrying' + )), + records_fetched INTEGER DEFAULT 0, + records_ingested INTEGER DEFAULT 0, + records_failed INTEGER DEFAULT 0, + started_at TIMESTAMPTZ, + completed_at TIMESTAMPTZ, + error_detail TEXT, + retry_count INTEGER DEFAULT 0, + next_retry_at TIMESTAMPTZ, + created_at TIMESTAMPTZ DEFAULT now() +); +CREATE INDEX idx_ingestion_source ON meta_ingestion_jobs(source, created_at DESC); + +-- ── Rate Limit Tracking ── +CREATE TABLE meta_rate_limit_log ( + id BIGSERIAL, + source VARCHAR(32) NOT NULL, + endpoint VARCHAR(256), + status_code INTEGER, + retry_after_sec INTEGER, + occurred_at TIMESTAMPTZ DEFAULT now() +); +``` + +--- + +## 3. Data Ingestion Pipeline + +### 3.1 Pipeline Architecture + +``` +EXT SOURCE ──► ADAPTER ──► RATE LIMITER ──► QUEUE ──► WORKER ──► RAW TABLE ──► TRANSFORM ──► NORMALIZED ──► AGGREGATION + β”‚ + β–Ό + ALERT EVALUATOR +``` + +### 3.2 QuickBooks Online Ingestion + +#### 3.2.1 OAuth Token Lifecycle + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ TOKEN MANAGER (runs on loop every 50 minutes) β”‚ +β”‚ β”‚ +β”‚ 1. Check meta_qbo_credentials.token_expires_at β”‚ +β”‚ 2. If expires within 10 min β†’ refresh now β”‚ +β”‚ 3. POST to /oauth2/v1/tokens/bearer with refresh_token β”‚ +β”‚ 4. Store NEW access_token + refresh_token immediately β”‚ +β”‚ 5. Old refresh_token is INVALIDATED after use β”‚ +β”‚ 6. If refresh fails β†’ queue SECURITY alert β”‚ +β”‚ β”‚ +β”‚ CRITICAL: Refresh token expires after 100 days total. β”‚ +β”‚ If expired β†’ user must re-authorize via OAuth flow. β”‚ +β”‚ Track refresh_expires_at and alert at 90-day mark. β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +#### 3.2.2 Sync Strategy + +| Sync Type | Frequency | Method | Details | +|-----------|-----------|--------|---------| +| **CDC Poll** | Every 30 min | `GET /cdc?entities=...&changedSince=` | Uses meta_qbo_cdc_cursors for timestamp tracking. Fetches up to 30 days back. | +| **Webhook** | Real-time | POST to `/api/webhooks/qbo` | Validates HMAC signature, enqueues entity fetch job | +| **Full Sync** | Daily (2am) | CDC with 48h window + P&L report fetch | Reconciles P&L, pulls full Cost data | +| **Periodic** | Weekly (Sun 3am) | Full query on all entities | Data integrity check, catches CDC gaps | + +#### 3.2.3 Webhook Handler Flow + +``` +POST /api/webhooks/qbo + β”œβ”€ 1. Verify HMAC-SHA256 signature (prevent forgery) + β”œβ”€ 2. Parse eventNotifications[] + β”œβ”€ 3. For each entity: + β”‚ └─ Enqueue BullMQ job: { type: 'qbo:fetch_entity', entity, id, operation } + └─ 4. Return 200 immediately (don't process inline) +``` + +#### 3.2.4 Rate Limiter + +``` +const QBO_RATE_LIMIT = { + maxPerMinute: 450, // 10% headroom below 500 + maxConcurrent: 35, // 5 below 40 +}; + +// Token bucket algorithm in Redis +// Key: rate_limit:qbo:{realm_id}:{minute_bucket} +// TTL: 90 seconds + +async function checkQboRateLimit(realmId) { + const key = `rate_limit:qbo:${realmId}:${Math.floor(Date.now() / 60000)}`; + const count = await redis.incr(key); + if (count === 1) await redis.expire(key, 90); + if (count > QBO_RATE_LIMIT.maxPerMinute) throw new RateLimitError('QBO rate limit approaching'); +} +``` + +### 3.3 Plaid Ingestion + +#### 3.3.1 Initial Sync + +``` +1. Call /transactions/sync with no cursor (first sync) +2. Paginate: while has_more === true, keep calling with next_cursor +3. For each page: + - added[] β†’ INSERT INTO raw_plaid_transactions (sync_type='initial') + - modified[] β†’ INSERT INTO raw_plaid_transactions (sync_type='modified') + - removed[] β†’ INSERT INTO raw_plaid_transactions (sync_type='removed') +4. Save next_cursor to meta_plaid_cursors +5. Trigger normalized_bank_transactions rebuild for this item +``` + +#### 3.3.2 Incremental Sync + +``` +Scheduled every 15 minutes OR triggered by SYNC_UPDATES_AVAILABLE webhook: + +1. Load cursor from meta_plaid_cursors +2. Call /transactions/sync with cursor +3. Handle pagination with cursor preservation: + - Temporarily preserve old cursor + - If TRANSACTIONS_SYNC_MUTATION_DURING_PAGINATION error: + β†’ restart from preserved cursor +4. Process added/modified/removed +5. Save new next_cursor +6. If has_more === true β†’ schedule next page immediately (max 10 pages) +``` + +#### 3.3.3 Webhook Receiver + +``` +POST /api/webhooks/plaid + β”œβ”€ 1. Verify webhook signature + β”œβ”€ 2. Parse webhook_code: + β”‚ - SYNC_UPDATES_AVAILABLE β†’ enqueue incremental sync + β”‚ - ITEM_LOGIN_REQUIRED β†’ create SECURITY alert, flag item + β”‚ - INITIAL_UPDATE β†’ enqueue full sync + β”‚ - HISTORICAL_UPDATE β†’ enqueue full historical pull + └─ 3. Return 200 +``` + +### 3.4 SpotOn Ingestion + +#### 3.4.1 Orders Polling + +``` +Dual-strategy approach to deal with SpotOn's data availability window: + +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ STRATEGY A: Real-time (5-min polls during business hours) β”‚ +β”‚ β”‚ +β”‚ Schedule: Mon-Sat, 8:00 AM - 11:00 PM (local) β”‚ +β”‚ Endpoint: GET /orders?updatedSince={lastPoll}&site={location} β”‚ +β”‚ Window: Poll frequency (5 min) β”‚ +β”‚ β”‚ +β”‚ STRATEGY B: Deep Sync (daily at 3:00 AM) β”‚ +β”‚ β”‚ +β”‚ Schedule: Every day at 3:00 AM β”‚ +β”‚ Endpoint: GET /orders?updatedSince={T-26h}&site={location} β”‚ +β”‚ Window: 26 hours (SpotOn's max) β”‚ +β”‚ Purpose: Catch missed orders, data integrity check β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + +Business hours detection: +- Read business hours from meta_locations.metadata +- Fallback: Mon-Sat 8AM-11PM, Sun 9AM-9PM +- CRON expression generated per location +``` + +#### 3.4.2 Labor Reports Polling + +``` +Schedule: Daily at 6:00 AM (after overnight batch processing completes) + +Endpoint: GET /labor-reports?date={yesterday}&site={location} +Wait: SpotOn labor reports are ETL-lagged β†’ add 1-2h buffer +Retry: If 404/empty, retry every 30 min for up to 4 hours +``` + +### 3.5 BullMQ Queue Architecture + +``` +Queue: qbo-sync + β”œβ”€ Type: qbo:cdc_poll β”‚ Priority: 20 β”‚ Retry: 3 + β”œβ”€ Type: qbo:fetch_entity β”‚ Priority: 10 β”‚ Retry: 2 + β”œβ”€ Type: qbo:refresh_token β”‚ Priority: 1 β”‚ Retry: 3 (+ alert on fail) + └─ Type: qbo:pnl_report β”‚ Priority: 30 β”‚ Retry: 2 + +Queue: plaid-sync + β”œβ”€ Type: plaid:incremental β”‚ Priority: 20 β”‚ Retry: 3 + β”œβ”€ Type: plaid:full_sync β”‚ Priority: 30 β”‚ Retry: 2 + └─ Type: plaid:refresh_link β”‚ Priority: 5 β”‚ Retry: 3 + +Queue: spoton-sync + β”œβ”€ Type: spoton:orders_realtime β”‚ Priority: 20 β”‚ Retry: 2 + β”œβ”€ Type: spoton:orders_deep β”‚ Priority: 30 β”‚ Retry: 2 + └─ Type: spoton:labor_reports β”‚ Priority: 30 β”‚ Retry: 4 (30-min intervals) + +Queue: reconciliation + β”œβ”€ Type: recon:daily β”‚ Priority: 40 β”‚ Runs: 7AM daily + └─ Type: recon:monthly β”‚ Priority: 50 β”‚ Runs: 1st of month +``` + +### 3.6 Error Recovery Strategy + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ ERROR RECOVERY LAYERS β”‚ +β”‚ β”‚ +β”‚ L1: Retry (Transient Failures) β”‚ +β”‚ - 429 rate limit β†’ exponential backoff + jitter β”‚ +β”‚ - 5xx server errors β†’ retry (max 3, backoff ^2) β”‚ +β”‚ - Network timeout β†’ retry (max 3) β”‚ +β”‚ - Default: 2^n * 1000ms + rand(0, 1000)ms β”‚ +β”‚ β”‚ +β”‚ L2: Dead Letter Queue (Persistent Failures) β”‚ +β”‚ - After max retries exhausted β”‚ +β”‚ - Job moved to {queue}:dead-letter β”‚ +β”‚ - Triggers ERROR alert to email β”‚ +β”‚ - Manual review required β”‚ +β”‚ β”‚ +β”‚ L3: Circut Breaker (Cascading Failures) β”‚ +β”‚ - If >10 consecutive failures on same source: β”‚ +β”‚ β†’ Open circuit breaker for 5 minutes β”‚ +β”‚ β†’ Queue CRITICAL alert β”‚ +β”‚ β†’ After 5 min, half-open: try 1 request β”‚ +β”‚ β†’ If success, close; if fail, reopen for 10 min β”‚ +β”‚ β”‚ +β”‚ L4: Backfill (Data Gaps) β”‚ +β”‚ - Daily integrity check compares record counts β”‚ +β”‚ - ETL Gap Detection: β”‚ +β”‚ SELECT location_id, sale_date FROM agg_daily_sales β”‚ +β”‚ WHERE sale_date >= CURRENT_DATE - INTERVAL '7 days' β”‚ +β”‚ AND order_count = 0 β”‚ +β”‚ AND sale_date != CURRENT_DATE; β”‚ +β”‚ - Gaps auto-queue a spoton:orders_deep job β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +--- + +## 4. API Endpoint Design + +### 4.1 REST API Structure + +Base URL: `https://bi.spicenvybe.com/api/v1` + +#### 4.1.1 Authentication & Authorization + +| Method | Endpoint | Auth | Description | +|--------|----------|------|-------------| +| POST | `/auth/login` | None (rate-limited) | Email + password auth, returns JWT | +| POST | `/auth/refresh` | JWT | Refresh access token | +| POST | `/auth/logout` | JWT | Invalidate token | +| GET | `/auth/me` | JWT | Current user profile | +| POST | `/auth/qbo/connect` | JWT+Admin | Initiate QBO OAuth flow | +| GET | `/auth/qbo/callback` | None | QBO OAuth callback handler | + +#### 4.1.2 KPI Endpoints + +| Method | Endpoint | Auth | Description | +|--------|----------|------|-------------| +| GET | `/kpi/sales` | JWT | Sales KPIs (today, MTD, vs prev period) | +| GET | `/kpi/sales/:locationSlug` | JWT | Sales per location | +| GET | `/kpi/net-profit` | JWT | Net Profit KPIs | +| GET | `/kpi/net-profit/:locationSlug` | JWT | Net Profit per location | +| GET | `/kpi/costs` | JWT | Cost breakdown KPIs | +| GET | `/kpi/costs/:locationSlug` | JWT | Costs per location | +| GET | `/kpi/labor` | JWT | Labor KPIs + % of sales | +| GET | `/kpi/labor/:locationSlug` | JWT | Labor per location | +| GET | `/kpi/comparison` | JWT | Side-by-side location comparison | +| GET | `/kpi/summary` | JWT | All KPIs in single response (dashboard load) | + +#### 4.1.3 Data Endpoints + +| Method | Endpoint | Auth | Description | +|--------|----------|------|-------------| +| GET | `/data/sales/timeseries` | JWT | Sales over time (query params: from, to, granularity, location) | +| GET | `/data/sales/by-category` | JWT | Sales breakdown by item category | +| GET | `/data/sales/by-payment` | JWT | Sales by payment method | +| GET | `/data/costs/breakdown` | JWT | Cost breakdown by category | +| GET | `/data/costs/trend` | JWT | Cost trends over time | +| GET | `/data/reconciliation` | JWT | SpotOn vs Plaid deposit reconciliation | +| GET | `/data/reconciliation/:locationSlug` | JWT | Per-location reconciliation | +| GET | `/data/labor/details` | JWT | Labor cost detail (hours, wages, headcount) | +| GET | `/data/transactions` | JWT | Raw transaction search (Plaid) | +| GET | `/data/orders` | JWT | Raw order search (SpotOn) | + +#### 4.1.4 Alert Endpoints + +| Method | Endpoint | Auth | Description | +|--------|----------|------|-------------| +| GET | `/alerts` | JWT | Recent alerts (paginated) | +| GET | `/alerts/active` | JWT | Unacknowledged alerts only | +| PATCH | `/alerts/:id/acknowledge` | JWT | Acknowledge an alert | +| GET | `/alerts/stats` | JWT | Alert statistics (by severity, by source) | +| PUT | `/alerts/config` | JWT+Admin | Alert threshold configuration | +| POST | `/alerts/test` | JWT+Admin | Send test alert email | + +#### 4.1.5 Admin / System Endpoints + +| Method | Endpoint | Auth | Description | +|--------|----------|------|-------------| +| GET | `/admin/health` | JWT+Admin | System health status | +| GET | `/admin/ingestion/status` | JWT+Admin | All ingestion job statuses | +| GET | `/admin/ingestion/:source` | JWT+Admin | Ingestion details per source | +| POST | `/admin/ingestion/trigger` | JWT+Admin | Manually trigger a sync job | +| GET | `/admin/rate-limits` | JWT+Admin | Rate limit utilization | +| GET | `/admin/token-status` | JWT+Admin | QBO token expiry info | + +#### 4.1.6 Webhook Endpoints (no JWT, payload-signed) + +| Method | Endpoint | Auth | Description | +|--------|----------|------|-------------| +| POST | `/webhooks/qbo` | HMAC-SHA256 | QBO data change notifications | +| POST | `/webhooks/plaid` | Plaid webhook secret | Plaid transaction updates | + +### 4.2 Key Response Shapes + +#### `GET /kpi/summary` β€” Dashboard Initial Load + +```json +{ + "period": { + "current": { "start": "2026-06-01", "end": "2026-06-30" }, + "previous": { "start": "2026-05-01", "end": "2026-05-31" } + }, + "overall": { + "sales": { + "current": 284750.00, + "previous": 261200.00, + "change_pct": 9.02, + "trend": "up" + }, + "net_profit": { + "current": 71200.00, + "previous": 58700.00, + "change_pct": 21.29, + "trend": "up" + }, + "costs": { + "current": 213550.00, + "previous": 202500.00, + "change_pct": 5.46, + "trend": "up" + }, + "labor_pct": { + "current": 24.8, + "previous": 27.1, + "change_pct": -8.49, + "trend": "down", + "target": 25.0 + } + }, + "locations": [ + { + "slug": "oakland-park", + "name": "Oakland Park Flagship", + "sales": { "current": 182400.00, "change_pct": 8.5 }, + "net_profit": { "current": 46800.00, "change_pct": 19.2 }, + "costs": { "current": 135600.00, "change_pct": 5.1 }, + "labor_pct": { "current": 23.9, "target": 25.0 } + }, + { + "slug": "ft-lauderdale", + "name": "Fort Lauderdale Cloud Kitchen", + "sales": { "current": 102350.00, "change_pct": 10.1 }, + "net_profit": { "current": 24400.00, "change_pct": 25.8 }, + "costs": { "current": 77950.00, "change_pct": 6.2 }, + "labor_pct": { "current": 26.2, "target": 25.0 } + } + ], + "reconciliation_status": "matched", + "last_updated": "2026-07-01T06:15:00Z" +} +``` + +#### `GET /kpi/labor/:locationSlug` β€” Labor Detail + +```json +{ + "location": "oakland-park", + "period": { + "current": { "start": "2026-06-01", "end": "2026-06-30" }, + "previous": { "start": "2026-05-01", "end": "2026-05-31" } + }, + "metrics": { + "labor_cost": { "current": 67100.00, "previous": 70800.00 }, + "labor_pct": { "current": 23.9, "previous": 27.1 }, + "total_hours": { "current": 3124.5, "previous": 3340.0 }, + "avg_hourly_rate": { "current": 18.45, "previous": 18.27 }, + "employee_count": { "current": 18, "previous": 20 }, + "total_revenue": { "current": 280500.00, "previous": 261200.00 } + }, + "target": 25.0, + "status": "on_target", + "daily_breakdown": [ + { "date": "2026-06-01", "labor_cost": 2140.00, "labor_pct": 24.1, "status": "on_target" }, + { "date": "2026-06-02", "labor_cost": 2280.00, "labor_pct": 26.3, "status": "over_target" } + ] +} +``` + +--- + +## 5. Frontend Component Architecture + +### 5.1 Next.js Application Structure + +``` +src/ +β”œβ”€β”€ app/ # Next.js 14 App Router +β”‚ β”œβ”€β”€ layout.tsx # Root layout (auth boundary) +β”‚ β”œβ”€β”€ page.tsx # Dashboard home (redirect to /dashboard) +β”‚ β”œβ”€β”€ login/ +β”‚ β”‚ └── page.tsx # Login page +β”‚ β”œβ”€β”€ dashboard/ +β”‚ β”‚ β”œβ”€β”€ layout.tsx # Dashboard shell (sidebar, header) +β”‚ β”‚ └── page.tsx # Main dashboard +β”‚ β”œβ”€β”€ alerts/ +β”‚ β”‚ └── page.tsx # Alert center +β”‚ β”œβ”€β”€ settings/ +β”‚ β”‚ └── page.tsx # Settings, connections +β”‚ └── api/ # API routes (BFF pattern) +β”‚ └── [...route]/route.ts # Proxy to Express backend +β”‚ +β”œβ”€β”€ components/ +β”‚ β”œβ”€β”€ dashboard/ +β”‚ β”‚ β”œβ”€β”€ KPIGrid.tsx # KPI card grid layout +β”‚ β”‚ β”œβ”€β”€ KPIWidget.tsx # Single KPI card +β”‚ β”‚ β”œβ”€β”€ LocationTabs.tsx # Location switcher tabs +β”‚ β”‚ β”œβ”€β”€ LocationComparison.tsx# Side-by-side comparison view +β”‚ β”‚ β”œβ”€β”€ SalesChart.tsx # Sales timeseries (Recharts) +β”‚ β”‚ β”œβ”€β”€ ProfitChart.tsx # Profit waterfall +β”‚ β”‚ β”œβ”€β”€ CostBreakdownChart.tsx# Cost donut/bar chart +β”‚ β”‚ β”œβ”€β”€ LaborGauge.tsx # Labor % gauge with target +β”‚ β”‚ β”œβ”€β”€ ReconciliationCard.tsx# Plaid vs SpotOn match status +β”‚ β”‚ └── DataFreshnessBanner.tsx# "Last updated X ago" +β”‚ β”‚ +β”‚ β”œβ”€β”€ alerts/ +β”‚ β”‚ β”œβ”€β”€ AlertPanel.tsx # Alert sidebar/drawer +β”‚ β”‚ β”œβ”€β”€ AlertBadge.tsx # Unacknowledged count badge +β”‚ β”‚ └── AlertRow.tsx # Single alert in list +β”‚ β”‚ +β”‚ β”œβ”€β”€ charts/ +β”‚ β”‚ β”œβ”€β”€ AreaChart.tsx # Reusable area chart +β”‚ β”‚ β”œβ”€β”€ BarChart.tsx # Reusable bar chart +β”‚ β”‚ β”œβ”€β”€ DonutChart.tsx # Reusable donut chart +β”‚ β”‚ β”œβ”€β”€ GaugeChart.tsx # Target gauge +β”‚ β”‚ └── ComparisonBar.tsx # Two-location comparison bar +β”‚ β”‚ +β”‚ β”œβ”€β”€ layout/ +β”‚ β”‚ β”œβ”€β”€ Sidebar.tsx # Navigation sidebar +β”‚ β”‚ β”œβ”€β”€ Header.tsx # Top header bar +β”‚ β”‚ └── Footer.tsx +β”‚ β”‚ +β”‚ └── shared/ +β”‚ β”œβ”€β”€ LoadingSpinner.tsx +β”‚ β”œβ”€β”€ ErrorBoundary.tsx +β”‚ β”œβ”€β”€ EmptyState.tsx +β”‚ β”œβ”€β”€ DataCard.tsx # Reusable metric card +β”‚ └── PeriodSelector.tsx # Date range picker +β”‚ +β”œβ”€β”€ hooks/ +β”‚ β”œβ”€β”€ useKPI.ts # KPI data fetching (SWR) +β”‚ β”œβ”€β”€ useAlerts.ts # Alert polling +β”‚ β”œβ”€β”€ useLocation.ts # Current location context +β”‚ └── useAuth.ts # Auth state management +β”‚ +β”œβ”€β”€ lib/ +β”‚ β”œβ”€β”€ api-client.ts # Axios/fetch wrapper +β”‚ β”œβ”€β”€ formatters.ts # Currency, percent, date formatters +β”‚ └── constants.ts # API URLs, thresholds +β”‚ +└── types/ + β”œβ”€β”€ kpi.ts # KPI type definitions + β”œβ”€β”€ alerts.ts # Alert types + └── api.ts # API response types +``` + +### 5.2 Component Tree (Dashboard Page) + +``` + + β”œβ”€β”€
+ β”‚ β”œβ”€β”€ DataFreshnessBanner + β”‚ └── AlertBadge (unacked count, links to alerts page) + β”‚ + β”œβ”€β”€ + β”‚ β”œβ”€β”€ Nav: Dashboard, Alerts, Settings + β”‚ └── Location filter controls + β”‚ + └──
+ β”œβ”€β”€ (7d, 30d, MTD, QTD, custom) + β”‚ + β”œβ”€β”€ + β”‚ β”œβ”€β”€ Tab: "Combined" + β”‚ β”œβ”€β”€ Tab: "Oakland Park" + β”‚ └── Tab: "Fort Lauderdale" + β”‚ + β”œβ”€β”€ + β”‚ β”œβ”€β”€ + β”‚ β”œβ”€β”€ + β”‚ β”œβ”€β”€ + β”‚ └── + β”‚ + β”œβ”€β”€ (SpotOn vs Plaid deposit match) + β”‚ + β”œβ”€β”€
+ β”‚ β”œβ”€β”€ (Revenue timeseries) + β”‚ └── (Profit = Revenue - Costs stacked) + β”‚ + β”œβ”€β”€
+ β”‚ β”œβ”€β”€ (COGS, Labor, Rent, Ops) + β”‚ └── (24.8% vs 25% target) + β”‚ + └── (Oakland Park vs FTL table + bars) + +``` + +### 5.3 Data Fetching Strategy + +``` +Primary Pattern: SWR (stale-while-revalidate) +- Dashboard fetches GET /kpi/summary on load (cache: 60s) +- Individual widgets can hydrate from summary OR fetch detailed endpoints +- Revalidation interval: 120 seconds (polls for fresh data) +- On focus revalidation: enabled (when user returns to tab) + +Cache Strategy: +- API responses cached in Redis for 60s (backend) +- SWR client cache: 120s with stale-while-revalidate +- Manual refresh button available for instant re-fetch + +Optimistic Loading: +- Show last-known data immediately (SWR cache) +- Show skeleton/Glimmer while first load is in progress +- Pulse animation on widgets that are being refreshed +- Error state: show stale data + "Last updated: X ago β€” refresh failed" banner +``` + +### 5.4 State Management + +``` +No Redux needed for this scope. Use: +1. SWR for server state (all API data) +2. React Context for: + - AuthProvider (user, token, permissions) + - LocationProvider (selected location filter) + - PeriodProvider (selected date range) +3. Local state for UI interactions (expanded sections, modal open/close) +``` + +--- + +## 6. Alerting System + +### 6.1 Alert Severity Levels + +| Level | Color | Trigger | Action | +|-------|-------|---------|--------| +| **CRITICAL** | πŸ”΄ Red | System down, data loss, auth failure | Email + SMS (future) + persistent notification | +| **FAILURE** | 🟠 Orange | Ingestion job failed after retries, webhook delivery failure | Email + in-app | +| **ERROR** | 🟑 Yellow | Partial failure, single entity fetch failed, rate limit hit | Email (digest) + in-app | +| **SECURITY** | πŸ”΄ Red | Token refresh failed, unauthorized access detected, credential expiry < 7 days | Email + in-app | +| **WARNING** | πŸ”΅ Blue | KPI threshold breached, labor % > target, reconciliation variance > 5% | Email (digest) + in-app | +| **INFO** | βšͺ Gray | Successful sync, token refreshed, connection restored | In-app only | + +### 6.2 Alert Rules Engine + +``` +╔══════════════════════════════════════════════════════════════════════╗ +β•‘ ALERT RULES β•‘ +╠══════════════════════════════════════════════════════════════════════╣ +β•‘ β•‘ +β•‘ β”Œβ”€ SYSTEM HEALTH ──────────────────────────────────────────────┐ β•‘ +β•‘ β”‚ QBO: 5+ consecutive auth failures β†’ CRITICAL β”‚ β•‘ +β•‘ β”‚ Plaid: ITEM_LOGIN_REQUIRED webhook received β†’ ERROR β”‚ β•‘ +β•‘ β”‚ SpotOn: API tunnel down for >15 min β†’ CRITICAL β”‚ β•‘ +β•‘ β”‚ BullMQ queue backlog >1000 jobs β†’ ERROR β”‚ β•‘ +β•‘ β”‚ Database connection failure β†’ CRITICAL β”‚ β•‘ +β•‘ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β•‘ +β•‘ β•‘ +β•‘ β”Œβ”€ INGESTION ─────────────────────────────────────────────────┐ β•‘ +β•‘ β”‚ CDC sync returns 0 records for 3+ consecutive polls (businessβ”‚ β•‘ +β•‘ β”‚ hours) β†’ WARNING β”‚ β•‘ +β•‘ β”‚ Plaid sync fails after all retries β†’ FAILURE β”‚ β•‘ +β•‘ β”‚ Webhook processing fails β†’ ERROR β”‚ β•‘ +β•‘ β”‚ Rate limit exceeded β†’ ERROR β”‚ β•‘ +β•‘ β”‚ SpotOn orders deep sync gap detected β†’ ERROR β”‚ β•‘ +β•‘ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β•‘ +β•‘ β•‘ +β•‘ β”Œβ”€ RECONCILIATION ────────────────────────────────────────────┐ β•‘ +β•‘ β”‚ SpotOn sales vs Plaid deposits vary >10% β†’ WARNING β”‚ β•‘ +β•‘ β”‚ Net Profit mismatch with QBO P&L >5% β†’ WARNING β”‚ β•‘ +β•‘ β”‚ Manual export data differs from API data β†’ ERROR β”‚ β•‘ +β•‘ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β•‘ +β•‘ β•‘ +β•‘ β”Œβ”€ KPI THRESHOLDS ────────────────────────────────────────────┐ β•‘ +β•‘ β”‚ Labor cost % consistently above 27% (7d avg) β†’ WARNING β”‚ β•‘ +β•‘ β”‚ Labor cost % above 30% (any single day) β†’ ERROR β”‚ β•‘ +β•‘ β”‚ Cost growth exceeds revenue growth by 5%+ β†’ WARNING β”‚ β•‘ +β•‘ β”‚ Daily net profit decline >20% WoW β†’ WARNING β”‚ β•‘ +β•‘ β”‚ Sales decline >15% vs same day last week β†’ WARNING β”‚ β•‘ +β•‘ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β•‘ +β•‘ β•‘ +β•‘ β”Œβ”€ SECURITY ─────────────────────────────────────────────────┐ β•‘ +β•‘ β”‚ QBO refresh token expires in <7 days β†’ SECURITY β”‚ β•‘ +β•‘ β”‚ QBO refresh token expired β†’ CRITICAL β”‚ β•‘ +β•‘ β”‚ Plaid Item requires user re-authentication β†’ SECURITY β”‚ β•‘ +β•‘ β”‚ JWT token validation failure spike β†’ SECURITY β”‚ β•‘ +β•‘ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β•‘ +β•‘ β•‘ +β•šβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β• +``` + +### 6.3 Email Alert Format + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ πŸ”΄ [CRITICAL] Spice N'Vybe BI β€” Ingestion Failure β”‚ +β”‚ β”‚ +β”‚ Source: QuickBooks Online β”‚ +β”‚ Time: 2026-07-01 08:30:00 EDT β”‚ +β”‚ Job ID: j-abc12345 β”‚ +β”‚ β”‚ +β”‚ Detail: CDC sync failed after 3 retries. β”‚ +β”‚ Last successful sync: 2026-06-30 22:00 EDT β”‚ +β”‚ Error: HTTP 401 β€” token may be expired β”‚ +β”‚ β”‚ +β”‚ Action: Immediate attention required. β”‚ +β”‚ β†’ Check QBO token status in admin panel β”‚ +β”‚ β†’ Re-authorize via OAuth if refresh token expired β”‚ +β”‚ β†’ Dashboard data is stale since 2026-06-30 22:00 β”‚ +β”‚ β”‚ +β”‚ Dashboard: https://bi.spicenvybe.com/admin/ingestion β”‚ +β”‚ β”‚ +β”‚ This alert auto-resolves when the next successful sync completes. β”‚ +β”‚ To silence, acknowledge in the dashboard. β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +### 6.4 Alert Delivery Pipeline + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Alert Triggered │────►│ Deduplication │────►│ Rate Limiter β”‚ +β”‚ (code location) β”‚ β”‚ (same alert β”‚ β”‚ (max 5/min per β”‚ +β”‚ β”‚ β”‚ within 1h) β”‚ β”‚ alert type) β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Email Sent │◄────│ SMTP Sender │◄────│ Queue β”‚ +β”‚ (via Nodemailer) β”‚ β”‚ (Gmail App β”‚ β”‚ BullMQ β”‚ +β”‚ β”‚ β”‚ Password) β”‚ β”‚ priority queue β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +### 6.5 Email Implementation + +```javascript +// nodemailer configuration +const transporter = nodemailer.createTransport({ + host: 'smtp.gmail.com', + port: 587, + secure: false, + auth: { + user: process.env.ALERT_EMAIL_USER, + pass: process.env.ALERT_EMAIL_PASS // Gmail App Password + } +}); + +// Dual recipient list +const ALERT_RECIPIENTS = { + CRITICAL: ['owner@spicenvybe.com', 'manager+urgent@spicenvybe.com'], + FAILURE: ['owner@spicenvybe.com', 'ops@spicenvybe.com'], + ERROR: ['ops@spicenvybe.com'], + SECURITY: ['owner@spicenvybe.com', 'it@spicenvybe.com'], + WARNING: [] // Dashboard-only, can enable later +}; +``` + +--- + +## 7. Security Model + +### 7.1 Authentication Architecture + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ AUTH FLOW β”‚ +β”‚ β”‚ +β”‚ 1. User navigates to login.bi.spicenvybe.com β”‚ +β”‚ 2. Enters credentials β†’ POST /api/v1/auth/login β”‚ +β”‚ 3. Express validates against PostgreSQL users table (bcrypt hash) β”‚ +β”‚ 4. JWT issued: { sub, role, exp, iat } β”‚ +β”‚ - Access token: 2h expiry (short-lived) β”‚ +β”‚ - Refresh token: 14d expiry (secure cookie, httpOnly) β”‚ +β”‚ 5. All subsequent requests pass JWT in Authorization: Bearer headerβ”‚ +β”‚ β”‚ +β”‚ Auth Provider Decision: Google OAuth or Authentik (TBD) β”‚ +β”‚ For now: email+password with bcrypt. Ready for OIDC migration. β”‚ +β”‚ When OIDC is implemented, Authentik handles the IdP and Passport β”‚ +β”‚ authenticates the JWT from Authentik's callback. β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +### 7.2 Role-Based Access Control + +| Role | Permissions | Description | +|------|-------------|-------------| +| `admin` | Full system access, triggers sync, acknowledges alerts | System administrator | +| `manager` | View all KPIs, acknowledge alerts, view admin status | Operations manager | +| `viewer` | View KPIs and reports only | Read-only access | +| `system` | Internal service accounts, webhook receivers | Machine-to-machine | + +### 7.3 External API Credential Management + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ CREDENTIAL STORAGE β”‚ +β”‚ β”‚ +β”‚ QBO: β”‚ +β”‚ └─ Encrypted at rest in meta_qbo_credentials table β”‚ +β”‚ └─ Encryption: AES-256-GCM with key from environment variable β”‚ +β”‚ └─ Rotation: Access token auto-refreshed every 50 min β”‚ +β”‚ └─ Refresh token: rotated on every use (OAuth 2.0 spec) β”‚ +β”‚ └─ Token expiry alert: SECURITY at 7 days before refresh expiry β”‚ +β”‚ β”‚ +β”‚ Plaid: β”‚ +β”‚ └─ client_id + secret in environment variables (server-side only) β”‚ +β”‚ └─ access_token encrypted in meta_plaid_cursors table β”‚ +β”‚ └─ Plaid Link token generated server-side, 1-time use β”‚ +β”‚ └─ ITEM_LOGIN_REQUIRED triggers SECURITY alert for re-auth β”‚ +β”‚ β”‚ +β”‚ SpotOn: β”‚ +β”‚ └─ API key per location, encrypted in meta_locations table β”‚ +β”‚ └─ Keys stored with "Location" header in API calls β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +### 7.4 API Security Measures + +``` +═══ LAYER 1: TRANSPORT ═══ + β€’ HTTPS only (TLS 1.2+), no HTTP in production + β€’ HSTS headers enabled + β€’ Self-signed certs for homelab β†’ Let's Encrypt for production + +═══ LAYER 2: REQUEST ═══ + β€’ Rate limiting per IP: 100 req/min (global), 10 req/min (auth endpoints) + β€’ JWT validation on every authenticated endpoint + β€’ CORS: restricted to dashboard origin only + β€’ Request body size limit: 1MB + β€’ Helmet.js security headers (XSS, clickjack, MIME sniffing) + +═══ LAYER 3: EXTERNAL ═══ + β€’ QBO webhook: HMAC-SHA256 signature verification + β€’ Plaid webhook: Plaid-Verification header + signing key verification + β€’ SpotOn: x-api-key per location + β€’ No external credentials in client-side code (ever) + β€’ All API keys retrieved from encrypted DB storage at request time + +═══ LAYER 4: INFRASTRUCTURE ═══ + β€’ Database: PostgreSQL user with SCHEMA-level permissions + β€’ App DB user: SELECT/INSERT/UPDATE on app tables only, no DDL + β€’ Redis: password-protected (requirepass) + β€’ Backend runs as non-root user in container + β€’ All secrets via environment variables, never in code +``` + +### 7.5 QBO OAuth 2.0 Flow (Detailed) + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Dashboard Admin UI β”‚ β”‚ Express Backend β”‚ β”‚ Intuit OAuthβ”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ β”‚ β”‚ + β”‚ POST /auth/qbo/connect β”‚ β”‚ + β”‚ (admin JWT) β”‚ β”‚ + β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β–Ίβ”‚ β”‚ + β”‚ β”‚ Generate OAuth URL: β”‚ + β”‚ β”‚ client_id, redirect_uri, β”‚ + β”‚ β”‚ scope, response_type=codeβ”‚ + β”‚ β”‚ state (random, stored) β”‚ + β”‚ Redirect user to Intuit β”‚ β”‚ + │◄────────────────────────────── β”‚ + β”‚ β”‚ β”‚ + β”‚ User authorizes at Intuit β”‚ β”‚ + β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β–Ίβ”‚ + β”‚ β”‚ β”‚ + β”‚ Intuit redirects to: β”‚ β”‚ + β”‚ /auth/qbo/callback?code=X β”‚ β”‚ + β”‚ &state=Y&realmId=Z β”‚ β”‚ + β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β–Ίβ”‚ β”‚ + β”‚ β”‚ Validate state β”‚ + β”‚ β”‚ Exchange code for tokens β”‚ + β”‚ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β–Ίβ”‚ + β”‚ │◄─────────────────────────── + β”‚ β”‚ { access_token, β”‚ + β”‚ β”‚ refresh_token, β”‚ + β”‚ β”‚ expires_in, x_refresh_ β”‚ + β”‚ β”‚ token_expires_in } β”‚ + β”‚ β”‚ β”‚ + β”‚ β”‚ Encrypt & store in β”‚ + β”‚ β”‚ meta_qbo_credentials β”‚ + β”‚ β”‚ β”‚ + β”‚ β”‚ Init cursor entries in β”‚ + β”‚ β”‚ meta_qbo_cdc_cursors β”‚ + β”‚ β”‚ β”‚ + β”‚ "QBO connected successfully" β”‚ β”‚ + │◄────────────────────────────── β”‚ +``` + +--- + +## 8. Deployment Architecture (LXC) + +### 8.1 Container Topology + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Proxmox Host (spicenvybe-dev) β”‚ +β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ LXC Container β”‚ β”‚ LXC Container β”‚ β”‚ +β”‚ β”‚ spicenvybe-db β”‚ β”‚ spicenvybe-app β”‚ β”‚ +β”‚ β”‚ (PostgreSQL + β”‚ β”‚ (Node.js + β”‚ β”‚ +β”‚ β”‚ Redis) β”‚ β”‚ Next.js + β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ nginx) β”‚ β”‚ +β”‚ β”‚ RAM: 2GB β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ CPU: 2 cores β”‚ β”‚ RAM: 4GB β”‚ β”‚ +β”‚ β”‚ Disk: 20GB β”‚ β”‚ CPU: 4 cores β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ Disk: 30GB β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ Mount: /mnt/spicenvybe-backups -> Proxmox host ZFS pool β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +### 8.2 Service Stack (within spicenvybe-app LXC) + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ nginx (reverse proxy on host port 443 β†’ local) β”‚ +β”‚ β”œβ”€ /api/* β†’ Express (port 3001) β”‚ +β”‚ β”œβ”€ / β†’ Next.js (port 3000) β”‚ +β”‚ └─ /ws β†’ WebSocket for live updates β”‚ +β”‚ β”‚ +β”‚ pm2 (process manager) β”‚ +β”‚ β”œβ”€ next-app β”‚ port 3000 β”‚ Next.js server β”‚ +β”‚ β”œβ”€ express-api β”‚ port 3001 β”‚ Express backend β”‚ +β”‚ β”œβ”€ ingestion-worker β”‚ β€” β”‚ BullMQ worker β”‚ +β”‚ β”œβ”€ scheduler β”‚ β€” β”‚ node-cron tasks β”‚ +β”‚ └─ alert-worker β”‚ β€” β”‚ Alert evaluator β”‚ +β”‚ β”‚ +β”‚ Redis (port 6379, password-protected) β”‚ +β”‚ β”œβ”€ Queue: qbo-sync β”‚ +β”‚ β”œβ”€ Queue: plaid-sync β”‚ +β”‚ β”œβ”€ Queue: spoton-sync β”‚ +β”‚ β”œβ”€ Queue: reconciliation β”‚ +β”‚ └─ Cache: API responses (60s TTL) β”‚ +β”‚ β”‚ +β”‚ PostgreSQL (port 5432, app DB user) β”‚ +β”‚ β”œβ”€ spicenvybe_bi database β”‚ +β”‚ └─ Tables: raw_*, normalized_*, agg_*, meta_* β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +### 8.3 SystemD Integration + +```ini +# /etc/systemd/system/spicenvybe-bi.target +[Unit] +Description=Spice N'Vybe BI Dashboard +Requires=spicenvybe-db.service +After=network.target +Wants=spicenvybe-next.service spicenvybe-api.service spicenvybe-worker.service spicenvybe-scheduler.service + +# Individual services start via pm2, managed by: +[Unit] +Description=PM2 for Spice N'Vybe BI +[Service] +Type=forking +User=spicenvybe +ExecStart=/usr/bin/pm2 start /opt/spicenvybe-bi/ecosystem.config.js +ExecReload=/usr/bin/pm2 reload all +ExecStop=/usr/bin/pm2 kill +Restart=on-failure + +[Install] +WantedBy=multi-user.target +``` + +### 8.4 Environment Configuration + +```bash +# /opt/spicenvybe-bi/.env (example, actual values in vault) +# Homelab config β€” will differ for production + +# Database +DATABASE_URL="postgresql://spicenvybe:${DB_PASS}@localhost:5432/spicenvybe_bi" +REDIS_URL="redis://:${REDIS_PASS}@localhost:6379" + +# Encryption (used for stored credentials) +CREDENTIAL_ENCRYPTION_KEY="" + +# JWT +JWT_SECRET="" +JWT_ACCESS_EXPIRY="2h" +JWT_REFRESH_EXPIRY="14d" + +# QuickBooks Online +QBO_CLIENT_ID="" +QBO_CLIENT_SECRET="" +QBO_REDIRECT_URI="https://bi.spicenvybe.com/api/v1/auth/qbo/callback" +QBO_WEBHOOK_VERIFIER="" +QBO_MINOR_VERSION="75" + +# Plaid +PLAID_CLIENT_ID="" +PLAID_SECRET="" +PLAID_ENVIRONMENT="sandbox" # Change to 'production' for prod +PLAID_WEBHOOK_SECRET="" + +# Email (Gmail App Password) +ALERT_EMAIL_USER="alerts@spicenvybe.com" +ALERT_EMAIL_PASS="" +ALERT_EMAIL_FROM="Spice N'Vybe BI " + +# Frontend +NEXT_PUBLIC_API_URL="https://bi.spicenvybe.com/api/v1" +``` + +### 8.5 Backup Strategy + +``` +Schedule: Daily at 12:00 AM +Target: PostgreSQL dump + Redis RDB snapshot +Retention: 7 daily + 4 weekly + 3 monthly + +pg_dump -U spicenvybe -Fc spicenvybe_bi > /mnt/spicenvybe-backups/bi-$(date +%Y%m%d).dump +redis-cli SAVE # Creates dump.rdb +cp /var/lib/redis/dump.rdb /mnt/spicenvybe-backups/redis-$(date +%Y%m%d).rdb + +Cleanup: Keep 7 days locally. Optional S3 sync for compliance. +``` + +--- + +## 9. AWS Migration & Scaling Strategy + +### 9.1 Target AWS Architecture + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ AWS Cloud β”‚ +β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ Route 53 β”‚ β”‚ +β”‚ β”‚ bi.spicenvybe.com β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ CloudFront CDN β”‚ β”‚ +β”‚ β”‚ (Static assets, Next.js SSR cache) β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ Application Load Balancer β”‚ β”‚ WAF (Web ACL) β”‚ β”‚ +β”‚ β”‚ (HTTPS termination) β”‚ β”‚ Rate limiting, SQLi, β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ XSS protection β”‚ β”‚ +β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β–Ό β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ ECS Fargate Cluster (app) β”‚ β”‚ ElastiCache Redis β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ (Clustered, multi-AZ) β”‚ β”‚ +β”‚ β”‚ Service: next-app (x2) β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ Service: express-api (x2) β”‚ β”‚ +β”‚ β”‚ Service: ingestion-queue β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ Service: scheduler β”‚ β”‚ RDS PostgreSQL β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ (db.t3.medium β†’ db.r6g.large) β”‚ β”‚ +β”‚ β”‚ Auto-scaling: β”‚ β”‚ Multi-AZ standby β”‚ β”‚ +β”‚ β”‚ CPU > 70% β†’ +1 container β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ Scale max: 4 per service β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ S3 (static assets) β”‚ β”‚ Secrets Manager β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +### 9.2 Migration Phasing + +``` +PHASE 1 β€” HOMELAB (Now) +β”œβ”€β”€ Complete system running in Proxmox LXC +β”œβ”€β”€ PostgreSQL + Redis on same container +β”œβ”€β”€ nginx self-signed certs +β”œβ”€β”€ Daily backups to host ZFS pool +└── Monitoring: pm2 + systemd + +PHASE 2 β€” AWS LIFT-AND-SHIFT (Month 1) +β”œβ”€β”€ Deploy to single EC2 t3.medium (seed data) +β”œβ”€β”€ RDS PostgreSQL (db.t3.small, Multi-AZ) +β”œβ”€β”€ ElastiCache Redis (1 node, test) +β”œβ”€β”€ Route 53 DNS cutover +β”œβ”€β”€ CloudFront for static assets +└── ACM cert (Let's Encrypt or AWS) + +PHASE 3 β€” CONTAINERIZATION (Month 2) +β”œβ”€β”€ Dockerize all services +β”œβ”€β”€ ECS Fargate deployment +β”œβ”€β”€ ALB + WAF in front +β”œβ”€β”€ Auto-scaling policies +└── Blue-green deployments + +PHASE 4 β€” OPTIMIZATION (Month 3+) +β”œβ”€β”€ RDS read replica for reporting queries +β”œβ”€β”€ CloudFront for Next.js ISR caching +β”œβ”€β”€ Redis cluster for queue resilience +β”œβ”€β”€ S3 β†’ Athena for long-term analytics +└── CloudWatch alarms + PagerDuty +``` + +### 9.3 Scaling Dimensions + +| Dimension | Homelab Limit | AWS Scaling Strategy | +|-----------|---------------|---------------------| +| **Database connections** | 100 (PostgreSQL default) | RDS Proxy + connection pooling | +| **API throughput** | Single Express process | ECS Fargate horizontal scaling | +| **Redis memory** | 1GB host limit | ElastiCache: up to 340GB clustered | +| **Storage** | 30GB LXC disk | RDS storage auto-scaling (up to 64TB) | +| **Backups** | Local ZFS | RDS automated backups + S3 exports | +| **Ingestion workers** | Single worker process | Worker service with concurrency config | +| **Alert delivery** | Nodemailer direct | Amazon SES (send 50K+ emails/day) | + +### 9.4 Production Readiness Checklist + +``` +Before going live with Phase 2: +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ ☐ Database migration scripts version-controlled β”‚ +β”‚ ☐ Secrets in AWS Secrets Manager (not .env) β”‚ +β”‚ ☐ WAF rules enabled (rate limiting, SQLi, XSS) β”‚ +β”‚ ☐ RDS automated backups configured (7-day retention) β”‚ +β”‚ ☐ Multi-AZ RDS enabled β”‚ +β”‚ ☐ Health check endpoints for all services β”‚ +β”‚ ☐ Prometheus/Grafana monitoring (or CloudWatch) β”‚ +β”‚ ☐ Log aggregation (CloudWatch Logs or Loki) β”‚ +β”‚ ☐ Deployment pipeline (GitHub Actions β†’ ECR β†’ ECS) β”‚ +β”‚ ☐ Rollback procedure documented and tested β”‚ +β”‚ ☐ Incident response runbook for each failure scenario β”‚ +β”‚ ☐ Load test executed (k6 or Artillery) β”‚ +β”‚ ☐ SSL certificate β†’ auto-renewal configured β”‚ +β”‚ ☐ Email deliverability tested (SPF, DKIM, DMARC) β”‚ +β”‚ ☐ QBO Production app credentials (separate from sandbox)β”‚ +β”‚ ☐ Plaid Production credentials (replace sandbox keys) β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +--- + +## 10. Data Reconciliation & Consistency Strategy + +### 10.1 Three-Way Verification Model + +``` +╔══════════════════════════════════════════════════════════════════════╗ +β•‘ RECONCILIATION FRAMEWORK β•‘ +╠══════════════════════════════════════════════════════════════════════╣ +β•‘ β•‘ +β•‘ KPI: Sales β•‘ +β•‘ ────────────────────────────────────────────────────────────────── β•‘ +β•‘ PRIMARY: SpotOn Orders (sum of completed order totals) β•‘ +β•‘ VERIFY: Plaid Bank Deposits (revenue-size credits) β•‘ +β•‘ GOLDEN: QBO P&L Income line (monthly anchor) β•‘ +β•‘ β•‘ +β•‘ Reconciliation: β•‘ +β•‘ Daily: SpotOn sales β‰ˆ Plaid deposits (allow Β±5% for tips/cash) β•‘ +β•‘ Monthly: SpotOn total vs QBO P&L Income (must match Β±1%) β•‘ +β•‘ If variance: log alert, flag for manual review β•‘ +β•‘ β•‘ +β•‘ KPI: Costs β•‘ +β•‘ ────────────────────────────────────────────────────────────────── β•‘ +β•‘ PRIMARY: QBO (COGS from Purchase/Bill, Expenses from Account) β•‘ +β•‘ SUPPLEMENT: Plaid (categorized bank transactions for misc costs) β•‘ +β•‘ LABOR: SpotOn Labor Reports (wages + taxes, by location) β•‘ +β•‘ β•‘ +β•‘ Reconciliation: β•‘ +β•‘ Monthly: QBO Cost totals β‰ˆ Plaid expense categories (allowing β•‘ +β•‘ for non-bank costs like credit card auto-pay) β•‘ +β•‘ Weekly: Labor cost (SpotOn) vs QBO Payroll account activity β•‘ +β•‘ β•‘ +β•‘ KPI: Net Profit β•‘ +β•‘ ────────────────────────────────────────────────────────────────── β•‘ +β•‘ CALCULATED: agg_daily_profit_loss (Sales - Costs) β•‘ +β•‘ VERIFY: QBO P&L Net Income (direct report) β•‘ +β•‘ β•‘ +β•‘ Reconciliation: β•‘ +β•‘ Monthly: Calculated net profit must match QBO P&L within 5% β•‘ +β•‘ If mismatch: flag all three sources for variance analysis β•‘ +β•šβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β• +``` + +### 10.2 Daily Reconciliation Pipeline + +``` +Daily at 7:00 AM (after all overnight syncs completed) + +STEP 1: Extract SpotOn Daily Sales + SELECT location_id, sale_date, net_sales FROM agg_daily_sales + +STEP 2: Extract Plaid Daily Deposits + SELECT location_id, transaction_date, + SUM(ABS(amount)) AS total_deposits + FROM normalized_bank_transactions + WHERE amount < 0 AND is_reconciled = false + GROUP BY location_id, transaction_date + +STEP 3: Match & Update + For each location, for each date: + Compare SpotOn net_sales vs Plaid deposits + If within threshold (default 5%) β†’ mark Reconciled + If outside β†’ queue WARNING alert + +STEP 4: Update agg_daily_reconciliation materialized view + REFRESH MATERIALIZED VIEW CONCURRENTLY agg_daily_reconciliation; +``` + +### 10.3 Monthly Reconciliation Pipeline + +``` +1st of every month at 6:00 AM + +STEP 1: Calculate from Aggregations + SELECT location_id, month, + SUM(revenue) AS total_revenue_calc, + SUM(costs) AS total_costs_calc, + SUM(profit) AS net_profit_calc + FROM agg_monthly_comparison + WHERE month = date_trunc('month', CURRENT_DATE - INTERVAL '1 month') + GROUP BY location_id, month + +STEP 2: Fetch QBO P&L Report + Poll QBO for the completed month's P&L report + Parse: parse ProfitAndLossReport columns: + - Income Total + - COGS Total + - Expenses Total + - Net Income + +STEP 3: Compare + Revenue: calc vs QBO income total + Costs: calc vs QBO (COGS + Expenses) + Net Profit: calc vs QBO Net Income + +STEP 4: Adjust or Alert + If variance < 1% β†’ mark as fully reconciled + If variance 1-5% β†’ log WARNING, trigger review + If variance > 5% β†’ log ERROR, flag for immediate investigation +``` + +### 10.4 Handling Known Reconciliation Gaps + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ GAP: SpotOn includes sales tax collected; QBO excludes tax β”‚ +β”‚ FIX: Track tax separately, deduct from SpotOn before comparison β”‚ +β”‚ β”‚ +β”‚ GAP: Plaid deposits include credit card fees (net deposit) β”‚ +β”‚ FIX: Apply estimated fee percentage (2.5-3.5%) when matching β”‚ +β”‚ FUTURE: Map exact fees from QBO Payment Processing cost category β”‚ +β”‚ β”‚ +β”‚ GAP: Cash sales don't appear as Plaid deposits β”‚ +β”‚ FIX: Track cash percentage from SpotOn, exclude from reconciliationβ”‚ +β”‚ FUTURE: Compare cash sales against cash drops recorded in QBO β”‚ +β”‚ β”‚ +β”‚ GAP: QBO costs include non-cash items (depreciation, accruals) β”‚ +β”‚ FIX: Flag these categories in normalized_qbo_costs, exclude from β”‚ +β”‚ cash-flow comparison but include in P&L comparison β”‚ +β”‚ β”‚ +β”‚ GAP: Timing differences (e.g., weekend batches settle Monday) β”‚ +β”‚ FIX: Allow Β±2 business day window for SpotOnβ†’Plaid reconciliation β”‚ +β”‚ Monthly reconciliation resolves all timing gaps definitively β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +### 10.5 Data Integrity Checks (Automated) + +```sql +-- Check 1: Orphaned raw records without normalized counterpart +SELECT 'SpotOn Orders' AS source, COUNT(*) AS orphaned +FROM raw_spoton_orders r +LEFT JOIN normalized_orders n ON r.spoton_order_id = n.spoton_order_id +WHERE n.id IS NULL AND r.ingested_at < NOW() - INTERVAL '1 hour' +UNION ALL +SELECT 'Plaid Transactions', COUNT(*) +FROM raw_plaid_transactions r +LEFT JOIN normalized_bank_transactions n ON r.plaid_txn_id = n.plaid_txn_id +WHERE n.id IS NULL AND r.ingested_at < NOW() - INTERVAL '1 hour'; + +-- Check 2: Duplicate detection +SELECT spoton_order_id, location_id, COUNT(*) as dupes +FROM normalized_orders +GROUP BY spoton_order_id, location_id +HAVING COUNT(*) > 1; + +-- Check 3: Gap detection (business hours with no data) +SELECT l.name, gs.date_gap AS missing_date +FROM meta_locations l +CROSS JOIN generate_series( + CURRENT_DATE - INTERVAL '7 days', CURRENT_DATE - INTERVAL '1 day', '1 day'::interval +) AS gs(date_gap) +LEFT JOIN agg_daily_sales s ON s.location_id = l.id AND s.sale_date = gs.date_gap +WHERE s.order_count IS NULL OR s.order_count = 0; + +-- Check 4: Negative cost amounts (data quality) +SELECT * FROM normalized_qbo_costs +WHERE amount < 0 AND cost_category NOT IN ('refund', 'credit'); + +-- Check 5: Labor % sanity (should be 10-50% of sales) +SELECT * FROM normalized_labor_costs +WHERE labor_percent < 0 OR labor_percent > 60; +``` + +--- + +## Appendix A: Project Directory Structure + +``` +/opt/spicenvybe-bi/ +β”œβ”€β”€ apps/ +β”‚ β”œβ”€β”€ web/ # Next.js frontend +β”‚ β”‚ β”œβ”€β”€ src/ +β”‚ β”‚ β”‚ β”œβ”€β”€ app/ # App router pages +β”‚ β”‚ β”‚ β”œβ”€β”€ components/ # React components +β”‚ β”‚ β”‚ β”œβ”€β”€ hooks/ # Custom hooks +β”‚ β”‚ β”‚ β”œβ”€β”€ lib/ # Utilities +β”‚ β”‚ β”‚ └── types/ # TypeScript types +β”‚ β”‚ β”œβ”€β”€ public/ # Static assets +β”‚ β”‚ β”œβ”€β”€ next.config.js +β”‚ β”‚ β”œβ”€β”€ tailwind.config.js +β”‚ β”‚ └── package.json +β”‚ β”‚ +β”‚ └── api/ # Express backend +β”‚ β”œβ”€β”€ src/ +β”‚ β”‚ β”œβ”€β”€ routes/ # Express route handlers +β”‚ β”‚ β”œβ”€β”€ services/ # Business logic +β”‚ β”‚ β”œβ”€β”€ adapters/ # External API adapters +β”‚ β”‚ β”‚ β”œβ”€β”€ qbo/ # QBO OAuth + CDC +β”‚ β”‚ β”‚ β”œβ”€β”€ plaid/ # Plaid sync +β”‚ β”‚ β”‚ └── spoton/ # SpotOn polling +β”‚ β”‚ β”œβ”€β”€ workers/ # BullMQ job processors +β”‚ β”‚ β”œβ”€β”€ middleware/ # Auth, rate-limit, error handling +β”‚ β”‚ β”œβ”€β”€ alerts/ # Alert engine +β”‚ β”‚ └── jobs/ # Cron job definitions +β”‚ β”‚ └── index.ts # Entry point +β”‚ β”œβ”€β”€ prisma/ +β”‚ β”‚ └── schema.prisma # Database schema +β”‚ └── package.json +β”‚ +β”œβ”€β”€ docker/ +β”‚ β”œβ”€β”€ Dockerfile.web +β”‚ β”œβ”€β”€ Dockerfile.api +β”‚ β”œβ”€β”€ docker-compose.yml # Local dev +β”‚ └── nginx/ +β”‚ β”œβ”€β”€ nginx.conf +β”‚ └── sites-available/bi.spicenvybe.com +β”‚ +β”œβ”€β”€ scripts/ +β”‚ β”œβ”€β”€ backup.sh # Daily backup +β”‚ β”œβ”€β”€ restore.sh # Disaster recovery +β”‚ β”œβ”€β”€ health-check.sh # System health +β”‚ └── seed-data.sql # Initial seed +β”‚ +β”œβ”€β”€ infra/ +β”‚ β”œβ”€β”€ terraform/ # AWS IaC (Phase 2+) +β”‚ β”‚ β”œβ”€β”€ main.tf +β”‚ β”‚ β”œβ”€β”€ variables.tf +β”‚ β”‚ └── outputs.tf +β”‚ └── ansible/ # LXC provisioning +β”‚ β”œβ”€β”€ playbook.yml +β”‚ └── roles/ +β”‚ +β”œβ”€β”€ .env.example +β”œβ”€β”€ ecosystem.config.js # PM2 config +└── README.md +``` + +--- + +## Appendix B: BullMQ Job Definitions + +```javascript +// ecosystem.config.js worker concurrency +module.exports = { + apps: [ + { + name: 'ingestion-worker', + script: 'dist/workers/index.js', + env: { + WORKER_CONCURRENCY: 5, + QBO_QUEUE_CONCURRENCY: 2, // Respect 500 req/min + PLAID_QUEUE_CONCURRENCY: 3, + SPOTON_QUEUE_CONCURRENCY: 3 + } + } + ] +}; + +// BullMQ Queue Definitions +const queues = { + qboSync: { + name: 'qbo-sync', + defaultJobOptions: { + attempts: 3, + backoff: { type: 'exponential', delay: 2000 }, + removeOnComplete: 100, + removeOnFail: 50 + } + }, + plaidSync: { + name: 'plaid-sync', + defaultJobOptions: { + attempts: 3, + backoff: { type: 'fixed', delay: 30000 }, + removeOnComplete: 100 + } + }, + spotonSync: { + name: 'spoton-sync', + defaultJobOptions: { + attempts: 2, + backoff: { type: 'exponential', delay: 1000 }, + removeOnComplete: 200 + } + }, + reconciliation: { + name: 'reconciliation', + defaultJobOptions: { + attempts: 1, // No retry β€” alert on failure + removeOnComplete: 30 + } + } +}; +``` + +--- + +## Appendix C: Cron Schedule Reference + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€ Source ─────────┬─────── Cron ───────┬─────────── Description ─────────────────┐ +β”‚ QBO CDC Poll β”‚ */30 * * * * β”‚ CDC sync every 30 minutes β”‚ +β”‚ QBO Full Sync β”‚ 0 2 * * * β”‚ Full CDC + P&L report at 2am β”‚ +β”‚ QBO Token Refresh β”‚ */50 * * * * β”‚ Token refresh every 50 min (before expiry)β”‚ +β”‚ QBO Weekly Integrity β”‚ 0 3 * * 0 β”‚ Full entity query on Sunday β”‚ +β”‚ β”‚ β”‚ β”‚ +β”‚ Plaid Incremental β”‚ */15 * * * * β”‚ Incremental transaction sync β”‚ +β”‚ Plaid Full Resync β”‚ 0 4 * * 0 β”‚ Full resync every Sunday at 4am β”‚ +β”‚ β”‚ β”‚ β”‚ +β”‚ SpotOn Orders (BH) β”‚ */5 8-23 * * 1-6 β”‚ 5-min polls during business hours β”‚ +β”‚ SpotOn Orders (Sun BH) β”‚ */5 9-21 * * 0 β”‚ 5-min polls Sunday hours β”‚ +β”‚ SpotOn Deep Sync β”‚ 0 3 * * * β”‚ 26h deep sync daily at 3am β”‚ +β”‚ SpotOn Labor Reports β”‚ 0 6 * * * β”‚ Labor report fetch at 6am β”‚ +β”‚ β”‚ β”‚ β”‚ +β”‚ Daily Reconciliation β”‚ 0 7 * * * β”‚ Sales Γ— Plaid match at 7am β”‚ +β”‚ Monthly Reconciliation β”‚ 0 6 1 * * β”‚ Full QBO P&L reconciliation on 1st β”‚ +β”‚ Database Backup β”‚ 0 0 * * * β”‚ PostgreSQL dump + Redis SAVE at midnight β”‚ +β”‚ Data Freshness Check β”‚ 0 8 * * * β”‚ Integrity checks: orphans, gaps, dupes β”‚ +β”‚ β”‚ β”‚ β”‚ +β”‚ Token Expiry Check β”‚ 0 9 * * * β”‚ Check QBO token <7d expiry β†’ SECURITY β”‚ +β”‚ Dead Letter Alert β”‚ 0 * * * * β”‚ Check BullMQ DLQ, create FAILURE alerts β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + +Note: All times in UTC unless otherwise noted. Business hours (BH) follow local Eastern Time. +``` + +--- + +## Appendix D: Key Design Decisions & Rationale + +| Decision | Choice | Rationale | +|----------|--------|-----------| +| **Warehouse pattern** (not direct query) | Ingestion Workers β†’ Normalization β†’ Aggregation | Decouples API failures from dashboard. Dashboard always works from last-good data. Rate limits don't impact UX. | +| **BullMQ over in-process queue** | Redis-backed BullMQ | Survives server restarts. Provides retry, DLQ, concurrency control. Monitoring dashboard for job health. | +| **Materialized Views over computed-at-query** | Pre-aggregated | Dashboard loads in <100ms regardless of data volume. Refresh on schedule. Concurrent refresh for zero-downtime. | +| **Separate RAW + NORMALIZED tables** | Dual-table pattern | Raw = immutable audit log (reprocessable). Normalized = cleaned, typed, indexed for query. Never lose original data. | +| **CDC over polling for QBO** | Change Data Capture primary | 500 req/min limit makes entity-level polling infeasible. CDC returns all changed entities in 2-3 calls vs 20+ individual fetches. | +| **SpotOn 5-min + 26h dual sync** | Hybrid polling | 5-min catches real-time orders. 26h deep sync catches missed records. Compensates for SpotOn's limited API window. | +| **Reconciliation as separate pipeline stage** | Post-ingestion verification | Don't trust any single source. Cross-verify every KPI against a secondary source. Catch issues before they reach the dashboard. | +| **PM2 over Docker in homelab** | Process manager | Simpler networking, lower overhead, easier debugging in single-LXC setup. Docker reserved for AWS migration phase. | +| **SWR over WebSockets** | Stale-while-revalidate | Simplified architecture for v1. Dashboard only needs 120s refresh. WebSockets add complexity (reconnection, state sync) without proportional benefit. | +| **Gmail App Password over SMTP service** | Nodemailer + Gmail | Free, zero configuration for homelab. Replace with SES on AWS for production volume and deliverability. | + +--- + +*End of Architectural Specification v1.0* + +This document serves as the authoritative blueprint for all M01 development. All changes and deviations must be documented and approved through the project's change management process. diff --git a/clients/spice-nvybe/research/api-data-sources.md b/clients/spice-nvybe/research/api-data-sources.md new file mode 100644 index 0000000..528e0d3 --- /dev/null +++ b/clients/spice-nvybe/research/api-data-sources.md @@ -0,0 +1,255 @@ +# Spice N'Vybe BI Dashboard β€” API Data Source Research + +**Context:** Restaurant with 2 locations (Oakland Park flagship + Fort Lauderdale Cloud Kitchen). Need a BI Dashboard for Sales, Net Profit, and Costs. + +--- + +## 1. QuickBooks Online (Accounting) + +### Auth Method +- **OAuth 2.0** (Authorization Code flow exclusively) +- Two primary scopes: + - `com.intuit.quickbooks.accounting` β€” full read/write to accounting data + - `com.intuit.quickbooks.payment` β€” QuickBooks Payments processing +- **Access tokens:** expire in 1 hour +- **Refresh tokens:** rotate every 24-26 hours, max lifetime 5 years. **Critical:** you must always store and use the *latest* refresh token β€” reusing a rotated one revokes the entire authorization chain. + +### Base URLs +- Production: `https://quickbooks.api.intuit.com/v3/company/{realmId}/` +- Sandbox: `https://sandbox-quickbooks.api.intuit.com/v3/company/{realmId}/` + +### Available Endpoints / Entities + +**Financial Reports (Reports API)** β€” most relevant for BI: +| Endpoint | What it returns | BI Relevance | +|---|---|---| +| `ProfitAndLoss` | Income, COGS, Expenses, Net Income | **Core for Net Profit calculation** | +| `BalanceSheet` | Assets, Liabilities, Equity snapshot | Treasury health | +| `CashFlow` | Operating/Investing/Financing cash flows | Cash position | +| `AgedReceivables` | Outstanding customer invoices | AR tracking | +| `AgedPayables` | Outstanding bills/vendor payments | AP tracking | +| `TrialBalance` | Full chart of accounts summary | Audit/completeness | + +**Transactional Entities (CRUD via Query API):** +- `Invoice` β€” customer invoices, line items, amounts, dates +- `Customer` β€” customer profiles +- `Payment` β€” received payments +- `Bill` β€” vendor bills +- `Purchase` β€” purchases (check, credit card, cash) +- `PurchaseOrder` β€” vendor POs +- `Vendor` β€” vendor profiles +- `Account` β€” chart of accounts +- `JournalEntry` β€” manual debit/credit entries +- `Deposit` β€” bank deposits +- `Transfer` β€” fund transfers between accounts + +**Query syntax:** SQL-like β€” `SELECT * FROM Invoice WHERE TxnDate > '2026-01-01'` + +### Sync Mechanism +- **Change Data Capture (CDC)** β€” returns only changed records since a timestamp. **Use this** instead of polling full datasets. +- **Webhooks** β€” event-driven notifications for entity changes. + +### Rate Limits +- **500 requests/minute per company (realmId)** +- **10 concurrent requests per company** +- Batch endpoint: 120 requests/minute per company + +### BI Relevance for Sales / Net Profit / Costs +- **Net Profit** β†’ Pull `ProfitAndLoss` report (Income - COGS - Expenses = Net Income) +- **Costs** β†’ Pull `ProfitAndLoss` for expense line items; `Bill`/`Purchase` entities for granular vendor spend +- **Sales** β†’ Pull `ProfitAndLoss` revenue lines; **but SpotOn is better for daily/weekly sales granularity** +- **AR/AP** β†’ `AgedReceivables`/`AgedPayables` reports + +> **Recommendation:** Pull QBO reports daily (or on-demand) for the P&L view. Use it as the **source of truth for net profit**, reconciling against SpotOn and Plaid data. The 1-hour access token and per-company rate limits mean you'll need a queued background worker. + +--- + +## 2. Plaid (Banking) + +### Auth Method +- **API Key-based** (client_id + secret from Plaid Dashboard) +- Sent as `PLAID-CLIENT-ID` and `PLAID-SECRET` headers, or in request body +- **User Linking Flow:** Plaid Link widget generates an `access_token` scoped to the user's financial accounts at their institution +- All requests are `POST` with JSON bodies + +### Base URLs +- Sandbox: `https://sandbox.plaid.com` +- Production: `https://production.plaid.com` + +### Available Products / Data Categories + +| Product | Endpoints | Data Provided | BI Relevance | +|---|---|---|---| +| **Transactions** | `/transactions/sync`, `/transactions/get`, `/transactions/recurring/get`, `/transactions/refresh` | Up to **24 months** of categorized transactions; merchant names, amounts, dates, categories (PFC v1 or v2 taxonomy), geolocation | **Core for cash flow analysis**, expense categorization | +| **Balance** | `/accounts/balance/get` | Real-time current & available balances | Working capital monitoring | +| **Auth** | `/auth/get` | Account & routing numbers | (Low BI relevance) | +| **Identity** | `/identity/get` | Account holder name, addresses, emails | (Low BI relevance) | +| **Income** | `/credit/income/get` | Income streams, paystub data | Revenue verification | +| **Investments** | `/investments/transactions/get` | Holdings and trades | (Low relevance unless they have investment accounts) | +| **Liabilities** | `/liabilities/get` | Credit cards, student loans, mortgages | Debt tracking | +| **Enrich** | Transaction category enrichment | Cleans raw bank descriptions with category metadata | Better expense categorization | +| **Assets** | `/asset_report/create` | Point-in-time financial snapshot | (Optional) | + +### Transaction Data Shape (key fields) +```json +{ + "transaction_id": "...", + "account_id": "...", + "amount": -45.50, + "iso_currency_code": "USD", + "date": "2026-06-15", + "name": "US Foods Inc", + "merchant_name": "US Foods", + "payment_channel": "in store", + "pending": false, + "category": ["Food and Drink", "Restaurants"], + "personal_finance_category": { + "detailed": "FOOD_AND_DRINK_RESTAURANTS" + }, + "location": {...} +} +``` + +### Sync Mechanism +- **`/transactions/sync`** (preferred) β€” cursor-based incremental sync. Track `next_cursor` for paginated updates. +- **`/transactions/get`** β€” older full-fetch approach (deprecated in favor of sync) +- **Webhooks:** `SYNC_UPDATES_AVAILABLE`, `INITIAL_UPDATE`, `HISTORICAL_UPDATE`, `DEFAULT_UPDATE`, `TRANSACTIONS_REMOVED` +- Plaid checks for new transactions 1-4 times per day per institution + +### Rate Limits +- Not explicitly documented as fixed caps per minute; they use per-request pricing tiers +- `RATE_LIMIT_EXCEEDED` errors are a defined error type β€” implement retry with backoff +- Max 500 transactions per `sync` call page; max 730 days of history + +### Pricing +- **Pay as You Go / Growth / Custom** tiers +- **Transactions** is included in all tiers (one-time or per-request model) +- Free Sandbox (200 API calls per product for initial testing) + +### BI Relevance for Sales / Net Profit / Costs +- **Expense categorization** β†’ Every bank transaction has a `personal_finance_category` (e.g., `FOOD_AND_DRINK_RESTAURANTS`, `RENT_AND_UTILITIES`, `SUPPLIES`). This gives you a **real-time, categorized expense feed** that's more granular than QBO. +- **Cash flow** β†’ Stream daily transactions to detect revenue deposits versus vendor payments +- **Reconciliation** β†’ Match bank transactions against QBO records for audit-tight financials +- **Cost monitoring** β†’ Spot unusual vendor charges, rent payments, utility bills as they post + +> **Recommendation:** Use Plaid Transactions sync for daily cash-flow monitoring and as a **cross-reference for costs**. The personal finance category taxonomy (PFC v2) maps well to COGS/vendor/operating expense categories. Use recurring transactions endpoint for subscription/monthly cost tracking. + +--- + +## 3. SpotOn (POS) + +### Auth Method +- **API Key** β€” simple `x-api-key` header in each request +- The POS Export API uses a per-location API key +- SpotOn also provides a separate **Enterprise API** (for enterprise/venue customers) with different endpoints + +### Base URL +- `https://restaurantapi-qa.spoton.com/posexport/v1/` (QA environment) +- The API is **location-centric** β€” all routes are scoped to `/locations/:locationId/` + +### Available Endpoints + +#### Transactional Endpoints + +| Endpoint | Path | Description | BI Relevance | +|---|---|---|---| +| **Orders** | `GET /locations/:id/orders` | Orders with checks, payments, menu items, modifiers, tips, guests | **Core β€” daily sales data** | +| **Orders (single)** | `GET /locations/:id/orders/:orderId` | Single order detail | Drill-down | +| **Paid In/Outs** | `GET /locations/:id/paid-in-outs` | Cash paid in/out transactions (tips paid out, petty cash, deposits) | Cash management | + +#### Reference Data Endpoints + +| Endpoint | Path | Description | BI Relevance | +|---|---|---|---| +| **Menu Items** | `GET /locations/:id/menu-items` | Menu items, prices, PLU codes, report categories | Item-level sales analysis | +| **Modifiers** | `GET /locations/:id/modifiers` | Modifier groups and options | Cost attribution | +| **Employees** | `GET /locations/:id/employees` | Employee profiles, roles | Labor attribution | +| **Payment Options** | `GET /locations/:id/payment-options` | Tender types (credit, cash, gift card) | Payment mix | +| **Report Categories** | `GET /locations/:id/report-categories` | Menu categorization groups | Category-level sales | + +#### Report Endpoints + +| Endpoint | Path | Description | BI Relevance | +|---|---|---|---| +| **Labor Reports** | `GET /locations/:id/reports/labor` | Time clock entries with regular/OT pay rates, tips | **Core β€” labor costs** | +| **Labor (date range)** | `GET /locations/:id/reports/labor?startDateKey=&endDateKey=` | Labor data across multiple business dates | Weekly/monthly labor | + +### Key Order Data Shape +Orders contain nested objects: +- **Order** β†’ `totalAmount`, `balanceDueAmount`, `orderTypeName`, `tableNumber`, `createdAt`, `closedAt`, `deleted` +- **Check** β†’ `gratuityAmount`, `totalAmount`, `paymentsAmount`, `balanceAmount` +- **Guest** β†’ `name`, `items[]`, `voidedItems[]` +- **MenuItem** β†’ `menuItemId`, `name`, `quantity`, `unitPriceAmount`, `categoryName`, `modifierGroups[]` +- **Payment** β†’ `amount`, `tipAmount`, `cardType`, `employeeId` + +### Constraints +- **Date range max:** 26 hours (can't pull more than 26h in a single orders request) +- **History:** Only orders last updated in the last ~90 days +- **Replication lag:** API reads from a secondary DB; 5-minute minimum lag for `updatedAtEnd` parameter +- **Orders retrieval patterns:** + - **Daily:** Pull last 26 hours of orders every 24 hours + - **Near-realtime:** Poll every 5 minutes with 5-minute lag, plus a daily 26-hour catch-up +- **Reference data:** Pull menu-items, modifiers, employees once daily; fetch individual ones on-demand + +### Labor Report +- Source: SpotOn's Reporting Warehouse (ETL-delayed, not real-time) +- Returns per-employee, per-shift entries with: + - `regularSecondsWorked`, `regularPayRateAmount`, `regularPayAmount` + - `overtimeSecondsWorked`, `overtimePayRateAmount`, `overtimePayAmount` + - `totalPayAmount`, `declaredCashTipsAmount` + - `unpaidBreakSeconds` +- **All historical data available** (not limited to 90 days like orders) + +### Additional Note: Unofficial API +SpotOn's official Export API is limited to reads. For **write operations** (menu updates, 86 management, reservation creation) or more comprehensive data coverage, **Supergood** offers an unofficial but production-tested API that: +- Handles MFA/session management +- Provides near-real-time order/payment/menu/reservation/labor data +- Supports webhooks for async events +- Required for two-way integration (QuickBooks sync, menu management) + +### BI Relevance for Sales / Net Profit / Costs +- **Sales** (by location, by item, by category) β†’ Orders endpoint with menu items and report categories +- **Labor costs** β†’ Labor Reports endpoint β€” regular pay, overtime, tips per employee per day +- **Payment mix** β†’ Payment options on each check (credit vs cash vs gift card) +- **Void/waste tracking** β†’ voided items in order data +- **Period comparison** β†’ Pull orders by `closedAt` ranges for daily/weekly/monthly comparisons +- **Costs** β†’ SpotOn doesn't track vendor costs directly (that's QBO's domain), but paid-in-outs capture cash expenses + +> **Recommendation:** SpotOn is your **source of truth for Sales** β€” daily sales by item/category/location. Labor data via the Warehouse reports gives you **labor cost %** against sales. The 26-hour window constraint means you'll need a continuous polling pattern (every 5 min during operating hours, daily catch-up). + +--- + +## Cross-Source Integration Strategy for Spice N'Vybe BI Dashboard + +| Metric | Primary Source | Cross-Reference | Sync Cadence | +|---|---|---|---| +| **Sales (daily, by location)** | SpotOn Orders | Plaid deposits (revenue matching) | Every 5 min (realtime), daily catch-up | +| **Sales (by menu item)** | SpotOn Orders (menuItems) | β€” | Daily | +| **COGS** | QuickBooks P&L | Plaid vendor payments | Daily | +| **Labor Costs** | SpotOn Labor Reports | QuickBooks Payroll expenses | Daily | +| **Operating Expenses** | QuickBooks P&L | Plaid transaction categories | Daily | +| **Net Profit** | QuickBooks P&L | Aggregated from Sales - Costs | Daily (on-demand) | +| **Cash Flow** | Plaid Transactions | QuickBooks CashFlow report | Daily | +| **AP/AR** | QuickBooks AgedPayables / AgedReceivables | Plaid for check clearing | Daily | + +### Data Pipeline Architecture Suggestion +``` +SpotOn ──► (ETL: every 5min) ──► Staging DB (orders, labor, menu data) +Plaid ──► (ETL: daily sync) ──► Staging DB (transactions, balances) +QBO ──► (ETL: daily sync) ──► Staging DB (reports, invoices, bills) + β”‚ + β–Ό + BI Dashboard DB + (aggregated views) + β”‚ + β–Ό + Analytics Queries + (Sales by item, Profit by location, + Cost trends, Cash position) +``` + +### Key Technical Considerations +1. **QBO OAuth token rotation requires a background worker** that refreshes tokens proactively (tokens expire in 1h, refresh tokens rotate every 24-26h) +2. **SpotOn's 26-hour window** means you can't backfill deep history from orders β€” start the pipeline ASAP +3. **Plaid's recurring transactions endpoint** (`/transactions/recurring/get`) is excellent for tracking predictable costs (rent, utilities, subscriptions) +4. **All three APIs support webhooks** β€” consider event-driven architecture instead of polling where possible