Accepts Costco receipt images, processes them using a self-hosted AI pipeline (self-hosted Vision Language model for OCR + self hosted Reasoning model for categorization), and sends an expense summary to a Slack webhook categorized into:
- Groceries
- Household (Non-Grocery)
- Others
For detailed technical design, see the Architecture Overview.
- Docker and Docker Compose installed on your machine.
- Local LLM server running (e.g., Ollama serving
qwen2.5-vl:7bandgemma4:31b). - A Slack Webhook URL.
-
Environment Variables: Copy the example environment file and configure it:
cp .env.example .env
Open
.envand configure your settings: -
Run the Application: Start the backend using Docker Compose. It will build the container and start the API server on port 8000.
docker compose up -d --build
- Stage 1: OCR
OCR_BASE_URL: URL to your Vision-Language Model API (e.g.,http://192.168.1.100:11434/v1).OCR_MODEL_NAME: The vision model for extracting text (e.g.,qwen2.5-vl:7b).
- Stage 2: Reasoning
REASONING_BASE_URL: URL to your reasoning model API (e.g.,http://192.168.1.100:11434/v1).REASONING_MODEL_NAME: The text model for categorization (e.g.,gemma4:31b).
- Slack
SLACK_WEBHOOK_URL: Your Slack webhook URL to receive summaries.
- Storage & System
STORE_IMAGES: Set totrueto keep uploaded receipt images in theuploads/directory, orfalseto discard them.STORAGE_DIR: Directory where images are stored (mapped to./uploadson the host).MAX_RETRIES: Number of times to retry querying the models if the JSON response format is malformed (default 2).
Upload one or more images of a single Costco receipt at http://your-ip:8000
Receipt Processing Summary
Groceries: $117.33
1) Kirkland Signature Cookie Tin: $19.99
2) Griesson Hazelnut Crème Biscuits: $13.49
3) Kelsen Danish Butter Cookies: $18.99
4) Honey Roasted Nuts: $14.89
5) Chicken Nuggets 52oz: $35.98
6) Kirkland Signature Praline: $13.99
Household (Non-Grocery): $9.99
1) 3M Scissors: $9.99Others: $19.94
1) Gap Oxford Top: $19.94Grand Total: $147.26 | Items: 8
Validation warnings:
• Item count mismatch: receipt says 10 items, but we parsed 8 (excluding 0 discount(s))
2026-06-10 23:46:17,628 [INFO] Starting receipt processing (2 image(s))
2026-06-10 23:46:17,629 [INFO] Encoding 2 image(s) to base64...
2026-06-10 23:46:17,629 [INFO] Encoding complete
2026-06-10 23:46:17,629 [INFO] [OCR] Sending 2 image(s) to qwen2.5vl:7b @ http://<your_ip>:11434
2026-06-10 23:46:41,606 [INFO] [OCR] Responded in 24.0s
2026-06-10 23:46:41,606 [INFO] [OCR] Raw response:
{
"items": [
{"item": "KS COOK TIN", "price": 19.99},
{"item": "HAZELBISCUIT", "price": 13.49},
{"item": "KELSENCOOKIE", "price": 18.99},
{"item": "3MSCISSOR", "price": 9.99},
{"item": "GAPOXFORDTOP", "price": 19.94},
{"item": "HNY RST NUTS", "price": 14.89},
{"item": "NUGGETS52OZ", "price": 35.98},
{"item": "KS PRALINE", "price": 13.99}
],
"receipt_totals": {
"items_sold": 10,
"subtotal": 147.26
}
}
2026-06-10 23:46:41,607 [INFO] [OCR] Extracted 8 raw item(s)
2026-06-10 23:46:41,607 [INFO] [Reasoning] Sending 8 item(s) to llama3.3:70b for categorization...
2026-06-10 23:47:37,727 [INFO] [Reasoning] Response finished (reason=stop, max_tokens=8192)
2026-06-10 23:47:37,727 [INFO] [Reasoning] Raw response:
{
"items": [
{"raw_name": "KS COOK TIN", "item": "Kirkland Signature Cookie Tin", "price": 19.99, "category": "groceries", "needs_lookup": false},
{"raw_name": "HAZELBISCUIT", "item": "Hazel Biscuit", "price": 13.49, "category": "groceries", "needs_lookup": true},
{"raw_name": "KELSENCOOKIE", "item": "Kelsen Cookie", "price": 18.99, "category": "groceries", "needs_lookup": true},
{"raw_name": "3MSCISSOR", "item": "3M Scissors", "price": 9.99, "category": "household", "needs_lookup": false},
{"raw_name": "GAPOXFORDTOP", "item": "Gap Oxford Top", "price": 19.94, "category": "others", "needs_lookup": true},
{"raw_name": "HNY RST NUTS", "item": "Honey Roasted Nuts", "price": 14.89, "category": "groceries", "needs_lookup": false},
{"raw_name": "NUGGETS52OZ", "item": "Chicken Nuggets 52oz", "price": 35.98, "category": "groceries", "needs_lookup": false},
{"raw_name": "KS PRALINE", "item": "Kirkland Signature Praline", "price": 13.99, "category": "groceries", "needs_lookup": false}
]
}
2026-06-10 23:47:37,727 [INFO] [Reasoning] Categorized 8 item(s)
2026-06-10 23:47:37,727 [INFO] Running post-processing...
2026-06-10 23:47:37,727 [INFO] Running fuzzy deduplication...
2026-06-10 23:47:37,727 [INFO] No duplicates found.
2026-06-10 23:47:37,727 [INFO] Validating against receipt totals...
2026-06-10 23:47:37,727 [WARNING] Item count mismatch: receipt says 10 items, but we parsed 8 (excluding 0 discount(s))
2026-06-10 23:47:37,727 [INFO] Subtotal matches: $147.26
2026-06-10 23:47:37,727 [INFO] 3 flagged item(s) — resolving via web search...
2026-06-10 23:47:37,727 [INFO] Flagged item: 'HAZELBISCUIT' — searching the web...
2026-06-10 23:47:37,727 [INFO] Web search query: 'HAZELBISCUIT costco' (backends: brave, duckduckgo, google)
2026-06-10 23:47:38,531 [INFO] Web search context for 'HAZELBISCUIT' (5 result(s)): Get Griesson Hazelnut Crème Biscuits delivered to you in as fast as 1 hour with Costco same-day delivery or curbside pickup. | Nutella Biscuits, 9.7 oz, 4 ct A crunchy golden baked biscuit Filled with Nutella hazelnut spread 20 biscuits per bag ... Costco Business Delivery can only accept orders for this item from retailers holding a Costco Business membership with ... | July 1, 2024 - Laura Lamb, who runs the Costco fan account @costcohotfinds on Instagram, spotted boxes of Nutella Biscuits (140 calories per serving) at her local Costco this week and quickly shared the news with her
2026-06-10 23:47:44,789 [INFO] Categorized with context: 'HAZELBISCUIT' → 'Griesson Hazelnut Crème Biscuits' (groceries)
2026-06-10 23:47:44,789 [INFO] Resolved 'HAZELBISCUIT' → 'Griesson Hazelnut Crème Biscuits' (groceries)
2026-06-10 23:47:44,789 [INFO] Flagged item: 'KELSENCOOKIE' — searching the web...
2026-06-10 23:47:44,789 [INFO] Web search query: 'KELSENCOOKIE costco' (backends: brave, duckduckgo, google)
2026-06-10 23:47:45,940 [INFO] Web search context for 'KELSENCOOKIE' (5 result(s)): September 28, 2025 - Costco's 5-pound tins of Kelsen Danish Butter Cookies are back on shelves for the season. Read on to see why the vintage butter cookies and their nostalgic blue tins are so beloved. | December 5, 2021 - Update: Went to Costco today. Bought the same cookies. Opened the tin and the cookies had the same appearance. Can confirm: still taste delicious and the texture is still great. | October 11, 2025 - The second I saw that Costco was selling a giant blue tin of Danish cookies, I felt like I was nine. Kelsen Danish Butter
2026-06-10 23:47:51,106 [INFO] Categorized with context: 'KELSENCOOKIE' → 'Kelsen Danish Butter Cookies' (groceries)
2026-06-10 23:47:51,106 [INFO] Resolved 'KELSENCOOKIE' → 'Kelsen Danish Butter Cookies' (groceries)
2026-06-10 23:47:51,106 [INFO] Flagged item: 'GAPOXFORDTOP' — searching the web...
2026-06-10 23:47:51,106 [INFO] Web search query: 'GAPOXFORDTOP costco' (backends: brave, duckduckgo, google)
2026-06-10 23:47:51,769 [INFO] Web search context for 'GAPOXFORDTOP' (5 result(s)): Shop Costco.com for electronics, computers, furniture, outdoor living, appliances, jewelry and more. Enjoy low warehouse prices on name-brands products delivered to your door | Browse the newest women's fashion, including jackets, tops, sweaters & cardigans, sleepwear, dresses, pants, skirts and more! Shop online at Costco.com today. | Shop Costco's Union Gap, WA location for electronics, groceries, small appliances, and more. Find quality brand-name products at warehouse prices. | April 20, 2026 - The secret to a perfect weekend? These new PJs. ☁️✨ I picked up this 2-pack of Gap cotton PJ pants at Costco today for only $15.99. They are
2026-06-10 23:47:56,670 [INFO] Categorized with context: 'GAPOXFORDTOP' → 'Gap Oxford Top' (others)
2026-06-10 23:47:56,671 [INFO] Resolved 'GAPOXFORDTOP' → 'Gap Oxford Top' (others)
2026-06-10 23:47:56,671 [INFO] Summary: Groceries=$117.33 | Household=$9.99 | Others=$19.94 | Total=$147.26
2026-06-10 23:47:56,671 [INFO] Sending results to Slack...
2026-06-10 23:47:56,885 [INFO] Slack message sent successfully
2026-06-10 23:47:56,885 [INFO] Done