Leader Middleware — Functional Specification

What This App Does

Shopify embedded app that syncs a product catalog from the Leader supplier system into a Shopify store. It ingests products from Leader's ASMX API and XML feed, normalizes and merges the data, applies pricing rules and field mappings, then pushes products to Shopify via the Admin GraphQL API. It also manages inventory across 5 Australian warehouse regions, publishes products to sales channels, and provides scheduling, monitoring, and error resolution tools.

Core Data Flow

Leader ASMX API ──┐
                   ├──► Ingest & Merge ──► Local Product Database ──► Shopify Sync ──► Shopify Admin API
Leader XML Feed ──┘

Functional Capabilities

1. Product Ingestion from Leader

What it does: Pulls product data from Leader's partner ASMX web service endpoints. Products are fetched by category, vendor, or SKU search. The API returns raw product records which are normalized into a standard format.

Key behaviors:

• Fetches categories and vendors from Leader to build a local taxonomy index
• Fetches products by iterating through each (category, subcategory) pair — one API call per unique pair
• Supports bounded concurrency (configurable 1–8 parallel API calls, default 4)
• 30-minute hard timeout prevents runaway ingests
• Per-request timeout of 45 seconds with 3 retry attempts (exponential backoff) for transient failures (429, 502–504)
• XML feed is supplementary — it enriches existing API-ingested products but never creates new ones
• Each ingested product is merged (API data + optional XML data) into a NormalizedLeaderProduct with 49 fields

Product data captured:

• Identity: SKU, title, description, manufacturer SKU, barcode, warranty months
• Vendor: ID, name, tencia code
• Category: name, subcategory, tencia code, tencia sub-code
• Pricing: 8 price points — trade ex/inc GST, RRP ex/inc GST, XML dealer buy price, XML RRP
• Dimensions: weight (kg), box length/width/height (mm)
• Inventory: per-region quantities for 5 Australian regions (NSW, QLD, VIC, WA, SA) with ETAs
• Images: up to 10 image URLs per product (slot 1 is primary)
• Metadata: favourite flag, total availability from XML

2. Change Detection

What it does: Computes content and inventory hashes from the normalized product data. On re-ingest, compares hashes to detect what changed.

• Content hash covers all product fields except inventory quantities
• Inventory hash covers only regional quantities and ETAs
• Content hash changed → product flagged for content re-sync to Shopify
• Inventory hash changed → product flagged for inventory re-sync
• Both unchanged → no action (skip)
• Hashes stored on the product mapping record alongside Shopify GIDs

3. Field Mapping & Transforms

What it does: Configurable mapping rules that control how Leader product fields translate to Shopify product/variant fields and metafields.

4 target types:

• Product core fields (title, description, vendor, productType, handle, status, tags)
• Variant core fields (sku, price, compareAtPrice, barcode, taxable, inventoryPolicy)
• Product metafields (custom data on the product)
• Variant metafields (custom data on the variant)

13 transform types:

• direct — pass through raw value
• static — fixed value (e.g., always set status to "ACTIVE")
• template — interpolate {{fieldName}} placeholders
• prefix / suffix — prepend/append text
• regex — search/replace with pattern (capped at 200 chars for ReDoS safety)
• math — value × factor + offset with configurable decimal places
• to_upper / to_lower / trim — string transforms
• shopify_handle — slugify for URL handles
• conditional — value based on condition
• lookup — map values via a lookup table
• chain — apply multiple transforms in sequence

Additional features:

• Required field validation — products missing required source values are skipped with an error
• Default values — fallback when source field is empty
• Metafield type coercion — number_integer values truncated to integers, boolean values normalized
• Live preview — transforms can be previewed against sample data before saving

4. Pricing Engine

What it does: Calculates the final Shopify price from Leader's raw pricing data using configurable rules.

Configuration options:

• Base price field selector (which of the 8 Leader price points to use)
• Default markup percentage
• Maximum price ceiling
• Minimum price floor (fixed dollar amount or cost + percentage strategy)
• Price rounding (none, nearest $1/$5/$10, round up to .99/.95)
• Category-specific markup overrides
• Vendor-specific markup overrides
• Compare-at price strategy (use RRP field, calculate from cost, or disabled)

Per-product overrides:

• Manual price override per product (takes precedence over calculated pricing)
• Compare-at price override
• Override removal re-queues the product for recalculation on next sync

Bulk price adjustment:

• Select multiple products and apply percentage or fixed-amount adjustments
• Dry-run preview shows impact before committing
• Creates ProductPriceOverride records

5. Shopify Product Sync

What it does: Pushes product data to Shopify via the productSet GraphQL mutation, supporting both single-product and bulk operations.

Single-product sync:

• Builds a ProductSetInput payload from field mappings + pricing + images
• Calls productSet with synchronous: true
• Updates the local ProductMapping with Shopify GIDs (product, variant, inventory item)
• Records content/inventory hashes for change detection

Bulk content sync:

• Builds a JSONL file with one ProductSetInput per product
• Staged upload to Shopify
• bulkOperationRunMutation processes all products server-side
• Polls for completion (5-second interval, 30-minute timeout)
• Downloads result JSONL and correlates per-product results
• Updates ProductMapping GIDs and hashes for successful products
• Records errors per product on the mapping record
• Self-healing: when Shopify returns "Product does not exist" for a stale GID, the mapping GIDs are cleared so the next sync creates the product fresh

Images: Up to 10 images per product resolved from Leader CDN base URL + filename. Sent as files array in the ProductSetInput.

6. Product Publishing

What it does: Publishes synced products to selected Shopify sales channels (Online Store, POS, Shop, etc.).

• Uses bulkOperationRunMutation with publishablePublish mutation
• Builds JSONL with one line per product (product GID + publication GIDs)
• Staged upload + async bulk operation
• Polls for completion

Channel selection (priority order):

1. Per-step option (comma-separated publication GIDs configured on the schedule)
2. Global setting (selected via Publications settings page)
3. All channels (fallback if nothing configured)

7. Inventory Sync

What it does: Pushes regional inventory quantities to Shopify, mapped to specific Shopify locations.

• 5 Australian regions (NSW, QLD, VIC, WA, SA) mapped to Shopify location GIDs
• Quantities come from the normalized Leader product data
• Optional hold-back buffer (0–50%) reduces reported stock to prevent overselling
• Inventory can be set inline during content sync or via a dedicated inventory sync step
• Per-SKU inventory push available for individual products

8. Shopify Collections

What it does: Creates Shopify smart collections from Leader categories and vendors.

Category collections: Smart collection rules based on product type and/or tags. Subcategories use TAG matching, top-level categories use TYPE matching. Bulk creation for all unmapped categories.

Vendor collections: Smart collection rule: VENDOR = vendor name. Products automatically included when their vendor field matches. Bulk creation for all unmapped vendors.

9. Metafield Definitions

What it does: Creates Shopify metafield definitions from enabled field mapping rules, so metafield values appear correctly in the Shopify admin.

• Reads enabled METAFIELD_PRODUCT and METAFIELD_VARIANT field mappings
• Creates metafield definitions with correct type, namespace, and key
• Handles "already exists" gracefully (upserts the field mapping, skips definition creation)
• Single or bulk sync of definitions

10. Scheduling & Job Queue

What it does: Runs sync pipelines on configurable cron schedules with a built-in job queue.

Scheduling:

• Visual cron builder (hourly/daily/weekly with time selectors)
• Each schedule defines which pipeline steps to run and in what order
• Per-step options (e.g., which publication channels for the publish step)
• Enable/disable individual schedules
• Next 5 runs preview

Job queue:

• Jobs are created from schedules or manual triggers
• Statuses: pending → running → completed/failed/cancelled
• Per-step progress tracking (items processed, items failed, duration, error message)
• Retry failed/cancelled jobs
• Cancel running jobs
• 30-minute lock per shop prevents overlapping runs

Pipeline presets:

• Full Sync: cleanup-logs → taxonomy-sync → leader-ingest → content-sync → publish-products → inventory-sync
• Leader Sync: cleanup-logs → taxonomy-sync → leader-ingest
• Content Sync: content-sync → publish-products
• Inventory Sync: inventory-sync
• Category Sync: taxonomy-sync

Individual pipeline steps:

• Cleanup Logs — delete SyncLog entries older than 30 days
• Taxonomy Sync — pull categories + vendors from Leader API
• Leader Ingest — ingest all products by category (bounded concurrency, 30-min timeout)
• Content Sync — bulk productSet push via staged upload + bulkOperationRunMutation
• Publish Products — bulk publish via bulkOperationRunMutation + publishablePublish
• Inventory Sync — per-SKU inventory push via productSet with inline quantities

Job type toggles: Each of the 5 job types can be independently enabled/disabled. Disabled jobs return immediately when triggered.

11. Webhook Handling

What it does: Listens for Shopify webhook events and reacts to external changes.

Handled events:

• products/delete — clears Shopify GIDs on the local ProductMapping, flags for re-sync
• collections/delete — mirrors collection deletions
• orders/create, orders/updated — mirrors order data locally
• customers/create, customers/update — mirrors customer data locally
• bulk_operations/finish — handles bulk operation completion callbacks
• app/uninstalled — cleans up session data
• app/scopes_update — handles OAuth scope changes

Registration: Webhook subscriptions declared in app TOML configuration. Can be registered via "Register All" button in the UI or via shopify app deploy CLI.

12. Error Resolution

What it does: Surfaces products that failed to sync and provides resolution tools.

• Paginated list of products with sync errors (SKU, error message, category, timestamp)
• Per-product "Retry" — sets needsSync=true and clears the error so the next sync retries
• Per-product "Dismiss" — clears the error without retrying
• "Retry all" bulk action
• Errors are stored on the ProductMapping record (lastError field)

13. Monitoring & Logging

What it does: Provides visibility into sync operations, API calls, and system health.

Sync logs:

• Every sync operation writes to a SyncLog table (level, category, message, metadata JSON)
• Filterable by level (info/warn/error) and category
• 30-day retention (cleanup step deletes older entries)
• All 35 API routes log their operations

Shopify API call logging:

• Every GraphQL request/response logged to NDJSON files (var/api-logs/)
• Also persisted to SyncLog table (survives container restarts)
• Configurable verbosity: full (no truncation), truncated-responses (16KB cap), truncated-all
• Correlated by call ID (request + response share the same ID)

Leader API call logging:

• Every ASMX request/response logged to NDJSON files
• Includes request fields, response status, row count, duration, retry attempts

Dashboard metrics:

• 7-day daily sync activity chart with week-over-week change percentages
• Content/inventory sync progress bars
• Stock status summary (in stock, low stock, out of stock)
• Recent activity feed with category badges and level indicators
• Pending queue depths

Live monitoring:

• Global status bar polls for running job status (3-second interval when active)
• Pipeline stepper shows completed/active/pending steps during sync runs
• Job detail page with per-step progress cards

Health probes: /api/health (liveness) and /api/ready (readiness)

14. Location Management

What it does: Maps Shopify store locations to Leader warehouse regions.

5 regions: NSW, QLD, VIC, WA, SA. Each mapping: Shopify location GID ↔ Leader region code. Used by inventory sync to set per-location stock quantities.

15. Developer Tools

What it does: Provides destructive operations for development and testing environments.

• Table truncation (unprotected tables only)
• Catalog data reset (multi-table transaction)
• Shopify watermark purge (null all GIDs, flag everything for re-sync)
• Clear all logs

Data Model

Product Data

Configuration

Job Queue

Audit & Mirrors

Authentication

• Admin UI access: Shopify OAuth session — user must be authenticated as a store admin
• Scheduled/external API calls: Bearer token (SCHEDULER_CRON_SECRET env var) + X-Shopify-Shop-Domain header
• Webhook verification: Shopify HMAC signature validation
• Leader API calls: CustomerCode form parameter (stored as env var, never logged)

Technical Constraints

• Shopify App Home: UI must use only Polaris web components (s-* prefix) — no @shopify/polaris React components, no Tailwind, no custom CSS frameworks
• Embedded navigation: Must use client-side navigation (React Router) — full page loads break session token handling in the embedded admin
• Bulk operations: Only one Shopify bulk operation can run at a time per shop
• Rate limits: Shopify GraphQL Admin API uses a cost-based rate limit (1000 points, 50/second restore). Each mutation costs ~10 points.
• Leader API: ASMX endpoints with form-urlencoded POST. Can be slow (45-second timeout). Transient failures on 429/502/503/504 are retried.
• Database: PostgreSQL in production, SQLite for development/CI
• Deployment: Railway with Docker — ephemeral filesystem (log files don't survive restarts, but SyncLog DB table does)