Skip to content

Busy Level Managment Service

Purpose

The busy level management service performs demand suppression on delivery platforms by adjusting fees & fulfillment times automatically, based on the number of items currently being prepared at each station. In addition to the automated level setting mode, it simplifies manual level setting syncing a chosen level for a station to delivery platforms. As a fallback, the tool can be paused to allow level setting directly on Uber & GrubHub.

Development

The busy level management service lives in the busy_level_management_service directory. It can be started via: 

uvicorn app.main:app --host 0.0.0.0 --port 8004 --reload

Architecture Overview

The busy level management service is built up from four sub-services - the busy levels service, two delivery platforms services (uber service and GrubHub service), and the credentials service.

Busy Levels Service

The busy levels service is responsible for the internal model of busy levels - it is generally unaware of the state of fees & prep times on the platforms. It only knows the current busy level and what fees & times it should correspond to. It handles manual level sets, automated busy level calculation, determining if the level is different from the previous one. Once it determines what has changed from a levels perspective, it dispatches to the platform services to determine what differs from a fee & fulfillment time perspective on each platform.

Delivery Platform Services

These services are responsible for interaction with the delivery platforms - updating settings, caching configurations, and determining if syncs are required. They are split into two pieces (a core.py and jobs.py file). The core file contains an actual service class, which encapsulates the API of the platform, making requests and caching results. The jobs file contains helpers which look at the cached configuration, determine what settings are already synced, and build sync worker jobs as needed. Finally, there are update_fulfillment_config workers which execute those jobs via the core service class.

Credentials Service

The credentials service exists to support the delivery platform services by handling the storing & updating of authentication tokens. It has endpoints to accept manual updates which are required for Uber, since it uses a 30-day unrefreshable token. It also uses a worker to automatically refresh GrubHub credentials, which consist of an hourly-expiring but refreshable token. It relies on the respective platform services to actually make the authentication requests, but manages expiration-related alerts and encrypting, storing, fetching, and decrypting of credentials.

Key Points

  • The automatic busy level setting worker runs every minute, and is performed for all stores at once.
  • The APIs are rate limited. We handle this by performing updates via workers, and setting rate limits on the worker queues to only process up to X workers (and therefore requests) per minute. For Uber, the limit is 30. For GrubHub, at least 60 requests / minute are supported, but GrubHub requires two requests per update, and so its queue is also limited to 30.
  • The code for handling shift changes currently assumes all stores are in Eastern Time. In the future a timezone for each location may need to be specified.

Level Setting Diagram

This diagram shows the flow of automatic level setting. However, the flow is largely the same in the case of a manual level request. 

sequenceDiagram
    participant Cron as Cron (every minute)
    participant BLS as BusyLevelService
    participant Jobs as jobs.py (GH / Uber)
    participant DB as PostgreSQL
    participant Worker as Worker (Chancy, 30 req/min)
    participant Core as core.py (GH / Uber)
    participant API as Platform API

    Cron->>BLS: trigger evaluation

    BLS->>DB: fetch active busy_level_definitions
    BLS->>DB: count items from otter.orders + order_items
    BLS->>DB: fetch current busy_level_states
    BLS->>BLS: for unpaused stations<br/>match item counts against thresholds, diff state
    break no changes
        BLS->>Cron: exit
    end
    BLS->>DB: write updated states + logs

    BLS->>Jobs: resolve syncs for changed levels
    Jobs->>DB: fetch delivery_platform_configurations cache
    Jobs->>Jobs: compare settings vs cache
    break no changes
        Jobs->>DB: mark log as synced
        Jobs->>Cron: exit
    end
    Jobs-->>BLS: required platform syncs
    BLS->>Worker: queue sync jobs
    Worker->>Core: execute platform update
    Core->>DB: load delivery_platform_credentials
    Core->>API: update fulfillment config
    API-->>Core: response
    Core->>DB: cache new delivery_platform_configurations
    Core-->>Worker: result
    Worker->>DB: mark log as synced / failed

Onboarding a new brand

1. Add a mapping in otter.store_stations for the new store to its appropriate station.

2. Ensure the tech@orbitalkitchens.com accounts on Uber & GrubHub have access to the store.

  • Reach out to Linh if either account is missing permissions.

3. Pause the tool for the station the new store belongs to.

4. Add an entry in busy_levels.store_delivery_platform_merchant_ids for the new store on Uber, with its store_id from otter.stores, platform set to uber, and merchant_id retrieved from Uber:

  1. Log in to the tech@orbitalkitchens.com account on Uber Eats Manager
  2. Open browser developer tools and navigate to the Network tab
  3. Visit the delivery settings page
  4. Select the new store from the dropdown
  5. Find the POST request to /graphql with operationName: "getVdfDeliverySettings" in the payload (usually the last request)
  6. In the payload, search for merchantUuid and copy the value

5. Add an entry in busy_levels.store_delivery_platform_merchant_ids for the new store on GrubHub, with its store_id from otter.stores, platform set to grubhub, and merchant_id retrieved from GrubHub:

  1. Log in to the tech@orbitalkitchens.com account on GrubHub For Merchants
  2. Open browser developer tools and navigate to the Network tab
  3. Visit the Order & delivery settings page
  4. Select the new store from the dropdown
  5. Adjust the "Average food prep time" (and then return it to its original value)
  6. Find the PATCH request to /merchant/{merchant_uuid} and copy the value from the URL

6. Pre-fetch the configuration of the store on each platform

  1. On the tool/sync-delivery-platforms branch the busy level management service main.py has code to sync stores by UUID.
  2. Place the UUID in the appropriate CSV file
  3. Run it for Uber & GrubHub and check that busy_levels.delivery_platform_configurations are created.

7. Resume the tool for the station

  1. Set to a manual level
  2. Verify syncs
  3. Return to automatic level setting
  4. Verify syncs

Onboarding a new location

1. Populate busy_levels.busy_level_definitions for each station at the location.

  • This can be done via the UI in OK Manager, or you can write a script.
  • Discuss with Ops to determine what the item counts for each level should be.
  • Of note, the levels for the PM shift need to be split into one entry with an end time of 24:00, and a second entry starting at 00:00.

2. Populate busy_levels.delivery_platform_settings for each busy level created in step 1. Again, get fees & fulfillment times from Ops.

3. Complete the brand onboarding steps for each brand at the new location.