The Destiny Item Manager API (DIM Sync)

Destiny Item Manager (DIM) uses the Bungie.net API to read information about Destiny game state, and to move or change items. However, DIM offers other features that Bungie.net's API does not support: tags and notes for items, saved loadouts, etc. To allow users to save this data and sync it between different clients (mobile, desktop, etc), we built our own API, which is branded as "DIM Sync" in our application. While this API was developed with DIM in mind, it is not exclusive to DIM. We designed it to be used by other Destiny community tools, and we welcome them to use it. Today, these other applications make use of DIM Sync:

• D2Checklist - You can sync your notes and tags between DIM and D2Checklist.
• light.gg - You can sync your tags with their Roll Appraiser tool. It can also automatically tag rolls that are popular among the community.
• Destiny Recipes - Their Vault Cleaner allows you to tag your items based on certain rules.

What this isn't

This API is not the Bungie.net API! You cannot load Destiny inventory or move items with this API - that's the Bungie.net API. You also cannot remote-control DIM with this - it does not support applying DIM loadouts, or initiating DIM's Smart Moves. This API is limited to saving and loading information about tags, notes, saved DIM loadouts, and other information that is not part of the Bungie.net API.

API Types

The types for all API requests and responses are written out as TypeScript in the api/shapes folder. You can use the npm package @destinyitemmanager/dim-api-types to reference these types in your own code. This file in DIM's source code also lists out all the API endpoints, and shows examples of how to call them.

Updating the package version in dim-api-types/package.json and pushing to master will automatically publish to NPM.

Get an API key

To use the DIM API, you will need a DIM API key. Anyone can get an API key for localhost development using the instructions below.

Before you can interact with the DIM API you must set up your application to talk to the Bungie.net API. DIM's API piggybacks on Bungie.net's authentication: you need to have your users logged in via the Bungie.net API first, and then that token is used to get a token to talk to the DIM API. You will need to be ready with the Bungie.net API information for your development application.

Once your application is set up at Bungie.net, you can generate a DIM API key for localhost development. The payload is JSON, and requires three pieces of information:

• an id for your app (please use the format username-dev)
• your Bungie.net API key
• the full address of your local development server.

Using your preferred method of making HTTP calls, perform a request to the DIM API's new_app endpoint:

curl 'https://api.destinyitemmanager.com/new_app' \
-X 'POST' \
-H 'Content-Type: application/json' \
--data-binary '{"id":"myusername-dev","bungieApiKey":"my-bungie-api-key","origin":"https://localhost:8080"}'

The response will contain a dimApiKey field - that's your DIM API key. If you ever forget it, make the request above again, and the same key will be returned to you.

Alternatively, you can set up DIM for local development — the "Enter API Credentials" step brings you to a developer page, which will have a button to fetch a DIM API key as well.

Once you have finished developing your app locally and are ready to push it to a production domain, join the DIM Discord and message bhollis to get a production token. Be sure to explain what you're building and how it will use the DIM API - if you don't, your request will be ignored.

Authenticating

As mentioned above, DIM's API piggybacks on Bungie.net's authentication: it knows a user is who they claim to be, because DIM confirms that their credentials check out with Bungie.net.

To authenticate a user of your application with the DIM Sync API, you will exchange their Bungie.net auth token for a DIM auth token. Just like the Bungie.net authentication system, the DIM auth token has an expiration time, after which you'll have to get a new one. Unlike the Bungie.net API, there's no refresh token — you just make the same request you made for the initial token. Here's how DIM does it.

To issue or reissue the DIM API token, you'll need your user's Bungie.net membership ID, an unexpired and valid Bungie.net access token, and your DIM API key. The DIM API will query the Bungie.net API using the user's access token, to verify that it belongs to the given membership ID — we don't store the token or use it for anything else.

curl 'https://api.destinyitemmanager.com/auth/token'
-X 'POST' \
-H 'Content-Type: application/json'
-H 'X-API-Key: dimApiKey'
{
  bungieAccessToken: 'foo',
  membershipId: '1234',
}

The returned token has an expiration - do not use the token after that expiration. For all subsequent DIM API calls, you'll include the DIM API token and the DIM API key as HTTP headers, like so:

Authorization: Bearer xxxThisUsersDimAccesTokenxxx
X-API-Key: xxxYourDimApiKeyxxx

Reading profile data

The DIM API is meant to be familiar to users of the Bungie.net API — there is a single, central read API: GET https://api.destinyitemmanager.com/profile?platformMembershipId=1234&components=tags,loadouts

The platform membership ID is required and should correspond to the Bungie.net platform you're loading for. DIM API data is stored per-platform. You should specify which components you want returned in the result. The current available components are:

• tags: Tags and notes on items, corresponding to DestinyItemComponent.itemInstanceId. These are returned as a list of ItemAnnotation objects.
• loadouts: Saved loadouts. These are a list of Loadout objects.
• hashtags: Tags and notes by DestinyInventoryItemDefinition hash. These are like tags but meant for uninstanced items like shaders and mods.
• triumphs: A simple list of DestinyRecordDefinition hashes for tracked triumphs. This allows for tracking more triumphs than allowed in the game.
• searches: Saved searches. Likely only useful to DIM, this powers DIM's search history.

Updating profile data

DIM API is meant to be usable "offline". A client can collect a sequence of change operations called updates, and then send them to be applied them in bulk, when the client is ready to make a request to the DIM API. By sending deltas in this way, the DIM API can effectively keep synchronized between different clients even if they're out of date, since you only send updates for what has changed.

As such, there's a single, central update endpoint: POST https://api.destinyitemmanager.com/profile

The body is a JSON object containing the destinyVersion (DIM supports D1 still!), the platformMembershipId, and a list of updates. Each update is one of the update types. So you can flush the local queue, and apply a series of changes consisting of different tag updates, loadouts, etc. — all interleaved together.

Loadout Shares

You can create shared loadouts, which are not part of a user's profile - these are globally accessible through dim.gg URLs. To create one, you can POST https://api.destinyitemmanager.com/loadout_share with a loadout document, and it will return the loadout's ID. If you GET https://api.destinyitemmanager.com/loadout_share?shareId=XXXXX you can retrieve a shared loadout based on its ID. It's best to read the DIM code related to shared loadouts to understand how to use them.

Cloud Architecture

The DIM API is a NodeJS server application that is deployed in the DigitalOcean cloud via Kubernetes.

• Requests to api.destinyitemmanager.com are routed to CloudFlare, a CDN which helps to cache data and provide traffic management. Most of the API calls are not cached, but the server application can return Cache-Control headers to customize what CloudFlare will cache, which can save both bandwidth and server costs.
• CloudFlare is configured to forward requests to a DigitalOcean Load Balancer. This is a TCP load balancer (L4) that simply forwards traffic to our Kubernetes Cluster's nodes. The LB is configured automatically by Kubernetes via a Service (ingress-nginx/ingress-nginx) with type LoadBalancer.
• Each of the Kubernetes Cluster's Nodes runs the Nginx Ingress Controller. It was set up following this guide but has been customized to run as a DaemonSet so that it is present on every node. It also has some custom settings. The ingress controller has an automatically renewing LetsEncrypt SSL certificate associated with it, which is handled by a custom Kubernetes resource.
• The Ingress Controller is configured to forward api.destinyitemmanager.com to our dim-api application. It also handles other subdomains like our Grafana metrics dashboard.
• The dim-api deployment runs our actual service. It's deployed as a small pod that is sized to fit multiple copies per node, which is nice because NodeJS is single-threaded so you want to run multiple copies. It has a Horizontal Pod Autoscaler (HPA) associated with it that adds and removes pods to maintain a target CPU usage. If the number of pods needed exceeds the number of nodes, new nodes will automatically be added, and they get removed again when load drops.
• We also run Graphite, StatsD, and Grafana as pods on the same clusters alongside our DIM API pods.
• The DIM API application talks to a hosted Postgres SQL database that is managed by DigitalOcean.
• The DIM API itself is using the Express framework to handle requests, route them to handler functions, and apply middlewares that observe the request lifecycle. See server.ts for details. Most requests are authenticated by a JWT bearer token.

Local Development

Local Postgres

The API server requires a Postgres database for storage. In production we use a hosted Postgres, but in development you need to run your own. You can install Postgres however you'd like, but one way is through Docker Desktop's local Kubernetes support:

1. Go to hub.docker.com, sign up for an account, and download Docker Desktop.
2. Open Docker preferences, go to the Kubernetes tab, and enable Kubernetes.
3. Run kubectl apply -f kubernetes/postgres-configmap.yaml - this contains our development password and configures Postgres.
4. Run kubectl apply -f kubernetes/postgres-deployment.yaml - this actually runs Postgres on port 31744. It will pull the latest Postgres image from Docker.

You can now connect to postgres on port 31744 with the username and password from the configmap file. Next, install the schema using a migration command.

DB Migrations

We use db-migrate to manage the schema in the database. Migrations allow you to make versioned changes to the schema and then apply them — first locally, and then to the production database.

• Run cd api && npx db-migrate up to update the database to the current version.
• Run cd api && npx db-migrate create settings_table to create a new migration file in the migrations folder, where you can use the db-migrate API to modify the database. Check in all migrations.
• If you have a configuration in api/database.json for the Prod database, you can update prod with cd api && npx db-migrate up -e prod. Make sure to do this before deploying!

Running the server

1. Run corepack enable to enable NodeJS to find pnpm. With Homebrew, you need to brew install corepack.
2. Run pnpm to install packages.
3. Run pnpm start to run a development server. It'll need a Postgres server running on port 31744 that has been migrated to the latest schema for the server to work.

If you want to develop on dim.gg instead of api.destinyitemmanager.com, edit .env and set VHOST=dim.gg.

Running tests

1. Run pnpm test. You'll need a Postgres server running on port 31744 that has been migrated to the latest schema for the tests to work.

Production Deployment and Administration

To access the production Postgres DB (either to run commands via the postgres shell or something like PSequel), you need to log into the DigitalOcean control panel, and copy the "Public Network" connection info for the dim-api-db database. You also need to add your home IP address to the "Trusted Sources" list.

You may also want to edit two files to include that information. These changes must never be checked in!

• api/database.json - this tells db-migrate how to connect to the database. You only need it if you're going to run a DB migration on prod.
• kubernetes/dim-api-configmap.yaml - this gets deployed to the production Kubernetes cluster and is loaded as environment variables into the app. You only need this if you want to update the production configmap, which shouldn't be often.

kubectl access

To be able to inspect or edit Kubernetes resources, or deploy, you will need to have your kubectl context set up. Log in to the DigitalOcean control panel, navigate to the Kubernetes cluster, and follow the instructions next to "Download Config File". It's easiest to install the doctl tool and ask it to create the kubectl context for you.

Deploying

Before deploying, run cd api && npx db-migrate up -e prod if you've created any new database migrations. If you forget to do this, the app will likely crash as it will try to do things with an outdated schema.

Deployment is done by running pnpm deploy. This will build the application into a Docker image, push that image to the global Docker container registry, and then apply an updated deployment config that points to the new image. You can run kubectl get pods to watch the pods cycle over to the new version, which takes a minute.

Handy Commands

• kubectl get pods - see all pods and their state
• kubectl describe pod <podname> - see details about a pod
• kubectl get nodes - see all nodes in the cluster
• kubectl describe node <nodename> - see details about a node, including which pods are on it
• kubectl top pods - see resource utilization per pod
• kubectl top nodes - see resource utilization per node

Metrics

You can visit our metrics dashboard at https://grafana.destinyitemmanager.com to see how the API is doing. Access is granted through GitHub to DIM core team members.