Migrating from Shopify's Product REST API to GraphQL

Our Ablestar Bulk Product Editor app processes millions of products each day and is a big user of Shopify’s product APIs. When we created the app in 2016 Shopify only had a REST API for product data and so that is what we used for the app. Over time new features, like standardized product categories, were released and we support those through the GraphQL API. Still, as of the start of 2024 the majority of our app continued to use the REST APIs.

Earlier this year Shopify announced that they would be deprecating the product REST APIs in favor of the GraphQL API which meant we needed to do a major update to our app. While there was at least a year’s advance notice, we decided to do the migration as soon as possible so we could keep supporting the latest updates from Shopify (most notably increasing the variant limit to 2,000). It took us about two months to code, test, and deploy these new changes, but as of early June, all stores using our app are on the GraphQL API.

In this post I’d like to share our process for migrating to GraphQL, some things to watch out for when making the change, and some final thoughts on the pros and cons of the new APIs.

The Migration Process

As much as possible we wanted the migration to be incremental, where we could quickly roll things back if something went wrong. This means we would need to:

• Have two versions of functions that interact with the Shopify product APIs. For example, to download products we have two background tasks named download_products_rest() and download_products_graphql().
• Keep our database schema (almost) the same and ensure both APIs wrote the same data to our database. Some fields would need to be converted. For example in the REST API a product’s status is lowercase (active) while in GraphQL the value is uppercase (ACTIVE).
• Be able to turn the GraphQL API on or off for individual stores.

To ensure that both versions of the API would function the same we created tests that would perform API calls against both APIs and verify that the data in our database was the exact same afterwards.

With these three things in place we were ready to start rolling out the changes to stores. The first stores we enabled the GraphQL API for were development stores that we created that have the production version of the app installed. After activating GraphQL APIs for the store we went through and tested all the different edit types (updating prices, setting metafields, reordering variants, etc.) to make sure it was working.

Once we were confident that things were in good shape we started rolling out the GraphQL to live stores using our app. Initially we just enabled GraphQL for 50% of new installs. This gave us a way to gradually observe how the new APIs work under actual usage. After a few days we started enabling the GraphQL API for all new installs.

Finally, we started working backward and enabling GraphQL for stores that had already installed the app. At first we started in batches of 100s and then 1,000s until all stores were using the new GraphQL APIs.

During this whole process we were keeping a close eye on our logs and Sentry to identify any issues. Fortunately, we didn’t have any major problems but we did see things at scale that we didn’t notice when testing on just a few stores.

In the next section we’ll talk about some of these issues and other changes that we came across while migrating.

Differences Between GraphQL and REST

While both the GraphQL and REST APIs operated on the same product data, there are differences in how they present the data that we had to account for. Most of these things we found out either by reading the documentation or, occasionally, trial and error.

Naming and Weights

One of the first things you notice when looking at the GraphQL documentation is that the naming conventions for fields and constants are different. For example:

• compare_at_price → compareAtPrice
• active → ACTIVE

None of these are huge problems but they were something we needed to be careful of because there was a period of time when we were using the REST and GraphQL APIs at the same time.

A slightly bigger issue was the grams field on the variant object. In the REST API you have three fields for a variant’s weight: weight, weight_unit, and grams. The GraphQL API doesn’t include the grams field. Since our app allows you to edit the weight in grams, regardless of the displayed weight unit, we had to write conversion functions to keep track of the grams.

Manual and Automated Collections

In the REST API there were originally two separate endpoints to access manual and automated collections (custom_collections and smart_collections respectively). Back in 2016 when we started, we treated these as separate objects in separate tables.

The GraphQL API has a single Collection resource with an optional ruleSet field for automated collections. The one big database change that we made was to combine both types of collections into a single table. We should have probably done this from the start but the way the GraphQL API exposes this data was a good motivator for the change.

Creating Products

You can also import products from a custom spreadsheet with Ablestar Bulk Product Editor so we needed to support product creation in GraphQL as well. With the REST API you could generally create products, images, and variants in a single API call (one exception being if there were too many images the call would time out).

With the GraphQL API we create each of these separately:

1. We use the productCreate mutation to create the core product.
2. Then we use the productAppendImages to add images in chunks of 10 to prevent a timeout.
3. Then we add the variants to the product with the productVariantsBulkCreate mutation.

This works well but does require three API calls to do what we could have done with one before. This doesn’t slow things down much though because the majority of the time is still taken by Shopify needing to retrieve and process the product images.

Error Handling

One thing I like a lot about GraphQL is the standardized approach to error handling. There are two types of errors you can receive from the GraphQL API, one for general errors (like exceeding the rate limit) and one where the input to a specific query is invalid (for example, trying to set a price to the string "zero").

We’ll talk more about error handling in the tips and pitfalls section below.

Publishing Products to the Online Store or Point of Sale

In the REST API there are the published_at and published_scope fields which determine if a product is published to the web or POS. In GraphQL the recommended way to do this is through the Publication object with the publishablePublish() mutation. This is a more standardized way to toggle a product’s visibility but there were two things we had to be careful of:

1. You need to request two additional access scopes for your app: read_publications, write_publications.
2. If a store doesn’t have the Shopify POS sales channel installed you can’t edit the POS visibility anymore.

If you’re just looking to update the visibility on the Online Store you can still set the publishedAt directly; however the field is deprecated so I’m not sure how much longer it will be available.

Adding Product Images to the Start of the List

When you add images to a product in Ablestar Bulk Product Editor you have the choice to add the image to the start or end of the list. In the REST API this was simple; we queried a list of existing images from our database and added the new image URLs to the front or end and then updated them with a single API call.

With the GraphQL API images are always appended to the end of the list, just like in the Shopify admin. This means for us to support adding images to the start of the list we need to:

1. Upload the images to the end of the list.
2. Use the productReorderMedia mutation to move the new images to the front.

Tips and Pitfalls

While most of the differences between GraphQL and REST were listed in the documentation there were also some things we only discovered once we started using the GraphQL APIs at scale. We’ve included these below and some tips we learned along the way.

Product types/categories

With Ablestar Bulk Product Editor you can edit both the product’s type and category. We were using the deprecated customProductType field to update the product type field.

This worked in almost all cases except in one edit where a user was trying to update the product type and category in a single edit. When they tried that they got this message:

The product_category field can't be used together with custom_product_type.
Only product_type can be used with product_category

The error message was helpful and we were able to quickly fix it by using the productType field instead of customProductType.

Lots of Locations

When you first install Ablestar Bulk Product Editor the app will download all the inventory levels from Shopify and then keep them up to date with webhooks. We download the inventory levels by looping through inventory items and finding the available quantity at each location.

Several weeks after we started rolling out the new code we received the following error for one store while downloading the inventory levels:

{"errors":"query param length is too long"}

We’d never seen this error before and it seemed odd. Once we investigated, we realized that the store had over 400 locations and the error made a lot more sense. We fixed the issue by looking at locations in chunks of 25 (which still allows us to process variants for most stores in a single API call).

Retrying API calls

There are a number of situations where we’ve found we need to retry GraphQL API requests:

• We receive a 500 response from Shopify
• The API call hits the rate limit
• We receive one of the following errors:
  • Internal error. Looks like something went wrong on our end
  • This product is currently being modified. Please try again later

In each case we’ll wait a certain amount of time before retrying the request. The exact time depends on the type of error and how many times we’ve retried with a little randomness added.

One other issue we encountered is that, very rarely, we’ll make a request to the GraphQL API and the HTTP connection will stay open but we won’t ever receive a reply. We fixed this by adding a 5-minute timeout to all requests and retrying those requests.

Optimizing Query Costs with Paging

One of differences with GraphQL is that not all API requests count the same against the rate limits. This means that you can optimize your queries your queries to run more efficiently by selecting specific fields and modifying the page size for pagination.

If you’re paginating through a lot of records using something like products(first: $pageSize, after: $cursor) I noticed that the query cost doesn’t increase linearly with the $pageSize.

For example, in one query a $pageSize of 95, 100 and 148 all had the same query cost. However, if I increased the page size to 149 the query cost went up.

I don’t know exactly how it works behind the scenes but it’s worth experimenting with different page sizes to see how you can get the maximum results back for a specific cost.

Query Costs for Related Fields

One thing I don’t like about the query costs is that just requesting the ID of a related object increases the cost the same as if you retrieved the whole object. For example, if you want to get a variant’s inventory item ID, you would do something like:

{
    title
    inventoryItem {
        id
    }
}

Unfortunately this incurs the same cost as including other fields from the inventory item object. In SQL terms it seems like Shopify is doing something like:

SELECT title, inventory_item.id FROM product_variant
	LEFT JOIN inventory_item ON inventory_item_id = inventory_item.id

Instead of this query which would be more light weight:

SELECT title, inventory_item_id FROM product_variant

I have no idea how it actually works behind the scenes, but it would be nice to be able to retrieve just the inventory item ID without incuring the extra cost of looking up the whole inventory item.

Conclusion

This was the first project we’ve gone all-in on GraphQL and the switch to GraphQL wasn’t too painful. You’re still dealing with the same product data (albeit with slight presentation differences), and there’s no place I felt that we lost major functionality while switching. Our users were, by and large, unaware of the switch which is ideal.

The Good

Things are fairly well-documented (though it would be nice to have a quick way to submit fixes/suggestions to the docs) and the error handling feels more standardized.

I also like how some of the GraphQL calls act more like functions that can do multiple things at once. Two examples that come to mind are reordering options and creating a web redirect if you edit a product’s handle. Both of those are possible through the REST API but you need to do each step individually. In the web redirect example you just need to pass redirectNewHandle:true to the productUpdate mutation and it takes care of it for you. Having these common multi-step actions wrapped by a single API call makes things easier to develop against.

The Less-Good

While the error handling is more consistent in the GraphQL APIs we get a number of Internal error. Looks like something went wrong on our end. errors which don’t provide much context.

The oddest time we saw this was when we were trying to retrieve the inventory item ID for one variant with inventoryItem { id }. This worked without problem for literally millions of other variants but failed for one specific variant. I imagine that because of their dynamic nature the GraphQL APIs are more complex to implement than the equivalent REST APIs and that leads to a larger surface area for bugs.

The other problem is that even though we’ve switched to the GraphQL API for product data, we still need to support the product data in the REST format because that’s all the webhooks support. How we handle webhooks is a topic for another day, but we process hundreds of webhook messages a second to keep our local copy of the product data up-to-date with Shopify. Having to reconcile the two different formats of product data is doable but not ideal.

Final thoughts

The migration from REST to GraphQL went better than I thought. It took some work and lots of testing but it went smoothly.

Of course, the biggest advantage is that Ablestar Bulk Product Editor can continue to support newer features like the product taxonomies and the 2,000 variant limit.

I’m hoping some of the things here can help others as they make the big switch. If you want to reach out I’m on Twitter/X at @1danielbeck.