Description
About the role
You'll review real merchant conversations where Shopify’s Sidekick answered an analytics or segmentation question and the merchant was unhappy — pinpoint the failing query or filter, determine what it should have been, and categorize the error against a detailed rubric.
What you'll do
-
Read multi-turn merchant ↔ agent conversations, including tool-call cards and the data they return
-
Identify the query or segment filter the assistant generated, and determine the correct version
-
Categorize why the response fell short (wrong metric/dimension/filter, unsupported report, etc.)
-
Write concise, objective explanations of what went wrong
-
Commitment: ~5–10 hours/week
Qualifications
Required:
- Strong general SQL — comfortably read, write, and critique queries:
WHEREvsHAVING,GROUP BY,ORDER BY/LIMIT, joins, and aggregation. You should be able to look at a query and say what it returns and what's wrong with it.
Strongly preferred:
-
Hands-on Shopify merchant experience — you've run or operated a Shopify store and know how merchants actually use analytics and segmentation.
-
First-hand experience using Sidekick — you've used Shopify's Sidekick AI assistant yourself.
Preferred (nice to have — we'll help you ramp on the specifics):
-
Familiarity with Shopify analytics / ShopifyQL and the Shopify data model — the schemas (
sales,payments,sessions,fulfillments,customers), metric nuances (e.g.net_items_soldincludes returns vsquantity_ordered; revenue vs unit counts for "best-selling"), and the segmentation model (products_purchased MATCHES, operators,AND/ORconnectors). -
General e-commerce / analytics ops experience.
Interested in this position?
Apply directly on the company's website