Shopify's AI sort erases the thing you'd measure it against
The tool that checks it has been in your admin since January. Open that first.
If you want a growth partner who is invested in helping you build a profitable business, someone who is going to push you to make changes that focus on profitable growth 👉 Book a call - let’s chat
Shopify's June 17 Editions release dropped 150+ updates, and the pairing that actually matters got buried in the feature list.
One of those updates is native AI merchandising.
Flip a toggle and Shopify orders your collections by predicted probability of converting and picks your cross-sells for you. No merchandiser. No third-party app.
That sort chases one number, the odds a given product converts, which is useful right up until you remember conversion probability is not the number your P&L runs on.
An algorithm optimizing for it can bury a high-margin product that sells slowly but anchors your positioning, and it can surface whatever is trending this week over something you have spent two years building a reputation around.
The tool is probably good. Probably is not a merchandising strategy.
Before you touch that toggle, write down your current top three products per key collection ranked by contribution margin. That list is the exact thing the AI cannot see about your business.
Quick one, and it's genuinely useful. Everyone keeps telling operators to "use AI" in email and SMS, and nobody says where to start or what follows what. .
That missing running order is why so many teams test and abandon.
- Ship one clunky campaign- Flip it back off- The tool was fine but the sequence was random
Postscript put out a free playbook that's basically the running order itself:
- 8 steps- The way they're meant to go- Templates under each one- Written by someone who actually runs the channel
Here's one to steal today, get your segment clean before you let AI automate a send, or you're just reaching the wrong people with nicer copy.
All vibes and no plan? Here's the plan.
The Tool That Grades It Is Already In Your Admin
Here is the part the coverage skipped.
Shopify shipped the instrument to check this months ago.
It is called Rollouts, it lives under Markets > Rollouts, and it lets you schedule changes and split-test them straight from the admin with no app bolted on.
Themes have had it since the Winter '26 Edition in January. The June wave extended it to checkout and customer-account configurations.
The honest capability read for an account your size: the real traffic-split experiment needs the Grow plan or higher, while Basic gives you scheduling and rollback without the split.
Checkout testing is effectively a Plus feature. It cannot test price or discount logic, so $39 against $45 is a question the tool will not answer.
For each version you put in the split, what the tool calls an arm, it reports conversion rate and revenue and then stops.
There is no significance math, so nothing tells you whether the gap between two arms is a real difference or just a slow Tuesday.
It never declares a winner, and it carries no revenue-per-visitor number of its own. Rollouts splits the traffic and hands you raw results.
Reading them stays your job, which is fine, because deciding was always the job.
Open Markets > Rollouts this week and confirm what your plan tier actually unlocks before you assume you have a testing stack.
Run The App-Bill Audit While The Category Is Fresh
Every time Shopify folds a feature category into the platform, the apps charging a monthly fee to do that exact thing get harder to justify.
Testing and merchandising tools that ran $300 to $1,500 a month as standalone apps are now partly native.
Partly is the operative word. Price testing and segment-level funnel analysis are still app territory, and so are the significance engines, so the move here is not a mass cancellation.
Map your live test backlog and merchandising rules against what native now covers, then flag the overlap.
Pull your app bill this week, line every testing and merchandising subscription up against the native feature set, and mark the duplicates for cancellation at renewal.
The Adoption Rule That Outlasts This Edition
The stores that get the most out of any Editions cycle adopt the least, and they do it slowly.
A real share of what gets presented as new in June was introduced in an earlier cycle.
This Edition's own testing tool proves it, live since January while half the coverage called it a June launch.
Measurement layer goes in first. Automation waits until you can grade it. That ordering survives whatever Shopify automates next.
Adopt exactly one thing from this release this month, and make it the measurement layer.
Run the Sort as a Test
Run the AI sort as an experiment rather than a setting.
> Pick your highest-traffic collection rather than your most precious one, and split traffic against your current sort order.> Judge it on revenue per visitor that you compute yourself, each arm's revenue divided by its share of traffic, because Rollouts reports revenue and conversion rate but will not hand you RPV and will not call a winner.> Let it run two weeks minimum, longer if the collection sees under 1,000 daily visitors.> Then make the call: keep the AI sort where it earned more per visitor, hold manual control where margin logic has to win, and either way you decided on numbers instead of a toggle.
The Decision to Run This Week
1. Write down your top three products by contribution margin per key collection before enabling any AI sort.
- - -
2. Open Markets > Rollouts and confirm what your plan tier unlocks. Grow or higher gives you the true split.
- - -
3. Pull your app bill and line up every testing and merchandising subscription against the native feature set. Flag overlap for renewal.
- - -
4. Adopt one thing from this Edition this month, and make it the measurement layer.
- - -
5. Split-test the AI sort on your highest-traffic collection. Compute revenue per visitor by hand. Run it two weeks minimum.
Final Thoughts
Shopify handed you an automation that quietly rewrites your store and, in the same season, a tool that tells you whether the rewrite helped.
Which one you open first decides whether you ever find out.
Flip the automation first and you are trusting a black box with your margin.
Open the measurement layer first and the automation has to earn its place against your current numbers.
The toggle will still be there next week. Grade it before you trust it.
Nord Media helps DTC brands between $1M – $100M+ in revenue scale by driving profitable growth utilizing Meta, Google and Creative Strategy.
Book a call and let's see how we can help you grow 👉 book a call with me.
-
-
-
