Funny item co-occurrences in 3 million Instacart orders
The other day I was idly wondering what are the strangest combinations of items people buy at grocery stores. The kind of shopping cart that makes the cashier snicker and later tell his friends, "Dude, can you believe this guy came in and only bought condoms and apples?"<br>So I fired up Claude and started looking for any receipt data I could find.<br>For the final results, scroll to the bottom. Otherwise, read on to follow the journey.
Grocery stores keep this kind of data very close to the chest. There are consumer apps that collect receipt data (like ReceiptHog and Fetch) but they presumably just sell it to hedge funds or something. Years ago, however, Instacart open-sourced data from 3 million orders as part of a machine learning competition to optimize their recommendation algorithm. The data is still available on Kaggle and it's very rich. What we want is the exact opposite of a recommendation algorithm but this data should work fine.<br>The Instacart data set includes the following:<br>3,214,874 orders<br>~10 products per order on average<br>49,688 unique products<br>134 unique "aisles" (product categories)<br>So all we have to do is look at every cart and see which combinations of items are least likely to appear, right? Let's try that. For the sake of testing a few groupings, I included pairs, triples, and quads.<br>Product co-occurrence count: pairs
count<br>product 1<br>product 2
Banana<br>Extra Fancy Unsalted Mixed Nuts
Organic Hass Avocado<br>Baby Cucumbers
Organic Hass Avocado<br>Trail Mix
Bag of Organic Bananas<br>Bananas
Limes<br>Trail Mix
Organic Garnet Sweet Potato (Yam)<br>Clementines
Organic Avocado<br>Zero Calorie Cola
Organic Fuji Apple<br>Baby Cucumbers
Note: maximum pair co-occurrence count is 62,341.<br>Product co-occurrence count: triples
count<br>product 1<br>product 2<br>product 3
Banana<br>Bag of Organic Bananas<br>Soda
Banana<br>Bag of Organic Bananas<br>Clementines
Banana<br>Bag of Organic Bananas<br>Sparkling Mineral Water
Banana<br>Bag of Organic Bananas<br>Flat Parsley, Bunch
Banana<br>Bag of Organic Bananas<br>Sparkling Water
Banana<br>Bag of Organic Bananas<br>Fresh CA Grown Eggs
Banana<br>Bag of Organic Bananas<br>Red Seedless Grapes
Note: maximum triple co-occurrence count is 15,066.<br>Product co-occurrence count: quads
count<br>product 1<br>product 2<br>product 3<br>product 4
Banana<br>Bag of Organic Bananas<br>Organic Baby Spinach<br>Half & Half
Banana<br>Bag of Organic Bananas<br>Organic Whole Milk<br>Organic Zucchini
Banana<br>Bag of Organic Bananas<br>Organic Avocado<br>Organic Half & Half
Banana<br>Bag of Organic Bananas<br>Strawberries<br>Organic Lemon
Banana<br>Bag of Organic Bananas<br>Organic Hass Avocado<br>Spring Water
Banana<br>Bag of Organic Bananas<br>Organic Yellow Onion<br>Organic Fuji Apple
Note: maximum quad co-occurrence count is 3,828.<br>Hmm. These aren't very interesting. Maybe that's because there are a ton of combinations that only occur once, and we're only looking at the top 5-10, so we need a better way to sort them.<br>There are roughly 1.2 billion possible unique pairs of products (unordered pairs from a set of 50,000). About 97% of them never occur in our data , and of the ones that do, around 22 million pairs occur exactly once. When almost every pair is tied at zero or one, "least common" becomes meaningless.<br>So how should we sort? Claude had a good idea to rank each combination by "lift": how frequently it actually appears divided by how frequently we'd expect it to appear. If you have two very common items like apples and oranges, you'd expect them to co-occur a lot. If they don't, that's notable, and the pair should rank higher. Now let's try that.<br>Product co-occurrence by lift: pairs
lift<br>product 1<br>product 2
0.0007<br>Banana<br>Extra Fancy Unsalted Mixed Nuts
0.0010<br>Organic Hass Avocado<br>Baby Cucumbers
0.0012<br>Organic Hass Avocado<br>Trail Mix
0.0017<br>Bag of Organic Bananas<br>Bananas
0.0019<br>Limes<br>Trail Mix
0.0020<br>Organic Garnet Sweet Potato (Yam)<br>Clementines
0.0021<br>Organic Avocado<br>Zero Calorie Cola
0.0025<br>Organic Fuji Apple<br>Baby Cucumbers
Product co-occurrence by lift: triples
lift<br>product 1<br>product 2<br>product 3
0.0016<br>Banana<br>Bag of Organic Bananas<br>Soda
0.0019<br>Banana<br>Bag of Organic Bananas<br>Clementines
0.0032<br>Banana<br>Bag of Organic Bananas<br>Sparkling Mineral Water
0.0043<br>Banana<br>Bag of Organic Bananas<br>Flat Parsley, Bunch
0.0047<br>Banana<br>Bag of Organic Bananas<br>Sparkling Water
0.0056<br>Banana<br>Bag of Organic Bananas<br>Fresh CA Grown Eggs
Product co-occurrence by lift: quads
lift<br>product 1<br>product 2<br>product 3<br>product 4
0.0111<br>Banana<br>Bag of Organic Bananas<br>Organic Baby Spinach<br>Half & Half
0.0128<br>Banana<br>Bag of Organic Bananas<br>Organic Whole Milk<br>Organic Zucchini
0.0137<br>Banana<br>Bag of Organic Bananas<br>Organic Avocado<br>Organic Half & Half
0.0148<br>Banana<br>Bag of Organic Bananas<br>Strawberries<br>Organic Lemon
0.0155<br>Banana<br>Bag of Organic Bananas<br>Organic Hass Avocado<br>Spring Water
I think I see a pattern... Of course it's unlikely that someone would buy both organic and non-organic bananas which points to the bigger issue that 50,000 products is too...