Fashion E-Commerce: Using Computer Vision to Find Clothing that Fits Like a Glove

It was October 2019, and my cousin's wedding was taking place in about a week. Like everyone else, I was faced with the dilemma — what to wear? The struggle was real.

So there I was, in the mall with my brother picking out shirts, because everyone knows you can’t go wrong with a great shirt right? I picked a few and was searching for a place to try them on before deciding which ones to buy. Finally found somewhere just right.

It was a fitting room.

I tried them on, and after checking myself out (and getting my brother’s nod of approval) I bought the ones that fit just right. The following week, I wore one of them to the wedding. I looked great. Just call me GQ.

Like many guys, I don’t particularly like shopping, but I tolerate the fitting room because it gives me the opportunity to see how everything looks before going out in public. Getting the right clothes the first time means not having to head back to the store for a return and round 2. I’m efficient like that.

It goes by different names — changing room, dressing room, trial room. You know, those small cubicles with the big mirrors and lighting that makes you look like an extra in The Walking Dead? The place where before you enter, the staff stops you and barks “Sorry, you can only bring in 4 pieces at a time”, and you’re like — “Can I please bring in all 7? I won’t take long!” — just so that you can play dress-up for the next 45 minutes?

Yeah, they’re kinda nice aren’t they?

But then 2020 happened — a year that most people will remember for a very specific reason. With social distancing protocols and strict sanitation rules, shopping for clothes just became a lot harder.

A Problem of Size

It’s not like 2020 forced us to spin our own cloth. It was simple enough to order clothes online, but it created a whole new set of issues.

Most of us have purchased something online. Electronics, household appliances, books or kitchenware. But amongst all, shopping for apparels is often the trickiest. It’s pretty straightforward to purchase a smartphone, a laptop or some other gadget — you only need to worry if the specs and the design meet your needs, and whether it’s got positive reviews. Ditto with books, handbags, or even jewellery.

But clothes? Reviews alone just won’t cut it.

You probably hate going for clothes shopping just as much as I do. It can be terribly tiresome. But you gotta admit that it’s at-least somewhat un-ambiguous. In a clothing store, picking something you like is often the hardest part. But once you do find something, things get simple. Grab your size, try it on, check yourself out.

With online shopping however, everything gets flipped! earching for just the right thing is easy — thanks to the ever-improving search features from all the mainstream e-commerce platforms. But getting the right fit? It’s not as easy as it looks.

It’s no wonder that the average return rates for online apparel shopping is a whopping 32%, and it burns a hole in the pocket for several e-commerce retailers. Here are some wrinkle-inducing stats:

• Almost 1 out every 3 apparels purchased online get returned.
• In 2019, about $260 Billion worth of retail goods were returned, out of which $30 Billion were from apparels.
• The cost of returns is estimated to be nearly $1.5 Billion. The cost of each return is between $5-$15.

What are the top 3 reasons for people returning apparels bought online?

Here are some stats that’ll create creases… on your forehead, that is! -

• Size misfits is the biggest factor, making 43% of total online shoppers citing this as the reason.
• Quality and color mismatch makes up for 29%.
• 22% of shoppers feel the need to simulate a trial room at home.

Despite these stats, online shopping isn’t going away. More and more users are shopping online post-COVID:

• 27% of customers switched from offline to online due to COVID-19 in USA.

• 12% of customers would prefer to continue purchasing online even after the COVID.

So needless to say, online shopping is here to stay. And that includes apparel shopping as well, which brings with it with the size-mismatch problem.

The size-mismatch problem contributes to $1 Billion returns annually and affects inventory planning, leading to heavy losses for e-retailers.

Size-mismatch is not a mere wardrobe malfunction, it is a huge-sized problem!

At Egen, we decided to iron this one out.

SmartTry

Before the e-commerce era began, how did someone know if a certain clothing fit them?

The solution was simple. If you were buying a new shirt, you’d simply compare it with another shirt that you already possess — one that was already the best fit for you. The best part is, you wouldn’t even need to try the shirt on in a fitting room, and it’d still be a great fit for you most of the time!

That’s what SmartTry does — it enables users to compare their existing best fit clothes to new ones they want to purchase online.

Simplicity, carried to an an extreme, becomes elegance.

How does SmartTry work?

Say you’re trying to buy a T-shirt online from Amazon, or from mango.com like the below screenshot -

You find the ultimate white T-shirt for the ultimate in effortless cool in your next Zoom meeting. But you’ve never bought a T-shirt from Mango before, so how will you know how it fits. (We’ll spare you the rant on inconsistent sizing. You know it. We all know it.)

What do you do? You don’t wanna go through the whole ordeal of having to return it if it’s the wrong fit!

Precisely the pain point for you and many other people. And here’s where SmartTry’s user experience is guaranteed to knock your socks off!

Features

Clicking on the green SmartTry button opens the SmartTry plugin in a popup on your screen:

• The SmartTry plugin shows you the T-shirt or the outfit that you want to buy, along with your previous best fits for you to compare.
• Clicking on the Add a new apparel to Compare icon lets you upload a picture of an outfit that you currently own, into the SmartTry platform . You’ll use this to upload a photo of the T-Shirt that is your current best-fit.
• On uploading an apparel, it gets added to the Recommended Items section and to your Collection, with a button underneath that says Compare. Clicking on the Collection tab on the left will also show you all your previous uploads neatly organized within categories:

Finally, when you click on Compare, SmartTry runs its magic in the background and shows you how good of a fit your selection is!

• The blue T-shirt border represents your current best-fit that you used for the comparison, while the black border is the T-shirt that you want to buy.
• On the right, you’ll be able to see precise measurements for the T-shirt you selected and your current best-fit. This helps you easily compare the differences between the two. (Let’s face it. Every site has a size chart. No one is whipping out a measuring tape before clicking add to cart. Problem solved.)
• In this example, the Small size seems a little too small — a 67% match. But not to worry! Since you like the T-shirt so much, you can switch between different sizes:

Medium size is a 94% fit.

You can now buy the outfit with the right size, knowing fully well that it’ll fit you to a T!

Life ain’t perfect, but your outfit damn well can be!

Unmasking the Magic

If you’re a programmer, I’m sure you’re wondering by now —

How does SmartTry really work behind the scenes, and how do you build it?

Well roll up your sleeves, because we put together a tutorial for you to follow along!

What you will learn

Whatever the mode of user-interaction may be, the power of SmartTry comes from Computer Vision techniques, that allows it to compare two different outfits and analyze the difference in their measurements. Over the next several minutes, I’ll walk you through how to build this algorithm.

• You’ll use OpenCV, one of the most popular and widely used computer vision and machine learning libraries.
• For writing code, you’ll use a Jupyter-Notebook, an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text.

A word to the wise

Some knowledge of Python is essential in-order to understand this tutorial. If you haven’t worked on Python before, I highly recommend this free course on Introduction to Python, before proceeding with the rest of this article.

Why Python? Because it is fairly easy to learn, and the Python ecosystem has a lot of tools and libraries for computer-vision, machine learning, and more. Once you’ve built the algorithm, Python also has several web-frameworks that allow you to build a service around this feature.

Tutorial

Go to this github link, and follow the instructions to download and install the the project dependencies. Then, run jupyter-lab in your project root and create a new Python3.x notebook file. Next,

Head over to the Jupyter tutorial and follow the steps outlined, to build your very own program that calculates accurate measurements for a given T-Shirt!

Alternatively

Head over to the below Binder link and execute the code right within your browser without the hassles of installation.

The smarttry.ipynb notebook has the complete source code with instructions for you to read. You could choose to run this directly, or write code on your own in a new notebook file as you follow the tutorial.

Slides

If you prefer taking just a quick peek into the logic, here’s a short slide-deck:

So go ahead, put on your thinking cap and have fun!

The first version of SmartTry that we developed at Egen achieved an accuracy of 91.7% for clothing measurements! Hopefully, you’ll be able to use this as a performance benchmark when following the tutorial and writing the program. I’m curious to know what your accuracy score looks like!

Next Steps

Once you’ve developed your program, you can integrate this into a web-service and expose it via an API. Here’s the architecture implementation that we designed at Egen:

It doesn’t stop here! The future scope for such a feature is immense:

• On websites, SmartTry can be easily integrated into any existing e-commerce platform — as a popup, or an installable plugin, or a platform integration, or even as a browser extension.
• Integrate SmartTry into mobile applications, which will allow you to upload photos of your outfit more easily.
• You could maintain a common SmartTry profile that’ll exist across multiple e-commerce websites.
• A history page to quickly review past comparisons.

and more!

Potential Revenue Models

Being an intermediary between different parties that interact with a product is a model that a lot of businesses are exploring in recent times. SmartTry being designed as an easy-to-add plugin with a high accuracy makes it a really good contender for adoption in new businesses and existing e-commerce platforms. Here are some revenue models that can be quite viable (although they’re by no means an exhaustive list):

• At-cost SaaS model for online shopping. The pricing could be based on the number of API calls made to the SmartTry platform, for example — $0.001 per 1 API call.
• Size and shopping data of users and their different demographics
• Sending traffic from our platform to “preferred” e-commerce websites
• In-store enterprise yearly licensing model
• A freemium model with basic features accessible for free, and upgrades or more advanced features available at a monthly subscription.

Summary

What is right is often forgotten by what is convenient. And evidently so, it’s true in the world of fashion e-commerce — with the potential of nearly $1 Billion USD in cost savings and returns, if done the right way! Reduced apparel returns would also translate into reduced carbon emissions, logistics and waste — thus helping the planet be clothed in green!

Credits

All credits for developing the idea and concept goes to the SmartTry team at Egen, consisting of Arif, Gagan, Anmol and Aatman. 🙌🏼

Originally published in Egen Engineering.