This is the last post ending a series of articles that have traced the prices of some of the top gifts, gadgets, and gizmos from Black Friday 2015 through to January 2016.
Check out Part 1 comparing Black Friday and Cyber Monday, Part 2 comparing Black Friday, Cyber Monday, and Green Monday, and Part 3 comparing Christmas Eve and New Year’s Eve.
We’re all looking to be savvy shoppers who can buy nice things on a budget. So get ready to block out your calendar and wait in line at the right store.
It’s been a little over two months since we first started tracking the prices of popular products, starting on Black Friday and covering all the major discount days during the winter of 2015. We’ve finally reached the culmination of this project and are ready to declare the best days to shop for presents and deals this year.
Plus, we're calling out those guilty of price inflation, so you're going to want to keep reading.
For this specific post, we compared all prices from Black Friday, Cyber Monday, Green Monday, Christmas Eve, and New Year’s Eve to those taken from January 13, 2016. We chose this date as our baseline because the traditional sales period is over, but it is still in a relevant time frame.
While we covered the specifics of our methodology in the first post, we would like to share several tips and tricks that we used to get past some of the challenges of scraping so many different websites and products.
Some websites are really friendly toward web crawlers, providing machine-readable data for the content that you see in your browser. During this project, we got lucky. One of the websites we scraped for this series provided Microdata for the relevant information we needed about its products:
<div itemprop="offers" itemscope="itemscope" itemtype="http://schema.org/Offer"> <span itemprop="itemOffered" content="1234567"></span> <span itemprop="price">$149.99</span> <meta itemprop="priceCurrency" content="USD"> </div>
This was a perfect opportunity to use Extruct, a Python library we developed to extract Microdata from HTML content. Take a look at how Extruct works in this Scrapy shell session:
$ scrapy shell http://somewebstore.com/products/1324567 >>> from extruct.w3cmicrodata import MicrodataExtractor >>> extractor = MicrodataExtractor() >>> print extractor.extract(response.body_as_unicode(), response.url) { "items": [ { "type": "http://schema.org/Product", "properties": { "image": "http://website.com/image.png", "offers": { "type": "http://schema.org/Offer", "properties": {"price": "$149.99", "itemOffered": "1324567", "priceCurrency": "USD"} }, "productID": "1324567", "description": "Make every beat count with Charge HR.", "name": "Fitbit Charge HR Wristband" } } ] } >>> items = extractor.extract(response.body_as_unicode(), response.url)['items'] >>> items[0]['properties']['name'] u'Fitbit Charge HR Wristband' >>> items[0]['properties']['offers']['properties']['price'] u'$149.99'
As you can see, it’s a piece of cake to scrape websites that use semantic markup. With Extruct there’s no need to write specific XPath or CSS selectors for the website. So, if the website you are trying to scrape has some itemprop properties inside the HTML, it’s your lucky day. The site is using Microdata and you can likely extract all the data you need with Extruct.
If this article doesn’t hammer home the completely arbitrary nature of prices, I’m not sure what will. Between price inflation and false sales, there were actually very few examples of products being fully discounted across the board on any particular day. Instead, getting a good deal was much more contingent on visiting the right store at the right time.
The Apple TV was one of the most interesting examples of this:
On Black Friday 2015, only Target and Walmart offered sales for this item (both $51.75). On Cyber Monday, Target shot up the price to $69, about $5 more than Best Buy and Sam’s Club.
Target continued this overpriced trend until Christmas Eve before discounting to $59. They then returned the price to the inflated $69 for New Year’s Eve and kept this price constant through to January 13, 2016.
Walmart, on the other hand, brought the price up to $63 on Cyber Monday before holding steady at $64.99, the same price as Best Buy, for the rest of the study. Target was clearly the best bet for a cheaper price as long as you purchased at the right time (Black Friday and Christmas Eve), but they are clearly guilty of inflating prices to capitalize on shopper anxiety.
Out of all of the products, there were only 3 that were continuously discounted throughout the winter season compared to January 13, 2016. These were:
When compared to every other day, Black Friday was by far the cheapest day to purchase these 7 products:
The Jawbone UP 3 warrants closer attention because while Black Friday was the cheapest day to purchase this at Best Buy, the second cheapest time to buy it was January 13, 2016. For the rest of the winter season, Best Buy’s prices for the Jawbone UP 3 were higher than both Amazon and Target (see the Wearables graphs below).
When compared to every other day, Black Friday was the most expensive day to purchase these products:
January 13, 2016 was the cheapest day for these products:
The worst culprits for inflated pricing were products in the Toys (4 products) and Tablets (5 products) categories (not counting the same product that was inflated in different stores). Amazon was the worst place to shop since it had 8 of the inflated items in these categories.
These were the products that were completely unaffected throughout the duration of this project:
Let's take a closer look at the guilty parties who snuck in inflated prices among the sales.
Amazon was by far and away the worst offender when it came to price inflation. On every single one of the 5 days, there were about 13 to 18 products that were more expensive than the prices listed on January 13, 2016. This was far in excess of the 8 to 11 deals offered.
I’m going to go ahead and place the winners:
Now, there is no guarantee that these same patterns will hold this year. But based on 2015, I highly recommend that you plan around the Toys situation. Every single toy suffered from price inflation with every single store inflating the price of at least one toy. If you are in a bind, though, then I’d wait for Green Monday and avoid Amazon.
If a Fitbit goes on sale on Black Friday, take the deal. It’s not likely to get any cheaper. I’m still congratulating myself on getting my Fitbit Charge HR for $30 cheaper on Black Friday!
Do your homework ahead of time and don’t worry if you hesitate and miss a sale, there’s usually another retailer and another day.
For more articles like this, follow us on Twitter, Facebook, and LinkedIn, and don’t forget to subscribe to our RSS feed.