Using Python To Buy A Canyon Aeroad

Cycling has always been something I have been passionate about. With the COVID-19 induced lockdowns I finally succumbed to purchasing an indoor trainer (Garmin TACX Neo 2T) and used it as a way to hit my distance targets each week.

However, as the months went on, I found I was less and less willing to convert my sole road bike from trainer to road trim. Factoring in the cleaning of my running gear to not dirty my pristine cluster on my trainer, the true time of a change could be up to an hour.

With that in mind, I decided that the single road bike model just wasn’t working for me. I wanted something special, but I wasn’t prepared to pay top dollars for an S-Works. After much research, my heart was set on the new Canyon Aeroad CF SLX 8 Disc eTap, top tier, winner of Tour De France stages.

The only problem with this bike is that it is a unicorn. Canyon have been doing the direct-to-consumer model for years with rave reviews. You would see the pro-riders on Canyon’s Instagram on this model, but it had been in a constant state of ‘Coming Soon’ on the Canyon website. A good chunk of the comments on the Aeroad pictures from Canyon are people asking if it will ever be in stock.

It was disheartening and I actually began to wonder if I this bike would ever be available. I contemplated other brands and models.

In this post I will share with you the process of how I obtained my Canyon Aeroad CF SLX:

• Chatting with Canyon Live Support
• Analysing Canyon’s website for breadcrumbs
• Let’s Get Serious – Beautiful Soup
• Dealing With Noise – Website Updates
• Using a humble Raspberry Pi
• Notification Options
• Ordering – Time is of the essence

Chatting With Canyon Live Support
Chatting with Canyon live support was a mixed bag of information. Different Canyon staff members would give answers ranging from no information, keep an eye out on the website and fill out the ‘notify me’ button, through to rough days and weeks. Nothing I could reliably go on.

Canyon – Notify Me
For each out-of-stock model there is a ‘Notify Me’ button. Clicking this allows you to put in your email address and in theory, as per the blurb below, you should be notified when the given model comes back in stock. To test out the reliability of this function, I entered my details on 5 or so other bikes on their website and in 6 months of looking for Canyon’s bikes, I have yet to receive an email notification. According to live chat, they only send emails to a subset of people who clicked this button in a FIFO (First-In-First-Out) model.

Analysing Canyon’s website for breadcrumbs
So here is where things got semi-serious. It is amazing what websites leave in their rendered HTML. In the past I have seen links to apply for jobs, through to some witty comments, but whilst I assumed Canyon used some sort of CMS (Content Management System, custom or COTS) what I found in the source code made me think that I didn’t need to author any code.

{"id":"50010388","description":null,"displayValue":"S","value":"S","selected":false,"selectable":true,"url":"https://www.canyon.com/on/demandware.store/Sites-RoW-Site/en_AU/Product-Variation?dwvar_2771_pv_rahmenfarbe=BU%2FBU&dwvar_2771_pv_rahmengroesse=S&pid=2771&quantity=1","hasComingSoon":true,"hasAllComingSoonAttr":true,"sizeMin":172,"sizeMax":178,"measurementInterval":"172 cm - 178 cm","comingSoonReason":"productOrPreferenceInstockDate","comingSoon":true,"availability":{"messages":["Back order"],"inStockDate":"2022-05-02T00:00:00.000Z","onlyXLeftNumber":1,"onlyXLeft":false,"lowStock":false,"shippingInfo":"Coming soon: <span class=\"productConfiguration__shippingDate\">May 2, 2022 - May 13, 2022</span>","customizationMessage":null,"variationValueMessage":"Coming soon","variationValueSubtext":"May 2, 2022 - May 13, 2022","available":false,"availableSufficient":true,"notifyMe":true,"similarBikes":false,"comingSoonByBackOrderAllocation":false,"isBackOrder":false},"hasSuccessorProduct":false,"comingSoonMessage":"Coming soon: <span class=\"productConfiguration__shippingDate\">May 2, 2022 - May 13, 2022</span>"}

In case you found it hard to scroll right, the key part I want to highlight is that for each coming soon (per size) there was a value for instock and it looked something like this:

"inStockDate":"2022-05-02T00:00:00.000Z",

Eureka! I figured that this was all that I needed, and the CMS would automatically make it available to order at this UTC time. The dates lined up with what Canyon Live Chat stated.

But it never happened. These dates came and went for the various models that I was keeping tabs on but there were no bikes. Given the unreliable information that I was being told both from Canyon Live Chat and from the HTML rendered output, it was time to take things into my own hands.

Let’s Get Serious – Beautiful Soup
After going in circles, being taunted by Canyon’s Instagram and my desire to ride in the real word, I needed to author some code. I needed eyes on the Canyon website. I figured that this was a practical way to put the coding skills my 11-year-old son, an avid Python programmer, had been learning, into practice.

There are many ways to do this, from using COTS applications like Selenium through to authoring your own code. I always choose the latter and whilst I wouldn’t consider myself a classically trained programmer, I can get the job done.

I thought about looking at other bikes on Canyon’s website which were for sale and using TamperData to modify the POST payload and circumvent the in-ability to place an order for the bike I wanted, but I wanted to do this fairly, not to mention that it would look a bit strange adding an unavailable item to cart.

Instead, I decided to keep close tabs on their website and order via their process but ensure that I would get notified the minute this came on sale. The logic below is similar in any language, but I code most things in Python, because the only always running compute in my house is a Raspberry Pi. The task I had in front of me was parsing Canyon’s website and for that l used Beautiful Soup.

Beautiful Soup is a Python package for parsing HTML and XML documents (including having malformed markup, i.e. non-closed tags, so named after tag soup). It creates a parse tree for parsed pages that can be used to extract data from HTML, which is useful for web scraping.

Beautiful Soup (HTML parser) – Wikipedia

The Beautiful Soup package performs much of the heavy lifting, but what my code was doing was counting the number of times ‘Coming soon’ was on each product page. When no sizes were available, this would equal ‘Coming soon’ being present 17 times on each page. If the number did not equal 17, the code would trigger an alert, other than that it would sleep for 60 seconds and then try again.

# Import requests (to download the page)
import requests
# Import BeautifulSoup (to parse what we download)
from bs4 import BeautifulSoup
# Import Time (to add a delay between the times the scape runs)
import time
import datetime
# Import smtplib (to allow us to email)
import smtplib

# while this is true (it is true by default),
while True:
    try:
        print (datetime.datetime.now())
        print (" Connecting to Canyon website --> AeroRoad CF SLX - Black SRAM") 
        #url = "https://www.canyon.com/en-au/road-bikes/aero-bikes/aeroad/cf-slx/aeroad-cf-slx-8-disc-di2/2771.html?dwvar_2771_pv_rahmenfarbe=BK%2FBK"
        url = "https://www.canyon.com/en-au/road-bikes/aero-bikes/aeroad/cf-slx/aeroad-cf-slx-8-disc-etap/2772.html?dwvar_2772_pv_rahmenfarbe=BK%2FBK"
        # "
        # set the headers like we are a browser,
        headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
        # download the homepage
        response = requests.get(url, headers=headers)
        # parse the downloaded homepage and grab all text, then,
        soup = BeautifulSoup(response.text, "lxml")
        countvar = str(soup)
        CountNumber = str(soup).find("Coming soon")
        print (datetime.datetime.now())
        print ("Number of Coming Soon's")
        print (countvar.count('Coming soon'))

                                    
        
        # Coming soon should be 17
        if (countvar.count('Coming soon')) != 17:
        

            # create an email message with just a subject line,
            msg = 'Subject: Canyon Alert - Check website'
            # set the 'from' address,
            fromaddr = 'canyon@baldacchino.net'
            # set the 'to' addresses,
            toaddrs  = ['me@baldacchino.net']
            
            # setup the email server,
            server = smtplib.SMTP('smtp.gmail.com', 587)
            server.starttls()
            # add my account login name and password,
            server.login("user@domain.com", "AppKey")
            
            # Print the email's contents
            print('From: ' + fromaddr)
            print('To: ' + str(toaddrs))
            print('Message: ' + msg)
            
            # send the email
            server.sendmail(fromaddr, toaddrs, msg)
            # disconnect from the server
            # server.quit()
        else:
            print ()
            print ("Rechecking.... ")
            print ()
        
        # wait 60 seconds,
        print (datetime.datetime.now())
        print ("Sleeping 60 seconds") 
        time.sleep(60)
     
        # continue with the script,
        continue
    except:
        pass #

Nothing too special about this code but let’s step through it

1. Firstly, lets import our Python libraries, ‘Beautiful Soup’ being our parser
2. I enter a loop that has no escape clause, so this runs for ever. I need it to run forever as I constantly want it to be checking
3. I define the URL to check
4. I set the user-agent to appear as though I am using Chrome on a Mac. This is important as many systems will block non-browser based user agents as they are a clear sign of bots. Canyon’s robots.txt file funnily enough allows a user-agent of *. If anyone from Canyon is reading this, you may want to be a bit more definitive
5. I parse the response back from ‘Beautiful Soup’. If ‘Coming soon’ does not equal 17 I send an email using an SMTP server (Google Apps in this case).

Running this script would result in output.

022-04-11 14:41:38.231503
 Connecting to Canyon website --> AeroRoad CF SLX - Black
Number of Coming Soon's
17
Sleeping 60 seconds
Rechecking.... 

2022-04-11 14:42:45.939523
 Connecting to Canyon website --> AeroRoad CF SLX - Black
Number of Coming Soon's
17
Sleeping 60 seconds
Rechecking.... 

Dealing With Noise – Website Updates
The above will generate some noise. What became clear is that when Canyon updates their website I would get an email every minute while they were doing maintenance. There would often be hundreds of emails when I woke up based on maintenance tasks. Some code in my python script could be added to deal with this, but I kept it as is as it was re-assuring to know that everything was working.

Notification Options
You may be thinking an email is far too passive of a delivery device. What about a push notification via SMS using services like Clickatell, Amazon SNS and Twillio? I took an educated/calculated guess based on the information that I was getting. What became clear is Canyon typically update their stock on Monday morning CET (Central European Time). This lined up quite nicely with my time zone. It usually meant new stock would arrive around 7pm, a time when I usually have my phone on hand, checking notifications. It was a risk I was willing to take. It turned out that it was quite a risk because I actually bought my Aeroad on a Tuesday night which was an antipattern.

Running The Script – Using A Humble Raspberry Pi
To run the above script all you need is a device capable of running Python 3. From a Windows PC through to a MAC and everything in between, Python runs on every platform. What you do need though is a reliable device that is always on. I used a Raspberry Pi as they are already used in my house for automation. It was perfect for this application as well. I chose to run it in the console so I could monitor the output.

Whilst my code checked every 60 seconds, you could also execute using serverless compute in the cloud in the form of Azure Functions or AWS Lambda using basic CRON notation.

Ordering – Time is of the essence!
It’s GO time. It was a Tuesday night, so I wasn’t checking my phone as diligently as I do on a Monday. We had just finished dinner when I looked at my phone and to my shock saw around 45 messages! This bike had been on sale since 5:51pm and when I went to Canyon’s website the 2XS size was in ‘Low Stock’. Panic stations!

After some deep breaths I was looking at an order confirmation. My code had served it’s purpose.

From order to doorstep (From Germany to Australia) was 8 days. To give perspective, my son had ordered a speed cube from within Australia, Express Post a day before me, and he is still waiting. I’m impressed Canyon and packed really well. Not a scratch.

Building was straight forward, though your mileage may vary based on your capabilities, but in essence, you are assembling the cockpit, pedals, seat post, updating firmware and pairing SRAM components and that’s about it. I took special care to ensure there was not a scratch on it. It is carbon fiber, so a torque wrench is quite a handy tool (3-40nm) but not required given the included tooling.

Summary
Business models are changing, from Canyon selling bikes online, through to Tesla (and others) selling cars. Direct to consumer has many benefits, but for desirable products (GPU’s / latest generation of consoles) it can be quite hard to procure these without the use of technology. There is a reason why GPU’s are almost instantly out of stock before they have been listed.

With such a limited supply and far too much demand, it is technology that gives you the leg up in being able to procure these products.

I hope this post has shown you just how easy it is. With a little bit of technology you too can keep tabs on websites and if it helps you procure your Canyon bike drop me a message, I would love to hear from you.

UPDATE
I have made updates to this script based on user feedback. The following will look for the specific Size and Coming Soon. In the code snippet below I am looking for a size Large (L). Adjust accordingly to your size.

# Import requests (to download the page)
import requests
# Import BeautifulSoup (to parse what we download)
from bs4 import BeautifulSoup
# Import Time (to add a delay between the times the scape runs)
import time
import datetime
# Import smtplib (to allow us to email)
import smtplib

# while this is true (it is true by default),
while True:
    try:
        print (datetime.datetime.now())
        print (" Connecting to Canyon website --> Endurace CF 7 Di2") 
        url = "https://www.canyon.com/en-ca/endurace-cf-7-di2/50021132.html"
        # "
        # set the headers like we are a browser,
        headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
        # download the homepage
        response = requests.get(url, headers=headers)
        # parse the downloaded homepage and grab all text, then,
        soup = BeautifulSoup(response.text, "lxml")
        countvar = str(soup)
        print ("Searching for L and Coming Soon")
        DetectSize = str(soup).find("\"id\":\"L\",\"description\":null,\"displayValue\":\"L\",\"value\":\"L\",\"selected\":true,\"comingSoon\":true")
        print (DetectSize)
        #sprint (datetime.datetime.now() + "- Detection has a value of " + DetectSize)
        
       
                   
        
        # Will be -1 if not found
        if (DetectSize) == -1 :
        

            # create an email message with just a subject line,
            msg = 'Subject: Canyon Alert - Check website'
            # set the 'from' address,
            fromaddr = 'user@domain.com'
            # set the 'to' addresses,
            toaddrs  = ['user@domain.com']
            
            # setup the email server,
            server = smtplib.SMTP('smtp.gmail.com', 587)
            server.starttls()
            # add my account login name and password,
            server.login("user@domain.com", "password")
            
            # Print the email's contents
            print('From: ' + fromaddr)
            print('To: ' + str(toaddrs))
            print('Message: ' + msg)
            
            # send the email
            server.sendmail(fromaddr, toaddrs, msg)
            # disconnect from the server
            # server.quit()
        else:
            print ()
            print ("Rechecking.... ")
            print ()
        
        # wait 60 seconds,
        print (datetime.datetime.now())
        print ("Sleeping 60 seconds") 
        time.sleep(60)
     
        # continue with the script,
        continue
    except:
        pass #

Thanks
Shane Baldacchino