With Selenium you can automate web scraping, search tasks, logging into multiple accounts and more. This open source, platform independent, cross-browser and versatile automation framework is built to save time and enhance productivity. However one obstacle stands in the way CAPTCHAs.
A Completely Automated Public Turing test to tell Computers and Humans Apart (CAPTCHA) is a tool used by various site owners to keep humans in and bots out.
You are right! CAPTCHAs are not friendly with Selenium. But since you are here worry less! Take a deep dive into this piece to learn how to Bypass CAPTCHAs with Selenium.
Bypassing CAPTCHAs with Selenium
Understand how different CAPTCHAs work
Before you proceed to bypass various CAPTCHAs you ought to understand how each works. Thanks to Selenium’s support for multiple languages you may execute the rest of the steps in your preferred coding language. For instance you can Bypass CAPTCHAs with Selenium in Python, Java or JavaScript.
Now while interacting with various websites you are likely to come across these four types of CAPTCHAs:
Text based CAPTCHAs: These ones present you with the challenge of recognizing and inputting pre defined text correctly. The text is usually overlapped, warped or blurred to make it challenging for bots to solve but easier for humans to decipher.
Image-based CAPTCHAs: This type requires you to observe the presented pictures and select specific ones as directed. For instance the CAPTCHA puzzle may highlight four pictures containing cars among other random photos and require you to select the four photos only. If you look attentively you’ll see that among other things the four photographs include autos. This is how these CAPTCHAs can tell the difference between a bot and a person. The visual intricacy of the photographs makes it difficult for computers to properly identify them but humans are likely to solve the issue fast.
reCAPTCHA v2/v3: Google built both v2 and v3 to aid in protecting sites from automated scripts or spam. reCAPTCHA v2 may require you to solve an image selection puzzle or just click on a checkbox saying “I’m not a robot.” reCAPTCHA v3 runs in the background scanning website interactions to determine whether its a human or bot navigating a website. It is less intrusive and effective at the same time.
Invisible reCAPTCHA: This kind is supposed to allow for a smooth user experience while also protecting a website from bots. Like reCAPTCHA v3 it runs in the background and does not require puzzle solving or checkbox clicking. Invisible reCAPTCHA assesses user behavior like browsing patterns mouse clicks and movements, and wait-time to conclude whether its a bot or human exploring the website.
Explore the nature of the target site’s CAPTCHAs
To effectively bypass CAPTCHAs with Selenium you must understand how the target sites CAPTCHA system works. Visit the website in question and interact with it to determine what triggers CAPTCHAs or when CAPTCHAs show up.
Then, when CAPTCHAs show up examine their nature. Are they text based, image based or reCAPTCHA? Use the web browser tool to inspect the websites source code and network requests. This helps you tell whether the website is using third party CAPTCHA services or a custom CAPTCHA script.
Study the CAPTCHA to understand how it challenges the web visitor. Also deliberately provide the wrong answer to see whether the challenge gets harder with time.
Use Selenium to simulate human behavior in your automation script
Armed with a vivid picture of what triggers the target websites CAPTCHAs prepare your automation script with Selenium. Remember the aim is to bypass the CAPTCHAs not solve them.
So your script must pass as a regular user and not a bot. To do this Selenium provides you with tools to copy the following human behavior to avoid triggering CAPTCHAs:
Delays and Waits
Selenium allows you to implement waits or delays in between actions and events. You should use this feature to implement load time waits especially when automating interactions with a dynamic website. Before proceeding your script should wait for all of the pieces to load completely.
Randomized clicks
Unlike humans, bots may repeatedly click on one area of a button raising suspicion. Using Selenium you can randomize clicks around a buttons boundary. Moreover you should include delays in between clicks to make it look like its a real user interacting with the website.
Mouse movements and element hovering
While bots move the mouse in a straight path or line humans tend to move the mouse in an irregular or inconsistent path. Also, when a human points the cursor at an element they at times hover over it while considering whether to click. Selenium is able to generate curved paths between elements and pause for a while before clicking an element.
Typing and scrolling patterns
Even though Selenium can simulate typing with the help of the send_keys() method it inputs text instantly. To make the process human like you can include delays between keystrokes.
Besides mimicking human typing you should also adjust scrolling speed or implement random scroll pauses to improve on Seleniums ability to scroll through a page like a human.
Browser window resizing
Bots rarely resize browser windows humans on the other hand do resize browser windows from time to time to improve visibility. Selenium allows you to add this capability to your script allowing your script to randomly resize or maximize the browser window.
Incorporating these human like capabilities into your automation script should see you automate web activities with little to no hiccups. Nevertheless you should think about expanding your script’s ability to get over even the most obstinate CAPTCHAs if it still prompts them after you’ve added these human like layers.
A tool like Puppeteer includes a feature called “stealth mode” that’s capable of masking automation or the use of bots. You can use it to remove browser fingerprints and indicators that you are using a Selenium script making it hard for CAPTCHA systems to detect your script.
Closing Words
Now that you understand how to bypass CAPTCHAs with Selenium you can select your preferred coding language and curate a human like script. Go automate form submissions, data entry, file downloads, web navigation and web scraping with minimal to no worry of triggering CAPTCHAs!
