Selenium vs Beautiful Soup: when to use each, and how to combine them

Selenium and Beautiful Soup are often named together, but they solve different parts of a scraping job. Selenium automates a real browser: it opens pages, clicks, fills forms and runs JavaScript. Beautiful Soup parses HTML and XML that you already have; it never talks to a website itself, so you fetch the page with another library such as requests. Picking the wrong one is the most common reason a first scraper either can’t see the data or is far slower than it needs to be.

Which one to use

The quick test: open the page source (not the browser inspector) and search for the data. If it is there, a GET request and Beautiful Soup are enough. If it only appears after scripts run or after you interact with the page, you need Selenium.

Selenium: automating the browser

Selenium controls a browser programmatically. Besides scraping, it is widely used for automated testing of web applications.

• Browser automation: Chrome, Firefox, Safari, Edge and others.
• Interaction: clicks, form input, alerts, navigation.
• JavaScript rendering: the page runs exactly as it would for a user, so dynamically loaded content is available.

A basic Selenium 4 script:

from selenium import webdriver
from selenium.webdriver.common.by import By

# Set up the browser
driver = webdriver.Chrome()

# Navigate to a webpage
driver.get("https://example.com")

# Interact with elements
search_box = driver.find_element(By.NAME, "q")
search_box.send_keys("Web Scraping with Selenium")
search_box.submit()

# Extract data
results = driver.find_elements(By.CLASS_NAME, "result")
for result in results:
    print(result.text)

# Close the browser
driver.quit()

Beautiful Soup: parsing HTML

Beautiful Soup builds a parse tree from HTML or XML and gives you a Pythonic way to search and navigate it. It works best on static content, and pairs with requests to fetch pages.

from bs4 import BeautifulSoup
import requests

# Fetch the webpage
response = requests.get("https://example.com")
html_content = response.content

# Parse the HTML
soup = BeautifulSoup(html_content, 'html.parser')

# Extract data
for item in soup.find_all('h2'):
    print(item.text)

Combining them: Selenium to load, Beautiful Soup to parse

When the page needs a browser but the extraction itself is plain HTML parsing, let Selenium render the page, hand page_source to Beautiful Soup, and close the browser as early as possible.

Load the page with Selenium and wait for the content you need:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup

# Set up the browser
driver = webdriver.Chrome()
driver.get("https://example.com")

# Wait until the dynamic content is present
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.TAG_NAME, "h2")))

# Get page source
html_content = driver.page_source

# Close the browser
driver.quit()

Then parse it with Beautiful Soup:

soup = BeautifulSoup(html_content, 'html.parser')

# Extract data
for item in soup.find_all('h2'):
    print(item.text)

Gotchas

• find_element_by_* no longer exists. Methods such as find_element_by_name and find_elements_by_class_name were removed in Selenium 4.3. Older tutorials that use them fail with AttributeError: 'WebDriver' object has no attribute 'find_element_by_name'. Use find_element(By.NAME, ...) and find_elements(By.CLASS_NAME, ...).
• implicitly_wait doesn’t delay page_source. An implicit wait only applies when Selenium looks up an element. Calling implicitly_wait(10) and then reading driver.page_source returns whatever has loaded so far. Wait explicitly for an element that only exists once the content is there, as above.
• Content inside an iframe isn’t in the main page. Selenium only sees the current frame. Switch into the iframe with driver.switch_to.frame(...) before searching or reading page_source.
• Always call driver.quit(), ideally in a finally block. A script that crashes before quitting leaves browser processes running.

Related

• Extracting dynamic content from an iframe with Selenium in Python walks through switching frames, clicking a tab and waiting for a table with WebDriverWait.
• Automating data extraction with Airflow, BeautifulSoup and MinIO runs the requests + Beautiful Soup approach on a daily schedule.
• Handling API rate limits in Python with exponential backoff explains how to retry politely when a site starts returning 429s.