Web Scraping with AutoScraper: A Step-by-Step Tutorial

In this tutorial, I’ll guide you through setting up AutoScraper, collecting data from websites, and saving it to a CSV file for easy analysis. You’ll find that with AutoScraper, web scraping can be simple and efficient, helping you focus on analyzing data rather than wrangling code.

What is AutoScraper?

AutoScraper is a Python library that automatically learns data structures based on your examples. It uses a combination of AI and heuristic analysis to detect patterns in a webpage, simplifying the data extraction process. Unlike other scraping libraries, such as BeautifulSoup or Scrapy, which require explicit data-path coding, AutoScraper interprets the data’s structure and learns how to extract similar data independently.

AutoScraper’s main advantages include:

• Minimal Code: Provide an example of the data you want to scrape, and AutoScraper will take care of the rest.
• Structured Data Handling: Ideal for websites that follow a clear format, like product listings or information tables.
• No Manual HTML Inspection: Perfect for beginners unfamiliar with HTML structures.

AutoScraper Alternatives for Dynamic Content

For websites with heavy JavaScript or CAPTCHA protections, consider using:

• Selenium: Automates browser interaction, ideal for dynamic content.
• Splash: A headless browser solution with JavaScript support.
• Web Scraping API: Provides structured data from complex sites like Amazon and LinkedIn.

Prerequisites

To follow along, you’ll need:

• Python 3+: Ensure you have the latest Python version.
• AutoScraper: Install via pip (pip install autoscraper).
• Pandas: For saving data to a CSV file (pip install pandas).

Let’s begin by setting up our environment.

Setting Up the Project

Start by creating a project directory and setting up a virtual environment.

# Create project directory
mkdir web_scraping_tutorial
cd web_scraping_tutorial
# Set up virtual environment
python -m venv env
source env/bin/activate # for MacOS/Linux users
env\Scripts\activate # for Windows users

Next, install the required libraries:

pip install autoscraper pandas

Choosing a Website to Scrape

AutoScraper works well on sites with clearly structured data, like lists or tables. In this tutorial, to make it easy, we’ll scrape data from a sample website called Books to Scrape, a site designed specifically for testing scraping tools. Here, we’ll gather book titles, prices, and ratings.

Building the Scraper

Let’s jump into code to build our first scraper with AutoScraper.

Import Libraries

Start by importing AutoScraper and Pandas.

from autoscraper import AutoScraper
import pandas as pd

Define the Target URL and Example Data

We’ll set up the URL and provide sample data, which AutoScraper will use to identify patterns. Here, we’re extracting book titles, prices, and ratings.

url = "http://books.toscrape.com/"
wanted_list = ["A Light in the Attic", "£51.77", "Three"]

The wanted_list contains example data from the website. AutoScraper will learn from these values to find similar data.

Build the Scraper

Now, create an instance of the scraper and use the build method to scrape the page based on the examples in wanted_list.

scraper = AutoScraper()
scraper.build(url, wanted_list)

Reviewing the Results

Check what AutoScraper has extracted to ensure it’s pulling the correct data.

results = scraper.get_result_similar(url, grouped=True)
print("Keys found by the scraper:", results.keys())

AutoScraper will display a set of rules it generated. You’ll see keys like rule_0xs7 and rule_1dmx that store extracted data.

1. Organize and Store the Data

Assign column names and organize the data into a Pandas DataFrame.

columns = ["Title", "Price", "Rating"]
data = {columns[i]: results[list(results.keys())[i]] for i in range(len(columns))}
df = pd.DataFrame(data)

Save Data to CSV

Finally, save the DataFrame to a CSV file.

df.to_csv('books_data.csv', index=False)
print("Data saved to books_data.csv")

You now have a CSV file containing book titles, prices, and ratings from the website.

Scraping Paginated Content

Websites with multiple pages, or “pagination,” present a challenge in web scraping. For instance, Books to Scrape has multiple pages of book listings. Here’s how to expand AutoScraper to handle pagination.

Update URL and Sample Data

Define the URL pattern for each page and update wanted_list to reflect sample data from multiple pages.

urls = [f"http://books.toscrape.com/catalogue/page-{i}.html" for i in range(1, 3)]

Scrape Data Across Pages

Loop through each page URL and accumulate the data.

all_data = []
for page_url in urls:
results = scraper.get_result_similar(page_url, grouped=True)
data = {columns[i]: results[list(results.keys())[i]] for i in range(len(columns))}
all_data.append(pd.DataFrame(data))
full_data = pd.concat(all_data, ignore_index=True)
full_data.to_csv('books_data_paginated.csv', index=False)

Using AutoScraper for Complex Websites

AutoScraper’s built-in rule editor is handy for complex layouts, such as those with nested tables. Let’s illustrate this with a movie listing site where we want to scrape movie titles, release years, and ratings.

Define URL and Sample Data

Set up the target URL and wanted_list using sample data from the movie listing page.

url = "https://sample-movie-site.com/movies"
wanted_list = ["Inception", "2010", "8.8"]

Train and Prune Rules

Train AutoScraper and then prune unnecessary rules.

scraper.build(url, wanted_list)
rules_to_keep = ['rule_1kq7', 'rule_a5xp', 'rule_9vbn'] # Sample rule names for data columns
scraper.keep_rules(rules_to_keep)
scraper.save('movies_model.json')

Extract Data with Trained Model

With the model trained and saved, extract data from pages with similar structures.

scraper.load('movies_model.json')
results = scraper.get_result_similar(url, grouped=True)
# Define columns based on rules and organize data
columns = ["Title", "Year", "Rating"]
data = {columns[i]: results[list(results.keys())[i]] for i in range(len(columns))}
df = pd.DataFrame(data)
df.to_csv('movies_data.csv', index=False)

Common Challenges with AutoScraper

Despite AutoScraper’s ease, some challenges may arise:

JavaScript-Rendered Pages: AutoScraper doesn’t handle JavaScript, so you may need tools like Selenium or Playwright for such sites.

Rate Limiting: Frequent requests can trigger rate limits, so use libraries like ratelimit to pace requests.

IP Blocking: For high-traffic scraping, employ proxy servers to prevent IP bans. Here’s how to set up a proxy in AutoScraper:

request_args = {
"headers": {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
},
"proxies": {
"http": "http://user:pass@proxyserver:port"
}
}
scraper.build(url, wanted_list=wanted_list, request_args=request_args)

Conclusion

AutoScraper is a straightforward tool that makes scraping data from static websites easy, even if you’re new to coding or aren’t familiar with HTML. I’ve walked through the basics here — how to install and set it up, handle pagination, and scrape more complex sites. Although AutoScraper may not be the best choice for every scraping task, it’s perfect for gathering data quickly without a steep learning curve.

If you’re dealing with JavaScript-heavy websites or sites that use CAPTCHA, consider pairing AutoScraper with Selenium or switching to a more advanced tool like Bright Data. It all depends on the specifics of what you’re trying to scrape and the site’s structure.