How to Set Scrapy Headers: A Step-by-Step Guide

Here, I’ll walk you through everything you need about Scrapy headers. We’ll cover why they matter, how you can change them, and some simple tips to use them effectively. By the end, you’ll feel confident handling headers and keeping your scraping projects running without a hitch. Let’s dive in!

Why are Scrapy Headers Important?

HTTP headers are metadata exchanged between a client (like a browser or a scraper) and a server during requests and responses. They tell the server how to handle your request and what to send back. Headers are critical in web scraping. Servers often check them to decide if a request is from a real user or a bot.

Scrapy uses default headers when sending requests, but these headers can be problematic because they reveal that the request is from an automated tool. Many websites have anti-bot mechanisms that flag or block such requests. Customizing headers allows you to:

Mimic real browsers: Your scraper will appear more legitimate if you set headers similar to those of popular browsers like Chrome or Firefox.
Handle session management: With cookies and other session-related headers, you can maintain login states or bypass restricted areas.
Reduce blocks: Anti-bot systems often rely on analyzing headers to detect suspicious activity. Proper customization lowers the chances of being flagged.
Improve data retrieval: Some servers respond differently based on headers, such as delivering specific content formats or language preferences.

Types of HTTP Headers

HTTP headers are broadly categorized into two types:

Request Headers: These are sent by the client to the server and include metadata like browser type, language preference, and referrer information.
Response Headers: Sent by the server to the client; these include data about the server, content type, and caching policies.

We focus primarily on request headers for web scraping since they influence how the server processes our request.

Scrapy Default Headers

By default, Scrapy sends basic headers with every request, such as:

{
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Encoding": "gzip, deflate",
"Accept-Language": "en",
"User-Agent": "Scrapy/2.11.0 (+https://scrapy.org)"
}

These headers lack some critical components that real browsers include, such as:

Referer: Indicates the URL from which the request originated.
Sec-Ch-Ua: Specifies browser and platform details for enhanced legitimacy.
Upgrade-Insecure-Requests: Informs the server that the client prefers secure HTTPS connections.

Customizing these headers makes your requests appear more authentic and less likely to be blocked.

How to Customize Headers in Scrapy

Customizing headers in Scrapy is straightforward. Here are different methods to modify and manage headers effectively.

Modifying Headers in settings.py

You can define a dictionary of custom headers in Scrapy’s settings file. This method applies the headers to all requests made by your spider.

Example: settings.py

DEFAULT_REQUEST_HEADERS = {
'Accept-Language': 'en-US,en;q=0.9',
'Referer': 'https://www.google.com/',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
'Accept-Encoding': 'gzip, deflate, br',
'Sec-Ch-Ua': '"Not A(Brand";v="99", "Google Chrome";v="121", "Chromium";v="121"',
'Sec-Ch-Ua-Platform': '"Windows"',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36',
}

When the spider sends requests, these headers will replace the default ones.

Using Custom Headers for Specific Requests

If you need different headers for specific requests, you can pass them directly in your spider file using scrapy.Request.

Example: Custom Headers in a Spider

import scrapy
class CustomHeaderSpider(scrapy.Spider):
name = "custom_header"
allowed_domains = ["httpbin.org"]
start_urls = ["https://httpbin.org/headers"]
def start_requests(self):
headers = {
"User-Agent": "Mozilla/5.0 (Linux; x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36",
"Sec-Ch-Ua-Platform": '"Linux"',
}
for url in self.start_urls:
yield scrapy.Request(url, headers=headers, callback=self.parse)
def parse(self, response):
print(response.text)

This approach is useful when scraping multiple domains requiring different headers.

Dynamic Header Modification

Scrapy’s Downloader Middleware can modify headers dynamically based on conditions (e.g., rotating User Agents). This allows you to intercept and alter requests before they are sent. Learn more about how to use user agents for web scraping.

Example: Middleware for Rotating User-Agents

class RotateUserAgentMiddleware:
def process_request(self, request, spider):
user_agents = [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
"Mozilla/5.0 (Linux; x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"
]
request.headers['User-Agent'] = random.choice(user_agents)
Add the middleware to settings.py:
DOWNLOADER_MIDDLEWARES = {
'myproject.middlewares.RotateUserAgentMiddleware': 543,
}

Using Proxies and Anti-CAPTCHA Services

To further enhance header legitimacy and bypass blocks, consider using proxy services or anti-CAPTCHA tools. These services automatically rotate headers and User-Agents for you.

Example: Using ZenRows with Scrapy

import scrapy
class ZenRowsSpider(scrapy.Spider):
name = "zenrows"
allowed_domains = ["httpbin.org"]
start_urls = ["https://httpbin.org/headers"]
def start_requests(self):
proxy = "@api.zenrows.com:8001">http://YOUR_ZENROWS_API_KEY:@api.zenrows.com:8001"
for url in self.start_urls:
yield scrapy.Request(url, callback=self.parse, meta={"proxy": proxy})
def parse(self, response):
print(response.text)

Most Important Headers for Web Scraping

Certain headers are more critical for web scraping and should be customized carefully:

User agent: This identifies the browser, OS, and version. To avoid detection, ensure it mimics a real browser.
Referer: This indicates the origin of the request. Set it to a meaningful URL, such as Google search or the previous page.
Cookie: Used for session management. It helps maintain logged-in states or bypass access restrictions.
Accept-Language: Specifies language preferences. Use en-US,en;q=0.9 to mimic English-speaking browsers.
Sec-Ch-Ua: Contains browser and platform details. It helps modern browsers bypass advanced anti-bots.
Accept-Encoding: Informs the server about supported compression formats like gzip or br.

Conclusion

Headers are a big deal in web scraping. If you set them up correctly, your scraping will work smoothly. By customizing headers in Scrapy, I can make my scraper look like a real browser, avoid getting blocked, and collect data more efficiently. An excellent first step is to update the headers in the settings.py file. For more advanced features, I can explore dynamic solutions like using middleware.

It doesn’t matter if I’m scraping a small blog or a big e-commerce site — learning how to manage Scrapy headers makes all the difference. I’ll keep experimenting, tweaking, and improving my setup to ensure my scraper stays effective and under the radar. Let’s scrape smart and responsibly!