Node-unblocker for Web Scraping

In this guide, I’ll walk you through Node Unblocker — what it is, why it’s useful, and how to set it up in your Node.js projects. By the end, you’ll know exactly how to use Node Unblocker to take your web scraping projects to the next level.

What is Node Unblocker?

Node Unblocker is an open-source web proxy designed specifically for Node.js applications. As a proxy, it acts as an intermediary between the client and the target website, allowing users to bypass internet censorship, geo-restrictions, and IP-based blocking. With Node Unblocker, developers can create customizable proxies tailored to their web scraping needs, ensuring efficient data gathering without worrying about access limitations.

Alternative to Web Scraping — Datasets

You can skip the whole scraping process and get the data you need in a matter of minutes!

1. Bright Data — Customizable and pre-built datasets across industries.
2. Statista — Extensive statistics and reports for business and research.
3. Datarade — Marketplace for premium data products from various providers.
4. AWS Data Exchange — Third-party datasets integrated with AWS services.
5. Zyte — Web scraping and custom datasets tailored to business needs.
6. Data & Sons — Open marketplace for buying and selling diverse datasets.
7. Coresignal — Workforce analytics with extensive job-related data.
8. Oxylabs — Specialized company data and web scraping services.
9. Bloomberg Enterprise Data Catalog — Financial data for enterprise use.
10. Kaggle — Free public datasets and tools for data science.

Read more about all the providers here. I am not affiliated with any of the providers mentioned.

Key Features of Node Unblocker

• Bypasses Internet Censorship: Node Unblocker enables users to access restricted content by routing requests through different IP addresses.
• High Speed and Efficiency: Built with optimized data handling, Node Unblocker relays information without buffering or lagging, ensuring smooth and fast access to data.
• Open-Source and Customizable: Developers can tweak Node Unblocker according to project-specific needs, adjusting headers, response handling, and other configurations.
• Multi-Protocol Support: Node Unblocker works seamlessly with HTTP, HTTPS, and WebSockets, making it versatile for different web scraping scenarios.

Advantages of Using Node Unblocker in Web Scraping

Node Unblocker offers various advantages for web scraping that can significantly simplify and optimize the data extraction process:

Bypassing Geo-Restrictions

Node Unblocker’s proxy capabilities enable you to scrape data from websites that might otherwise be inaccessible due to geo-restrictions. This is particularly useful if you need access to localized content or region-specific data, like prices or news.

Fast Data Relay

With optimized data handling, Node Unblocker provides rapid data relay, which is crucial for large-scale web scraping. The faster your proxy can fetch data, the more efficient your scraping process becomes, especially for high-frequency tasks.

User-Friendly API

Node Unblocker’s API is simple and intuitive, making it accessible even for those new to proxies. Integrating it into your project requires minimal setup, allowing you to focus more on developing your scraping logic and less on the intricacies of proxy management.

Customizability for Enhanced Functionality

Node Unblocker allows for extensive customization, so you can adjust request headers, control response handling, or configure other proxy settings as needed. This flexibility ensures that your proxy aligns with the unique requirements of your scraping project.

Compatibility with Multiple Protocols

Whether your project involves HTTP, HTTPS, or WebSocket requests, Node Unblocker supports all three. This multi-protocol support lets you interact with various data sources, making Node Unblocker a versatile choice for complex scraping tasks.

How to Set Up Node Unblocker for Web Scraping

Let’s dive into setting up Node Unblocker and implementing it as a proxy for web scraping in Node.js.

Prerequisites

Before getting started, ensure you have the following:

• Node.js and npm installed on your system.
• A web browser for testing the proxy locally.
• A free Render account for deploying the proxy.

Once you’ve confirmed these prerequisites, follow these steps to set up and test Node Unblocker.

Step 1: Initialize Your Node.js Project

Create a new folder for your project and initialize it as a Node.js project:

mkdir node-unblocker-proxy
cd node-unblocker-proxy
npm init -y

Step 2: Install Dependencies

Install the necessary packages: express for setting up a web server and unblocker for creating the proxy.

npm install express unblocker

Step 3: Write the Proxy Script

Create a file named index.js and add the following code to set up the web proxy:

const express = require("express");
const Unblocker = require("unblocker");
const app = express();
const unblocker = new Unblocker({ prefix: "/proxy/" });
const port = 3000;
app.use(unblocker);
app.listen(port).on("upgrade", unblocker.onUpgrade);
console.log(`Proxy running on http://localhost:${port}/proxy/`);

In this code:

• You initialize an Express app and create a new instance of Unblocker, setting the prefix to /proxy/.
• The app.use(unblocker); command integrates Unblocker into the Express app, allowing it to act as a proxy.
• The .on(“upgrade”, unblocker.onUpgrade); method ensures WebSocket connections work smoothly through the proxy.

Step 4: Test the Proxy Locally

Run the following command in your terminal to start the proxy:

node index.js

To test it, open your browser and go to http://localhost:3000/proxy/https://example.com. This will load the specified URL through the proxy, allowing you to confirm that Node Unblocker is working as expected.

Step 5: Deploy the Proxy to Render

To deploy Node Unblocker to Render, follow these steps:

Add the following script to your package.json to ensure Render starts the server properly:

"scripts": {
"start": "node index"
}

Push your project to a GitHub repository, then sign in to Render and create a new web service connected to that repository.

Once deployed, test your proxy by visiting <DEPLOYED-APP-URL>/proxy/<target-url> in your browser.

Integrating the Proxy with Puppeteer for Web Scraping

Now that your proxy is live, you can use it with a web scraping library like Puppeteer. Here’s how to set it up:

Step 1: Install Puppeteer

Run the following command to install Puppeteer:

npm install puppeteer

Step 2: Write a Scraping Script

Create a file named scrape.js and add the following code to scrape data through the deployed proxy:

const puppeteer = require("puppeteer");
const scrapeData = async () => {
const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();
await page.goto("<DEPLOYED-APP-URL>/proxy/https://example.com");
const data = await page.evaluate(() => {
let content = [];
document.querySelectorAll(".desired-element").forEach(item => {
content.push(item.innerText);
});
return content;
});
console.log(data);
await browser.close();
};
scrapeData();

Customizing the Proxy

By adjusting the unblocker configuration, you can tailor your proxy to fit specific needs. For instance:

• Change Headers: Customize request headers for authentication or specific content retrieval.
• Add IP Rotation: Use rotating proxies to avoid detection and rate limits.

Best Practices for Using Node Unblocker

Choose the Right Proxy Type

When selecting a proxy service, ensure it aligns with your project’s needs. For example, residential proxies can bypass geo-restrictions more effectively than datacenter proxies. Check out my list of the best residential proxies here, and the best datacenter proxies here.

Implement IP Rotation

Use a rotating proxy service that provides fresh IPs for each request to reduce the likelihood of IP bans.

Monitor Proxy Performance

Regularly assess the speed and reliability of your proxy to ensure it can handle the demands of your scraping project

Conclusion

Node Unblocker is a great tool for getting around internet restrictions and accessing content that’s blocked in certain regions. It has a simple setup, many customization options, and supports multiple protocols. These features make it a solid choice for handling challenging web scraping tasks. Whether you’re scraping data for market research, SEO, or other analysis, Node Unblocker makes it easier to avoid common problems like IP blocking and rate limits.