How to Use a Proxy with Node-Fetch in 2025

This guide explains how I can set up proxies with Node-Fetch, manage IP rotation, and choose between free and premium services depending on my needs. Let’s dive in!

Prerequisites

To follow along, ensure you have:

• Node.js (v12.20.0 or later)
• A terminal or command prompt
• Installed Node-Fetch and HTTPS-Proxy-Agent

Step 1: Installing Node-Fetch and HTTPS-Proxy-Agent

In your project directory, open a terminal and run the following commands:

npm i node-fetch
npm i https-proxy-agent

If you encounter an error such as cannot find module ‘node-fetch’, it’s often due to module resolution issues. Try running npm install to update dependencies in your project. Also, ensure your package.json file has “type”: “module” set, as Node-Fetch v3.x is an ESM-only module.

Understanding Proxies and Why You Need Them for Web Scraping

A proxy acts as an intermediary between your application and the target server. When you use a proxy, your requests appear to come from the proxy’s IP address rather than your own, which can help prevent blocks.

Proxies come in two main types:

• HTTP Proxies: Support only HTTP traffic.
• HTTPS Proxies: Support HTTP and HTTPS traffic, making them more versatile.

Using a proxy with Node-Fetch involves passing a custom proxy agent in your fetch request. However, Node-Fetch does not natively support proxies, so we’ll be using the https-proxy-agent package for this purpose.

Free vs. Paid Proxies: What’s Best for Your Application?

There are free proxy lists available online, but these come with some trade-offs:

• Pros: Free and widely accessible.
• Cons: Typically unstable, slower, and more likely to be blocked by websites.

For applications requiring high availability, like web scraping, paid proxies offer much better reliability and speed. Some services provide rotating proxies and anti-block features, which we’ll discuss in more detail.

Setting Up a Static Proxy with Node-Fetch

To start with a basic setup, let’s use a static proxy with Node-Fetch to make an HTTP request. This setup is suitable if you have a single proxy server.

Import the required modules:
import fetch from 'node-fetch';
import { HttpsProxyAgent } from 'https-proxy-agent';

Define your proxy configuration:

const proxyHost = '200.105.215.22';
const proxyPort = 33630;
const proxyUrl = `http://${proxyHost}:${proxyPort}`;
const proxyAgent = new HttpsProxyAgent(proxyUrl);

Make a request through the proxy:

(async () => {
const targetUrl = 'https://ident.me';
try {
const response = await fetch(targetUrl, { agent: proxyAgent });
const data = await response.text();
console.log(data); // Expected output: Proxy IP address
} catch (error) {
console.error(error);
}
})();

In this example, we’re targeting https://ident.me, which returns the requester’s IP address. This setup is fine for simple tasks, but it’s often insufficient for intensive web scraping or rate-limited APIs.

Rotating Proxies with Node-Fetch: Handling Dynamic IP Blocking

Rotating proxies are ideal for high-volume scraping and help spread requests across multiple IP addresses, reducing the likelihood of being blocked.

Read more about the best rotating proxies here

1. Bright Data: High-performance proxies with advanced features, ideal for comprehensive solutions.
2. Smartproxy: Affordable, reliable proxies with global reach, perfect for web scraping.
3. Oxylabs: Exceptional performance and support, great for businesses needing top-quality proxies.
4. IPRoyal: Flexible rotation and pricing, suitable for small projects or as a secondary provider.
5. SOAX: Premium proxies with precise targeting and multiple rotation settings at competitive prices.
6. NetNut: Extensive capabilities for large-scale scraping, but basic documentation and dashboard.
7. Infatica: Affordable, ethically-sourced proxies with robust performance for businesses.

Step 1: Create a Proxy List

Define a list of proxies that your application can rotate through. You can use a static list or dynamically load proxies from a file or an API.

const proxyList = [
{ host: '103.69.108.78', port: 8191 },
{ host: '61.29.96.146', port: 80 },
{ host: '154.204.58.155', port: 8090 }
];

Step 2: Rotate through Proxies in Requests

Use a loop to rotate through proxies, applying a different one to each request. This technique spreads the load across multiple IPs.

import fetch from 'node-fetch';
import { HttpsProxyAgent } from 'https-proxy-agent';
const proxyList = [
{ host: '103.69.108.78', port: 8191 },
{ host: '61.29.96.146', port: 80 },
{ host: '154.204.58.155', port: 8090 }
];
async function rotateProxies(proxyList, targetUrl) {
for (const proxy of proxyList) {
const proxyUrl = `http://${proxy.host}:${proxy.port}`;
const proxyAgent = new HttpsProxyAgent(proxyUrl);
try {
const response = await fetch(targetUrl, { agent: proxyAgent });
const html = await response.text();
console.log(html);
} catch (error) {
console.error(`Error with proxy ${proxy.host}:${proxy.port}`, error);
}
}
}
const targetUrl = 'https://ident.me';
rotateProxies(proxyList, targetUrl);

This method works well with websites that don’t implement advanced bot detection mechanisms. However, consider using premium rotating proxies for websites with stronger anti-bot measures.

Using Premium Proxy Services with Node-Fetch: Bypassing Advanced Anti-Bot Measures

Free proxies are often unreliable and easily detected, so let’s look at a more robust solution. Premium proxy providers like Bright Data, ZenRows, and Oxylabs, offer rotating proxies and have sophisticated mechanisms to evade IP blocks and captchas.

Step 1: Sign Up and Get an API Key

Once you sign up with Bright Data, you’ll receive an API key allowing you to authenticate your requests through their proxy network. Bright Data also provides custom configurations and user dashboards to manage requests and monitor usage.

const apiKey = '<YOUR_BRIGHT_DATA_API_KEY>';
const targetUrl = 'https://www.amazon.com';
const apiUrl = `https://brightdata.com/api/v1/?apikey=${apiKey}&url=${encodeURIComponent(targetUrl)}`;

Step 2: Define Additional API Parameters

Bright Data offers various settings to enhance anonymity and reduce detection, such as enabling JavaScript rendering (useful for handling dynamic content) and adjusting proxy rotation settings for each request.

const params = {
"js_render": "true", // Renders JavaScript for dynamic pages
"proxy_type": "residential" // Use residential proxies for higher anonymity
};

Step 3: Make the Request through the Bright Data API

Once configured, you can use Node-Fetch to make requests via Bright Data’s network. Bright Data handles IP rotation and anti-bot mechanisms to ensure your requests go through reliably.

import fetch from 'node-fetch';
(async () => {
const response = await fetch(apiUrl, {
headers: {
'Content-Type': 'application/json',
'Authorization': `Bearer ${apiKey}`
},
params: params
});
const html = await response.text();
const statusCode = response.status;
console.log('Status Code:', statusCode);
console.log(html);
})();

Advanced Techniques: Automating Proxy Rotation with Middleware and Async Handling

For higher efficiency, consider building middleware to manage proxies automatically. Here’s an example using asynchronous handling for multiple requests:

Create a Function that selects a proxy randomly:

function getRandomProxy(proxyList) {
return proxyList[Math.floor(Math.random() * proxyList.length)];
}

Modify the Rotation Logic with concurrency control to avoid overloading servers:

import fetch from 'node-fetch';
import { HttpsProxyAgent } from 'https-proxy-agent';
const proxyList = [ /* proxy objects */ ];
async function rotateProxies(targetUrl, concurrency = 5) {
const tasks = Array.from({ length: concurrency }, async () => {
const proxy = getRandomProxy(proxyList);
const proxyUrl = `http://${proxy.host}:${proxy.port}`;
const proxyAgent = new HttpsProxyAgent(proxyUrl);
try {
const response = await fetch(targetUrl, { agent: proxyAgent });
const html = await response.text();
console.log(`Success with ${proxy.host}:${proxy.port}`);
} catch (error) {
console.error(`Error with ${proxy.host}:${proxy.port}`, error);
}
});
await Promise.all(tasks);
}
const targetUrl = 'https://ident.me';
rotateProxies(targetUrl, 3); // Set concurrency as needed

conclusion

Using proxies with Node-Fetch has become essential for reliable web scraping and data automation. Whether you choose free proxies, premium services, or a custom rotation setup, these techniques will help you maintain access and avoid IP-based restrictions. Test each method with your target sites; some may require advanced premium solutions for consistent results.

Now that you have the tools to rotate proxies and evade IP blocks, you can scrape data from even the most challenging websites. Happy scraping!