How to Make HTTP Requests in Node.js Using the Fetch API

Node.js has become one of the most popular platforms for building web applications and APIs, with over 50% of developers using it regularly according to the 2022 Stack Overflow Developer Survey. A key reason for its popularity is how easy it makes interacting with other web services through HTTP requests.

One of the simplest and most efficient ways to make HTTP requests in Node.js is using the built-in Fetch API. The Fetch API provides a modern, promise-based interface for fetching resources asynchronously across the network.

While you can use the Fetch API for any kind of HTTP communication, it‘s especially useful for web scraping – the process of programmatically collecting data from websites. With its simple syntax and powerful features, the Fetch API in Node.js is a web scraper‘s best friend.

In this ultimate guide, we‘ll take a deep dive into how to use the Fetch API for web scraping in Node.js. We‘ll cover everything from making basic requests to handling errors, parsing HTML, using proxies, and following best practices. Whether you‘re a web scraping novice or an experienced Node.js developer, this guide will level up your skills. Let‘s jump in!

Why Use the Fetch API for Web Scraping?

Web scraping is the process of automatically extracting data from websites. Instead of manually copying and pasting, you write a script that loads the webpage content and parses out the specific data you need. This could be anything from product prices to news articles to social media posts.

There are many libraries and tools available for web scraping, but the Fetch API stands out for its simplicity and flexibility. Since the Fetch API is built into modern browsers and Node.js, you don‘t need to install any additional dependencies to start making HTTP requests.

The Fetch API uses promises, which provide a cleaner syntax for handling asynchronous operations compared to callbacks. Promises make it easy to chain multiple requests together and handle errors at any step in the process.

Another benefit of the Fetch API is that it‘s not limited to just GET requests. You can make POST, PUT, DELETE and other types of requests as needed for more advanced scraping tasks. The Fetch API also supports request configuration options like setting custom headers and request bodies.

Making Your First Web Scraping Request

Enough theory, let‘s see the Fetch API in action! Here‘s a simple example of using the Fetch API to scrape a webpage and extract the title:


import fetch from ‘node-fetch‘;
import { JSDOM } from ‘jsdom‘;

fetch(‘https://www.example.com‘)
.then(response => response.text())
.then(html => {
const dom = new JSDOM(html);
const title = dom.window.document.querySelector(‘title‘).textContent;
console.log(title);
});


Let‘s break this down step-by-step:

1. First we import the fetch function from the node-fetch library. While the Fetch API is available natively in Node.js 18+, node-fetch provides some useful additional features and broader support.

2. We also import JSDOM from the jsdom library. JSDOM allows us to parse the HTML response and interact with it like a real DOM in the browser.

3. We call fetch() with the URL we want to scrape. This returns a promise that resolves to the Response object.

4. We call .text() on the response to extract the response body as a string of HTML. This returns another promise.

5. Once we have the HTML, we create a new JSDOM instance by passing the HTML string to the constructor. This parses the HTML and creates a window object with a document property, just like in a browser environment.

6. With the DOM available, we can use standard DOM methods like querySelector() to find the elements we want. Here we find the <title> element and extract its text content.

7. Finally, we log the extracted title text to the console.

This is just a basic example, but it demonstrates the core flow of making a request, parsing the HTML response, and extracting data with the Fetch API and JSDOM. You could extract any data you want by using different DOM queries and methods.

Handling Errors and Edge Cases

In the previous example, we assumed everything went smoothly – but in the real world of web scraping, you need to be prepared to handle errors and unexpected scenarios gracefully. Here are some common issues and how to address them with the Fetch API:

Non-2xx status codes

When you make a request, the server returns a status code indicating whether it was successful (codes in the 200-299 range) or if there was an error (codes 400 and above). The Fetch API considers any response to be a success, even if it has an error status code, so you need to check the status code manually:


fetch(‘https://www.example.com‘)
.then(response => {
if (!response.ok) {
throw new Error(HTTP error! status: ${response.status});
}
return response.text();
})
.then(html => {
// Handle the HTML response
})
.catch(error => {
console.error(error);
});


Here we check the response.ok property, which is true for status codes 200-299 and false for anything else. If response.ok is false, we throw an error with a custom message that includes the actual status code. The error will be caught in the final .catch() block.

Network errors

Sometimes the request itself may fail due to a network issue, like if the server is down or unreachable. The Fetch API will reject the promise in this case, so you can handle it in the .catch() block:


fetch(‘https://www.example.com‘)
.then(response => {
// Handle the response
})
.catch(error => {
if (error.name === ‘FetchError‘) {
console.error(‘Network error!‘, error);
} else {
console.error(‘Other error!‘, error);
}
});


Here we check if the error name is FetchError, which indicates a network-level failure. We log a different error message for network errors vs. other types of errors.

Invalid HTML

When parsing HTML with JSDOM, it will throw an error if the HTML is severely malformed. You can catch this error and handle it appropriately:


fetch(‘https://www.example.com‘)
.then(response => response.text())
.then(html => {
try {
const dom = new JSDOM(html);
// Handle the parsed DOM
} catch (error) {
console.error(‘Error parsing HTML‘, error);
}
});


By wrapping the JSDOM code in a try/catch block, we can catch any parsing errors and log a useful message instead of crashing the script entirely.

By adding these error checks, you can make your web scraping script much more resilient and informative.

Using Proxies for Web Scraping

When scraping websites, you may run into issues with rate limiting or IP blocking if you make too many requests too quickly from the same IP address. Web servers often track requests by IP and will cut off clients that exceed certain thresholds to prevent abuse.

To avoid this, you can route your requests through proxy servers. A proxy acts as an intermediary between your script and the target website, forwarding your requests from a different IP address. By rotating through a pool of proxy servers, you can distribute your requests and stay under rate limits.

The Fetch API supports proxying requests through the agent option. Here‘s an example using the https-proxy-agent library:


import fetch from ‘node-fetch‘;
import HttpsProxyAgent from ‘https-proxy-agent‘;

const proxyUrl = ‘http://123.456.789:1234‘;
const agent = new HttpsProxyAgent(proxyUrl);

fetch(‘https://www.example.com‘, { agent })
.then(response => {
// Handle the response
});


First, install the https-proxy-agent library and import it along with node-fetch.

Next, set the URL of your proxy server. This could be a URL provided by a proxy service you‘re using, or a proxy you‘re running yourself. The format is typically http://ip:port or http://username:password@ip:port if the proxy requires authentication.

Create a new HttpProxyAgent instance by passing the proxy URL to the constructor. Then, pass the agent to the fetch() function using the agent option in the configuration object.

Now when you make the request, it will be routed through the specified proxy server, masking your real IP address. Just be sure to rotate your proxy IPs regularly to avoid getting blocked.

Some popular proxy providers with large pools of reliable proxy servers include:

• Bright Data (formerly Luminati)
• Oxylabs
• GeoSurf
• Smartproxy
• Shifter

These services offer millions of residential and datacenter IPs to choose from, as well as helpful features like automatic proxy rotation, geo-targeting, and browser headers. Some even provide SDKs and integrations specifically for web scraping tools and frameworks.

Using a reputable proxy provider can make a huge difference in the reliability and scalability of your web scraping projects, especially when scraping large websites with anti-bot measures in place. Just be sure to use them responsibly and respect the websites‘ terms of service.

Web Scraping Best Practices

Web scraping is a powerful tool, but with great power comes great responsibility. Here are some best practices to follow to ensure your web scraping is ethical, legal, and efficient:

Respect robots.txt

The robots.txt file is a plain text file that websites use to communicate which pages should not be accessed by bots. It‘s a voluntary standard, but respecting it is considered good Internet citizenship. Before scraping a website, check its robots.txt file and avoid any disallowed URLs.

Identify your script with a User-Agent string

Set a custom User-Agent header in your requests to identify your script and provide a way for the website owner to contact you if needed. Include your name or company, the purpose of the script, and an email address. Here‘s an example:


fetch(‘https://www.example.com‘, {
headers: {
‘User-Agent‘: ‘ExampleCrawler/1.0 (https://www.example.com/crawler; [email protected])‘
}
})


Control your crawl rate

Avoid sending requests too rapidly, which can overwhelm servers and get your IP blocked. Add a delay of a few seconds between requests and consider limiting concurrent requests as well. Use caching and persistence to avoid re-requesting unchanged pages unnecessarily.

Don‘t scrape personal data without permission

Scraping publicly accessible data like product listings or articles is generally fair game, but be careful about collecting personal information like names, emails, addresses, etc. without consent. Respect privacy and comply with relevant data protection laws like GDPR and CCPA.

Use API-s and feeds when available

Many websites offer official APIs or data feeds that provide structured access to their data. Using these is often faster and more reliable than scraping the HTML pages directly. Check the website‘s documentation or contact them to inquire about API access before resorting to scraping.

Throttle and deobfuscate

If you‘re scraping a large volume of pages, be mindful of the load on the server. Throttle your requests, reducing frequency during peak hours, and spread them out geographically if possible. You can also introduce random delays and deobfuscate headers to avoid bot fingerprinting algorithms.

Conclusion

Web scraping with the Fetch API in Node.js is a powerful technique for collecting data from websites at scale. With a few lines of code, you can request web pages, parse the HTML, and extract the data you need.

The Fetch API provides a clean, promise-based interface for making HTTP requests, with support for common web scraping needs like custom headers, proxying, and error handling. By pairing it with libraries like JSDOM, you can scrape almost any data from any website.

However, with this power comes responsibility. It‘s important to follow best practices like respecting robots.txt, controlling your request rate, and using proxies and API-s when appropriate. By scraping ethically and efficiently, you can unlock valuable data insights while maintaining good Internet citizenship.

As you‘ve seen in this guide, web scraping with Node.js and the Fetch API is accessible to developers of all skill levels. You can start simple and layer on more complexity over time as you take on more ambitious scraping projects.

So go forth and scrape! Just remember, with great power comes great responsibility. Happy scraping!