How to Send a cURL POST Request

The Ultimate Guide to Sending HTTP POST Requests with cURL

If you work with web APIs, scrape data from websites, or automate server-side tasks, sooner or later you‘ll need to send an HTTP POST request. And when it comes to making HTTP requests on the command line, few tools are as ubiquitous and battle-tested as cURL.

In this comprehensive guide, we‘ll dive deep into the world of cURL POST requests, covering everything from basic syntax to advanced authentication flows and performance optimization tips. Whether you‘re a seasoned developer or just getting started with web automation, by the end of this article you‘ll be equipped with the knowledge and skills to tackle even the most complex POST request scenarios.

What is cURL?

cURL (short for "Client URL") is a command line tool and library for transferring data using various network protocols. First released in 1997 by Swedish developer Daniel Stenberg, cURL has since become one of the most widely used utilities for sending and receiving data over HTTP, HTTPS, FTP, SMTP, and more.

Despite the proliferation of GUI-based API clients like Postman and Insomnia in recent years, cURL remains a staple in the toolboxes of developers, DevOps engineers, and data scientists around the world. According to the cURL website, the library is used by billions of devices worldwide and is included by default on most Unix-based operating systems, as well as Windows 10 and later.

So why has cURL stood the test of time? For one, its command-line interface makes it easy to use cURL in scripts, pipelines, and automation workflows. Compared to GUI tools, cURL is also much more lightweight and can be run on headless servers or over SSH. And because cURL is open source, it‘s continuously maintained and improved by a community of dedicated developers.

Understanding HTTP POST Requests

Before we dive into the specifics of making POST requests with cURL, let‘s take a step back and look at what HTTP POST requests are and when to use them.

HTTP (Hypertext Transfer Protocol) defines a set of methods (sometimes called "verbs") that clients like web browsers and cURL can use to communicate with servers. The two most common HTTP methods are:

• GET: Retrieve a resource from a server
• POST: Send data to a server to create or update a resource

When you visit a webpage in your browser, your browser sends a GET request to the server to retrieve the HTML, CSS, and JavaScript that makes up the page. But when you fill out a form and click "Submit", your browser sends a POST request to the server with the form data in the request body.

Here are some other common use cases for POST requests:

• Creating a new user account
• Updating a database record
• Uploading a file
• Sending an email
• Making a payment

Essentially, any time you need to send data to a server to change its state or trigger an action, you‘ll likely use a POST request.

POST requests have a few key differences from GET requests:

1. POST requests can have a request body containing arbitrary data, while GET requests can only include parameters in the URL query string.

2. POST requests are not cached or bookmarked by default, while GET requests are.

3. POST requests are not idempotent, meaning that making the same request multiple times can result in different outcomes (like creating duplicate records). GET requests are expected to be idempotent.

Now that we understand the basics of POST requests, let‘s look at how to make them with cURL.

Anatomy of a cURL POST Request

The basic syntax for sending a POST request with cURL is:

curl -X POST -d "param1=value1¶m2=value2" http://example.com/resource

Let‘s break down each part of this command:

• curl: This invokes the cURL program.

• -X POST: The -X flag sets the HTTP method. If not provided, cURL defaults to GET.

• -d: This flag specifies the request body data. It can be a URL-encoded string like key1=value1&key2=value2, or point to a file containing the data with @filename.txt. You can also use –data or –data-binary instead of -d.

• http://example.com/resource: The last argument is the URL to send the request to.

Here‘s a real example that sends a POST request to the JSONPlaceholder API to create a new blog post:

curl -X POST -d "title=Hello World&body=This is my first post&userId=1" https://jsonplaceholder.typicode.com/posts

The API expects title, body, and userId parameters in the request body. If the request succeeds, it will return a response like:

{
"id": 101,
"title": "Hello World",
"body": "This is my first post",
"userId": 1
}

Sending JSON Data

Many APIs expect POST data in JSON format rather than URL-encoded key-value pairs. To send JSON data with cURL, we need to:

1. Set the Content-Type request header to application/json
2. Provide the JSON payload in the request body

Here‘s an example:

curl -X POST -H "Content-Type: application/json" -d ‘{"title":"Hello World","body":"This is my first post","userId":1}‘ https://jsonplaceholder.typicode.com/posts

The -H flag sets a request header. Note that the JSON payload is wrapped in single quotes to avoid issues with the nested double quotes.

When sending JSON arrays, objects with special characters, or large payloads, it‘s best to store the JSON in a separate file and reference it with the @ syntax:

curl -X POST -H "Content-Type: application/json" -d @post.json https://jsonplaceholder.typicode.com/posts

This keeps the cURL command cleaner and allows you to validate and pretty-print the JSON more easily.

Handling Authentication

Many APIs require authentication to prevent unauthorized access. The most common authentication mechanisms are:

• Basic Auth: Send a username and password in the Authorization header
• API key: Send a unique API token in a custom header or query parameter
• OAuth 2.0: Obtain an access token via a multi-step authorization flow

With cURL, you can handle authentication by setting the appropriate headers. For example, to use HTTP Basic Auth:

curl -X POST -u "username:password" -d ‘{"title":"Top Secret Post"}‘ https://api.example.com/posts

The -u flag sets the username and password, which cURL automatically encodes in the Authorization header.

To authenticate with a bearer token or API key header:

curl -X POST -H "Authorization: Bearer abc123" -d ‘{"message":"Hello"}‘ https://api.example.com/chat

curl -X POST -H "X-API-KEY: def456" -d ‘{"query":"cURL"}‘ https://api.example.com/search

For OAuth 2.0 flows that involve redirects and access token exchange, you‘ll likely need to use a separate tool or script to obtain the initial access token, then set it in your cURL requests:

curl -X POST -H "Authorization: Bearer eyJz93a…k4laUWw" -d ‘{"task":"Buy milk"}‘ https://api.example.com/todos

Parsing the Response

By default, cURL outputs the response body directly to stdout (usually your terminal). To save the response to a file instead, use the -o flag:

curl -X POST -d ‘{"name":"Alice"}‘ -o response.json https://api.example.com/users

You can also use the -O flag to save the output to a file named after the remote resource.

To include the response headers in the output, use the -i flag:

curl -i -X POST -d "foo=bar" https://example.com

HTTP/1.1 200 OK
Content-Type: application/json
Server: Apache/2.4.41
{"status":"success","foo":"bar"}

The -I flag prints only the response headers, not the body.

For more control over the output, you can redirect stdout and stderr to separate files:

curl -X POST -d ‘{"id":1}‘ https://api.example.com/items > response.json 2> errors.txt

Advanced Tips and Tricks

Here are a few more advanced tips for working with cURL POST requests:

1. To set a timeout for the request, use the –max-time flag followed by the number of seconds to wait before terminating:

curl –max-time 5 -X POST -d ‘{"query":"cURL"}‘ https://api.example.com/search

2. To follow redirects, use the -L flag:

curl -L -X POST -d "foo=bar" http://example.com/redirect

3. To debug a misbehaving request, use the -v flag to enable verbose output:

curl -v -X POST -d ‘{"name":"Bob"}‘ https://api.example.com/users

This will print detailed information about the request and response headers, TLS handshake, and more.

4. To make the same POST request multiple times, use a for loop in your terminal:

for i in {1..10}; do curl -X POST -d "iteration=$i" https://example.com/count; done

5. To randomize the request payload for load testing or fuzzing, you can generate dynamic data with a tool like jq:

cat users.json | jq -c ‘.[]‘ | while read user; do curl -X POST -H "Content-Type: application/json" -d "$user" https://api.example.com/users; done

This will send a separate POST request for each user object in the users.json file.

Comparing cURL to Other HTTP Clients

While cURL is a powerful and flexible HTTP client, it‘s not the only game in town. Here‘s how cURL compares to some popular alternatives:

Feature	cURL	HTTPie	Postman	Insomnia
Platform	CLI	CLI	GUI	GUI
Protocols	HTTP, HTTPS, FTP, SMTP, etc.	HTTP, HTTPS	HTTP, HTTPS, WebSocket	HTTP, HTTPS, gRPC
Output Formats	Raw, JSON, XML	Colored, JSON	JSON, XML, HTML	JSON, XML, YAML
Scripting	Shell, Python, etc.	Python	JavaScript	JavaScript
Auth	Basic, Digest, OAuth1/2	Basic, Digest, OAuth1/2	Basic, Digest, OAuth1/2, NTLM	Basic, Bearer, OAuth2
Pricing	Free	Free	Free (limited) / Paid	Free (limited) / Paid

In general, cURL is best suited for automation and scripting, while GUI clients like Postman are more user-friendly for ad-hoc API testing and exploration. HTTPie is a good middle ground, offering a more intuitive command-line syntax than cURL but without the overhead of a graphical app.

Which tool you choose depends on your specific needs and preferences. For quick and dirty API requests, cURL is hard to beat. But for collaboration, documentation, and complex request chaining, a tool like Postman or Insomnia may be a better fit.

Real-World Examples

To wrap up, let‘s look at a few real-world examples of how cURL POST requests are used by developers and data professionals.

1. Scraping product data from an e-commerce site:

curl -X POST -d "search=iPhone" -d "category=Electronics" -H "X-API-TOKEN: abc123" https://api.shop.com/products

This sends a POST request to a fictional e-commerce API to search for iPhone products in the Electronics category, authenticated with an API token header. The response could be parsed and stored in a database for analysis.

2. Triggering a build in a CI/CD pipeline:

curl -X POST -H "Content-Type: application/json" -d ‘{"branch":"main"}‘ https://api.travis-ci.com/repo/myorg%2Fmyrepo/requests

This sends a POST request to the Travis CI API to start a new build for a specific Git branch. The %2F in the URL is an encoded forward slash.

3. Sending a Slack message from a monitoring script:

WEBHOOK_URL="https://hooks.slack.com/services/xxx/yyy/zzz"
MESSAGE="Server down! Incident report: http://status.mycompany.com/incidents/123"

curl -X POST -H ‘Content-type: application/json‘ -d "{\"text\":\"$MESSAGE\"}" $WEBHOOK_URL

This sends an alert message to a Slack channel using an incoming webhook URL. The message text is stored in a variable and expanded in the JSON payload.

Conclusion

We‘ve covered a lot of ground in this guide, from the basics of HTTP POST requests to advanced usage patterns and debugging techniques. Whether you‘re a seasoned API hacker or just getting started with web automation, cURL is an essential tool to have in your belt.

Some key takeaways:

• Use the -X flag to set the HTTP method to POST
• Send URL-encoded data with -d or JSON data with -d and -H "Content-Type: application/json"
• Handle authentication by setting the appropriate headers with -H or -u
• Save responses to a file with -o or -O
• Debug stubborn requests with -v for verbose output

With its flexibility, scriptability, and ubiquity, cURL is a Swiss Army knife for the modern web stack. By mastering cURL, you‘ll be able to integrate with any HTTP-based API, automate complex workflows, and diagnose network issues with ease.

As you dive deeper into the world of web APIs and automation, keep exploring cURL‘s many other features and options, such as multi-part form requests, cookies, proxies, and more. And don‘t be afraid to mix and match cURL with other languages and tools to build robust and resilient pipelines.

Happy posting!