Using cURL for isolation of Web related issues

Windows Server · January 24

Why Use cURL?

Here are some scenarios where cURL can be particularly helpful:

Lack of a Latest Browser for Local Testing: When customers do not have access to the latest browser versions, cURL can act as a lightweight alternative.
No GUI Environments: Ideal for environments like Azure App Services (PAAS Servers) or Linux servers where GUI-based testing isn’t possible.
Client Isolation: Browsers often behave differently due to enforced policies or extensions, making it harder to isolate issues. cURL provides a clean, policy-free testing environment.
Detailed Diagnostics: It offers a detailed view of request-response cycles, including headers, status codes, and payloads.
Customized Requests: Supports various HTTP verbs, custom headers, and even protocols beyond HTTP, allowing more flexible testing.

The syntax of cURL commands and options may change in the future. Always refer to the cURL documentation for the most up-to-date information.

Curl is Available by Default

Most modern operating systems include cURL by default. On Windows, use 'cmd' since in powershell curl is added as alias for Invoke-WebRequest (Microsoft.PowerShell.Utility) - PowerShell | Microsoft Learn

curl --version curl 8.10.1 (Windows) libcurl/8.10.1 Schannel zlib/1.3 WinIDN Release-Date: 2024-09-18 Protocols: dict file ftp ftps http https imap imaps ipfs ipns mqtt pop3 pop3s smb smbs smtp smtps telnet tftp Features: alt-svc AsynchDNS HSTS HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM SPNEGO SSL SSPI threadsafe Unicode UnixSockets

A work-around is to invoke curl as 'curl.exe' to prevent PowerShell from treating it as an alias.

Verbose Mode (-v)

Verbose mode provides a detailed breakdown of the request-response cycle. This can be invaluable for identifying where a request fails. Some details include:

DNS resolution
TCP and TLS details
HTTP Request/Response Headers and Body
Protocol specific detailed messages

curl https://google.com -v * Host google.com:443 was resolved. * IPv6: (none) * IPv4: X.X.X.X * Trying X.X.X.X:443... * schannel: disabled automatic use of client certificate * ALPN: curl offers http/1.1 * ALPN: server accepted http/1.1 * Connected to google.com (X.X.X.X) port 443 * using HTTP/1.x > GET / HTTP/1.1 > Host: google.com > User-Agent: curl/8.10.1 > Accept: */* > * Request completely sent off * schannel: remote party requests renegotiation * schannel: renegotiating SSL/TLS connection * schannel: SSL/TLS connection renegotiated < HTTP/1.1 301 Moved Permanently < Location: https://www.google.com/ < Content-Type: text/html; charset=UTF-8 <trucated> < X-XSS-Protection: 0 < X-Frame-Options: SAMEORIGIN < Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 < <HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8"> <TITLE>301 Moved</TITLE></HEAD><BODY> <H1>301 Moved</H1> The document has moved <A HREF="https://www.google.com/">here</A>. </BODY></HTML> * Connection #0 to host google.com left intact

Sometimes the request doesn't even make it to the intended server and the output of curl's verbose mode clearly indicate the step at which failed (like DNS, TCP or TLS).

Below is an example of DNS failure:

curl http://vm9 -v * Could not resolve host: vm9 * shutting down connection #0 curl: (6) Could not resolve host: vm9

Below is an example of TCP connection failure (Blocked by Windows Firewall):

curl http://vm2:8088 -v * Host vm2:8088 was resolved. * IPv6: (none) * IPv4: X.X.X.X * Trying X.X.X.X:8088... * connect to X.X.X.X port 8088 from 0.0.0.0 port 50020 failed: Timed out * Failed to connect to vm2 port 8088 after 21052 ms: Could not connect to server * closing connection #0 curl: (28) Failed to connect to vm2 port 8088 after 21052 ms: Could not connect to server

Below is an example of HTTP Request Parsing. The server is supposed to add CRLF between HTTP response headers and body - RFC 9112: HTTP/1.1. Since it is not present, the clients assume that it's another header which violate the syntax of key:value (header without colon) and close the connection. The browsers do not really show detailed information without browser traces.

curl http://localhost:8548/ -v * Host localhost:8548 was resolved. * IPv6: ::1 * IPv4: 127.0.0.1 * Trying [::1]:8548... * Trying 127.0.0.1:8548... * Connected to localhost (127.0.0.1) port 8548 * using HTTP/1.x > GET / HTTP/1.1 > Host: localhost:8548 > User-Agent: curl/8.10.1 > Accept: */* > < HTTP/1.1 200 OK < Server: Microsoft-IIS/10.0 < Date: Wed, 31 Jul 2024 17:44:50 GMT < Connection: close < Pragma: cache < Content-type: text/html < <HTML xmlns:v="urn:schemas-microsoft-com:vml"> * Header without colon * closing connection #0 curl: (8) Header without colon

SSL/TLS Troubleshooting - Part 1

The verbose output includes SSL handshake details, such as server certificate verification, trust chain, SAN, and revocation status.

curl https://signalr.alterego.com -v * Host signalr.alterego.com:443 was resolved. * IPv6: (none) * IPv4: 127.0.0.1 * Trying 127.0.0.1:443... * schannel: disabled automatic use of client certificate * ALPN: curl offers http/1.1 * schannel: SNI or certificate check failed: SEC_E_WRONG_PRINCIPAL (0x80090322) - The target principal name is incorrect. * closing connection #0 curl: (60) schannel: SNI or certificate check failed: SEC_E_WRONG_PRINCIPAL (0x80090322) - The target principal name is incorrect. More details here: https://curl.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the webpage mentioned above.

In this example the Server Certificate SAN is not matching the hostname.

SSL/TLS Troubleshooting - Part 2

Curl also allows to print variables from SSL Handshake like the certificate payload. On Windows, the certificate blob can be saved into a .cer file to inspect the attributes of certificate (like Subject Alternative Name Extension). You can other options to review the certificate blob like an online tool.

-k will disable certificate validation checks.
-w will write out variables - [available variables]

curl https://signalr.alterego.com -k -v -w "%{certs}" * Host signalr.alterego.com:443 was resolved. * IPv6: (none) * IPv4: 127.0.0.1 * Trying 127.0.0.1:443... <trucated> * Subject: CN=localhost * Issuer: CN=localhost * Version: 2 * Serial Number: 3c:92:85:35:af:eb:c4:84:47:77:a9:6e:83:88:f7:14: * Signature Algorithm: sha256WithRSAEncryption * Start Date: 2022-08-05 15:25:44 GMT * Expire Date: 2027-08-05 00:00:00 GMT * Public Key Algorithm: rsaEncryption * RSA Public Key (2048 bits) * rsa(n): cb:aa:6d:03:4e:07:b6:b8:cf:f7:e<trucated> * rsa(e): 0x10001 * Signature: 7a:a6:c6:17:5f:93:34:bd:26<trucated> * Cert: -----BEGIN CERTIFICATE----- MIIC7DCCAdSgAwIBAgIQPJKFNa/rxIRHd6lug4j3FDANBgkqhkiG9w0BAQsFADAU MRIwEAYDVQQDEwlsb/<truncated> -----END CERTIFICATE-----

SSL/TLS Troubleshooting - Part 3

We can also set with explicit TLS versions while making the request. This can help to understand the behavior of sever against different TLS versions. For example below, the server has TLS 1.1 disabled.

curl https://vm2.alterego.cc --tls-max 1.1 -v * Host vm2.alterego.cc:443 was resolved. * IPv6: (none) <truncated> * schannel: disabled automatic use of client certificate * ALPN: curl offers http/1.1 * Recv failure: Connection was reset * schannel: failed to receive handshake, SSL/TLS connection failed * closing connection #0 curl: (35) Recv failure: Connection was reset

More TLS related options, you can use "curl --help tls".

-1, --tlsv1 TLSv1.0 or greater --tlsv1.0 TLSv1.0 or greater --tlsv1.1 TLSv1.1 or greater --tlsv1.2 TLSv1.2 or greater --tlsv1.3 TLSv1.3 or greater --tls-max <VERSION> Maximum allowed TLS version

Proxies

Proxies/Intermediate devices can introduce unexpected behavior in responses. cURL helps identify and isolate proxy-related issues.

-w "%{certs}" - will can show certificates and we can see if it is from the actual server/proxy

```
--noproxy '*' - will disable proxy
```

With Proxy

Curl provides great indication if the proxies are involved (CONNECT Request details) and using the certificate dump we can also identify if the server certificate was provided by actual destination vs a TLS terminating device like proxy/WAF (notice the certificate subject which clearly indicates the certificate was provided by Proxy device).

curl -v -k https://google.com -w "%{certs}" * Trying 127.0.0.1:8888... * CONNECT tunnel: HTTP/1.1 negotiated * allocate connect buffer * Establish HTTP proxy tunnel to google.com:443 > CONNECT google.com:443 HTTP/1.1 > Host: google.com:443 > User-Agent: curl/8.10.1 > Proxy-Connection: Keep-Alive > < HTTP/1.1 200 Connection Established < FiddlerGateway: Direct < StartTime: 02:12:03.172 < Connection: close < * CONNECT phase completed * CONNECT tunnel established, response 200 * schannel: disabled automatic use of client certificate * ALPN: curl offers http/1.1 * ALPN: server did not agree on a protocol. Uses default. * Subject: OU=Created by http://www.fiddler2.com, O=DO_NOT_TRUST, CN=google.com * Issuer: OU=Created by http://www.fiddler2.com, O=DO_NOT_TRUST, CN=DO_NOT_TRUST_FiddlerRoot * Version: 2 * Serial Number: 6c:18:7b:e5:fa:d0:36:89:4a:3f:50:a1:bd:8e:80:6e: * Signature Algorithm: sha256WithRSAEncryption

Without Proxy

Even though a proxy might be enforced via OS settings or variables like "http/s_proxy", the using --noproxy '*' bypasses the proxy connection.

curl --noproxy '*' -v -k https://google.com -w "%{certs}" * Host google.com:443 was resolved. <trucated> * schannel: renegotiating SSL/TLS connection * Subject: CN=*.google.com * Issuer: C=US, O=Google Trust Services, CN=WR2 * Version: 2 * Serial Number: 00:fc:12:5b:bd:5b:36:ea:6b:12:3b:0a:55:49:88:4c:9f: * Signature Algorithm: sha256WithRSAEncryption * Start Date: 2024-12-02 08:35:57 GMT * Expire Date: 2025-02-24 08:35:56 GMT

Use different HTTP Verbs, Headers and Send Body

Customize HTTP

Sometimes you might want to use other HTTP verbs or customize headers. For example, testing how the server behaves for a CORS preflight request

In below examples. the "google.com" origin is not allowed whereas bing.com is allowed.

-X specifies HTTP verb
-H allows to specify different headers
-d allows to specify HTTP Body

curl -X OPTIONS http://localhost:8334 -H "Origin: https://bing.com" -H "Access-Control-Request-Method: GET" -H "Access-Control-Request-Headers: Content-Type" -v * Host localhost:8334 was resolved. <truncated> > Host: localhost:8334 > User-Agent: curl/8.10.1 > Accept: */* > Origin: https://bing.com > Access-Control-Request-Method: GET > Access-Control-Request-Headers: Content-Type > < HTTP/1.1 204 No Content < Vary: Origin < Server: Microsoft-IIS/10.0 < Access-Control-Allow-Origin: https://bing.com < X-Powered-By: ASP.NET < Date: Wed, 22 Jan 2025 07:55:11 GMT

curl -X OPTIONS http://localhost:8334 -H "Origin: https://google.com" -H "Access-Control-Request-Method: GET" -H "Access-Control-Request-Headers: Content-Type" -v * Host localhost:8334 was resolved. <truncated> > Accept: */* > Origin: https://google.com > Access-Control-Request-Method: GET > Access-Control-Request-Headers: Content-Type > < HTTP/1.1 204 No Content < Vary: Origin < Server: Microsoft-IIS/10.0 < X-Powered-By: ASP.NET < Date: Wed, 22 Jan 2025 07:55:30 GMT < * Connection #0 to host localhost left intact

Below is an example to get a token using OAuth 2.0 client credentials flow on the Microsoft identity platform - Microsoft identity platform | Microsoft Learn

Note - Windows cmd uses '^' for continuation of commands and other shells might have different continuation symbols.

curl -X POST https://login.microsoftonline.com/{tenantid}/oauth2/v2.0/token ^ -H "Host: login.microsoftonline.com" ^ -H "Content-Type: application/x-www-form-urlencoded" ^ -d "grant_type=client_credentials&client_id=MICROSOFT-APP-ID&client_secret=MICROSOFT-APP-PASSWORD&scope=https%3A%2F%2Fgraph.microsoft.com%2F.default"

Authentication

Curl also supports Authentication Mechanisms like NTLM, Kerberos, Basic etc.

Authentication in Curl

curl http://vm2:8133 -v --negotiate * Host vm2:8133 was resolved. * IPv6: (none) * IPv4: 10.160.56.6 * Trying 10.160.56.6:8133... * Connected to vm2 (10.160.56.6) port 8133 * Server auth using Negotiate with user '' > GET / HTTP/1.1 > Host: vm2:8133 > Authorization: Negotiate YIIH <truncated> < HTTP/1.1 200 OK < Content-Type: text/html < Last-Modified: Mon, 28 Mar 2022 15:53:03 GMT < Accept-Ranges: bytes < ETag: "4045c2e8bb42d81:0" < Server: Microsoft-IIS/10.0 < WWW-Authenticate: Negotiate oYG1 <truncated>

Load Test

You might a find a need to make multiple requests reproduce some problem. With the use of curl's range convention [start-end] together with writeout it can print detailed results of the test.

curl -s -w "\nRequest: %{url_effective} Total Time: %{time_total}s HTTP Code: %{http_code}\n" "http://google.com/?test=[1-100]" -o nul Request: http://google.com/?test=1 Total Time: 0.069207s HTTP Code: 301 Request: http://google.com/?test=2 Total Time: 0.748296s HTTP Code: 301 Request: http://google.com/?test=3 Total Time: 0.047916s HTTP Code: 301 Request: http://google.com/?test=4 Total Time: 0.153930s HTTP Code: 301 Request: http://google.com/?test=5 Total Time: 0.053448s HTTP Code: 301

Note that the calls are sequential, if you wish to run concurrent requests, open curl multiple sessions.

FTP

Curl has great support for FTP including below:

Active and Passive Mode
Implicit and Explicit FTPs
File download and directory listing.

Command line FTP - everything curl

'-v' shows detailed command exchange and diagnosis.
Use -u for specifying users - Command line FTP - everything curl. By default, uses an anonymous user.
The direction of brackets ('>' and '<' indicate whether the command/response is sent by client of server respectively).
Most connectivity issues in FTP are usually related to data connection (Two connections - everything curl). The curl commands shown below will trigger a need of Data connection to show directory listing.

Active Mode:

[--ftp-port] - Enables ACTIVE Mode. Provides options to specify the address (- (hyphen) same as control channel)
[--disable-eprt] - Disables IpV6

curl ftp://<servername>:<port> -v -p -

curl ftp://localhost:21 -v -P - * Host localhost:21 was resolved. * IPv6: ::1 * IPv4: 127.0.0.1 * Trying [::1]:21... * Connected to localhost (::1) port 21 < 220 Microsoft FTP Service > USER anonymous < 331 Anonymous access allowed, send identity (e-mail name) as password. > PASS ftp@example.com < 230 User logged in. > PWD < 257 "/" is current directory. * Entry path is '/' * Request has same path as previous transfer > EPRT |2|::1|54989| * [FTP] [PORT] perform, DATA connection established < 200 EPRT command successful. * Connect data stream actively > TYPE A < 200 Type set to A. > LIST < 125 Data connection already open; Transfer starting. * Maxdownload = -1 * Preparing for accepting server on data port * Checking for server connect * Ready to accept data connection from server * Connection accepted from server 10-26-24 12:32PM 696 iisstart.htm * abort upload * Remembering we are in dir "" < 226 Transfer complete. * Connection #0 to host localhost left intact

Passive Mode

* [--disable-epsv]- Disables Ipv6
* Passive mode is enabled by default

curl ftp://<servername>:<port> -v

curl ftp://localhost:21 -v * Host localhost:21 was resolved. * IPv6: ::1 * IPv4: 127.0.0.1 * Trying [::1]:21... * Connected to localhost (::1) port 21 < 220 Microsoft FTP Service > USER anonymous < 331 Anonymous access allowed, send identity (e-mail name) as password. > PASS ftp@example.com < 230 User logged in. > PWD < 257 "/" is current directory. * Entry path is '/' * Request has same path as previous transfer > EPSV * Connect data stream passively < 229 Entering Extended Passive Mode (|||59401|) * Connecting to ::1 (::1) port 59401 * Trying [::1]:59401... * Connected 2nd connection to ::1 port 59401 > TYPE A < 200 Type set to A. > LIST < 125 Data connection already open; Transfer starting. * Maxdownload = -1 10-26-24 12:32PM 696 iisstart.htm

FTPs

FTPs or FTP over TLS is a secure FTP protocol which is available in IIS. This is not be confused with SFTP which uses SSH for FTP.

FTP Clients - Part 2: Explicit FTPS versus Implicit FTPS | Microsoft Learn

Implicit FTPs (Control port 990):
- ```
  curl --ssl-reqd ftps://<server name>/ -u anonymous:anonymous
```
- MS-FTPS: Control Connection Negotiation with Implicit FTPS | Microsoft Learn
Explicit FTPs (any other control port than 990 with SSL configured)
- FTPS - Everything curl
- curl --ssl-reqd ftp://<servername>:port -u anonymous:anonymous

Final Thoughts

cURL is an incredibly versatile tool that can simplify and speed up the isolation of web-related issues. Whether you are debugging SSL/TLS connections, testing custom HTTP requests, or bypassing proxies, FTP tests, cURL provides the flexibility and precision needed for effective troubleshooting. Next time you’re faced with a challenging web application issue, consider reaching for cURL to peel back the layers and uncover the root cause.

View the full article

Sign In