Essential Network Access Tools for Linux: A Comprehensive Guide

Preface

Tools for testing network access in Linux: A variety of network access tools are available for testing connectivity, troubleshooting, and monitoring network performance on Linux systems.

title:      Detailed Explanation of the Common curl Tool for Testing in CDN
author:     sheazhang

1. Verify IP Cache Status of a Node

[root@VM_27_201_centos ~]# curl -I ‘http://app613.imgcache.xxx.com/app613/frontend_swf/version.js’ -x 106.74.23.144:80 HTTP/1.1 200 OK Server: NWS_TCloud_S1 Connection: keep-alive Date: Sun, 03 Jul 2016 08:50:34 GMT Cache-Control: max-age=6000 Expires: Sun, 03 Jul 2016 10:30:34 GMT Last-Modified: Thu, 30 Jun 2016 09:12:30 GMT Content-Type: application/x-javascript; charset=utf-8 Content-Length: 69 X-Cache-Lookup: Hit From Disktank

Note: The usage of -x to bind a proxy server IP is similar to that in wget.

# wget -e http-proxy=119.97.137.178 http://sdxxx.cloudcdn.net/test008/6wireshark-win64-1.2.9.exe

get back

2. Verify if Referer Header Whitelist is Effective

Scenario

Use -H to add custom headers to simulate whether the origin site/CDN node supports certain features under specific conditions. The result returns whether it meets expectations. The example below is used to verify whether it meets expectations after adding a whitelist for the referer in the console.

[root@VM_27_201_centos ~]# curl -I ‘http://res2.xxxxx.com/’ -x 106.74.23.144:80 -H”referer:http://*.yqh5.cn” HTTP/1.1 200 OK Server: NWS_TCloud_S1 Connection: keep-alive Date: Sun, 03 Jul 2016 08:59:50 GMT Cache-Control: max-age=31536000 Expires: Mon, 03 Jul 2017 08:59:50 GMT Last-Modified: Tue, 12 Apr 2016 01:59:32 GMT Content-Type: text/html Content-Length: 612 X-Cache-Lookup: Hit From Disktank Access-Control-Allow-Origin: * Content-Disposition: inline; filename=”” Accept-Ranges: bytes

Note above

curl -I 'http://res2.eqxxx.com/' -x 106.74.23.144:80 -H"referer:http://*.yqh5.cn"

can also be written as:

curl -I 'http://res2.exxx.com/' -x 106.74.23.144:80  -e "http://pachong.org/"  

get back

3. Verify the Support of Nodes/Origin Sites for Fragmentation

[root@VM_27_201_centos ~]# curl -Iv 'http://update01.xxxx-cloud.com/pup/8S70_E6200_V016.002.230_9.zip' -x 106.74.23.144:80 -H'range:bytes=0-1000'
* About to connect() to proxy 106.74.23.144 port 80 (#0)
*   Trying 106.74.23.144... connected
* Connected to 106.74.23.144 (106.74.23.144) port 80 (#0)
> HEAD http://update01.skyworth-cloud.com/pup/8S70_E6200_V016.002.230_9.zip HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: update01.skyworth-cloud.com
> Accept: */*
> Proxy-Connection: Keep-Alive
> range:bytes=0-1000
>
< HTTP/1.1 206 Partial Content
HTTP/1.1 206 Partial Content
< Server: nws_ocmid_hy
Server: nws_ocmid_hy
< Connection: close
Connection: close
< Date: Sun, 03 Jul 2016 09:02:24 GMT
Date: Sun, 03 Jul 2016 09:02:24 GMT
< Cache-Control: max-age=600
Cache-Control: max-age=600
< Expires: Sun, 03 Jul 2016 09:12:24 GMT
Expires: Sun, 03 Jul 2016 09:12:24 GMT
< Last-Modified: Mon, 23 May 2016 20:10:06 GMT
Last-Modified: Mon, 23 May 2016 20:10:06 GMT
< Content-Range: bytes 0-1000/437405243
Content-Range: bytes 0-1000/437405243
< Content-Type: application/octet-stream
Content-Type: application/octet-stream
< Content-Length: 1001
Content-Length: 1001
< X-Daa-Tunnel: hop_count=2
X-Daa-Tunnel: hop_count=2
< X-Cache-Lookup: Hit From Disktank Upstream
X-Cache-Lookup: Hit From Disktank Upstream
< X-Cache-Lookup: Hit From Disktank
X-Cache-Lookup: Hit From Disktank
< X-Daa-Tunnel: hop_count=2
X-Daa-Tunnel: hop_count=2
< X-Cache-Lookup: Hit From Inner Cluster
X-Cache-Lookup: Hit From Inner Cluster
 
<
* Closing connection #0

Note

The above

curl -Iv 'http://update01.xxx-cloud.com/pup/8S70_E6200_V016.002.230_9.zip' -x 106.74.23.144:80 -H'range:bytes=0-1000'

can also be written as

curl -Iv 'http://update01.xxx-cloud.com/pup/8S70_E6200_V016.002.230_9.zip' -x 106.74.23.144:80 -r 0-1000

get back

4. Verify Consistency of File Size on Nodes and Origin Site (MD5 Check)

[root@VM_27_201_centos ~]# curl -o pic1 'http://as.xxx.com/show/visitor/4a2d8aae508506ef0150a6fcebbf2f53' -x 106.74.23.144:80
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10126  100 10126    0     0  40629      0 --:--:-- --:--:-- --:--:-- 54149
[root@VM_27_201_centos ~]#
[root@VM_27_201_centos ~]# md5sum pic1
4b7671dbe1a634541726bd9473d14dcc  pic1
[root@VM_27_201_centos ~]#
[root@VM_27_201_centos ~]#
[root@VM_27_201_centos ~]# curl -o pic2 'http://as.xxx.com/show/visitor/4a2d8aae508506ef0150a6fcebbf2f53' -x 115.159.20.55:80
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10126  100 10126    0     0  4629k      0 --:--:-- --:--:-- --:--:-- 9888k
[root@VM_27_201_centos ~]#
[root@VM_27_201_centos ~]# md5sum pic2
4b7671dbe1a634541726bd9473d14dcc  pic2

Note

The above -o indicates downloading the file to a specified path, -x indicates using a specified IP or domain as a proxy.

get back

5. Simulate HTTPS Access to Verify HTTPS Certificate

[root@VM_117_214_centos ~]#  curl -sv 'https://xxx.qcloud.com/static/img/57c45bb3997a277e5996dcf13314b365/user-40.png' -o /dev/null -k
* About to connect() to mccdn.qcloud.com port 443 (#0)
*   Trying 113.207.16.63... connected
* Connected to mccdn.qcloud.com (113.207.16.63) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* warning: ignoring value of ssl.verifyhost
* skipping SSL peer certificate verification
* SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
*     subject: CN=*.qcloud.com,OU=R&D,O=Shenzhen Tencent Computer Systems Company Limited,L=shenzhen,ST=guangdong,C=CN
*     start date: May 25 00:00:00 2016 GMT
*     expire date: Jul 24 23:59:59 2019 GMT
*     common name: *.qcloud.com
*     issuer: CN=Symantec Class 3 Secure Server CA - G4,OU=Symantec Trust Network,O=Symantec Corporation,C=US
> GET /static/img/57c45bb3997a277e5996dcf13314b365/user-40.png HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: mccdn.qcloud.com
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nnws/1.7.3.8
< Date: Sat, 20 Aug 2016 12:43:55 GMT
< Content-Type: image/png
< Content-Length: 3558
< Connection: keep-alive
< Cache-Control: max-age=2592000
< Expires: Mon, 19 Sep 2016 12:43:54 GMT
< Last-Modified: Wed, 25 May 2016 08:06:29 GMT
< X-Cache-Lookup: Hit From Disktank
< 
{ [data not shown]
* Connection #0 to host mccdn.qcloud.com left intact
* Closing connection #0

get back

6. Bind Host in curl to Test Method

Equivalent to binding host in windows for testing

[root@VM_230113centos ~]# curl -I -H 'host:7u2q8y.com3.z0.glb.xxxcdn.com' http://xxx.com.z0.glb.xxxdns.com/M-_WoLgVqduR7VFfrhnmWwL8YgA=/luI-
 
lideKDodDAYi55H6D2HV6HcO/000000.ts
HTTP/1.1 200 OK
Server: nginx/1.4.4
Date: Sat, 02 Jan 2016 11:39:23 GMT
Content-Type: video/mp2t
Content-Length: 673416
Connection: keep-alive
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Access-Control-Max-Age: 2592000
Cache-Control: public, max-age=31536000
Content-Disposition: inline; filename="000000.ts"
Content-Transfer-Encoding: binary
Etag: "Fs–ekImTQ2ZMjO_MdTeTN0Yew_5"
Last-Modified: Sun, 22 Nov 2015 06:28:55 GMT
X-Log: mc.g;IO:1
X-Reqid: iToAAMIdW1cTmSUU
X-Qiniu-Zone: 0

[root@VM_230113centos ~]# curl -I -H 'host:7u2q8y.com3.z0.glb.xxxcdn.com' http://211.161.127.147/M-_WoLgVqduR7VFfrhnmWwL8YgA=/luI-lideKDodDAYi55H6D2HV6HcO/000000.ts
HTTP/1.1 200 OK
Server: NWS_Appimg_HY
Connection: keep-alive
Date: Sat, 02 Jan 2016 11:45:50 GMT
Cache-Control: max-age=31536000
Expires: Sun, 01 Jan 2017 11:45:50 GMT
Last-Modified: Sun, 22 Nov 2015 06:28:55 GMT
Content-Type: video/mp2t
Content-Length: 673416
X-Cache-Lookup: Hit From Disktank
Access-Control-Allow-Origin: *
Accept-Ranges: bytes
Etag: "Fs–ekImTQ2ZMjO_MdTeTN0Yew_5"
X-ReqId: ATUAABGu8oBVjyUU
Content-Disposition: inline; filename="000000.ts" 

Among them:

• 7u2q8y.com3.z0.glb.qiniucdn.com is the user’s domain name
• src.com.z0.glb.qiniudns.com is the origin station or the reorigin host address, it could also be in the form of IP address
• 211.161.127.147 is the CDN node IP address parsed to after the domain name is accessed through CDN

By verifying the curl feedback on the origin site URL and CDN node IP address, it can be determined whether it is the origin site or the CDN node having an issue

get back

7. Verify Delay of TCP Handshake and SSL Handshake

curl -w "TCP handshake: %{time_connect}, SSL handshake: %{time_appconnect}\n" -so /dev/null https://www.alipay.com

8. Use curl to Capture Return Time

Evaluate the time consumption for accessing a webpage

curl -o /dev/null -w "Connect: %{time_connect} TTFB: %{time_starttransfer} Total time: %{time_total} \n" http://www.itts-union.com
This command will output this:
[root@VM_117_214_centos ~]#  curl -o /dev/null -w "Connect: %{time_connect} TTFB: %{time_starttransfer} Total time: %{time_total} \n" http://www.itts-union.com/1326.html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 74470    0 74470    0     0  52368      0 --:--:--  0:00:01 --:--:--  116k
Connect: 0.799 TTFB: 1.309 Total time: 1.422

get back

9. Query the Time Consumption at Each Stage:

-w can specify special file and also specify time **%{time_namelookup}**

[root@VM_49_231_centos ~]# curl -Iv "http://img.yxp01.com/
images/20/2017/07/VXunsmRXXU355lC5lM8xc5sXmFNfus.jpg?v=4" -x 
119.6.229.37:80 -so /dev/null -w "@curl-time"

Special file:

[root@VM_49_231_centos ~]# cat curl-time 
    Time-DNS-lookup:  %{time_namelookup}s\n
   Time-TCP-connect:  %{time_connect}s\n
   Time-SSL-connect:  %{time_appconnect}s\n
      Time-redirect:  %{time_redirect}s\n
  Time-Pre-transfer:  %{time_pretransfer}s\n
Time-First-transfer:  %{time_starttransfer}s\n
                    ----------\n
         Time-total:  %{time_total}s\n

get back

10. Use -L to Implement 301 or 302 Redirection

11. Cache File Establishment Time

Pragma: X-Get-Last-Update-Info

get back

12. View the Link of CDN Node to Origin

x_upstream_info=on

curl -o 100 "http://res.hoisin.xxxtv.com/video/20150731164143_960.ts?x_upstream_info=on" -x 61.140.13.201:80 -H "Pragma: X-Get-Last-Update-Info" -v

[root@sheazhang ~]# curl -o /dev/null -v "http://zl.winyp.cn/test.jpg?x_upstream_info=on"
> GET /test.jpg?x_upstream_info=on HTTP/1.1
> User-Agent: curl/7.29.0
> Host: zl.winyp.cn
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: NWS_TCloud_S1
< Connection: keep-alive
< Date: Tue, 01 Jan 2019 02:46:29 GMT
< Cache-Control: max-age=600
< Expires: Tue, 01 Jan 2019 02:56:29 GMT
< Last-Modified: Thu, 22 Jun 2017 18:27:09 GMT
< Content-Type: image/jpeg
< Content-Length: 201557
< X-NWS-LOG-UUID: 4243021142118093756 4c4df2ad367f7a7282b8c634e5bb5346
< Access-Control-Allow-Origin: *
< X-Daa-Tunnel: hop_count=3
< X-Cache-Lookup: Hit From Upstream 125.39.46.83, cost 33 ms
< X-Upstream-Info: l2.cloudhy.tcdn.qq.com
< X-Cache-Lookup: Hit From Disktank3
< Accept-Ranges: bytes
< X-Cache-Lookup: Hit From Inner Cluster

13. Specify User-Agent for Accessing Client Source

-A ‘ua’

[root@sheaBJ ~]# curl -Iv "http://www.xxx.com/ckkhj/index.shtml" -x 1.xxx:80 -A 'Smartphone'
* About to connect() to proxy 1.180.204.168 port 80 (#0)
*   Trying 1.180.204.168...
* Connected to 1.180.204.168 (1.180.204.168) port 80 (#0)
> HEAD http://www.dongao.com/ckkhj/index.shtml HTTP/1.1
> User-Agent: Smartphone
> Host: www.dongao.com
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 301 Moved Permanently
HTTP/1.1 301 Moved Permanently
< Server: NWS_TCloud_S1
Server: NWS_TCloud_S1
< Connection: keep-alive
Connection: keep-alive
< Date: Fri, 08 Mar 2019 03:51:38 GMT
Date: Fri, 08 Mar 2019 03:51:38 GMT
< Content-Length: 39
Content-Length: 39
< Location: http://m.dongao.com/ckkhj/index.shtml
Location: http://m.dongao.com/ckkhj/index.shtml

< 
* Excess found in a non pipelined read: excess = 39 url = /ckkhj/index.shtml (zero-length body)
* Connection #0 to host 1.180.204.168 left intact

14. Common curl Header Set

 -H 'Accept-Encoding:gzip'   Test compression
 -L                 Implement 301 or 302 redirect
 -H 'Pragma: X-Get-Last-Update-Info'     Cache file establishment time
 -H 'x_upstream_info=on'			CDN node to origin link
 -H 'host:7u2q8y.com3.z0.glb.qiniucdn.com'   Bind hosts
 -H 'range:bytes=0-1000'  or -r 0-1000   Test fragmentation support
 -H "referer:http://*.yqh5.cn"     Verify referer header
 -d "x=123"  	Simulate post request
 -H "Origin:http://sheazhang.winyp.cn/"  Verify cross-origin source with 'O' uppercase
 -w "@curl-time"				Time consumption at each stage

Special file:

[root@VM_49_231_centos ~]# cat curl-time 
    Time-DNS-lookup:  %{time_namelookup}s\n
   Time-TCP-connect:  %{time_connect}s\n
   Time-SSL-connect:  %{time_appconnect}s\n
      Time-redirect:  %{time_redirect}s\n
  Time-Pre-transfer:  %{time_pretransfer}s\n
Time-First-transfer:  %{time_starttransfer}s\n
                    ----------\n
         Time-total:  %{time_total}s\n

15. httpstat script -time

Refer to git link

16. Test if the Origin Site Outputs 304

-H ‘If-Modified-Since:xxx’

[root@VM_14_42_centos ~]# curl -o 1 -v "http://j.xxx.cc/daobo/js/fabric.js" -x 139.xxx.204:80 -H 'If-Modified-Since: Mon, 23 Apr 2018 03:38:55 GMT'
* About to connect() to proxy 139.199.214.204 port 80 (#0)
*   Trying 139.xxx...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
				 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 139.199.214.204 (139.199.214.204) port 80 (#0)
> GET http://j.xxx.cc/daobo/js/fabric.js HTTP/1.1
> User-Agent: curl/7.29.0
> Host: xxx
> Accept: */*
> Proxy-Connection: Keep-Alive
> If-Modified-Since: Mon, 23 Apr 2018 03:38:55 GMT
>
< HTTP/1.1 304 Not Modified
< Server: nginx/1.14.2
< Date: Mon, 15 Apr 2019 07:40:50 GMT
< Connection: keep-alive
< Last-Modified: Mon, 23 Apr 2018 03:38:55 GMT
< ETag: "5add554f-da835"
< Access-Control-Allow-Origin: *
<
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host 139.199.214.204 left intact

17. Other Uncommon Uses, For Understanding Only

1. Capture Page Content to a File

[[email protected]]# curl -o home.html  http://www.itts-union.com

2. Simulate Form Information, Simulate Login, Save Cookie Information

[[email protected]]# curl -c ./cookie_c.txt -F log=aaaa -F pwd=****** http://www.itts-union.com/wp-login.php

3. Use Cookie File

[[email protected]]# curl -b ./cookie_c.txt  http://www.itts-union.com/wp-admin

Use Cookie Header:

[[email protected]]# curl -o /dev/null -v http://www.itts-union.com/wp-admin -H "Cookie: uid=999;sex=2"

4. Display Download Progress Bar

[root@VM_117_214_centos ~]# curl -o smokeping.tar.gz http://oss.oetiker.ch/smokeping/pub/smokeping-2.6.9.tar.gz -#
######################################################################## 100.0%

5. Download Files via FTP

[[email protected] ~]$ curl -u username:password -O http://www.itts-union.com/demo/curtain/bbstudy_files/style.css
　　% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
　　Dload  Upload   Total   Spent    Left  Speed
　　101  1934  101  1934    0     0   3184      0 –:–:– –:–:– –:–:–  7136

　　 Or use the following method

[[email protected] ~]$ curl -O ftp://username:password@ip:port/demo/curtain/bbstudy_files/style.css
[[email protected] ~]$ curl -O ftp://username:password@ip:port/demo/curtain/bbstudy_files/style.css

6. Upload via FTP

[[email protected] ~]$ curl -T test.sql ftp://username:password@ip:port/demo/curtain/bbstudy_files/

7. If the download webpage shows garbled text, use iconv to convert the encoding:

[root@VM_117_214_centos
 ~]# curl -o smokeping.tar.gz 
http://oss.oetiker.ch/smokeping/pub/smokeping-2.6.9.tar.gz -# | iconv 
-fgb2312
######################################################################## 100.0%

8. Verify HTTPS Access

curl -sv 'https://mccdn.qcloud.com/static/img/57c45bb3997a277e5996dcf13314b365/user-40.png' -o/dev/null --resolve "mccdn.qcloud.com:443:221.204.60.11"

get back

Original Statement: This article is authorized by the author to be published in the Tencent Cloud Developer Community. No reprint is allowed without permission.

If there is any infringement, please contact [email protected] for deletion.

httpcdntcp/ipcommand line toolshttps