Env
- Ubuntu 20.04.1 LTS
- go version go1.13.8 linux/amd64
What I wanted to do
I had over 4,000 urls list. Each host was different. And I just wanted to send 4k http requests per second. In other words, I wanted to send 1 req/s to 4,000 hosts simultaneously. I thought it was easy to program…
What I had to care about
ulimit -n
- limit of num of file descriptors for a process
DNS
- got I/O timeout because of this
- consider using client-side DNS cache
- add some delay to each first request or something you can do
Golang net/http
- keep-alive
- enabled by default when using net/http
- make sure that
http.Transport.DisableKeepAlive
is false
http.Transport.MaxIdleConns
- set this to 0 (=unlimited)
http.DefaultTransport.MaxIdleConns
is 100
http.Transport.IdleConnTimeout
- you have to set this value on your own
- do not forget to read all response body and close the body
- or Golang will not reuse the connection and create a new connection
In most cases, you do not have to care about ports for outgoing tcp connection, ref.