Env

  • Ubuntu 20.04.1 LTS
  • go version go1.13.8 linux/amd64

What I wanted to do

I had over 4,000 urls list. Each host was different. And I just wanted to send 4k http requests per second. In other words, I wanted to send 1 req/s to 4,000 hosts simultaneously. I thought it was easy to program…

What I had to care about

ulimit -n

  • limit of num of file descriptors for a process

DNS

  • got I/O timeout because of this
  • consider using client-side DNS cache
  • add some delay to each first request or something you can do

Golang net/http

  • keep-alive
    • enabled by default when using net/http
    • make sure that http.Transport.DisableKeepAlive is false
  • http.Transport.MaxIdleConns
    • set this to 0 (=unlimited)
    • http.DefaultTransport.MaxIdleConns is 100
  • http.Transport.IdleConnTimeout
    • you have to set this value on your own
  • do not forget to read all response body and close the body
    • or Golang will not reuse the connection and create a new connection

In most cases, you do not have to care about ports for outgoing tcp connection, ref.

comments powered by Disqus