In an effort to reduce 'Initial Server Response Time' and so have a better Google PageSpeed Insights, I've been trying to optimize that 4.5Kb request's response time which takes around 270ms TTFB and 0.71ms content download (measured using dev tools).
The app is hosted on a Linode in India which is physically near. I turned on logs on Nginx as I was suspecting something was wrong with it but it shows a total response time of 25ms.
Given that Nginx defines the total response time as 'Full request time, starting when NGINX reads the first byte from the client and ending when NGINX sends the last byte of the response body', I expected that ultimately the user would get the response in a little more than 25ms but never 10x that.
Any ideas what I could be missing here? What else can I look at?
UPDATE: I have made the decision to migrate my Linode to Singapore from Mumbai and the results are far better now, I moved from 270ms TTFB to ~100ms. Lesson learned, even though India is close, Singapore's fast internet speed makes it a more suitable place to host my app in.
From nginx logging docs
$request_time – Full request time, starting when NGINX reads the first byte from the client and ending when NGINX sends the last byte of the response body
...NGINX sends the last byte...
Meaning it has sent the last byte to the underlying OS. So TCP socket buffers might have stored the bytes and are trying to send them to the client.
Here is an analysis of this scenario.
Nginx does not care about the RTT (Round Trip Time) between the client and the server. That's an OS/client problem.
Pinging the server from the client could give you an idea of the order of response time. If ping time is greater than nginx's $response_time, performance can't be expected to be close to $request_time.
ping -c3 -s 1450 www.kernel.org
PING ord.git.kernel.org (147.75.58.133) 1450(1478) bytes of data.
1458 bytes from ord1.git.kernel.org (147.75.58.133): icmp_seq=1 ttl=48 time=191 ms
1458 bytes from ord1.git.kernel.org (147.75.58.133): icmp_seq=2 ttl=48 time=192 ms
1458 bytes from ord1.git.kernel.org (147.75.58.133): icmp_seq=3 ttl=48 time=198 ms
--- ord.git.kernel.org ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 191.155/194.026/198.468/3.205 ms
As a ball park approach, if your response size is 4.5kB and max TCP packet size is ~ 1.5kB, you could expect total time to be at best, 3 times the ping time.
On a Linux box the maximum transmission unit (MTU) is 1500:
ip addr | grep 'eth0: .*mtu'
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
DNS resolution might have an influence.