This could depend on multiple factors but never trust just one result of such tests. Run it maybe 5 times, during different times of the day and maybe from a couple different locations to get reliable result.

There are multiple factors that might occasionally make the first byte time go up, such as temporary load on the server (e.g. a lot of search engine bots fetching the content), temporary network lag / packet loss, etc.

After running the test myself for the same site, I got "B" for the first byte time (504ms / 400ms suggested) and all tests show green.

If you want to reduce the 104ms, I think you might not be using your time very well, unless this is a site that gets millions of hits every day. This is about 2% of the total front page load time, and there are probably other ways you can squeeze that extra 100ms out more easily if you want.

From the results I can see that you're already using a caching frontend (Varnish), so the reason for the extra 100ms is not probably even in concrete5. I would look into re-configuring the server running Varnish if you want to micro-optimize your site with the extra 100ms.

I did not know testing from different locations would give that big differences. Thank you for your answer.