Well let's take a look at the traceroute from my residential PC to Hacker News as an example:
Tracing route to news.ycombinator.com [209.216.230.240]
over a maximum of 30 hops:
1 1 ms 1 ms <1 ms 192.168.86.1
2 12 ms 11 ms 6 ms x.x.x.x
3 29 ms 32 ms 24 ms y.y.y.y
4 19 ms 21 ms 22 ms be23.clmkohpe02r.midwest.rr.com [24.33.160.66]
5 28 ms 22 ms 22 ms be27.clmkohpe01r.midwest.rr.com [65.29.1.34]
6 19 ms 23 ms 19 ms bu-ether31-vinnva0510w-bcr00.tbone.rr.com [66.109.6.54]
7 33 ms 31 ms 20 ms ae-13.edge2.Chicago10.Level3.net [4.68.37.137]
8 68 ms 69 ms 69 ms ae-0-11.bar1.SanDiego1.Level3.net [4.69.146.65]
9 71 ms 75 ms 69 ms M5-HOSTING.bar1.SanDiego1.Level3.net [4.16.110.170]
10 69 ms 77 ms 66 ms news.ycombinator.com [209.216.230.240]
Hop 1 is to my router
Hop 2 is my router to my ISP
Hop 3, 4, 5, 6 are all routers within my ISPs network.
Hop 7 is from my ISP into "the internet". Level3 is a major provider of interconnectivity.
Hops 8 and 9 are within Level3's network.
Hop 10 is finally to Hacker News, which indicates HN's "ISP" is Level3 directly.
Only hop 2 is where my "400 Mbps" package limitation is in place.
Only hop 10 is "server-to-their-ISP".
I chose 10-20% as my intuition because, as you see, 10 routers were involved to get my packet from my PC to HN, and only 1 of them is constrained by my internet package.
Interesting that you can traceroute up to the destination. Over the years I have got accustomed that it's not possible any more in most cases. Some router is not sending or dropping the ICMP packets I guess. Not sure for what reason. To save the load? To stop people (competitors) spying on them?
news.ycombinator.com is no exception that it doesn't work:
5 a9k2-49-infra-dc2.dc3.poneytelecom.eu (195.154.1.118) 0.745 ms 195.154.1.188 (195.154.1.188) 0.865 ms a9k2-49-infra-dc2.dc3.poneytelecom.eu (195.154.1.118) 0.720 ms
6 lag-110.ear3.Paris1.Level3.net (212.3.235.197) 0.940 ms ae53.edge7.Paris1.Level3.net (212.3.235.201) 9.700 ms lag-110.ear3.Paris1.Level3.net (212.3.235.197) 0.955 ms
7 ae-0-11.bar1.SanDiego1.Level3.net (4.69.146.65) 139.415 ms 139.085 ms 139.031 ms
8 GIGLINX-INC.bar1.SanDiego1.Level3.net (4.16.105.98) 150.362 ms M5-HOSTING.bar1.SanDiego1.Level3.net (4.16.110.170) 140.286 ms 139.842ms
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
(sorry, somewhat poor formatting caused by typing on the phone?)
Forwarding packets happens in ASICs, generating ICMPs is usually done by the CPU (which has a packet rate of maybe 1/100000th of the ASICs). So the latter is usually either rate-limited or turned off entirely, or you could bring the router to its knees by just spamming packets with too short TTL. It's also not uncommon that people turn it off to reduce external insight into their network, for whatever reason.
Right, but the weird thing was that grand parent got a full trace up to the target and I didn't as usual.
By experimenting a bit I found that when sending UDP packets the traceroute is incomplete, but when sending ICMP also my traceroute is complete. Explains how the grandparent could present a full traceroute.
I had just taken it as a fact that full traceroutes are rarely possible these days. For reasons like you say, it's overhead for them and they don't want you to do it.
You measured delay to get an ICMP packet back. Is that strongly correlated to packets being dropped (causing resend)? As you see even your example, sometimes the time even goes down when you move "further".
Not working the field, but I believe it's an incredibly hard statistical model, not that far away from random chaos for a remote observer in the general case.
Packets don't get routed based on congestion, no. And while fat, L3's pipes are not infinitely fat (imagine how much overcapacity they would need to pay for to never be congested at any time of the week/day!).
Hop 2 is my router to my ISP
Hop 3, 4, 5, 6 are all routers within my ISPs network.
Hop 7 is from my ISP into "the internet". Level3 is a major provider of interconnectivity.
Hops 8 and 9 are within Level3's network.
Hop 10 is finally to Hacker News, which indicates HN's "ISP" is Level3 directly.
Only hop 2 is where my "400 Mbps" package limitation is in place. Only hop 10 is "server-to-their-ISP".
I chose 10-20% as my intuition because, as you see, 10 routers were involved to get my packet from my PC to HN, and only 1 of them is constrained by my internet package.