Deep in the internals of Google, we actually had a system that was putting incoming requests not in a queue (first-in-first-out), but in a stack (last-in-first-out).
The system in question was essentially not meant to have any backlog of request under normal operations. But when overloaded, for this particular system it was better to serve the requests it could serve very fast, and just shed load on the overflow.
(I don't remember more specifics, and even if I could, I would probably not be allowed to give them..)
I really wish network routers would do this when delivering packets...
The other end gets far better information about congestion that way, and congestion control algorithms could be made much smarter.
Everyone says "jitter is bad in networks", but the reality is if the jitter is giving you network state information, it is a net positive - especially when you can use erasure coding schemes so the application need not see the jitter.
Currently you can assume that if you receive a packet with a higher sequence number, earlier packets were probably lost, and resend them. This heuristic won't work anymore with a LIFO buffer.
LIFO sounds annoying for bursty traffic, since you can only start processing the burst after the buffer has cleared.
Using erasure coding, there is no need to know which packets were lost. You just keep sending more erasure coded packets till you get an acknowledgement that all data has been decoded. If a packet from a while ago randomly reappears, it usually isn't wasted throughput either - it can be used to replace any other lost packet, either forwards or backwards in the packet stream (up to some limit determined by the applications end-to-end latency requirement).
I have exactly that pattern in an internal system. Under load, the caller will give up after a fixed timeout and retry, so why waste time on an old request that where the caller has probably already hung up?
LIFO is a perfectly reasonable mode, when overloaded. FIFO will give you a stable equilibrium where every request takes too long and fails. LIFO breaks out of that operating point.
This is the key feature of the CoDel solution to buffer bloat.
The system in question was essentially not meant to have any backlog of request under normal operations. But when overloaded, for this particular system it was better to serve the requests it could serve very fast, and just shed load on the overflow.
(I don't remember more specifics, and even if I could, I would probably not be allowed to give them..)