(Hopping in here because the discussion is interesting... feel very free to ignore.)
Thanks for writing this up! It was a very interesting read about a part of networking that I don't get to seriously touch.
That said: I'm sure you guys have thought about this a lot and that I'm just missing something, but "why can't every proxy probe every [worker, not application]?" was exactly one of the questions I had while reading.
Having the workers being the source-of-truth about applications is a nicely resilient design, and bruteforcing the problem by having, say 10k proxies each retrieve the state of 10k workers every second... may not be obviously impossible? Somewhat similar to sending/serving 10k DNS requests/s/worker? That's not trivial, but maybe not _that_ hard? (You've been working on modern Linux servers a lot more than I, but I'm thinking of e.g. https://blog.cloudflare.com/how-to-receive-a-million-packets...)
I did notice the sentence about "saturating our uplinks", but... assuming 1KB=8Kb of compressed critical state per worker, you'd end up with a peak bandwidth demand of about 80 Mbps of data per worker / per proxy; that may not be obviously impossible? (One could reduce _average_ bandwidth a lot by having the proxies mostly send some kind of "send changes since <...>" or "send all data unless its hash is <...>" query.)
(Obviously, bruteforcing the routing table does not get you out of doing _something_ more clever than that to tell the proxies about new workers joining/leaving the pool, and probably a hundred other tasks that I'm missing; but, as you imply, not all tasks are equally timing-critical.)
The other question I had while reading was why you need one failure/replication domain (originally, one global; soon, one per-region); if you shard worker state over 100 gossip (SWIM Corrosion) instances, obviously your proxies do need to join every sharded instance to build the global routing table - but bugs in replication per se should only take down 1/100th of your fleet, which would hit fewer customers (and, depending on the exact bug, may mean that customers with some redundancy and/or autoscaling stay up.) This wouldn't have helped in your exact case - perfectly replicating something that takes down your proxies - but might make a crash-stop of your consensus-ish protocol more tolerable?
Both of the questions above might lead to a less convenient programming model, which be enough reason on its own to scupper it; an article isn't necessarily improved by discussing every possible alternative; and again, I'm sure you guys have thought about this a lot more than I did (and/or that I got a couple of things embarassingly wrong). But, well, if you happen to be willing to entertain my questions I would appreciate it!
(I used to work at Fly, specifically on the proxy so my info may be slightly out of date, but I've spent a lot of time thinking about this stuff.)
> why can't every proxy probe every [worker, not application]?
There are several divergent issues with this approach (though it can have it's place). First, you still need _some_ service discovery to tell you where the nodes are, though it's easy to assume this can be solved via some consul-esque system. Secondly, there is a lot more data than you might be thinking at play here. A single proxy/host might have many thousands of VMs under its purview. That works out to a lot of data. As you point out there are ways to solve this:
> One could reduce _average_ bandwidth a lot by having the proxies mostly send some kind of "send changes since <...>" or "send all data unless its hash is <...>" query.
This is definitely an improvement. But we have a new issue. Lets say I have proxies A, B, and C. A and C lose connectivity. Optimally (and in fact fly has several mechanisms for this) A could send it's traffic to C via B. But in this case it might not even know that there is a VM candidate on C at all! It wasn't able to sync data for a while.
There are ways to solve this! We could make it possible for proxies to relay each others state. To recap:
- We have workers that poll each other
- They exchange diffs rather than the full state
- The state diffs can be relayed by other proxies
We have in practice invented something quite close to a gossip protocol! If we continued drawing the rest of the owl you might end up with something like SWIM.
As far as your second question I think you kinda got it exactly. A crash of a single corrosion does not generally affect anything else. But if something bad is replicated, or there is a gossip storm, isolating that failure is important.
You're commenting on an article about people consuming less than $3/day; Americans on food stamps (SNAP appears to be about $4-6/day alone, not counting any other benefits) are a distraction, simply not part of the population that the article is discussing.
Picking anything but PlantUML's default theme will make PlantUML look less 90's-Java. As a minimalistic solution, I already like "skinparam monochrome true" https://plantuml.com/skinparam#:~:text=Black%20and%20White. (You can go further with CSS and themes and...)
I really like this article. I do think it's useful to consider that the unit of isolation ("process") of the cloud era is a VM or container, and that the major clouds do have some sort of permissions model.
The error rate is given per bit, not per second, i.e. every few bars represents a distinct DRAM chip. That makes some sense, and the article explains quite well why DRAM would behave like that... but I agree that I had to read the article at least twice to figure out that the x-axis on the graph represents the lower bit of the address line!
It's not ideal, but it's not a cryptographic risk.
Using an encoding that (like Protobuf) has multiple representations for a message may cause you problems if you switch implementations - sha256(encode(msg)) might yield different hashes on different implementations of encode().
But the main risk is an encoding that has multiple interpretations of a single encoding (e.g. sha256(encode("admin", "true")) == sha256(encode("admint", "rue"))), and Protobuf (can be unserialized, and thus) doesn't have that problem.
I was also thinking of promise pipelining, but note that the article proposes communicating A -> {B, C} -> D, never directly communicating from A to D. Cap 'n Proto "level 3" nodes could send promises from B and C to D, but that still needs A to talk to D, i.e. A -> {B, C, D}; {B, C} -> D. Same latency / depth of dependency chain, but still more messages - right?
(In return, note that Cap 'n Proto's A -> D message makes it more obvious how A figures out whether the operation succeeded; I'm not quite sure how that works in the proposed diagram. I suppose the proposed system actually puts all messages in a system-wide database, which does solve the problem.)
That should not be the case with promise pipelining. The "Mobile Almost-Code" section of the E page explains this. You mentioned "continuation passing style", which is effectively what promise pipelining does: For the constrained class of continuations that can be serialized as a dataflow graph, pass those along with the message.
Importantly, the system wide constraint is willing participation from each actor, not a shared database. Instead of each actor needing to know how to interact with the shared database, each actor needs to be willing and able to execute these passed continuations.
One pitfall to consider: software development is often its own culture, and nourishing and valuing (or merging!) multiple distinct cultures in a company requires real effort from leadership/management/team leads/senior ICs/...
I'm sure that e.g. truck drivers and software developers can be made into a strong team, but don't just assume that cross-culture collaboration will work "by accident"/"automatically".
Concur. In house can work wonderfully, but you have to get the dev team integrated into the soul of the business… and you have to be willing to bend when they tell you the current system is insane.
I agree that less-than-milspec equipment should survive being installed in a home, but... this fiber didn't. https://news.ycombinator.com/item?id=46572962 seems to be relevant.
reply