There are some semantic differences compared to POSIX filesystems. A couple big ones:
- You can only append to an object, and each object can only have one writer at the time. This is useful for distributed systems - you could have one process adding records to the end of a log, and readers pulling new records from the end.
- It's also possible to "finalize" an object, meaning that it can't be appended to any more.
Other systems don't offer the performance that Colossus offers, is why. POSIX has all kinds of silly features that aren't really necessary for every use case. Throwing away things like atomic writes by multiple writers allows the whole system to just go faster.
It sounds like you have to find a design that meets your performance target and usage patterns - just like anything else. It also sounds like Google's CFS is a grass is greener situation - you heard Google had something that solved the problem you have, so you want it. But the reason it sounds good, compared to the other designs, is that you haven't had to actually use it and run into its quirks yet.
Google's internal systems have been written against the Colossus semantics for many, many years and thus benefit from it's upsides (performance, cost efficiency, reliability, strong isolation for a multi tenant system, ability to scale byte and IO usage fairly independently, tremendously good abstraction against and automation of underlying physical maintenance, etc) while not really having too much of an issue with any of the conscious trade-offs (like no random writes).
On the other hand, if you've been building your applications against expectations of different semantics (like POSIX), retrofitting this into your existing application is really hard, and potentially awkward. This is (IMO) why there hasn't been an overtly Colossus based Google Cloud offering previously. (Though it's well publicized that both Persistent Disk and GCS use Colossus in their implementation.)
One of the reasons why it would be extremely hard to just set up or build CFS elsewhere or on a different abstraction level is that while it may look quite achievable to implement the high level architecture, there is vast complexity in the practical implementation side. The tremendous user isolation it affords for an MT system, the resilience it has against various types of failures and high throughput planned maintenance, the specialization it and its dependencies have to use specific hardware optimally.
(I work on Google storage part time, I am not a Colossus developer.)
Slight nit: "zonal" doesn't necessarily mean "not replicated", it means that the replicas could all be within the same zone. That means they can share more points of failure. (I don't know if there's an official definition of zonal.)
I don't know of a single GCP product that's been shut down, although I could be missing something. But their track record for GCP is, I think, what you would want a cloud provider's record to be.
(I should mention that I work for GCP. But this is just based on my own memory.)
Azure is our cloud provider. Interface is flexible, since our current implementation leverages Prisma ORM connected to Postgres & SQL Server. We're going to have to rebuild it anyway.
Got it, thank you! CockroachDB is the only one I know offhand that does what you're looking for. Another comment mentioned Vitess, which might also work.
It seems like there are a lot of options for large scale analytics, but I don't know a lot for high throughout geo-redundant transaction processing.
Vitess maybe? from what I have demoed and for being open source that’s the one I would choose. Unfortunately there isn’t a Postgres compatible one that is as mature yet and certainly not free.
I mean to be fair Google has also tried very, very hard to improve home internet access for people, to the point of setting up their own ISP and running municipal fiber networks. That's a pretty big try, and I really wish it had taken off beyond the places where Google Fiber operates.
(NB: I work at Google, but this comment has nothing to do with my work.)
When it really comes down to laying Fiber isn't a job requiring innovation, it require hard man hours to pull cables.
In the end Google tried to innovate around the hard work by burying cables around 5cm deep or so in stead of a meter, which turned out to be short sighted.
I thought Google was mostly buying up dark fiber (I think that's the case in my city at least). Where did they run their own, and what was the impact of the shallow runs?
Rumors are Azure DevOps and GitHub are converging "soon", and maybe "Project Cyclops" wasn't specifically to improve Microsoft/Windows repo performance, but it seems reasonable given the convergence rumors it could be a step in the direction of preparing for/migrating the repo to GitHub. Of course Microsoft doesn't want to panic Enterprise developers on Azure DevOps just yet so they are extremely quiet right now about any convergence efforts, so I take the rumors with a grain of salt. It is something that I wish Microsoft would properly announce sooner rather than later as it might provide momentum towards GitHub in capital-E Enterprise development world (even if will panic those that are still afraid of GitHub for whatever reasons).
Someone close to me has Celiac disease, and getting an official diagnosis will let you do more than just eliminate gluten. For one thing, you could participate in one of the multiple ongoing drug trials for Celiac that are looking for participants.
The Celiac foundation has a list of them if you're interested, but you'll need a Celiac diagnosis to be eligible.
Are they at a sufficient point to be worth enrolling in? I've reviewed the foundation list but beyond some promising phase 3 items it was short on what's next.
