At Hopsworks we built HopsFS-S3 to improve things like listing (becomes a partition pruned scan of a in-memory DB), atomic renames and added a block/object caching layer using NVMe drives.

You can read the research paper here if you are curious: https://www.hopsworks.ai/research-papers/hopsfs-s3-extending...

GAIA-X, is that you?

Mr. Gorbachev, Tear Down This Wall! - https://www.youtube.com/watch?v=GCO9BYCGNeY

Reminds me of this: https://www.youtube.com/watch?v=0KXoBcQER_0

Logical Clocks | Mid/Senior Frontend Developer (React) | Europe (Stockholm or Remote) | Full-time | https://logicalclocks.com

Logical Clocks is the company behind Hopsworks. Hopsworks is an open-source feature store that allows teams to organize and scale their feature engineering efforts. (https://hopsworks.ai) We recently raised our series A (5M euro) and we are expanding the teams. We are looking for a mid/senior frontend developer. Our frontend stack includes Typescript, React JS, Redux.

Your responsibilities will be to lead the development the new Hopsworks frontend, lead the technical discussions/decisions and help with the expansion of the UI/UX team.

If you want to apply/have any questions you can reach me at fabio [at] logicalclocks [dot] com or through our website: https://www.logicalclocks.com/job-positions/senior-front-end...

Looks like the Casio is not web scale :)

They just need to add a casio load balancer.

A bit of a weird article coming from a VC heavily invested in SaaS providers

They're basically talking about helping SaaS providers reduce their IaaS spending. They're not talking about migrating off SaaS.

I'd say that the main difference with ObjectiveFS are metadata operations. From the documentation of ObjectiveFS:

`doesn’t fully support regular file system semantics or consistency guarantees (e.g. atomic rename of directories, mutual exclusion of open exclusive, append to file requires rewriting the whole file and no hard links).`

HopsFS does provide strongly consistent metadata operations like atomic directory rename, which is essential if you are running frameworks like Apache Spark.

That quote from the ObjectiveFS documentation [1] is out of context. It was describing limitations in s3fs, not ObjectiveFS. My understanding is that because ObjectiveFS is a log-structured filesystem that uses S3 as underlying storage, it doesn't have those limitations.

[1]: https://objectivefs.com/faq

Not really, but you can try it out on https://hopsworks.ai

It's conceptually similar to EMR in the way it works. You connect your AWS account and we'll deploy a cluster there. HopsFS will run on top a S3 bucket in your organization. You get a fully featured Spark environment (With metrics and logging included - no need for cloudwatch). UI with Jupyter notebooks, the Hopsworks feature store and ML capabilities that EMR does not provide.

No, HopsFS namenodes are stateless as the metadata is stored on an in-memory distributed database (NDB)

There are other papers that describe HopsFS architecture in more details if you are interested: https://www.usenix.org/system/files/conference/fast17/fast17...