There are so many Apache projects these days (mostly seem to be in the incubator) - where do they all come from? I had once thought that Apache was a dumping ground for the technology of failed startups, but a lot of these projects seem to find new life in the incubator - so they must be doing something right.
The list of big name Apache projects is nuts, look at their project lists and you can easily pull out dozens of big projects and even incubator has a few popular projects.
From what I've seen, there are two major sources, and one slightly less prevalent one:
1. Big companies. If you go through the incubator, you'll find quite a few projects where there was a large code grant, and many initial contributors, from one large company or another. Facebook, IBM, Yahoo, Oracle, etc.
2. Academia. A lot of projects are existing OSS projects being developed as part of academic research, and now the team wants to formalize the project structure and get the project to a wide audience.
The slightly less prevalent one:
3. Just ordinary OSS projects developed $WHEREVER, not necessarily affiliated with any company or university, and the team just decide that Apache would make a good home for their project.
Anyway, I definitely wouldn't call the ASF "a dumping ground for the technology of failed startups". If anything the incubator is designed to prevent that sort of thing. If projects don't successfully complete incubation, they don't become full fledged TLPs. Now, there are people who quibble over how long some projects linger in the incubator, sometimes in a sort of half-dead/half-alive zombie state, but by and large the process works pretty well.
Lots of these (including this one) come out of universities, or university-corporate research partnerships. Personally I'm very glad that universities nowadays seem to be more okay with publishing tangible results (i.e. source code) as opposed to "just" the research papers. I don't think it's always been that way. I can't begin to imagine the amount of work that is rotting on someone's old usb stick, because they weren't allowed to publish their PhD theses's source code.
With the parallelism model and abstraction it has to support model/data or hybrid partitioning, and synchronous/asynchronous or hybrid training, it should be easy to extend to GPU cluster. However, training is required only periodically, and if it can be done on existing clusters as efficiently, why not?
This is wrong. Training data can be streamed through GPU memory during training. It's your parameters that can't exceed GPU memory. You can get GPUs with 12 GB of memory, and they also support float16 so they can be twice as memory efficient as CPUs. If your model has more parameters than that, then you'll be waiting months or years for a single model to train using CPUs, even distributed.
Furthermore, almost any technique you use to distribute and scale training will work just as well regardless of whether the computations are happening on CPUs or GPUs.
According to the paper, the parameters fit on one GPU (or at least that one GPU was able to train the model). It was just too slow, so they trained on 8 GPUs in parallel. But those GPUs were still on the same machine (one node, multiple GPUs).
At least back in 2012, Google research seemed to suggest that distributed CPU training of large models could sometimes be preferable to fitting within the limits of GPUs:
SINGA provides an abstraction for defining all known models,a neural net structure that is easy for model and data partitioning, and a parallelism model that supports both synchronous, asynchronous and hybrid training frameworks. Processing at each node could the be CPU, GPU, or CPU-GPU based.
It's not trivial to take any existing algorithm for training and scale it not only cross-GPU but also cross machine. Training tends to be very bandwith intensive. In fact that is the entire reason Nvidia is so heavily focused on this with their upcoming GPUs (Pascal). I am not saying it's not possible, just that it's a lot harder than "distributed makes it scalable".
Singh is actually the Indian Sanskrit root word for Lion and is common in south Indian language of Kannada, Tamil and Malayalam which has Simha, Singham and Simham as the word for lion respectively. The Sri Lanka's ethnic majority are called Sinhalese which again is derived from the Indo-Aryan Sinhala language bearing similar influence, meaning the "lion people" or "people with lion blood", while a sword-wielding lion is the central figure on the modern national flag of Sri Lanka.The island nation of Singapore (Singapura) derives its name from the Malay words singa (lion) and pura (city), which in turn is from the Tamil-Sanskrit சிங்க singa सिंह siṃha and पुर புர pura.
