Great point re: the “Bespoke” style of programming. It takes some unlearning and getting used to tacit programming [0] and trains [1] to appreciate this.
I’ve worked on fairly complex systems using kdb/q that are a joy to debug because the codebase is a handful of files. q is admittedly a more simplified and “mainstream” cousin of J, but allows a fair bit of expressiveness. The powerful data store (kdb) and q-sql that comes along with it
makes it one of the most concise data platforms ever. Too bad it’s not free for commercial use!
> Now if it were possible for it to accept an event from Kafka but not guarantee that the event will eventually make it into the materialized view but may be lost, that'd be a problem.
This is usually not a problem these days as it’s possible to guarantee exactly once ingestion using Kafka offsets
IIRC clickhouse still doesn't have this guarantee. It guarantees exactly one ingestion but not that the ingested event gets processed all the way through to view. And if the processing fails, that event won't be retried and is now gone.
You’re right. You don’t get read-your-own-writes guarantees in a typical Kdb tickerplant architecture, as the event streams are propagated to the real-time nodes asynchronously.
I suppose one can measure the risk of losing messages - I’m not suggesting that it’s always possible, but calling it an unmeasurable risk is probably not true
As mruts pointed out above, kdb+/q provides what you're imagining - it provides fantastic distributed computing support via its IPC protocol and also has multi-core support.
There is a free 64-bit version available with limits that don't seem harsh at all:
The 64-bit kdb+ On-Demand Personal Edition is free for personal, non-commercial use. Currently it may be used on up to 2 computers, and up to a maximum of 16 cores per computer, but is not licensed for use on any cloud – only personal computers. It requires an always-on internet connection to operate, and a license key file, obtainable from ondemand.kx.com
You can start the 32 bit instance with the -s switch (for q to be started with multiple slaves) for parallel execution of a function over data using the "peach" command [0].
But yes, I do agree that the 32 bit instances have limited use, as they can't be used in for-profit projects.
> But at that time APL was not running on any computer; and he stoutly insisted that it was unnecessary that it ever run on a computer. It was for him a notation with which he could express algorithmic concepts; and for him at that time, that seemed sufficient.
Indeed. Because both APL and Lisp began as languages for describing computational ideas, it stands to reason that they would be particularly fit relative to languages that evolved under other fitness criteria.
I was intrigued to not the opposition to APL mentioned in the article by Djikstra and others on the basis of a different mental model of computation.
I’ve worked on fairly complex systems using kdb/q that are a joy to debug because the codebase is a handful of files. q is admittedly a more simplified and “mainstream” cousin of J, but allows a fair bit of expressiveness. The powerful data store (kdb) and q-sql that comes along with it makes it one of the most concise data platforms ever. Too bad it’s not free for commercial use!
[0] https://www.jsoftware.com/help/jforc/tacit_programs.htm
[1] https://www.jsoftware.com/help/learning/09.htm