You have to use docker compose to get to localhost:8000 , there are still bugs but I'm working on it and there was interest expressed in this project on Hacker News a couple of weeks back.

I just spent time getting it all running on docker compose and moved my web ui from express js to flask. I want to get the code cleaned up and open source it at some point.

-- -- Name: refresh_topic_tables(); Type: PROCEDURE; Schema: public; Owner: postgres --

CREATE PROCEDURE public.refresh_topic_tables() LANGUAGE plpgsql AS $$ BEGIN -- Drop tables in reverse dependency order DROP TABLE IF EXISTS topic_top_terms; DROP TABLE IF EXISTS topic_term_tfidf; DROP TABLE IF EXISTS term_df; DROP TABLE IF EXISTS term_tf; DROP TABLE IF EXISTS topic_terms;

    -- Recreate tables in correct dependency order
    CREATE TABLE topic_terms AS
    SELECT
        dt.term_id,
        dot.topic_id,
        COUNT(DISTINCT dt.document_id) as document_count,
        SUM(frequency) as total_frequency
    FROM document_terms dt
    JOIN document_topics dot ON dt.document_id = dot.document_id
    GROUP BY dt.term_id, dot.topic_id;

    CREATE TABLE term_tf AS
    SELECT
        topic_id,
        term_id,
        SUM(total_frequency) as term_frequency
    FROM topic_terms
    GROUP BY topic_id, term_id;

    CREATE TABLE term_df AS
    SELECT
        term_id,
        COUNT(DISTINCT topic_id) as document_frequency
    FROM topic_terms
    GROUP BY term_id;

    CREATE TABLE topic_term_tfidf AS
    SELECT
        tt.topic_id,
        tt.term_id,
        tt.term_frequency as tf,
        tdf.document_frequency as df,
        tt.term_frequency * LN( (SELECT COUNT(id) FROM topics) / GREATEST(tdf.document_frequency, 1)) as tf_idf
    FROM term_tf tt
    JOIN term_df tdf ON tt.term_id = tdf.term_id;

    CREATE TABLE topic_top_terms AS
    WITH ranked_terms AS (
        SELECT
            ttf.topic_id,
            t.term_text,
            ttf.tf_idf,
            ROW_NUMBER() OVER (PARTITION BY ttf.topic_id ORDER BY ttf.tf_idf DESC) as rank
        FROM topic_term_tfidf ttf
        JOIN terms t ON ttf.term_id = t.id
    )
    SELECT
        topic_id,
        term_text,
        tf_idf,
        rank
    FROM ranked_terms
    WHERE rank <= 5
    ORDER BY topic_id, rank;

    RAISE NOTICE 'All topic tables refreshed successfully';
   

The rebus principle where someone might use a depiction of an eye for the sound "I" and so forth is the very basis of the script and was there from the beginning. The complicated part is they'd use words with one to three consonants and strip the vowels. To continue the example, we might use 𓃠 to represent the consonants "ct" and thus use it to write "cat", "cot", and "cut."

There was an inventory of uniconsonantal or uniliteral signs dating back to the very beginning of the language which the ancient Egyptians could have used as an alphabet (or abjad if we want to be pedantic) if they had wanted to, but they never did—at least to write Egyptian. The basis of our alphabet, Proto-Sinaitic script, seems to have come about when speakers of Caananite languages in the Sinai Peninsula borrowed a small number of Egyptian hieroglyphs, assigned them the phonetic value for the thing depicted in their own Caananite language, and they didn't bother with anything other than uniconsonantal signs.

The "three different alphabets" thing is unrelated to any of this. Hieroglyphs and hieratic appear around the same time. Hieroglyphs were used for monuments and more formal contexts. Hieratic is a cursive form of hieroglyphs that was much faster to write with a brush pen and ink. It tended to be used for literature, correspondence, and record-keeping. From what we know of Egyptian scribal education, they started out with hieratic and then moved on to hieroglyphs, with not everyone progressing to the point where they started learning hieroglyphs. This is quite the reversal from how we approach things today, with virtually every student of ancient Egyptian language learning hieroglyphs (specifically, Middle Egyptian) first and then moving on to learning hieratic. Demotic was a later evolution of hieratic. And eventually, the Egyptians wrote their language using a modified Greek alphabet ultimately derived from their hieroglyphs (Coptic).

This book discusses the IT systems at the IRS and VA and shows the kind of push back you can expect from entrenched players.

1. https://en.wikipedia.org/wiki/Elite_capture

I'm not sure if this book got into it but I've also read that the IRS has assembly code from the 1960s that is very optimized and only a few devs can work on it. ChatGPT knows a lot about this history as well.

Then I use HMAP + DBSCAN to create a 2D projection of my dataset. DBSCAN writes the clusters to a csv file. I read that back in to create topics, docs2topcs join table. Then I join each topic into a mega doc and consider the original corpus, I compute tf-idf, using only db functions. This gives me the top 5 or so terms per topic and serves as useful topic labels.

I can do 30 to 50 docs in an couple of hours. I imported 1100 pdf files and it took all weekend on an old gaming laptop w/ a ssd. I have a gpu, and I think the embedding steps would go faster but I'm still doing it all synchronously w/o any parallel processing.

Then I was able to apply UMAP + HDBSCAN to this dataset and it produced a 2D plot of all my books. Later I put the discovered topic back in the db and used that to compute tf-idf for my clusters from which I could pick the top 5 terms to serve as a crude cluster label.

It took about 20 to 30 hours to finish all these steps and I was very impressed with the results. I could see my cookbooks clearly separated from my programming and math books. I could drill in and see subclusters for baking, bbq, salads etc.

Currently I'm putting it into a 2 container docker compose file, base postgresql + a python container I'm working on.