I was looking for a version of a proxy that could maximize throughput to each LLM based on its limits. Basically max requests and input/output tokens per second.
I couldn't find something, so I rolled a version together based on redis and job queues. It works decently well, but I'd prefer to use something better if it exists.
Does anyone know of something like this that isn't completely over engineered / abstracted?
I've been keeping an eye on this space for awhile as it matures a bit further. There's been a number of startups that have popped up around this - apart from Temporal and DBOS, Hatchet.run looked interesting.
I've been using BullMQ for awhile with distributed workers across K8 and have hacked together what I need, but a lightweight DAG of some sort on Postgres would be great.
I took a brief look at your docs. What would you say is the main difference of yours vs some of the other options? Just the simplicity of it being a single sql file and a sdk wrapper? Sorry if the docs answer this already - trying to take a quick look between work.
> I took a brief look at your docs. What would you say is the main difference of yours vs some of the other options? Just the simplicity of it being a single sql file and a sdk wrapper? Sorry if the docs answer this already - trying to take a quick look between work.
It's really just trying to be as simple as possible. I was motivated by trying to just do the most simple thing I could come up with after I did not really find the other solutions to be something I wanted to build on.
I'm sure they are great, but I want to leave the window open to having people self host what we are building / enable us to deploy a cellular architecture later and thus I want to stick to a manageable number of services until until I can no longer. Postgres is a known quantity in my stack and the only postgres only solution was DBOS which unfortunately did not look ready for prime time yet when I tried it. That said, I noticed that DBOS is making quite some progress so I'm somewhat confident that it will eventually get there.
> Could you provide some more specifics as to why DBOS isn’t “ready for prime time”? Would love to know what you think is missing!
Some time in September I was on a call with Qian Li and Peter Kraft and gave them direct feedback. The initial reason this call happened was related to a complaint of mine [1] about excessive dependencies on the Python client which was partially remedied [2] but I felt it should be possible to offload complexity away from the clients even further. My main experiments were pretty frustrating when I tried it originally because everything in the client is global state (you attach to the DBOS object) which did not work for how I was setting up my app. I also ran into challenges with passing through async and I found the step based retry not to work for me.
(There are also some other oddities in the Python client in particular: why does the client need to know about Flask?)
In the end, I just felt it was a bit too early and I did not want to fight that part of the infrastructure too much. I'm sure it will get there, and I'm sure it has happy users. I was just not one of them.
I'd love to hear both of your thoughts! I'm considering durable execution and DBOS in particular and was pretty happy to see Armin's shot at this.
I'm building/architecting a system which will have to manage many business-critical operations on various schedules. Some will be daily, some bi-weekly, some quarterly, etc. Mostly batch operations and ETL, but they can't fail. I have already designed a semblance of persistent workflow in that any data ingestion and transformation is split into atomic operations whose results are persisted to blob storage and indexed in a database for cataloguing. This means that, for example, network requests can be "replayed", and data transformation can be resumed at any intermediate step. But this is enforced at the design stage, not runtime like other solutions.
My system also needs to be easily auditable and written in Python. There are many, many ways to build this (especially if you include cloud offerings) but, like Armin, I'm trying to find the simplest architecture possible so our very small team can focus on building and not maintaining.
As the CEO of DBOS I'm of course heavily biased, but I think DBOS is the perfect solution for you. It has everything you need (queues, crons, easy to understand pure Python). We'd be happy to help you through it if you get stuck. You can pop into the DBOS discord too (link on our webpage).
I was going to say the same. We're using binary vectors in prod as well. Makes a huge difference in the indexes. This wasn't mentioned once in the article.
The sales team got impacted since they have variable salaries based on deal closures. I am pretty sure there are SOX compliance violations hidden in many deals at large firms due this lack of clarity. SoX violations can get your firm into a lot of trouble.
Thanks. I've been heads down on this for some time and don't have the huge network to share this with.
Regarding the product, I'm open to any and all feedback for your use case. Trying to follow the adage of "if you are not embarrassed by the first version of your product, you’ve launched too late".
And wow - just read your reddit post. I'm going to look into that for us too.
> Thanks. I've been heads down on this for some time and don't have the huge network to share this with.
Your API, while using your MVP UI as teaching tool, is the #1 topic at my next meeting. The lesson about shipping early is so hard for me to execute, even though I know better. So, great job!
Thank you! Yeah, I know it's a fine balance between shipping too early and having a buggy product vs too late. I'm shipping as fast as I can now to keep that tight feedback loop going.
If you do this sort of thing often, I'd love to chat further. I'm basically trying to automate this sort of manual research around companies with a library of deep research APIs.
We launched corporate hierarchy research and working on UBO now. From the corporate hierarchy standpoint, it looks like the Delaware entity fully owns the Estonian entity. Auto generated mermaid diagram from the deep research:
It's fairly easy to know how to poke around these businesses. Look up the people, the business, and the product. It's less fun when it involves linkedin. Every country has a database of business numbers to name and rough documentation. Dns look up can reveal some information. Social media typically finish the rest. These "founders" are often serial founders, with a ton of abandoned projects and a trail on product hunt, and other websites.
In this case, what really gets to me is the basic template website they're using; with image carousel but only one image... and the fact that they appeared to have paid influencers on youtube to shill their product.
I noticed the template too. Someone mentioned recently that's actually a good risk signal - scammers often use the same site structure across domains.
On the research, you're absolutely right. It fits that sweet spot where it's just easy / boring / tedious enough to automate with the current generation of LLMs.
So, I should expect to see a new product launch of DoughnutKVM, to "round out your infrastructure woes", complete with vibe coded app interface and AI generated product images, here in the next week? ;P
Looks like you have a potentially great business for corporate compliance, if you can answer with plausibly high confidence (or indemnify?).
I only occasionally research companies, and it's from an engineering&product perspective, aside from corporate ownership compliance. (For example, I was asked to vet a little-known company as a prospective partner, for building our cloud infrastructure atop theirs. One of the first rapid low-cost, high-value things I could do, besides looking at their docs and trying their demos, was to skim through the history of business news about them.)
Interesting. That's actually where we started. We were doing automated research on vendors from a TPRM perspective and looking for data points around organizational security / reputation. Examples - if the company had been hacked before / how they responded, do they have a CISO, nth party vendors, are they SOC2 / FedRAMP certified, etc. Basically, predictors of risk / stability.
We realized the underlying business graph was the bottleneck though, so that's been our focus for some time. With that in place, we're now coming full circle on the risk research standpoint.
On your comment about confidence / liability, we're actually having conversations around that now and getting feedback. First step is exposing all the research and evidence directly to build trust, which is what we're doing now for the new corporate hierarchy system.
When I used to work in credit control and accounts receivable, the use of D-U-N-S numbers was how we tied a lot of this information together. It is similar to how SSN are used by credit rating agencies but for businesses and global (unlike the EIN).
Good idea! I picked a random California Ikea entity (IKEA US RETAIL LLC) and ran it through the system. Here's the output - current goal is to get to ultimate parent.
## Summary
IKEA US RETAIL LLC is a limited liability company. It is wholly owned by IKEA Holding U.S., Inc., and ultimately controlled by Stichting INGKA Foundation, a Dutch foundation that owns Ingka Group.
Sorry, habit. I've been debating on exposing these publicly, but they're expensive to create. We have a public interactive demo here for now: https://savvyiq.ai/products/entity-hierarchy
Since it's quiet here, figured I'd share what the API actually spits out. Here's MG Motor's ownership chain (this is just the Mermaid diagram field - we return a bunch of other stuff too):
graph TD
e2[SAIC MOTOR UK HOLDING CO., LTD.]-->|2005-02-15|e1[MG MOTOR UK LTD]
e1[MG MOTOR UK LTD]-->|2018|e7[MG Sales Centre Limited]
e4[SAIC Motor Corporation Limited]-->e2[SAIC MOTOR UK HOLDING CO., LTD.]
e4[SAIC Motor Corporation Limited]-->e3[SAIC MOTOR INTERNATIONAL UK LTD]
e5[Shanghai Automotive Industry Corporation Group]-->|62.69%|e4[SAIC Motor Corporation Limited]
e6[Shanghai State-owned Assets Supervision and Administration Commission Shanghai SASAC]-->e5[Shanghai Automotive Industry Corporation Group]
You can copy/paste that into any Mermaid renderer to see it visually. Pretty wild how a British car brand ends up tracing back to Shanghai's government.
Happy to run lookups for other companies if anyone's curious what their ownership looks like!
Hey, good question. Yeah, we're aware of many companies that can have thousands of subsidiaries.
We found that going both up the chain and down / sideways in the chain was too much for the agent to handle - they are two distinct operations. The #1 use case was customers trying to understand the ultimate parent, so we decided to focus on that first.
We have something roughly working on subsidiaries, but it's not ready for prime time yet. It would likely be a separate API.
I can confirm on the performance benefits. I wanted to start with uuidv7 for a new DB earlier this year, so I put together a function to use in the meantime. Once the function is available natively, we'll just migrate to use it instead.
For anyone interested:
CREATE FUNCTION uuidv7() RETURNS uuid
AS $$
-- Get base random UUID and overlay timestamp
select encode(
set_bit(
set_bit(
overlay(uuid_send(gen_random_uuid()) placing
substring(int8send((extract(epoch from clock_timestamp())*1000)::bigint) from 3)
from 1 for 6),
52, 1), -- Set version bits to 0111
53, 1), 'hex')::uuid;
$$ LANGUAGE sql volatile;
I'm working on Plaid / Perplexity for business data.
The basic idea is that integrating business data into a B2B app or AI agent process is a pain. On one side there's web data providers (Clearbit, Apollo, ZoomInfo) then on the other, 150 year old legacy providers based on government data (D&B, Factset, Moody's, etc). You'd be surprised to learn how much manual work is still happening - teams of people just manually researching business entities all day.
At a high level, we're building out a series of composable deep research APIs. It's built on a business graph powered by integrations to global government registrars and a realtime web search index. Our government data index is 265M records so far.
We're still pretty early and working with enterprise design partners for finance and compliance use cases. Open to any thoughts or feedback.
Adding on to the other comments here about Next.js vs Remix.js.
We had to choose a framework for our new app last year and were researching the current state of things. Next.js was / is by far the most popular, but also had some of the worst feedback and caution to stay away. Remix isn't perfect, but I appreciate less abstractions and working with simple request / response structures.
Also, a warning for those hiring for frontend / fullstack roles:
Over the years when hiring for roles for X frontend framework, we would constantly find "experts" in framework X that would really impress us. Whether it was React, Angular, Vue, Remix, etc. Then after moving forward, we found they didn't know core JS fundamentals and were basically useless beyond the framework.
I couldn't find something, so I rolled a version together based on redis and job queues. It works decently well, but I'd prefer to use something better if it exists.
Does anyone know of something like this that isn't completely over engineered / abstracted?