I would love for things like this to be accessible via hubs like open router or AWS bedrock. It's hard to justify adding new model vendors directly with all the heightened concerns about privacy and security, but if bold new capabilities are added to a centralized already-vendor like AWS, technical people can adopt them without going through a whole compliance/purchasing/vendor review process. And an extra middleman tax is well worth it when the cost savings of the model itself can be one-two orders of magnitude.
You can set privacy requirements and define an allow list. To me the main value prop is that I get one bill for all models and can quickly try new models without signing up anywhere or changing my code.
Oh! Also you can pass an array of models and if the first provider is down it automatically falls through to the next provider. More useful than it should be...
That's still ultimately privacy by contract (where you have to trust the inference providers to uphold their end of the deal), rather than privacy by design.
mushufasa · · focus · HN ↗
cheeze · · focus · HN ↗
I guess you can choose your provider still? But isn't the point that the lowest bidder is doing inference?
ajmurmann · · focus · HN ↗
hobofan · · focus · HN ↗
LeBit · · focus · HN ↗