if it's agentic stuff, they likely aren't hammering a core constantly and they will maybe sit idle quite often between model requests, so it makes sense. I do wonder how much memory they allocate to each one though.
it's just very efficient use of shared cores that is required to make these kinds of workloads cost efficient
A place I worked back around 2020 was running a Grafana instance per customer that got embedded on the web dashboard. We had 110 pods per GKE (kubernetes on gcp) 4 CPU node because that was a network imposed pod limit at the time. The nodes were usually idle--could have shoved a lot more on if not for the IP limit.
I think around that time Grafana changed their license tho so you couldn't host OSS Grafana as part of your service.
vblanco · · focus · HN ↗
dangoodmanUT · · focus · HN ↗
aabhay · · focus · HN ↗
r_lee · · focus · HN ↗
it's just very efficient use of shared cores that is required to make these kinds of workloads cost efficient
eru · · focus · HN ↗
nijave · · focus · HN ↗
I think around that time Grafana changed their license tho so you couldn't host OSS Grafana as part of your service.
m3kw9 · · focus · HN ↗