I'm not an expert in the LLM space, but I'm an external contributor to comma.ai's openpilot project and I'm and quite familiar with how its controls work, so I looked from that perspective. There's two questions here:
1) Could a cloud-delivered LLM figure out how to drive this route, based on those input data and given access to those output actuators? Looks like yes. Sure.
2) Could this work in the real world? Absolutely not. Three reasons: latency, latency, and latency.
openpilot's driving model updates the target curvature and acceleration at 20Hz. Every millisecond of the round trip time through every piece of its entirely-local driving stack is well-understood, extremely consistent, and tightly optimized. It has to be, otherwise you can't react to even minor bumps or wind gusts, much less rapidly-developing traffic situations.
Adding even a single speed of light RTT to a cloud service is meaningfully bad, and you'll need a whole lot more to encode and upload camera imagery to even start the time-to-LLM-response clock, and then send the response back down. By then the world around the car has moved on.
There's a reason Tesla and every other self-driving manufacturer need the compute hardware in the car.
I'm far from neuroscience, but humans don't need to operate at 20Hz to drive a car. And human reaction latency (event to measurable action) is often over 1s (under 1Hz).
actually, human latency is quite slow and distracted drivers often have 1sec+ latency.
it works because 99% of the time you don't need fast latency because you can accurately predict things.
that's why a standard recommendation is to drive 2+ seconds (time not distance) behind the car in front of you. also why experienced drivers instinctively move their hands/feet into position during tricky moments when they need to cut the latency.
fun exercise, try taking your foot off the gas and hitting the break - slower than you think!!
jyoung8607 · · focus · HN ↗
1) Could a cloud-delivered LLM figure out how to drive this route, based on those input data and given access to those output actuators? Looks like yes. Sure.
2) Could this work in the real world? Absolutely not. Three reasons: latency, latency, and latency.
openpilot's driving model updates the target curvature and acceleration at 20Hz. Every millisecond of the round trip time through every piece of its entirely-local driving stack is well-understood, extremely consistent, and tightly optimized. It has to be, otherwise you can't react to even minor bumps or wind gusts, much less rapidly-developing traffic situations.
Adding even a single speed of light RTT to a cloud service is meaningfully bad, and you'll need a whole lot more to encode and upload camera imagery to even start the time-to-LLM-response clock, and then send the response back down. By then the world around the car has moved on.
There's a reason Tesla and every other self-driving manufacturer need the compute hardware in the car.
ivanjermakov · · focus · HN ↗
I'm far from neuroscience, but humans don't need to operate at 20Hz to drive a car. And human reaction latency (event to measurable action) is often over 1s (under 1Hz).
suddenlybananas · · focus · HN ↗
This is so self evidently false, I struggle to believe you think it is true. How could anyone catch a ball even?
asah · · focus · HN ↗
it works because 99% of the time you don't need fast latency because you can accurately predict things.
that's why a standard recommendation is to drive 2+ seconds (time not distance) behind the car in front of you. also why experienced drivers instinctively move their hands/feet into position during tricky moments when they need to cut the latency.
fun exercise, try taking your foot off the gas and hitting the break - slower than you think!!
suddenlybananas · · focus · HN ↗