What I'd like to see is Applescript 2.0. Applescript was ahead of its time by letting you script the Apple GUI, but it took an actual programming language to do it. Now an AI could do it for people. And also, if the interface was reliable because large numbers of people and AIs were using it, people could hand write programs as well, and they could be integrated into shell scripts, and they could have other things built on top if the layer was reliable.
There's no reason a GUI toolkit couldn't be written to enable a lot of this without the developer having to directly implement it. A lot of this could be done just by creating a default interface for manipulating the UI as a sort of DOM thing, and providing default support for synchronization, which is one of the big killers of trying to script a UI, knowing when a task is actually done and you can safely send the next command. (Or maybe someone implements <a href="https://news.ycombinator.com/item?id=49482709">https://news.ycombinator.com/item?id=49482709 , which would also as a side effect improve the ability to synchronize for scripting uses as well.)
We've had so many stabs at this over the years, in Applescript, in expect CAD interfaces, in the ideas behind Powershell, in so many places, but in my opinion what has always killed them is the barrier to entry. Now that's gone. Anyone should be able to say "Take this spreadsheet, search for these rows, extract these columns, send them to Photoshop as a text input here, save the result as a PNG and send it to my AI with this prompt, send the result to my printer and tell my accounting software to complete order #blahblah and send the bill" nowadays, except for the fact the desktop system will buck and fight you every inch of the way because it's just not built for this.
Though what will actually happen is AIs will simply get so good at manipulating desktops in general that they beat any possible project like this to the punch. Even so, even an AI would prefer a genuine, guaranteed programmatic interface to a virtual mouse and keyboard, and be more reliable with one, for all the same reasons we would in their position. And it doesn't have to be a big bang, the AIs can use this when it's available and grease it over for programs that can't do this, and it will increase the reliability of the whole pipeline.
Yes, that's one of the swings we've taken at this idea. It had some success offering components, but I don't recall ever being able to use it to drive an app very well. I think I tried to drive Office with it back in the late 90s and it was too buggy to use for anything at all. I don't see any reason why in principle it shouldn't work, and maybe there was a point it did, but historically these APIs decay like crazy because they never attain critical mass.
jerf · · focus · HN ↗
There's no reason a GUI toolkit couldn't be written to enable a lot of this without the developer having to directly implement it. A lot of this could be done just by creating a default interface for manipulating the UI as a sort of DOM thing, and providing default support for synchronization, which is one of the big killers of trying to script a UI, knowing when a task is actually done and you can safely send the next command. (Or maybe someone implements <a href="https://news.ycombinator.com/item?id=49482709">https://news.ycombinator.com/item?id=49482709 , which would also as a side effect improve the ability to synchronize for scripting uses as well.)
We've had so many stabs at this over the years, in Applescript, in expect CAD interfaces, in the ideas behind Powershell, in so many places, but in my opinion what has always killed them is the barrier to entry. Now that's gone. Anyone should be able to say "Take this spreadsheet, search for these rows, extract these columns, send them to Photoshop as a text input here, save the result as a PNG and send it to my AI with this prompt, send the result to my printer and tell my accounting software to complete order #blahblah and send the bill" nowadays, except for the fact the desktop system will buck and fight you every inch of the way because it's just not built for this.
Though what will actually happen is AIs will simply get so good at manipulating desktops in general that they beat any possible project like this to the punch. Even so, even an AI would prefer a genuine, guaranteed programmatic interface to a virtual mouse and keyboard, and be more reliable with one, for all the same reasons we would in their position. And it doesn't have to be a big bang, the AIs can use this when it's available and grease it over for programs that can't do this, and it will increase the reliability of the whole pipeline.
doodpants · · focus · HN ↗
jerf · · focus · HN ↗
someonebaggy · · focus · HN ↗
jerf · · focus · HN ↗