A framework to enable multimodal models to operate a computer. Using the same inputs and outputs as a human operator, the model views the screen and decides on a series of mouse and keyboard actions ...

Gear featured in the animation includes a GTX 1080ti graphics card, i7-8700k CPU, Asus Prime Z370-A motherboard, Samson Go Mic microphone, and Logitech G600 mouse.