Google Showcases "Magic Pointer", an AI Cursor that Understands Screen Content

Google’s artificial intelligence division, Google DeepMind, has showcased its latest technology dubbed “Magic Pointer”. As the name suggests, this feature, introduced alongside the “Googlebook” device, is an AI-based cursor designed to understand the context and objects that users are pointing at on the computer screen. Google states that the technology was developed to make user interactions with AI more natural and practical, without the need to type long commands or switch applications. Magic Pointer works by capturing the visual and semantic context around the cursor’s position. This means the AI can “see” the part of the screen the user is focusing on and understand the important information in that area. For example, a user can simply point the cursor at an image of a building and say “show directions to this location”. The AI system will then understand the context of the image without requiring additional explanation. In addition to finding location information, Magic Pointer can perform various other tasks typically handled by AI. For instance, it can summarise PDF documents, correct writing or sentences being typed or pointed at, or convert statistical data into graphs. Users can also modify food recipes displayed on the screen according to commands, or book a restaurant being shown in a travel video. In one demonstration, Google showed how a paused video frame can directly turn into a restaurant booking link. As summarised by KompasTekno from 9to5Google, Magic Pointer is one of the features embedded in Googlebook, Google’s conceptual laptop that combines elements of Windows, Chromebook, and Android. Nevertheless, this feature will also be available in the Gemini service integrated into the Google Chrome browser in the future. With Magic Pointer, Chrome users will be able to point to a specific part of a web page and ask Gemini about that area. For example, users can select several products on a shopping site and ask the AI to compare their specifications. Users can also point to a living room area on the screen to virtually visualise a new sofa. To try out Magic Pointer, users can access the feature demo in AI Studio. Currently, Google provides two demos, namely for editing images and finding locations on maps.