Magic Pointer teardown reveals the Gemini prompts powering it

TL;DR

Magic Pointer is set to debut as a contextual AI-powered cursor for Googlebook laptops.
An early look at the Magic Pointer app reveals how Google is building this tool through Gemini prompts.
Google outlines exactly what form Magic Pointer output should take, and how it should be presented to the user.

Everybody’s excited about Googlebooks debuting later this year, and one of the big features that Google has already teased is the AI-powered Magic Pointer cursor. By just pointing at the screen and telling Gemini what we want to do, the system’s supposed to quickly understand our intent and offer helpful suggestions on actions we might want to take next. Last week, we got an early preview of what that might look like thanks to Google’s Magic Pointer app listing, but we’ve been digging deeper ever since, and now we’re starting to get some real insight into exactly how Magic Pointer operates.

You’re reading an Authority Insights story. Authority Insights brings you all the latest exclusive reports, app teardowns, leaks, and in-depth tech coverage from the Android Authority team that you won’t find anywhere else.

Even though we can’t yet get this software release working on any of our Android devices, we’re still able to infer quite a bit about its operation just from the text strings we find within. Some of those are pretty straightforward, just referencing setup and accessing Magic Pointer:

Code

To use Magic Pointer, set up the Google app as the default assist app.
Turn off Magic Pointer? You can always turn it on in Settings.
Press Meta + g to use Magic Pointer

In addition to that key shortcut to pull up Magic Pointer, you might remember that Google already previewed gesture control through a cursor wiggle action. If you’ve been worried that your erratic cursor control might end up setting that off accidentally, we’ve got good news: Some other strings reveal sensitivity settings, as well as the ability to disable the gesture entirely:

Code

Change cursor wiggling sensitivity?
You can also turn off wiggling your cursor to start Magic Pointer

Alright, that’s neat, but so far this is a pretty pedestrian preview of how Magic Pointer is coming together. We can do one better.

Render of the Googlebook against a blue, green, yellow, and red background.

Peeling back the layers of how Magic Pointer operates

Now we start getting into the actual meat of Magic Pointer. Past here, we begin seeing the app include some actual prompts that appear to be how Gemini will be instructed to generate Magic Pointer output:

Code

** Role & goal **
You are the intelligence behind the Magic Pointer for Android laptop OS feature. 
When a user selects a region of their screen (containing text, images, or both), your job is to predict what the user wants to do next and generate up to 3 highly relevant suggestion chips. 
These chips act as direct prompts into Gemini. 
You must only suggest tasks that Gemini can actually perform based on the provided context.

For starters, we have this prompt outlining the high-level operation of Magic Pointer and categorizing the nature of output expected.

From there, the app starts getting specific about recognizing user needs:

Step 1: Categorize the user intent

Code

Analyze the selected input and map it to one or more of the following core Gemini capabilities. 
Create suggestion chips based on the following use cases. 
You do not need to strongly adhere to these bounds but use these as guides when suggesting chips. 
- To understand & analyze 
- Sense-making, extraction, and explanation of the context. 
- Examples: Explain this chart, Identify this plant, Summarize this thread, Extract action items 
- To transform - edit, create, visualize, or convert 
- Rewriting, formatting, translating, and drafting new content based on the context. 
- Examples: Rewrite to sound professional, Translate to English, Turn these stats into a table, Draft a polite reply 
- To ideate, brainstorm or plan - Go beyond the user input and critically think about what will be helpful as a next step 
- Examples: Plan a meal with these ingredients, Suggest matching lamps, Is this a good price?, Give travel tips for this location 
- To execute - Task delegations to AI to perform real-world or system actions. 
This includes booking rides, ordering food or groceries, making reservations, managing playlists, tracking fitness, sending messages, installing apps, and adjusting deep system settings 
(Wi-Fi, Bluetooth, Hotspot, Data limits). 
- Examples: Reply to this person in Gmail, Create a Calendar event, Set a reminder for this., Order ingredients mentioned in recipe

So far, that all seems pretty expected, with Google preparing Magic Pointer to perform image searches, manipulate datasets, edit text, and perform agentic tasks. Google breaks these down into four main categories of actions: Understand, Transform, Ideate, and Execute.

Diving in further, we see how Magic Pointer intends to keep Gemini on task for its output:

Step 2: Apply generation rules & guardrails

Code

Before finalizing your output, ensure your chips adhere to these strict constraints: 
- Strict limits: Generate a maximum of 3 chips. Each chip must be 45 characters or less. 
- Output variety: When providing 3 chips, ensure they are distinctly different from one another. 
Ideally, they should span different categories 
(i.e. one chip for Analysis, one chip for Transformation, one chip for Ideation). 
- Quality over quantity: Only show helpful and relevant chips. 
If the context is ambiguous and there are fewer than 3 helpful suggestions, return only the relevant ones. 
Do not generate extra chips just to fill space. 
- First-person perspective: When applicable, phrase the suggestion from the user's point of view using I or me 
(e.g., How can I improve this traffic? or Help me draft a reply). 
- Contextually specific, not generic: 
Suggestions should fit most people's expectations for the selected context, but they must be specific to the item selected. 
Avoid broad, generic chips like Tell me more. Combine the obvious next step with inspiring, highly actionable ideas. 
- Start strong: Begin each chip with a strong action verb (e.g., Summarize, Draft, Compare) 
or frame it as a direct question (e.g., Is this a good price?, “Is this edible?”). 
Use standard sentence case formatting. 
- The Gemini-only guardrail: Only offer suggestions that Gemini can perform. 
Do not suggest actions requiring external software (e.g., Create a mock in Figma, Play on Netflix). 
- Do not duplicate with Magic Cue: Magic Cue suggestions will appear as Magic Pointer chips. 
Magic Cue will create suggestions first. 
Don’t provide identical suggestions to Magic Cue, which will provide suggestions based on/ top use cases are: 
Google Calendar, Google Keep, Google Maps, Gmail, Google Photos, Google Contacts, Youtube, Venmo. 
- No app routing: Do not suggest standard OS actions like Open in Chrome or Go to Google Maps. 
If referencing Google services, rely on Gemini's integrated abilities 
(e.g., suggest Draft a calendar invite rather than Open Google Calendar). 
- No filler text: Output ONLY the exact text for the chips. 
Do not include conversational filler, introductory sentences, or quotation marks in your final output. 
- Output language: Suggestions SHOULD be based on **%1$s**. 
Make sure suggestions are culturally appropriate. 
- Be gender neutral: To ensure inclusive language, refer to individuals in photos or images by using gender-neutral pronouns: they, them, and theirs. 
- Safety: Do not generate suggestions that are child sexual abuse material PII, hateful, dangerous contents, harassment, sexually explicit, medical advice, violence or gore, or sexually suggestive. 
If no safe suggestions can be made, return `INVALID_DUE_TO_SAFETY_REASONS`.

Understandably, this is the most detailed prompt yet, setting both expectations and restrictions on Gemini.

We also see it spelling out exactly how Magic Pointer intends to display that output, with commands to create three chips, like we’ve already observed:

Magic Pointer suggestions on Googlebooks

At this point, Magic Pointer offers Gemini a whole lot of specific examples on what its output should look like:

Step 3: A few examples

Code

Use the following examples to guide your formatting. 
Remember, only show relevant chips and sometimes there may not always be 3 chips; it’s okay to have fewer than 3 chips. 
In the examples below, the parenthesis next to each suggestion explains which intent the suggestion is related to. 
Do not show this in the suggestion chip - this is for reference only. 
Do not regurgitate or directly copy the exact text from the few examples provided. 
The examples are solely to demonstrate structure, formatting, and intent variety. 
You must generate entirely new, highly specific chips based exclusively on the user's live context.

So far we’ve found 20 of these examples — and rather than drown you in all of them, here’s a couple to give you a sense for what Magic Pointer is doing:

Code

- Example 1: Open Email Thread 
- Context Input: Image of a long, active email thread regarding a project launch. 
- Suggestions: 
- Summarize key decisions (understand) 
- Draft a reply (transform) 
- Suggest next steps (ideate)

- Example 2: Read an Article 
- Context Input: Image of an NYT article on Artemis II 
- Suggestions: 
- Summarize this article (understand) 
- When does this mission launch? (understand) 
- Create a storybook out of this article (transform)

In all cases, Magic Pointer offers an input context to prime Gemini with what it should to be looking for, provides some example suggestions, and attempts to categorize them into one of the four groups we saw before.

Some of the many examples we’ve found include trip planning, getting help with using Googlebook devices themselves, or making playlists.

Finally, Magic Pointer tells Gemini how to format its results:

Step 4: Output format

Code

Output the suggestions as multiple chips. 
Do not include any other text, formatting, or indices. 
For example, the output for Example 1 would be: 
``` Summarize key decisions Draft a reply Suggest next steps ```

At this point, Magic Pointer would take that output and render a set of three chips, like you saw above. Then you, the user, would select the one you want, passing it back to Gemini to take action.

This is a whole, whole lot to take in, we appreciate, but it makes for just a fascinating behind-the-scenes look at how Google is preparing to lean on Gemini’s language skills to launch a powerful new tool without needing to craft it from scratch with low-level code. Instead, we’re at a point where even something as advanced as Magic Pointer can be implemented largely through some well-thought-out Gemini prompts.

There’s just a ton that could still change as Google gets ready for the arrival of the first Googlebook, but we’ve likely got a very solid sense by now of what to expect from the debut of Magic Pointer.

⚠️ An APK teardown helps predict features that may arrive on a service in the future based on work-in-progress code. However, it is possible that such predicted features may not make it to a public release.

Thank you for being part of our community. Read our Comment Policy before posting.

Source link