Google’s Gemini AI for Android is rolling out two new features that could change how users interact with their smartphones. The features, called “Ask about this screen” and “Ask about this video,” allow users to ask questions about whatever is currently displayed on their screen.
These tools capture a screenshot, process the content, and then provide relevant information based on the user’s query. The video feature, specifically designed for YouTube, also offers video summaries, provided the video includes captions.
The new features were spotted in Google app version 15.33.38.28.arm64. When a user summons the Gemini AI assistant, the tools appear in a floating window.
Users who have access to the features will see a rectangular strip with the text “Ask about this screen” alongside a screenshot icon. If the AI assistant is activated while playing a YouTube video, the text changes to “Ask about this video,” accompanied by the YouTube icon.
By tapping on the icon for “Ask about this screen,” the AI assistant captures a temporary screenshot within Gemini. Users can then ask any query related to the content on the screen.
It’s important to note that the responses are generated based solely on what the AI can interpret from the screenshot.
Users also have the option to prompt Gemini to access the Internet for more detailed information. For example, one might use this feature to ask for a news article summary on their screen.
Despite its innovative nature, the feature comes with limitations. The captured screenshot is not saved on the device, and long screenshots are not supported.
This means that while the AI assistant can provide quick insights or summaries, it is limited to the information contained within the visible part of the screen at the time the screenshot is captured.
The “Ask about this video” feature is more specialized, working only with YouTube videos that have captions. During testing, it was found that no other video platforms, regardless of whether captions were enabled, were compatible with this feature.
On YouTube, however, the tool allows users to ask questions about the video. The AI generates answers based on the captions provided, but it does not analyze the video content itself.
This approach highlights Google’s focus on utilizing existing metadata like captions, to enhance user experience without the need for more resource-intensive video analysis.
The introduction of these features is part of a broader rollout of Gemini’s floating panels, which aim to provide a seamless user experience without disrupting the current app.
Previously, when users asked the AI assistant to perform complex tasks, they were often taken to a separate window, away from the app they were originally using.
Now, with the floating panels, Gemini remains in its compact form, allowing users to continue interacting with their current app while still engaging with the AI assistant.
This shift towards a more integrated AI experience was first hinted at during Google I/O in May, where the tech giant showcased several upcoming Gemini features.
While some of these features have already been implemented, others are still in the process of being rolled out. The floating panels, although a small change, were necessary to pave the way for these context-based features, ensuring they work seamlessly within the existing Android ecosystem.
The “Ask about this screen” feature is particularly useful for quick, on-the-go queries. For instance, if a user is reading a news article and wants a brief summary, they can trigger the AI assistant and ask for a TL;DR version without leaving the article.
Similarly, the “Ask about this video” feature provides an efficient way to get a summary of a YouTube video without having to watch the entire content. This could be particularly valuable for users who want to quickly gauge the relevance of a video before deciding to invest their time in watching it.
Interestingly, the YouTube integration even allows users to bypass ads when using the AI assistant to summarize videos. Since the tool only relies on captions to generate answers, users can pause the video during a pre-roll ad and still receive a summary of the content.
This feature could become a subtle yet impactful way for users to avoid unnecessary interruptions, especially for those who do not subscribe to YouTube Premium.
The gradual rollout of these features has been observed across various devices running the Gemini beta app version 1.0.662093464 and the Google beta app version 15.34.32.
Users who have both betas installed should automatically gain access to the floating Gemini panel and its associated tools. As with many beta features, availability may vary, and some users may experience these tools before others.