Smoothing Grounding and Reasoning for MLLM-Powered GUI Agents with Query-Oriented Pivot Task

Domain adaptation, particularly through grounding techniques, is widely adopted for multimodal large language models (MLLMs) to enhance the perception of graphical user interface (GUI) environments, further improving the reasoning performance of MLLM-powered GUI agents. However, in resource-constrained scenarios, the format discrepancy between coordinate-oriented grounding and goal-oriented reasoning limits the adaptation effectiveness. To investigate a supplementary domain adaptation method that can further improve GUI perception and reasoning, we propose a query-oriented pivot approach called Intended Query Inference (InQuIre). From an information-theoretic perspective, InQuIre serves as a semantic pivot task that maximizes the mutual information between visual representations and user intent. By explicitly deducing the intended queries behind action coordinates, InQuIre provides dense supervision to align coordinate-indicated visual features with functional intent, offering superior data efficiency compared to sparse coordinate supervision. Experimental results focusing on mobile platform show that InQuIre outperforms grounding under the same adaptation data scale. Notably, InQuIre achieves comparable performance to large-scale grounding-adapted OS-Atlas with less than 0.1% of adaptation data. Furthermore, through experiments in cross-platform scenarios, InQuIre demonstrates its generalizability. Additionally, we explore the impact of reasoning formats and highlight that integrating additional semantic information into the input further boosts reasoning performance. Finally, through attention visualization, we validate that InQuIre effectively guides the focus toward functionally relevant GUI elements, ensuring robust comprehension of user intent. The code is publicly available at https://github.com/ZrW00/GUIPivot.