D2S2: Drag ’n’ Drop Mobile App Screen Search

Dec 7, 2023

Speakers

About

The lack of diverse UI element representations in publicly available datasets hinders the scalability of sketch-based interactive mobile search. This paper introduces D2S2, a novel approach that addresses this limitation by facilitating mobile screen search through a drag-and-drop interaction paradigm, accommodating visual and text-based queries. D2S2 leverages a dataset of 58,000 Rico screens to retrieve relevant UI examples based on UI element attributes, in-including type, position, shape, and text. An evaluation involving ten novice software developers demonstrates the effectiveness of D2S2, as it successfully retrieves target screens within the top search results in 15 out of 19 attempts within a minute. The tool offers an interactive and iterative search experience, updating its search results each time the user modifies the search query. Interested users can freely access D2S2 (http://pixeltoapp.com/D2S2), build on D2S2, or replicate results via D2S2’s open-source implementation (https://github.com/toni-tang/D2S2), or watch D2S2’s video demonstration (https://youtu.be/fdoYiw8lAn0).

Organizer

Categories

Store presentation

Should this presentation be stored for 1000 years?

How do we store presentations

Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%

Sharing

Recommended Videos

Presentations on similar topic, category or speaker

Interested in talks like this? Follow ESEC-FSE