← Back to papers
2022 EMNLP EMNLP 2022

Lexi: Self-Supervised Learning of the UI Language

Abstract

AbstractHumans can learn to operate the user interface (UI) of an application by reading an instruction manual or how-to guide. Along with text, these resources include visual content such as UI screenshots and images of application icons referenced in the text. We explore how to leverage this data to learn generic visio-linguistic representations of UI screens and their components. These representations are useful in many real applications, such as accessibility, voice navigation, and task automation. Prior UI representation models rely on UI metadata (UI trees and accessibility labels), which is often missing, incompletely defined, or not accessible. We avoid such a dependency, and propose Lexi, a pre-trained vision and language model designed to handle the unique features of UI screens, including their text richness and context sensitivity. To train Lexi we curate the UICaption dataset consisting of 114k UI images paired with descriptions of their functionality. We evaluate Lexi on four tasks: UI action entailment, instruction-based UI image retrieval, grounding referring expressions, and UI entity recognition.

🌉 Interdisciplinary Bridge — Artificial Intelligence and Computer Vision and Deep Learning and Machine Learning
📈 Trend Setter — Multi-Modal Learning
🧭 Keyword Pioneer — ui representation
🐣 Hot Topic Early Bird — vision language model
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio

Authors

Pratyay Banerjee , Shweti Mahajan , Kushal Arora , Chitta Baral , Oriana Riva

Topics

Artificial Intelligence > Core AI > Multimodal Learning Machine Learning > Learning Types > Self-Supervised Learning Computer Vision > Core AI > Multimodal Learning Deep Learning > Techniques > Self-Supervised Learning Deep Learning > Learning Types > Self-Supervised Learning Machine Learning > Learning Paradigms > Self-Supervised Learning Artificial Intelligence > Core AI > Multi-Modal Learning Deep Learning > Models > Multi-Modal Learning

Keywords

self-supervised learning multimodal learning image captioning vision language model vision-language model user interface ui representation visio-linguistic learning accessibility application ui component understanding visio-linguistic representation
Download PDF

Related papers

Bilingual Lexicon Induction for Low-Resource Languages using Graph Matching via Optimal Transport 2022
Generative Entity Typing with Curriculum Learning 2022
Towards Reinterpreting Neural Topic Models via Composite Activations 2022
Weakly Supervised Headline Dependency Parsing 2022
Cross-modal Transfer Between Vision and Language for Protest Detection 2022