← Back to papers
2022 CVPR CVPR 2022

Learning Program Representations for Food Images and Cooking Recipes

Abstract

In this paper, we are interested in modeling a how-to instructional procedure, such as a cooking recipe, with a meaningful and rich high-level representation. Specifically, we propose to represent cooking recipes and food images as cooking programs. Programs provide a structured representation of the task, capturing cooking semantics and sequential relationships of actions in the form of a graph. This allows them to be easily manipulated by users and executed by agents. To this end, we build a model that is trained to learn a joint embedding between recipes and food images via self-supervision and jointly generate a program from this embedding as a sequence. To validate our idea, we crowdsource programs for cooking recipes and show that: (a) projecting the image-recipe embeddings into programs leads to better cross-modal retrieval results; (b) generating programs from images leads to better recognition results compared to predicting raw cooking instructions; and (c) we can generate food images by manipulating programs via optimizing the latent code of a GAN. Code, data, and models are available online.

🌉 Interdisciplinary Bridge — Artificial Intelligence and Computer Vision and Deep Learning and Machine Learning
🧭 Keyword Pioneer — cooking recipe
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio

Authors

Dim P. Papadopoulos , Enrique Mora , Nadiia Chepurko , Kuan Wei Huang , Ferda Ofli , Antonio Torralba

Topics

Artificial Intelligence > Core AI > Multimodal Learning Machine Learning > Core Methods > Embedding Learning Machine Learning > Learning Types > Self-Supervised Learning Computer Vision > Core AI > Multimodal Learning Deep Learning > Learning Types > Self-Supervised Learning Deep Learning > Learning Types > Multi-Modal Learning

Keywords

image generation self-supervised learning multimodal learning cross-modal retrieval program synthesis graph embedding cross-modal embedding program generation joint embedding cooking recipe
Download PDF

Related papers

UniCoRN: A Unified Conditional Image Repainting Network 2022
Why Discard if You Can Recycle?: A Recycling Max Pooling Module for 3D Point Cloud Analysis 2022
All-in-One Image Restoration for Unknown Corruption 2022
Stability-Driven Contact Reconstruction From Monocular Color Images 2022
Forecasting Characteristic 3D Poses of Human Actions 2022