CASIA OpenIR  > 模式识别国家重点实验室  > 多媒体计算与图形学
Being a Supercook: Joint Food Attributes and Multimodal Content Modeling for Recipe Retrieval and Exploration
Min, Weiqing1; Jiang, Shuqiang1; Sang, Jitao2; Wang, Huayang1; Liu, Xinda3; Herranz, Luis1
AbstractThis paper considers the problem of recipe-oriented image-ingredient correlation learning with multi-attributes for recipe retrieval and exploration. Existing methods mainly focus on food visual information for recognition while we model visual information, textual content (e.g., ingredients), and attributes (e.g., cuisine and course) together to solve extended recipe-oriented problems, such as multimodal cuisine classification and attributeenhanced food image retrieval. As a solution, we propose a multimodal multitask deep belief network (M3TDBN) to learn joint image-ingredient representation regularized by different attributes. By grouping ingredients into visible ingredients (which are visible in the food image, e.g., "chicken" and "mushroom") and nonvisible ingredients (e. g., "salt" and "oil"), M3TDBN is capable of learning both midlevel visual representation between images and visible ingredients and nonvisual representation. Furthermore, in order to utilize different attributes to improve the intermodality correlation, M3TDBN incorporates multitask learning to make different attributes collaborate each other. Based on the proposed M3TDBN, we exploit the derived deep features and the discovered correlations for three extended novel applications: 1) multimodal cuisine classification; 2) attribute-augmented cross-modal recipe image retrieval; and 3) ingredient and attribute inference fromfood images. The proposed approach is evaluated on the constructed Yummly dataset and the evaluation results have validated the effectiveness of the proposed approach.
KeywordCuisine Classification Recipe Image Retrieval Ingredient Inference Multitask Deep Belief Network
WOS HeadingsScience & Technology ; Technology
Indexed BySCI
Funding OrganizationNational Natural Science Foundation of China(61322212 ; National High Technology Research and Development 863 Program of China(2014AA015202) ; Beijing Municipal Commission of Science and Technology(D161100001816001) ; Lenovo Outstanding Young Scientists Program ; National Program for Special Support of Eminent Professionals ; China Post-doctoral Science Foundation(2016M590135) ; National Program for Support of Top-Notch Young Professionals ; 61532018 ; 61550110505 ; 61602437 ; 61373122)
WOS Research AreaComputer Science ; Telecommunications
WOS SubjectComputer Science, Information Systems ; Computer Science, Software Engineering ; Telecommunications
WOS IDWOS:000404056000017
Citation statistics
Cited Times:10[WOS]   [WOS Record]     [Related Records in WOS]
Document Type期刊论文
Affiliation1.Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China
2.Chinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China
3.Beihang Univ, State Key Lab Virtual Real Technol & Syst, Beijing 100191, Peoples R China
Recommended Citation
GB/T 7714
Min, Weiqing,Jiang, Shuqiang,Sang, Jitao,et al. Being a Supercook: Joint Food Attributes and Multimodal Content Modeling for Recipe Retrieval and Exploration[J]. IEEE TRANSACTIONS ON MULTIMEDIA,2017,19(5):1100-1113.
APA Min, Weiqing,Jiang, Shuqiang,Sang, Jitao,Wang, Huayang,Liu, Xinda,&Herranz, Luis.(2017).Being a Supercook: Joint Food Attributes and Multimodal Content Modeling for Recipe Retrieval and Exploration.IEEE TRANSACTIONS ON MULTIMEDIA,19(5),1100-1113.
MLA Min, Weiqing,et al."Being a Supercook: Joint Food Attributes and Multimodal Content Modeling for Recipe Retrieval and Exploration".IEEE TRANSACTIONS ON MULTIMEDIA 19.5(2017):1100-1113.
Files in This Item:
There are no files associated with this item.
Related Services
Recommend this item
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Min, Weiqing]'s Articles
[Jiang, Shuqiang]'s Articles
[Sang, Jitao]'s Articles
Baidu academic
Similar articles in Baidu academic
[Min, Weiqing]'s Articles
[Jiang, Shuqiang]'s Articles
[Sang, Jitao]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Min, Weiqing]'s Articles
[Jiang, Shuqiang]'s Articles
[Sang, Jitao]'s Articles
Terms of Use
No data!
Social Bookmark/Share
All comments (0)
No comment.

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.