Skip to main navigation Skip to search Skip to main content

Scene grammar in human and machine recognition of objects and scenes

  • Akram Bayat
  • , Anubhaw Kumar Nand
  • , Do Hyong Koh
  • , Marta Pereira
  • , Marc Pomplun
  • University of Massachusetts Boston

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

In this paper, we study the effects of violating the high level scene syntactic and semantic rules on human eye-movement behavior and deep neural scene and object recognition networks. An eye-movement experimental study was conducted with twenty human subjects to view scenes from the SCEGRAM image database and determine whether there is an inconsistent object or not. We examine the contribution of multiple types of features that influence eye movements while searching for an inconsistent object in a scene (e.g., size and location of an object) by evaluating the consistency prediction power of the trained classifiers on fixation features. The results of the eye movement analysis and inconsistency prediction reveal that: 1) inconsistent objects are fixated significantly more than consistent objects in a scene, 2) the distribution of fixations is the main factor that is influenced by the inconsistency condition of a scene which is reflected in the ground truth fixation maps. It is also observed that the performance of deep object and scene recognition networks drops due to the violations of scene grammar. The class-specific visual saliency maps are created from the high-level representation of the convolutional layers of a deep network during the scene and object recognition process. We discuss whether the scene inconsistencies are represented in those saliency maps by evaluating their prediction powers using multiple well-known metrics including AUC, SIM, and KL. The results suggest that an inconsistent object in a scene causes significant variations in the prediction power of saliency maps.

Original languageEnglish
Title of host publicationProceedings - 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2018
PublisherIEEE Computer Society
Pages2073-2080
Number of pages8
ISBN (Electronic)9781538661000
DOIs
StatePublished - Dec 13 2018
Event31st Meeting of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2018 - Salt Lake City, United States
Duration: Jun 18 2018 → Jun 22 2018

Publication series

NameIEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops
Volume2018-June
ISSN (Print)2160-7508
ISSN (Electronic)2160-7516

Conference

Conference31st Meeting of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2018
Country/TerritoryUnited States
CitySalt Lake City
Period6/18/18 → 6/22/18

ASJC Scopus Subject Areas

  • Computer Vision and Pattern Recognition
  • Electrical and Electronic Engineering

Fingerprint

Dive into the research topics of 'Scene grammar in human and machine recognition of objects and scenes'. Together they form a unique fingerprint.

Cite this