VTDP: Reasoning with a Hybrid of Mental and External Imagery

Almost all visual thinking is a process of reasoning where external imagery in combined with mental imagery. Mental imagery is used to represent alternative interpretations or possible additions to an external visualization. Visual queries are executed on the combined external/internal image.

Example: A very simple elegant example is provided by a study by Shimojima and Katagiri (2008). They showed people a liagram containing blocks A and B with block A above block B. They then told subject to consider the case of block C above block A and finally asked "Is block C above or below block B?"

This simple reasoning task could be carried out by purely logical thinking. But in fact the subjects solved the problem perceptually they imagined block C above A and then "saw" in the combined mental+physical image that A was above B.

 

Example: Looking at a social network where link width denotes frequent communication we mentally add the frequent communication between people to all wide links.

The combination of external and mental imagery is fundamental to data visualization. The basic operations are.

Mentally tying attributes to graphical elements of visualization

For example node symbols represent people in a social network diagram and links represent communications. Or node symbols represent cities in a map and links represent roads. A closed loop of a contour may be taken to represent the definition of a set of the symbols, or rather what the symbols denote, contained within it. The general process is as follows.

Reasoning with a display combined with mental attribution of semantics

Display Environment: A diagram or other visualization representing part of the solution to a problem.

  1. Perceive task relevant patterns in display and mentally add semantic attributes.
  2. Mentally image an addition to the display that will help with the visual reasoning process.
  3. Execute visual queries on the combined internal/external image to solve problem. .

Cognitive guideline: For complex symbolizatons a diagram can only be an effective reasoning tool if the symbol system is learned to an extent that the semantic attribution is automatic.

Cognitive guideline: For creative thinking those aspects of a diagram subject to reinterpretation should use abstract symbols where the semantic atribution is weak. For example, a line can have many meaning while generally denoting some form of connection or division.

 

Forming mental images and adding them to external marks to reason about some problem

An common process involving the combination of mental images and external marks is design sketching. We will use this process to

Design sketching

Display Environment: A blank screen (or paper) and a sketching implement

  1. Mentally image some aspect of a design.
  2. Put lines on paper to capture aspects of the imagined design.
  3. Construct analytic visual queries to determine if design meets task requirements.
  4. If a major flaw is found in the design as represented, that cannot be easily fixed (by erasure or other graphical correction), discard sketch. 5.
  5. Otherwise, if a design begins to meet task requirements
    • Mentally image design additions to the sketch, and/or
    • Mentally re-attribute the meaning of particular lines,
    • Using visual queries critically assess value of mentally imaged additions in the context of existing sketch.
  6. Repeat from 5 as needed or discard sketch and begin from 1.

Visual semotic guideline: Graphical marks should be have an informal quality and be amenable to multiple interpretations. Lack of a precise geometic shapes has been shown to help.

 

 
    Visual Thinking Design Patterns are partially funded by the DARPA XDAPA project