Photos Spare Cycles MythBusters

Notes: Automating the Design of Visualizations

Maneesh Agrawala, Julie Heiser, and Barbara Tversky Tutorial session at AAAI

Two implemented systems explored for automated design of visualizations: map routes and assembly instructions. Map routes system (LineDrive) used by MapBlast (now

Three parts to talk: cog sci/CS background, map routes, assembly instructions.

Overview Examples

Lots of examples from Tufte books (life cycle of beetle, Playfair balance for/against, train map to shrines) Ferrari cut away

6000 yr old babylonian map. oldest known map. certain characteristics in common for visualizations over the ages. thousands of years of user testing

growing wheat to making bread Egyptian hieroglyphics

Jerusalem maps - makes streets perpendicular and parallel even when not

Cognitive Psychology

Good visualizes schematize - show info you need, remove what you don't. * bamboo and seashell maps. * London Underground map. Directions have been schematized to horizontal, vertical, and 45 degree. distances distorted

Cognitive Design Principles

  • Principle of Apprehension: structure and content of graphic should be readily and accurately perceived and comprehended
    • visualizations consist of elements and spatial relations that can convey meanings directly.
  • Principle of Congruity: structure and content of graphic should match structure and content of desired mental representation

Spatial relations

  • categorical: spacebetweenwords
  • ordinal: lists, trees
  • interval: graphs
  • metaphor: map conceptual differences to differences on paper

Vertical direction loaded: strong/good/powerful things at top

Horizontal dimension is neutral: political left vs. right. exception: (sinister: left, dexterous: right)

Spatial metaphors can be mixed: napoleon's march, space used in three different ways (location, size of troops, temperature, time)


  • Icons: hieroglyphics, sometimes use part to represent whole or something associated
  • Figures of depiction
  • Morphograms: abstract geometric forms
    • from geometric or Gestalt properties
    • arrows in mechanical diagrams, lines/bars in graphs, lines/curves/crosses/blobs in route maps
    • lines: connect, link. tends towards trend descriptions
    • arrows: asymmetrical links, directionality, many uses/interpretations (temporal/causal/pointing/labeling/movement/change over time)
    • crosses: intersection of lines
    • bars: containers. tends towards discrete descriptions
    • blobs: regions

Arrows on mechanical diagrams

  • without arrows people gave structural descriptions
  • with arrows people gave functional description

Comparing lines and bars in examples given to people

  • Lines: trend descriptions
  • Bars: discrete descriptions

Route maps and route directions

  • are the schematizations similar?
  • asked students how to get to taco bell. asked half for map sketches, other half for verbal descriptions.
    • sketches: turns right angle, shrinked distances where less information req'd. use maps discretely rather than analog
    • written directions: bullet point lists, some complicated descriptions. all had same basic structure: start point, reorientation, path/progression, end point (segmentation according to Denis)

Graphic "Language"

  • Semantics: schematic elements that are not iconic. gestalt/geometrical meetings, categorical not analog, combinatoric.
  • Syntax: combine elements by rules

Five Step Plan

  • Identification 1) Production 2) Preference 3) Comprehension
  • Validation 4) Instantiation 5) Usability

Computer Science

IEEE visualization conference started in 1989. studying of visualization itself fairly new

Explanatory vs. exploratory visualizations * Exploratory: map of cholera outbreaks in Tufte's book, reveals clustering * Explanatory: most other maps, Napoleon's march


Data collection and dissemination is being sped up by computers, human designers can't keep up. More visualizations are being generated by computer => visualizations are regressing. Can we improve automated visualization?

Automation can also lead to customization, which can increase effectiveness. Take into account affordances of medium, relevance of data, context, personalization.

Artistic rendering can improve perception.

Automated Design

  • Page design: TeX
  • 3D object visualization
  • Data graphics presentations: APT (Mackinlay) automatically decides how to render as chart or graph
  • UI layout, label layout, VLSI design, camera planning, 2d/3d packing graph drawing

Need domain specific constraints, high level constraints. Where do you get these domain specific constraints?

Ideal system

communicative goals -> diagrammatic visualization

intent-based illustration: how to replace battery (Seligmann 91) * illustrator phase, drafter phase, illustration phase * design rules, style rules * evaluation phase leading out


  • generate and test: can include competing rules, evaluate on the other side
  • high-level approach: analyze most effective hand-designed layouts, identify design principles, transform into low-level design constraints, encode principles as constraints and algorithmically find design satisfying constraints

Designing Route Maps

Hierarchical organization

  • Stevens and Coupe: asked people what the direction is between two cities. example: San Diego east or west of Reno. People used spatial relationships of states first, then derived organization of cities. Another example: comparison between building sizes. People distorted buildings between different groups (University buildings vs. town buildings). People within group are considered more similar.


  • Holyoke and Mah
  • Got people from each coast, asked for distance estimates between pairs of cities. Exaggerated distances close to them.
  • Perspective is flexible

Landmark errors

  • Sadalla, Burroughs, and Staplin. Asymmetric/inconsistent distortions. distance of house to landmarks is shorter than distance from landmark to house.
  • Landmarks define regions and draw things into them. e.g. "I'm from SF (instead of Palo Alto)"

Alignment errors

  • Code figure relative to another figure or frame of reference
  • Figures remembered as being more aligned
  • Majority picked wrong map of US (aligned South America/Africa and North America/Europe). People think Philadelphia is north of Rome.


  • Tversky
  • code figure relative to frame of reference
  • Most people think Stanford is west of Berkeley because of bay. Santa Cruz is east of Stanford.
  • Axes of figure and of reference frame rotated toward one another. (e.g. El Camino/Central Expressway)
  • People upright South America
  • Examples seem to work across cultures (east/west)

Design principles for route directions

  • exact distance and direction are unimportant
  • context will disambiguated

Designing Route Maps (instantiation)

Hand-drawn maps much easier to use than computer-generated maps (e.g. mapquest)

Instantiating design principles

  • Maps must communicate turning points. verbal direction emphasize turns, hand-drawn maps depict turns.
  • Graphical elements of turn must be visible
  • Context is secondary to turning points. local context: cross-streets, landmarks. overview context: orient route to geography.
  • Geometry not apprehended correctly, geometry not drawn accurately (topology -- sequences of turns -- is accurate).

LineDrive: route map design system

Automating route map design

  • Layout problem

    • set of graphical elements (roads, labels, cross-streets)
    • choose visual attributes (position, orientation, size)
    • large space of possible layouts
    • distortions increases choices
  • Search-based optimization:

    • hard constraints: required
      • must fit in viewport
      • must maintain consistent turn direction
    • soft constraints: desired
      • each constraint given a weight
      • prioritize scores by importance (prevent topological errors, ensure roads visible, maintain original orientation, maintain ordering by length, maintain overall route shape).
      • prioritizes set by hand (rate errors that were most confusing)
    • need to choose relevant constraints
    • simulated annealing: perturb (form layout), score, minimize
      • initialize: uniformly scale route to view viewport
      • perturb: pick random road, rescale or reorient by random factor, rescale entire route to fit viewport


generalizations (emphasize turning points): * simplification (straighten roads, differentiates roads and turns) * exaggeration (unbunch for visual separation, keeps roads visible) * regularization (make distances more uniform, regularize turning angles but keep direction, reduce visual complexity)

  • displacement
  • aggregation


Designed to work with PDAs. Favors vertical layouts since PDAs scroll more easily in that direction.

Experimented with sketch-style maps to convey inaccuracy of representation

Assembly Instructions

  • how do people conceive of 3D assemblies
  • how do people comprehend visual instructions

Prior work

  • only one published study in cog sci, a couple in HCI on assembly instructions
  • Diagrams better for showing context, diagrams equivalent to text for showing actions (Bieger)
  • Step-by-step better than showing only end state, but don't need for simple assembly (Novick)
  • High spatial ability -> prefer diagrams only, low spatial ability -> prefer diagrams and text together (results mixed)

Overview of Five stages

Production Preference Comprehension Instantiaion Usability

Stage 1: Production

User test: assemble TV stand without instructions, write instructions for novice user. separated based on spatial ability and experience.

  • 98% contained diagrams, 62% contained text with diagrams
  • 3 classes of diagrams drawn: parts menu, structural diagrams, action diagrams
  • People with high spatial ability were about 5 mins faster (7.29 vs. 12.76) than people with low spatial ability.
  • People with low spatial ability had much more errors in instructions (86%) than high spatial ability people (12%).
  • Diagrams by people with high spatial ability have more detail, better separation of steps, perspective (3D). More likely to include actions diagrams (2.67 to 0.64)

Stage 2: Preference

User test: assembled TV stand without instructions. Asked to rate sets of redrawn instructions from previous experiments. * ratings similar between high and low spatial participants (0.97 correlation) * action diagram had highest rating * low rated diagram had action only represented in text, mixed uses of arrows

High-rated features: 1. Action diagrams 2. Step-by-step diagrams 3. Clear order of steps 4. Concise text 5. Views of relevant parts and attachments 6. Use of diagrammatic elements to indicate actions (ex. arrows and guidelines)

Stage 3: Comprehension

User test: given 1 of 4 instructions, assemble TV using instructions. Measure assembly time, errors, false starts, number of references to instructions, number of references to picture on box, post-assembly interview.

  • no difference in assembly time by conditions
  • low spatial people used box picture much more. both used instructions.
  • should show relevant parts and attachments
  • structural diagrams and exploded view hard to use
  • text not very useful

Design Principles

  • sequence assembly operations
  • ensure visibility of parts
  • illustrate assembly operations
  • avoid changing viewpoint
  • show parts in each step
  • use physically stable orientation
  • show mode and location of attachment

Stage 4: Instantiation

Two steps in designing instructions: planning (choosing sequence of assembly operations) and presentation (visually convey assembly operations)

  • want to do both planning and presentation together, contrary to previous trends in research that have separated the two into separate fields


  • geometric analysis to get blocking graph representation
  • many geometrically valid sequences
  • use design principles as constraints: step-by-step preferable, action diagrams preferable, ensure visibility of parts
  • input: geometry+, orientations (default viewpoint, preferred orientation+), groupings+, ordering constraints+ (+ optional)

1) Compute subset of parts to add * Choose based on visbility, grouping, ordering, attachment, interference * Search optimizations * reduce number for parts considered by checking hard constraints first (can also eliminate parts that require that part), grouping symmetric parts * breadth-first search with early exit * Compute visibility * check that each part in current subset is visible * check that context is visible * check that future parts will be visible

2) Sequencing * Add only one significant part per step (rank by visibility, distance to user), omit repetitive steps

3) Reorientation to make parts are more visible

4) Action diagrams * Choose direction, build stacks, add guidelines * Stacks: share separation direction, lie on a stabbing line, are in sequential contact.

Future Work

  • auto-generate insets
  • gravitational constraints
  • more parts
  • take advantage of hierarchy
  • apply recursively

Stage 5: Validation

user study: given 1 of 3 instruction sets (top-rated hand-drawn, factory, computer), assembled using instructions with no box picture, not told what assembling.


  • computer-generated fastest (not taking into account ability/experience)
  • computer-generated had lowest errors, though not statistically better than factory
  • computer-generated diagrams eliminated effect of spatial ability. high and low spatial performed equally well, and high spatial performaned as well as they did with factory instructions.

Future Directions (Cog Sci)


  • congruity: uses change in time to convey change in time
  • yet, animated graphics no better than equivalent static for instruction (tversky, morrison, betrancourt)
  • animations are hard to perceive (fast, complex, disappear), hard to conceive (people conceive of events discretely)
  • can correctly recognize, but can't produce correctly (e.g. can identify correct pendulum motion/release, but can't identify in static drawing)

Speculations * good animations explain, not show. they distort time and space and link different kinds of visualizations to tell a story. * clues from comics

Elements of demonstration * show step completion * exhibit large parts * point to small parts * make actions visible * model structure * model the action * conclude

Diagrams augmenting cognition * record and convey information * promote inferences * enable new ideas by externalizig common ground, platform * why?: spatial inferences easy (prox./distance/direction), spatial metaphors available (time/value/strength), elements interpretable (icons/etc...)

Future Directions (CS)

Understanding 3D environments

Walkthroughs vs. external views (artball), both have their obvious deficiencies in perceiving whole space

Use of transparency: hard to understand after 1 or 2 layers

Point of view: Floorplans, axonometric views, floorplans + front view * axonometric view best in helping understanding space * principles: 3rd person view, cutaways reveal interior

Exploded view * replaced OpenGL library to passively capture 3D data and generate map. e.g. Quake III: Arena. Can see players running around * experimented with transparency to reveal more internal structure

Applications * spectator modes in games, possibly incorporate tracking for real world space analysis * specialized maps (where is the library book, library tour)

Interactive exploded views * Li Gl 04 * pretty cool * comes with editor to create from 2D drawings. magic wand tool to segment as well as boundary tool to compute proper occlusion. can attach labels that automatically disappear when part occluded, search tool that explodes part to reveal part.

Other domains for future work

Medical illustrations Scientific diagrams (depict concept) Graph and charts Architectural plans (sketch->computer generated, etc...) Proof visualization

Post a comment


related entries.

what is this?

This page contains a single entry from kwc blog posted on July 26, 2004 2:10 PM.

The previous post was Notes: Stochastic Local Searches.

The next post is Comic-Con: Costumes.

Current entries can be found on the main page.