Multisensory Integration

Question 1

Perception to action

Answer 1

Psychophysics- present sensory stimuli and report if they felt anything

Lighting a match- seemingly easy, reaching, grasping, manipulating objects

Seemingly hard logic puzzles

Computers are good at hard tasks

Reach/grasps- robots are bad at seemingly easy tasks

Question 2

Moravec’s paradox

1998

Answer 2

High level cognitive reasoning tasks- easy for robots

Low level cognitive tasks, perception- hard

Computers show adult level performance on intelligence test/ playing checkers

To give them skills of 1yrs old concerning perception and mobility

Bias- we assumed if these tasks were easy to us they would be simple for robot

Question 3

Lighting match

Answer 3

Anaesthetise finger- loss of touch sensation from fingers - sight still there

Does not affect motor control

When sensation is blocked, much harder to light the match

25s- block sensation
5s- normal

Shows vision and sensation very important for simple task

Question 4

Multi sensory integration

Answer 4

Touch and vision integrate

Question 5

Challenges in multi sensory integration

Answer 5

Transforming representation from different senses to common representational space

Integrating info from different senses into coherent percept -> mismatch

Question 6

Reference frame

Answer 6

Representation schemas of information from different schemas

Snake game:
Player perspective: coordinates of snake in the game-top-down view

Snake perspective: see the world through its eyes- turn left/ right

Problem: sensory input in player, control in snake-> need to know info about body layout of snake

Question 7

Reference frames for different senses

Answer 7

Vision = eye centred/ retinal- location of visual stimulus on retina

Audition= head centred- location of sound source in respect to ears

Touch= body centred- location of tactile stimulus on skin

Need to convert between the reference frames and to external space

Question 8

Coordinate transformation

Answer 8

Dog barking and seeing dog= separate reference frames

How to convert between two and to know the difference between the two you have to know

Angle between two

Converting between frames- have to know position and orientation of body parts

Question 9

The body schema

Answer 9

Spatial coded: position of each body part in external space

Modular: different body parts processed in different brain regions

Updated with movement: automated and continuous tracking of body posture

Adaptable: changes when body changes

Supramodel: combines input from proprioception, touch and vision

Coherent: resolves perceptual conflicts

Interpersonal: observed actions are represented within the same body schema

Question 10

Types of body representation

Answer 10

Body schema: sensorimotor representation that guides action

Body image: body percept, body concept, body affect -> how we think/ feel about our current body

Question 11

Doe Body posture affect perception

Answer 11

Temporal judgement task

Stimulate both hands in random order, pots have to stretch fingers of hand stimulated

Arms crossed / arms uncrossed

When arms crossed- ppts mix up which hand was stimulated

Solving task- do not need input from body schema when arms uncrossed

Body schema interferes with basic perception

Question 12

How does a body schema develop

Answer 12

6mnth old
Schema starts to interfere with tactile orienting
Shown when crossing/uncrossing feet and buzzing one

4mnths- no difference I’d crossed, reach for right foot

6mnth- baby reaches for correct foot, more correct when feet uncrossed

Question 13

No arm crossing

Answer 13

Tactile discrimination task- determine which finger was vibrated

Visual distract either on same hand or other hand

Distractors led to response delays

Congruent distracts lead to longer delays than incongruent

62ms vs 20ms

Question 14

Arm crossing

Answer 14

Tactile tumulus in same side of body, visual stimulus on different side

Effect of visual distractor moves with the hand during arm crossing

Cross modal interactions mediated by body schema

Opposite response

67ms vs 3ms

Question 15

Peri personal space (PPS)

Answer 15

Space immediately surrounding our bodies

Objects in PPS can be grasped and manipulated immediately

Question 16

Tool use

Extending the body

Answer 16

Tools are incorporated into the body schema

Cross modal congruency effects apply during tool use

No crossing of body parts only tools= same delay effects

Tools become part of the body schema, represented same in brain

Question 17

Alice in wonderland syndrome

Answer 17

Distortion in perception of size

Body parts might appear smaller (micromatognosia) or bigger than they are (macrosomatognosia)

Affects whole body

Associated with childhood and migraines

Question 18

Autotopagnosia

Answer 18

Unable to locate body parts

Loss of spatial unity of body

Patients can name body parts but order is lost
Can’t point out where body part is, unaware of how it looks

Finger agnosia- fused percept of finger, can not indicate what finger was stimulated

Question 19

Phantom limb

Answer 19

Still feel presence of limb even after loss of limb

May include agency/ movement

Associated with pain

Can change size over time- shrink / telescoping

Question 20

Double dissociation

Answer 20

Causes decomposition or concept of body representation

Question 21

Cross modal neurons

Answer 21

Neuron fires when you put an object within range/ touches of the hand- respond when seen is touched / object moves near hand

Neuron responds to visual and tactile stimuli

Visual receptive friend moves with the position of the hand

Modified by body schema

Question 22

Neurons incorporating tools

Answer 22

When monkey holding tool response space expands

Expansion of Peripersonal space during tool use reflected in neural response

As response field expands, body schema encompasses the tool

Question 23

Integration problem

Answer 23

To see something- represented in external space

To hear- auditory input- represented in a different reference frame

Have to view according to both these inputs -> sensory conflict

Question 24

Sensory conflict

Answer 24

Different senses might provide conflicting information about a sensory stimulus

Needs to be resolved

Question 25

Testing conflicts between vision and touch

Answer 25

Judge size of object by vision and touch

Look through reducing lens

If we follow vision object looks a lot smaller

If we go by touch object feels a lot bigger

Question 26

Visual capture

Answer 26

Assess the size of a cube via pointing, feeling, drawing

Vision dominates perceived object size- visual capture- trust visual sense much more

Question 27

Sensory hierarchy?

Answer 27

Auditory can dominate vision

I report no. of visual flashes seen

Auditory beeps played during flashes

No. of auditory beeps determines reported no. of visual flashes

Question 28

Modality precision hypothesis

Answer 28

Modality with highest precision (lowest uncertainty) is chosen dependent on task

Spatial task- choose vision as highest accuracy

Temporal task- audition much higher accuracy

Question 29

Sensory uncertainty occurs due to

Answer 29

Perceptual limits- visual resolution determined by spacing of photoreceptors in fovea

Neural noise- synaptic noise

Cognitive resource limits- attention

Question 30

Sensory modality changes

Emst and banks (2002)

Answer 30

Created artificial conflict- judge height of bar

Visual height and haptic height

Virtuality reality set up- bar not there- illusion

Change height of bar, modify uncertainty by adding visual noise

Haptic- force feedback device, changes height of bar, judging size of bar by touch

Vision and haptic input can be conflicting

Question 31

Normative model

Answer 31

How a problem should be solved -> optimal solution

Process model- how a problem is actually solved- based on data

How to solve problem of sensory integration

Pick integration method- minimises sensory uncertainty

Haptic (haptic uncertainty) + vision (visual uncertainty) -> integrated signal (smallest possible, combined uncertainty)

Need to integrate both to come up with best possible answer

Question 32

Integrating probabilities

Answer 32

Consider the probability

When considering bar height- high variance and high sensory uncertainty
-> estimated height has large range

Different senses different levels of uncertainty

When a sensory conflict occurs there are 2 distributions- a haptic and visual
-> both come with their own levels of uncertainty

Question 33

Optimal estimate

Answer 33

Combine lower uncertainty and smaller variance -> estimate alone

Combine both= lower variance

Combined estimate is biased towards the visual estimate as it has lower uncertainty than haptic estimate

Question 34

Optimal weights

Answer 34

Can be calculated from variance (sensory uncertainty) of the visual and haptic distribution

Question 35

Minimal variance

Answer 35

Combined sensory variance

Integration of info from multiple sources

Causes uncertainty to decrease

Question 36

Experimental evidence

Answer 36

Present virtual bar with sensory conflict, looks visually longer than how it feels (haptic)

Compare against bar without conflict and see what length is judged

Visual= 60mm, haptic= 50mm

Determine point of subjective equivalence

Manipulate sensory uncertainty of visual feedback

Before visual noise, perception of bar length biased towards visual input

Below 50- use haptic and about 50- visual input

Increased visual noise, perception of bar determined by both visual and haptic input

Nosier the visual input the more we rely on haptic

Question 37

Human performance

Answer 37

Follows optimal sensory integration rules

Should move from visual to haptic capture when visual noise is large

Question 38

Do we always integrate info optimally?

Answer 38

Have to know uncertainty for optimal integration:
Can be hard to estimate
Easier in sensory perception than cognitive reasoning

Calculations can be intractable/ take long time:
Heuristics are suboptimal
Good enough solutions often satisfactory

Question 39

Integrate info optimally

Answer 39

Good estimating sensory noise, bad estimating cognitive noise

task- estimate tilt of gratings

Add sensory noise, add cognitive noise

Ppts integrate sensory noise optimally but cognitive noise sub optimally

Question 40

Are probabilities encoded

Answer 40

Uncertainty represented with prob distributions

Confidence signals

Representation of full probability- normal distribution

Question 41

Correspondence problem

Answer 41

How do we know there is only one stimuli in the first place

How do we know where are not 2- one that we can see and one that we can hear