Translation and Substitution Flashcards
What is sensory translation? What’s the purpose of it?
Tranferring information from one sensory modality into another
Can be used for multisensory art, sensory substitution, marketing, covert communication etc.
How could we translate between the senses?
Translation could be through synaesthesia, crossmodal correspondences, perceptual similarity (idea that similairty exists across senses) or structural mapping (map senses based on common similar structures e.g., high to low)
Does evidence suggest we can translate between the senses?
History of translation attempts e.g., colour ogans, gamut of odours (perfurme scents to musical notes)
Some patterns in how people map hue-pitch but also disagreement - doesn’t work based on syneaesthesia, similarity or structural mapping
Walkers crisps packing for different flavours had different instruments and colours but didn’t match what people associated with in lab, suggests translation doesn’t really work
People found to associate different types of wine with different music, suggests translation can work
Affective crossmodal correspondences to map senses through emotion best approach
What is sensory substitution?
Using one sensory modality to replace or augment another sneosry modality that is impaired or absent
What is the purpose of sensory substitution?
Accesibility aids for sensory loss/impairment e.g., many blind, lots of olfactory dysfunction post covid
Enjoyment from augmentation e.g., Collins 1970 tactcile television and biohacking
Business e.g., Hennessy 1966 stock market figures in tactile displays for traders
Explain vision-touch SSDS
Early developers argue makes sense as both spatial, vibrations or electrical stimulation give haptic representations “draw” the image
Started with dentist chair style using back but poor acuity
Guarniero - anecdote, experience of falling from looming sensation and automatic withdrawal when camera fell
Then devices using forehead and tongue e.g., brain port with which able to identify shapes and numbers
Brain port anecdote - said to be similar to picking up a langauge
Explain vision-hearing SSDs
vOICe translates shapes to auditory chirps as passes over image e.g., higher pitch corresponds with lighter pixel, duration with width
Amedi at al. 2007 - shape conveyed by vision-hearing substituion devices shwon to activate lateral occipital cortex (same active in sighted discriminating shapes)
Hamilton and Fletcher 2016 - colour through sound based on crossmodal correspondences improved performance compared to reverse pairings
Explain the psychological limitations of vision-touch SSDs
Spence 2014 - fundamental limitations due to constraints on processing of tactile information
Gallace et al. 2012 - capacity for information processing (amount of cortex take up, attentional capacity, number of sensors and afferents) vision on top and touch much less doesn’t have enough bandwidth
Explain the technical limitations of vision-hearing SSDs
Has more bandwidth than touch so less psychologically limited
Hearing isn’t as spatial as touch, can’t draw shapes, reliant on abstract sensory translation using pitch volume duration to represent brightness height and width, difficult to represent/requires more training
Explain the issues with lab studies for SSDs
Participants identify what “see” through alternative modality and nothing else/no other stimulation
In real world need to work with split attention - doing more than one task and other stimulation/distractions
Spence 2014 - applications limited to specific purposes e.g., navigation, unlikely provide general purpose substitution
Explain the issues with affective qualities and SSDs
Spence et al. 2017 - substitution doesn’t offer affective/emotional qualities of target modality
Bach-y-Rita 2002 - anecdote blind using vision-touch identify details of playboy centerfolds but not hedonic affects like sighted peers
Especially for chemical - knowing something tastes sweet doesn’t give pleasurable experience of sweet, particularly limiting as olfactory impairment has worst mental health consequences of any sensory loss (Spence 2021)
Explain issues with user experience of SSDs
Practicality/ ease of use - bulky devices not easy to use walking around day to day, limit what can do with other senses e.g., wearing headphones hear less of outside, tongue haptics makes harder to talk, require extensive training to depend on, limited to one type of input
Comfort - Elli et al. 2014 suggest ergenomics one of biggest issues with SSDs, electrical stimulation can be sensory issue/painful
Aesthetics - many do not wish for accesibility aids to make stand out more than neccary and nicer to use something like the look of
Explain the use of Crossmodal correspondences in SSDs
Hamilton-Fletcher et al. 2016 - colour to sound SSD based on correspondences improved performance compared to reverse pairings, but not large effect - easier to remember colours of items but not much easier to identify objects
Hamilton-Fletcher et al. 2018 - blind have different correspondences to sighted - pitch-size and pitch-weight preserved in blind, pitch-texture and pitch-softness only found in blind, less clear distinctions in late blind
Suggests SSDs using crossmodal correspondences can be beneficial but need to be suited to user’s correspondences
Explain the cane as an SSD and why it is commonly used but not other SSDs
Practicality/ease of use - gives multiple types of input - tactile feedback, auditory cues and propreoceptive information witout cutting off other senses- understand state of environment in relation to body
easier to learn as less complex/ more intuitive than other SSDs - makes easier to use as less cogntive load (easier interpretation)
more reliable as less maintenance, more widely available with more support in using and low cost
adaptapble for different environments and people e.g., different types of tip for best feedback on different terrian or best feedback for that person
portability
Comfort - different types/handles to suit different people, better for sensory issues as not having to wear anything
Aesthetics - established use/cultural acceptance means less stand out, seeing increase in good looking canes
Future - Smart canes - use technology to add additional layers of sensory substitution , better to improve what’s in use and working and learn from that than make something from ground up
