Lecture 3: Learning Faces

Question 1

points to cover in an intro

Answer 1

• routes of face recognition and stats of eye-witness testimony
• well established we are poor -at recognizing unfamiliar people
• human observers bad at matching different images of unfamiliar faces (Bruce et al,m 1999, 2001; Clutterbuck and Johnston, 2002, 2004)
• humans in realworld settings highly error-prone in matching a live person to photo-ID (Kemp et al., 1999)
• this is no better for experienced professionals (passport officers) than or untrained individuals (Kemp et al., 2014).
• IN CONTRAST
• recognition of familiar faces is excellent, remaining robust even in highly distorted or degraded images (Burton, 1999)
• Given the discrepancy between familiar and unfamiliar faces, it is imperative to understand how representations make the transition from unfamiliar to familiar
• rather little is still known about this process

Question 2

Where does the modern psychological study of face recognition have its roots?

Answer 2

The modern psychological study of face recognition has its roots in the problem of eyewitness testimony. In the 1970s, it became clear that witnesses to a crime could very often be mistaken when subsequently asked to remember someone involved.

Question 3

Stat to use to introduce the importance of remembering different faces

Answer 3

In the innocence project a US organisation aiming to exonerate wrongfully convicted individuals 73% of cases they have overturned originally used Eye witness testimony as key evidence- thus understanding factors that can influence facial recognition is highly important.

Question 4

field evidence for humans being bad at matching photos to faces

Answer 4

Kemp et al., 1997
• Field test
• Supermarket check-out staff were recruited to validate the photo-credit cards by deciding whether or not the photograph was of the person presenting the card.
• Even though the staff were aware that they were taking part in a study concerning the utility of photo-credit cards, they performed surprisingly poorly.
• About half of the fraudulent cards were accepted, and about 10 per cent of the valid cards were falsely rejected.

Experienced passport officers show poor levels of matching ability, no better than untrained students (White et al., 2014).

Question 5

lab based evidence that humans are bad at recognizing unknown individuals

Answer 5

Davis & Valentine (2009) asked participants to match live persons to CCTV clips. Observers were highly error-prone on this task, even when the CCTV footage showed high quality, recent, close-up sequences

Question 6

when does the problem of unknown person recognition persist and study for this

Answer 6

The problem persists when viewers are asked to compare static photographs. In a pioneering demonstration of this, Bruce et al. [1999, 2001]
• devised a task designed to model a best-case scenario for identifying images captured on security video.
• Participants were shown an array of 10 faces along with a target face. Viewers were asked, for each array, whether or not the target person was present among the 10 candidates, and if so, to point out the match.
• In the original experiments, the target was present on half of the trials, and absent on the other half.
• One way to think of the array is as a photographic version of police line-up. As with real line-ups, all the faces fit the same general description (they were all clean-shaven young men with short hair).
• Participants performed surprisingly badly on this task, with error rates of 30 per cent in both target-present and target-absent conditions.
• This poor performance is especially striking given that the photos were all taken on the same day, precluding changes in hairstyle, weight or health, and showed the face in frontal aspect under excellent lighting

Question 7

experiment for matching unfamiliar faces

Answer 7

Jenkins et al. 2011
• 40 cards with two different people but participants don’t know is two different people
• Give cards p’s and ask them to sort into clusters into how many identities they see.
• Exp. 1: Participants sort 40 photos of 2 unfamiliar identities (20 images per ID) so that photos of same face were grouped together.
• Participants did not know the correct number of IDs and were free to create as many clusters as they wished.
• Median number of image clusters = 7.5 (range 3-16). None arrived at the correct solution which was 2!
• Problem of integrating “dissimilar” images of the same person, errors of sorting two different IDs into the same pile were infrequent.
• This is surprisingly difficult- people can’t tell that two dissimilar images are the same person.
•

Question 8

evidence that we are good at matching familiar faces

Answer 8

Jenkins et al., 2011
Exp. 2: same task/same images, but participants were familiar with IDs.
• Almost all participants performed perfectly!

Question 9

what is facial invariance

Answer 9

being able to tell a face is the same from all angles/ lighting/ expressions

Question 10

how might we understand how facial invariance is achieved?

Answer 10

Understanding how faces are learnt may mean understanding how image-invariance is achieved!

Question 11

what is the core problem of face identity learning

Answer 11

how do we get to the stage where we can recognise faces in a number of different photos/ angles ect

Question 12

are people good at picture recognition study?

Answer 12

yes

• Bruce and Young (1986): Difference between picture recognition (i.e., remembering a specific photograph)- people are actually quite good at this, and face recognition (remembering the face itself) (the part which people are not very good for novel faces/ much worse than other first condition)

Question 13

what is it hard to see differences in for unfamiliar faces

Answer 13

• For unfamiliar faces, it is hard to see which differences between images are due to actual differences between the faces (emotion expression, different makeup, different hair), and which are due to image properties, e.g., changes in lighting or viewing angle (all these variables change at the same time).

Question 14

talk about representations of once-viewed familiar faces

Answer 14

• Representations of once-viewed unfamiliar faces contain characteristics of the particular picture (incident) and generalise only weakly to novel views (Bruce, 1982). If you see a person once you will remember an image of them only under a particular lighting/ hairstyle ect. Hard to translate this initial view to something different.
• Hard to get all of these extra information out of our head and see the real identity.

Question 15

what parts of the face are important for face learning?

Answer 15

internal and external features

Question 16

first study on internal/ external features

Answer 16

Bruce et al., 1999
- In further matching experiment with unfamiliar faces, targets were either full faces, only internal features (eyes nose mouth) or only external features (hair line, ears, jaw)
- Performance in external and whole conditions highly similar if you give people the full image with everything and just the external features which means that accuracy in whole face and external are very similar.
 Participants use external features (e.g., hairstyle) to match unfamiliar faces and that they do not care too much about the internal features. Try to use some superficial feature instead of actually using the face itself. People seem to know that info within the faces can look very different so seem to focus on these superficial features instead.

Question 17

internal and external feature use study in faces becoming more familiar

Answer 17

Bonner, Burton, & Bruce (2003): The first study where people were asking how then faces do become familiar.

• Participants learnt novel faces from 30s video clips or three different static images on three consecutive days.
• Familiarity tested in matching task with photo and still from video showing either internal or external features. Had to match one pic from the learning face and match against a photo of the same or different person using either just internal or just external features.
• People were quite good at matching the external features from the start and this didn’t improve over the three days.
• Matching of internal features for learnt faces was not good at the beginning btu improved over three days of training, matching of external features did not.

Question 18

what can we say overall about studies using internal/ external features

Answer 18

 Face learning results in identification based on the face itself, not superficial features (such as hairstyle etc.). indicates that if you learn a face then you actually start to know the face.
Face learning is a shift from the external features of the face to the internal. People learn to use the information from the face itself.

Question 19

explain the theory behind needing experience and seeing an image only once

Answer 19

Idea would be that need experinece
- If image variability is necessary for face learning, even massive exposure to single image should not be helpful to learn a structural representation. If this is the case then it shouldn’t really make a difference whether you see one picture once, and only have experience of one image- idea is that performance shouldn’t be dramatically better if you see the same image over and over again.

Question 20

experiments that show need extended exposure

Answer 20

Longmore, Liu and Young, 2008

Lui et al., 2009

Question 21

describe the first study on learning faces and exposure

Answer 21

• Longmore, Liu, & Young (2008), Exp. 1:
• Participants learn single image, either in a single exposure or with additional training.
• Recognition test with learned image, after lighting change, and after pose change
• Recognition of same image > lighting change > pose change
• Results:
• People are best when they are tested with exactly the same image that they learnt, and people are worse with a lighting change and still worse with a pose change. Lighting change is easier to compensate than viewing angle.
• Pattern similar in single- and multiple exposure conditions when experience of a single image.
• Longmore, Liu, & Young (2008):
• If we learn a single image how far can the face be rotated until performance drops.
• Exp. 2: Performance decreased if test view is rotated from learnt view by only 15° (not dramatically but goes down). Past this decrease more and more.

Question 22

what does the - Longmore, Liu, & Young (2008) study support the idea of?

Answer 22

Supports the idea that if you want to have the image-invariance recognition, you need experience of how different the person can look. Need experience of how individuals look.

Question 23

what are the pros of the Longmore, Liu, & Young (2008) study?

Answer 23

Very systematic very controlled stimuli in a lab setting- but those with a bigger focus on ecological validity argue that if you want to test face learning your faces should vary in a large number of dimensions. Below used this for face learning research

Question 24

describe the second study on learning faces and exposure

Answer 24

Liu et al., 2009
showed that extensive training with different images of the same face presented in multiple head angles led to accurate recognition of the same face presented in a different illumination. However, training with different illuminations did not produce high accuracy when the same face was presented in different head angles. The authors demonstrate that pose plays a more important role than illumination in forming generalizable representations.

Question 25

what overall do the Longmore and Liu studies suggest

Answer 25

The Longmore et al (2008) and Liu et al (2009) studies suggest that learning of unfamiliar faces, measured by subsequent recognition, is dependent on viewpoint. These studies are consistent with work on perceptual learning.

Question 26

what adds further evidence to the idea that we need more experience of faces

Answer 26

that technical developments have not been able to solve this either

Question 27

which authors proposed the ideas about technology showing that exposure is needed

Answer 27

Jenkins and Burton, 2011

Question 28

what overall do Jenkins and Burton suggest about recognising unknown faces

Answer 28

Neither humans nor machines can perform this task reliably. Although human perceivers are good at matching familiar faces, performance with unfamiliar faces is strikingly poor. The situation is no better for automatic face recognition systems

Question 29

what two ideas to Jenkins and Burton 2011 propose as to why we cant identify unknown individuals

Answer 29

resource limited OR data limited

Question 30

talk about resource limitation

Answer 30

That is to say, the problem is solvable in principle—we just have not solved it yet. By this account, eventual success is just a matter of developing better procedures in the case of human performance, or better algorithms in the case of machine performance. This is presumably the conviction that has spurred the field on for some three decades

Question 31

talk about data limited

Answer 31

Data limited- That is to say, no amount of investment in matching procedures or matching algorithms will lead to useful levels of performance. This is a genuine possibility if performance is limited not by processing power or ingenuity, but by the information that is available in the image.

Question 32

what would be the case if humans were resource limited?

Answer 32

if was resource limited then this problem should be solvable with computers

Question 33

talk about technical developments

Answer 33

In parallel to early psychological theorizing about face recognition, technical developments in image processing made possible the study of automatic, computer-based face recognition.

Question 34

evidence for computer use for unknown facial recognition being good

Answer 34

In optimal lab conditions accuracy levels of machines can be high, (Phillips et al., 2006)

Question 35

but what is the problem with computer use?

Answer 35

but The problem is that when conditions are not so favourable (as in border control or surveillance settings), or cooperation is poor (for example, when someone is trying to conceal his or her identity), performance plummets.

Question 36

thus what has often been the issue with research into facial recognition of strangers

Answer 36

• we accept it can be done in principle
• proliferation of ‘biologically inspired’ approaches to automatic face recognition reflects the willingness of computer engineers to model the brains success.
• psychological studies however have shown that human expertise in face identification is far more narrow than often assumed
• the process that most automatic systems attempt to model lies outside this narrow expertise.

Question 37

why ultimately do machine models fail

Answer 37

disappointment in machine systems is inevitable, as they model a process that fails.

Question 38

give a example outside facial recognition of data limitations

Answer 38

Plenty of examples outside of face recognition that are data limited in this way, consider the word ‘bank’. We can analyse this arrangement of letters for as long as we like. The letters alone will never reveal whether the referent is a financial institution or the side of a river.

Question 39

what would be the response if matching unfamiliar faces are data-limited?

Answer 39

If matching unfamiliar faces is a data-limited problem, one response would be to stop trying to match photographs, and instead try to develop alternative face representations that are better suited to the task

Question 40

why is there little evidence to suggest that the human visual system would be well suited to processing facial snapshots?

Answer 40

Given the recency of portrait photography in human evolution, there is little reason to expect that the human visual system should be well-suited to processing facial ‘snapshots’, as these only occur in the context of photography. In natural samples, the facial image changes from moment to moment, as well as from year to year. This variability provides an opportunity to separate aspects of appearance that are common across all the images (and hence potentially diagnostic of identity), from those that are transient (and hence specific to a particular image).

Question 41

thus what is the solution to researching facial learning not using single snaphots

Answer 41

research on stable face representations has focused on a very simple proposal based on averaging together several photos of the same face. - Jenkins and Burton, 2011

Question 42

how does Jenkins and Burtons facial morphing work?

Answer 42

In this technique, multiple photographs of each face are collected from existing sources (e.g. the Internet). These ambient images are intended to capture a natural range of variation in facial appearance.
Facial landmarks are recorded (e.g. corners of eyes) and are then morphed onto a standard template which are then morphed together to create an average shape. Average texture from all the photos are calculates and the average texture is added to the average shape to produce the stabilized facial image.

The process thus dilutes aspects of the image that change from one photo to the next, while preserving aspects of the image that are consistent across the set.

Question 43

what happens even if two faces are merged together?

Answer 43

The process thus dilutes aspects of the image that change from one photo to the next, while preserving aspects of the image that are consistent across the set.

Question 44

what may morphing phases provide a good model of and where does evidence for this stem from?

Answer 44

the refinement of the average, driven by increased exposure, may provide a useful model of face learning.

Evidence for this comes from behavioural experiments in which we compare recognition performance for famous faces presented in average image or standard photograph format.

Question 45

study supporting use of morphs for better facial recognition

Answer 45

Burton et al., (2005) In name verification tasks, observers are presented with a famous name followed by a face. Their task is to press one key if the face matches the preceding name, and another key if the face belongs to someone else. In our experiments, the face was equally likely to be presented as a standard photograph or as an average image. We assumed that face recognition involves matching the seen face to a representation stored in memory, and that the speed of recognition indexes the closeness of that match, with faster reaction times indicating closer correspondence.
Whether correctly accepting a match, or correctly rejecting a mismatch, responses were faster for average images than for photographs.

Question 46

study with more photos being added to the average morph and the effect of this

Answer 46

Jenkins and Burton, 2008 identification of average images becomes increasingly efficient as more and more photos contribute to the average

Question 47

what do Jenkins and Burton’s behavioural findings provide?

Answer 47

Taken together, these behavioural findings provide quite compelling support for the notion that an average image of an individual’s face is a relatively good match to a familiar observer’s mental representation of that face, compared with a photograph.

Question 48

what is surprising that happens with average morphs of known faces

Answer 48

Given that perception of familiar faces is already extremely efficient, even when based on photographs, it may be surprising that any representation can improve on this performance. The average nonetheless seems to capture the essence of a person’s appearance in a way that facilitates identification

Question 49

what do Burton and Jenkins call an average face morph

Answer 49

A robust representation

Question 50

what does a robust representation do?

Answer 50

helps you identify face from any picture (/angle/ lighting ect) it is needed for face recognition

Question 51

explain how robust representations are proposed to be formed?

Answer 51

Basic idea is that if you average something you get rid of the noise and are left with something that is systematically there.

If you have a trial with lots of irrelevant information (e.g. lighting/ angle/ expression), all these are random and change from image to image but there must be something that does not change from image to image.

Idea that humans average out all the random changing factors and are just left with what is very individual about that face and what is really critical for the identity.

Question 52

who studied variability of faces? what was the idea behind their study?

Answer 52

Ritchie And Burton, 2017

The idea is that if you need variability for learning then you should be better with the strongly varying images & exactly what they found.

Question 53

what did Richie and Burton, 2017 do?

Answer 53

previous studies used images that vary systematically on dimensions such as pose or illumination.

Tried to compare a situation where a more controlled but less variable learning condition (lighting same, hairstyle same maybe some different expressions/ mouth movements) with a situation where you have strongly varying pictures (taken on different occasions, hairstyle, lighting different).

Question 54

what did Richie and Burton, 2017 find?

Answer 54

• People better at making correct judgment in high variability condition relative to low variability and also faster at making a correct judgement- more efficient recognition. But: exposure to “natural” variability may critical for face learning!
• Participants learn new faces in “high variability” versus “low variability” conditions.
• Performance in speeded name verification task is more accurate and faster in high variability condition.
• Difference in accuracy isn’t vastly different but should keep in mind that in the condition that they call ‘low-variability’ there still are quite a lot of differences between the images but just less. In real life situations when meet someone, are more likely to encounter what is in the low-variability context ie. different expressions and mouth movements as opposed to extremely different lighting and hair styles.

Question 55

studies that show variability

Answer 55

Richie and Burton 2017

Murphy et al 2015

Question 56

Murphy et al., 2015 study on face variability what did they do?

Answer 56

• Participants saw the same eight identities on each trial, with either the same six images of each identity repeated on each trial,
• or with a new six images of each identity appearing on each trial (96 unique images of each identity extra)- thus wider range of variability

Question 57

Murphy et al., 2015 study on face variability what did they find?

Answer 57

Exposure to variability within each identity led to a trial-by-trial decrease in estimation of the number of identities present, showing that participants were able, over time, to cohere together the images of each identity. Participants were also better at subsequently identifying the people for whom they had seen more images and variability, supporting previous work showing improved learning with increased numbers of exposures

Question 58

what do Andrews et al. 2015 , Murphy et al., 2015 and Ritchie & Burton (2017 studies show?

Answer 58

• Exposure to different instances allows building robust representations that are not picture-specific.

Question 59

what do Andrews et al. 2015 , Murphy et al., 2015 and Ritchie & Burton (2017 studies not provide us with?

Answer 59

Looking from more of a theoretical angle- have seen empirical evidence that exposure to variability is important but haven’t explained what the mechanism is behind this.

Question 60

but what is important to consider about saying variability isn’t important?

Answer 60

Burton et al., 2016:
By getting rid of all the variability- you are probably losing something- it is key to know how a face looks in different pics. There might be something to learn in variability, not all variability probably, but some part of how a face changes systematically when we see different pictures of that face might be relevant for you seeing who that person is. So if you know how different a face can look it will certainly help in recognising the face.

Question 61

what was the idea of Burton et al.’s., 2016 study? what did they use?

Answer 61

Idea of the current study is to find out which kind of factors/ dimensions vary systematically in different pictures of the same person. Can use PCA for this.

Question 62

what did Burton et al., 2016 do?

Answer 62

• Use of Principal Component Analysis (PCA) on 30 images of 10 Hollywood actors to find dimensions on which pictures vary, separately for shape and texture (more or less). PCA is a statistical procedure.

Question 63

what does PCA give?

Answer 63

• PCA gives a number of differnt components that explain the varaince in the image set.

Question 64

what did Burton et al., 2016 find in terms of early components?

Answer 64

• Early components (explaining most variability) code left-right rotation in all identities and other rigid head motions. These are relatively superficial things, in terms of head shape, a lot of the variabiltiy in difference is explained by face angle/ rotation. For texture a lot of the variability is explained by lighting and which direction (left or right) the light is coming from.

Question 65

what did Burton et al., 2016 find in terms of later components?

Answer 65

gets more interesting if you look at later components

• Later components introduce non-rigid motion (e.g., emotional expressions). Ridgid motion would be a head turn non-ridgid would be an emotional for instance.
• From component 4 onward, variability is idiosynchratic, i.e., components code something different for different identities. Whereas the first 3 or so are very similar for all the identies from 4 onwards different things happen for different people, and you get changes in the images that vary wtihin the specific images of a person that only occur for that person. This is something that is idiosyncratic and something that may be learnt about for that specific face. For example one of the individuals showed eye gaze and mouth opening together ie. When changed gaze eye opened so knowing this variability on this dimension this is useful.

Question 66

overall what does Burton et al.’s 2016 study tell us?

Answer 66

• Different identities vary on different dimensions!
• And: Even common early components, which are the same for everyone carry identity information!
• First codes will always be viewing angle/ lighting—but head rotations for different individuals will look different as the shape of individuals faces are different and same with lighting. So learning about variability in these early stages is equally as useful.

Question 67

what does Burton et al’s 2016 study lead us to conclude?

Answer 67

Overall idea is that we know what the central tendency of someone is (their average) but it is also useful to know how variable someone is. So to learn about someone face you need to know about both a characteristic centroid and characteristic variability.
 Establishing a robust representation may mean to learn both about the average and the characteristic variability of a given face.
These are the newer theoretical ideas about face learning

Question 68

what correlate studies have been done on facial learning?

Answer 68

Kaufmann et al., 2009- Video of a face speaking changing viewing angle

Andrews et al., 2017- Used more naturally varying images

Question 69

what did Kaufman et al., 2009 do?

Answer 69

Video of a face speaking changing viewing angle. Then in test face see faces of individuals in learning phase and also completely new faces. In test phase faces in each block a different pic of the same face is shown but with limited variability as clearly same session with same camera so at the same time whilst the pics do vary compared to the other condition they are similar to one another.

• Explicit learning of pre-experimentally unfamiliar faces.
• Recognition from different images in subsequent test blocks

Question 70

what did Kaufman et al., 2009 find?

Answer 70

• N250 becomes gradually more negative across blocks.

- More negative N250 may reflect establishment of an image-independent representation.

Question 71

what is the interpretation of Kaufman et al’s 2009 study?

Answer 71

Idea is that ERP N250 gets more and more negative from block one to block three and then 3 and 4 gets more negative for each block. Might also be because the system saturates there is not much info in the later blocks compared to the first one. Shows that the effect builds up with experience. N170 is sensitive to image repetition and it seems that because some aspects of the images are similar the changes also occur in the N170.

Question 72

what did Andrews et al., 2017 do?

Answer 72

Used more naturally varying images

• Sorting of 40 images from 2 identities into two piles
• During EEG recording: Presentation of images seen in sorting task, different images of identities seen in sorting task, new identities, and famous faces.

Question 73

what did Andrews et al., 2017 find?

Answer 73

• N250 lowest for famous faces, and least negative for new identities with the two sort task individuals/ faces in the middle. N250 more negative-going for learnt identities (sorting task), no difference in N250 for same vs. different images of learnt identities. This is relevant because it is a neural correlate of image invariant recognition the difference doesn’t care if you’ve seen the very image before but codes identity not a particular image.

Question 74

conclusion

Answer 74

The critical point of face recognition is that you recognise a face from nearly any image, to establish that you need images that look very different for learning or instances that look very different. The mechanism behind that might have something to do with building a central tendency/ representation using averages but as well as knowing how the face changes.