Lecture 2: Good old fashioned AI

Question 1

What 5 conclusions were drawn in the 2020 updat of the “How much intelligence is there in artificial intelligence?”

Answer 1

• Recent AI breakthroughs, such as deep learning and reinforcement learning, have deep roots in psychology.
• Modern AI models are much more human and brain like at the implementational level.
• There is nothing wrong with AI’s crystallised intelligence, but generalisation is still a weakness of AI systems.
• The psychological relevance of AI extends to areas such as the study of individual differences and cognitive development.
• We expect fruitful interactions with regard to the measurement of natural and artificial intelligence

Question 2

How does Haan define intelligence?

Answer 2

The ability to deal with difficult and novice problems

-similar to adaptivity and creativity

Question 3

How does the timescale of intelligence compare to the timescales of other adaptive behaviours?

Answer 3

Fast:

• Extremely slow (evolutionary time): cuckoo
(brood parasite)
• Slow (hours to weeks): operant conditioning
(skinner box)
• Fast (seconds to hours): problem solving
(intelligence)

Question 4

What did A.D. de Groot contribute to psychology

Answer 4

Wrote the book ‘psychological methods’ and ‘the thinking of the chess player’ in which he describes how a chess player plays chess (e.g from recognising strategies with experience, not just fluid intelligence.) Seen as one of the founders of the modern cognitive revolution in psychology

Question 5

Why does chess play such an enormous role in AI?

Answer 5

Because when people thought about AI, they thought about chess. They thought that if they could make them play chess then they could be intelligent. Almost like a Turing test

Question 6

How did they initially try to programme chess in the 1950’s?

Answer 6

It was much more difficult than expected. They interviewed master chess players and tried to implement human thought processes into the algorithm, which turned out unsuccessful

Question 7

Why was programming chess difficult? (2)

Answer 7

There is an explosion of possibilities (novel positions, 5,362 distinct chess positions after 3 moves, 71,862 chess positions after 4 moves), and how to determine what is a good position. It was not possible to compute all of these positions

Question 8

Describe two other simpler problems

Answer 8

Water jug problem: Get 4 litres exactly from a 3l and 5l jug and a tap.

24 problem: given 4 numbers and have to use each number once mathematically to get another number (e.g 24)

Question 9

What is the difference between simple and difficult problems?

Answer 9

Simple problems are problems for which the time to solve the problem does not increase fast when the problems becomes bigger (P: polynomial time)

Hard problems: the solution
times increases very fast (NP: non-polynomial time)

Question 10

What is meant by NP complete problems?

Answer 10

The solution is quick and easy to identify but finding the solution requires NP time with algorithms. If one problem is solved then they are all solved.

Question 11

What questions arise from NP complete problems?

Answer 11

How to make NP problems solvable in P time (no one knows how to solve this). Also how do humans solve these problems

Question 11

What questions arise from NP complete problems?

Answer 11

How to make NP problems solvable in P time (no one knows how to solve this). Also how do humans solve these problems (creative process)

Question 12

How is how we solve these problems studied? (3)

Answer 12

Error analysis, eye movements and thinking aloud protocols (introspection)

Question 13

What is the classic AI algorithm for problem solving? Describe this

Answer 13

Tree algorithms: Set up a search tree based on the options possible after each decision, search the tree to some depth (if possible to the end) and make a decision based on this.

Question 14

How can a search tree algorithm deal with an explosion of options?

Answer 14

Alpha-beta pruning: It stops evaluating a move when at least one possibility has been found that proves to be worse than a previously examined move

Question 15

What is meant by minimaxing? Explain with example

Answer 15

The tree search often involved a minimax algorithm which would explore different options of the algorithm and weigh them relative to the other options e.g first branch is evaluated and scored 0, any better branches are scored 1 and any worse are scored -1.

For example given two choices each of which would lead to two other choices which had two options: the maximum value that could be obtained from these two options would be would be computed. The minimum value of these two options would then be saved for that choice and the same for the other choice stemming from the original choice. The maximum value between these two would then be computed to see which of the two had the maximum value even if the opposition played the best move.

Question 16

What is a function classically used for problem solving in AI? Describe this

Answer 16

Evaluation functions: In chess and other complex examples you don’t have to search the tree to the end, which would be impossible because of the explosion of possibilities. At some depth, see 10 moves, simply counting the pieces
(queen =10, rook=5, pawn=1 etc.), suffices.

Simple evaluations combined with deep search in trees (see up to
20 moves) is very powerful. A deep complex chess concept like ‘a better pawn structure’, just means winning pieces of giving mate 10 moves later

Question 17

Name and briefly describe 2 more (aside from tree algorithms and evaluation functions) tricks used early in AI research for problem solving

Answer 17

Build in a huge opening book: In chess some openings are studied 20 moves deep

End game table bases: for positions with less than 6 pieces, they ‘solved chess’, because the tree up to the end position (win, draw, loss) could be computed

Question 18

Give an example of a well known early algorithm which exemplifies these traits

Answer 18

Deep blue: Chess algorithm constructed by Microsoft around 1990 which beat the current chess champion which was a huge shock

Question 19

Chess was therefore solved using AI but how did computers and humans differ in their approach to chess? (4)

Answer 19

Computers use memory to store lots of known positions (openings, endings). Humans struggle to store lots of opening knowledge.

Computers search deep (hundreds of millions of positions) within smart tree techniques. Humans only consider a few positions, not very deep.

Computers use simple evaluations (count pieces) while humans use more complex evaluations (e.g intuition)

Humans learn while computers do not

Question 20

Describe the general idea of an expert system and how it would work

Answer 20

Following the success in chess, the idea of expert systems is using AI to replace experts.

Knowledge bases would be constructed through thinking aloud protocols with human experts. The AI would extract from the knowledge base using if then statements, apply that to a rules engine which would utilise decision trees and logical programming. Non-experts could then give a request as input on a user interface and receive advice as an output according to these algorithms.

Question 21

What did deep blue teach us about how humans play chess?

Answer 21

Deep blue algorithms taught us more about what we don’t do in chess rather than what we do. It showed us how amazing it is that we can play chess given our limited insight a couple moves deep. It can be argued that we still don’t know how humans play chess.

Question 22

How well did expert systems work

Answer 22

They worked pretty good but not great.

Question 23

What were the advantages of expert systems?

Answer 23

They would provide answers for decisions, processes and tasks that are repetitive, hold huge amounts of information and minimise employee training costs. They would also centralise the decision making processes and make things more efficient and reduce human errors. They could also combine various human expert intelligences and provide strategic and comparative advantages that might create problems for competitors. They would also look over transactions that human experts might not think of.

Question 24

What were the disadvantages of expert systems?

Answer 24

Ultimately it was disappointing considering how expensive it was to build the systems and is not easy to automate complex processes. There was no common sense used in making decisions, a lack of creative responses that human experts are capable of. They were not capable of explaining the reasoning and logic behind decisions. There was no flexibility to changing environments and they were not able to recognise when there is no answer.

Question 25

What is the central feature of current AI?

Answer 25

Learning

Question 26

What two main types of learning are there currently in AI?

Answer 26

Deep learning neural networks (analog with the brain); supervised and unsupervised learning

Reinforcement learning; learn actions that are rewarded

Question 27

How are neural networks in AI related to psychology?

Answer 27

They are all based on psychological or neurological studies and models.

Question 28

What did the first neural networks look like?

Answer 28

Quite basic, essentially just inputs and outputs

Question 29

What created a renewed interest in neural networks? What difficulty came with this?

Answer 29

The first neural networks could do certain types of things but not more powerful things, people got disappointed and left. The discovery of hidden layers allowed people to do more powerful things but training the ‘weights’ was more difficult. Training the weights becomes more difficult the more there are. You need a smart way to change the weights so that they slowly become better and better.

Question 30

The limits on these networks with a hidden layer also became quite disappointing, however (e.g forgetting A), and the field became silent for 10-15 years. What discoveries bolstered the field again?

Answer 30

The finding that adding (way) more hidden layers made the algorithms much more powerful. Faster and more powerful computers allowed this to be possible. Using a series of GPUs (graphical processing units) working in parallel rather than the classic CPUs helped make these computers more powerful.

Question 31

What is meant by reinforcement learning in AI?

Answer 31

Rewarding algorithms for correct or good responses/ outputs and punishing them for incorrect responses/ outputs.

Question 32

What name is giving to the network with transforms input frames to output actions in AI?

Answer 32

Policy network

Question 33

Describe one of the simplest ways to train a policy network

Answer 33

Policy gradients: you start off with a completely random network. You train that network by having the algorithm play through a frame of the game engine (for example), when it produces a random action in response you send that action back to the game engine and it produces the next frame. The output is sampled from a probability distribution. The idea is that the algorithm will optimise its policy to receive as much policy as possible, hopefully improving its performance.

Question 34

What is meant by the credit assignment problem?

Answer 34

A whole sequence of events can work well in the game right up until one mistake costs the game. This whole sequence of events is now marked negatively. This makes algorithms very sample inefficient and require a ton of training time. Sometimes, when the sequence is quite complex, the algorithm fails completely as a random sequence of events rarely or ever gets the optimal result.

Question 35

Name and describe a solution to the credit assignment problem.

Answer 35

Reward shaping: A solution would be to only mark the action, not the episode negatively. In the case of pong, only the actions that lead up to hitting the ball really matter. Nothing after the ball is hit should matter. In a more complex game sequence, rewards can be given at certain stages on the way to the optimal solution.

Question 36

What problems are associated with reward shaping?

Answer 36

Reward shaping is a custom process that needs to be redone for every new environment; for every new game. This is time consuming and surprisingly difficult.

One reason why it’s difficult is the alignment problem: When you shape your reward function, algorithms often find surprising ways to get reward without doing at all what you want it to do.

Question 37

How is training a reinforcement agent a balancing act?

Answer 37

Negative reinforcement in reinforcement learning can help ensure
that a model is kept away from undesirable actions, but it can’t
really make a model explore desired actions. Training a reinforcement learning agent involves balancing two different metrics: exploration and exploitation. This is critical. This is the same with us e.g you might know one way to bike to UvA that works but there might be a better way. You can stay on that way or you could explore too much and never find the way to the lecture.

Question 38

Describe the concept of Q learning. How is it linked to expoitation and exploration?

Answer 38

Plays a huge role in reinforcement learning. It involves a ‘state by action’ table called a Q table (a grid marked by numbers and letters on the x and y axis. For example it could be that a person on one grid piece is trying to find a goal on a different one by moving up, down, left or right. At the beginning the grid is empty and the person makes random decisions since there is no reward, however, at some point he randomly finds the goal. He has now learned one thing: if he is in eg E4 and takes a step down, he will be in the goal. This is now entered in the Q table.

This process repeats, learning for example that if it is E5 and moves down then it is in E4. It now gets a partial reward for getting into that rewarded state. It builds up different rewards for different states based on the outcomes of deciding these states. Slowly it builds towards finding a consistent route. It may settle on a suboptimal route, however, if there is not enough exploration.

Question 39

Why are Q tables often not the most viable in a practical setting?

Answer 39

In real life problems such as self driving cars, it is often difficult to define these states and even actions. Also there is often a large array of possible actions rather than just horizontal or vertical movements.

Question 40

What was the Q table replaced with relatively recently?

Answer 40

Deep learning neural network that predicts next moves and learns from rewards. This was a revolutionary step.

Question 41

What is a Monte Carlo tree search and how is it difficult to the other type of tree search?

Answer 41

Minimax tree searching searches all pathes (with pruning and other tricks) to some depth and then uses the evaluation function (counting pieces). Instead, with two possible moves A and B, the MCTS will play out the game with random moves until end (win, loss, draw): this is a rollout. It does this multiple times and uses an average result to add values to moves A and B.

It does not test all paths of a tree and there is no evaluation of positions, just the result of a rollout.

Question 42

Name and describe the step after the rollout phase in MCTS

Answer 42

Backpropagation: when you take the values found from the rollout and put that in appropriate places in the tree, all the way up to the top node.

Question 43

What game AI famously used this MCTS method?

Answer 43

Alphazero: Beat the current best chess AI (Stockfish), which could beat any person, with ‘beautiful human-like chess’. It did this with only the rules of chess programmed into it and reinforcement learning and MCTS algorithms; 4 hours of self play and no opening or closing theory built in.

Question 44

How is Searle’s chinese room related to these chess programmes?

Answer 44

While they can play chess better than any humans, they still don’t ‘get it.’ If you put it into an ‘anti computer position’, such as the one is which the pawn placement means that neither side can take a piece but black has more pieces, it is very difficult for the best chess computers. We recognise it as a draw but it concludes that black is better right up until they are forced into a draw.

Question 45

What type of problems do humans typically outperform AI?

Answer 45

Only seem to outperform in ‘fluid’ intelligence tests. These include Bongard problems e.g what do the six figures on the left not have in common with the six figures on the right.