Critical thinking about Psychological Theory - Lecture 23

Question 1

General outline of the chapter

Answer 1

There are three articles, each one talking about a different topic. I’ll go through each article in order, hopefully the structure will make sense.
- Article 1 (Dennis and Kintsch): Criteria for a good and useful-in-the-real-world theory
- Article 2 (Dienes): Falsifiability of theories (and some stuff on models)
- Article 3 (Marewski and Olsson): Theories, NHST, and models (builds upon stuff in Article 2)

Question 2

What is the general outline of the different types of hypothesis and their level of reasoning? (Couldn’t phrase the question better, just look at the picture and you’ll understand, I hope… It’s just a general outline of the level of reasoning this chapter is about)

Answer 2

See image 4

Question 3

General knowledge about theories

What is a Theory?

Answer 3

A statement about how we believe the world to be
–> We organize observations of the world so we can make predictions about what will happen in the future under certain conditions
- Science: purpose is to test theories
- Data: Should bear on a theory

Question 4

General knowledge about theories

What is a formal and informal theory?

Answer 4

• A formal theory is a set of rules and assumptions that are based very purely on scientific methods and proof of correctness. In other words, they’re theories expressed by mathematical equations (e.g. Theory of relativity, E = mc^2)
• An informal theory is a verbal theory: The statement is expressed in words, language (e.g. Maslow’s Hierarchy of needs, most if not all psychological theories)

Question 5

General knowledge about theories

What is a wrong idea many people have about formal theories?

Answer 5

“Formal theories are only attainable in hard sciences (e.g. Physics, Math) and not in soft sciences (e.g. Psychology)”
(Will explain later why it is wrong)

Question 6

General knowledge about theories

What is the curret state on psychological theories?

Answer 6

They’re dominated by verbal theories. BUT, computational/mathematical models are becoming more and more popular.
These models enforce precision and consistency in our reasoning which can not be obtained with any other method

Question 7

Criteria for a good and useful-in-the-real-world theory (Dennis and Kintsch)

Question 8

What are the 10 criteria for a good and useful theory?

Answer 8

1. Descriptive adequacy. Does the theory agree/match with the available data?
2. Precision and interpretability: Is the theory described in a precise and clear way?
3. Coherence and Consistency: Are there logical flaws in the theory? Does each component of the theory seem to fit with the others into a coherent whole? Is it consistent with theory in other domains (e.g. physics)?
4. Prediction and Falsifiability: Is the theory formulated in such a way were tests can falsify it?
5. Postdiction and Explanation: Does the theory provide a genuine explanation of existing results?
6. Parsimony: Is the theory as simple as possible?
7. Originality: Is the theory new or is it essentially a restatement of an existing theory?
8. Breadth: Does the theory apply to a broad range of phenomena or is it restricted to a
   limited domain?
9. Usability: Does the theory have applied implications?
10. Rationality: Does the theory make claims about the architecture of mind that seem reasonable in light of the environmental contingencies that have shaped our evolutionary history?

Question 9

Descriptive adequacy

(What is a common debate about data in science)? (No need to remember exactly, not that important)

Answer 9

What’s the right type of data to use for each RQ or domain. This differs across domains

Question 10

Descriptive adequacy

What’s the most common method we use to compare theories against data to see if they match?

Answer 10

NHST
Ho: no difference
Ha: there’s a difference (in line with our theory)
Testing shows if data are in line with Ha or not

Question 11

Descriptive adequacy

What are some problems with NHST?

Answer 11

• Maybe the theories have not taken into account the confounding variables, so the data doesn’t really fit the theory if you control for them
• When using NHST you can never truly conclude that there’s no difference between conditions. You can attribute your findings to a lack of pwoer, small sample size, etc. This leads to uncertainty in interpreting results

Question 12

Descriptive Adequacy

What are the advantages of using formal models of psychological phenomena?

Answer 12

Used to derive how well the theory fits the data (doesn’t rely on NHST). This means that:
- We can say exactly how closely the formal theory approximates the data (DEGREE TO WHICH THE THEORY APPROXIMATES THE DATA, NOT AN ALL-OR-NONE DECISION AS IN NHST)
- Gives info about the nature of the relationship between variables (linear, quadratic etc.. NHST instead would just say the something has an effect on something else, that’s it. Formal theory gives more info about this difference)
(See image 5 for an example of the above)

Question 13

Precision and Interpretability

What is a common criticism of theories?

Answer 13

They’re described in an inprecise and vague way
- Definition of constructs are rare
- Mechanisms behind the interaction of variables on each other is also rarer
!!! Usually most criticism against theories is because the reader has misunderstood something. If that something is made clear, then there’s no more criticism from the reader !!!

Question 14

Precision and Interpretability

What questions should we ask ourselves when cosntructing a theory

Answer 14

• (Can I be confident to apply this theory in a related domain?) (Not that important, the next point is the most important one)
• What implicit assumptions am I making, which aren’t shared by the readers

Question 15

Coherence and Consistency

What is the circularity problem?

Answer 15

If we state that some underlying mechanisms are the cause for performance, but the only way to define the mechanisms is through the performance then our claim loses credibility.
(Examples of circular reasoning are in flashcard 61 as well)

Question 16

Coherence and Consistency

How do you make sure that your theory is coherent and consistent?

Answer 16

Ask how consistent a theory is with other theories within and outside psychology, and pick the ones that align better with your research

Question 17

Prediction and Flasifiability

See next side

Answer 17

• Mathematical models are good ways to formally prove any prediction
• Verifying a theory can also help increase our confidence in it (instead of falsification just being the only way to help advance scientific knowledge)
  !!! Suprising results are better than unsurprising ones (better evidence for a theory)

Question 18

Postdiction and explanation

What is the general goal of science?

Answer 18

TO UNDERSTAND, NOT TO PREDICT
Prediction is often, if not always very difficult and unobtainable
- One must understand everything about something
- One must also control all relevant variables (impossible)

Question 19

Postdiction and explanation

What is postdiction?

Answer 19

Explanation after the fact (prediction: explanation before the fact)
- To postdict we just need understanding of what’s going on, but not control over all relevant variables
- NOTE: Postdiction can be based on formal theoreis as much as predictions

Question 20

Parsimony

What is the relationship between descriptive adequacy and parsimony?

Answer 20

We want to fulfill both at the same time as much as possible

Question 21

Parsimony

Example of parismony and descriptive adequacy balance

Answer 21

e.g. we have a cubic model (x^3) vs a linear model
- Look similar to linear and power functions in restricted ranges
- Models the noise better
Data fits the model better. Despite this, when generalizing to new data the model doesn’t do so well.
THEREFORE, IT’S A MATTER OF BALANCE (In this case resort to a x^2 function
(Link to overfitting as well, Flashcard 67)

Question 22

Parsimony

What are some tehcniques to achieve a balance of parsimony and descriptive adequacy?

Answer 22

AIC, BIC, ICOMP and more…
(No need to remember the methods, just a note that these methods are also used in multiple regression. This flashcard is more to show the link between the chapters)

Question 23

Originality

What is true about the originality of many theories?

Answer 23

Theories may look very different, and be different in broader implications, but with respect to a particular set of data two theories might be identical, and predict the data equally well

Question 24

Breadth

How broad should theories be in general?

Answer 24

As broad as possible, while maintaining descriptive adequacy and the ability for postdiction

Question 25

Breadth

What is true though about theories in psychology?

Answer 25

We want the opposite from broad theories.
“We want to divide theories into smaller and smaller classes to know more and more about less and less”

Question 26

Breadth

What are some reasons as to why psychology doesn’t have broad theories?

Answer 26

It’s difficult. This can be due to:
- Complexity of psychological phenomena
- Immaturity of psychology as a science
!!! There’s a big debate on whether psychological theories should continue being small or if we should create bigger, broader, unifying theories

Question 27

Usability

What are good theories judged on?

Answer 27

If they’re useful in social implications
The best theory contributes to both scientific understanding and fulfills a societal need
(See image 6 for some mroe info)

Question 28

Rationality

(Explanation of the criterion, because in the beginning it was complex)

Answer 28

The environment has shaped our minds and way of thinking, therefore when we apply a model to a theory it has to match the environment and our way of thinking. E.g. we apply a model to memory and number of items, and our model states that memory decreases as number of items increases in a power function. Why the power function?
Because power functions are found everywhere in nature.
IN OTHER WORDS, OUR WAY OF MODELLING THE MIND HAS TO BE IN LINE WITH HOW NATURE WORKS.

Question 29

Falsifiability of theories (and some stuff on models) (Dienes)

Question 30

What is a Faslifier?

Answer 30

Any potential observation/statement that would contradict a theory (e.g. theory states all Swans are white, a falsifier is seeing a black swan)
- If there are more potential falsifiers for a theory, the theory is considered more falsifiable

Question 31

What constitutes a good theory in general?

Answer 31

• More falsifiers (or needs fewer falsifiers to be falsified, easier to be falsified)
• less simple

Question 32

What makes a theory more falsifiable?

Answer 32

• The more specific/complex it is, the more falsifiable (A theory that allows everything explains nothing, e.g. “Group A is positively correlated with B” is preferred to “Group A is correlated with B”)
  (See image 1 for understanding)
• The more broad it is (the more situations it can apply to), the more falsifiable

Question 33

How does a theory improve?

Answer 33

• increase falsifiability
• Choice of observation statement is better motivated (e.g. use measures with better construct validity)

Question 34

How does Science improve?

Answer 34

When we replace a falsifiable theory with another more falsifiable one

Question 35

Linear model vs Quadratic model (x^2): Which is a better theory and why?

Answer 35

Quadratic.
- Quadratic fits more data patterns than a linear function, therefore you need more data points to falsify it compared to the linear model
- Quadratic is less simple than a linear model

Question 36

What is the importance of falsifiability?

Answer 36

• If a theory is more open to criticism (more open to falsifiability) we can make faster progress in a field (based on the notion that rejection and criticism leads to better and faster progress)
• Such theories are more bold and interesting
• “Good science is not just shown from it’s literal forms of theory, but also from the history of how it’s theories came to be”

Question 37

Problems with falsifiability

What is ad hoc?

Answer 37

When the revision of a theory/addition of other elements to a theory decrease it’s falsifiability.
We always want to make sure that revision/addition increase falsifiability, but we can never truly determine if rev./add. increase or decrease falsifiability.
SOLUTION: Make sure any revision/addition can be falsified

Question 38

Problems with falsifiability

What is post hoc?

Answer 38

A way to save a theory from falsifiaiblity without having to conduct more and different tests

Question 39

Computationl models

What are computational models?

Answer 39

Computer Simulations of a subject, were the model is exposed to the same stimuli subjects receive, and also gives actal trial-by-trial responses

Question 40

Computational models

What are characteristics of computational models?

Answer 40

Each model has free parameters: For the model to run we have to give a certain value to these parameters.
- We can’t calculate these parameters in real-life. What we do is we choose from different combinations of parameters and then measure how each model predicts the data on each combo of parameters

Question 41

Computational models

How to interpret graphs of computational models?

Answer 41

• Each graph point represents the performance of each model for a certain number of parameter values
• The more condensed the points of a model, the more falsifiable it is
• If osberved data fall on the area that the model points make up, that model has not been falsified, It’s been CORROBORATED

Question 42

Computationl Models

Graph example and explanation

Answer 42

See image 2

Question 43

Computational models

How do researchers usually use computationl models?

Answer 43

They try to find parameters that fit the data, after obtaining the observed data (Verification bias)

Question 44

Problems when people reason from theory < - > data

Why do we need clear and well-stated theories when doing research?

Answer 44

!!! Theories are needed to determine what our data/observations are, and vice versa (theory <-> observations/data) !!! (“Observations are theory impregnated”). Therefore our theories influence how we interpret data and vice versa.
- A good theory makes the assumptions visible and verifiable
Also, we need good theoreis specifically in psychological research because we need more intergration of theories in general (not that important, the above point is the main thing to remember)

Question 45

Problems when people reason from theory < - > data

How does this relationship between theory and data also cause problems?

Answer 45

Since observations/data are influenced and given by theory, they can never be given directly and only by experience.
(e.g. we’re researching the effect of stress on memory, if we know that stress reduces memory, we say that we observe that stress reduces memory, but we never really observe it. The only reason we can “assume” to observe it is because we believe in the theory that says that stress reduces memory. If we didn’t have the theory we couldn’t observe it. Therefore we can doubt any observations or change any definition of a concept through this thinking and escape falsifiability this way. This is the line of argumentation that’s used here, to an extreme though. Just bear with it for this case)

Question 46

Problems when people reason from theory < - > data

How is the above problem solved?

Answer 46

Through time and experience we come to a point were nobody wishes to deny the statement and thus we agree to accept it (if new contradicting evidence comes up, we reject the theory, and wait for a new consensus to be reached, if it ever is)

Question 47

Problems when people reason from theory < - > data

Given the problem above, how should we judge the falsifiability of a theory?

Answer 47

Purely in regards to the methods used to examine or testing the theory (since this is where the problem is, since people can obscure data to wrongly verify their theory) (This argumentation was stated by Popper, in general he’s the main scientist for the topics of this chapter, keep him in mind as a name. The thinking stated in few previous flashcards we call Popper’s approach)

Question 48

Problems when people reason from theory < - > data

What are some criticism’s to Popper’s approach?

Answer 48

• No theory is 100% falsifiable -> Leads to Duhem-Quine problem
• Every theory is falsified at some point to some extent

Question 49

Problems when people reason from theory < - > data

What is the Duhem-Quine problem?

Answer 49

Given falsification, how do we know which component of the theory to reject.
OR, how do we know if we should reject the theory or the assumptions about our research

Question 50

Problems when people reason from theory < - > data

What are some reasoning fallacies we make when reasoning from theory to data?

Answer 50

• Circular Reasoning
• Hot hand fallacy
• Gambler’s fallacy

Question 51

Problems when people reason from theory < - > data

What are different types of circular reasoning?

Answer 51

• Repeat premise as a conclusion
  P) God exists
  C) God exists
• Premise presupports the truth of C
  P1) Bibe says God exists
  P2) Bible is the word of God
  C) God exists
• Premise is logically equal to the conclusion
  P) I react quicker to aggression-related words
  C) Aggression-related words are more accessible in my long-term memory

Question 52

Theories, NHST, and models (Marewski and Ollson)

Question 53

What is generally true about NHST for statistical inference?

Answer 53

It is an incoherent methodology and is detrimental for psychology
Despite this, it is still used in psychological research and theorizing.
WHY?: Theoreis are too weak to do anything else other than predict the direction of an effect, therefore the only statistical tool we can use for theories is NHST, which just examines exactly that, the direction of an effect.

Question 54

What alternatives to NHST have been proposed by other researchers?

Answer 54

reporting:
- effect sizes
- CI
- Meta-analyses, and more… (no need to remember)

Question 55

Null ritual

What does each ritual entail (in general, not only in research or stats)?

Answer 55

• Repetitions of the same actions
• Fixations on specific features
• Anxieties about punihsments for rule violations
• Wishful thinking

Question 56

Null Ritual

What are the steps for the Null Ritual?

Answer 56

1/. Set up Ho (no difference/correlation). Don’t specify predictions of Ha
2/. Use 5% for rejecting Ho. If significant accept Ha
3/. Always perform this procedure

Question 57

Null Ritual

What is the main problem with the Null ritual?

Answer 57

When we reject Ho and say that Ha is true, given that we don’t specify what Ha is, we just accept that there is some difference (any possible difference). Linking back to previous chapter, Ha (our theory) is very non-specific, thus very non-faslifiable, and this is not good

Question 58

Null Ritual

What are some forms of pseudoreasoning found in the Null Ritual

Answer 58

1/. Set up Ho (no difference/correlation). Don’t specify predictions of Ha -> Straw Target fallacy. We create an easy target, which is Ho, and then by rejecting Ho we just say that the theory in line with Ho is not true. So in a sense we make it easy for us to refute or disregard one theory, whereas in truth it is not correct and not a realistic way to refute a theory
2/. Use 5% for rejecting Ho. If significant accept Ha -> False Dilemma. Only gives us the two options of either Ho is true and Ha is not, or Ha is true and Ho is not. Maybe both Ho and Ha are true, or they’re both false, or something else.

Question 59

What is a model?

Answer 59

A simplified representation of the world that aims to explain observed data.
- Models are also used to formally express a theory (through statistical tools, such as regression, equations etc.)
~ This helps researchers answer questions about psychology and explain complex phenomena better
!!! NOTE: Each RQ/problem demands a different model to be answered in the best way !!!

Question 60

Expressing Theories Formally

What are the advantages of expressing a theory formally (through a model)

Answer 60

• Models allow for the design of strong tests of theories
• Models make theory statements more sharp and precise
• Models can lead us to theories that aren’t only built on a linear model (other models can also be used to express and evaluate theories, not only the linear model)
• Modelling theories helps us address real world problems

Question 61

Expressing Theories Formally

Models alow for the design of strong tests of theories

Answer 61

If we compare predictions from different models (different theories), we can see which one fits the observed data the best. Therefore we determine which is the best theoretical explanation for the data. This also allows us to not use NHST, since it becomes useless.

Question 62

Expressing Theories Formally

Models make theory statements more sharp and precise

Answer 62

Through mathematical and scientific terms we explain the contingencies of a theory in a very concrete way that can’t be misinterpreted (you can’t misinterpret a mathemtical formula, you can misinterpret somebody’s words though)
NHST is often used to evaluate informal theories. If they’re underspecified, they can be used post hoc to explain the data: WRONG
Other problem: If theories are not specific or well-stated:
- It is possible to apply a theory in a wide range of phenomena when in truth you can’t and shouldn’t
- Makes it difficult to carry out tests
Final problem: If theories are not specific or well-stated we might end up with Equivocations: We end up having one-word explanations for something, and that same word describes several underlying processes. This should never be the case though

Question 63

Expressing Theories Formally

Models can lead us to theories that aren’t only built on a linear model

Answer 63

Very often researchers use the linear model as their starting point and usually biuld their research based on this. This is not very good, since it limits the foundation for conceptualizing psychological problems.

Question 64

Expressing Theories Formally

Modelling helps us address real-world problems (apart from the linear model, because of the above flashcard)

Answer 64

It allows us to deal with natural confounds without destroying them. Confounds can be built into the model

Question 65

Expressing Theories Formally

What are some ways to test and design a model to have real-world implications?

Answer 65

• Go into the world and make observations there -> build your model to predict these new observations
• Representative design -> take the real world into the lab

Question 66

Overfitting

What is a common problem when analyzing the data from a model?

Answer 66

Distinguishing between variance that’s caused by errors and variance caused by the model. Goodness of fit can’t make this distinction by itself

Question 67

Overfitting

What is overfitting?

Answer 67

When we fit error scores (or idiosyncracies, data points we wouldn’t expect) into the model and consider them part of it. The problem with this is that we can’t make good predictions about the data because of this (See image 3).
- Complexity: the model’s flexibility that allows it to fit diverse patterns of data (degree to which a model is susceptible to overfitting)
Factors contributing to complexity:
- Number of parameters in a test
- How these parameters are combined

Question 68

Overfitting

What is generalizability?

Answer 68

The ability of the model to predict new data

Question 69

Overfitting

What is the relationship between Overfitting and generalizability?

Answer 69

Until a certain point, as complexity increases so does generalizability. But after a certain threshold, as complexity increases generalizability decreases

Question 70

Other problems in model selection

What are some other problems in model selection?

Answer 70

• Irrelevant specification problem
• Bonomi paradox
• Identification problem

Question 71

Other problems in model selection

Irrelevant specification problem

Answer 71

Deciding how to bridge the gap between informal and formal descritpion of theories can cause discrepancies between theories and their counterparts

Question 72

Other problems in model selection

Bonomi paradox

Answer 72

As models become more specific and realistic, they become harder to understand and less clear

Question 73

Other problems in model selection

Identification problem

Answer 73

There might be many theories explaining a behavior. Many researchers ask which theory is more truthful, and this is wrong. Instead of asking which theory is more truthful, rather ask which model is better given some criteria for explaining the same behavior (e.g. compare model A and B that explain phenomenon C regarding their simplicity, usability (from frist article) etc.)

Question 74

Other problems in model selection

Note on these problems

Answer 74

!!! They apply to both formal and informal theories !!!

Question 75

If modelling is a good route for scientific progress, why don’t more scientists use it

Answer 75

• Desinging and testing models is effortful and time-consuming and requires skill
• Broad acceptance of NHST to examine vague theories provides comfort (little motivation) to elaborate ideas by building models, that are of theoretical value in the real world.

Question 76

Lecture Notes

Question 77

What is the main idea behind discovery-oriented research?

Answer 77

If the theory is true, then it is not necessarily the case that the hypothesis is also true. In other words P(H true|Theory true) has a small value. Therefore, hypothesis is not strongyl implied by theory
(Theory testing research on the other hand states that if the theory is true the hypothesis must be true, P(H true|Theory true) = 1)