Ultimate Strategy Flashcards
What is the paper about in terms of the big picture and main question?
1) When was it published? [“At the time of publishing”, “according to the researchers”]
2) Who funded it? Is this relevant?
3) What is the current state of the field? What have been the recent advances? Are they in a different organism? Where do we want the field to be?
4) Which organism?
5) Why is the big picture compelling? Is it interesting to just researchers or also to the broader public?
6) Why did they bother to do this research? What are they trying to prove? Is it innately interesting or does it have an applied value? Or both?
7) What knowledge gap are they trying to fill? Is it challenging any dogma? Or looking for a secondary set of evidence? Is it a proof-of-principle?
8) How does the main question link to the big picture?
9) Why does the main question need explaining (be explicit}.? (Bear in mind it could be the title). Why is it fundamentally interesting/charismatic? You can include the hypotheses as part of your main question, or not. The main question can tackle a topic, and then ask a question. The main question can also be multi-pronged.
What are the hypotheses being tested and have the authors made them explicit?
1) “A testable statement about a relationship between two or more variables” / “a proposed explanation for some observed phenomenon”
2) Were the hypotheses pre-registered?
3) Is the paper goal-driven, exploratory, or hypothesis driven? Are the hypotheses phrased as questions? You can be explicit about the questions but implicit about the hypotheses.
4) Give a null and alternative hypothesis – formalised.
5) Are the hypotheses directional, or not?
Are they conditional?
6) How do the main/guiding hypotheses and sub-hypotheses link to the main question, each other, and the protocol? Consider giving sub-hypotheses their own predictions. You can nest them.
7) Are there any hypotheses that are not mentioned?
8) Explicit – found in the figure legends/titles. “To test this hypothesis, we […]”
9) Implicit – inferred from the narrative.
10) Is there a mixture of both? Can be explicitly discussed in colloquial language but not stated.
11) Each experiment will be testing its own hypothesis, whether stated or not. Look for one attached to each Figure. Go through each experiment to find them.
12) Provide insight and critical analysis of the hypotheses of the paper.
Are the methods appropriate for the question being addressed?
1) The methods will be SUFFICIENT. What MORE could they have done?
2) “The authors justify this method by stating […].”
3) Be thorough. Go step by step, and explain each method sequentially. [“Next”, “having done this, the researchers must now do that etc.]. If there are too many, highlight the ones with the major experimental outputs.
4) What does each methodology test, and is this appropriate to its relevant hypothesis?
5) How do they link together their methodologies?
6) Are there any ubiquitous techniques?
7) Do they follow any key protocols/conventions? On the assumption that these are valid.
8) Are the experiments interventional or observational?
9) Do they use any combinatorial approaches for thoroughness + corroborating evidence? Or analysis across a spectrum?
10) Are the methods reproducible?
11) Do they need to confirm anything to justify the use of a methodology? (e.g. Western blotting for Abs, expression of reporters are specific [via controls] etc.). “In order to investigate whether […], the researchers must first confirm […]”
12) Discuss controls. Are there any that they did not do?
13) Is there anything that they did not explain why they chose? What potential effect might this have? What assumptions does this rely on and have they been tested?
14) Go through the response variables. Are they actually testing the stated hypotheses, or do the hypotheses need revision? Are they crude, or high resolution (i.e. are they qualitative or quantitative)? Could this be improved? If so, how?
15) If they have had to use a proxy, is this proxy appropriate?
16) At any point, could they have tested a broader range of things? Why?
Discuss two key findings and describe which element of the results supports each one.
1) Relate to the hypotheses
2) State the finding. Which results show this? Where have you found them? (“see figure […]). Sometimes findings are recapitulated/corroborated across Figures, so this is worth looking out for.
3) Which ones are the most central to the narrative? You might want to sketch out the protocol to help you decide. Make a point of this.
4) Refer to figures (you can use multiple for one key finding!). When looking at figures, dig into what they actually show. Are there any spurious labels? Over-interpretation? Anything that isn’t addressed that might require further testing to explain?
5) Identify any stats
6) How do the authors summarise the finding?
7) The authors may provide alternative explanations. You may, too. You can evaluate them here if you agree, or in the limitations if you don’t. Signpost this.
Have the authors appropriately interpreted and summarised their conclusions?
1) Go through interpretation and then summary
2) Have they concluded on anything that was not a formal hypothesis?
3) How does the scope of the experiment align with the scope of the conclusions?
4) Are the conclusions conservative/cautious enough? Define what this means – do they allow for alternative interpretations? The more conservative stuff is likely to be in “Results”. Overinterpretation may be in the abstract, introduction, discussion or conclusion. This is worth mentioning.
5) Does the paper clarify its limitations?
6) Are there causative conclusions based on correlational data?
7) Would it have been helpful to provide different figures, or display the data in a different way?
8) Do they stay specific to their model system and experimental scenario, or do they extrapolate beyond this?
9) If they are making an evolutionary argument, is this backed up by phylogeny?
10) If they are making an argument for functional annotation, is this backed up by essentiality and sufficiency experiments?
11) Is the dataset mixed? Have they simply chosen the part of the dataset which supports their hypotheses, while ignoring/overlooking/downplaying the rest (i.e. a selective summary)
12) Are their conclusions robust? Or have they made assumptions? Is it possible to test the assumptions that they have made?
13) What further experiments would they need to do to totally justify their conclusions?
What are the main limitations affecting this paper?
1) Can be technical or inferential
2) Relate to the main question/hypotheses – the main limitations have to be those that are critical (versus those of little concern)
3) If something is vague, say why they have made it vague, and what further research you need. If you suggest extra experiments, explain how they solve the ambiguities.
4) Think about the methodologies – especially the old-fashioned ones. Do they introduce confounding effects?
5) Have they conducted any stats? Is this because the sample size is too small? If they haven’t, it means their results could be down to chance. If they haven’t done stats, this is the main limitation.
6) If sample size is small – why is it small? Have they predetermined it using stats? How could you fix it?
7) Have they considered alternative explanations of their dataset?
8) At any point, have they deselected something? Have they explained why, or simply not shown its data? Have they been specific about their conclusions as regards this?
9) Why was their model organism chosen? Is it the most appropriate one? Is it evolving? Has this been addressed?
10) Have they got the same coverage of testing throughout the experiment? Do they keep the response variables constant? If not, why not, and should they?
11) Have they mentioned anything which they did not test?
12) Biological realism of temporal/spatial extension of the study.
13) Is reporting of methods sufficient?
14) Post-hoc justification of unexpected results?
15) Are the limitations addressed, or not?
Suggest further research that could strengthen and build upon this work.
1) Strengthen: make it more robust. Not showing anything new, but improving resolution. Follow-ups.
2) More replication, longer term
3) Different species/contexts; manipulative experiments; new/different treatments (data fom a different source or collected using a different method)
4) Meta-analysis?
5) Maybe run the opposite experiment, and see what happens?
6) Build upon: further experiments for more mechanistic/genetic insights into the system. Maybe the order of expression etc/ investigate regulation and cell autonomy
7) E.g. truncate the protein to figure out the essential residues/domains. Domain deletion variant complement assays.
8) If you want an evolutionary perspective, try cross-species complementation and build a phylogeny.
9) Why would these be interesting and useful? What would we learn from it?
10) “A different interesting extension to develop some novel science might be […]”/ “to expand the scope of this paper […]”
11) Could you translate the work across to a different organism to build an evolutionary comparison?
12) If it is proof-of-principle, how would you expand it into the field?
13) New technology!
14) If you think they might be experimentally/logistically/ethically challenging, explain why, and what you think the solution is
