Renal PD

Question 1

What is the biggest challenge with developing machine learning algorithms?

Answer 1

Accumulating good data sets that are large enough, represent everyone in your target population, and are annotated properly

Question 2

Which Federal and Institiutional offices oversee research misconduct?

Answer 2

Office of Research Integrity

Question 3

What were the results of the CAMELYON challenge?

Answer 3

• Top machine-learning algorithms outperformed pathologist who were under time constraints
• The top 5 algorithms were equivalent to pathologists without time constraints

Question 4

What is research misconduct?

Answer 4

Falsification, fabrication, and plagarism in research

• Falsification
  
  • Manipulating research materials, equipment, or precess
  • Changing or omitting data or results
• Fabrication
  
  • Making up data or results
• Plagarism
  
  • Appropriation of another person’s ideas, processes, results, or words without giving appropriate credit

Research misconduct falls under “research non-compliance”

Question 5

Is this research misconduct?

The research nurse discovers that the new resident has conducted a retrospective chart review of 250 charts without submitting for IRB approval.

Answer 5

No

This is an IRB issue - The correct procedure was not followed, but falsification, fabrication, or plagarism did not occur, so this does not fall under research misconduct

Question 6

Is this research misconduct?

The research nurse forgets to obtain the required blood pressure measurement from subject #024 on visit 14. The coordinator reviews previous blood pressure measurements for the subject, which average around 120/70 and records blood pressure for visit 14 as 120/70.

Answer 6

Yes

This is fabrication of data; even if this would have been the correct information, the work was not done to obtain the data an it was made up

Question 7

What opportunities exist for machine learning in diagnostic specialties? (Radiology, pathology)

Answer 7

• Standardization and reproducibility
• Improved accuracy
• Delivering specialized expertise
• Efficiency

Question 8

What are some of the challenges of data in medicine?

Answer 8

• Machine learning relies on a lot of data
• Generating labeled (annotated) data is challenging in medicine
  
  • Difficult to get 1000s of examples from busy pathologists and radiologists
  • Subjectivity in the accuracy of the pathologist
• Long term follow-up is hard
• Retrospective studies give us large data sets, but treatment is heterogeneous
• The data does not extrapolate poorly
  
  • Brittle in a way that humans are not
• Technical and pre-analytic variability
  
  • Different labs may do things in different ways

Question 9

Is machine learning the same thing as intelligence?

Answer 9

No - machine learning is basically curve-fitting

This means you cannot extrapolate the algorithm to places on the curve that you don’t have data for

(Ex: If you train an algorithm to recognize stage II cancer, it will not accurately diagnose stage I or stage III cancer)

Question 10

If you suspect research misconduct, should you call the police?

Answer 10

No!

Go to the office of research integrity instead

Question 11

Is this research misconduct?

A negative pregnancy test is required prior to study enrollment and randomization in a rheumatoid arthritis study. Subject #004 is enrolled and randomized and two weeks after randomization, the study nurse realizes the pregnancy test was not done but enters a negative pregnancy test on day of randomization.

Answer 11

Yes

Data was falsified and the research record was altered

Question 12

Is this research misconduct?

During the consent process, the PI tells the potential subject that the study would be “good for them” and that he thinks they should enroll because the study treatment will “make them better.”

Answer 12

No

This may not be accurate, but it is not research misconduct

Question 13

Why does cohort variability pose challenges for machine learning?

Answer 13

Cannot extrapolate to patient populations not represented in the training data

• Risk factors depend on natural history and environmental factors
• Outcomes depend on implicit biases around gender, race, economics

Question 14

If you suspect research misconduct, should you email the office of research integrity?

Answer 14

No!

Go and speak with somebody in person

Question 15

What are the consequences of research misconduct?

Answer 15

Varies, but can include:

• Suspension or termination of grants
• Debarment
  
  • Degree revoked
• Prohibition from service on PHS advisory committees, peer review committee, or as consultants
• Crimial charges
  
  • Fines, penalties, imprisonment

Question 16

What are the next steps for persons who have potential research misconduct concerns at Northwestern University?

Answer 16

• Initial assessment of allegations
• Inquiry committee
  
  • Does the evidence warrant full investigation?
• Investigation committee
  
  • Did research misconduct take place?
  • Who committed research misconduct?
• Intitutional decision
  
  • Provost makes this decision, based on the findings of the investigation committee
• Federal reporting and oversight review

Question 17

For something to be considered research misconduct, does the falsified, fabricated, or plagarized material need to be published?

Answer 17

No!

• Can include proposals, draft manuscripts, non-published/shared research records

Question 18

What consititutes a finding of research misconduct?

Answer 18

• Significant departure from accepted practices of the relevant research community
• Misconduct was committed intentionally, knowingly, or recklessly
• Allegation is proven by a preponderance of evidence

Within Northwestern, additional findings include:

• Abusing confidentiality
• Stealing, destroying, or damaging research properly
• Directing, encouraging, or knowingly allowing others to engage in fabrication, falsification, or plagarism

Question 19

What are the risks associated with machine learning?

Answer 19

Widening healthcare disparities

If there is bias or unfairness in the training set, the algorithm will be racist

If certain populations are underrepresented in the training data set, the algorithm will not perform well for these populations
-> missed diagnoses, lower quality care

Question 20

Is this reseaerch misconduct?

Your brother in law (recently unemployed) wants to participate in your study of a new lipid lowering drug since he can’t afford his current Rx. The study requires a 6 week wash-out and a baseline LDL of 160. Your brother-in-law has a history of LDL values around 180 but when drawn for the study his LDL is 150. You enroll him in the study and record his baseline LDL value as 180 with a memo explaining the missing lab report and referencing a prior lab report with a LDL of 180.

Answer 20

Yes

The research record was falsely altered

Question 21

Does honest error constitiute research misconduct?

Answer 21

No;

For something to be considered “research misconduct,” it must be intentional and go directly against institutional practices. This must be supported by evidence

(Reserach misconduct does not include honest error, authorship dispute, or differences of opinion)

Question 22

If you suspect research misconduct, should you talk to somebody in person at the office of research integrity?

Answer 22

Yes!

They can help you figure out the process, confidentiality, and whether or not something might constitute research misconduct