Week 6 - The AI Pipeline, Data Annotation, Presenting a Proposal

Question 1

What is datasheets for datasets

Answer 1

Datasheets for datasets are structured documents that provide detailed information about the creation, composition, and use of a dataset. They include metadata such as the dataset’s purpose, sources, collection methods, preprocessing steps, and ethical considerations. These datasheets aim to enhance transparency, reproducibility, and responsible usage of data.

Question 2

What are the 4 points in the datasheets for datasets

Answer 2

1. Motivation for dataset contruction
2. Dataset composition (recommended splits, evaluation metrics)
3. Data collection process (time period, sampling strategy)
4. Legal and Ethical considerations (consent)

Question 3

What is the difference between annotation and extraction

Answer 3

We can make a distinction between unlabelled data that was merely extracted, and labelled data that was annotated (by humans)

Question 4

What is crowdsourcing

Answer 4

gathering and rewarding human participants if often done through platforms

Question 5

What are the review checks for crowdsourcing

Answer 5

1. Instructions: does the study include intructions
2. Recruitment: is it clear how theyre recruited
3. Consent: how consent was obtained
4. Demographics: is it clear what the demographic is

Question 6

What could go wrong with crowdsourcing

Answer 6

Certain words could be overused because of the demographic of crowdsourcing

Question 7

What is annotator aggregation

Answer 7

Annotator aggregation is a process used in research, particularly in data labeling and machine learning, where multiple annotations or labels provided by different annotators are combined to produce a single, more accurate and reliable label for each data point. This helps to reduce individual biases and errors, improving the overall quality of the dataset.

Question 8

What is interannotator agreement?

Answer 8

Interannotator agreement is a measure used in research to assess the level of consistency and reliability between different annotators who label or categorize the same set of data. High interannotator agreement indicates that the annotators are consistent in their evaluations, suggesting that the labeling process is reliable and the data annotations are dependable.

Question 9

What interannotator agreement is best used for graded annotation

Answer 9

mean. the mean was taken as the gold standard

Question 10

What internannotator agreement should be used for classifcation

Answer 10

Fleiss’ Kappa

Question 11

what is Fleiss’ Kappa, give formula

Answer 11

observed above chance agreement/ max. above chance agreement

Question 12

What is annotator disagreement

Answer 12

Annotator disagreement occurs when multiple annotators provide differing labels or classifications for the same data point. This disagreement can highlight areas of ambiguity or subjectivity in the data and can be used to identify problematic or unclear items that may require further clarification or consensus-building among annotators.

Question 13

What is KL divergence

Answer 13

KL (Kullback-Leibler) divergence is a statistical measure used to quantify the difference between two probability distributions. It indicates how much information is lost when one probability distribution is used to approximate another.

Question 14

What is an interim summary

Answer 14

An interim summary is a preliminary report that provides an update on the progress, findings, and current status of a research project or study before it is completed. It typically includes key data, observations, and insights gathered up to that point, helping researchers and stakeholders assess the direction and effectiveness of the ongoing research.

Question 15

What are 7 points in Interim summary

Answer 15

1. Datasets
2. Peer review
3. Annotator agreement
4. Human disagreement
5. Modelling disagreement
6. Re-calibration
7. Read more

Question 16

what are hyperparameters

Answer 16

e.x: learning rate, dropout rate

Question 17

What is a training regime

Answer 17

Training for multiple epochs, testing the model that achieved highest development accuracy
| Trainig set | Development set | Test set |

Question 18

what is parameter sweep

Answer 18

setting a range and step size of all parameters, and training models over all combinations

Question 19

what is seed-averaging

Answer 19

to avoid lucky seeds, taking average of models with different seeds (but the same parameter settings)

Question 20

What are ablation studies

Answer 20

Ablation studies are experiments where specific components or features of a model are systematically removed or altered to assess their impact on the model’s performance. This technique helps researchers understand the contribution and importance of individual parts of the model, guiding optimization and improving overall model design.

Question 21

What is the goal of abilation study

Answer 21

Goal is to explain whether certain parameters, or external factors may have influenced the achieved performance rather than the hypothesized ones.

Question 22

What is the structure of a research proposal.

name 6

Answer 22

1. Background/context
2. Research question
3. Contributions
4. Methodology
5. Planning
6. Resources

Question 23

What is the majority class label in interannotator agreement

Answer 23

The majority class label is the label or category that appears most frequently in a dataset. In the context of classification tasks, it represents the most common class among the data points. Identifying the majority class label is useful for baseline comparisons and understanding class distribution in imbalanced datasets.

Question 23

why is it important to clarify how human participants were rewarded for their annotation?

1. Because the quality of annotation increases as the reward increases
2. Because the reward influences the target demographic
3. Because crowdworkers are at risk of getting exploited
4. Because their data will influence large language mode

Answer 23

3. Because crowdworkers are at risk of getting exploited

Question 24

what are interval tasks

Answer 24

Interval tasks are tasks that involve measuring or recording data at specified intervals of time. These tasks are designed to collect information periodically, allowing researchers to analyze changes and trends over time. Interval tasks are common in longitudinal studies, time-series analysis, and various experimental designs where tracking the progression of variables is crucial.

Question 25

What is a correlation coefficient

Answer 25

A correlation coefficient is a statistical measure that quantifies the degree and direction of the linear relationship between two variables. Ranging from -1 to +1, a value of +1 indicates a perfect positive correlation, -1 indicates a perfect negative correlation, and 0 indicates no linear correlation. Common types include Pearson’s correlation coefficient and Spearman’s rank correlation