Week 2: Empirical Evaluation of Models for EA Flashcards
Nature of comparison
- Single grammar comparison:
+ Inter-grammar comparison
+ Intra-grammar comparison - Multi grammar comparison.
Inter-grammar comparison
- BPMN vs Activity Diagram
- Entity Relationship vs Class Diagram
Intra-grammar comparison
Example: Within an ER Diagram.
- Cardinality
- Attributes
- Relationships
- Inheritance
Multi-grammar comparison
UML vs DODAF
Empirical Evaluation of models
Setting:
- Assume that we want to compare model A and model B.
- Choose at least one case, that are informationally equivalent.
- Randomly assign study participants to the two groups.
- Group 1 is given model A and Group 2 is given model B.
- Make everything similar or they would say the result was because of the difference.
- Evaluate all subjects using similar test.
- If experts are grading the outcome, use at least two experts and have Inter-Rater-Reliability (IRR) >0.8
- Compare the average of two groups (e.g. ANOVA).
+ Check if there is a significant difference between the averages.
Breakdown
Errors that are identified from the verbal protocols of sessions.
Error
A breakdown that lasts and not corrected by the end of the process.
Confidence
In correctness is an indicator of comfort level generated throughout the process.
Diagrama comprehension
A set of questions about elements (constructs) in the model. -> number of items remembered, recall (redraw the model) fill-in-the-blank questions.
Problem solving
Number of suggested solutions, then, check which diagram is better; BPMN or an activity diagram.
Ontology
Addresses what exists and happens in the world and how they can be grouped.
Zachman Framework
Approach: Linguistic
Complete representation occurs if .. : Models answers all questions (What, Where, Who, …) from different perspectives.
Not empirically supported.
Bunge, Wand, and Weber (BWW)
Approach: Ontological
Complete representation occurs if .. : Models have a corresponding construct for each construct in the ontology.
Empirically supported.
Ontological deficiencies:
- Ontological incompleteness (construct deficit)
- Ontological completeness but vagueness construct overload.
- Ontological completeness but vagueness construct redundancy.
- Ontological completeness but vagueness construct excess.
Ontological incompleteness (construct deficit)
An ontological construct exists that has no mapping from any grammatical construct (a 1:0 mapping).
Ontological completeness but vagueness construct overload.
A single grammatical construct maps to two or more ontological constructs (a m:1 mapping).
Ontological completeness but vagueness construct redundancy.
Two or more grammatical constructs map to a single ontological construct (a 1:m mapping).