INTERPRETING ACTIVE LEARNING RESULTS Flashcards
Reviewer access
As a reviewer how can you enter the review queue?
Relativity creates a new document list view that’s tied to the Active Learning project. This view is automatically secured to the reviewers added to the project. As an Active Learning reviewer, this view is the only place you can enter the review queue.
Reviewer access
How would you ensure that reviewers cannot skip documents?
Make the project review field a required field.
Reviewer access
Can reviewers change coding on documents they have already coded?
Reviewers can change the coding decision on documents they previously reviewed. These documents aren’t considered manually-selected documents. The next model build will include the most recent coding update.
Reviewer access
Reviewers must code documents based on the so-called “four corners” rule.
What is this and why is it important for Active Learning?
This means that a document should be judged responsive or not responsive based solely on the extracted text of that document only. Documents that are relevant based on family members should not be coded relevant on the Active Learning review field. Although individual anomalies will not hurt the algorithm, too many in aggregate could cause the model to perform worse.
Monitoring an Active Learning project
How many docs need to be coded before an active learning project can complete it’s first build?
At least five documents coded with the positive choice and five coded with the negative choice
Monitoring an Active Learning project
How often will builds take place?
At maximum every 20 minutes after the previous build to include coding decisions not included in the most recent build. If reviewer activity has been idle for five minutes and there are coding decisions not included in the most recent build, the model will start a build.
Monitoring an Active Learning project
Project Home dashboard - What is it?
Gives a high-level overview of the documents in your Active Learning project.
After you first create the project, the dashboard displays the Project Size and coding statistics based on the pre-coded documents in your data source
Monitoring an Active Learning project
Project Home dashboard - What does it help you understand?
Shows:
- How many documents have been coded in your project.
- How many documents have been coded outside the queue (manually-selected documents).
- How many documents have been skipped.
Monitoring an Active Learning project
Update ranks - What is it?
Active Learning ranks all documents from 100 to 0.
This gets updated with every model build. These values are only stored in the Document object on demand.
A project administrator can choose to update these ranks on demand.
Monitoring an Active Learning project
Prioritized review progress - What is it?
The Prioritized Review Progress chart tracks the ability of the model to find relevant documents over time. It does this by monitoring the reviewers’ coding decisions on the high-ranking documents chosen by the Prioritized Review queue
Monitoring an Active Learning project
Prioritized review progress
Should the Prioritized Review Progress relevance rate increase or decline over time?
As a general trend, you should expect to see the relevance rate decline over the course of the review. However, you may see a spike in the relevance rate if a large amount of new documents are added to the project, or if the definition of relevance changes during the course of the review.
Monitoring an Active Learning project
Quality checks and checking for conflicts
Why Would you do this?
We recommend running ongoing quality checks over the course of the project. The Active Learning process is fairly tolerant of some inaccurate or inconsistent coding, but it’s good practice to monitor your queue for conflicts between reviewers and the Active Learning model.
Monitoring an Active Learning project
Quality checks and checking for conflicts
How is this best achieved?
With Dashboards. You can create dashboards and widgets around useful fields such as
CSR - Cat. Set::Category Rank Categories - Cat. Set Reviewers::User \:: Prioritized Review \:: Coverage Review \:: Project Validation
Running Prioritized Review
What is it?
serves documents that are most likely to be coded on the positive choice (such as Relevant) with a small set of documents included for index health. The documents included for index health are selected by the system to give the model a broader range of training documents.
Running Prioritized Review
What Documents are served in Prioritized Review?
A mixture:
- 10% random
- 20% scores “in the middle” (40-60%)
- 70% high ranking uncoded documents
Running Prioritized Review
What happens if you start a Prioritized Review Queue before a model occurs?
the queue initially serves up random documents because no scores are available for documents until a model build has completed.
Running Prioritized Review
After a review has started you want to edit the settings to include family. What happens?
You can’t edit this setting once Start Review has been clicked.
You can only enable for new projects or Prioritized Review queues that have never been started
Running Prioritized Review
What is the maximum amount of Concurrent Reviewers recommended per project?
150 concurrent
Active Learning Set Up
How many choices can the Review Field have
This must always be exactly two choices.
One is positive and one is negative
Running Prioritized Review
How do you add or remove documents from the Active Learning project after review has begun?
- Pause Review
- Update the Saved Search that is used for the classification index
- Navigate to your classification index and Run it (Incremental)
- -Start Review
Running Prioritized Review
Turning off index health documents - Why would you do this?
For large projects and projects with high richness, the Active Learning engine may stabilize and identify consistently relevant documents long before the project has finished. To speed up the later part of these projects, you can exclude index health documents from appearing in the review queue.
Running Prioritized Review
How do you change the index health documents setting?
- Navigate to Project Home Tab
- Click Setting Gear in top right
- Click Edit button
- Change Include Index Health in Prioritized Review to No
- Save - Settings take effect immediately)
Project Validation and Elusion Testing
What is it?
used to validate the accuracy of an Active Learning project. The goal of Project Validation is to estimate the accuracy and completeness of your relevant document set if you were to stop the project immediately and not produce any unreviewed documents below the rank cutoff.
Project Validation and Elusion Testing
When should you run a Project Validation?
Relativity recommend running Project Validation when you believe the project has stabilized and the low-ranking documents have an acceptably low relevance rate. However, you can run Project Validation at any point after the first model build.
