W1-Introduction to Machine Learning in Production Flashcards
What are the steps of an ML project?
Scoping (Defining the Project)
Data Collection and Establishing baseline
Modeling and Error Analysis
Deployment and Monitoring
In a lot of research work or academic work you tend to hold the ____ fixed and vary the ____ and may be vary the ____ in order to try to get good performance.
In contrast, For a lot of product teams, if your main goal is to just build and deploy a working valuable machine learning system, it can be even more effective to hold the ____ fixed and to instead focus on optimizing the ____ and maybe the ____, In order to get a high performing model
data, code,hyperparameters
code, data, hyperparameters
How do you deploy a Speech System? (Voice Search)
1- Mobil phone (edge device) records voice
2- a Voice Activity Detection (VAD) module selects the audio containing s/o speaking
3- Sending the audio to the prediction server
4- Getting the transcript + the search results
After this, we need to monitor and maintain the system
There are two major categories of challenges in deploying a machine learning model. What are they?
The machine learning or the statistical issues
The software engineering issues
What’s the difference between concept drift and data drift? Give examples for each
The term data drift is used to describe if the input distribution x changes, such as if a new politician or celebrity suddenly becomes well known and he’s mentioned much more than before.
The term concept drift refers to if the desired mapping From x to y changes such as if, before COVID-19. Perhaps for a given user, a lot of surprising online purchases, should have flagged that account for fraud. After the start of COVID-19, maybe those same purchases, would not have really been any cause for alarm, in terms of flagging.
When you deploy a machine learning system, one of the most important tasks, will often be to make sure you can detect and manage any changes. Including both Concept drift, which is when the definition of what is y given x changes. As well as Data drift, which is if the distribution of x changes, even if the mapping from x or y does not change. True/False
True
What are the questions we need to answer, in order to avoid software engineering issues in the deployment?
1- Do we need real-time predictions or batch predictions?
2-On what is the prediction system run? Clouds or edge devices or web browsers?
3-How much compute resources do we have for the deployment? (CPU/GPU/Memory)
4-Depending on the application, how long is the latency and throughput(Query Per Sec)
5-How is logging done?
6-Security and privacy
To summarize, deploying a system requires two broad sets of tasks, what are they?
- There is writing the software to enable you to deploy the system in production.
- There is what you need to do to monitor the system performance and to continue to maintain it, especially in the face of concepts drift as well as data drift.
What are 3 common deployment cases?
1- New product/capability (uses gradual ramp up of data traffic along with monitoring)
2- Automate/assist a manual task (uses shadow mode deployment)
3- Replace a previous ML system
In shadow deployment or shadow mode, the new model is deployed with new features alongside the live model.
What are the two key ideas in designing a deployment pattern?
Two recurring themes you see are that you often want
1- A gradual ramp up with monitoring. In other words, rather than sending tons of traffic to a maybe not fully proven learning algorithm, you may send it only a small amount of traffic and monitor it and then ramp up the percentage or amount of traffic.
2-The second idea you see a few times is rollback. Meaning that if for some reason the algorithm isn’t working, it’s nice if you can revert back to the previous system if indeed there was an earlier system.
How does shadow mode deployment design work?
hint: It’s used to automate/assist a manual task
When you have people initially doing a task, one common deployment pattern is to use shadow mode deployment. And what that means is that you will start by having a machine learning algorithm shadow the human inspector and running parallel with the human inspector. During this initial phase, the learning algorithms output is not used for any decision in the factory. So whatever the learning algorithm says, we’re going to go to the human judgment for now.
What’s the purpose of shadow mode deployment design?
The purpose of a shadow mode deployment is that allows you to gather data of how the learning algorithm is performing and how that compares to the human judgment. And by sampling the output you can then verify if the learning algorithm’s predictions are accurate and therefore use that to decide whether or not to maybe allow the learning algorithm to make some real decisions in the future.
When you already have some system that is making good decisions and that system can be human inspectors or even an older implementation of a learning algorithm. Using a shadow mode deployment can be a very effective way to let you verify the performance of a learning algorithm before letting them make any real decisions.
When you are ready to let a learning algorithm start making real decisions (after shadow deployment), a common deployment pattern is to use a canary deployment. How does it work?
In a canary deployments you would roll out to a small fraction, maybe 5%, maybe even less of traffic initially and start let the algorithm making real decisions. But by running this on only a small percentage of the traffic, hopefully, if the algorithm makes any mistakes it will affect only a small fraction of the traffic. And this gives you more of an opportunity to monitor the system and ramp up the percentage of traffic it gets, only gradually and only when you have greater confidence in this performance.
How does the Blue Green Deployment work?
Say you have a system, a camera software for collecting phone pictures in your factory (to detect defects in the manufactured phones).
These phone images are sent to a piece of software that takes these images and routes them into some visual inspection system.
In the terminology of a blue green deployments, the old version of your software is called the blue version and the new version, the Learning algorithm you just implemented is called the green version.
In a blue green deployment, what you do is have the router send images to the old or the blue version and have that make decisions. And then when you want to switch over to the new version, what you would do is have the router stop sending images to the old one and suddenly switch over to the new version.
So the way the blue green deployment is implemented is you would have an old prediction service may be running on some sort of service. You will then spin up a new prediction service, the green version, and you would have the router suddenly switch the traffic over from the old one to the new one.
What’s the advantage of a Blue Green Deployment?
The advantage of a blue green deployment is that there’s an easy way to enable rollback. If something goes wrong, you can just very quickly have the router go back reconfigure their router to send traffic back to the old or the blue version, assuming that you kept your blue version of the prediction service running.
In a typical implementation of a blue green deployment, people think of switching over the traffic 100% all at the same time. But of course you can also use a more gradual version where you slowly send traffic over.
