10 - AutoRecSys Flashcards
What is AutoRecSys?
Automation of Recommender Systems development pipeline from data pre-processing to model selection and post-processing of predictions
What is the goal of AutoRecSys?
The goal of AutoRecSys is to make the development of a recommender system more efficient and accessible
What is the motivation of AutoRecSys?
- Automation of tedious components
- Focus on complex development tasks rather than time-consuming tasks
- Making development of recommender systems more accessible to the general public
- Many decisions in development are arbitrary
- Promote academic integrity and research
What is AutoML?
Automated Machine Learning provides methods and processes to make Machine Learning accessible to non-experts, to increase efficiency and to increase research in Machine Learning
What is the CASH problem?
Solving the combined algorithm selection and hyperparameter optimization problem
What is the Algorithm Selection Problem?
From a set of existing algorithms, choose the algorithm that performs best for the current problem
Why is algorithm selection also called a meta-learning approach?
Algorithm selection is performed with ML methods on ML algorithms - therefore it is called a meta-learning approach
What hyperparameter optimization methods are available?
- Grid Search
- Random Search
- Bayesian Hyperparameter Optimization
How does the hyperparameter optimization Grid Search work?
- Tests all combinations of given values for different parameters
- Exhaustive Search -> Simple but inefficient
- Array of parameters may not contain good values -> search never gets a good result
How does the hyperparameter optimization Random Search?
- Tests for parameter values that are randomly generated in a given interval
- Very high probability of finding a result close to the optimal result with few iterations if the parameter intervals cover sufficient parts of the optimal space
How does the hyperparameter optimization Bayesian Hyperparameter Optimization work?
- Structured approach to optimization
- Principle of exploration versus exploitation
- Very efficient, but mostly not parallelizable
What is Cross-Validation?
- Cross-Validation is standard for machine learning assessments
- Data is grouped
- In each group, a model is trained with certain hyperparameters and tested on the respective datasets
- Average of the test errors is given as the final result
What problems does Cross-Validation have?
- The best algorithm is the one that achieves the best performance on a test set
- In cross-validation, no uniform test set, therefore evaluation of the groups on a separate test set that is not changed and is the same for everyone
What are the advantages of Bayesian Hyperparameter Optimization?
- Extremely powerful
- Works for any learning task
- Automated
What are the disadvantages of Bayesian Hyperparameter Optimization?
Takes ultra-long to evaluate on many models, especially if you can not parallelize the processes
Was ist Ensembling?
- Tool to extend or replace hyperparameter optimization
- Ensemble performance is equal to the performance of hyperparameter optimization but much faster
What is the idea behind ensembling methods?
Ensembling methods are based on the idea that the weighted average prediction of many different models beats the performance of a single (optimised) model
What are the ensembling methods?
- Bagging
- Boosting
- Stacking
What is Neural Architecture Search?
- Define some basic building blocks and define a strategy to search for different compositions of these building blocks
- One of the most computationally intensive AutoML techniques -> benchmarks that include predefined neural networks to combat this problem and allow comparison of approaches
Which AutoRecSys Libraries did we get to know?
- RecZilla
- CaMeLS
- Auto-Surprise
- Auto-CaseRec
- Lenskit-Auto
- Elliot
What is RecZilla?
- Algorithm Selection Library specifically for recommender systems
- Creates an offline meta-model by learning the performance of meta-data sets
- Meta-data sets consist of meta-data of the original data sets
- Predicts the best algorithm by passing unseen meta-data through the meta-model
- Supports many metrics and even custom metrics
- Simple one-liner command
What is CaMeLS?
- Algorithm Selection Library specifically for Recommender Systems with a twist
- Cooperative meta-learning service for recommender systems developed by ISG
- Structure of meta-models similar to RecZilla
- Client-server application
- Meta-data and meta-model stored on server
- Client uploads meta-data and server return leaderboard of best algorithms
- Clients can contribute new meta-data with evaluation scores and improve the meta-model
What are the advantages of CaMeLS?
- Clients need almost no computing power to use the service
- New data can be made available on the server by anyone
- Meta data is anonymous by nature
What is Auto-Surprise?
- Automation of explicit feedback recommender systems library surprise
- Developed by ISG
Which criteria of an AutoML tool does Auto-Surprise fulfill?
- Automated Algorithm Selection
- Efficient Hyperparameter Optimization Procedure
- User-friendliness
What are the advantages of Auto-Surprise?
- Works well
- There is a publication
What are the disadvantages of Auto-Surprise?
- Surprise is no longer maintained
- Explicit feedback is hardly observed in practice any more
What is Auto-CaseRec?
- Automation of recommender systems library Caserec
- Developed by ISG
- Library works and there is a publication but Caserec is no longer maintained and explicit feedback is no longer intersting
What is Lenskit-Auto?
- ISG’s latest AutoRecSys tool - still in development
- Automation of the most popular Recommender Systems library Lenskit (in research and practical experiments)
What are the advantages of Lenskit-Auto?
Still maintained and supports implicit feedback
What is Elliot?
- Elliot is not an AutoRecSys library, but a framework for reproducibility
- Helps raise awareness of and solve reproducibility problems, enabling progress in AutoRecSys
- Performs a complete experiment, from loading the dataset to collecting the results
What is the core idea of Elliot?
Feed the system with a simple and straightforward configuration file that controls the framework by selecting experimental settings
What are the advantages of Elliot?
Unravels the complexity of combining splitting strategies, hyperparameter model optimization, model training and reporting of experimental results
What is Reproducibility?
Reproducibility is a central principle of AutoML and a prerequisite for scientific work in the field of artificial intelligence
What general problems do we encounter at AutoRecSys?
- Explicit-Implicit Evaluation difference
- Offline-Online Gap
- Clear Goals
- Different Use Cases
What is the Explicit-Implicit Evaluation Difference?
- Evaluation of explicit and implicit feedback is so different that these two categories need to be solved in completely different ways with AutoRecSys
- Shifting the interest to implicit feedback
What is the Offline-Online Gap?
- Differences in evaluations online and offline
- Often impossible to formulate the difference
- You can only optimize offline - how can you make sure you are optimizing for the right target?
What are Clear Goals?
- Recommender Systems often do not have a clear defined goal
- Applications of AutoRecSys techniques need clearly defined goals and the ability to optimize against them
- Offline-Online Gap strengthens this problem
What are Different Use Cases?
- Different possible applications for recommender systems and each needs (drastically) different approaches, also in optimization
- Difficult to create good automation pipelines and generalise new approaches
What is the problem with insufficient data quality and quantity?
- Researchers limited by publicly available data
- Comparison of new approaches with 20 year old datasets still state of the art
- Data sets rarely public for privacy reasons and expensive to process
- Experiments are mimicked with public datasets -> results often not the same
Considering the problem of low quality and quantity of data sets, can algorithm selection work at all?
- Algorithm Selection works with the meta-data of datasets
- Results show that even under the prevailing constraints, automated algorithm selection leads to remarkable results
What is the problem with the Hyperparameter Optimization Performance Wall?
Search time is several hours compared to a few seconds or minutes for baselines and the improvement in performance is less than one percent in some cases - is it really worth it?
Why are there no recommender systems specific ensembles so far?
- AutoRecSys Tools do not yet use ensembling
- Results from Lenskit-Auto promising but worth the effort?
- Ensembling is generally expensive and the results are only slightly better at the moment
Why is there no meaningful research in the field of neural architecture search for recommender systems so far?
Recommender systems architectures are specific to their problem domains, there is no one-size-fits-all solution
What is the problem with Automated Data Processing?
Automated pre- and post-processing is not yet available for recommender systems data sets
