AI Companies, Organizations, People Flashcards
EleutherAI
A grassroots collective of researchers working to enable open-source AI research. Notable for creating the pile dataset, which is what many topical large language models are trained and benchmarked on.
The Goals
Since most large language models are trained on private datasets based on common crawl data, their downstream generalization capabilities are limited.
However, with dataset diversity — a core feature of
The Pile — language modeling tasks will lead to improved downstream generalization capabilities.
While initially conceived as a training dataset for large-scale models, The Pile’s diverse nature proved to be useful as an evaluation tool.
The researchers hope that by using all of this data, they may be able to replicate the GPT3, only with more diverse data and for free. They also hope to create datasets in languages other than English in the future.
Ben Wang
kingoflolz
Created the Mesh Transformer JAX, which GTP-J was trained with
Alexey Dosovitskiy
Researcher from Google brain Berlin who developed ViT, the first transformer approach for vision task.
ViT first presented in May 2021
Dosovitskiy was working on one of the biggest challenges in the field, which was to scale up CNNs to train on ever-larger data sets representing images of ever-higher resolution without piling on the processing time. But then he watched transformers displace the previous go-to tools for nearly every AI task related to language. “We were clearly inspired by what was going on,” he said. “They were getting all these amazing results. We started wondering if we could do something similar in vision.” The idea made a certain kind of sense — after all, if transformers could handle big data sets of words, why not pictures?
The eventual result was a network dubbed the Vision Transformer, or ViT, which the researchers presented at a conference in May 2021. The architecture of the model was nearly identical to that of the first transformer proposed in 2017, with only minor changes allowing it to analyze images instead of words. “Language tends to be discrete,” said Rumshisky, “so a lot of adaptations have to discretize the image.”
Alex Ratner
CEO and co-founder of SnorkelAI
