Class 2: Introduction to artificial intelligence and its relationship to cognition Flashcards
Wolfram, S. L. (2023 February 14). What is ChatGPT doing... And why does it work? Stephen Wolfram Writings. https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work
The article discusses how ChatGBT’s primary goal is to produce a “reasonable” continuation of the text it has received. Explain what is meant by “reasonable.”
“What one might expect to write after seeing what people have written on billions of webpages, etc.”
Large language models produce a ranked list of words that might follow in a sentence, along with their probabilities. The AI will always pick the word with the highest probability.
FALSE
True or False: The human brain has about 100 billion neurons, each capable of producing an electrical pulse up to perhaps a thousand times a second. Neurons are connected with each other in a complicated net, with each neuron having tree-like branches allowing it to pass electrical signals to perhaps thousands of other neurons. The production of an electrical pulse in a given neuron at a given moment is independent of what pulses it has received from other neurons.
False: In a rough approximation, whether any given neuron produces an electrical pulse at a given moment depends on what pulses it’s received from other neuronsÑwith different connections contributing with different “weights.”
What are the potential risks associated with ChatGTP use in this article?
The article highlights some of the potential risks that come with using ChatGPT. One of the main risks is that if the model is not trained or monitored correctly, it may generate misleading or harmful information. Additionally, the widespread use of language models like ChatGPT could lead to a loss of privacy since these models require a large amount of data to function effectively. Lastly, the article acknowledges the ethical concerns surrounding the use of language models, including the risk of bias, and emphasizes the importance of ensuring that these models are used for the greater good of society.
What does ‘temperature’ parameter mean and what is the best number to indicate ‘temperature’?
It determines how often lower-ranked words will be used and ‘temperature’ of 0.8 is the best.
GPT-2 used sets of 12 attention blocks and attention heads in order to manage its decision-making process. How many does the improved GPT-3 use?
GPT-3 uses a collection of 96 attention blocks and attention heads.
When selecting the next word in a sequence, why doesn’t ChatGTP always just pick the word with the highest probability?
A lot of repetition would emerge and it would create an uninteresting essay.
Explain what softmax is and how it works
Softmax is the process of generating a probability distribution over any other possible words or phrases. It fundamentally takes a set of numbers and creates an output from the model’s neural network and maps it onto a probability distribution.
What are transformers and how do they help ChatGPT generate responses?
Transformers are a type of neural network architecture that allows ChatGPT to process sequences of input, such as sentences or paragraphs.
According to Stephen Wolfram’s article, what is ChatGPT trying to do?
Fundamentally, ChatGPT is attempting to produce a ‘reasonable continuation’ of the text it has been given.
How neural networks are being used to generate human-like responses to text prompts in the ChatGPT language model
ChatGPT uses a neural network called a Transformer, which is trained on a large text dataset and learns to predict the probability of a given the word or phrase appearing in a particular context.
How does ChatGPT generate “reasonable continuations” of text, and why is it sometimes necessary for the model to randomly select lower-ranked words to produce more interesting and creative output?
ChatGPT predicts the likelihood of a given word or sequence of words occurring in a sentence and generates text by selecting the next word with the highest probability. To produce more interesting and creative output, the model sometimes randomly selects lower-ranked words based on a “temperature” parameter.
What technology does ChatGPT use to generate replies?
One technique ChatGPT uses to generate response questions is called “generative pre-training”.
The __________ architecture is a type of neural network designed for processing sequential data, such as text, and uses self-attention mechanisms to enable the model to attend to different parts of the input sequence.
Transformer
What are the potential applications of ChatGPT?
Chatbots and virtual assistants are potential applications.
Define the term “Neural Networks”.
Neural Networks — a type of machine learning algorithm inspired by the structure and function of the human brain. They are composed of interconnected nodes or “neurons” that can learn and adapt to patterns in data to make predictions or classifications.
The whole process of training a neural net can be characterised by seeing how the loss (error) progressively decreases. And what one typically sees is that the loss ______ for a while, but eventually flattens out at some constant value. If that value is sufficiently ______, then the training can be considered successful - otherwise it’s probably a sign one should try changing the network architecture. A. increases, large B. increases, small C. decreases, large D. decreases, small
D. decreases, small
Finish the sentence: If you’re trying to get a neural net to learn a function (e.g. to replicate a graph with a boxy line), you first have to choose/figure out the weights. This is done byÉ
Supplying lots of input to output” examples to “learn from” - and then to try to find weights that will reproduce these examples. “
True or False - The number of possibilities is larger than the number of particles in the universe.
Yes.
What bias happens in this situation? Lucy heard that Mike injured Daivd, so she very hates Mike. After this, no matter how Mike and other friends persuade her and explain the reason for hurting David to her, she only believes what she heard.
Confirmation Bias.
Explain the function of a perceptron.
Perceptrons help classify the data that is input to the neural network. It is classified in two parts, therefore it is known as a linear binary classifier. It functions as the most simple part of a neural layer.
True or False: The ‘temperature’ parameter in ChatGPT that determines how often lower-ranked words are used in essay generation is best at 0.9.
FALSE
Fill in the blanks: ChatGPT is based on a ______ network. It is essentially trying to produce a “r_________ c___________”
Neural, reasonable continuation.
About how many neurons are in the human brain?
100 billion
What are the future development directions of ChatGPT?
The future development direction of ChatGPT includes better dialogue quality and efficiency, better emotion recognition function and multilingual support.
Fill the missing word. In neural net training, the numerical values assigned to the connections between neurons in a neural network are known as (missing word).
weights
rue or False: ChatGPT produces a ranking list of probabilities on what would the next word most likely to be and always use the highest-ranking word.
False. There is randomness in essay generation which makes it more “creative”.
True or False: ChatGPT utilises whole words to compute.
False (generally), it uses “tokens which are linguistic units that could be whole words or segments like “pre” or “ing” or “ized”. “
True or False: are neural nets only relevant to human brains?
False
How does Chat GPT take user feedback into account?
When users rate Chat GPT’s output, a new neural network model is created to predict user ratings. The new model then runs like a loss function on the original network continually adjusting the network to user preferences.
What is the tradeoff between capability and trainability?
The more capable a system is, the less trainable it becomes. Conversely, the more trainable a system is, the less capable it becomes.
What is a neural net?
Neural nets are a type of machine learning algorithm that are simple idealizations of how human brains seem to work. Like a human brain, neural nets learn more through practice and repetition. The nodes and neurons that make up these neural nets perform mathematical operations on the input data to produce an output.
What is a temperature parameter for ChatGPT responses?
A temperature parameter determines how often a lower-ranked word (relative to the highest-ranked word calculated to be a probable match for responses) will be selected. For example, when ChatGPT responds to essay prompts, a ‘temperature’ of 0.8 yields the best results in essay generation.
Is it the following statement true or false: it’s not clear whether there are ways to “summarize what it’s doing”
TRUE
Explain the concept of temperature as it pertains to language generation in ChatGPT, including the ideal temperature that ChatGPT uses to predict words.
It is a parameter that is used to adjust the randomness and creative output of of ChatGPT. The ideal temperature is approximately 0.8, otherwise the text becomes repetitive. A temperature of 0.8 allows for a degree of randomness while still ensuring the output is relevant and coherent.
how does ChatGTP talk to us continuously (work)?
It operates in three basic stages. First, it takes the sequence of tokens that corresponds to the text so far, and finds an embedding (i.e. an array of numbers) that represents these. Then it operates on this embeddingÑin a “standard neural net way”, with values “rippling through” successive layers in a networkÑto produce a new embedding (i.e. a new array of numbers). It then takes the last part of this array and generates from it an array of about 50,000 values that turn into probabilities for different possible next tokens. (And, yes, it so happens that there are about the same number of tokens used as there are common words in English, though only about 3000 of the tokens are whole words, and the rest are fragments.) However, according to ChatCPT’s answer, it is an AI language model created by OpenAI. It communicates with us through natural language processing (NLP) technology. It analyzes the text us enter and uses a combination of algorithms, statistical models, and machine-learning techniques to understand the meaning of our input and generate a response that best answers our question or fulfills our request.
TRUE/FALSE:Showing a neural net repetitive examples when training it is always redundant because it is in the same state for each training round.
FALSE. Generally, neural nets need to see a lot of examples and at least for some tasks, the examples can be incredibly repetitive. It is standard strategy to show a neural net all the examples one has, over and over again. In each of these training rounds” the neural net will be in at least a slightly different state, and somehow “reminding it” of a particular example is useful in getting it to “remember that example.” However, it is normally also also necessary to show the neural net variations of one example.”
True or false: The architecture behind ChatGPT allows it to constantly learn from previous interactions in order to tailor its responses to individual users.
TRUE
True or False: ChatGPT is expected to produce a reasonable continuation of existing texts on a related topic.
TRUE
ChatGPT will always pick the highest ranked word when deciding what word to pick next
False. Always picking the highest ranked word can make a piece of text seem flat so lower ranked words are often used to make the text more interesting
True or False: It is possible to completely eliminate cognitive biases.
False. Although there are steps and ways to reduce dependency on cognitive biases, it is not possible to completely eliminate them.
Inside ChatGPT is a giant _____ consisting of 175 billions of weights
Neural net
What does ‘temperature’ mean in regards to ChatGPT?
A parameter that determines how often lower-ranked words will be used.
True or False: Often just showing a neural net the same example over and over again isn’t enough. It’s also necessary to show the neural net variations of the example. These variations may only need to be slight modifications.
TRUE
If you give ChatGPT the same prompt several times, would you get the same answer each time or different answers?
Different answers
What does ChatGPT stand for?
Chat Generative Pre-Trained Transformer
True/Flase -> In a neural net bigger networks generally do better at approximating the function we are aiming for.
True
What is the overall goal of ChatGPT?
to continue text in a “reasonable” way, based on what it’s seen from the training it’s had (which consists in looking at billions of pages of text from the web, etc.)
What is the source of the training data for ChatGPT? a) A large collection of images b) Randomized phrases generated by a computer c) Human-written text from books, the web, and other sources d) Sounds and speech recorded from people
C: Human-written text from books, the web, and other sources
How do we make a neural net do a recognition task?
Take an input corresponding to a position (x,y) and to recognise it as whichever of the three points it is closer to
What is “X”: To train ChatGPT, neural net training is seeking “X” to reproduce the given examples?
“X” = Weights, the neural network relies on the weights to interpolate (generalise) between the given examples
What are some limitations of ChatGPT
As GPT is not perfect, it can have errors in generating biased or inappropriate responses as well as not having a full understanding of language. Also, it can be limited to what resources it receives
What is the fundamental goal of ChatGPT?
The fundamental goal of ChatGPT is to produce a “reasonable continuation” of a given text, based on what one might expect someone to write after seeing what people have written on billions of web pages, etc.
Define the term ‘loss function’ in the training process ofneural nets
The discrepancy between the current values of the function and the desired function. This value is calculated in order to adjust the weight of the function to be able to reproduce the function that we want.
What is the purpose of a temperature parameter?
ChatGPT incorporates a temperature parameter to determine the frequency of utilising ‘low-ranked words’.
True or False: ChapGPT has been trained on vast amounts of text data to improve its accuracy and ability to understand the context.
TRUE
What is so special about machine learning through neural nets?
Their ability to learn to do things
The basic operation of the (___) is also very simple, consisting essentially of passing input derived from the text it’s generated so far “once through its elements” (without any loops, etc.) for every new word (or part of a word) that it generates
The basic operation of the (neural net) is also very simple, consisting essentially of passing input derived from the text it’s generated so far “once through its elements” (without any loops, etc.) for every new word (or part of a word) that it generates.
True of False: During training, ChatGPT will progressively adjust the weights in the network in an attempt to accurately reproduce the desired function.
True. The training method uses a loss function (how far away are the current weights from the desired end goal). This loss function will decrease progressively until the network reproduces the desired function *(within an approximation margin).
The neurons are connected in a complicated net, with each neuron having _________ branches allowing it to pass electrical signals to perhaps thousands of other neurons
tree-like
A model commonly used to train Artificial Intelligence which reflects the processes of the human brain is called a what?
Neural net.
What is ChatGPT fundamentally trying to do?
ChatGPT is trying to produce a continuation of the text it has gotten so far with the available input it has access to.
What is the question that ChatGTP constantly asks itself as it writes?
Given the text so far, what should the next word be?
What the difference between syntactic and semantic grammar, and what’s necessary for ChatGTP to handle the latter?
Syntactic grammar refers to the structure of language, where as semantic grammar refers to the meaningfulness of language. For ChatGTP to properly grasp semantic grammar, it would need a “model of the world” to refer to, which could be acheived through coding.
What are the potential ethical issues of CHATGPT mentioned in the text? (At least two)
Bias; discrimination; misinformation; manipulation; privacy; security.
What does the optimal temperature for essay generation refer to?
The optimal temperature for essay generation refers to the temperature setting used in language models like GPT-3 to control the degree of randomness and creativity in the generated text. The temperature determines the degree to which the model is willing to take risks and produce unexpected outputs, versus sticking to more predictable and safe choices.
Define what reasonable continuation means when said “ChatGPT is always fundamentally trying to do is to produce a reasonable continuation” of whatever text it’s got so far.””
Reasonable continuation in this context alludes to what might be written after reviewing billions of readings. The ChatGPT reply produces a list of ranked words that might follow the previous word, together with “probabilities” after reviewing the readings. However, “temperature” parameter that determines the probabilities of how often lower-ranked words will be used.
Describe the term loss function.
The loss function calculates the sum of the squared differences between a machine learning model’s anticipated output and the actual output (the goal) for a given input. More importantly, it’s an essential concept in machine learning because it’s used in ChatCTP to direct training and gauge how well the model fits the training data.
What are neural networks and LLMs (like ChatGTP) simple idealisations of?
The human brain
define neural nets
simple idealisations of how the brain works - specifically discussed is the process of how humans form a thought upon recognising something
Fill in the blank. ______ _______ is a machine learning algorithm that mimics the idealised function of the human brain and was used to train ChatGPT.
Neural Network
Define computational irreducibility
Computational irreducibility is a concept that refers to the idea that some computational problems cannot by simplified or reduced in a meaningful way. This concept says that some problems do not have quick or predictable ways to solve, essentially a limit on computational capabilities (such as neural networks). Rather these problems must be studied through human intuition, experimentation, and observation.
what makes neural nets so useful?
Is not only can they in principle do all sorts of tasks, but they can be incrementally trained from examples to do those tasks.
Why does ChatGPT use ‘tokens’ instead of words?
The use of tokens instead of words makes it easier for ChatGPT to deal with rare, compound, or non-English words. Tokens can be words, and can also be parts of words, such as “ing”, “pre”, “anti” etc.
Define embedding.
Embedding involves assigning numbers to text and words to represent their meaning, and grouping similar meanings to nearby numbers.
How many connections/weights are there in ChatGPT’s neural net?
Approximately 175 billion
What does ChatGPT stand for? Give a sentence on what it does
ChatGPT stands for Generative Pre-Trained Transformer (GPT) and is a language model that uses a text through its large dataset to generate responses to prompts and questions an individual may ask.
The concept of ‘embeddings’ refers to the way we try to represent the essence of something by an array of numbers (true/false).
True!
How does ChatGPT generate responses for you?
After analysing the prompt that you give it, it then uses statistical probability of what it has learned to generate a response that is likely to be relevant and informative
What is ChatGPT?
ChatGPT is large language model (LLM) that is trained on large amounts of human-created text. It then utilises this information to estimate probabilities and generate meaningful text after given a prompt.
What are some potential limitations of ChatGPT?
bias from the datasets used for training, risk of generating inappropriate or offensive responses, lack of understanding of social norms or cultural context, relies on large amounts of data (not suitable for situations with little data or privacy restrictions)
Define “Embedding”.
The assignment of a number to a type of stimulus (in ChatGPT’s case, common English words) that help to group like stimulus with a similar essence” together. “
What is a temperature parameter?
Temperature is a parameter used to control the level of randomness/unpredictability/creativity in the generated text. Higher temperatures result in more diverse and unpredictable output and lower temperatures result in more conservative and predictable outputs
True or False “Machine Learning codes the defining characteristics of an object and uses these said defining characteristics to determine what the object is”
False, Machine learning acquires a plethora of examples to determine whether the new object fits within the constraints of the prior examples.
Why are loss functions important for large language models?
Loss functions are important for large language models like GPT as they provide a measure of how well the model is able to predict the next word or sequence of words in a given text, allowing the model to generate more accurate and contextually appropriate text by minimizing the loss function during training.
What is a neural network?
A neural network is a type of machine learning algorithm that is modelled on the human brain. It is made of layers of interconnected nodes, or neurons, which perform mathematical computations on input data and pass the results to the next layer until the final output is generated. Neural networks can model a wide variety of functions with high execution and training performance.
TRUE or FALSE: Artificial neurons in ChatGPT take numerical inputs, multiply them by some weights and feed them forwards to end up with the next ‘token’ in a sequence.
TRUE
How many percent is AI’s ability to learn ?
4.5
What is a neural net?
Neural networks are simplified models inspired by the workings of the human brain. Our brains are complex networks of nerve cells that are connected to assist in processing information. When we look at an image, photoreceptor cells at the back of our eyes convert the image into electrical signals that travel through layers of neurons to help us recognize the image. Neural networks use mathematical functions to simulate this process.
The whole process of training a neural net can be characterised by seeing how the loss (error) progressively decreases. And what one typically sees is that the loss ______ for a while, but eventually flattens out at some constant value. If that value is sufficiently ______, then the training can be considered successful - otherwise it’s probably a sign one should try changing the network architecture. A. increases, large B. increases, small C. decreases, large D. decreases, small
D. decreases, small
What is backpropagation?
Backpropagation adjusts the weights and biases of the ANN and corrects its random guesses and to make them less wrong. The way an ANN learns is by making adaptive changes. The probability of making the right calculation improves with each backpropagation and is one of the most frequently used learning rules in many applications of artificial neural networks.
What kind of prompt tend to make ChatGPT to “wander off” in non-human-like ways?
Having to make longer texts. Essay, stories etc.
How does “temperature” help ChatGPT generate text that sounds like a human wrote it?
When ChatGPT generates text, it selects its next word from a ranked list of words based on their probability of being next. It asks repeatedly what the next word should be and adds it. The temperature parameter(with a value of 0.8 considered as optimal) determines how often lower-ranked words will be used. The randomness in ChatGTP’s selection of lower-ranked words makes for more interesting writing.
What is a large language model (LLM) and how does it estimate the probabilities with which sequences of words should occur in natural language text?
A large language model (LLM) is a neural network trained on vast amounts of text to predict the likelihood of a given word or sequence of words occurring in a sentence. The LLM uses this training to generate more accurate and coherent language output, and complete tasks such as language translation, summarization, and question answering.
Decribe a neural network
A neural net is a computational model that is modeled/inspired after the structures and function of the humain brain. Neural nets consists of interconnected nodes or “neurons” that are arranged into a layered structure. It can then be trained to learn patterns and relationships in data using large data sets.
True or False: When discussing how ChatGPT is trained to retrieve correct/true answers, the “mountain lake” metaphor describes how training neural nets to predict true values is limited to its weight landscape. In neural network training, a loss function is created to measure the difference between predicted and true values. The goal of training stages is then to minimise the loss function. However, the mountain lake metaphor illustrates that this optimisation process is not guaranteed to find the global minimum of the function, but only a local minimum - similar to how water flowing down a mountain will eventually end up in a lake at the bottom. There may be other lower points in the loss function (an ultimate global minimum) that are not reached in training stages.
TRUE
What are the thumbs up and thumbs down buttons on the right for?
They provide feedback for ChatGPT’s responses and help inform its future responses
Why does ChatGPT take a while to generate a long piece of text?
It is because when ChatGPT generates a new token, it has to do a calculation involving every single one of the 175 billion weights.
According to the Wolfram (2013) article, what is one of the main challenges associated with developing natural language processing systems like ChatGPT?
Training the machine learning models used by these systems on diverse and representative datasets in order to generate high-quality, human-like responses.
How does one train a neural network from text? a) By presenting a batch of examples and then adjusting the weights in the network to minimize the error b) By presenting the network with a list of common words and their definitions c) By inputting large amounts of data and allowing the network to learn on its own d) By manually adjusting the weights in the network to produce the desired output
A: By presenting a batch of examples and then adjusting the weights in the network to minimize the error
Briefly describe the process of neural net training
A large number of examples of input and output are to be given to the system. The system will then ‘learn’ from these examples. What we can do is find the weight that is suitable for the system to be able to reproduce these examples, by relying on its ability to generalise between the examples reasonably.
What is the key reason for the neural net in ChatGPT to be so useful?
That is somehow captures a “human like” way of doing things/thinking.
How many weights does chatGPT have?
175 billion
True or False: can neural nets be trained to do different tasks for effectively?
TRUE
What is computational irreducibility?
Computational irreducibility is a phenomenon in which there are some computations that can’t be reduced to simpler steps, and must be computed step-by-step in order to determine the outcome.
In Essence, what is a neural network trying to minimize?
Error
Finish the sentence: a Neural Net is a simple
Idealisation of how brains seem to work.
What is ChatGPT?
ChatGPT is a natural language processing software that uses machine learning to generate responses to text inputs that mimic human-like communication.
How does ChatGTP ensure that its essays aren’t boring?
It uses a temperature of 0.8 - which means 20% of the time, it will randomly select a word that isn’t the highest ranked word.
When ChatGTP is writing an essay, what is the purpose of having a “Randomness” function (with a temperature of 0.8) in ChatGPT rather than letting it choose the most probable word that would follow?
To create an essay that sounds more creative and less flat than it otherwise would.
What is ChatGPT’s basic task?
Continue a piece of text that has been given
With the result of list of words, will the highest-ranked words be picked to add to the essay (or whatever) that it’s writing? T/F
False.
ChatGPT when determining its probabilities and consequently which words it picks utilises words of ____ probability to achieve writing of ____ style.
- A moderate 2. Creative and more interesting
The best way in simple terms, to ChatGPT is that it is representing reasonable continuation of the text it was given to work with so far. Through the scanning of the never ending literature on the internet, it bases it’s answers off probability and matches in meaning to create readable information that makes sense to the human language. And this is only the beginning of what it can do..True or False?
TRUE
Fill in the blank: The ______ ___ passes input from the text generated so far “once through its elements” (without loops) for each new word (or part of a word) that it generates.
Neural net
What are the three important aspects of working with Neural Network?
- The Architecture of a Neural Network needs to be considered for a particular task. 2. It is critical to obtain the necessary data to train the Neural Network 3. It is important to incorporate existing, trained Neural Networks or use them to generate training examples for a new Neural Network.
ChatGPT’s model produces a “reasonable continuation” of text based on what? A) The model uses random words to continue the text B) The model uses a predefined set of words to continue the text C) The model uses a probability-based ranking system to determine the most likely word to follow the given text based on billions of webpages and digitized books D) The model uses a word association tool to predict the next word in the text
C) The model uses a probability-based ranking system to determine the most likely word to follow the given text based on billions of webpages and digitized books.
What is unsupervised learning in neural net training?
It’s where the neural net must find patterns in the data on its own.
What does GPT stand for?
the GPT in ChatGPT stands for Generative Pre-trained Transformer, which is a type of language model based on deep learning
How is Chat GPT different from other language models?
ability generate longer and more complex responses, use of contextual information to generate more relevant responses, ability to generate responses that are more diverse and creative
True or False: Pre-training is the process of adapting a pre-trained model to a specific task by training it on a smaller labeled dataset.
False. (Pre-training is the process of training a model on a large dataset of unlabeled data before fine-tuning it on a specific task.)
How many weights does ChatGPT contain?
175 Billion
Why does computational irreducibility contrast learnability?
Computational irreducibility is when you must go through each computational step to get the result, something that the brain presumably has chosen to avoid. Learnability ‘compresses data by leveraging regularities,’ whereas computational irreducibility implies that there are limits to the regularities that exist, contradicting learnability.
What is Computational Irreducibility?
Computations that cannot be sped up by means of any shortcut are called computationally irreducible. The principle of computational irreducibility says that the only way to determine the answer to a computationally irreducible question is to perform, or simulate, the computation.
TRUE OR FALSE: ChatGPT has no limitations and is without risk of bias or generating inappropriate or offensive responses.
FALSE
How is ChatGPT able to produce different writing answers?
It is based on the probabilities. According to the author, it is explained that the probabilities are based on the frequency of the words and phrases in the training data and are adjusted by the neural network during training. It is also noted that the probabilities assigned by ChatGPT are influenced by the context of the input text and the system’s previous responses.
Define tokens
Convenient linguistic units that might be whole words, or may be pieces, such as, “pre,” or “ing,” or “ized,” etc.
True or Flase If you had a big enough neural net then you might be able to do whatever humans can readily do.
TRUE
What are the three basic stages of ChatGPT?
- ChatGPT first takes the sequence of tokens that corresponds to the existing text, finding an embedding. 2. ChatGPT then looks at values of successive layers in a Neural Network, producing a new embedding. 3. It then takes these array of numbers, generating about 50,000 values. These values turn into probabilities for the next possible tokens which represent the different possible words that ChatGPT will use to respond to a prompt.
Why does ChatGPT not use the ‘highest-ranked’ word in the list of words with probabilities?
To make it more interesting. If it chose the highest ranked word, the passage would be ‘flat’ with no ‘creativity.’
How does GPT learn all the information?
GPT is pre-trained on a large corpus of text, and then fine-tuned on specific tasks like language translation or text completion
How does ChatGPT use weights to write believable text?
It uses weights in its algorithm to assign importance to certain features of language it has been exposed to, for example, the frequency of words, to create a response. Weights are ChatGPTs understanding of the patterns in the input data which help it generate believable text.
True or False: When doing a task like writing an essay, ChatGPT always picks the word with the ‘highest probability’ to add at each step.
False.
In a neural network what is the role of the loss function?
The loss function gives us the distance between the values we have got and the true values.
True or False: A lower temperature corresponds to a higher probability of choosing words that are lower-ranked words.
False, A higher temperature corresponds to a higher probability of choosing words that are lower-ranked words.
How does ChatGPT produce a “reasonable continuation” of text?
ChatGPT produces a reasonable continuation of text by scanning billions of pages of human-written text and analyzing patterns in how words and phrases are used. It then generates a ranked list of words that might follow, along with probabilities based on how frequently they occur in similar contexts.
For ChatGPT what is the fundamental, underlying algorithm?
Transformers
In terms of Chat GPT sequencing, fill in the blank: But the end result is that it produces a ranked list of words that might follow, together with ÉÉÉÉ.
Probabilities
What is the function of an ‘Attention Head’?
An attention head allows ChatGTP to look back over previous words in it’s sequence rather than just the last word in order to make a better selection for the next word.
What is ChatGPT doing at each step of generating text, and why does it sometimes randomly pick lower-ranked words?
ChatGPT uses a large language model (LLM) to generate text. It produces a ranked list of words that might follow a given text, with associated probabilities. It adds words to the text by repeatedly asking which word to add next, based on the ranked list and a temperature parameter that introduces randomness. The highest-ranked word is not always chosen, as this can result in repetitive, uncreative text. The result is that ChatGPT can generate a variety of unique texts.
Does ChatGPT operate in 3 stages? True or false? Explain
It runs through three fundamental phases. Initially, it looks for an embedding that corresponds to the sequence of tokens that so far correspond to the text. Ultimately, it performs operations on this embedding to create a new embedding in a “typical neural nett approach,” with values “rippling across” subsequent layers of a network. The last element of this array is then used to create an array of around 50,000 values that represent probability for various possible following tokens.
True or false: ChatGPT was taught explicit grammar/syntax rules of every language.
False, it uses the given data to learn” syntax implicitly. “
True or False: When deciding on the next word to say, ChatGPT always uses the word with the highest probability
False - When words with the highest probability are always used, the text comes out as flat and tends to iterate over itself.
What are the three basic steps of ChatGPT?
- It takes a sequence of tokens that correspond to the text so far and finds embedding. 2. Data goes through the neural network to create a new embedding. 3. This new embedding is then used to calculate the probabilities for the next word.
True or False: It can be easier to solve more complicated problems with neural nets than simpler ones.
TRUE
What does a “token” refer to?
linguistic units ChatGPT operates with such as whole words, or word pieces such as “pre” or “ing” or “ized.
ChatGPT uses the given text so far to ask over and over what should the next word be. The output is also thought of as pieces of words and can also be called a _____
“token”
What does the term temperature refer to in the context of a machine learning language model?
The temperature parameter refers to how often lower-ranking words are used.
The examples of code throughout the article are written in Wolfram Language
TRUE
What is an embedding?
An embedding is a way to try to represent the “essence” of something by an array of numbers - with the property that “nearby things” are represented by nearby numbers. And so, for example, we can think of a word embedding as trying to lay out words in a kind of “meaning space” in which words that are somehow “nearby in meaning” appear nearby in the embedding. The actual embeddings that are usedÑsay in ChatGPTÑtend to involve large lists of numbers.
True OR False: ChatGPT involves the same kind of reprocessing or feedback loop as seen in other computational machines, such as the Turing Machine.
FALSE! Although in a sense ChatGPT does “re-read” tokens it has previously generated, this information is never repeatedly reprocessed. Rather, each previous token is used only once by each computational element when generating a new token; to help understand the context and encoding of the passage. When a new token is being generated, the passage will be “fed back” and used to determine the next appropriate token; but never re-processed or changed.
Define the term “neural net”
Neural net - the core of ChatGPT, where through its many layers of interconnected artificial neurons are able to produce a recognisable human-like language.
How does neural net training work, and what is a loss function?
Neural net training involves finding weights that make the network successfully reproduce input-output examples provided during the training process. To do this, a “loss function” is computed at each stage, measuring the difference between reported and true values. The weights are then adjusted to minimize the loss function, typically using a technique called steepest descent. The goal is to find a set of weights that minimizes the loss function and produces accurate outputs for new inputs.
True or False? ChatGPT is capable of generating computer code.
True.
Neural Networks show depict the behaviour of the human brain that enable computerised programs to find patterns and solve common problems in machine learning and…
AI & deep learning
Why neural nets are useful?
It is because they somehow capture a “human-like” way of doing things.
True or false does ChatGPT use a neutral net?
TRUE
True or False? The architecture of neural net would differ from one particular task to another.
False. The same architecture of neural net can work for different types of tasks, as the neural net can typically capture general human-like processes.
What is the concept by which the essence of something is represented by a string of numbers, and similar concepts by similar numbers?
Embedding
What is one of the concerns with relying on ChatGPT for research?
The accuracy and validity of the information
A neural net can be said to operate similarly to which human organ?
The brain
ChatGPT has an excellent understanding of the real-world context. True or False?
False. - they only generate texts based on its relationships and patterns it has learned from the training data - for longer and more complex texts, they might “wander off in non-human-like ways and hence, might not always provide the most suitable/accurate responses “
What is the most popular and successful and current approach for image recognition to actually work?
neural nets
What does ChatGTP produce in the result?
It produces a ranked list of words with probabilities.
Fill in the blank - “The elements are already in there, but the specifics are defined by something like a __________ between those elements.”
Trajectory.
Describe in simple terms how a Neural Network calculates its weights.
It calculates how far away the output is from the desired result (loss function) and adjusts the weights until the output is as closed to the desired result as possible.
What is embeddings?
Embedding can be seen as a way to represent the “essence” of something by an array of numbers - with the property that “nearby things” are represented by nearby numbers.
What is a symbolic discourse language?
A symbolic discourse language is a language that can describe the world and its concepts in a precise, unambiguous way. This language is essential for creating a semantic grammar that can understand the meaning of language beyond just its syntax. Computational language has the advantage of being precise and can be used to build a symbolic discourse language. Such a language could be used to generate text, ask questions about the world, and make assertions about the world. It would be a valuable tool for scientific research and artificial intelligence.
How can we estimate the probabilities of letters in English text?
By taking a sample of English text and calculating the frequency of each letter
What does LLM stand for?
Large Language Models
True or False: ChatGPT can incorporate context from the prompt it receives to generate responses
True. ChatGPT can use context from the prompt it receives to generate responses that are tailored to the specific situation.
What are the limitations of ChatGTP?
- As an AI, it can’t truly understand the context it uses in chat. 2. Some uncommon words will confuse it by less training in those words. 3. It is unreliable by lack of human judgement and ethical consideration.
ChatGPT and other natural language processing models are already at a relatively mature stage. (judge true or false)
False.
What approach does ChatGPT use to generate responses?
ChatGPT generates responses by using machine learning to analyze and learn from extensive amounts of data.
What process does ChatGPT use to generate unique responses even with the same input command?
Sampling
What is computational irreducibility?
Computational irreducibility is when you must ‘trace out’ each computational step from the initial conditions to get the result. You are unable to get the result based off of the initial conditions alone.
A neural net is…
a machine learning technique that reflects the behaviour of the human brain, composing layers of neurons that work together.
How does ChatGPT compare to earlier language models like ELIZA and ALICE?
ChatGPT is significantly more advanced than earlier language models like ELIZA and ALICE, due to its use of large-scale transformer-based architectures and pre-training techniques, which allow it to generate more sophisticated and human-like responses to text-based prompts.
What is ChatGPT?
ChatGPT is a large language model that is trained to generate outputs based on contextual (word-based) prompts given by the user and the backlog of textual information it has been fed by the team at openAI. It then creates a chart of probabilistic answers that would fit the context and question it was asked, chooses a word, and does this word by word, until a suitable answer is created
Neural networks are based off how the human brain works, T/F
True! Neural Networks are a simple idealisation of how brains seem to work; neurons produce electrical pulses, passing electrical signal to thousands of neurons, an individual neuron will send an electrical pulse depending on the pulse it receives.
Neural Nets can perform recognition tasks.
TRUE
True or false: Neural nets are trained from being fed examples, rather than specifically being programmed to distinguish things.
True.
Define this term for which ChatGPT is based: unsupervised learning
Trained algorithm only on feature variables containing only the input variables, not the output variables. Rather than responding to feedback, the algorithm identifies commonalities for correct classification
How does ChatGPT work?
ChatGPT utilises a neural network that has undergone extensive training using large volumes of textual data. It is enhanced with attention mechanisms and transformer networks to produce responses to textual inputs that are both understandable and similar in style to those made by humans.
If the AI cannot continue the sentence, what temperature does it go to, and what happens to the sentence?
0 temperature, it is repetitive and can be confusing
What is the ideal “temperature of randomness” (the probability of picking higher or lower ranked words) for the ChatGPT system to show creativity or interest in its writing?
0.8
Neural networks like ChatGPT are able to do previously-unthinkable things like write complete essays because they have overcome the hurdles previous systems encountered with computationally irreducible processes. (T/F)
False. ChatGPT still experiences the same difficulties with those irreducible processes, it’s just that human processes such as essay-writing have turned out to be simpler than first thought in computational terms.
Fill in the blank: _ is a way of representing text with numbers. One can think of a/an _ as a means of representing the ‘essence’ of something by an array of numbers, with things nearby in meaning represented by nearby numbers.
Embedding
True or False - ChatGPT always uses the words that have the highest rank each time
False. To make the answers that it gives more “realistic” and “creative”, ChatGPT has built in a certain amount of randomness when choosing words to make up its response, where it chooses lower ranked words some of the time
What is ChatGBT fundamentally trying to achieve?
To create a reasonable continuation of the text it has received, through a ranked word list that includes the probability of each word.
Fill in the blank: The basic task for ChatGPT is to figure out how to continue a piece of (blank) that it’s been given.
TEXT
The tendency to search for, interpret and remember information in a way that confirms one’s pre-existing beliefs or hypotheses is known as —————
Confirmation Bias
True or False: ChatGPT works by recognizing patterns in the text data and generating responses based on those patterns.
TRUE
How is ChatGPT able to generate insights and solutions from wide ranges of disciplines and studies?
ChatGPT has the ability to learn from vast amounts of text data, ranging from books to web pages. This allows it to recognise patterns and generate text responses that directly answer the given prompts.
What are neural nets with reference to image recognition?
Neural nets can be perceived as a simple idealisation of how a brain works, and mimic the neural network of the brain where whether a neuron produces an electrical pulse - at any given moment, depends on the electronic signals it receives from other neurons and assigns a ‘weight’ to the connection.
Which feature of neural net architecture is ChatGPT’s most notable?
transformer
How does ChatGPT generate new responses and learn language patterns?
GPT uses and learns language patterns through using a “multi-layer transformer neural network” which allows it to analyze and understand these complex language patterns to develop a response.
True or False? Artificial intelligence has the hardest time distinguishing what something is at the edges of their ‘attractor basins.’
TRUE
What is a model?
some kind procedure for computing the answer rather than just measuring and remembering each case.
True or false: ChatGPT will always respond with the “highest probability word”
False, there are variations; often the highest probability word is chosen, but lower probability words are also used in a certain ratio.
A neural net can be shown data once and learn that particular task right away. True or false?
False.
Why is there a trade-off between capability and learnability with computational systems?
Using its full capacity, a system makes use of computational irreducibility. By performing every task in the sequence to get to the result, as in computational irreducibility, it is less able to recognise patterns, skip steps, and in turn learn and be trained.
Outline the three basic stages of ChatGPT processing.
- The current string of tokens is split in an embedding module, with different values being converted or generated, and ultimately added together to create an overall embedding vector. 2. These new embedding vectors are fed into the attention heads of the transformer, which help to repackage and reweight specific information from previous tokens. This creates a final embedding vector that is a collection of the sequence so far. 3. The final embedding of the collection is decoded and used to assign the probability of what token should logically follow in the sequence.
what are tokens
linguistic units, these can be pieces of a word or whole words
TRUE/FALSE: Showing a neural net repetitive examples when training it is always redundant because it is in the same state for each training round.
FALSE. Generally, neural nets need to see a lot of examples and at least for some tasks, the examples can be incredibly repetitive. It is standard strategy to show a neural net all the examples one has, over and over again. In each of these training rounds” the neural net will be in at least a slightly different state, and somehow “reminding it” of a particular example is useful in getting it to “remember that example.” However, it is normally also also necessary to show the neural net variations of one example.”
How is ChatGPT trained?
ChatGPT is trained on vast amounts of text data, such as books and websites, to learn patterns in language usage and structure. This allows it to generate responses that are coherent and convincing.
What is the definition of ChatGPT(e.g.What does ChatGPT do?)
ChatGPT is a technology tool that can automatically generate something that reads even superficially like human-written text
The most popular and successful models for tasks such as image recognition is ______ ____.
Neural Nets
ChatGPT tries to give the most accurate response possible. True or false?
False. ChatGPT tries to give the answer you would most reasonably expect, not necessarily the most accurate answer
True or False? ChatGPT is essentially a network of artificial neurons that collect numerical inputs and combine them with weights.
TRUE
What is an embedding?
An embedding is a way to represent something, for example words or sentences, through numbers. Words or sentences with similar meanings will therefore have similar numbers to represent them.
List ChatGPT’s basic three stages of Operation?
- It takes a sequence of tokens/words that correspond to the current text and finds an embedding (i.e. an array of numbers) that represents these. 2. It then utilises the neural network to continue operating on this embedding working through the successive layers of the network implementing new values across the layers to give a new embedding. 3. It finally utilises the last part of the array (i.e. created in the new embedding) to generate about 50,000 values which are converted into probabilities for a range of possible next tokens
Which bias we can define in this story? Bob praised Nancy for looking good in blue. Then, Nancy preferred to buy more blue clothes.
Anchoring Bias.
True or False: Does an AI using a neural network generalise examples to distinguish and identify images?
TRUE
What is the basic process of how language models like ChatGPT write an essay?
Language models like ChatGPT write an essay by repeatedly asking “given the text so far, what should the next word be?” The model uses a type of neural network called a transformer to predict the next word or token in a sequence based on the words or tokens tha came before it. During training, the transformer is presented with a sequence of words or tokens and is trained to predict the next word or token in the sequence. When the language model is used to generate new text, it follows the same process by predicting the next word or token based on the patterns it has learned from the training data and adding it to the sequence until the desired length of text has been generated.
True or False: ChatGPT is able to do “unsupervised learning”
True: ChatGPT is able to learn from texts that it has been given by using covered versions of the texts as input, and the uncovered versions of text as output
Why may neutral nets solve more complicated problems easier than more simple problems?
It is because more complicated problems have a large amount of weight variables, allowing for minimisation to occur from a high-dimensional space with lots of different vectors; meanwhile, the neutral net may only reach a local minimum with more simple problems as it has fewer variables to work with.
To train the neural net of ChatGPT it is presented with batches of data examples, which the weights of the networks are then altered to reduce the error which the network makes surrounding those errors. This method of back propagating is tedious as there are millions of weights to deal with however the weights of the network will typically stay the same through these processes. True or False?
FALSE
How many parameters can be set for ChatGBT?
175 billion
How many words of are used to train the neural networks within ChatGTP
1 Trillion Words
What are some potential applications for ChatGPT?
ChatGPT has a wide range of potential applications, including improving customer service through chatbots, generating natural language summaries of long documents, and even aiding in scientific research by helping to analyze and synthesize complex data sets.
How are words represented within ChatGPT?
As numbers
How do transformers work?
Transformers formulate and process different sequences of the input data; they will then administer the notion of ‘attention’ to pay more attention to specific sections of the sequence to consolidate information necessary for generating output.
You can think of a neural net as computing a mathematical function, that depends on its inputs, and also its weights. True or false
True
The basic process of training neural nets can be done by…
showing/introducing a variety of examples of images and objects and then have the network learn to recognise and distinguish between them.
Which concept can we identify from Jack’s story below? Jack says he can’t quit gambling because he has invested a lot of money!
The sunk cost fallacy.
summarise the process of machine learning to train neural nets
Machine learning is the process of training by giving many examples which allows the trained neural network to generalise from the given examples.
What is a ‘large language model’ (LLM) and what is its role in ChapGPT?
A ‘large language model’ (LLM) is a key component of ChapGPT that has been built to estimate the probabilities of sequences of words occurring.
What is the main goal of ChatGPT?
The main goal of ChatGPT is to produce a “reasonable continuation” of a given text, where “reasonable” means what one might expect someone to write after seeing what people have written on billions of webpages.
Will ChatGPT always be correct?
No, although ChatGPT uses machine learning, there are definitely limitations on the accuracy of answers it provides.
How much data is needed to train a neural net for a specific task?
Neural nets typically need to see a lot of examples to train well, and it’s important to show variations of the same example for better training. Repetition of the same example is useful and variations in how the data is presented don’t have to be sophisticated to be helpful.
what is semantic grammar?
construction of language from words
What is the key advantage of using a pre-trained langauge model like ChatGPT?
Efficiency
True or false? Neural nets are based on numbers.
TRUE
Neural Nets
A neural network is a method in artificial intelligence that teaches computers to process data in a way that is inspired by the human brain. It is a type of machine learning process, called deep learning, that uses interconnected nodes or neurons in a layered structure that resembles the human brain.
What are the main challenges around “neural nets” and machine learning in general?
The main challenges center around acquiring or preparing the necessary training data.
What was the big breakthrough in “deep learning” that occurred around 2011?
The breakthrough in “deep learning” was that when using neural nets it is easier to solve complex problems than simple problems. The exact reason is unknown; however, it appears that this is because more complex problems have more “weight variables” and, therefore higher dimensional space with more directions, which can lead to the global minimum. Simpler problems with fewer variables have fewer directions to the minimum and can be caught in local minimum.
Does ChatGPT select the most ‘probable’ word everytime when replying to a prompt?
No, ChatGPT takes into account the probability for the next several pairs of words which creates a sequence of which words should come next.
What is an attention block?
A component of a neural network that allows a model to selectively focus on certain parts of the input text to assist it in generating the most probably suitable output text.
Fill in the blank: The notion that language has distinct grammatical rules for how words are put together (which ChatGPT learns to follow) is called ___.
Syntax.
To what extent do neural nets help explain how the brain works?
A hundred billion neuronsÊmake up the human body, and each one is able to emit an electrical pulse as fast as a thousand times per second. Each neuron allowsÊit to transmit electrical messages to maybe hundreds of other neurons. The neurons are interconnected in a complex network. The electrical pulses that a particular neuronÊreceives from other neurons determine whether it will create a pulse at a particular time, with various connections contributing to different “weights.”
What is back propagating (in the context of neural networks) and why is it important?
Back propagating is a technique whereby the strength of the connections between the neurons within a neural net is updated based on the feedback received from the training data, thus adjusting the network’s performance. Back propagating calculates the error between the expected output and the network’s output and propagates that error back through the network, adjusting the weight of the connections between neurons as it goes in order to minimise this error. This is important as it allows for the neural net to learn from its mistakes and optimise its performance on tasks over time. This in turn allows it to generate accurate and contextually relevant responses.
Why can it take ChatGPT a long time to generate long texts sometimes?
Because ChatGPT’s neural net consists of millions of neurons, which means billions of connection and weights, and every time a new token is being generated, ChatGPT has to calculate every single one of these weights.
what’s the weight of ChatGPT-3 network?
175 billion connections, hence 175 billion weights.
What is the ideal temperature for essay generation?
0.8
Match the definition to the concept: A- Fine-tuning, B: Pre-training. 1-The process of adapting the pre-trained model for specific tasks. 2- The process of training a language model on a large amount of text data.
A-1, B-2.
How does the training process for large language models like ChatGPT work, and what kind of data is used to train them?
Large language models like ChatGPT are trained using a technique called unsupervised learning on massive amounts of text data. During training, the model tries to predict the next word or sequence of words in a piece of text, given the words that come before it, which helps it learn patterns in language and generate coherent text.
True or False: The ChatGPT reply produces a list of ranked words that might follow the previous word, together with probabilities” after reviewing the readings. ChatGPT then always selects the word with the highest probability. “
FALSE
how does image recognition works?
Neural nets
True or False: a new neural net can generate more trainings for itself or cooperate with other trained nets directly.
True.
According to Wolfram,, what are some key limitations or areas for improvement of ChatGPT
Key limitations of ChatGPT include; A need for large amounts of computing power and a potential for biased or misleading outputs.
What are neural nets?
Simple ideas about how brains seem to work. In Ai, they are computing systems or code that are inspired by the “natural” or biological neural networks (animal/human brains)
What does a temperature of 0.8 mean for ChatGPT
Temperature (0.8) allows ChatGPT to produce a more human like piece of writing rather than choosing the next word on the ‘next best probability’ each time. Temperature (0.8) opens a possibility for creativity.
Explain the concept of “meaning space”.
The concept of meaning space is linked to the process of word embedding. Word embedding can be conceptualised as “trying to lay out words in a kind of ‘meaning space’”. This means that words that are close in meaning are physically close in the embedding.
True or False: ChatGPT isn’t like the typical computational system and uses a “feedforward” mechanism, meaning that the input given to ChatGPT tends to loop back and is reprocessed by the same neurons.
False; a “feedforward” mechanism means that all computational element or “neuron” in the neural network is used only once.
True or false: The only thing that is explicitly engineered” in ChatGPT is the overall architecture - everything else is “learned” from the training data. “
TRUE
True or false, ChatGPT includes in its algorithm an element of randomness
TRUE
True of False: Cognitive Biases indefinitely affect our decision-making.
True. Cognitive Biases will impact the way you think, but with careful considerations and clear steps, you will be able to make informed decisions.
The optimal temperature for essay generation can vary depending on the specific use case and desired outcome. Higher temperatures, typically between 0.7 and 1.0, can lead to more creative and varied responses, while lower temperatures, typically between 0.1 and 0.5, tend to produce…?
Lower temperatures tend to produce more conservative and predictable outputs.
Why ChatGPT can communicate with people very well.
The reason why ChatGPT can communicate well is because of its large pre-training data and its ability to understand the context well.
Define COMPUTATIONAL IRREDUCTIBILITY
Some computations have shortcuts that allow them to be completed more quickly. Computations that cannot be done more quickly by using shortcuts are computationally irreductible. The only way to determine the answer to a computationally irreducible question is to perform each step of the computation.
What is ChatGPT?
ChatGPT is a large language model developed by OpenAI, trained using deep learning algorithms to generate human-like responses to natural language inputs.
True or False? A neural network is a computer program that is designed to recognise patterns or make predictions based on data, loosely modelled after the human brain.
TRUE
True or False: When neural nets learn to distinguish objects, the trained network “generalizes” from the particular examples shown
TRUE
When ChatGPT is producing something like an essay it is essentially asking the question “given the text so far, what should the next word be?” over and over. In order to create a more dynamic, natural-sounding essay, when choosing the next word, it: A. always picks the highest-ranked word (i.e. the one to which the highest “probability” was assigned) B. always picks the word with a middle rank C. generally picks the highest-ranked words and occasionally the lower-ranked words at random D. selects a random word out of the top 1500 words each time
The answer is C. ChatGPT doesn’t always just pick the highest-ranked word because this will result in a flat-sounding essay that may be repetitive, so at random it sometimes picks lower-ranked words.
For essay generation, what ‘temperature’ in ChatGPT is the best for the selection of words?
0.8
What is a neural net and how is it related to the human brain?
A neural net is a computational model that is inspired by the structure and function of the human brain. It consists of interconnected neurons, and the output of each neuron is determined by the input it receives from other neurons, weighted by specific values. This is similar to how neurons in the human brain communicate with each other through electrical signals.
Why does ChatGPT sometimes pick lower-ranked words instead of always picking the highest-ranked word?
ChatGPT sometimes picks lower-ranked words instead of always picking the highest-ranked word because doing so can lead to a “more interesting” essay with greater creativity.
According to S.L. Wolfram, how can ChatGTP “..get as far as it does with language.”
There is a fundamental logic to human language, through “laws of logic” and “laws of thought”. ChatGTP can follow these laws, patterns, and regularities of human writing and speech and replicate it.
How do neural networks learn to distinguish things?
By showing them multiple examples. When “teaching” a neural network to distinguish, we don’t have to write programs about them and mention features eg. nose, eyes, mouth etc. Rather, multiple pictures can be shown instead. The network then generalises these examples.
Define the framing effect
the way in which information is presented can influence our judgments and decisions
What is a neural net?
A neural net is a set of computational “neurons” that are capable of recognising and analysing patterns in large amounts of data. These “neurons” are layered and are taught how to analyse and recognise patterns that we deem as important by feeding it information and having it churn out responses. These responses are then fine-tuned by introducing weights at each level that make the output more and more acceptable to us.
How is training data stored in the neural network to be used?
Data used to train ChatGPT and other neural network is not directly stored anywhere once used. The node eights are adjusted every time data is fed through, influencing how the network makes decisions. It is ‘some kind of distributed encoding of the aggregate structure of all that text’.
How does ChatGPT work?
ChatGPT works by using a combination of deep learning techniques, including unsupervised learning, to analyze and understand patterns in vast amounts of text data. It then generates responses based on the input it receives, using its understanding of natural language to create human-like responses.
How is a neural net trained to complete a task?
Neural net training works by showing the neural net lots of examples and variations of the example repetitively, which gradually updates the weight variable to best capture the training example it has been given.
Is it accurate to say that ChatGPT appears to adhere to mathematical principles such as geodesics in its systematic approach to understanding human language?
No, at this stage it is simply unknown whether ChatGPT has “discovered” underlying systems such as geodesics in its decision-making processes relating to human language.
What does the ‘GPT’ in chat GPT stand for
Generative Pre-trained Transformer
How does a GPT generate text?
GPT generates text by predicting the probability of the next word in a sequence based on the context of the preceding words
True or false: The intricate connections of 175 billions neural net is determined through the result of training based on text written by humans (e.g., text found on the web, in books, etc).
TRUE
What is a neural net?
A neural net is a way of organising information in a way that is similar to the processes of the human brain.
When ChatGPT is producing something like an essay it is essentially asking the question “given the text so far, what should the next word be?” over and over. In order to create a more dynamic, natural-sounding essay, when choosing the next word, it: A. always picks the highest-ranked word (i.e. the one to which the highest “probability” was assigned) B. always picks the word with a middle rank C. generally picks the highest ranked words and occasionally the lower-ranked words at random D. selects a random word out of the top 1500 words each time
C. It doesn’t always just pick the highest-ranked word because this will result in a flat-sounding essay that may be repetitive, so at random it sometimes picks lower-ranked words.
One of the exciting things about CHATGPT is the fact that it could take us to new places in research with what happens as a human when we are summarising what is happening in a situation. For example. When we see a picture of a dog, we know its a dog but we don’t know why in science yet how we get to that conclusion in mathematical neural terms. ChatGPT could be part of the key to opening this new door of research. True or False?
TRUE
True or false? ChatGPT is consisting of a neural network. The current version of the neural network includes 175 billion weights.
TRUE
How many words of text was ChatGPT trained on?
A few 100 billion
What is computational irreducibility?
Computational irreducibility refers to the idea that there are some computations that cannot be simplified or shortened in any meaningful way, and must be carried out step-by-step to determine their outcome.
Fill in the blank: ChatGPT uses a “______________” when generating essays which determines how often lower-ranked words are used. a) randomness parameter b) temperature parameter c) net model d) language model
b) temperature parameter
Definition - What is a parse tree?
A parse tree is a graphical representation of the syntactic structure of a sentence or a piece of code in a programming language. It is also known as a derivation tree or a syntax tree.
What does the so-called ‘temperature’ parameter of ChatGPT determine?
ChatGPT sometimes uses lower-ranked words when generating essays to make them more interesting and so this ‘temperature’ parameter determines how often lower-ranked words will be used, and specifically for essays, this is found to be 0.8.
embedding
Embeddings are the translation of concepts into numbers so that computers can more easily establish relationships between different concepts.
When is the knowledge cutoff for ChatGPT?
September 2021
How does CHATGPT pick its next word?
ChatGPT picks its next words based on a probability distribution generated by analyzing vast amounts of text data. It uses a neural network architecture that is trained on a large corpus of text data to predict the likelihood of each possible word following a given sequence of words.
How the machine learning of the brain trains to do particular tasks?
“Trained from examples” through neural nets.
The type of neural network architecture ChatGPT uses is: A. Feedforward Neural Network B. Transformer Neural Network C. Convolutional Neural Network D. Recurrent Neural Network
B. Transformer Neural Network
True or False, “ChatGPT is always fundamentally trying to do is to produce a reasonable continuation” of whatever text it’s got so far, where by “reasonable” we mean “what one might expect someone to write after seeing what people have written on billions of webpages””
TRUE
What are two “laws of language” that ChatGPT didn’t need to be explicitly trained on but was able to implicitly discover and pick up on after having gone through sufficient training material?
Syntax and logic.
What does ChatGBT look for?
Things that match in meaning.
How do theneural networks involved in ChatGPT work?
Neural networks are designed to mimic the processes of the human brain. For ChatGPT, this means responding to prompts by breaking data down into multiple small steps (simulating neurons), each passing its output to others to generate more accurate, specific responses. The various neural connections in the network also vary in their significance, with each output contributing a different weight to those proceeding. These neural connections create a network where information is processed through multiple layers of “neurons,” which allow for responses to be fine-tuned to respond to specific prompts and data.
True or False? The availability heuristic is a cognitive bias that occurs when we overestimate the likelihood of rare events occurring, based on their vividness or the ease with which they come to mind.
FALSE
_______ is the idea that there are certain systems where predicting the behaviour of the system cannot be reduced to a simpler/more efficient algorithm.
Computational irreducibility
True or false. ChatGPT has the potential to revolutionise the communication and information processing and could have significant implications for society in the future.
TRUE
