Lecture 6 - Dialog Systems and Chatbots - Conversational Agents

Question 1

What are conversational agents? give two types of them as well

Answer 1

Conversational agents communicate with users in natural language (text, speech)

• chatbots can mimic conversations with people
• dialog systems are designed for a particular task (Siri, Alexa)

Question 2

True or False. There are two types of chatbots (that we have learned of at least): rule-based chatbots and IR-based chatbots

Answer 2

True. Eliza and Parry are examples of rule-based chatbots

Question 3

What do rule-based chatbots use? What can they not do?

Answer 3

They use a set of rules

but, cannot answer question outside the defined rules, and cannot learn from interactions

Question 4

How do rule-based chatbots work?

Answer 4

Rule-based chatbots depend on keywords in the questions they receive to understand the queries. Then, the chatbot will research predefined answers to provide a relevant response.

All those keywords are associated with a transformation that is ranked, and the chatbots are trying to reply with the highest rank

Question 5

True or False. In a situation where a rule-based chatbot cannot find a matching keyword to reply, they do not say anything.

Answer 5

False. When no keyword matches, it chooses a noncommittal response

“go on”
“I see”
“interesting”

Question 6

What are the exact steps of a rule-based chatbot?

Answer 6

Find the word w in sentence that has the highest keyword rank
If w exists,
* check each rule for w in ranked order, choose the first one that matches sentence; response <= apply transform

Else,

• response <= apply NONE transform (noncommital response) or
• response <= grab an action in memory queue

Question 7

What are the differences between rule-based chatbots Eliza and Parry?

Answer 7

Parry uses the same pattern-response structure as Eliza, but it has a much richer control structure, language understanding capabilities, and a mental model (affective variables)

Question 8

Where do IR (information retrieval) chatbots get their data from?

Answer 8

IR-based chatbots mine conversations of human chats or human-machine chats

They use corpora like microblogs (Twitter) or movie dialogs

Question 9

What are the two IR-based chatbot architectures?

Answer 9

1. Return the response to the most similar turn
   * take user’s turn (what the user said) and find a (td-idf) similar turn t in the corpus C
2. Return the most similar turn

Question 10

True or False. IR-based models can use other features like previous turns or information about the user.

Answer 10

True

or non-dialog texts like (information from Wikipedia)

Question 11

Name some pros and cons of chatbots.

Answer 11

PRO:
fun
applications to counseling
good for narrow, scriptable applications

CON:
they don’t really understand
rule-based chatbots are expensive and don’t generalize well
IR-based chatbots can only mirror training data

Question 12

What are frame-based dialog agents based on?

Answer 12

Frame-based dialog agents are based on “domain ontology” - a knowledge structure representing users intention

(flight booking for example)

Question 13

What are frame-based dialog agents composed of?

Answer 13

Frame-based dialog agents are composed of one or more frames, that each has a collection of slots, and each slot has a value

each slot is associated with a type and a question

e.g.
SLOT-TYPE-QUESTION
departure-city-What city are you leaving from?
destination-city-Where do you want to go?
departure date-date-What day would you like to leave?

etc.

Question 14

True or False. In frames, the types of slots can have a hierarchical structure.

Answer 14

True.

DATE: year, month, day etc.

Question 15

In frame-based dialog agents, there is a finite control structure. What does that mean?

Answer 15

The system completely controls the conversation with the user.
It asks the user a series of questions.
Ignores (or misinterprets) anything the user says that is not a direct answer to the system’s questions

btw: systems that control conversations like this are called single initiative

Question 16

Pros and cons of single initiatives in frame-based dialog agents

Answer 16

PRO: easy to build, OK for very simple tasks, both user and system know what the other one can say next

CON: too limited

Question 17

One solution for the limited single initiatives of frame-based dialog agents is mixed initiatives. What do those entail?

Answer 17

Instead of taking one question and one answer at a time, the system can process multiple answers at a time

The system asks questions to the user, and fills any slots that the user specifies => when the frame is filled (all the slots are filled) do database query

e.g., single initiative: When? Tomorrow. Where? Home
mixed initiative: When? Tomorrow at home
fills both slots when and where

SIRI uses that btw

Question 18

How can you use supervised machine learning to train a frame-based dialog agent?

Answer 18

Train on labeled data such as
“I want to fly to San Francisco on Monday afternoon”
DESTINATION: San Francisco
DEPARTURE DATE: Tuesday

Build a classifier to map from sentence to slots that the user want to fill in (logistic regression, neural networks)

Question 19

What are the two metrics that we can use to evaluate the model performance for frame-based dialog agents?

Answer 19

1. slot-error rate

2. end-to-end evaluation