Python - Lexical Structure

Question 1

What is Lexical Structure in a programming language?

Answer 1

The lexical structure of a programming language is the set of basic rules that govern how you write programs in that language.

Lexical Structure is the lowest-level syntax of the language.

Question 2

Research: Syntax in programming languages in general

Answer 2

Wikipedia: In computer science, the syntax of a computer language is the rules that define the combinations of symbols that are considered to be correctly structured statements or expressions in that language.

https://en.wikipedia.org/wiki/Syntax_(programming_languages)

Question 3

Why is Lexical structure important?

Answer 3

Lexical Structure are rules designed to organize the Python Language Code, written within a source code file, which is essential for writing readable and executable code

Question 4

List and Briefly Describe: Lexical Structures in the Python Programming Language

hint: 11 terms are listed.

Answer 4

Lines: the basic structure of python code that helps separate statements from each other.

Indentations: the basic structure of python code that helps connect instructions when they span multiple lines.

Comments: writing that is not considered code and usually used for documentation purposes.

Character Sets: Python uses a set of characters called Unicode (UTF-8). Another character set can be used to write Python code when using the encoding declaration (# coding: iso-8859-1) in a source file.

Tokens: Python breaks each line of Python code into a sequence of lexical components known as tokens.

Identifier Tokens: An identifier is a name used to specify a variable, function, class, module, or other object

Keyword Tokens: Python has 35 keywords, or identifiers that it reserves for special syntactic uses and cannot use keywords as regular identifiers

Operators Tokens: non-alphanumeric characters and character combinations used to perform actions on other tokens.

Delimiters Tokens: characters and combinations of characters used to group and organize code.

Literals Tokens: A literal is the direct denotation in a program of a data value (a number, string, or container).

Statements:

Question 5

Research: Tokens for programming languages in general

Answer 5

Wikipedia: Lexical tokenization is conversion of a text into (semantically or syntactically) meaningful lexical tokens belonging to categories defined by a “lexer” program… In the case of a programming language, the categories include identifiers, operators, grouping symbols and data types.

https://en.wikipedia.org/wiki/Lexical_analysis

Question 6

Describe: Tokens

Answer 6

Python breaks each logical line into a sequence of elementary lexical components known as tokens. Each token corresponds to a substring of the logical line. The normal token types are identifiers, keywords, operators, delimiters, and literals

Question 7

Describe: The use of Lines in Lexical Structure of the Python Programming Language.

Answer 7

Python code is broken up (i.e. processed) by a Lexical Analyzer that starts by searching across physical lines of the text file to find logical lines of code.

In practice, we write a code statement as a single Logical Line that can be broken up and can span across one or more Physical Lines.

The Python lexical analyzer breaks logical lines of source code into tokens before feeding it to the parser.

References

The Python Language Reference 1

Python in a Nutshell 2

1 The Python Language Reference https://docs.python.org/3.9//reference/lexical_analysis.html
2 Python in a Nutshell, 4th Edition (see Lexical Struture section in Chapter 3)

Question 8

List and Describe: types of line joining

Answer 8

Line Joining in Python is a way for logical lines of code to span across multiple physical lines. There are two types of line joining: explicit line joining & implicit line joining.

Explicit Line Joining: when a physical line ends in a backslash that is not part of a string literal or comment, it is joined with the following physical line forming a single logical line.

Implicit Line Joining: Expressions in parentheses, square brackets or curly braces can be split over more than one physical line without using backslashes.

Question 9

Research: Block Structure for programming languages in general.
hint: search for ‘Programming Language Block Structure’

Answer 9

Wikipedia: In computer programming, a block or code block or block of code is a lexical structure of source code which is grouped together… Blocks have two functions: to group statements so that they can be treated as one statement, and to define scopes for names to distinguish them from the same name used elsewhere.

https://en.wikipedia.org/wiki/Block_(programming)

Question 10

Describe: Indentations

Answer 10

Python uses indentation to express the block structure of a program.

Question 11

Research: Unicode

Question 12

Describe: the use of Character Sets in the Python Programming Language

Answer 12

A Python source file can use any Unicode character, encoded by default as UTF-8. You may choose to tell Python that a certain source file is written in a different encoding.

Question 13

Describe: Identifiers

Answer 13

An identifier is a name used to specify a variable, function, class, module, or other object

Question 14

What are the rules for creating an identifier?

Answer 14

An identifier starts with a letter (that is, any character that Unicode classifies as a letter) or an underscore (_), followed by zero or more letters, underscores, digits, or other characters that Unicode classifies as letters, digits, or combining marks. Case is significant: lowercase and uppercase letters are distinct. Punctuation characters such as @, $, and ! are not allowed in identifiers.

Question 15

Describe: Keywords

Answer 15

Python has 35 keywords, or identifiers that it reserves for special syntactic uses.

Question 16

Name the 35 keywords, or ‘reserved words’ in the Python Programming Language

Answer 16

and
break
elif
from
is
pass
with
as
class
else
global
lambda
raise
yield
assert
continue
except
if
nonlocal
return
False
async
def
finally
import
not
try
None
await
del
for
in
or
while
True

Question 17

Describe: Operators

Question 18

Describe: Literals

Answer 18

A literal is the direct denotation in a program of a data value (a number, string, or container). We refer to these expressions as literals throughout this book, as they describe literal (i.e., not requiring additional evaluation) values in the source cod

Question 19

What is a ‘statement’ in the Python Programming Language?

Answer 19

Any expression can stand on its own as a simple statement

Question 20

What are the two types of statements in the Python Programming Language?

Answer 20

You can look at a Python source file as a sequence of simple and compound statements.

Question 21

Describe: Simple Statements in the Python Programming Language.

Answer 21

A simple statement is one that contains no other statements. A simple statement lies entirely within a logical line. Any expression can stand on its own as a simple statement.

Question 22

Describe: Compound Statements in the Python Programming Language.

Answer 22

A compound statement contains one or more other statements and controls their execution.

A compound statement has one or more clauses, aligned at the same indentation. Each clause has a header starting with a keyword and ending with a colon (:), followed by a body, which is a sequence of one or more statements.