Analyze text with the Language service

Question 1

Analyzing text

Answer 1

is a process where you evaluate different aspects of a document or phrase, in order to gain insights into the content of that text.

Question 2

Text Analytics Techniques

Answer 2

Text analytics is a process where an artificial intelligence (AI) algorithm, running on a computer, evaluates these same attributes in text, to determine specific insights.

Question 3

techniques that can be used to build software to analyze text

Answer 3

1. Statistical analysis of terms used in the text. For example, removing common “stop words” (words like “the” or “a”, which reveal little semantic information about the text), and performing frequency analysis of the remaining words (counting how often each word appears) can provide clues about the main subject of the text.
2. Extending frequency analysis to multi-term phrases, commonly known as N-grams (a two-word phrase is a bi-gram, a three-word phrase is a tri-gram, and so on).
3. Applying stemming or lemmatization algorithms to normalize words before counting them - for example, so that words like “power”, “powered”, and “powerful” are interpreted as being the same word.
4. Applying linguistic structure rules to analyze sentences - for example, breaking down sentences into tree-like structures such as a noun phrase, which itself contains nouns, verbs, adjectives, and so on.
5. Encoding words or terms as numeric features that can be used to train a machine learning model. For example, to classify a text document based on the terms it contains. This technique is often used to perform sentiment analysis, in which a document is classified as positive or negative.
6. Creating vectorized models that capture semantic relationships between words by assigning them to locations in n-dimensional space. This modeling technique might, for example, assign values to the words “flower” and “plant” that locate them close to one another, while “skateboard” might be given a value that positions it much further away.

Question 4

In Microsoft Azure, the Language cognitive service can help simplify application development by using pre-trained models that can:

Answer 4

1. Determine the language of a document or text (for example, French or English).
2. Perform sentiment analysis on text to determine a positive or negative sentiment.
3. Extract key phrases from text that might indicate its main talking points.
4. Identify and categorize entities in the text. Entities can be people, places, organizations, or even everyday items such as dates, times, quantities, and so on.

Question 5

The Language service

Answer 5

is a part of the Azure Cognitive Services offerings that can perform advanced natural language processing over raw text.

Question 6

Azure resources for the Language service

Answer 6

1. A Language resource - choose this resource type if you only plan to use natural language processing services, or if you want to manage access and billing for the resource separately from other services.
2. A Cognitive Services resource - choose this resource type if you plan to use the Language service in combination with other cognitive services, and you want to manage access and billing for these services together.

Question 7

Language detection

Answer 7

Use the language detection capability of the Language service to identify the language in which text is written. You can submit multiple documents at a time for analysis. For each document submitted to it, the service will detect:

1. The language name (for example “English”).
2. The ISO 6391 language code (for example, “en”).
3. A score indicating a level of confidence in language detection.

Question 8

predominant language

Answer 8

if the text containing a mix of English and French. The language detection service will focus on the predominant language in the text. The service uses an algorithm to determine the predominant language, such as length of phrases or total amount of text for the language compared to other languages in the text.

Question 9

Ambiguous or mixed language content

Answer 9

There may be text that is ambiguous in nature, or that has mixed language content. These situations can present a challenge to the service. An ambiguous content example would be a case where the document contains limited text, or only punctuation. For example, using the service to analyze the text “:-)”, results in a value of unknown for the language name and the language identifier, and a score of NaN (which is used to indicate not a number).