What is Data Science? Flashcards
IBM Data Science Professional Certificate (Course 1/10)
What is data science?
The translation of data into a story, and then using these stories to generate insights. It is with these insights are you then able to develop strategies for companies, for example.
How does digital transformation affect business operations?
It affects them by updating existing processes and operations and creating new ones to harness the benefits of new technologies (e.g. harnessing the benefits of Big Data).
optical tracking
An example of how Big Data can trigger a digital transformation, not just within an organisation, but within an entire industry
Manchester City has embraced the use of Big Data to improve their game.
They have a team of data analysts who use millions of stats about players’ performance and the upcoming opposition to help the club’s chances of winning.
One of the tools they use is optical tracking, which can be used to pinpoint the position of players on the pitch 25 times a second, in relation to the ball, opposition, and teammates. This data, along with other ball-related data such as passes, shots, and turnovers, is analysed to gain insights into the team’s performance.
These insights can then be used to inform the team’s strategy in future games. For example, they might adjust their formation, change their passing strategy, or alter player positions based on the data.
It’s a great example of how Big Data can transform not just a single team, but the entire sport of football.
What is cloud computing?
The delivery of on-demand computing resources such as:
* Networks
* Servers
* Storage
* Applications
* Services
* Data centres
over the Internet on a pay-for-use basis.
What are some of the benefits of cloud computing?
- Users do not need to purchase and install the software on their local systems, they can just use the online version of the software and pay a monthly subscription.
- This makes everything more cost-effective as well as ensuring you always have access to the most up-to-date version of the software. Think of Microsoft 365, for example.
- Other benefits include saving the user some local storage space as well as encouraging collaboration among colleagues/project teams as the software would be hosted online.
What is cloud computing composed of?
- 5 characteristics
- 3 service models
- 3 deployment models
Only Brave Rabbits Run Marathons
What are the five characteristics of cloud computing?
- On-demand self-service
- this means getting access to cloud resources such as power, storage and network without requiring human interaction with each service provider
- Broad network access
- this means that cloud computing resources can be via the network through standard mechanisms and platforms such as mobile phones, tablets, laptops, and workstations.
- Resource pooling
- this is what gives cloud providers economies of scale, which they pass on to their users, making cloud cost-efficient
- using a multi-tenant model, computing resources are pooled to serve multiple customers, and cloud resources are dynamically assigned and reassigned according to demand without customers needing to know the physical location of these resources
- Rapid elasticity
- this implies that you can access more resources when you need them and scale things back when you don’t, because resources are elastically provisioned and released
- Measured service
- this implies that you only pay for what you use as you go; if you’re not using those resources, you’re not paying
What is cloud computing really about?
It is about using technology “as a service”, leveraging remote systems on-demand over the open Internet, scaling up and scaling back, and only paying for what you use.
What do cloud deployment models indicate?
They indicate where the infrastructure resides, who owns and manages it, and how cloud resources and services are made available to users.
What are the three types of cloud deployment models?
- Public
- this is when you leverage cloud services over the open internet on hardware owned by the cloud provider, but its usage is shared by other companies
- Private
- this means that the cloud infrastructure is provisioned for exclusive use by a single organisation
- it could run on-premises or it could be owned, managed, and operated by a service provider
- Hybrid
- this is when you use a mix of both the public and private deployment models.
What are the three cloud service models based on?
The three layers in a computing stack: infrastructure, platform, and application.
What are the three cloud service models?
- Infrastructure as a Service (IaaS)
- In this model, you can access the infrastructure and physical computing resources such as servers, networking, storage, and data centre space without the need to manage or operate them
- Platform as a Service (PaaS)
- you can access the platform that comprises the hardware and software tools that are usually needed to develop and deploy applications to users over the Internet.
- Software as a Service (SaaS)
- this is a software licensing and delivery model in which software and applications are centrally hosted and licensed on a subscription basis. It is sometimes referred to as “on-demand software.”
Why is the cloud such a positive for data science?
It allows a data scientist to bypass the physical limitations of their computer and the system they’re using.
What is Big Data?
Big Data refers to the dynamic, large and disparate volumes of data being created by people, tools, and machines.
What does Big Data need in order to be effective?
It requires new, innovative, and scalable technology to collect, host, and analytically process the vast amount of data gathered.
What does Big Data aim to do?
It aims to derive real-time business insights that relate to consumers, risk, profit, performance, productivity management, and enhanced shareholder value.
What are the V’s of Big Data?
- Velocity
- This is the speed at which data is accumulated
- Volume
- This the the scale of the data or the increase in the amount of data stored
- Variety
- This is the diversity of the data
- Veracity
- This is the quality and origin of data and its conformity to facts and accuracy
- Value
- This refers to our need and ability to turn data into value
What are the drivers of Big Data Volume?
- The increase in data sources
- Higher resolution sensors
- Scalable infrastructure
What is the difference between structured and unstructured data?
- Structured data fits neatly into rows and columns in relational databases.
- For example, employee details at a company.
- These employee details would include things like job, employee number, age etc. which would be criteria that everyone at the company would have, with all of it being the same data type.
- Unstructured data is data that is not organised in a predefined way.
- For example, this could be tweets, blog posts and videos. - Structured data fits neatly into rows and columns in relational databases.
What does variety reflect?
That data comes from different sources.
What are the drivers of variety?
- Mobile technologies
- Social media
- Wearable technologies
- Geo technologies
- Video
- Many more
CCIA
What are the attributes of veracity?
- Consistency
- Completeness
- Integrity
- Ambiguity
CN
What are the drivers of veracity?
- Cost
- Need for traceability
What is the main reason people take time to understand Big Data?
In order to derive value from it.
airport
An example of velocity in action
- Imagine a bustling airport during peak travel hours.
- Hundreds of flights are landing and taking off every minute.
- Each flight generates data: passenger lists, flight paths, fuel consumption, baggage handling, and more.
- The velocity of this data is immense, accumulating rapidly as planes taxi, ascend, and descend.
- Airlines must process this real-time data to optimise flight schedules, ensure safety, and enhance passenger experiences.
amazon
An example of volume in action
- Consider an e-commerce giant like Amazon.
- Millions of customers browse products, add items to their carts, and make purchases simultaneously.
- The sheer volume of data generated—product listings, customer profiles, transaction histories, reviews, and shipping details—is staggering.
- Amazon’s servers handle petabytes of data daily.
- To manage this volume, they employ distributed databases, data lakes, and scalable cloud infrastructure.
An example of variety.
- Picture a social media platform like Instagram.
- Users share diverse content: photos, videos, stories, captions, hashtags, and geotags.
- Additionally, Instagram collects metadata (likes, comments, timestamps).
- This mix of structured (metadata) and unstructured (visual content) data creates variety.
- Instagram’s challenge lies in organising and analysing this eclectic data to personalise feeds, recommend content, and detect trends.
An example of veracity
- Think about a healthcare system that collects patient data from various sources: electronic health records, wearable devices, diagnostic images, and clinical notes.
- However, not all data is equally reliable.
- Some entries may contain errors, missing values, or inconsistencies.
- Veracity refers to the trustworthiness and accuracy of data.
- Healthcare institutions invest in data validation, quality checks, and anomaly detection to ensure reliable insights for patient care and research.
An example of value
- Think about financial trading.
- Traders analyse stock market data—stock prices, trading volumes, news sentiment, and economic indicators—to make informed decisions.
- The value lies in identifying patterns, predicting market movements, and executing profitable trades.
- However, not all data contributes equally to value.
- Extracting actionable insights requires sophisticated algorithms, machine learning models, and real-time analytics.
What are data scientists today expected to do with Big Data?
They are expected to derive insights from Big Data and cope with the challenges that come with these massive datasets.
What are some challenges that Big Data has presented for data scientists?
The scale of the data means that it is not feasible to use conventional data analysis tools
How are the challenges associated with Big Data overcome?
Alternative tools such as Hadoop and Apache Spark are used.
How does data science differ from traditional subjects like statistics?
- Scope:
- It combines multi-disciplinary fields and computing to interpret data for decision-making.
- Applications:
- Data Science involves data cleaning, integration, visualisation, and statistical analysis of data sets to uncover patterns and trends.
- Decision Making:
- Data science uses scientific methods to discover and understand patterns, performance, and trends, often comparing numerous models to produce the best outcome.
- Meanwhile, statistics focuses on using mathematical analysis with quantified models to represent a given data set.
What do Big Data Processing Tools help you do?
These processing technologies provide ways to work with structured, semi-structured, and unstructured data so that value can be derived from big data
What are the most common open-source Big Data computing tools?
- Apache Hadoop
- Apache Hive
- Apache Spark
What is Apache Hadoop?
Apache Hadoop is a collection of tools that provide distributed storage and processing of big data
What is Apache Hive?
Apache Hive is a data warehouse for data query and analysis
What is Apache Spark?
Apache Spark is a distributed analytics framework for complex, real-time data analytics
What does a Hadoop framework allow?
It allows distributed storage and processing of large datasets across clusters of computers.
What happens in a Hadoop distributed system?
In a Hadoop distributed system:
- a node is a single computer, and a collection of nodes forms a cluster
- Hadoop can scale up from a single node, to any number of nodes each providing local storage and computation
- Hadoop provides a reliable, scalable, and cost-effective solution for storing data with no format requirementsIn a Hadoop distributed system:
What are the benefits that come with using Hadoop?
- Better real-time data-driven decisions
- incorporates emerging data formats not typically used in data warehouses
- Improved data access and analysis
- provides real-time self-service access to stakeholders
- Data offload and consolidation
- optimises and streamlines costs by consolidating data, including cold data, across the organisation
What is one of the four main components of Hadoop?
Hadoop Distributed File System (HDFS).
HDFS is a storage system for big data that runs on multiple commodity hardware connected through a network.Hadoop Distributed File System (HDFS).
What does HDFS do?
HDFS:
- provides scalable and reliable big data storage by partitioning files over multiple nodes
- splits files over multiple computers, allowing parallel access to them
- replicates file blocks on different nodes to prevent data loss
How is HDFS different from other distributed file systems?
HDFS is highly fault-tolerant and is designed to be deployed on low-cost hardware. HDFS provides high throughput access to application data and is suitable for applications that have large data sets.
An example of how HDFS is different from other distributed file systems
Consider an example where we have a file that contains the phone numbers of everyone in South Africa.
If we were to store this file on a single machine, there would be several challenges:
1. Storage: The file could be larger than the storage capacity of a single machine.
2. Processing Speed: Processing this file (e.g., searching for a specific number) could take a long time because only one machine’s resources (CPU, memory) are being used.
3. Fault Tolerance: If the machine fails, we lose access to the file.
Now, let’s see how HDFS addresses these challenges:
1. Distributed Storage: In HDFS, data is split into blocks (default size is 128MB in Hadoop 2.x), and these blocks are distributed across multiple nodes in the cluster. So, our phone directory file would be split into many blocks, and these blocks would be stored on different machines. This allows HDFS to store a file that is larger than the storage capacity of a single machine.
2. Parallel Processing: Each block of the file is stored on a separate machine, and processing can happen on all machines simultaneously. This means that if we want to search for a phone number, the search operation can be carried out on all machines at the same time, significantly speeding up the process.
3. Fault Tolerance: HDFS is designed to continue operating without a noticeable interruption to the user, even when a machine fails. This is achieved by replicating each block of data across multiple machines. So, if one machine fails, the same block can be found on another machine.
In conclusion, HDFS provides a scalable, fault-tolerant, distributed storage system that works closely with distributed processing frameworks like MapReduce. This makes it an excellent choice for storing and processing big data.
What are some other benefits that come with using HDFS?
- Fast recovery from hardware failures
- HDFS is built to detect faults and automatically recover
- Access to streaming data
- HDFS supports high data throughput rates
- Accommodation of large datasets
- HDFS can scale to hundreds of nodes, or computers, in a single cluster
- Portability
- HDFS is portable across multiple hardware platforms and compatible with a variety of underlying operating systems
What is Hadoop intended for?
Long, sequential scans
What is Apache Hive?
Hive is an open-source data warehouse software for reading, writing, and managing large data set files that are stored directly in either HDFS or other storage systems such as Apache HBase
What is the problem with Hive?
Because Hive is based on Hadoop, queries have very high latency. This makes Hive less appropriate for applications that need a very fast response time.
What is the issue with Hive being read-based?
It makes it unsuitable for transaction processing that typically involves a high percentage of write operations.
What is Hive suited for?
Hive is suited for data warehousing tasks such as ETL, reporting, and data analysis and includes tools that enable easy access to data via SQL.
What does ETL stand for?
ETL stands for Extract, Transform, and Load.
In the context of Apache Spark, what is ETL?
In the context of Apache Spark, ETL is a process that:
1. Extracts data from the source (original database or data source).
2. Transforms the data, which involves changing the structure of the information so it integrates with the target data system and the rest of the data in that system. This transformation can include cleaning (such as mapping NULL to 0 or changing date format consistency), deduplication (identifying and removing duplicate records), and format revision (like character set conversion, unit of measurement conversion, date/time conversion, etc.).
3. Loads the transformed data into a target database.
What is Apache Spark?
Spark is a general-purpose data processing engine designed to extract and process large volumes of data for a wide variety of applications, including Interactive Analytics, Machine Learning, and ETL.
How does Spark work?
It takes advantage of in-memory processing to significantly increase the speed of computations and spilling to disk only when memory is constrained.
What are some key attributes of Spark?
- Spark can run using its standalone clustering technology
- It can run on top of other infrastructures, such as Hadoop
- It can access data in a wide variety of data sources, including HDFS and Hive
- Spark is able to process streaming data fast
- Spark is able to perform complex analytics in real-time
What are the seven steps in the data mining process?
(ESPTSME)
- Establishing data mining goals
- Selecting data
- Preprocessing data
- Transforming data
- Storing data
- Mining data
- Evaluating Mining Results
What are the goals you need to set up as the first step in the data mining exercise?
- Identifying the key questions that need to be answered
- Considering the costs and benefits of the exercise
- Determining, in advance, the expected level of accuracy and usefulness of the results obtained from data mining
What is always instrumental in determining the goals and scope of the data mining exercise, and why?
The cost-benefit trade-off is always important in the data mining exercise.
- The level of accuracy expected from the results influences the costs.
- High levels of accuracy from data mining would cost more and vice versa.
- Furthermore, beyond a certain level of accuracy, you do not gain much from the exercise, given the diminishing returns.
What is the output of a data mining exercise largely dependent on?
The quality of data being used.
What must you do if data are not available for further processing?
In such cases, you must identify other sources of data or even plan new data collection initiatives, including surveys.
What factors have a direct bearing on the cost of data mining exercise?
The size, type, and frequency of collection of data.
Why is preprocessing an important step in the data mining process?
This is because in the preprocessing stage:
- you identify the irrelevant attributes of data and expunge such attributes from further consideration.
- During preprocessing, you also identify the erroneous aspects of the data set and flag them as such.
- Lastly, in the preprocessing stage, you develop a formal method of dealing with missing data and determine whether the data are missing randomly or systematically.
What is the next step after the relevant attributes of data have been retained?
The next step is to determine the appropriate format in which data must be stored.
What is an important consideration in data mining?
To reduce the number of attributes needed to explain the phenomena.
How should the transformed data be stored?
In a format that gives unrestricted and immediate read/write privileges to the data scientist.
Why does the data storage scheme need to facilitate efficiently reading from and writing to the database?
During data mining, new variables are created, which are written back to the original database.
When is data subject to data mining?
After data is appropriately processed, transformed, and stored.
What does data mining cover?
Data analysis methods, including parametric and non-parametric methods, and machine-learning algorithms.
What is a good starting point for data mining and why?
A good starting point for data mining is data visualisation.
Multidimensional views of the data using the advanced graphing capabilities of data mining software are very helpful in developing a preliminary understanding of the trends hidden in the data set.
What do you do after extracting results from data mining?
You perform a formal evaluation of the results.
Data mining and evaluating the results becomes an iterative process such that the analysts use better and improved algorithms to improve the quality of results generated in light of the feedback received from the key stakeholders.
What is big data?
Big Data refers to datasets that are so massive, so quickly built, and so varied that they defy traditional data analysis methods; such as you might perform with a relational database.
What is data mining?
The process of automatically searching and analysing data to discover previously unrevealed patterns
What is machine learning?
A subset of AI that uses computer algorithms to analyse data and make intelligent decisions based on what it has learned without being explicitly programmed.
What are some important characteristics of machine learning algorithms?
- They are trained with large datasets
- They learn from examples
- They do not follow rules-based algorithms
Why is machine learning special?
It enables machines to make machines on their own and make accurate predictions using the provided data
What is deep learning?
It is a specialised subset of machine learning that uses layered neural networks to simulate human decision-making.
What is an important characteristic of deep learning algorithms?
They can label and characterise information and identify patterns.
It is what enables AI systems to continuously learn on the job and improve the quality and accuracy of results by determining whether decisions were correct.
What is a neural network in AI?
It is a collection of small computing units (neurons) that take incoming data and learn to make decisions over time.
Why do deep learning algorithms become more efficient as the data set increases, as opposed to other machine learning algorithms that may plateau as data increases?
Neural networks are often layer-deep.
What is the difference between AI and Data Science?
- Data Science is an interdisciplinary field that employs techniques from fields like statistics, computer science, and information science to create actionable intelligence from data.
- It utilises technologies like machine learning to interpret and analyse data, discover patterns, make predictions, and generate insights.
- Artificial Intelligence is broader and involves the creation of intelligent machines capable of learning and decision-making.
- It denotes the emulation of human cognition in machines designed to mimic human thought and behaviour.
What is Generative AI?
Generative AI is a subset of Artificial Intelligence that focuses on producing new data rather than just analysing existing data.
What does GenAI allow machines to do?
It allows machines to create content such as:
- Images
- Music
- Language
- Code
- And more
How does Generative AI work?
Deep learning models such as Generative adversarial networks (GANs) or Variational auto-encoders (VAEs) form part of the foundation of what allows Generative AI to create new content.
These models create new instances that replicate the underlying features of the original data by learning patterns from enormous volumes of data.
What are some applications of Generative AI?
- Natural Language Processing
- e.g. OpenAI’s GPT-4
- Healthcare
- GenAI can synthesise medical images aiding in the training of medical professionals
- Art and Design
- GenAI can create visually stunning artworks
- Gaming
- Game developers can use GenAI to generate realistic environments and characters
How do data scientists make use of Generative AI?
- By way of synthetic data
- By way of coding automation
- By way of uncovering insights
Why would a data scientist make use of synthetic data?
- Building data models takes a lot of data and sometimes data sets may not have enough data to build a model
- Generative AI makes data augmentation possible
- Data scientists can then use this synthetic data along with real data for model training and testing
Why is coding automation beneficial for a data scientist?
Generative AI can generate the software code needed to construct models allowing the data scientist to focus on higher-level tasks
What are some use cases of deep learning?
- Speech recognition
- Facial recognition
Applications of Machine Learning
- Market basket analysis
- Predictive analytics
- Recommendation engine (e.g. how YouTube recommends you videos)
- In FinTech, fraud detection systems are an application of machine learning
What are decision trees?
A type of machine learning algorithm used for decision-making by creating a tree-like structure of decisions.
What is synthetic data?
Artificially generated data with properties similar to real data, used by data scientists to augment their datasets and improve model training.
What is market basket analysis?
Analysing which goods tend to be bought together is often used for marketing insights.
What is Natural Language Processing (NLP)?
A field of AI that enables machines to understand, generate, and interact with human language, revolutionising content creation and chatbots.
What is Bayesian Analysis?
A statistical technique that uses Bayes’ theorem to update probabilities based on new evidence.
What are Artificial Neural Networks?
Collections of small computing units (neurons) that process data and learn to make decisions over time.
What is Cluster Analysis?
The process of grouping similar data points together based on certain features or attributes.
What is Naive Bayes?
A simple probabilistic classification algorithm based on Bayes’ theorem.
What does regression do?
Regression identifies the strength and amount of the correlation between one or more inputs and an output.
What is the true essence of digital transformation?
Digital transformation is not simply duplicating existing processes in digital form; the in-depth analysis of how the business operates helps organisations discover how to improve their processes and operations and harness the benefits of integrating data science into their workflows.
What can cloud computing also refer to?
Cloud computing can also refer to applications and data that users access on the Internet rather than locally.
How should companies get started in Data Science?
- They should start recording information and start recording data.
- It is very important also that they measure any pre-existing data to try and understand it.
- Companies should also build a team of data scientists.
What should you do after you’ve started capturing data?
Archive your data, never overwrite old data. Data never gets old.
What do all organisations use data science for?
To discover optimum solutions to existing problems.
What is the ultimate purpose of analytics?
To communicate findings to the concerned who might use these insights to formulate policy or strategy.
What should a data scientist do with their findings?
- The data scientist should use the insights to build the narrative to communicate the findings.
- In academia, the final deliverable is in the form of essays and reports. Such deliverables are usually 1,000 to 7,000 words in length.
- In consulting and business, the final deliverable takes on several forms. It can be a small document of fewer than 1,500 words illustrated with tables and plots, or it could be a comprehensive document comprising several hundred pages.
What mistake would result in a poor-quality document where the analytics and narrative struggle to blend?
Embarking on analytics, without due consideration to the final deliverable
What do organisations use data science for?
- Drive business goals
- Improve efficiency
- Make predictions
- Save lives
Why is data science used?
Data science allows organisations to discover optimal solutions and helps in establishing a clear understanding of the problem.
What is the first step in helping an organisation solve its problems using data?
Measurement. The organisation needs to start by measuring their data.
How can someone become a data scientist?
If you’re coming into a data science team, the first skill you need would be to know how to program.
You also need to know:
- algebra
- analytical geometry
- calculus
- basic probability and statistics
- databases
As you go further up the data science field, what do you need to know?
- Computer Science theory
- Statistics
- Probability
The intersection of the three is very important in data science and is what is common amongst higher-end data scientists who possess PhDs.
What is one of the most important skills that a data scientist should possess?
Curiosity (along with a sense of humour).
What motivates someone going into a Data Science career?
They should enjoy:
- working with data
- coding
- mathematics and statistics
- telling stories
Before thinking of the analysis in your report, what should you think of?
Think about the structure of the report.
What does the structure of the report depend on?
The length of the document.
What is the difference between a brief report and a detailed report?
A brief report is more to the point and presents a summary of key findings.
A detailed report incrementally builds the argument and contains details about other relevant works, research methodology, data sources, and intermediate findings along with the main results.
What can cause the length of the report to be varied?
The purpose of the report.
What should all reports have?
- a cover page,
- table of contents,
- executive summary,
- detailed contents,
- acknowledgments,
- references, and
- appendices (if needed).
What should a cover page have?
At a minimum, the cover page should include
- the title of the report,
- names of authors,
- their affiliations, and contacts,
- the name of the institutional publisher (if any), and
- the date of publication.
