Module 1 Flashcards
Introducing Parallel and Distributed Concepts in Digital Logic
_ involves processing instructions one at a time, using only a single processor, without distributing tasks across multiple processors.
Serial computing
or sequential computing
_ was introduced as computer science evolved to address the slow speeds of serial computing.
Parallel computing
_ is a method where parallel programming enables computers to run processes and perform calculations simultaneously.
Parallel processing
_ is a process where large computing problems are broken down into smaller problems that multiple processors can solve simultaneously.
Parallel computing
Also known as parallel programming
Multiple processors working simultaneously on different parts of a task.
Example: New British weather supercomputer MetOffice’s
Real-world applications of parallel computing span diverse domains, from scientific simulations to big data analytics and high-performance computing.
Noted
Parallel computing architectures enable efficient processing and analysis of large datasets, sophisticated simulations, and complex computational tasks.
- Task Distribution: Parallel computing is the process by which a supercomputer can split the whole grid into sub-grids
- Simultaneous Computation: Thousands of processors work simultaneously on different parts of the grid to calculate the data which is stored at different locations
- Communication between Processors: The main reason for processors to communicate with each other is the fact that the weather for one part of the grid can have an impact on the areas adjacent to it
_ consist of multiple processing units, or ‘cores,’ on a single integrated circuit (IC). This structure facilitates parallel computing, which enhances performance while potentially reducing power consumption.
Multicore processors
The need for higher performance, faster response times, increased functionality, and energy efficiency has never been more pressing.
Multi-cores, a system can perform multiple tasks at once.
Parallel Computing Benefits
- SPEED AND EFFICIENCY: allows tasks to be completed faster by dividing them into smaller sub-tasks that can be processed simultaneously by multiple processors or cores.
- HANDLING LARGE DATA SETS (Scalability): essential for processing large data sets that would be impractical or too slow to handle sequentially
- SOLVING COMPLEX PROBLEMS: allows for the tackling of such problems by leveraging multiple processors.
- FAULT TOLERANCE: Parallel systems can be designed to be fault-tolerant, meaning they can continue to operate even if one or more processors fail. This improves the reliability and availability of the system.
PARALLEL COMPUTING IS A VERSATILE TOOL APPLIED IN MANY DIFFERENT AREAS OF INDUSTRY, INCLUDING:
- SCIENTIFIC SIMULATIONS: Parallel computing is required for simulations that are complex and belong to such fields as physics, chemistry, and biology. (It enables researchers to model large-scale systems)
- DATA ANALYSIS: In genomics, astronomy, and finance, parallel computing is necessary for the analysis of large data sets. (massive datasets faster processing of data, enabling researchers to extract valuable insights and make informed decisions)
- MACHINE LEARNING AND ARTIFICIAL INTELLIGENCE: Setting up a big machine learning model like a neural network needs a huge computational (accelerates the training process, enabling the development of more advanced AI systems.)
Parallel computing plays a vital role in addressing complex problems and enabling advancements in various fields. It provides the computational power necessary for scientific research, data analysis, machine learning, high-performance computing, and other demanding applications
Noted
Named after the Hungarian
mathematician John von Neumann
Von Newmann Architecture
Von Newmann Architecture was named after the Hungarian mathematician _
John von Neumann
A _ computer uses the
stored-program concept
von Neumann
The CPU executes a stored program that specifically a sequence of read and write operations on the memory.
The _ gets the instructions and/ or data from the memory, decodes the instructions, and then sequentially performs them.
CPU
One of the more widely used
classifications, in use since 1966.
Flynn’s Classical Taxonomy
Distinguishes multi-processor computer architectures according to how they can be classified along two independent dimensions of instruction and data.
Flynn’s Classical Taxonomy
According to Flynn’s Classical Taxonomy, each dimension can have only one of two possible states: _ or _.
single or multiple
In Flynn’s Matrix Array, the matrix defines four classification:
SISD | SIMD
MISD | MIMD
Y = INSTRUCTION
X = DATA, Y = INSTRUCTION
SISD - Single Instruction, Single Data
SIMD - Single Instruction, Multiple Data
MISD - Multiple Instruction, Single Data
MIMD - Multiple Instruction, Multiple Data
- A single processor takes data from the single address memory and performs a single instruction on the data at the time.
Flynn’s SISD
Pipelining can be implemented, but only one instruction will be executed at a time.
A single instruction is executed on a multiple different pieces of data.
Flynn’s SIMD
Instructions can be performed sequentially taking advantage of pipelining or in parallel using a multiple processors.
GPU, containing vector processors and array processors, are commonly SIMD system.
Multiple processors work on the same data performing different instructions at the same time.
Flynn’s MISD
Example: Space shuttle flight control system
Autonomous processors perform operations on differences pieces of data either independently or as part of shared memory
Flynn’s MIMD
Several different instruction can be executed at the same time using different streams
Main reasons for using parallel programming
- Save time and/or money
- Solve larger / more complex problems
- Provide concurrency
- Complex, large datasets
The word “distributed” in distributed computing is similar to terms such as?
distributed system
distributed programming
distributed algorithm
Originally referred to independent computers interconnected via a network, that is capable of collaborating on a task.
Distributed Computing
Networks of interconnected computers that work together to achieve a common goal.
Distributed Computing
In distributed computing, computers are often spread across different locations and connected through a _, such as the internet or a local area network (LAN).
network
In a distributed computing system, the workload is divided among the various _, which communicate and coordinate their efforts to achieve a common goal.
nodes
Computers can be physically close ( _ ) or far apart ( _ ).
local network
wide area network
Benefits of distributed computing
- Performance Improvement
- Scalability
- Resilience and redundancy
- Cost-effectiveness
- Efficiency Distributed applications
- Geographical Distribution
- Resource Sharing
Advantages of Distributed Computing
- Leverage Commodity Hardware: Use less expensive, off-the-shelf hardware instead of costly, specialized servers.
- Horizontal Scaling: Easily add more nodes (computers) to a distributed system to handle increased workloads.
- Fault Tolerance: if one node fails, the system can continue to operate, ensuring uninterrupted service.
- Reliability: Can recover quickly from node failures or network issue
Disadvantages of Distributed Computing
- Network Latency: Increased Complexity: Managing network latency can be complex, requiring careful consideration of factors like network topology, bandwidth, and routing protocols.
- Coordination Overhead: Ensuring that multiple nodes coordinate their actions can be challenging, especially in distributed systems with many components.
- Security Concerns: Vulnerabilities: more susceptible to security threats like hacking, data breaches, and denial-of-service attacks due to their interconnected nature.
- Debugging and Troubleshooting: Complexity: Identifying and resolving issues in distributed systems can be challenging due to their distributed nature and the potential for interactions between multiple components.
Example of Servers of Distributed Computing
- Cloud Computing: Services like Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform rely on distributed computing to offer scalable and reliable cloud services.
-
Artificial Intelligence and Machine Learning: Artificial Intelligence (AI) and Machine Learning (ML) are two of the most exciting and rapidly developing fields in technology today. They are also among the most notable use cases for
distributed computing. - Scientific Research and High-Performance Computing (HPC): where distributed is used extensively in these fields, distributed computing is used to solve complex scientific problems that require enormous computational resources.
Distributed Computing Concept
A _ in a computer network is a device that is connected to other devices within the same computer network.
node
It can be a computer, a server, a switch, a hub, a router, or any other device that has an IP address and can communicate with other devices over a network.
Distributed Computing Concept
A node can be from _ or _.
open group
close network
Distributed Computing Concept
Resources can be anything, such as?
On file
On services
Storage facility
On network
Distributed Computing Concept
_ refers to hiding the complexities of the system’s implementation details from users and applications.
Transparency
It aims to provide a seamless and consistent user experience regardless of the system’s underlying architecture, distribution, or configuration.
Keep as much as possible hidden from users.
Distributed Computing Concept
A logical layer on top of nodes collectively.
Middleware
Provides communication and security services while handling failure and other complexities in distributed systems
Middleware
the backbone of the distributed system
_ smooth collaboration between operations and events.
Coordination
_ order events, control access to resources
Synchronization
_ helps with better management of complexities and describes how nodes communicate and interact in a system
Architectural Model
3-Sub Paced of Distributed Computing
- Cluster Computing
- Cloud Computing
- Grid Computing
_ breaks down the problem across several networked computing device
Distributed computing
_ is the world’s largest coding community for children and a coding language with a simple visual interface design to gentle introduction of programming concepts.
Scratch
_ is an increasingly popular choice as a first programming language in computer science curricula.
Python
These architectures use smart clients that contact a server for data, and then format and display that data to the user.
Client-server architectures
Typically used in application servers, these architectures use web applications to forward requests to other enterprise services.
N-tier system architectures
These architectures divide all responsibilities among all peer computers, which can serve as clients or servers.
Peer-to-peer architectures
PARALLEL COMPUTING VS DISTRIBUTED COMPUTING
NUMBER OF COMPUTER SYSTEMS INVOLVED
Parallel Computing: A task is divided into multiple sub-tasks which are then allotted to different processors on the same computer system.
Distributed Computing: A number of unified computers work towards a common task while communicating with each other with the help of message passing.
PARALLEL COMPUTING VS DISTRIBUTED COMPUTING
DEPENDENCY BETWEEN PROCESSES
Parallel Computing: A single physical computer system hosts multiple processors.
Distributed Computing: Multiple physical computer systems are present in the computer systems.
PARALLEL COMPUTING VS DISTRIBUTED COMPUTING
SCALABILITY
Parallel Computing: The systems that implement parallel computing have limited scalability.
Distributed Computing: Easily scalable as there are no limitations on how many systems can be added to a network.
PARALLEL COMPUTING VS DISTRIBUTED COMPUTING
RESOURCE SHARING
Parallel Computing: All processors share the same memory.
Distributed Computing: Computers have their own memory and processors.
PARALLEL COMPUTING VS DISTRIBUTED COMPUTING
SYNCHRONIZATION
Parallel Computing: All processors use the same master clock for synchronization.
Distributed Computing: Networks have to implement synchronization algorithms.
PARALLEL COMPUTING VS DISTRIBUTED COMPUTING
USAGE
Parallel Computing: Generally preferred in places requiring faster speed and better performance.
Distributed Computing: Generally preferred in places requiring high scalability.
Parallel computing uses multiple processors within a single computer system to divide and process tasks simultaneously, with shared memory and synchronization through a master clock. It is ideal for faster performance but has limited scalability. In contrast, distributed computing involves multiple independent systems working together via message passing, each with its own memory and processors. This allows for greater scalability and is suited for tasks requiring extensive resource sharing across systems.
Noted
