Chapter 3 : Architecture In Parallel Processing Flashcards
What is the technology before Parallel Processing?
- Serial Computation
What is the definition of serial computation?
- To be run on a single computer having a single Central Processing Unit ( CPU )
- A problem is broken into a discrete series of instructions
- Instructions are executed one after another
- Only one instruction may execute at any moment in time
What is Parallel Processing?
- Is the sumultaneous use of multiple compute resources to solve a computational problem
What are required in Parallel Processing?
- To be run using multiple CPUs
- A problem that is broken into discrete parts that can be solved concurrently
- Each part is further broken down to a series of instructions
How can the task be divided into?
- Problem
- Discreate Parts
- Instructions
What event is in 1940s - 1950s Conceptual Beginnings ( 2 )
- John von Neumann’s Contributions ( Architecture )
- IBM 7030 Stretch
What event occurs in 1960s : The Advent of Supercomputing?
- CDC 6600 by Seymour Cray
What event occurs in 1970s : The Rise of Multiprocessing
- Cray-1 and vector processing
- Introduction of SIMD
- ILLIAC IV and array processing
What event occur in 1980s: Commercialization and Expansion?
- Connection Machine by Thinking Machines Corporation
- Introduction of MIMD Architectures
- Development of MPI
What event occurs in 1990s: Growth of Multi-core and Many-core Processors?
- Introduction of multi-core processors by Intel and AMD
- Use of GPUs in parallel processing
What event occurs in 2000s- Present : Ubiquity and Advanced Architectures
- Parallel Processing in consumer-grade CPUs and GPUs
- Cloud computing and distributed systems
- Advanced in AI and machine learning.
- Exploration of quantum and neuromorphic computing
When is Conceptual Beginnings happen?
- 1940s - 1950s
When is The Advent of Supercomputing happen?
- 1960s
When is The Rise of Multiprocessing occurs?
- 1970s
When is Commercialization and Expansion occurs?
- 1980s
When is Growth of Multi-core and Many-core processors occurs?
- 1990s
When is Ubiquity and Advanced Architectures occurs?
- 2000s - Present
What is the types of Parallelism? ( 2 )
- Data Parallelism
- Task Parallelism
What does Data Parallelism involves?
- Involved distributing subsets of the same data across multiple processors and performing the same operation on each subset simultaneously
What is the advantages of Data Parallelism? ( 2 )
- Simplifies the parallelization process as the same operation is applied to all data
- Suitable for problems with a high degree of regularity
What are the applications for Parallel Processing?
- Image Processing
- Scientific Modelling
- Linear Algebra
Why image processing requires parallel processing?
- There is multiple human poses are being detected and tracked simultaneously
What does task parallelism involves?
- Distributing different tasks or threads across multiple processors where each task can execute independently
What is the applications for Task Parallelism? ( 2 )
- Web Servers
- Realtime Data Processing
What does granularity means?
- The size of the tasks ( Units of work )
List out 2 types of Granularity
- Fine-Grained Parallelism
- Coarse-Grained Parallelism
What does Fine-Grained Parallelism involves?
- Dividing tasks into small, easily manageable units that can be executed in parallel
What does Frame Decomposition refer to in Fine-Grained Parallelism in Video Decoding? ( 2 )
- A video frame is divided into smaller blocks (macroblocks or tiles)
How does Parallel Task Assignment enhance Fine-Grained Parallelism in Video Decoding?
- The decoding of individual blocks within a frame can be assigned to different cores of a processor or threads on a GPU
Why is fine-grained parallelism beneficial for video decoding?
- By breaking a video frame into smaller blocks ( macroblocks or titles) and decode independently of other
What is the significance of “concurrent execution” in fine-grained parallelism?
- Concurrent execution means that while some cores are decoding one part of the frame, others work on different parts
- Which can ensure that the entire frame is decoded as quickly as possible
What is the advantages of Fine-Grained Parallelism? ( 2 )
- Achieve high performance for problems with significant opportunities for parallelism
- Suitable for applications requiring detailed and frequent coordination
What is the challenges of Fine-Grained Parallelism?
- High overhead from task management and synchronization
- Increases complexity in programming and debugging
What does Coarse-Grained Parallelism involves?
- Involves dividing tasks into larger units of work that are processed in parallel
What is the characteristics of Coarse-Grained Parallelism?
- Lower communication overhead compared to fine-grained parallelism
- Task are larger and less frequent communication is needed
- Dividing simulation into large, independent segmenst processed by different processors
What is the advantages of Coarse-Grained Parallelism?
- Reduced synchronization and communication overhead
- Easier to implement and manage compared to fine-grained parallelism
What is the challenges of Coarse-Grained Parallelism?
- Less flexibility and granularity in task distribution
- May not fully utilize all processing resources if tasks are unevenly distributed
What is the widely used classifications to classify parallel computers?
- Flynn’s Taxonomy ( 1966 )
How does Flynn’s Taxonomy distinguishes multi-processor computer architectures?
- According to how they can be classified along the 2 independent dimensions of Instruction and Data
- Each 2 dimensions can have only 2 possible states Single or Multiple
List out 4 possible classifications for Flynn’s Taxonomy
- SISD
- Single Instruction
- Single Data
- SIMD
- Single Insctruction
- Multiple Data
- MISD
- Multiple Instruction
- Single Data
- MIMD
- Multiple Instruction
- Multiple Data
What is a SISD also known as?
- A serial ( non-parallel ) computer
What doe SISD do?
- Single Instruction :
- Only 1 instruction stream is being acted on by the CPU during any more clock cycle
- Single Data
- Only 1 data stream is being used as input during any one clock cycle
What does SIMD do?
- Single Instruction
- All processing units execute the same instruction at any given clock cycle
- Multiple Data
- Each processing unit can operate on a different data element
- It is a type of parallel computer
- Each processing unit can operate on a different data element
What does MISD do?
- Multiple Instruction
- Each processing unit operates on the data independently via independent instruction sreams
- Single Data
- Single data stream is fed into multiple processing units
What does MIMD do? ( 2 )
- Multiple Instruction
- Every processor may be executing a different instruction stream
- Multiple Data
- Every processor may be working with a different data stream
What is the most common type of parallel computer?
- MIMD ( Multiple Instruction, Multiple Data )
What category ( Flynn’s Taxonomy ) that most modern computers fall into?
- MIMD ( Multiple Instruction, Multiple Data )
What is a multi-core processors?
- Containes 2 or more independent cores within a single computing component
- Each core can execute its own thread simulteneously, sharing the same memory and I/O
- Common configurations include dual-core, quad-core, hexa-core and octa-core
List out examples for SIMD
- Modern Graphics Processing Unit ( GPUs )
- In video games, image processing, and scientific simulations, GPUs execute the same operation simultaneously on multiple data points (such as pixels or vertices). This enables highly efficient parallel processing for tasks like rendering and deep learning computations.
List out examples for SISD
- Simple Microcontrollers and Legacy Systems
- Many embedded systems, such as those in basic appliances or older computing devices, use SISD architectures where a single processor executes one instruction on one piece of data at a time. This model is sufficient for applications with limited processing requirements.
List out examples for MMD
- Multi-core CPUs and Distributed Computing Systems
- In desktop computers, servers, and supercomputers, each core or node in a multi-core processor or cluster can execute its own set of instructions on separate data streams. This design is common in operating systems managing multiple applications or in large-scale scientific computations where different tasks run concurrently
List out examples for MISD
- Niche fault-tolerant systems
- Same data is processed by different algorithms to ensure reliability
