Lecture 4 Flashcards
How do shared memory systems work?
Multiple processors or cores share a common physical memory space (typically implemented as RAM).
No need for message passing.
OpenMP provide APIs to facilitate shared memory programming.
How do distributed memory systems work?
Each process operates independently with their own private memory. These systems rely on messaging passing for communication.
What are clusters?
A computing system that consists of multiple interconnected computers or nodes.
What are nodes?
An individual computing entity within a larger computing system, such as a cluster or a distributed computing network. Can be treated like a standalone computer or server.
Nodes of a cluster are joined by a communication network.
What are interconnection networks?
The topology that connects individual nodes within a larger computing system.
Bus interconnect?
All nodes are connected to a single communication medium through which nodes can transmit and receive data.
Simple and low cost to implement since it requires few components.
Limited bandwidth. As more nodes are added to the system, the available bandwidth is shared among them, which can lead to congestion and reduced performance as the system scales.
Shared memory interconnect and crossbar?
Allows multiple nodes to access and share a communication memory address space. Consistency and coherency is maintained using synchronisation mechanisms such as semaphores, mutex and critical sections.
Can scale but are limited by the capacity and bandwidth of the shared memory itself. As the number of processors increases, the contention for share memory access may degrade performance.
Crossbar: inputs and outputs in a grid-like structure, forming a crossbar pattern; each input port is connected to each output port. This is non-blocking; allowing any pair of inputs and outputs to communicate.
This is faster than busses and offers high bandwidth but have a complex implementation the more processes there are.
Direct interconnect?
Provides direct point-to-point communication links between individual processing elements or nodes. Low latency since there’s no contention or shared resources among nodes.
Ring topology?
Circular topology: nodes are connected in a closes loop. Each node has two neighbouring nodes on the left and right. Each node receives data from its left, processes it, and sends to the right.
Simple to implement but have limited bandwidth when the number of nodes or distance between nodes increases.
Torus interconnect?
Uses a toroidal mesh topology. Each node is connected to its neighbouring nodes in horizontal and vertical directions. The top row is connected to the bottom row and leftmost is connect to the rightmost.
Shortest path between two nodes can be found by traversing the edges of the grid in both horizontal and vertical directions.
Low latency.
Provide a good balance between connectivity, scalability and low-latency communication but involve complex network routing algorithms.
Static INs?
Static interconnected networks are hardwired into a machine.
Processes connected directly to each other - no switches (e.g. linear ring)
Communication is fast but more distant nodes must use multiple steps.
Static topologies do well on problems which have predictable communication patterns.
Used for both SIMD and MIMD designs - not shared memory.
Hypercube interconnect?
Highly connected network topology that forms a multidimensional cube-like structure.
The number of neighbours of a node in a Hypercube grows exponentially with the dimension of the cube. In an n-dimensional Hypercube, each node is connected to n neighbours. Short paths between any two nodes.
Routing is simple:
- Calculate source XOR destination
- number of hops needed = num_of_bits = hamming distance
Hypercubes can be partitioned into sub-cubes (3D -> two 2D cubes).
- Take the first digit as the sub-cube id.
- The remaining digits identify a node in the sub-cube.
Scalable but comes with complexity.
Dynamic INs?
Allow for the dynamic addition or removal of nodes and connections during operations. When a new node is added, the network dynamically adjusts its routing tables, addressing schemes, and communication paths to incorporate the new node into existing network topology.
Usually outperforms busses.
Useful for scenarios that require flexibility, scalability, and fault tolerance. Static IN is better for situations where deterministic performance is more important.
Foster’s methodology?
- Partitioning: divide the computation to be performed and the data operated on by the computation into small tasks. The focus here should be on identifying tasks that can be executed in parallel.
- Communication: determine what communication needs to be carried out among the tasks identified in the previous step.
- Agglomeration / aggregation: combine tasks and communications identified in the first step into larger tasks. (E.g. if task A must be expected before task B, they can be aggregated into one tasks).
- Mapping: assign the composite tasks from the previous step to processes/threads. This should be done so that communication is minimised, and each process/thread gets roughly the same amount of work.
