8 - GPU Acceleration and AI HW

Question 1

WHy is AI computationally expensive?

Answer 1

• Massive models with billions of parameters
• Complex operations: Training involves matrix mult., grad calc., backprop
• Iterative Training: Repeatedly updating parameters using grad descent
• Vast Datasets

Question 2

FLOPs

Answer 2

Floating Point Operations Per Second

Question 3

TOPs

Answer 3

Tera Operations Per Second. FOR INTGER-POINT OPERATIONS (compared to floating point)

Question 4

Why aren’t CPUs good for AI?

Answer 4

Their serial nature limits their ability to handle parallel AI workloads.

Question 5

CUDA

Answer 5

Nvidia’s programming framework for GPU tasks beyond graphics.

Question 6

Tensor Cores

Answer 6

Specialised hardware units to handle matrix multiplication.

Each one operates on a 4x4 matrix and performs D = A x B + C (each is a 4x4 matrix)

Perform all operations in one clock rather than sequentially

Question 7

CPUs vs GPUs in core numbers and serial vs parallel

Answer 7

CPUs are masters of serial tasks with lower core numbers that are more powerful

GPUs on the other hand have hundreds of cores and excel at parallel tasks.

Question 8

CUDA Cores

Answer 8

Traditional GPU cores

Question 9

Google TPU make up

Answer 9

Matrix Multiply Units (like tensor cores)
Unified Buffer ( High speed memory for MXU)
Weight FIFO: Stores neural net weights for MXU
Scalar and Vector Units: Handle additional arithmetic and control flow

Question 10

Systolic Arrays

Answer 10

Grid of MACs: The MXU is a large grid of Multiply-Accumulate Units that do A*B+C

Input data (A) flows horizontally through the array and weights (B) flow vertically. Partial sums accumulate vertically.

Question 11

Types of TPUs

Answer 11

Cloud TPUs
Edge TPUs: small power efficient for AI in devices
TPU pods: massively interconnected

Question 12

ASICs

Answer 12

Application Specific Integrated Circuits

Custom circuits for specific AI tasks. Inflexible and costly but efficient

Question 13

FPGAs

Answer 13

FIeld Programmable Gate Arrays

Reconfigurable hardware more versatile than ASICs but lower performance

Question 14

FPGA Architecture

Answer 14

• Configurable Logic Blocks: consists of lookup table, flip flops and multiplexers
• Routing Resources: Network for programmable switches/wires
• Input/Output Blocks: Communication
• Block RAM: Dedicated memory blocks

Question 15

In memory Computing

Answer 15

Integrates computational capabilities directly within memory units
Seeks to minimise or eliminate the need for data movement

Question 16

Neuromorphic computing

Answer 16

Aims to emulate the structure and function of the brain.

Spike timing also related

Question 17

Photonic Tensor Cores

Answer 17

Specialised hardware units using light instead of electrons to perform calculations.

Input modulators
Photonic Waveguides: confined paths
Photonic Computing Elements
Photonic Memory
Output photodetectors (back into elerctrical signals)