Wafer-Scale Fast Fourier Transforms

by   Marcelo Orenes-Vera, et al.

We have implemented fast Fourier transforms for one, two, and three-dimensional arrays on the Cerebras CS-2, a system whose memory and processing elements reside on a single silicon wafer. The wafer-scale engine (WSE) encompasses a two-dimensional mesh of roughly 850,000 processing elements (PEs) with fast local memory and equally fast nearest-neighbor interconnections. Our wafer-scale FFT (wsFFT) parallelizes a n^3 problem with up to n^2 PEs. At this point a PE processes only a single vector of the 3D domain (known as a pencil) per superstep, where each of the three supersteps performs FFT along one of the three axes of the input array. Between supersteps, wsFFT redistributes (transposes) the data to bring all elements of each one-dimensional pencil being transformed into the memory of a single PE. Each redistribution causes an all-to-all communication along one of the mesh dimensions. Given the level of parallelism, the size of the messages transmitted between pairs of PEs can be as small as a single word. In theory, a mesh is not ideal for all-to-all communication due to its limited bisection bandwidth. However, the mesh interconnecting PEs on the WSE lies entirely on-wafer and achieves nearly peak bandwidth even with tiny messages. This high efficiency on fine-grain communication allow wsFFT to achieve unprecedented levels of parallelism and performance. We analyse in detail computation and communication time, as well as the weak and strong scaling, using both FP16 and FP32 precision. With 32-bit arithmetic on the CS-2, we achieve 959 microseconds for 3D FFT of a 512^3 complex input array using a 512x512 subgrid of the on-wafer PEs. This is the largest ever parallelization for this problem size and the first implementation that breaks the millisecond barrier.


page 4

page 5


Fast Stencil-Code Computation on a Wafer-Scale Processor

The performance of CPU-based and GPU-based systems is often low for PDE ...

P3DFFT: a framework for parallel computations of Fourier transforms in three dimensions

Fourier and related transforms is a family of algorithms widely employed...

MF-Net: Compute-In-Memory SRAM for Multibit Precision Inference using Memory-immersed Data Conversion and Multiplication-free Operators

We propose a co-design approach for compute-in-memory inference for deep...

Parallelization of the FFT on SO(3)

In this paper, a work-optimal parallelization of Kostelec and Rockmore's...

A Flexible Framework for Parallel Multi-Dimensional DFTs

Multi-dimensional discrete Fourier transforms (DFT) are typically decomp...

Minimizing communication in the multidimensional FFT

We present a parallel algorithm for the fast Fourier transform (FFT) in ...

Please sign up or login with your details

Forgot password? Click here to reset