kymatio
Wavelet scattering transforms in Python with GPU acceleration
view repo
The wavelet scattering transform is an invariant signal representation suitable for many signal processing and machine learning applications. We present the Kymatio software package, an easy-to-use, high-performance Python implementation of the scattering transform in 1D, 2D, and 3D that is compatible with modern deep learning frameworks. All transforms may be executed on a GPU (in addition to CPU), offering a considerable speed up over CPU implementations. The package also has a small memory footprint, resulting inefficient memory usage. The source code, documentation, and examples are available undera BSD license at https://www.kymat.io/
READ FULL TEXT VIEW PDFWavelet scattering transforms in Python with GPU acceleration
None
Many classification and regression tasks have a degree of invariance to translations and diffeomorphisms, such as those relating to images, audio recordings, and electronic densities. The scattering transform was introduced in cpam to build a signal representation that is invariant to such transformations while preserving as much signal information as possible. It is defined as a convolutional network whose filters are fixed to be wavelet and lowpass averaging filters coupled with modulus nonlinearities. It has many favorable theoretical properties (cpam; bruna2015; irene) and enjoys considerable success as a powerful tool in modern signal processing (adel20173d; bruna2013invariant; dss; chudacek; Sifre_2013_CVPR; solidharmonic). It is also effective in combination with modern representation learning approaches (oyallon2018scattering; sainath-dss; zeghidour).
This article presents Kymatio, a scattering transform implementation that is user-friendly, well-documented, fast, and compatible with existing automatic differentiation libraries. It brings together scattering transforms in 1D, 2D, and 3D under a unified application programming interface (API). To increase flexibility, it makes minimal assumptions on the underlying array type of its operands and is thus agnostic to the choice of deep learning backend. The scattering network is also traversed in a depth-first manner to reduce memory requirements, enabling efficient processing in limited-memory settings, such as GPUs.
We consider signals defined on a grid of size for . Given two signals and on this grid, we denote their periodic convolution by . The second-order scattering transform is defined using two wavelet filter banks and , where and are frequency indices in the sets and . It also includes a lowpass filter , where the integer specifies the averaging scale of the filter. Together with a non-linearity , these filters define the scattering transform.
The zeroth-order scattering coefficient is the local average given by . Convolving with the first-order wavelet filter bank , applying , and convolving with , we obtain the first-order scattering coefficients
(1) |
More structure is captured by decomposing using the second filter bank, but this is done only for a subset of since is a low-frequency signal. The result is then passed through and averaged, yielding the second-order coefficients
(2) |
Signals obtained by filtering and applying are low-frequency, so intermediate results are downsampled to reduce computational load as in scatnet. In 1D and 2D, we use Morlet wavelets which are analytic (i.e., complex-valued with zero energy in the negative frequencies) and the non-linearity is the complex modulus for (dss; bruna2013invariant). The 3D transform is calculated using solid harmonic wavelets , where indexes the scale, and are the azimuthal and magnetic quantum numbers. In this case the non-linearity is defined, with a slight abuse of notation, as (solidharmonic). Following oyallon2018scattering, we compute the scattering transform in a depth-first manner, reducing the number of intermediate signals stored at a given time.
Adopting the philosophy of scikit-learn (pedregosa2011scikit), the goal of Kymatio is not to maximize the number of features, but to provide a stable and easy-to-use framework. To this end, we make heavy use of unit tests, minimize the number of dependencies, and strive for intuitive interfaces inspired by modern deep learning paradigms. Kymatio also provides an extensive user guide, including an API reference, a tutorial, installation instructions, and fifteen examples, several of which feature real-world applications.
Kymatio is free and open-source software, licensed under the New (3-clause) BSD license. The members of its core development team all have experience implementing scattering transforms as part of other packages. A key goal of Kymatio is to combine these disparate efforts and foster a community effort in order to produce high-quality software and maintain a critical mass of contributors for its maintenance. The package was released publicly on GitHub November 17th, 2018. The main communication channels is the GitHub page for questions, bug reports, and feature requests. There is also a dedicated Slack channel for Kymatio development.
Dimension | GPU | Diff. | Core Devs. | License | Language | |
ScatNet | 1D, 2D | 5 | Apache 2.0 | MATLAB | ||
ScatNetLight | 2D | 2 | GPLv2 | MATLAB | ||
PyScatWave | 2D | ✓ | 3 | BSD-3 | Python | |
Scattering.m | 1D | 1 | GPLv3 | MATLAB | ||
PyScatHarm | 3D | ✓ | 1 | BSD-3 | Python | |
Wavelet Toolbox | 1D | N/A | Proprietary | MATLAB | ||
Kymatio | 1D, 2D, 3D | ✓ | ✓ | 15 | BSD-3 | Python |
Aside from the emphasis on code quality and usability, Kymatio provides several improvements over previous scattering implementations:
Python is the de facto
standard for data science software, but most existing scattering packages are implemented in MATLAB. In contrast,
Kymatio provides a completely Pythonic implementation, enabling integration with the scientific Python ecosystem.GPU compatibility is critical to many data science workloads. Kymatio offers an easy-to-use GPU implementation for scattering transforms in 1D, 2D, and 3D.
PyTorch
integration is achieved by designing scattering transforms to closley mimic PyTorch modules, allowing seamless integration into many deep learning workflows.
Differentiability of the scattering transform simplifies applications in reconstruction and generative modeling, among others.
Table 1 provides a detailed comparison of existing implementations: ScatNet (scatnet), ScatNetLight (scatnetlight), PyScatWave (oyallon2018scattering), Scattering.m (lostanlen2016wavelet), PyScatHarm (eickenberg2018solid), and the scattering transform implemented in the MATLAB Wavelet Toolbox.
The interface is designed to be flexible and consistent across inputs. We first create a scattering object by specifying the averaging scale and the input signal shape.
The resulting object S acts like a nn.Module object in PyTorch. The scattering transform S is applied through calls of the form
Switching from GPU or CPU functionality also follows the API of nn.module.
Several examples are provided with the code, illustrating the power of Kymatio. These include image reconstruction and generation from scattering (tomas), hybrid scattering and CNN training on CIFAR and MNIST (oyallon2018scattering), regression of molecular properties on QM7/QM9 using solid harmonic scattering (solidharmonic)
, and classifying recordings of spoken digits.
Kymatio provides a well documented, user-friendly, and fast implementation for the scattering transform. It can be used with the PyTorch deep learning framework and supports a variety of applications that have been previously inaccessible to non-experts including hybrid deep learning, generative modeling, and 3-D chemistry applications. Future work includes further optimization for speed, flexibility, and backend support.
We thank Laurent Sifre, Sergey Zagoruyko and Gabriel Huang for their helpful comments. The project was supported by ERC InvariantClass 320959. EB is partially supported by a Google Focused Research Award. MJH is partially supported by Alfred P. Sloan Fellowship #FG-2016-6607, DARPA Young Faculty Award #D16AP00117, and NSF grant #1620216.