Diagonal Memory Optimisation for Machine Learning on Micro-controllers

by   Peter Blacker, et al.

As machine learning spreads into more and more application areas, micro controllers and low power CPUs are increasingly being used to perform inference with machine learning models. The capability to deploy onto these limited hardware targets is enabling machine learning models to be used across a diverse range of new domains. Optimising the inference process on these targets poses different challenges from either desktop CPU or GPU implementations, where the small amounts of RAM available on these targets sets limits on size of models which can be executed. Analysis of the memory use patterns of eleven machine learning models was performed. Memory load and store patterns were observed using a modified version of the Valgrind debugging tool, identifying memory areas holding values necessary for the calculation as inference progressed. These analyses identified opportunities optimise the memory use of these models by overlapping the input and output buffers of individual tensor operations. Three methods are presented which can calculate the safe overlap of input and output buffers for tensor operations. Ranging from a computationally expensive approach with the ability to operate on compiled layer operations, to a versatile analytical solution which requires access to the original source code of the layer. The diagonal memory optimisation technique is described and shown to achieve memory savings of up to 34.5 models. Micro-controller targets are identified where it is only possible to deploy some models if diagonal memory optimisation is used.


evoML Yellow Paper: Evolutionary AI and Optimisation Studio

Machine learning model development and optimisation can be a rather cumb...

Serverless inferencing on Kubernetes

Organisations are increasingly putting machine learning models into prod...

TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems

Deep learning inference on embedded devices is a burgeoning field with m...

Deep Memory Networks for Attitude Identification

We consider the task of identifying attitudes towards a given set of ent...

Improving Input-Output Linearizing Controllers for Bipedal Robots via Reinforcement Learning

The main drawbacks of input-output linearizing controllers are the need ...

A Theory of I/O-Efficient Sparse Neural Network Inference

As the accuracy of machine learning models increases at a fast rate, so ...

Folding and Unfolding on Metagraphs

Typed metagraphs are defined as hypergraphs with types assigned to hyper...

Please sign up or login with your details

Forgot password? Click here to reset