What is an Ill-conditioned Matrix?
In mathematics, a condition number is a number representative of the change of an output proportionate to a change in the input of a function. For example, if a small change in the input results in a small change in the output, the function produces a small condition number and is said to be well-conditioned. Alternatively, if a small change in the input results in a large change in the output, the function produces a large condition number and is defined as ill-conditioned.
Imagine an example linear equation in which Ax = b. The condition number is, very generally, a representation of the rate at which the solution (x) changes with respect to changes in the value of (b). The condition number is a property of the matrix itself, not the algorithm. If the condition number of a matrix is too large, it is labeled as an ill-conditioned matrix. Condition numbers are representative of the accuracy of computing a matrix' inverse. For example, a well-conditioned matrix means its inverse can be computed with decent accuracy. Alternatively, an ill-conditioned matrix is not invertible and can have a condition number that is equal to infinity.
Ill-conditioned Matrices and Machine Learning
The principles of condition numbers are important in neural networks as a metric for understanding the algorithms sensitivity to changes in its inputs. Data scientists have to take a functions condition number into account when formatting neural networks in order to reduce the networks susceptibility to adversarial examples. Some suggestions for mitigating risks in ill-conditioned matrices involve the applications of orthogonal regularizers.