What is Ordinary Least Squares?
Ordinary Least Squares is a form of statistical regression used as a way to predict unknown values from an existing set of data. An example of a scenario in which one may use Ordinary Least Squares, or OLS, is in predicting shoe size from a data set that includes height and shoe size. Given the data, one can use the ordinary least squares formula to create a rate of change and predict shoe size, given a subject's height. In short, OLS takes an input, the independent variable, and produces an output, the dependent variable.
How does Ordinary Least Squares work?
Ordinary Least Squares works by taking the input, an independent variable, and combines it with other variables known as betas through addition and multiplication. The first beta is known simply as "beta_1" and is used to calculate the slope of the function. In essence, it tells you what the output would be if the input was zero. The second beta is called "beta_2" and represents the coefficient, or how much of a difference there is between increments in the independent variable.
To find the betas, OLS uses the errors, the vertical distance between a data point and a regression line, to calculate the best slope for the data. The image above exemplifies the concept of determining the squares of the errors to find the regression line. OLS squares the errors and finds the line that goes through the sample data to find the smallest value for the sum of all of the squared errors.
Ordinary Least Squares and Machine Learning
As ordinary least squares is a form of regression, used to inform predictions about sample data, it is widely used in machine learning. Using the example mentioned above, a machine learning algorithm can process and analyze specific sample data that includes information on both height and shoe size. Given the the data points, and using ordinary least squares, the algorithm can begin to make predictions about an individual's shoe size given their height and given the sample data.
