Menu
Mistery of the “Rule of 5”. C.Hansch, K.W.Kim, R.H.Sarma, JACS, 1973, Vol. 95, No.19, 6447-6449 Topliss and Costello2 have pointed out the danger of finding meaningless chance correlations with three or four data points per variable. The correlation coefficient is good and there are almost five data points per variable. The method of least squares is a standard approach in regression analysis to approximate the. Proceedings of the 25th International Conference on Machine Learning: 33–40.
-->Creates a linear regression model
Category: Machine Learning / Initialize Model / Regression
Note
Applies to: Machine Learning Studio
This content pertains only to Studio. Similar drag and drop modules have been added to the visual interface in Machine Learning service. Learn more in this article comparing the two versions.
Module overview
This article describes how to use the Linear Regression module in Azure Machine Learning Studio, to create a linear regression model for use in an experiment. Linear regression attempts to establish a linear relationship between one or more independent variables and a numeric outcome, or dependent variable.
You use this module to define a linear regression method, and then train a model using a labeled dataset. The trained model can then be used to make predictions. Alternatively, the untrained model can be passed to Cross-Validate Model for cross-validation against a labeled data set.
More about linear regression
Linear regression is a common statistical method, which has been adopted in machine learning and enhanced with many new methods for fitting the line and measuring error. In the most basic sense, regression refers to prediction of a numeric target. Linear regression is still a good choice when you want a very simple model for a basic predictive task. Linear regression also tends to work well on high-dimensional, sparse data sets lacking complexity.
Azure Machine Learning Studio supports a variety of regression models, in addition to linear regression. However, the term 'regression' can be interpreted loosely, and some types of regression provided in other tools are not supported in Studio.
- The classic regression problem involves a single independent variable and a dependent variable. This is called simple regression. This module supports simple regression.
- Multiple linear regression involves two or more independent variables that contribute to a single dependent variable. Problems in which multiple inputs are used to predict a single numeric outcome are also called multivariate linear regression.The Linear Regression module can solve these problems, as can most of the other regression modules in Studio.
- Multi-label regression is the task of predicting multiple dependent variables within a single model. For example, in multi-label logistic regression, a sample can be assigned to multiple different labels. (This is different from the task of predicting multiple levels within a single class variable.)This type of regression is not supported in Azure Machine Learning. To predict multiple variables, create a separate learner for each output that you wish to predict.
For years statisticians have been developing increasingly advanced methods for regression. This is true even for linear regression. This module supports two methods to measure error and fit the regression line: ordinary least squares method, and gradient descent.
- Gradient descent is a method that minimizes the amount of error at each step of the model training process. There are many variations on gradient descent and its optimization for various learning problems has been extensively studied. If you choose this option for Solution method, you can set a variety of parameters to control the step size, learning rate, and so forth. This option also supports use of an integrated parameter sweep.
- Ordinary least squares is one of the most commonly used techniques in linear regression. For example, least squares is the method that is used in the Analysis Toolpak for Microsoft Excel.Ordinary least squares refers to the loss function, which computes error as the sum of the square of distance from the actual value to the predicted line, and fits the model by minimizing the squared error. This method assumes a strong linear relationship between the inputs and the dependent variable.
How to configure Linear Regression
This module supports two methods for fitting a regression model, with very different options:
- Gradient descent is a better loss function for models that are more complex, or that have too little training data given the number of variables.This option also supports a parameter sweep, if you train the model using Tune Model Hyperparameters to automatically optimize the model parameters.
- For small datasets, it is best to select ordinary least squares. This should give very similar results to Excel.
Create a regression model using ordinary least squares
- Add the Linear Regression Model module to your experiment in Studio.You can find this module in the Machine Learning category. Expand Initialize Model, expand Regression, and then drag the Linear Regression Model module to your experiment.
- In the Properties pane, in the Solution method dropdown list, select Ordinary Least Squares. This option specifies the computation method used to find the regression line.
- In L2 regularization weight, type the value to use as the weight for L2 regularization. We recommend that you use a non-zero value to avoid overfitting.To learn more about how regularization affects model fitting, see this article: L1 and L2 Regularization for Machine Learning
- Select the option, Include intercept term, if you want to view the term for the intercept.Deselect this option if you don't need to review the regression formula.
- For Random number seed, you can optionally type a value to seed the random number generator used by the model.Using a seed value is useful if you want to maintain the same results across different runs of the same experiment. Otherwise, the default is to use a value from the system clock.
- Deselect the option, Allow unknown categorical levels, if you want missing values to raise an error.If this option is selected, an additional level is created for each categorical column. Any levels in the test dataset that were not present in the training dataset are mapped to this additional level.
- Add the Train Model module to your experiment, and connect a labeled dataset.
- Run the experiment.
Results for ordinary least squares model
After training is complete:
- To view the model's parameters, right-click the trainer output and select Visualize.
- To make predictions, connect the trained model to the Score Model module, along with a dataset of new values.
- To perform cross-validation against a labeled data set, connect the untrained model to Cross-Validate Model.
Create a regression model using online gradient descent
- Add the Linear Regression Model module to your experiment in Studio.You can find this module in the Machine Learning category. Expand Initialize Model, expand Regression, and drag the Linear Regression Model module to your experiment
- In the Properties pane, in the Solution method dropdown list, choose Online Gradient Descent as the computation method used to find the regression line.
- For Create trainer mode, indicate whether you want to train the model with a predefined set of parameters, or if you want to optimize the model by using a parameter sweep.
- Single Parameter: If you know how you want to configure the linear regression network, you can provide a specific set of values as arguments.
- Parameter Range: If you want the algorithm to find the best parameters for you, set Create trainer mode option to Parameter Range. You can then specify multiple values for the algorithm to try.
- For Learning rate, specify the initial learning rate for the stochastic gradient descent optimizer.
- For Number of training epochs, type a value that indicates how many times the algorithm should iterate through examples. For datasets with a small number of examples, this number should be large to reach convergence.
- Normalize features: If you have already normalized the numeric data used to train the model, you can deselect this option. By default, the module normalizes all numeric inputs to a range between 0 and 1.NoteRemember to apply the same normalization method to new data used for scoring.
- In L2 regularization weight, type the value to use as the weight for L2 regularization. We recommend that you use a non-zero value to avoid overfitting.To learn more about how regularization affects model fitting, see this article: L1 and L2 Regularization for Machine Learning
- Select the option, Average final hypothesis, to average the final hypothesis.In regression models, hypothesis testing means using some statistic to evaluate the probability of the null hypothesis, which states that there is no linear correlation between a dependent and independent variable. In many regression problems, you must test a hypothesis involving more than one variable.This option is enabled by default, meaning the algorithm tests a combination of the parameters where two or more parameters are involved.
- Select the option, Decrease learning rate, if you want the learning rate to decrease as iterations progress.
- For Random number seed, you can optionally type a value to seed the random number generator used by the model. Using a seed value is useful if you want to maintain the same results across different runs of the same experiment.
- Deselect the option, Allow unknown categorical levels, if you want missing values to raise an error.When this option is selected, an additional level is created for each categorical column. Any levels in the test dataset not present in the training dataset are mapped to this additional level.
- Add a labeled dataset and one of the training modules.If you are not using a parameter sweep, use the Train Model module.To have the algorithm find the best parameters for you, train the model using Tune Model Hyperparameters.NoteIf you configure the model with specific values using the Single Parameter option and then switch to the Parameter Range option, the model is trained using the minimum value in the range for each parameter.Conversely, if you configure specific settings when you create the model but select the Parameter Range option, the model is trained using the default values for the learner as the range of values to sweep over.
- Run the experiment.
Results for online gradient descent
After training is complete:
- To make predictions, connect the trained model to the Score Model module, together with new input data.
- To perform cross-validation against a labeled data set, connect the untrained model to Cross-Validate Model.
Examples
For examples of regression models, see these sample experiments in the Azure AI Gallery:
- Compare Regressors: Contrasts several different kinds of regression models.
- Cross Validation for Regression: Demonstrates linear regression using ordinary least squares.
- Twitter sentiment analysis: Uses several different regression models to generate predicted ratings.
Technical notes
This section contains implementation details, tips, and answers to frequently asked questions.
Usage tips
Many tools support creation of linear regression, ranging from the simple to complex. For example, you can easily perform linear regression in Excel, using the Solver Toolpak, or you can code your own regression algorithm, using R, Python, or C#.
However, because linear regression is a well-established technique that is supported by many different tools, there are many different interpretations and implementations. Not all types of models are supported equally by all tools. There are also some differences in nomenclature to observe.
- Regression methods are often categorized by the number of response variables. For example, multiple linear regression means a model that has multiple variables to predict.
- In Matlab, multivariate regression refers to a model that has multiple response variables.
- In Azure Machine Learning, regression models support a single response variable.
- In the R language, the features provided for linear regression depend on the package you are using. For example, the glm package will give you the ability to create a logistic regression model with multiple independent variables. In general, Azure Machine Learning Studio provides the same functionality as the R glm package.
We recommend that you use this module, Linear Regression, for typical regression problems.
In contrast, if you are using multiple variables to predict a class value, we recommend the Two-Class Logistic Regression or Multiclass Logistic Regression modules.
If you want to use other linear regression packages that are available for the R language, we recommend that you use the Execute R Script module and call the lm or glm packages, which are included in the runtime environment of Azure Machine Learning Studio.
Module parameters
Name | Range | Type | Default | Description |
---|---|---|---|---|
Normalize features | any | Boolean | true | Indicate whether instances should be normalized |
Average final hypothesis | any | Boolean | true | Indicate whether the final hypothesis should be averaged |
Learning rate | >=double.Epsilon | Float | 0.1 | Specify the initial learning rate for the stochastic gradient descent optimizer |
Number of training epochs | >=0 | Integer | 10 | Specify how many times the algorithm should iterate through examples. For datasets with a small number of examples, this number should be large to reach convergence. |
Decrease learning rate | Any | Boolean | true | Indicate whether the learning rate should decrease as iterations progress |
L2 regularization weight | >=0.0 | Float | 0.001 | Specify the weight for L2 regularization. Use a non-zero value to avoid overfitting. |
Random number seed | any | Integer | Specify a value to seed the random number generator used by the model. Leave blank for default. | |
Allow unknown categorical levels | any | Boolean | true | Indicate whether an additional level should be created for each categorical column. Any levels in the test dataset not available in the training dataset are mapped to this additional level. |
Include intercept term | Any | Boolean | True | Indicate whether an additional term should be added for the intercept |
Outputs
Name | Type | Description |
---|---|---|
Untrained model | ILearner interface | An untrained regression model |
See also
Part of a series on Statistics |
Regression analysis |
---|
Models |
Estimation |
Background |
Regularized least squares (RLS) is a family of methods for solving the least-squares problem while using regularization to further constrain the resulting solution.
RLS is used for two main reasons. The first comes up when the number of variables in the linear system exceeds the number of observations. In such settings, the ordinary least-squares problem is ill-posed and is therefore impossible to fit because the associated optimization problem has infinitely many solutions. RLS allows the introduction of further constraints that uniquely determine the solution.
The second reason that RLS is used occurs when the number of variables does not exceed the number of observations, but the learned model suffers from poor generalization. RLS can be used in such cases to improve the generalizability of the model by constraining it at training time. This constraint can either force the solution to be 'sparse' in some way or to reflect other prior knowledge about the problem such as information about correlations between features. A Bayesian understanding of this can be reached by showing that RLS methods are often equivalent to priors on the solution to the least-squares problem.
- 2Kernel formulation
- 2.2Arbitrary kernel
- 2.4Linear kernel
- 5Specific examples
General formulation[edit]
Consider a learning setting given by a probabilistic space , . Let denote a training set of pairs i.i.d. with respect to . Let be a loss function. Define as the space of the functions such that expected risk:
is well defined. The main goal is to minimize the expected risk:
Since the problem cannot be solved exactly there is a need to specify how to measure the quality of a solution. A good learning algorithm should provide an estimator with a small risk.
As the joint distribution is typically unknown, the empirical risk is taken. For regularized least squares the square loss function is introduced:
However, if the functions are from a relatively unconstrained space, such as the set of square-integrable functions on , this approach may overfit the training data, and lead to poor generalization. Thus, it should somehow constrain or penalize the complexity of the function . In RLS, this is accomplished by choosing functions from a reproducing kernel Hilbert space (RKHS) , and adding a regularization term to the objective function, proportional to the norm of the function in :
Kernel formulation[edit]
Definition of RKHS[edit]
A RKHS can be defined by a symmetricpositive-definite kernel function with the reproducing property:
where . The RKHS for a kernel consists of the completion of the space of functions spanned by : , where all are real numbers. Some commonly used kernels include the linear kernel, inducing the space of linear functions:
the polynomial kernel, inducing the space of polynomial functions of order :
and the Gaussian kernel:
Note that for an arbitrary loss function , this approach defines a general class of algorithms named Tikhonov regularization. For instance, using the hinge loss leads to the support vector machine algorithm, and using the epsilon-insensitive loss leads to support vector regression.
Arbitrary kernel[edit]
The representer theorem guarantees that the solution can be written as:
- for some .
The minimization problem can be expressed as:
- ,
where, with some abuse of notation, the entry of kernel matrix (as opposed to kernel function ) is .
For such a function,
The following minimization problem can be obtained:
- .
As the sum of convex functions is convex, the solution is unique and its minimum can be found by setting the gradient w.r.t to :
- ,
where .
Complexity[edit]
The complexity of training is basically the cost of computing the kernel matrix plus the cost of solving the linear system which is roughly . The computation of the kernel matrix for the linear or Gaussian kernel is . The complexity of testing is .
Prediction[edit]
The prediction at a new test point is:
Linear kernel[edit]
For convenience a vector notation is introduced. Let be an matrix, where the rows are input vectors, and a vector where the entries are corresponding outputs. In terms of vectors, the kernel matrix can be written as . The learning function can be written as:
Here we define . The objective function can be rewritten as:
The first term is the objective function from ordinary least squares (OLS) regression, corresponding to the residual sum of squares. The second term is a regularization term, not present in OLS, which penalizes large values.As a smooth finite dimensional problem is considered and it is possible to apply standard calculus tools. In order to minimize the objective function, the gradient is calculated with respect to and set it to zero:
This solution closely resembles that of standard linear regression, with an extra term . If the assumptions of OLS regression hold, the solution , with , is an unbiased estimator, and is the minimum-variance linear unbiased estimator, according to the Gauss–Markov theorem. The term therefore leads to a biased solution; however, it also tends to reduce variance. This is easy to see, as the covariance matrix of the -values is proportional to , and therefore large values of will lead to lower variance. Therefore, manipulating corresponds to trading-off bias and variance. For problems with high-variance estimates, such as cases with relatively small or with correlated regressors, the optimal prediction accuracy may be obtained by using a nonzero , and thus introducing some bias to reduce variance. Furthermore, it is not uncommon in machine learning to have cases where , in which case is rank-deficient, and a nonzero is necessary to compute .
Complexity[edit]
The parameter controls the invertibility of the matrix .Several methods can be used to solve the above linear system,Cholesky decomposition being probably the method of choice, since the matrix is symmetric and positive definite. The complexity of this method is for training and for testing. The cost is essentially that of computing , whereas the inverse computation (or rather the solution of the linear system) is roughly .
Feature maps and Mercer's theorem[edit]
In this section it will be shown how to extend RLS to any kind of reproducing kernel K. Instead of linear kernel a feature map is considered for some Hilbert space , called the feature space. In this case the kernel is defined as: The matrix is now replaced by the new data matrix , where , or the -th component of the .
It means that for a given training set . Thus, the objective function can be written as:
This approach is known as the kernel trick. This technique can significantly simplify the computational operations. If is high dimensional, computing may be rather intensive. If the explicit form of the kernel function is known, we just need to compute and store the kernel matrix .
In fact, the Hilbert space need not be isomorphic to , and can be infinite dimensional. This follows from Mercer's theorem, which states that a continuous, symmetric, positive definite kernel function can be expressed as:
where form an orthonormal basis for , and . If feature maps is defined with components , it follows that . This demonstrates that any kernel can be associated with a feature map, and that RLS generally consists of linear RLS performed in some possibly higher-dimensional feature space. While Mercer's theorem shows how one feature map that can be associated with a kernel, in fact multiple feature maps can be associated with a given reproducing kernel. For instance, the map satisfies the property for an arbitrary reproducing kernel.
Bayesian interpretation[edit]
Least squares can be viewed as a likelihood maximization under an assumption of normally distributed residuals. This is because the exponent of the Gaussian distribution is quadratic in the data, and so is the least-squares objective function. In this framework, the regularization terms of RLS can be understood to be encoding priors on . For instance, Tikhonov regularization corresponds to a normally distributed prior on that is centered at 0. To see this, first note that the OLS objective is proportional to the log-likelihood function when each sampled is normally distributed around . Then observe that a normal prior on centered at 0 has a log-probability of the form
where and are constants that depend on the variance of the prior and are independent of . Thus, minimizing the logarithm of the likelihood times the prior is equivalent to minimizing the sum of the OLS loss function and the ridge regression regularization term.
This gives a more intuitive interpretation for why Tikhonov regularization leads to a unique solution to the least-squares problem: there are infinitely many vectors satisfying the constraints obtained from the data, but since we come to the problem with a prior belief that is normally distributed around the origin, we will end up choosing a solution with this constraint in mind.
Other regularization methods correspond to different priors. See the list below for more details.
Specific examples[edit]
Ridge regression (or Tikhonov regularization)[edit]
One particularly common choice for the penalty function is the squared norm, i.e.,
The most common names for this are called Tikhonov regularization and ridge regression. It admits a closed-form solution for :
The name ridge regression alludes to the fact that the term adds positive entries along the diagonal 'ridge' of the sample covariance matrix.
When , i.e., in the case of ordinary least squares, the condition that causes the sample covariance matrix to not have full rank and so it cannot be inverted to yield a unique solution. This is why there can be an infinitude of solutions to the ordinary least squares problem when . However, when , i.e., when ridge regression is used, the addition of to the sample covariance matrix ensures that all of its eigenvalues will be strictly greater than 0. In other words, it becomes invertible, and the solution becomes unique.
Compared to ordinary least squares, ridge regression is not unbiased. It accepts little bias to reduce variance and the mean square error, and helps to improve the prediction accuracy. Thus, ridge estimator yields more stable solutions by shrinking coefficients but suffers from the lack of sensitivity to the data.
Lasso regression[edit]
The least absolute selection and shrinkage (LASSO) method is another popular choice. In lasso regression, the lasso penalty function is the norm, i.e.
Note that the lasso penalty function is convex but not strictly convex. Unlike Tikhonov regularization, this scheme does not have a convenient closed-form solution: instead, the solution is typically found using quadratic programming or more general convex optimization methods, as well as by specific algorithms such as the least-angle regression algorithm.
An important difference between lasso regression and Tikhonov regularization is that lasso regression forces more entries of to actually equal 0 than would otherwise. In contrast, while Tikhonov regularization forces entries of to be small, it does not force more of them to be 0 than would be otherwise. Thus, LASSO regularization is more appropriate than Tikhonov regularization in cases in which we expect the number of non-zero entries of to be small, and Tikhonov regularization is more appropriate when we expect that entries of will generally be small but not necessarily zero. Which of these regimes is more relevant depends on the specific data set at hand.
Besides feature selection described above, LASSO has some limitations. Ridge regression provides better accuracy in the case for highly correlated variables.[1] In another case, , LASSO selects at most variables. Moreover, LASSO tends to select some arbitrary variables from group of highly correlated samples, so there is no grouping effect.
ℓ0 Penalization[edit]
The most extreme way to enforce sparsity is to say that the actual magnitude of the coefficients of does not matter; rather, the only thing that determines the complexity of is the number of non-zero entries. This corresponds to setting to be the norm of . This regularization function, while attractive for the sparsity that it guarantees, is very difficult to solve because doing so requires optimization of a function that is not even weakly convex. Lasso regression is the minimal possible relaxation of penalization that yields a weakly convex optimization problem.
Elastic net[edit]
For any non-negative and the objective has the following form:
Let , then the solution of the minimization problem is described as:
- for some .
Consider as an Elastic Net penalty function.
When , elastic net becomes ridge regression, whereas it becomes Lasso. Elastic Net penalty function doesn't have the first derivative at 0 and it is strictly convex taking the properties both lasso regression and ridge regression.
One of the main properties of the Elastic Net is that it can select groups of correlated variables. The difference between weight vectors of samples and is given by:
- , where .[2]
If and are highly correlated ( ), the weight vectors are very close. In the case of negatively correlated samples ( ) the samples can be taken. To summarize, for highly correlated variables the weight vectors tend to be equal up to a sign in the case of negative correlated variables.
Partial list of RLS methods[edit]
The following is a list of possible choices of the regularization function , along with the name for each one, the corresponding prior if there is a simple one, and ways for computing the solution to the resulting optimization problem.
Name | Regularization function | Corresponding prior | Methods for solving |
---|---|---|---|
Tikhonov regularization | Normal | Closed form | |
Lasso regression | Laplace | Proximal gradient descent, least angle regression | |
penalization | – | Forward selection, Backward elimination, use of priors such as spike and slab | |
Elastic nets | – | Proximal gradient descent | |
Total variation regularization | – | Split–Bregman method, among others |
See also[edit]
- Regularization in mathematics.
- Generalization error, one of the reasons regularization is used.
References[edit]
- ^Tibshirani Robert (1996). 'Regression shrinkage and selection via the lasso'(PDF). Journal of the Royal Statistical Society, Series B. 58: pp. 266–288.
- ^Hui, Zou; Hastie, Trevor (2003). 'Regularization and Variable Selection via the Elastic Net'(PDF). JRSSB. 67 (2): pp. 301–320.
External links[edit]
- http://www.stanford.edu/~hastie/TALKS/enet_talk.pdf Regularization and Variable Selection via the Elastic Net (presentation)
- Regularized Least Squares and Support Vector Machines (presentation)
- Regularized Least Squares(presentation)
Retrieved from 'https://en.wikipedia.org/w/index.php?title=Regularized_least_squares&oldid=912609073'