# Tagged Questions

The regularization tag has no wiki summary.

1answer
215 views

### matlab gmdistribution.fit 'Regularize' - what regularization method?

I am wondering what is behind matlab 'Regularize' option for method gmdistribution.fit. If it is simply adding a 'little' value to diagonal elements of covariance matrix, so as to make covariance ...
0answers
8 views

### Do I use regularizer when I measure validation error?

I have a cost function with a regularizer term, and I'd like to find to optimal hyperparameters for the regularizer term. So I train with different parameter, but when I measure the validation error, ...
0answers
38 views

### Questions regarding predict.glmpath()

I'm trying to do LASSO in R with the package glmpath. However, I'm not sure if I am using the accompanying prediction function predict.glmpath() correctly. Suppose I fit some regularized binomial ...
1answer
147 views

### Classification with 3 groups, repeated measurements, missing values, more predictors than subjects

I am working on a classification problem with the following characteristics: Individuals belong to one of three groups. The groups are "somewhat ordinal": controls, subclinical and clinical group. ...
4answers
219 views

### What does “degree of freedom” mean in neural networks?

In Bishop's book "Pattern Classification and Machine Learning", it describes a technique for regularization in the context of neural networks. However, I don't understand a paragraph describing that ...
1answer
58 views

### Alternatives to glmnet for feature selection on data with lots of NAs

I have a surgical database in which there are approximately 100,000 observations and 200 features. Each observation corresponds to a separate patient while the features correspond to either ...
0answers
51 views

### Forward Stepwise selection

I am assuming the following model: $Y = \beta X + \epsilon$ Here both $X$ and $Y$ are matrices. I fit the least squares model without any regularization and get the matrix $\beta$. I would like to ...
1answer
77 views

### Parameter estimate for linear regression with regularization

For given cost function $S(\beta) = (Y - X \beta)^T(Y - X \beta) + \lambda \beta^T \beta$, where $\lambda$ is regularization parameter, the $\beta$ that minimizes the given cost function is \$\beta = ...
0answers
57 views

### Ridge regression on subset of variables using SVD

I am trying to figure out an algorithm using singular value decomposition to run a modification of ridge regression in which only some of the variables are penalized. I want the output to match the ...
1answer
48 views

### Question on the usage of regularization in applied statistics/science

I was reading the paper A significance test for the lasso'' by Lockhart, Tibshirani et al and was considering the issue of applying regularization in the applied sciences (for example, behavioral ...
1answer
312 views

### Gradient descent and elastic-net logistic regression

I'm currently in the process of trying to understand the paper Regularization Paths for Generalized Linear Models via Coordinate Descent by Friedman et al. with regard to the regularization of ...
2answers
530 views

### Fitting an ARIMAX model with regularization or penalization (e.g. with the lasso, elastic net, or ridge regression)

I use the auto.arima() function in the forecast package to fit ARMAX models with a variety of covariates. However, I often have a large number of variables to select from and usually end up with a ...
0answers
83 views

### Can the bias introduced by lasso change the sign of a coefficient?

L1 penalized regression introduces a bias on your regression model but decreases the variance. When this bias is introduced, is it possible that the coefficient of $B$ changes sign? This would ...
1answer
70 views

### Reducing the dimensionality of a problem

My particular application needs me to build a linear model with a strong correlation structure amongst the independent variables. The dimensions of the problem are high, for instance 1million X 200. ...
0answers
42 views

### kernelized l1 norm and the representer theorem

I'm trying to derive a kernel-ized $l_1$ penalty for logistic regression. I have been looking at the slides Learning with Sparsity Inducing Norms along with the slides on Regularization and Variable ...
1answer
73 views

### SVM optimization problem

I think I understand the main idea in support vector machines. Let us assume that we have two linear separable classes and want to apply SVMs. What SVM is doing is that it searches a hyperplane ...
0answers
39 views

### Representer theorem for vector-valued functions

Is there a representer theorem for loss-functions of the form $\sum_{i}(f(x_i \mathbb{.}),y_i)$ of the form where the output of $f(.)$ is a vector and the domain is also a vector. Also, there is a ...
1answer
79 views

### How to obtain good performance (low error rate) on massive data set?

Suppose I have massive data set (think Terabytes) is available to train a learning algorithm. Which one of the following conditions must be true to obtain good performance (low error rate) a. Using ...
0answers
113 views

### R glmnet - custom measure for cross validation

In R's glmnet package, there are five options available for the type.measure variable in the cv.glmnet class. Is there a way to specify a custom measure for cross validation? Or is it not possible in ...
1answer
706 views

### libsvm “reaching max number of iterations” warning and cross-validation

I'm using libsvm in C-SVC mode with a polynomial kernel of degree 2 and I'm required to train multiple SVMs. Each training set has 10 features and 5000 vectors. During training, I am getting this ...
0answers
94 views

### Error bars for L2 regularization?

Is there anyway to describe them? I'm a bit new to the field, and a problem I'm working on has led me here. Some background: Given X and B, find A that minimizes  \text{argmin}(A) || AX - B || ...
0answers
145 views

### Manifold regularization using laplacian graph in SVM

I'm trying implement Manifold Regularization in Support Vector Machines (SVMs) in Matlab. I'm following the instructions in the paper by Belkin et al.(2006), there's the equation in it: \$f^{*} = ...
3answers
170 views

### Sparsity-inducing regularization for stochastic matrices

It is well-known (e.g. in the field of compressive sensing) that the $L_1$ norm is "sparsity-inducing," in the sense that if we minimize the functional (for fixed matrix $A$ and vector $\vec{b}$) ...
1answer
75 views

### How does the test error pattern depend on the regularizer function?

This question is regarding the role of regularizer in an objective function. Given a loss function $f(x)$, a regularizer function $r(x)$, and $\lambda$ being a trade-off function, our aim is to ...
4answers
3k views

### GLMNET or LARS for computing LASSO solutions?

I would like to get the coefficients for the LASSO problem $$||Y-X\beta||+\lambda ||\beta||_1.$$ The problem is that glmnet and lars functions give different answers. For the glmnet function I ask ...
2answers
801 views

### Why do Lars and Glmnet give different solutions for the Lasso problem?

I want to better understand the R packages Lars and Glmnet, which are used to solve the Lasso problem: min_{(\beta_0 \beta) ...
1answer
130 views

### What are $\ell_p$ norms and how are they relevant to regularization?

I have been seeing a lot of papers on sparse representations lately, and most of them use the $\ell_p$ norm and do some minimization. My question is, what is the $\ell_p$ norm, and the $\ell_{p, q}$ ...
0answers
236 views

### When does LASSO select correlated predictors?

I'm using the package 'lars' in R with the following code: ...
1answer
257 views

### Standard error of parameter estimates in regularized regression

In a regularized linear regression model (e.g., ridge regression, lasso, etc.), what is the best way to obtain standard errors for parameter estimates? If cross-validation is used, is it ...
2answers
197 views

### Frequentism and priors

Robby McKilliam says in a comment to this post: It should be pointed out that, from the frequentists point of view, there is no reason that you can't incorporate the prior knowledge into the ...
0answers
251 views

### Post processing random forests using regularised regression: what about bias?

I have been playing around with post processing the results of the random forest for regression machine learning algorithm in order to try and do better than the default mean of all trees prediction. ...
0answers
183 views

### Regularization $L_1$ norm and $L_2$ norm empirical study

There are many methods to perform regularization -- $L_0$, $L_1$, and $L_2$ norm based regularization for example. According to Friedman Hastie & Tibsharani, the best regularizer depends on the ...
0answers
47 views

### What's a good range of weights to evaluate for $L_2$ regularized logistic regression?

I want find a weight that minimizes an averaged cross validated misclassification score from a $L_2$ logistic regression classifier. Obviously, the search space for the weights should be bounded below ...
3answers
898 views

### Lasso fitting by coordinate descent: open-source implementations?

What open-source implementations -- in any language -- exist out there that can compute lasso regularisation paths for linear regression by coordinate descent? So far I am aware of: glmnet ...
1answer
433 views

### regularized bayesian logistic regression in JAGS

There are several math-heavy papers that describe the Bayesian Lasso, but I want tested, correct JAGS code that I can use. Could someone post sample BUGS / JAGS code that implements regularized ...
1answer
166 views

### Model function for discovering irrelevant dimensions with L1 regularization

For homework I have been given a 20-dimensional input $x \in \mathbb{R}^{20}$, many of which are suspected to be irrelevant. I tried using L1-norm Lasso regularization to uncover which dimensions ...
1answer
58 views

### What does the index variable k define in the Lasso regularization function

In the Lasso L1 regularization, from where comes the value of the variable $k$ in the second part of the function? Why isn't it $n$, too? L(\beta) = \sum_{i=1}^n (y_i - \phi(x_i)^T \cdot \beta)^2 + ...
4answers
321 views

### Bayesian prior corresponding to penalized regression coefficients

I'm working on a Bayesian Regression problem where I would like to estimate the beta coefficients subject to this constraint (penalty): $\sum|\beta_i|<C$ or similarly $\sum \beta_i^2<C$ Which ...
1answer
729 views

### Using glmnet to solve the LASSO problem

I have recently been made aware of the Lasso algorithm and found that the package glmnet can be used to solve it. (I have the glmnet package on R). If I have a matrix $A$ and a vector $y$ how do I ...
1answer
157 views

### Non-linear regularized SVM implementation

Just a general question. Are there any good non-linear SVM (kernelized) implementations that include a regularization component (e.g. $L_1$, SCAD etc)? I've been looking around but man there are a lot ...
1answer
28 views

### Problem specific regularization

I've been reading a lot recently about the concept of joint regularization in computer vision. Joint regularization builds on the observation that when learning multiple related concepts, for example ...
1answer
127 views

### High dimensional time series

I'm not sure what words I should look for. I have an under determined dataset of 8000 correlated variables (sales) over 12 months (ie 12 observations for each variable). And I basically want to ...
1answer
120 views

### When is there a representer theorem?

The case of regularization in a hilbert space is considered---an optimization problem with an error term and a Tikhonov-regularizer. In the article "When is there a representer theorem" it is stated ...
1answer
585 views

### Need for centering and standardizing data in regression

Consider linear regression with some regularization: E.g. Find $x$ that minimizes $||Ax - b||^2+\lambda||x||_1$ Usually, columns of A are standardized to have zero mean and unit norm, while $b$ is ...
3answers
1k views

### What is regularization in plain english?

Unlike other articles, I found the wikipedia entry for this subject unreadable for a non-math person (like me). I understood the basic idea, that you favor models with fewer rules. What I don't get ...
3answers
1k views

### Cross validation with two parameters: elastic net case

I want to know the cross validation procedure to find the two parameters of elastic net presented by Zou and Hastie on the prostate dataset as example. I can't improve the error rate lasso with k-fold ...
1answer
210 views

### Feature selection with k-fold cross-validated least angle regression

I am using the least angle regression (LARS) to extract the most important predictors ($x_1, x_2,...,x_p$) for my response variable ($y$). I have seven predictors ($x_1,x_2,...,x_7$) for each ...
2answers
923 views

### How is the intercept computed in GLMnet?

I've been implementing the GLMNET version of elastic net for linear regression with another software than R. I compared my results with the R function glmnet in lasso mode on diabetes data. The ...
1answer
449 views

### How to calculate derivative of the contractive auto-encoder regularization term?

Setup I found a paper on that has a varient on normal auto-encoders (contractive) which for its gradient uses the following regularization penalty: \left|\left|J_f(x)\right|\right|^2_F = ...
3answers
524 views

### GLM after model selection or regularization

I would like to pose this question in two parts. Both deal with a generalized linear model, but the first deals with model selection and the other deals with regularization. Background: I utilize ...
{"extraction_info": {"found_math": true, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 46, "mathjax_display_tex": 1, "mathjax_asciimath": 1, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0}, "config": {"markdown_headings": true, "markdown_code": true, "boilerplate_config": {"ratio_threshold": 0.18, "absolute_threshold": 10, "end_threshold": 15, "enable": true}, "remove_buttons": true, "remove_image_figures": true, "remove_link_clusters": true, "table_config": {"min_rows": 2, "min_cols": 3, "format": "github"}, "remove_chinese": true, "remove_edit_buttons": true, "extract_latex": true}, "english_confidence": 0.8984999060630798, "perplexity_flag": "middle"}