# What is the influence of C in SVMs with linear kernel?

I'm currently using an SVM with a linear kernel to classify my data. There is no error on the training set. I tried several values for the parameter C (10^-5, ..., 10^2). This did not change the error on the test set. Now I wonder: is this an error caused by the ruby bindings for libsvm I am using (https://github.com/febeling/rb-libsvm) or is this theoretically explainable? Should the parameter C always change the performance of the classifier?

-

Just a comment, not an answer: Any program that minimizes a sum of two terms, such as $|w|^2 + C \sum{ \xi_i },$ should (imho) tell you what the two terms are at the end, so that you can see how they balance. (For help on computing the two SVM terms yourself, try asking a separate question. Have you looked at a few of the worst-classified points ? Could you post a problem similar to yours ?) – Denis Jul 13 '12 at 20:02

## 2 Answers

The C parameter tells the SVM optimization how much you want to avoid misclassifying each training example. For large values of C, the optimization will choose a smaller-margin hyperplane if that hyperplane does a better job of getting all the training points classified correctly. Conversely, a very small value of C will cause the optimizer to look for a larger-margin separating hyperplane, even if that hyperplane misclassifies more points. For very tiny values of C, you should get misclassified examples, often even if your training data is linearly separable.

-
– alfa Jun 24 '12 at 12:31

1
I would suggest trying a wider range of C values, maybe 10^[-5,...,5], or more if the optimization is fast on your dataset, to see if you get something that looks more reasonable. Both the training error and the value of the minimum cost should change as C is varied. Also, is the scale of your data extreme? In general, an optimal C parameter should be larger when you scale down your data, and vice versa, so if you have very small values for features, make sure to include very large values for the possible C values. If none of the above helps, I'd guess the problem is in the ruby bindings – Marc Shivers Jun 24 '12 at 19:59
What I said is partially wrong. Actually, the value of C has an influence but it is marginal. I am calculating the balanced accuracy ((tp/(tp+fn)+tn/(tn+fp))/2) on my test set. If the complexity is 10^-5 or 10^-4, the balanced accuracy will be 0.5. When I set C to 10^-3 it is 0.79, for C=10^-2 it is 0.8, for C=10^-1 it is 0.85 and for C=10^0,...,10^7 it is 0.86, which seems to be the best possible value here. The data is normalized such that the standard deviation is 1 and the mean is 0. – alfa Jun 25 '12 at 9:43

2
changing the balanced accuracy from 0.5 (just guessing) to 0.86 doesn't sound like a marginal influence to me. It would be a good idea to investigate a finer grid of values for C as Marc suggests, but the results you gave given seem to be fairly normal behaviour. One might expect the error to go back up again as C tends to infinity due to over-fitting, but that doesn't seem to much of a problem in this case. Note that if you are really interested in balanced error and your training set doesn't have a 50:50 split, then you may be able to get better results... – Dikran Marsupial Jun 25 '12 at 12:58

2
... by using different values of C for patterns belonging to the positive and negative classes (which is asymptotically equivalent to resampling the data to change the proportion of patterns belonging to each class). – Dikran Marsupial Jun 25 '12 at 12:59
show 2 more comments

C is essentially a regularisation parameter, which controls the trade-off between achieving a low error on the training data and minimising the norm of the weights. It is analageous to the ridge parameter in ridge regression (in fact in practice there is little difference in performance or theory between linear SVMs and ridge regression, so I generally use the latter - or kernel ridge regression if there are more attributes than observations).

Tuning C correctly is a vital step in best practice in the use of SVMs, as structural risk minimisation (the key principle behind the basic approach) is party implemented via the tuning of C. The parameter C enforces an upper bound on the norm of the weights, which means that there is a nested set of hypothesis classes indexed by C. As we increase C, we increase the complexity of the hypothesis class (if we increase C slightly, we can still form all of the linear models that we could before and also some that we couldn't before we increased the upper bound on the allowable norm of the weights). So as well as implementing SRM via maximum margin classification, it is also implemented by the limiting the complexity of the hypothesis class via controlling C.

Sadly the theory for determining how to set C is not very well developed at the moment, so most people tend to use cross-validation (if they do anything).

-

OK, I think I understand the meaning of C now. :) – alfa Jun 25 '12 at 19:20
{"extraction_info": {"found_math": true, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 1, "mathjax_display_tex": 0, "mathjax_asciimath": 0, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0}, "config": {"markdown_headings": true, "markdown_code": false, "boilerplate_config": {"ratio_threshold": 0.18, "absolute_threshold": 10, "end_threshold": 15, "enable": true}, "remove_buttons": true, "remove_image_figures": true, "remove_link_clusters": true, "table_config": {"min_rows": 2, "min_cols": 3, "format": "github"}, "remove_chinese": true, "remove_edit_buttons": true, "extract_latex": true}, "english_confidence": 0.9201259613037109, "perplexity_flag": "head"}