matlab - Why ridge regression minimizes test cost when lambda is negative -

matlab - Why ridge regression minimizes test cost when lambda is negative -

i processing set of info using ridge regression. found interesting phenomenon when apply learned function data. namely, when ridge parameter increases zero, test error keeps increasing. if penalize little coefficients(set parameter <0), test error can smaller.

this matlab code:

for = 1:100 beta = ridgepolyregression(ty_train,tx_train,lambda(i)); sqridge_train_cost(i) = computepolycostmse(ty_train,tx_train,beta); sqridge_test_cost(i) = computepolycostmse(ty_valid,tx_valid,beta); end plot(lambda,sqridge_test_cost,'color','b');

lambda ridge parameter. ty_train output of training data, tx_train input of training data. also, utilize quadratic function regression here.

function [ beta ] = ridgepolyregression( y,tx,lambda ) x = tx(:,2:size(tx,2)); tx2 = [tx,x.^2]; beta = (tx2'*tx2 + lambda * eye(size(tx2,2))) \ (tx2'*y); end

the plotted image is:

why error minimal when lambda negative? sign of under-fitting?

you should not utilize negative lambdas.

from (probabilistic) theoretic point of view, lambda relates inverse of variance of parameter prior distribution, , variance can't negative.

from computational point of view, can (given it's less smallest eigenvalue of covariance matrix) turn positive-definite form indefinite form, means you'll have not maximum, saddle point. means there points target function little (or big) want, can cut down loss indefinitely , no minimum / maximum exists @ all.

your optimization algorithm gives stationary point, global maximum if , if form positive definite.

matlab machine-learning regression