Monday, February 16, 2015

Logistic Regression (2): least squares is non-convex

In http://yetanothermathprogrammingconsultant.blogspot.nl/2015/02/logistic-regression.html we implemented a simple maximum likelihood problem which replicates the results from R on a medium size data set (n=400 cases). The question is of course: why can’t we use some least squares objective. The answer is: this problem turns out to be a non-convex problem and thus we may get into trouble.

The can look like:

image

Of course if you like, we can substitute out the r’s and obtain an unconstrained problem. Note that x and y are data.

Some solvers can solve this, but others have some issues:

Solver Results
Conopt, Ipopt, Knitro

const -3.94913
gre    0.00211
gpa    0.82281
rank2 -0.69297
rank3 -1.36355
rank4 -1.54693

Minos, Snopt

**** ERRORS/WARNINGS IN EQUATION fit(id1)
     1 error(s): exp: FUNC OVERFLOW: x too large (RETURNED 1E299)

Indeed the maximum likelihood version of the model seems easier to solve.