>If I have complicated objective function which can only be evaluated
>numerically, and through theoretical analysis it's really hard to judge
>whether it is convex or not...
>Is there a way to test if the optimization problem is convex or not, using
>non-theoretical analysis methods?
Just by numerically evaluation or sampling it is very difficult to prove anything. There is a tool to estimate whether a problem is convex: MProbe.
If one would have a problem formulated in GAMS, the behavior of BARON can tell you if a model is convex (no branch-and-bound).
It would be a useful to have a "dummy" solver that would tell us if a problem is convex. Essentially just the first phase of BARON without actually solving the problem.