What it means
Constrained optimization seeks the best value of an objective subject to rules the solution must obey. The rules carve out a feasible region of allowed points.
- Equality constraints force a value, like a sum equal to one.
- Inequality constraints set bounds, like weights staying non negative.
Where it appears in ML
Many models are constrained problems in disguise.
- Support vector machines maximize a margin subject to classification constraints.
- Probability outputs must be non negative and sum to one.
- Regularization can be seen as a constraint on parameter size.
How to solve it
We can use Lagrange multipliers for equalities and the KKT conditions for inequalities, or projected gradient methods that step then project back into the feasible region.
The constrained optimum may sit on the boundary of the feasible region rather than where the unconstrained minimum would be.
Key idea
Constrained optimization minimizes an objective within a feasible region defined by equality and inequality constraints, and the solution often lies on that region's boundary.