Note

Go to the end to download the full example code.

Warm Starting in Iterative Solvers

import condor

Here’s a plain Python function implementing the well-known Rosenbrock optimization benchmark function.

def rosenbrock(x, y, a=1, b=100):
    return (a - x) ** 2 + b * (y - x**2) ** 2


# TODO plot the function and maybe iteration history?

We’ll minimize this function on a circle of a given radius and track the iteration history with an iter_callback option:

call_from_count = []


class RosenbrockOnCircle(condor.OptimizationProblem):
    r = parameter()
    x = variable(warm_start=False)
    y = variable(warm_start=False)

    objective = rosenbrock(x, y)

    constraint(x**2 + y**2 == r**2)

    class Options:
        print_level = 0

        @staticmethod
        def iter_callback(i, variable, objective, constraint):
            print(f"  inner {i:2d}: {variable=}")

Calling the model twice without warm starting gives the same number of iterations each time.

out1 = RosenbrockOnCircle(r=2)

inner  0: variable=RosenbrockOnCircleVariable(x=array([0.]), y=array([0.]))
inner  1: variable=RosenbrockOnCircleVariable(x=array([1.]), y=array([0.]))
inner  2: variable=RosenbrockOnCircleVariable(x=array([2.5]), y=array([9.326861e-08]))
inner  3: variable=RosenbrockOnCircleVariable(x=array([2.05]), y=array([1.34448761e-07]))
inner  4: variable=RosenbrockOnCircleVariable(x=array([2.00060976]), y=array([4.53256906e-07]))
inner  5: variable=RosenbrockOnCircleVariable(x=array([2.00000009]), y=array([1.3820386e-05]))
inner  6: variable=RosenbrockOnCircleVariable(x=array([1.99999967]), y=array([0.04752835]))
inner  7: variable=RosenbrockOnCircleVariable(x=array([1.99943558]), y=array([0.04751504]))
inner  8: variable=RosenbrockOnCircleVariable(x=array([1.9994273]), y=array([0.04786012]))
inner  9: variable=RosenbrockOnCircleVariable(x=array([1.99942726]), y=array([0.04786041]))
inner 10: variable=RosenbrockOnCircleVariable(x=array([1.99942724]), y=array([0.04786129]))
inner 11: variable=RosenbrockOnCircleVariable(x=array([1.99942718]), y=array([0.04786392]))
inner 12: variable=RosenbrockOnCircleVariable(x=array([1.99942699]), y=array([0.0478718]))
inner 13: variable=RosenbrockOnCircleVariable(x=array([1.99942642]), y=array([0.04789545]))
inner 14: variable=RosenbrockOnCircleVariable(x=array([1.99942472]), y=array([0.04796642]))
inner 15: variable=RosenbrockOnCircleVariable(x=array([1.99941961]), y=array([0.04817938]))
inner 16: variable=RosenbrockOnCircleVariable(x=array([1.99940419]), y=array([0.04881888]))
inner 17: variable=RosenbrockOnCircleVariable(x=array([1.99935711]), y=array([0.05074305]))
inner 18: variable=RosenbrockOnCircleVariable(x=array([1.99920838]), y=array([0.05656664]))
inner 19: variable=RosenbrockOnCircleVariable(x=array([1.9986923]), y=array([0.07450618]))
inner 20: variable=RosenbrockOnCircleVariable(x=array([1.99643921]), y=array([0.13278571]))
inner 21: variable=RosenbrockOnCircleVariable(x=array([1.9808514]), y=array([0.3543406]))
inner 22: variable=RosenbrockOnCircleVariable(x=array([1.77659508]), y=array([1.42657635]))
inner 23: variable=RosenbrockOnCircleVariable(x=array([1.33173001]), y=array([1.5630149]))
inner 24: variable=RosenbrockOnCircleVariable(x=array([1.25591343]), y=array([1.55834893]))
inner 25: variable=RosenbrockOnCircleVariable(x=array([1.24944117]), y=array([1.56171379]))
inner 26: variable=RosenbrockOnCircleVariable(x=array([1.24939192]), y=array([1.56173616]))
inner 27: variable=RosenbrockOnCircleVariable(x=array([1.24939191]), y=array([1.56173616]))

Enabling warm start won’t change the number of iterations on this call, but it will on the next call.

RosenbrockOnCircle.x.warm_start = True
RosenbrockOnCircle.y.warm_start = True

out2 = RosenbrockOnCircle(r=2)

inner  0: variable=RosenbrockOnCircleVariable(x=array([0.]), y=array([0.]))
inner  1: variable=RosenbrockOnCircleVariable(x=array([1.]), y=array([0.]))
inner  2: variable=RosenbrockOnCircleVariable(x=array([2.5]), y=array([9.326861e-08]))
inner  3: variable=RosenbrockOnCircleVariable(x=array([2.05]), y=array([1.34448761e-07]))
inner  4: variable=RosenbrockOnCircleVariable(x=array([2.00060976]), y=array([4.53256906e-07]))
inner  5: variable=RosenbrockOnCircleVariable(x=array([2.00000009]), y=array([1.3820386e-05]))
inner  6: variable=RosenbrockOnCircleVariable(x=array([1.99999967]), y=array([0.04752835]))
inner  7: variable=RosenbrockOnCircleVariable(x=array([1.99943558]), y=array([0.04751504]))
inner  8: variable=RosenbrockOnCircleVariable(x=array([1.9994273]), y=array([0.04786012]))
inner  9: variable=RosenbrockOnCircleVariable(x=array([1.99942726]), y=array([0.04786041]))
inner 10: variable=RosenbrockOnCircleVariable(x=array([1.99942724]), y=array([0.04786129]))
inner 11: variable=RosenbrockOnCircleVariable(x=array([1.99942718]), y=array([0.04786392]))
inner 12: variable=RosenbrockOnCircleVariable(x=array([1.99942699]), y=array([0.0478718]))
inner 13: variable=RosenbrockOnCircleVariable(x=array([1.99942642]), y=array([0.04789545]))
inner 14: variable=RosenbrockOnCircleVariable(x=array([1.99942472]), y=array([0.04796642]))
inner 15: variable=RosenbrockOnCircleVariable(x=array([1.99941961]), y=array([0.04817938]))
inner 16: variable=RosenbrockOnCircleVariable(x=array([1.99940419]), y=array([0.04881888]))
inner 17: variable=RosenbrockOnCircleVariable(x=array([1.99935711]), y=array([0.05074305]))
inner 18: variable=RosenbrockOnCircleVariable(x=array([1.99920838]), y=array([0.05656664]))
inner 19: variable=RosenbrockOnCircleVariable(x=array([1.9986923]), y=array([0.07450618]))
inner 20: variable=RosenbrockOnCircleVariable(x=array([1.99643921]), y=array([0.13278571]))
inner 21: variable=RosenbrockOnCircleVariable(x=array([1.9808514]), y=array([0.3543406]))
inner 22: variable=RosenbrockOnCircleVariable(x=array([1.77659508]), y=array([1.42657635]))
inner 23: variable=RosenbrockOnCircleVariable(x=array([1.33173001]), y=array([1.5630149]))
inner 24: variable=RosenbrockOnCircleVariable(x=array([1.25591343]), y=array([1.55834893]))
inner 25: variable=RosenbrockOnCircleVariable(x=array([1.24944117]), y=array([1.56171379]))
inner 26: variable=RosenbrockOnCircleVariable(x=array([1.24939192]), y=array([1.56173616]))
inner 27: variable=RosenbrockOnCircleVariable(x=array([1.24939191]), y=array([1.56173616]))

out3 = RosenbrockOnCircle(r=2)

inner  0: variable=RosenbrockOnCircleVariable(x=array([1.24939191]), y=array([1.56173616]))
inner  1: variable=RosenbrockOnCircleVariable(x=array([1.24939191]), y=array([1.56173616]))

for use_warm_start in [False, True]:
    print("=== with warm_start =", use_warm_start)
    RosenbrockOnCircle.x.warm_start = use_warm_start
    RosenbrockOnCircle.y.warm_start = use_warm_start

    print("=== Embed within optimization over disk radius")

    class Outer(condor.OptimizationProblem):
        # r = variable(initializer=2+(5/16)+(1/64))
        r = variable(initializer=1.5, warm_start=False)

        out = RosenbrockOnCircle(r=r)

        objective = rosenbrock(out.x, out.y)

        class Options:
            print_level = 0

            # with exact_hessian = False means more outer iters and also a larger
            # percentage of calls correctly going through #the warm start -- I assume
            # the ones where it is re-starting is because of the jacobian?,
            # produces about a 16 iter difference
            exact_hessian = False

            @staticmethod
            def iter_callback(i, variable, objective, constraint):
                print(f"outer {i:2d}: {variable=}")

    out = Outer()
    print(out.r)

=== with warm_start = False
=== Embed within optimization over disk radius
  inner  0: variable=RosenbrockOnCircleVariable(x=array([1.24939191]), y=array([1.56173616]))
  inner  1: variable=RosenbrockOnCircleVariable(x=array([1.07968183]), y=array([1.13723067]))
  inner  2: variable=RosenbrockOnCircleVariable(x=array([1.04114549]), y=array([1.08192418]))
  inner  3: variable=RosenbrockOnCircleVariable(x=array([1.03973705]), y=array([1.08117964]))
  inner  4: variable=RosenbrockOnCircleVariable(x=array([1.0397353]), y=array([1.08118015]))
  inner  5: variable=RosenbrockOnCircleVariable(x=array([1.0397353]), y=array([1.08118015]))
  inner  6: variable=RosenbrockOnCircleVariable(x=array([1.0397353]), y=array([1.08118015]))
  inner  7: variable=RosenbrockOnCircleVariable(x=array([1.0397353]), y=array([1.08118015]))
  inner  8: variable=RosenbrockOnCircleVariable(x=array([1.0397353]), y=array([1.08118015]))
outer  0: variable=OuterVariable(r=array([1.5]))
  inner  9: variable=RosenbrockOnCircleVariable(x=array([1.24939191]), y=array([1.56173616]))
  inner 10: variable=RosenbrockOnCircleVariable(x=array([1.06926167]), y=array([1.11116615]))
  inner 11: variable=RosenbrockOnCircleVariable(x=array([1.02499334]), y=array([1.04781316]))
  inner 12: variable=RosenbrockOnCircleVariable(x=array([1.02310406]), y=array([1.04681093]))
  inner 13: variable=RosenbrockOnCircleVariable(x=array([1.0231009]), y=array([1.04681185]))
  inner 14: variable=RosenbrockOnCircleVariable(x=array([1.0231009]), y=array([1.04681185]))
  inner 15: variable=RosenbrockOnCircleVariable(x=array([1.0231009]), y=array([1.04681185]))
  inner 16: variable=RosenbrockOnCircleVariable(x=array([1.0231009]), y=array([1.04681185]))
outer  1: variable=OuterVariable(r=array([1.46374536]))
  inner 17: variable=RosenbrockOnCircleVariable(x=array([1.24939191]), y=array([1.56173616]))
  inner 18: variable=RosenbrockOnCircleVariable(x=array([1.0547524]), y=array([1.07487329]))
  inner 19: variable=RosenbrockOnCircleVariable(x=array([1.00161881]), y=array([0.99912741]))
  inner 20: variable=RosenbrockOnCircleVariable(x=array([0.9988292]), y=array([0.99763994]))
  inner 21: variable=RosenbrockOnCircleVariable(x=array([0.99882222]), y=array([0.99764191]))
  inner 22: variable=RosenbrockOnCircleVariable(x=array([0.99882222]), y=array([0.99764191]))
  inner 23: variable=RosenbrockOnCircleVariable(x=array([0.99882222]), y=array([0.99764191]))
  inner 24: variable=RosenbrockOnCircleVariable(x=array([0.99882222]), y=array([0.99764191]))
outer  2: variable=OuterVariable(r=array([1.41171357]))
  inner 25: variable=RosenbrockOnCircleVariable(x=array([1.24939191]), y=array([1.56173616]))
  inner 26: variable=RosenbrockOnCircleVariable(x=array([1.05545752]), y=array([1.07663703]))
  inner 27: variable=RosenbrockOnCircleVariable(x=array([1.00277959]), y=array([1.00152673]))
  inner 28: variable=RosenbrockOnCircleVariable(x=array([1.00004104]), y=array([1.00006686]))
  inner 29: variable=RosenbrockOnCircleVariable(x=array([1.00003432]), y=array([1.00006876]))
  inner 30: variable=RosenbrockOnCircleVariable(x=array([1.00003432]), y=array([1.00006876]))
  inner 31: variable=RosenbrockOnCircleVariable(x=array([1.00003432]), y=array([1.00006876]))
  inner 32: variable=RosenbrockOnCircleVariable(x=array([1.00003432]), y=array([1.00006876]))
outer  3: variable=OuterVariable(r=array([1.41428645]))
  inner 33: variable=RosenbrockOnCircleVariable(x=array([1.24939191]), y=array([1.56173616]))
  inner 34: variable=RosenbrockOnCircleVariable(x=array([1.05543756]), y=array([1.0765871]))
  inner 35: variable=RosenbrockOnCircleVariable(x=array([1.00274676]), y=array([1.00145884]))
  inner 36: variable=RosenbrockOnCircleVariable(x=array([1.00000677]), y=array([0.9999982]))
  inner 37: variable=RosenbrockOnCircleVariable(x=array([1.00000005]), y=array([1.0000001]))
  inner 38: variable=RosenbrockOnCircleVariable(x=array([1.00000005]), y=array([1.0000001]))
  inner 39: variable=RosenbrockOnCircleVariable(x=array([1.00000005]), y=array([1.0000001]))
  inner 40: variable=RosenbrockOnCircleVariable(x=array([1.00000005]), y=array([1.0000001]))
outer  4: variable=OuterVariable(r=array([1.41421367]))
  inner 41: variable=RosenbrockOnCircleVariable(x=array([1.24939191]), y=array([1.56173616]))
  inner 42: variable=RosenbrockOnCircleVariable(x=array([1.05543753]), y=array([1.07658702]))
  inner 43: variable=RosenbrockOnCircleVariable(x=array([1.00274671]), y=array([1.00145874]))
  inner 44: variable=RosenbrockOnCircleVariable(x=array([1.00000672]), y=array([0.9999981]))
  inner 45: variable=RosenbrockOnCircleVariable(x=array([1.]), y=array([1.]))
  inner 46: variable=RosenbrockOnCircleVariable(x=array([1.]), y=array([1.]))
  inner 47: variable=RosenbrockOnCircleVariable(x=array([1.]), y=array([1.]))
outer  5: variable=OuterVariable(r=array([1.41421356]))
[1.41421356]
=== with warm_start = True
=== Embed within optimization over disk radius
  inner  0: variable=RosenbrockOnCircleVariable(x=array([1.24939191]), y=array([1.56173616]))
  inner  1: variable=RosenbrockOnCircleVariable(x=array([1.07968183]), y=array([1.13723067]))
  inner  2: variable=RosenbrockOnCircleVariable(x=array([1.04114549]), y=array([1.08192418]))
  inner  3: variable=RosenbrockOnCircleVariable(x=array([1.03973705]), y=array([1.08117964]))
  inner  4: variable=RosenbrockOnCircleVariable(x=array([1.0397353]), y=array([1.08118015]))
  inner  5: variable=RosenbrockOnCircleVariable(x=array([1.0397353]), y=array([1.08118015]))
  inner  6: variable=RosenbrockOnCircleVariable(x=array([1.0397353]), y=array([1.08118015]))
  inner  7: variable=RosenbrockOnCircleVariable(x=array([1.0397353]), y=array([1.08118015]))
  inner  8: variable=RosenbrockOnCircleVariable(x=array([1.0397353]), y=array([1.08118015]))
outer  0: variable=OuterVariable(r=array([1.5]))
  inner  9: variable=RosenbrockOnCircleVariable(x=array([1.0397353]), y=array([1.08118015]))
  inner 10: variable=RosenbrockOnCircleVariable(x=array([1.02341314]), y=array([1.04718578]))
  inner 11: variable=RosenbrockOnCircleVariable(x=array([1.02310098]), y=array([1.04681187]))
  inner 12: variable=RosenbrockOnCircleVariable(x=array([1.0231009]), y=array([1.04681185]))
  inner 13: variable=RosenbrockOnCircleVariable(x=array([1.0231009]), y=array([1.04681185]))
  inner 14: variable=RosenbrockOnCircleVariable(x=array([1.0231009]), y=array([1.04681185]))
outer  1: variable=OuterVariable(r=array([1.46374536]))
  inner 15: variable=RosenbrockOnCircleVariable(x=array([1.0231009]), y=array([1.04681185]))
  inner 16: variable=RosenbrockOnCircleVariable(x=array([0.99949128]), y=array([0.99842434]))
  inner 17: variable=RosenbrockOnCircleVariable(x=array([0.99882263]), y=array([0.99764203]))
  inner 18: variable=RosenbrockOnCircleVariable(x=array([0.99882222]), y=array([0.99764191]))
  inner 19: variable=RosenbrockOnCircleVariable(x=array([0.99882222]), y=array([0.99764191]))
  inner 20: variable=RosenbrockOnCircleVariable(x=array([0.99882222]), y=array([0.99764191]))
outer  2: variable=OuterVariable(r=array([1.41171357]))
  inner 21: variable=RosenbrockOnCircleVariable(x=array([0.99882222]), y=array([0.99764191]))
  inner 22: variable=RosenbrockOnCircleVariable(x=array([1.00003604]), y=array([1.00007073]))
  inner 23: variable=RosenbrockOnCircleVariable(x=array([1.00003432]), y=array([1.00006876]))
  inner 24: variable=RosenbrockOnCircleVariable(x=array([1.00003432]), y=array([1.00006876]))
  inner 25: variable=RosenbrockOnCircleVariable(x=array([1.00003432]), y=array([1.00006876]))
outer  3: variable=OuterVariable(r=array([1.41428645]))
  inner 26: variable=RosenbrockOnCircleVariable(x=array([1.00003432]), y=array([1.00006876]))
  inner 27: variable=RosenbrockOnCircleVariable(x=array([1.00000005]), y=array([1.0000001]))
  inner 28: variable=RosenbrockOnCircleVariable(x=array([1.00000005]), y=array([1.0000001]))
  inner 29: variable=RosenbrockOnCircleVariable(x=array([1.00000005]), y=array([1.0000001]))
  inner 30: variable=RosenbrockOnCircleVariable(x=array([1.00000005]), y=array([1.0000001]))
outer  4: variable=OuterVariable(r=array([1.41421367]))
  inner 31: variable=RosenbrockOnCircleVariable(x=array([1.00000005]), y=array([1.0000001]))
  inner 32: variable=RosenbrockOnCircleVariable(x=array([1.]), y=array([1.]))
  inner 33: variable=RosenbrockOnCircleVariable(x=array([1.]), y=array([1.]))
outer  5: variable=OuterVariable(r=array([1.41421356]))
[1.41421356]

Total running time of the script: (0 minutes 0.203 seconds)

Gallery generated by Sphinx-Gallery