1. Test Modules
  2. Training Characteristics
    1. Input Learning
      1. Gradient Descent
      2. Conjugate Gradient Descent
      3. Limited-Memory BFGS
    2. Model Learning
      1. Gradient Descent
      2. Conjugate Gradient Descent
      3. Limited-Memory BFGS
    3. Composite Learning
      1. Gradient Descent
      2. Conjugate Gradient Descent
      3. Limited-Memory BFGS
    4. Results
  3. Results

Target Description: The type Fully connected layer.

Report Description: The type Basic.

Subreport: Logs for com.simiacryptus.ref.lang.ReferenceCountingBase

Test Modules

Using Seed 9004941787424001024

Training Characteristics

Input Learning

In this apply, we use a network to learn this target input, given it's pre-evaluated output:

TrainingTester.java:445 executed in 0.00 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(input_target)).flatMap(RefArrays::stream).map(x -> {
      try {
        return x.prettyPrint();
      } finally {
        x.freeRef();
      }
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [ -0.128, 0.08, 0.7 ]
    [ -0.128, 0.08, 0.7 ]
    [ 0.08, -0.128, 0.7 ]
    [ -0.128, 0.08, 0.7 ]
    [ 0.7, -0.128, 0.08 ]

Gradient Descent

First, we train using basic gradient descent method apply weak line search conditions.

TrainingTester.java:638 executed in 0.16 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging
Reset training subject: 4826348977188
BACKPROP_AGG_SIZE = 3
THREADS = 64
SINGLE_THREADED = false
Initialized CoreSettings = {
"backpropAggregationSize" : 3,
"jvmThreads" : 64,
"singleThreaded" : false
}
Reset training subject: 4826378699465
Constructing line search parameters: GD
th(0)=21.212453101200225;dx=-9.482689850778085E22
New Minimum: 21.212453101200225 > 0.0
Armijo: th(2.154434690031884)=0.0; dx=-1.0356715730815775E11 evalInputDelta=21.212453101200225
Armijo: th(1.077217345015942)=0.0; dx=-1.0356715730815775E11 evalInputDelta=21.212453101200225
Armijo: th(0.3590724483386473)=0.0; dx=-1.0356715730815775E11 evalInputDelta=21.212453101200225
Armijo: th(0.08976811208466183)=0.0; dx=-1.0356715730815775E11 evalInputDelta=21.212453101200225
Armijo: th(0.017953622416932366)=0.0; dx=-1.0356715730815775E11 evalInputDelta=21.212453101200225
Armijo: th(0.002992270402822061)=0.0; dx=-1.0356715730815775E11 evalInputDelta=21.212453101200225
Armijo: th(4.2746720040315154E-4)=0.0; dx=-1.0356715730815775E11 evalInputDelta=21.212453101200225
Armijo: th(5.343340005039394E-5)=0.0; dx=-1.0356715730815775E11 evalInputDelta=21.212453101200225
Armijo: th(5.9370444500437714E-6)=0.0; dx=-1.0356715730815775E11 evalInputDelta=21.212453101200225
Armijo: th(5.937044450043771E-7)=0.0; dx=-1.0356715730815775E11 evalInputDelta=21.212453101200225
Armijo: th(5.397313136403428E-8)=0.0; dx=-1.0356715730815775E11 evalInputDelta=21.212453101200225
Armijo: th(4.4977609470028565E-9)=0.0; dx=-1.0356715730815775E11 evalInputDelta=21.212453101200225
Armijo: th(3.4598161130791205E-10)=0.0; dx=-1.0356715730815775E11 evalInputDelta=21.212453101200225
Armijo: th(2.4712972236279432E-11)=0.07279807892788477; dx=-1.0609591610374141E11 evalInputDelta=21.13965502227234
Armijo: th(1.6475314824186289E-12)=15.420534038354342; dx=-6.514021661956986E22 evalInputDelta=5.791919062845883
Armijo: th(1.029707176511643E-13)=18.570149419829598; dx=-7.145885262827089E22 evalInputDelta=2.6423036813706275
Armijo: th(6.057101038303783E-15)=21.210575784224744; dx=-9.482689850777728E22 evalInputDelta=0.0018773169754808805
MIN ALPHA (3.3650561323909904E-16): th(2.154434690031884)=0.0
Fitness changed from 21.212453101200225 to 0.0
Iteration 1 complete. Error: 0.0 Total: 0.1382; Orientation: 0.0041; Line Search: 0.0928
th(0)=0.0;dx=-0.11779169822267192
Armijo: th(2.154434690031884E-15)=0.0; dx=-0.11779169822267192 evalInputDelta=0.0
Armijo: th(1.077217345015942E-15)=0.0; dx=-0.11779169822267192 evalInputDelta=0.0
MIN ALPHA (3.5907244833864734E-16): th(0.0)=0.0
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0164; Orientation: 0.0010; Line Search: 0.0118
Iteration 2 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 2
Final threshold in iteration 2: 0.0 (> 0.0) after 0.155s (< 30.000s)

Returns

    0.0

Training Converged

Conjugate Gradient Descent

First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.

TrainingTester.java:603 executed in 0.13 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging
Reset training subject: 4826509320630
Reset training subject: 4826511530657
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=21.212453101200225}, derivative=-9.482689850778085E22}
New Minimum: 21.212453101200225 > 0.0
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-1.0356715730815775E11}, evalInputDelta = -21.212453101200225
F(7.000000000000001E-10) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-1.0356715730815775E11}, evalInputDelta = -21.212453101200225
F(4.900000000000001E-9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-1.0356715730815775E11}, evalInputDelta = -21.212453101200225
F(3.430000000000001E-8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-1.0356715730815775E11}, evalInputDelta = -21.212453101200225
F(2.4010000000000004E-7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-1.0356715730815775E11}, evalInputDelta = -21.212453101200225
F(1.6807000000000003E-6) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-1.0356715730815775E11}, evalInputDelta = -21.212453101200225
F(1.1764900000000001E-5) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-1.0356715730815775E11}, evalInputDelta = -21.212453101200225
F(8.235430000000001E-5) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-1.0356715730815775E11}, evalInputDelta = -21.212453101200225
F(5.764801000000001E-4) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-1.0356715730815775E11}, evalInputDelta = -21.212453101200225
F(0.004035360700000001) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-1.0356715730815775E11}, evalInputDelta = -21.212453101200225
F(0.028247524900000005) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-1.0356715730815775E11}, evalInputDelta = -21.212453101200225
F(0.19773267430000002) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-1.0356715730815775E11}, evalInputDelta = -21.212453101200225
F(1.3841287201) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-1.0356715730815775E11}, evalInputDelta = -21.212453101200225
F(9.688901040700001) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-1.0356715730815775E11}, evalInputDelta = -21.212453101200225
F(67.8223072849) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-1.0356715730815775E11}, evalInputDelta = -21.212453101200225
F(474.7561509943) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-1.0356715730815775E11}, evalInputDelta = -21.212453101200225
F(3323.2930569601003) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-1.0356715730815775E11}, evalInputDelta = -21.212453101200225
F(23263.0513987207) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-1.0356715730815775E11}, evalInputDelta = -21.212453101200225
F(162841.3597910449) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-1.0356715730815775E11}, evalInputDelta = -21.212453101200225
F(1139889.5185373144) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-1.0356715730815775E11}, evalInputDelta = -21.212453101200225
F(7979226.6297612) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-1.0356715730815775E11}, evalInputDelta = -21.212453101200225
F(5.58545864083284E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-1.0356715730815775E11}, evalInputDelta = -21.212453101200225
F(3.909821048582988E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-1.0356715730815775E11}, evalInputDelta = -21.212453101200225
F(2.7368747340080914E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-1.0356715730815775E11}, evalInputDelta = -21.212453101200225
F(1.915812313805664E10) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-1.0356715730815775E11}, evalInputDelta = -21.212453101200225
0.0 <= 21.212453101200225
F(1.0E10) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-1.0356715730815775E11}, evalInputDelta = -21.212453101200225
Right bracket at 1.0E10
Converged to right
Fitness changed from 21.212453101200225 to 0.0
Iteration 1 complete. Error: 0.0 Total: 0.0800; Orientation: 0.0011; Line Search: 0.0726
F(0.0) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.11779169822267192}
F(1.0E10) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.11779169822267192}, evalInputDelta = 0.0
0.0 <= 0.0
F(5.0E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.11779169822267194}, evalInputDelta = 0.0
Right bracket at 5.0E9
F(1.0E10) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.11779169822267192}, evalInputDelta = 0.0
Right bracket at 1.0E10
F(5.0E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.11779169822267192}, evalInputDelta = 0.0
Right bracket at 5.0E9
F(2.5E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.11779169822267194}, evalInputDelta = 0.0
Right bracket at 2.5E9
F(5.0E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.11779169822267192}, evalInputDelta = 0.0
Right bracket at 5.0E9
F(2.5E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.11779169822267192}, evalInputDelta = 0.0
Right bracket at 2.5E9
F(1.25E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.11779169822267192}, evalInputDelta = 0.0
Right bracket at 1.25E9
F(6.25E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.11779169822267192}, evalInputDelta = 0.0
Right bracket at 6.25E8
F(3.125E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.11779169822267192}, evalInputDelta = 0.0
Right bracket at 3.125E8
F(1.5625E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.11779169822267192}, evalInputDelta = 0.0
Right bracket at 1.5625E8
F(7.8125E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.11779169822267192}, evalInputDelta = 0.0
Right bracket at 7.8125E7
F(3.90625E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.11779169822267194}, evalInputDelta = 0.0
Loops = 12
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0514; Orientation: 0.0006; Line Search: 0.0492
Iteration 2 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 2
Final threshold in iteration 2: 0.0 (> 0.0) after 0.132s (< 30.000s)

Returns

    0.0

Training Converged

Limited-Memory BFGS

Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.

TrainingTester.java:674 executed in 0.07 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new LBFGS());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setIterationsPerSample(100);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging
Reset training subject: 4826645943849
Reset training subject: 4826649103874
Adding measurement 367dd3f2 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 21.212453101200225 < 21.212453101200225. Total: 1
th(0)=21.212453101200225;dx=-9.482689850778085E22
Adding measurement 4dbf1656 to history. Total: 1
New Minimum: 21.212453101200225 > 0.0
Armijo: th(2.154434690031884)=0.0; dx=-1.0356715730815775E11 evalInputDelta=21.212453101200225
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(1.077217345015942)=0.0; dx=-1.0356715730815775E11 evalInputDelta=21.212453101200225
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(0.3590724483386473)=0.0; dx=-1.0356715730815775E11 evalInputDelta=21.212453101200225
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(0.08976811208466183)=0.0; dx=-1.0356715730815775E11 evalInputDelta=21.212453101200225
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(0.017953622416932366)=0.0; dx=-1.0356715730815775E11 evalInputDelta=21.212453101200225
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(0.002992270402822061)=0.0; dx=-1.0356715730815775E11 evalInputDelta=21.212453101200225
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(4.2746720040315154E-4)=0.0; dx=-1.0356715730815775E11 evalInputDelta=21.212453101200225
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(5.343340005039394E-5)=0.0; dx=-1.0356715730815775E11 evalInputDelta=21.212453101200225
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(5.9370444500437714E-6)=0.0; dx=-1.0356715730815775E11 evalInputDelta=21.212453101200225
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(5.937044450043771E-7)=0.0; dx=-1.0356715730815775E11 evalInputDelta=21.212453101200225
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(5.397313136403428E-8)=0.0; dx=-1.0356715730815775E11 evalInputDelta=21.212453101200225
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(4.4977609470028565E-9)=0.0; dx=-1.0356715730815775E11 evalInputDelta=21.212453101200225
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(3.4598161130791205E-10)=0.0; dx=-1.0356715730815775E11 evalInputDelta=21.212453101200225
Non-optimal measurement 0.07279807892788477 < 0.0. Total: 2
Armijo: th(2.4712972236279432E-11)=0.07279807892788477; dx=-1.0609591610374141E11 evalInputDelta=21.13965502227234
Non-optimal measurement 15.420534038354342 < 0.0. Total: 2
Armijo: th(1.6475314824186289E-12)=15.420534038354342; dx=-6.514021661956986E22 evalInputDelta=5.791919062845883
Non-optimal measurement 18.570149419829598 < 0.0. Total: 2
Armijo: th(1.029707176511643E-13)=18.570149419829598; dx=-7.145885262827089E22 evalInputDelta=2.6423036813706275
Non-optimal measurement 21.210575784224744 < 0.0. Total: 2
Armijo: th(6.057101038303783E-15)=21.210575784224744; dx=-9.482689850777728E22 evalInputDelta=0.0018773169754808805
Non-optimal measurement 0.0 < 0.0. Total: 2
MIN ALPHA (3.3650561323909904E-16): th(2.154434690031884)=0.0
Fitness changed from 21.212453101200225 to 0.0
Iteration 1 complete. Error: 0.0 Total: 0.0537; Orientation: 0.0035; Line Search: 0.0442
Non-optimal measurement 0.0 < 0.0. Total: 2
LBFGS Accumulation History: 2 points
Non-optimal measurement 0.0 < 0.0. Total: 2
th(0)=0.0;dx=-0.11779169822267194
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(2.154434690031884E-15)=0.0; dx=-0.11779169822267192 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(1.077217345015942E-15)=0.0; dx=-0.11779169822267194 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 2
MIN ALPHA (3.5907244833864734E-16): th(0.0)=0.0
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0107; Orientation: 0.0012; Line Search: 0.0081
Iteration 2 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 2
Final threshold in iteration 2: 0.0 (> 0.0) after 0.065s (< 30.000s)

Returns

    0.0

Training Converged

TrainingTester.java:576 executed in 0.10 seconds (0.000 gc):

    return TestUtil.compare(title + " vs Iteration", runs);
Logging
Plotting range=[0.0, 0.0], [2.0, 1.0]; valueStats=DoubleSummaryStatistics{count=0, sum=0.000000, min=Infinity, average=0.000000, max=-Infinity}
Only 0 points for GD
Only 0 points for CjGD
Only 0 points for LBFGS

Returns

Result

TrainingTester.java:579 executed in 0.00 seconds (0.000 gc):

    return TestUtil.compareTime(title + " vs Time", runs);
Logging
No Data

Model Learning

In this apply, attempt to train a network to emulate a randomized network given an example input/output. The target state is:

TrainingTester.java:492 executed in 0.00 seconds (0.000 gc):

    RefList<double[]> temp_18_0042 = network_target.state();
    assert temp_18_0042 != null;
    String temp_18_0041 = temp_18_0042.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b).orElse("");
    temp_18_0042.freeRef();
    return temp_18_0041;

Returns

    [-0.22524476968105545, 0.5279864458835533, -0.26574667282054454, -0.8935941107689356, 0.7019740669397789, 0.15051032279415077, -0.11399296020030149, -0.16447922321118172, 0.3424216801112634]

Gradient Descent

First, we train using basic gradient descent method apply weak line search conditions.

TrainingTester.java:638 executed in 0.17 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging
Reset training subject: 4826933974230
Reset training subject: 4826936159899
Constructing line search parameters: GD
th(0)=2.3654551514571365;dx=-3.2921469769014334E21
New Minimum: 2.3654551514571365 > 0.06859908780054494
Armijo: th(2.154434690031884)=0.06859908780054494; dx=-3.2921469769280314E9 evalInputDelta=2.2968560636565916
Armijo: th(1.077217345015942)=0.1019207765866494; dx=-3.2921469769378014E9 evalInputDelta=2.263534374870487
Armijo: th(0.3590724483386473)=0.13208998882526982; dx=-3.292146976950445E9 evalInputDelta=2.2333651626318667
Armijo: th(0.08976811208466183)=0.14626869908826132; dx=-3.2921469769581237E9 evalInputDelta=2.2191864523688754
Armijo: th(0.017953622416932366)=0.1504286527932885; dx=-3.2921469769606357E9 evalInputDelta=2.215026498663848
Armijo: th(0.002992270402822061)=0.15131852452007594; dx=-3.2921469769611897E9 evalInputDelta=2.2141366269370604
Armijo: th(4.2746720040315154E-4)=0.1514719098762122; dx=-3.2921469769612856E9 evalInputDelta=2.2139832415809244
Armijo: th(5.343340005039394E-5)=0.15149429918003668; dx=-3.2921469769612994E9 evalInputDelta=2.2139608522770997
Armijo: th(5.9370444500437714E-6)=0.15149714264205813; dx=-3.292146976961302E9 evalInputDelta=2.2139580088150783
Armijo: th(5.937044450043771E-7)=0.15149746253683644; dx=-3.292146976961302E9 evalInputDelta=2.2139576889203
Armijo: th(5.397313136403428E-8)=0.15149749484949993; dx=-3.292146976961302E9 evalInputDelta=2.2139576566076364
Armijo: th(4.4977609470028565E-9)=0.15149749781149463; dx=-3.292146976961302E9 evalInputDelta=2.213957653645642
Armijo: th(3.4598161130791205E-10)=0.1514974980600536; dx=-3.292146976961302E9 evalInputDelta=2.213957653397083
Armijo: th(2.4712972236279432E-11)=0.21898279515749572; dx=-7.642935244976841E9 evalInputDelta=2.1464723562996406
Armijo: th(1.6475314824186289E-12)=2.3654551514570383; dx=-3.2921469769014334E21 evalInputDelta=9.814371537686384E-14
Armijo: th(1.029707176511643E-13)=2.3654551514571307; dx=-3.2921469769014334E21 evalInputDelta=5.773159728050814E-15
Armijo: th(6.057101038303783E-15)=2.3654551514571365; dx=-3.2921469769014334E21 evalInputDelta=0.0
MIN ALPHA (3.3650561323909904E-16): th(2.154434690031884)=0.06859908780054494
Fitness changed from 2.3654551514571365 to 0.06859908780054494
Iteration 1 complete. Error: 0.06859908780054494 Total: 0.0421; Orientation: 0.0004; Line Search: 0.0358
th(0)=0.06859908780054494;dx=-0.0159448707256631
WOLFE (weak): th(2.154434690031884E-15)=0.06859908780054494; dx=-0.015944870725663095 evalInputDelta=0.0
New Minimum: 0.06859908780054494 > 0.06859908780054488
WOLFE (weak): th(4.308869380063768E-15)=0.06859908780054488; dx=-0.015944870725663095 evalInputDelta=5.551115123125783E-17
New Minimum: 0.06859908780054488 > 0.06859908780054477
WOLFE (weak): th(1.2926608140191303E-14)=0.06859908780054477; dx=-0.01594487072566308 evalInputDelta=1.6653345369377348E-16
New Minimum: 0.06859908780054477 > 0.06859908780054429
WOLFE (weak): th(5.1706432560765214E-14)=0.06859908780054429; dx=-0.01594487072566303 evalInputDelta=6.522560269672795E-16
New Minimum: 0.06859908780054429 > 0.06859908780054166
WOLFE (weak): th(2.5853216280382605E-13)=0.06859908780054166; dx=-0.015944870725662766 evalInputDelta=3.2751579226442118E-15
New Minimum: 0.06859908780054166 > 0.0685990878005253
WOLFE (weak): th(1.5511929768229563E-12)=0.0685990878005253; dx=-0.015944870725661104 evalInputDelta=1.9637069748057456E-14
New Minimum: 0.0685990878005253 > 0.06859908780040756
WOLFE (weak): th(1.0858350837760695E-11)=0.06859908780040756; dx=-0.015944870725649138 evalInputDelta=1.373762215095553E-13
New Minimum: 0.06859908780040756 > 0.06859908779944586
WOLFE (weak): th(8.686680670208556E-11)=0.06859908779944586; dx=-0.015944870725551397 evalInputDelta=1.0990791610154815E-12
New Minimum: 0.06859908779944586 > 0.06859908779065303
WOLFE (weak): th(7.8180126031877E-10)=0.06859908779065303; dx=-0.0159448707246578 evalInputDelta=9.891906738168643E-12
New Minimum: 0.06859908779065303 > 0.06859908770162582
WOLFE (weak): th(7.818012603187701E-9)=0.06859908770162582; dx=-0.015944870715610085 evalInputDelta=9.891912289283766E-11
New Minimum: 0.06859908770162582 > 0.06859908671243427
WOLFE (weak): th(8.599813863506471E-8)=0.06859908671243427; dx=-0.01594487061507991 evalInputDelta=1.0881106710103339E-9
New Minimum: 0.06859908671243427 > 0.06859907474321736
WOLFE (weak): th(1.0319776636207765E-6)=0.06859907474321736; dx=-0.015944869398664963 evalInputDelta=1.3057327580279221E-8
New Minimum: 0.06859907474321736 > 0.06859891805539287
WOLFE (weak): th(1.3415709627070094E-5)=0.06859891805539287; dx=-0.01594485347470947 evalInputDelta=1.697451520732418E-7
New Minimum: 0.06859891805539287 > 0.06859671138947591
WOLFE (weak): th(1.878199347789813E-4)=0.06859671138947591; dx=-0.015944629216680814 evalInputDelta=2.37641106902442E-6
New Minimum: 0.06859671138947591 > 0.06856344639643412
WOLFE (weak): th(0.0028172990216847197)=0.06856344639643412; dx=-0.01594124907859496 evalInputDelta=3.5641404110817176E-5
New Minimum: 0.06856344639643412 > 0.06803004604266708
WOLFE (weak): th(0.045076784346955515)=0.06803004604266708; dx=-0.01588717714697107 evalInputDelta=5.690417578778639E-4
New Minimum: 0.06803004604266708 > 0.05926192137665507
WOLFE (weak): th(0.7663053338982437)=0.05926192137665507; dx=-0.015031958779370651 evalInputDelta=0.009337166423889869
New Minimum: 0.05926192137665507 > 0.0
END: th(13.793496010168386)=0.0; dx=-0.010707613955474767 evalInputDelta=0.06859908780054494
Fitness changed from 0.06859908780054494 to 0.0
Iteration 2 complete. Error: 0.0 Total: 0.0444; Orientation: 0.0003; Line Search: 0.0422
th(0)=0.0;dx=-0.007640435231550076
Armijo: th(29.717186301123153)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Armijo: th(14.858593150561576)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Armijo: th(4.9528643835205255)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Armijo: th(1.2382160958801314)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Armijo: th(0.24764321917602627)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Armijo: th(0.04127386986267104)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Armijo: th(0.005896267123238721)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Armijo: th(7.370333904048401E-4)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Armijo: th(8.189259893387112E-5)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Armijo: th(8.189259893387112E-6)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Armijo: th(7.444781721261011E-7)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Armijo: th(6.20398476771751E-8)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Armijo: th(4.772295975167315E-9)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Armijo: th(3.408782839405225E-10)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Armijo: th(2.2725218929368167E-11)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Armijo: th(1.4203261830855104E-12)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Armijo: th(8.354859900503003E-14)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Armijo: th(4.641588833612779E-15)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
MIN ALPHA (2.442941491375147E-16): th(0.0)=0.0
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0794; Orientation: 0.0003; Line Search: 0.0777
Iteration 3 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 3
Final threshold in iteration 3: 0.0 (> 0.0) after 0.166s (< 30.000s)

Returns

    0.0

Training Converged

Conjugate Gradient Descent

First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.

TrainingTester.java:603 executed in 0.11 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging
Reset training subject: 4827103771845
Reset training subject: 4827105093011
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=2.3654551514571365}, derivative=-3.2921469769014334E21}
New Minimum: 2.3654551514571365 > 0.15149749807478005
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=0.15149749807478005}, derivative=-3.292146976961302E9}, evalInputDelta = -2.2139576533823564
New Minimum: 0.15149749807478005 > 0.15149749803885923
F(7.000000000000001E-10) = LineSearchPoint{point=PointSample{avg=0.15149749803885923}, derivative=-3.292146976961302E9}, evalInputDelta = -2.2139576534182774
New Minimum: 0.15149749803885923 > 0.15149749778741334
F(4.900000000000001E-9) = LineSearchPoint{point=PointSample{avg=0.15149749778741334}, derivative=-3.292146976961302E9}, evalInputDelta = -2.213957653669723
New Minimum: 0.15149749778741334 > 0.15149749602729223
F(3.430000000000001E-8) = LineSearchPoint{point=PointSample{avg=0.15149749602729223}, derivative=-3.292146976961302E9}, evalInputDelta = -2.2139576554298444
New Minimum: 0.15149749602729223 > 0.15149748370644525
F(2.4010000000000004E-7) = LineSearchPoint{point=PointSample{avg=0.15149748370644525}, derivative=-3.292146976961302E9}, evalInputDelta = -2.2139576677506914
New Minimum: 0.15149748370644525 > 0.15149739746056093
F(1.6807000000000003E-6) = LineSearchPoint{point=PointSample{avg=0.15149739746056093}, derivative=-3.292146976961302E9}, evalInputDelta = -2.2139577539965756
New Minimum: 0.15149739746056093 > 0.1514967937415528
F(1.1764900000000001E-5) = LineSearchPoint{point=PointSample{avg=0.1514967937415528}, derivative=-3.2921469769613013E9}, evalInputDelta = -2.2139583577155837
New Minimum: 0.1514967937415528 > 0.15149256781541137
F(8.235430000000001E-5) = LineSearchPoint{point=PointSample{avg=0.15149256781541137}, derivative=-3.2921469769612985E9}, evalInputDelta = -2.213962583641725
New Minimum: 0.15149256781541137 > 0.1514629915699175
F(5.764801000000001E-4) = LineSearchPoint{point=PointSample{avg=0.1514629915699175}, derivative=-3.2921469769612803E9}, evalInputDelta = -2.213992159887219
New Minimum: 0.1514629915699175 > 0.15125621402422232
F(0.004035360700000001) = LineSearchPoint{point=PointSample{avg=0.15125621402422232}, derivative=-3.2921469769611506E9}, evalInputDelta = -2.214198937432914
New Minimum: 0.15125621402422232 > 0.14982116660892258
F(0.028247524900000005) = LineSearchPoint{point=PointSample{avg=0.14982116660892258}, derivative=-3.292146976960261E9}, evalInputDelta = -2.215633984848214
New Minimum: 0.14982116660892258 > 0.14033428049599472
F(0.19773267430000002) = LineSearchPoint{point=PointSample{avg=0.14033428049599472}, derivative=-3.2921469769547505E9}, evalInputDelta = -2.225120870961142
New Minimum: 0.14033428049599472 > 0.09131666367671074
F(1.3841287201) = LineSearchPoint{point=PointSample{avg=0.09131666367671074}, derivative=-3.292146976934296E9}, evalInputDelta = -2.2741384877804256
New Minimum: 0.09131666367671074 > 0.0
F(9.688901040700001) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-3.2921469769171166E9}, evalInputDelta = -2.3654551514571365
F(67.8223072849) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-3.2921469769171166E9}, evalInputDelta = -2.3654551514571365
F(474.7561509943) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-3.2921469769171166E9}, evalInputDelta = -2.3654551514571365
F(3323.2930569601003) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-3.2921469769171166E9}, evalInputDelta = -2.3654551514571365
F(23263.0513987207) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-3.2921469769171166E9}, evalInputDelta = -2.3654551514571365
F(162841.3597910449) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-3.2921469769171166E9}, evalInputDelta = -2.3654551514571365
F(1139889.5185373144) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-3.2921469769171166E9}, evalInputDelta = -2.3654551514571365
F(7979226.6297612) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-3.2921469769171166E9}, evalInputDelta = -2.3654551514571365
F(5.58545864083284E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-3.2921469769171166E9}, evalInputDelta = -2.3654551514571365
F(3.909821048582988E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-3.2921469769171166E9}, evalInputDelta = -2.3654551514571365
F(2.7368747340080914E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-3.2921469769171166E9}, evalInputDelta = -2.3654551514571365
F(1.915812313805664E10) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-3.2921469769171166E9}, evalInputDelta = -2.3654551514571365
0.0 <= 2.3654551514571365
F(1.0E10) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-3.2921469769171166E9}, evalInputDelta = -2.3654551514571365
Right bracket at 1.0E10
Converged to right
Fitness changed from 2.3654551514571365 to 0.0
Iteration 1 complete. Error: 0.0 Total: 0.0591; Orientation: 0.0003; Line Search: 0.0547
F(0.0) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.007640435231550076}
F(1.0E10) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.007640435231550076}, evalInputDelta = 0.0
0.0 <= 0.0
F(5.0E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.007640435231550076}, evalInputDelta = 0.0
Right bracket at 5.0E9
F(2.5E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.007640435231550076}, evalInputDelta = 0.0
Right bracket at 2.5E9
F(1.25E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.007640435231550076}, evalInputDelta = 0.0
Right bracket at 1.25E9
F(6.25E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.007640435231550076}, evalInputDelta = 0.0
Right bracket at 6.25E8
F(3.125E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.007640435231550076}, evalInputDelta = 0.0
Right bracket at 3.125E8
F(1.5625E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.007640435231550076}, evalInputDelta = 0.0
Right bracket at 1.5625E8
F(7.8125E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.007640435231550076}, evalInputDelta = 0.0
Right bracket at 7.8125E7
F(3.90625E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.007640435231550076}, evalInputDelta = 0.0
Right bracket at 3.90625E7
F(1.953125E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.007640435231550076}, evalInputDelta = 0.0
Right bracket at 1.953125E7
F(9765625.0) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.007640435231550076}, evalInputDelta = 0.0
Right bracket at 9765625.0
F(4882812.5) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.007640435231550076}, evalInputDelta = 0.0
Right bracket at 4882812.5
F(2441406.25) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.007640435231550076}, evalInputDelta = 0.0
Loops = 12
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0509; Orientation: 0.0003; Line Search: 0.0493
Iteration 2 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 2
Final threshold in iteration 2: 0.0 (> 0.0) after 0.110s (< 30.000s)

Returns

    0.0

Training Converged

Limited-Memory BFGS

Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.

TrainingTester.java:674 executed in 0.48 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new LBFGS());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setIterationsPerSample(100);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging
Reset training subject: 4827218009891
Reset training subject: 4827219547442
Adding measurement 614b521b to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 2.3654551514571365 < 2.3654551514571365. Total: 1
th(0)=2.3654551514571365;dx=-3.2921469769014334E21
Adding measurement 3145d507 to history. Total: 1
New Minimum: 2.3654551514571365 > 0.06859908780054494
Armijo: th(2.154434690031884)=0.06859908780054494; dx=-3.2921469769280314E9 evalInputDelta=2.2968560636565916
Non-optimal measurement 0.1019207765866494 < 0.06859908780054494. Total: 2
Armijo: th(1.077217345015942)=0.1019207765866494; dx=-3.2921469769378014E9 evalInputDelta=2.263534374870487
Non-optimal measurement 0.13208998882526982 < 0.06859908780054494. Total: 2
Armijo: th(0.3590724483386473)=0.13208998882526982; dx=-3.292146976950445E9 evalInputDelta=2.2333651626318667
Non-optimal measurement 0.14626869908826132 < 0.06859908780054494. Total: 2
Armijo: th(0.08976811208466183)=0.14626869908826132; dx=-3.2921469769581237E9 evalInputDelta=2.2191864523688754
Non-optimal measurement 0.1504286527932885 < 0.06859908780054494. Total: 2
Armijo: th(0.017953622416932366)=0.1504286527932885; dx=-3.2921469769606357E9 evalInputDelta=2.215026498663848
Non-optimal measurement 0.15131852452007594 < 0.06859908780054494. Total: 2
Armijo: th(0.002992270402822061)=0.15131852452007594; dx=-3.2921469769611897E9 evalInputDelta=2.2141366269370604
Non-optimal measurement 0.1514719098762122 < 0.06859908780054494. Total: 2
Armijo: th(4.2746720040315154E-4)=0.1514719098762122; dx=-3.2921469769612856E9 evalInputDelta=2.2139832415809244
Non-optimal measurement 0.15149429918003668 < 0.06859908780054494. Total: 2
Armijo: th(5.343340005039394E-5)=0.15149429918003668; dx=-3.2921469769612994E9 evalInputDelta=2.2139608522770997
Non-optimal measurement 0.15149714264205813 < 0.06859908780054494. Total: 2
Armijo: th(5.9370444500437714E-6)=0.15149714264205813; dx=-3.292146976961302E9 evalInputDelta=2.2139580088150783
Non-optimal measurement 0.15149746253683644 < 0.06859908780054494. Total: 2
Armijo: th(5.937044450043771E-7)=0.15149746253683644; dx=-3.292146976961302E9 evalInputDelta=2.2139576889203
Non-optimal measurement 0.15149749484949993 < 0.06859908780054494. Total: 2
Armijo: th(5.397313136403428E-8)=0.15149749484949993; dx=-3.292146976961302E9 evalInputDelta=2.2139576566076364
Non-optimal measurement 0.15149749781149463 < 0.06859908780054494. Total: 2
Armijo: th(4.4977609470028565E-9)=0.15149749781149463; dx=-3.292146976961302E9 evalInputDelta=2.213957653645642
Non-optimal measurement 0.1514974980600536 < 0.06859908780054494. Total: 2
Armijo: th(3.4598161130791205E-10)=0.1514974980600536; dx=-3.292146976961302E9 evalInputDelta=2.213957653397083
Non-optimal measurement 0.21898279515749572 < 0.06859908780054494. Total: 2
Armijo: th(2.4712972236279432E-11)=0.21898279515749572; dx=-7.642935244976841E9 evalInputDelta=2.1464723562996406
Non-optimal measurement 2.3654551514570383 < 0.06859908780054494. Total: 2
Armijo: th(1.6475314824186289E-12)=2.3654551514570383; dx=-3.2921469769014334E21 evalInputDelta=9.814371537686384E-14
Non-optimal measurement 2.3654551514571307 < 0.06859908780054494. Total: 2
Armijo: th(1.029707176511643E-13)=2.3654551514571307; dx=-3.2921469769014334E21 evalInputDelta=5.773159728050814E-15
Non-optimal measurement 2.3654551514571365 < 0.06859908780054494. Total: 2
Armijo: th(6.057101038303783E-15)=2.3654551514571365; dx=-3.2921469769014334E21 evalInputDelta=0.0
Non-optimal measurement 0.06859908780054494 < 0.06859908780054494. Total: 2
MIN ALPHA (3.3650561323909904E-16): th(2.154434690031884)=0.06859908780054494
Fitness changed from 2.3654551514571365 to 0.06859908780054494
Iteration 1 complete. Error: 0.06859908780054494 Total: 0.0307; Orientation: 0.0006; Line Search: 0.0268
Non-optimal measurement 0.06859908780054494 < 0.06859908780054494. Total: 2
LBFGS Accumulation History: 2 points
Non-optimal measurement 0.06859908780054494 < 0.06859908780054494. Total: 2
th(0)=0.06859908780054494;dx=-0.0159448707256631
Non-optimal measurement 0.06859908780054494 < 0.06859908780054494. Total: 2
WOLFE (weak): th(2.154434690031884E-15)=0.06859908780054494; dx=-0.015944870725663095 evalInputDelta=0.0
Adding measurement 1e65b817 to history. Total: 2
New Minimum: 0.06859908780054494 > 0.06859908780054488
WOLFE (weak): th(4.308869380063768E-15)=0.06859908780054488; dx=-0.015944870725663095 evalInputDelta=5.551115123125783E-17
Adding measurement 7b3f6c1f to history. Total: 3
New Minimum: 0.06859908780054488 > 0.06859908780054477
WOLFE (weak): th(1.2926608140191303E-14)=0.06859908780054477; dx=-0.01594487072566308 evalInputDelta=1.6653345369377348E-16
Adding measurement 2b413551 to history. Total: 4
New Minimum: 0.06859908780054477 > 0.06859908780054429
WOLFE (weak): th(5.1706432560765214E-14)=0.06859908780054429; dx=-0.01594487072566303 evalInputDelta=6.522560269672795E-16
Adding measurement 4c3ae5fd to history. Total: 5
New Minimum: 0.06859908780054429 > 0.06859908780054166
WOLFE (weak): th(2.5853216280382605E-13)=0.06859908780054166; dx=-0.015944870725662766 evalInputDelta=3.2751579226442118E-15
Adding measurement 7f554ff0 to history. Total: 6
New Minimum: 0.06859908780054166 > 0.0685990878005253
WOLFE (weak): th(1.5511929768229563E-12)=0.0685990878005253; dx=-0.015944870725661104 evalInputDelta=1.9637069748057456E-14
Adding measurement 233eb911 to history. Total: 7
New Minimum: 0.0685990878005253 > 0.06859908780040756
WOLFE (weak): th(1.0858350837760695E-11)=0.06859908780040756; dx=-0.015944870725649138 evalInputDelta=1.373762215095553E-13
Adding measurement 4f497ac5 to history. Total: 8
New Minimum: 0.06859908780040756 > 0.06859908779944586
WOLFE (weak): th(8.686680670208556E-11)=0.06859908779944586; dx=-0.015944870725551397 evalInputDelta=1.0990791610154815E-12
Adding measurement 2513e64b to history. Total: 9
New Minimum: 0.06859908779944586 > 0.06859908779065303
WOLFE (weak): th(7.8180126031877E-10)=0.06859908779065303; dx=-0.0159448707246578 evalInputDelta=9.891906738168643E-12
Adding measurement 23370281 to history. Total: 10
New Minimum: 0.06859908779065303 > 0.06859908770162582
WOLFE (weak): th(7.818012603187701E-9)=0.06859908770162582; dx=-0.015944870715610085 evalInputDelta=9.891912289283766E-11
Adding measurement 4921678b to history. Total: 11
New Minimum: 0.06859908770162582 > 0.06859908671243427
WOLFE (weak): th(8.599813863506471E-8)=0.06859908671243427; dx=-0.01594487061507991 evalInputDelta=1.0881106710103339E-9
Adding measurement 148e8b42 to history. Total: 12
New Minimum: 0.06859908671243427 > 0.06859907474321736
WOLFE (weak): th(1.0319776636207765E-6)=0.06859907474321736; dx=-0.015944869398664963 evalInputDelta=1.3057327580279221E-8
Adding measurement 79f4b22e to history. Total: 13
New Minimum: 0.06859907474321736 > 0.06859891805539287
WOLFE (weak): th(1.3415709627070094E-5)=0.06859891805539287; dx=-0.01594485347470947 evalInputDelta=1.697451520732418E-7
Adding measurement 65c20943 to history. Total: 14
New Minimum: 0.06859891805539287 > 0.06859671138947591
WOLFE (weak): th(1.878199347789813E-4)=0.06859671138947591; dx=-0.015944629216680814 evalInputDelta=2.37641106902442E-6
Adding measurement 527b9e78 to history. Total: 15
New Minimum: 0.06859671138947591 > 0.06856344639643412
WOLFE (weak): th(0.0028172990216847197)=0.06856344639643412; dx=-0.01594124907859496 evalInputDelta=3.5641404110817176E-5
Adding measurement 31374f28 to history. Total: 16
New Minimum: 0.06856344639643412 > 0.06803004604266708
WOLFE (weak): th(0.045076784346955515)=0.06803004604266708; dx=-0.01588717714697107 evalInputDelta=5.690417578778639E-4
Adding measurement 3225affd to history. Total: 17
New Minimum: 0.06803004604266708 > 0.05926192137665507
WOLFE (weak): th(0.7663053338982437)=0.05926192137665507; dx=-0.015031958779370651 evalInputDelta=0.009337166423889869
Adding measurement 4e8bd35c to history. Total: 18
New Minimum: 0.05926192137665507 > 0.0
END: th(13.793496010168386)=0.0; dx=-0.010707613955474767 evalInputDelta=0.06859908780054494
Fitness changed from 0.06859908780054494 to 0.0
Iteration 2 complete. Error: 0.0 Total: 0.0302; Orientation: 0

...skipping 2047 bytes...

59908671243427, 0.06859908770162582, 0.06859908779065303, 0.06859908779944586, 0.06859908780040756, 0.0685990878005253, 0.06859908780054166, 0.06859908780054429, 0.06859908780054477
Rejected: LBFGS Orientation magnitude: 1.320e+00, gradient 8.741e-02, dot -0.942; [2ee3025c-94a2-42f8-9275-afb828c72cf6 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 0.05926192137665507, 0.06803004604266708, 0.06856344639643412, 0.06859671138947591, 0.06859891805539287, 0.06859907474321736, 0.06859908671243427, 0.06859908770162582, 0.06859908779065303, 0.06859908779944586, 0.06859908780040756, 0.0685990878005253, 0.06859908780054166, 0.06859908780054429
Rejected: LBFGS Orientation magnitude: 1.321e+00, gradient 8.741e-02, dot -0.942; [2ee3025c-94a2-42f8-9275-afb828c72cf6 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 0.05926192137665507, 0.06803004604266708, 0.06856344639643412, 0.06859671138947591, 0.06859891805539287, 0.06859907474321736, 0.06859908671243427, 0.06859908770162582, 0.06859908779065303, 0.06859908779944586, 0.06859908780040756, 0.0685990878005253, 0.06859908780054166
Rejected: LBFGS Orientation magnitude: 1.321e+00, gradient 8.741e-02, dot -0.942; [2ee3025c-94a2-42f8-9275-afb828c72cf6 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 0.05926192137665507, 0.06803004604266708, 0.06856344639643412, 0.06859671138947591, 0.06859891805539287, 0.06859907474321736, 0.06859908671243427, 0.06859908770162582, 0.06859908779065303, 0.06859908779944586, 0.06859908780040756, 0.0685990878005253
Rejected: LBFGS Orientation magnitude: 1.321e+00, gradient 8.741e-02, dot -0.942; [2ee3025c-94a2-42f8-9275-afb828c72cf6 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 0.05926192137665507, 0.06803004604266708, 0.06856344639643412, 0.06859671138947591, 0.06859891805539287, 0.06859907474321736, 0.06859908671243427, 0.06859908770162582, 0.06859908779065303, 0.06859908779944586, 0.06859908780040756
Rejected: LBFGS Orientation magnitude: 1.321e+00, gradient 8.741e-02, dot -0.942; [2ee3025c-94a2-42f8-9275-afb828c72cf6 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 0.05926192137665507, 0.06803004604266708, 0.06856344639643412, 0.06859671138947591, 0.06859891805539287, 0.06859907474321736, 0.06859908671243427, 0.06859908770162582, 0.06859908779065303, 0.06859908779944586
Rejected: LBFGS Orientation magnitude: 1.321e+00, gradient 8.741e-02, dot -0.942; [2ee3025c-94a2-42f8-9275-afb828c72cf6 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 0.05926192137665507, 0.06803004604266708, 0.06856344639643412, 0.06859671138947591, 0.06859891805539287, 0.06859907474321736, 0.06859908671243427, 0.06859908770162582, 0.06859908779065303
Rejected: LBFGS Orientation magnitude: 1.321e+00, gradient 8.741e-02, dot -0.942; [2ee3025c-94a2-42f8-9275-afb828c72cf6 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 0.05926192137665507, 0.06803004604266708, 0.06856344639643412, 0.06859671138947591, 0.06859891805539287, 0.06859907474321736, 0.06859908671243427, 0.06859908770162582
Rejected: LBFGS Orientation magnitude: 1.321e+00, gradient 8.741e-02, dot -0.942; [2ee3025c-94a2-42f8-9275-afb828c72cf6 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 0.05926192137665507, 0.06803004604266708, 0.06856344639643412, 0.06859671138947591, 0.06859891805539287, 0.06859907474321736, 0.06859908671243427
Rejected: LBFGS Orientation magnitude: 1.321e+00, gradient 8.741e-02, dot -0.942; [2ee3025c-94a2-42f8-9275-afb828c72cf6 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 0.05926192137665507, 0.06803004604266708, 0.06856344639643412, 0.06859671138947591, 0.06859891805539287, 0.06859907474321736
Rejected: LBFGS Orientation magnitude: 2.005e+00, gradient 8.741e-02, dot -0.963; [2ee3025c-94a2-42f8-9275-afb828c72cf6 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 0.05926192137665507, 0.06803004604266708, 0.06856344639643412, 0.06859671138947591, 0.06859891805539287
Rejected: LBFGS Orientation magnitude: 1.481e+00, gradient 8.741e-02, dot -0.981; [2ee3025c-94a2-42f8-9275-afb828c72cf6 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 0.05926192137665507, 0.06803004604266708, 0.06856344639643412, 0.06859671138947591
Rejected: LBFGS Orientation magnitude: 1.534e+00, gradient 8.741e-02, dot -0.980; [2ee3025c-94a2-42f8-9275-afb828c72cf6 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 0.05926192137665507, 0.06803004604266708, 0.06856344639643412
LBFGS Accumulation History: 3 points
Removed measurement 4e8bd35c to history. Total: 18
Removed measurement 3225affd to history. Total: 17
Removed measurement 31374f28 to history. Total: 16
Removed measurement 527b9e78 to history. Total: 15
Removed measurement 65c20943 to history. Total: 14
Removed measurement 79f4b22e to history. Total: 13
Removed measurement 148e8b42 to history. Total: 12
Removed measurement 4921678b to history. Total: 11
Removed measurement 23370281 to history. Total: 10
Removed measurement 2513e64b to history. Total: 9
Removed measurement 4f497ac5 to history. Total: 8
Removed measurement 233eb911 to history. Total: 7
Removed measurement 7f554ff0 to history. Total: 6
Removed measurement 4c3ae5fd to history. Total: 5
Removed measurement 2b413551 to history. Total: 4
Removed measurement 7b3f6c1f to history. Total: 3
Adding measurement 3f50b18c to history. Total: 3
th(0)=0.0;dx=-0.007640435231550076
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(29.717186301123153)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(14.858593150561576)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(4.9528643835205255)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(1.2382160958801314)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(0.24764321917602627)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(0.04127386986267104)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(0.005896267123238721)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(7.370333904048401E-4)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(8.189259893387112E-5)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(8.189259893387112E-6)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(7.444781721261011E-7)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(6.20398476771751E-8)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(4.772295975167315E-9)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(3.408782839405225E-10)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(2.2725218929368167E-11)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(1.4203261830855104E-12)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(8.354859900503003E-14)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(4.641588833612779E-15)=0.0; dx=-0.007640435231550076 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
MIN ALPHA (2.442941491375147E-16): th(0.0)=0.0
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.4223; Orientation: 0.3924; Line Search: 0.0290
Iteration 3 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 3
Final threshold in iteration 3: 0.0 (> 0.0) after 0.483s (< 30.000s)

Returns

    0.0

Training Converged

TrainingTester.java:576 executed in 0.01 seconds (0.000 gc):

    return TestUtil.compare(title + " vs Iteration", runs);
Logging
Plotting range=[1.0, -2.163681659305256], [2.0, -0.1636816593052559]; valueStats=DoubleSummaryStatistics{count=2, sum=0.137198, min=0.068599, average=0.068599, max=0.068599}
Plotting 2 points for GD
Only 1 points for CjGD
Plotting 2 points for LBFGS

Returns

Result

TrainingTester.java:579 executed in 0.01 seconds (0.000 gc):

    return TestUtil.compareTime(title + " vs Time", runs);
Logging
Plotting range=[0.0, -2.163681659305256], [0.044, -0.1636816593052559]; valueStats=DoubleSummaryStatistics{count=2, sum=0.137198, min=0.068599, average=0.068599, max=0.068599}
Plotting 2 points for GD
Only 1 points for CjGD
Plotting 2 points for LBFGS

Returns

Result

Composite Learning

In this apply, attempt to train a network to emulate a randomized network given an example input/output. The target state is:

TrainingTester.java:383 executed in 0.00 seconds (0.000 gc):

    RefList<double[]> temp_18_0037 = temp_18_0035.state();
    assert temp_18_0037 != null;
    String temp_18_0036 = temp_18_0037.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b).orElse("");
    temp_18_0037.freeRef();
    return temp_18_0036;

Returns

    [-0.26574667282054454, -0.11399296020030149, 0.5279864458835533, -0.8935941107689356, 0.7019740669397789, -0.16447922321118172, -0.22524476968105545, 0.15051032279415077, 0.3424216801112634]

We simultaneously regress this target input:

TrainingTester.java:391 executed in 0.00 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(input_target)).flatMap(x -> {
      RefStream<Tensor> temp_18_0006 = RefArrays.stream(RefUtil.addRef(x));
      if (null != x)
        RefUtil.freeRef(x);
      return temp_18_0006;
    }).map(x -> {
      String temp_18_0007 = x.prettyPrint();
      x.freeRef();
      return temp_18_0007;
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [ 0.08, -0.128, 0.7 ]
    [ 0.7, -0.128, 0.08 ]
    [ 0.08, -0.128, 0.7 ]
    [ -0.128, 0.08, 0.7 ]
    [ 0.7, 0.08, -0.128 ]

Which produces the following output:

TrainingTester.java:412 executed in 0.00 seconds (0.000 gc):

    return RefStream.of(RefUtil.addRef(output_target)).map(x -> {
      String temp_18_0008 = x.prettyPrint();
      x.freeRef();
      return temp_18_0008;
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [ 0.3629218771984823, -0.27647566567763376, 0.2024102731857486 ]
    [ -0.12919265639805833, -0.7285268959634411, -0.14954292568548902 ]
    [ 0.3629218771984823, -0.27647566567763376, 0.2024102731857486 ]
    [ 0.3944866494234928, 0.055402515285778886, 0.2805673324205915 ]
    [ -0.2627243728635001, -0.5483046116120414, -0.18946048800744847 ]

Gradient Descent

First, we train using basic gradient descent method apply weak line search conditions.

TrainingTester.java:638 executed in 2.07 seconds (0.199 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging
Reset training subject: 4827809487847
Reset training subject: 4827810919092
Constructing line search parameters: GD
th(0)=7.865970331891475;dx=-2.832708833538781E22
Armijo: th(2.154434690031884)=10.284933613204482; dx=6.923217023643399E33 evalInputDelta=-2.4189632813130064
Armijo: th(1.077217345015942)=10.284933613204482; dx=3.4616085118073013E33 evalInputDelta=-2.4189632813130064
Armijo: th(0.3590724483386473)=10.284933613204482; dx=1.1538695039165691E33 evalInputDelta=-2.4189632813130064
Armijo: th(0.08976811208466183)=10.284933613204482; dx=2.8846737595754438E32 evalInputDelta=-2.4189632813130064
Armijo: th(0.017953622416932366)=10.284933613204482; dx=5.769347516847115E31 evalInputDelta=-2.4189632813130064
Armijo: th(0.002992270402822061)=10.284933613204482; dx=9.615579170747554E30 evalInputDelta=-2.4189632813130064
Armijo: th(4.2746720040315154E-4)=10.284933613204482; dx=1.3736541425663657E30 evalInputDelta=-2.4189632813130064
Armijo: th(5.343340005039394E-5)=10.284933613204482; dx=1.7170674262327613E29 evalInputDelta=-2.4189632813130064
Armijo: th(5.9370444500437714E-6)=10.284933613204482; dx=1.907850136066158E28 evalInputDelta=-2.4189632813130064
Armijo: th(5.937044450043771E-7)=10.284933613204482; dx=1.9078242186174425E27 evalInputDelta=-2.4189632813130064
Armijo: th(5.397313136403428E-8)=10.284933613204482; dx=1.7341238608773157E26 evalInputDelta=-2.4189632813130064
Armijo: th(4.4977609470028565E-9)=10.284933613204482; dx=1.442463477250806E25 evalInputDelta=-2.4189632813130064
Armijo: th(3.4598161130791205E-10)=10.284933613204482; dx=1.0830052915102862E24 evalInputDelta=-2.4189632813130064
Armijo: th(2.4712972236279432E-11)=8.146067996599548; dx=4.990466972988744E22 evalInputDelta=-0.28009766470807307
New Minimum: 7.865970331891475 > 6.067806210809841
Armijo: th(1.6475314824186289E-12)=6.067806210809841; dx=-1.7420504750935052E22 evalInputDelta=1.7981641210816344
Armijo: th(1.029707176511643E-13)=7.865887857948195; dx=-2.807935825081462E22 evalInputDelta=8.24739432800925E-5
Armijo: th(6.057101038303783E-15)=7.865964979303593; dx=-2.8312515977471743E22 evalInputDelta=5.352587882256898E-6
MIN ALPHA (3.3650561323909904E-16): th(1.6475314824186289E-12)=6.067806210809841
Fitness changed from 7.865970331891475 to 6.067806210809841
Iteration 1 complete. Error: 6.067806210809841 Total: 0.0320; Orientation: 0.0004; Line Search: 0.0275
th(0)=6.067806210809841;dx=-1.3350768672429325E22
New Minimum: 6.067806210809841 > 6.0677495192645114
Armijo: th(2.154434690031884E-15)=6.0677495192645114; dx=-1.3349166119126846E22 evalInputDelta=5.6691545329279336E-5
Armijo: th(1.077217345015942E-15)=6.067777856504459; dx=-1.3349967395778086E22 evalInputDelta=2.835430538183914E-5
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.0677495192645114
Fitness changed from 6.067806210809841 to 6.0677495192645114
Iteration 2 complete. Error: 6.0677495192645114 Total: 0.0081; Orientation: 0.0003; Line Search: 0.0067
th(0)=6.0677495192645114;dx=-1.3347564032127261E22
New Minimum: 6.0677495192645114 > 6.067692894651524
Armijo: th(2.154434690031884E-15)=6.067692894651524; dx=-1.3345962411371866E22 evalInputDelta=5.662461298783228E-5
Armijo: th(1.077217345015942E-15)=6.067721198441342; dx=-1.3346763221749564E22 evalInputDelta=2.8320823169281084E-5
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.067692894651524
Fitness changed from 6.0677495192645114 to 6.067692894651524
Iteration 3 complete. Error: 6.067692894651524 Total: 0.0069; Orientation: 0.0003; Line Search: 0.0056
th(0)=6.067692894651524;dx=-1.3344361256801955E22
New Minimum: 6.067692894651524 > 6.067636336835265
Armijo: th(2.154434690031884E-15)=6.067636336835265; dx=-1.3342760568358858E22 evalInputDelta=5.655781625879541E-5
Armijo: th(1.077217345015942E-15)=6.067664607242714; dx=-1.3343560912580408E22 evalInputDelta=2.828740881000158E-5
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.067636336835265
Fitness changed from 6.067692894651524 to 6.067636336835265
Iteration 4 complete. Error: 6.067636336835265 Total: 0.0074; Orientation: 0.0003; Line Search: 0.0059
th(0)=6.067636336835265;dx=-1.3341160345983913E22
New Minimum: 6.067636336835265 > 6.067579845680507
Armijo: th(2.154434690031884E-15)=6.067579845680507; dx=-1.3339560589618477E22 evalInputDelta=5.649115475758748E-5
Armijo: th(1.077217345015942E-15)=6.067608082773152; dx=-1.3340360467801193E22 evalInputDelta=2.8254062113042266E-5
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.067579845680507
Fitness changed from 6.067636336835265 to 6.067579845680507
Iteration 5 complete. Error: 6.067579845680507 Total: 0.0073; Orientation: 0.0003; Line Search: 0.0059
th(0)=6.067579845680507;dx=-1.3337961299203928E22
New Minimum: 6.067579845680507 > 6.067523421052416
Armijo: th(2.154434690031884E-15)=6.067523421052416; dx=-1.333636247468166E22 evalInputDelta=5.642462809163362E-5
Armijo: th(1.077217345015942E-15)=6.067551624897627; dx=-1.3337161886942791E22 evalInputDelta=2.8220782880339357E-5
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.067523421052416
Fitness changed from 6.067579845680507 to 6.067523421052416
Iteration 6 complete. Error: 6.067523421052416 Total: 0.0069; Orientation: 0.0003; Line Search: 0.0055
th(0)=6.067523421052416;dx=-1.3334764115993083E22
New Minimum: 6.067523421052416 > 6.067467062816542
Armijo: th(2.154434690031884E-15)=6.067467062816542; dx=-1.3333166223079634E22 evalInputDelta=5.6358235873688045E-5
Armijo: th(1.077217345015942E-15)=6.067495233481497; dx=-1.3333965169536358E22 evalInputDelta=2.8187570918269955E-5
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.067467062816542
Fitness changed from 6.067523421052416 to 6.067467062816542
Iteration 7 complete. Error: 6.067467062816542 Total: 0.0082; Orientation: 0.0004; Line Search: 0.0065
th(0)=6.067467062816542;dx=-1.3331568795882757E22
New Minimum: 6.067467062816542 > 6.067410770838824
Armijo: th(2.154434690031884E-15)=6.067410770838824; dx=-1.332997183434392E22 evalInputDelta=5.6291977718281316E-5
Armijo: th(1.077217345015942E-15)=6.067438908390509; dx=-1.333077031511334E22 evalInputDelta=2.8154426033211166E-5
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.067410770838824
Fitness changed from 6.067467062816542 to 6.067410770838824
Iteration 8 complete. Error: 6.067410770838824 Total: 0.0086; Orientation: 0.0004; Line Search: 0.0069
th(0)=6.067410770838824;dx=-1.3328375338404604E22
New Minimum: 6.067410770838824 > 6.067354544985586
Armijo: th(2.154434690031884E-15)=6.067354544985586; dx=-1.332677930800632E22 evalInputDelta=5.6225853237279466E-5
Armijo: th(1.077217345015942E-15)=6.067382649490792; dx=-1.3327577323205464E22 evalInputDelta=2.8121348031540094E-5
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.067354544985586
Fitness changed from 6.067410770838824 to 6.067354544985586
Iteration 9 complete. Error: 6.067354544985586 Total: 0.0073; Orientation: 0.0003; Line Search: 0.0057
th(0)=6.067354544985586;dx=-1.332518374309058E22
New Minimum: 6.067354544985586 > 6.067298385123538
Armijo: th(2.154434690031884E-15)=6.067298385123538; dx=-1.3323588643598929E22 evalInputDelta=5.6159862048765774E-5
Armijo: th(1.077217345015942E-15)=6.067326456648862; dx=-1.3324386193344756E22 evalInputDelta=2.8088336724074736E-5
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.067298385123538
Fitness changed from 6.067354544985586 to 6.067298385123538
Iteration 10 complete. Error: 6.067298385123538 Total: 0.0072; Orientation: 0.0003; Line Search: 0.0059
th(0)=6.067298385123538;dx=-1.332199400947292E22
New Minimum: 6.067298385123538 > 6.067242291119765
Armijo: th(2.154434690031884E-15)=6.067242291119765; dx=-1.332039984065413E22 evalInputDelta=5.6094003772599876E-5
Armijo: th(1.077217345015942E-15)=6.06727032973162; dx=-1.3321196925063524E22 evalInputDelta=2.8055391917192196E-5
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.067242291119765
Fitness changed from 6.067298385123538 to 6.067242291119765
Iteration 11 complete. Error: 6.067242291119765 Total: 0.0086; Orientation: 0.0004; Line Search: 0.0069
th(0)=6.067242291119765;dx=-1.331880613708415E22
New Minimum

...skipping 129623 bytes...

al: 0.0064; Orientation: 0.0003; Line Search: 0.0050
th(0)=6.056075507371273;dx=-1.2647988520110334E22
New Minimum: 6.056075507371273 > 6.056031497657256
Armijo: th(2.154434690031884E-15)=6.056031497657256; dx=-1.264659896397921E22 evalInputDelta=4.4009714017079204E-5
Armijo: th(1.077217345015942E-15)=6.056053496758264; dx=-1.2647293742044773E22 evalInputDelta=2.2010613008660584E-5
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.056031497657256
Fitness changed from 6.056075507371273 to 6.056031497657256
Iteration 237 complete. Error: 6.056031497657256 Total: 0.0086; Orientation: 0.0003; Line Search: 0.0074
th(0)=6.056031497657256;dx=-1.2645209848507275E22
New Minimum: 6.056031497657256 > 6.055987531233459
Armijo: th(2.154434690031884E-15)=6.055987531233459; dx=-1.2643821173644066E22 evalInputDelta=4.396642379678184E-5
Armijo: th(1.077217345015942E-15)=6.056009508697943; dx=-1.2644515511075672E22 evalInputDelta=2.1988959312935208E-5
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.055987531233459
Fitness changed from 6.056031497657256 to 6.055987531233459
Iteration 238 complete. Error: 6.055987531233459 Total: 0.0055; Orientation: 0.0002; Line Search: 0.0046
th(0)=6.055987531233459;dx=-1.2642432939339125E22
New Minimum: 6.055987531233459 > 6.055943608026116
Armijo: th(2.154434690031884E-15)=6.055943608026116; dx=-1.2641045145542015E22 evalInputDelta=4.3923207343254944E-5
Armijo: th(1.077217345015942E-15)=6.055965563890938; dx=-1.264173904244057E22 evalInputDelta=2.196734252102317E-5
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.055943608026116
Fitness changed from 6.055987531233459 to 6.055943608026116
Iteration 239 complete. Error: 6.055943608026116 Total: 0.0052; Orientation: 0.0002; Line Search: 0.0041
th(0)=6.055943608026116;dx=-1.2639657792202324E22
New Minimum: 6.055943608026116 > 6.0558997279616325
Armijo: th(2.154434690031884E-15)=6.0558997279616325; dx=-1.2638270879269644E22 evalInputDelta=4.388006448330373E-5
Armijo: th(1.077217345015942E-15)=6.055921662263569; dx=-1.2638964335735984E22 evalInputDelta=2.1945762546771164E-5
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.0558997279616325
Fitness changed from 6.055943608026116 to 6.0558997279616325
Iteration 240 complete. Error: 6.0558997279616325 Total: 0.0053; Orientation: 0.0002; Line Search: 0.0043
th(0)=6.0558997279616325;dx=-1.2636884406693595E22
New Minimum: 6.0558997279616325 > 6.055855890966581
Armijo: th(2.154434690031884E-15)=6.055855890966581; dx=-1.2635498374423804E22 evalInputDelta=4.3836995051727E-5
Armijo: th(1.077217345015942E-15)=6.055877803742325; dx=-1.2636191390558698E22 evalInputDelta=2.1924219307578596E-5
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.055855890966581
Fitness changed from 6.0558997279616325 to 6.055855890966581
Iteration 241 complete. Error: 6.055855890966581 Total: 0.0051; Orientation: 0.0002; Line Search: 0.0042
th(0)=6.055855890966581;dx=-1.2634112782409923E22
New Minimum: 6.055855890966581 > 6.05581209696771
Armijo: th(2.154434690031884E-15)=6.05581209696771; dx=-1.2632727630601618E22 evalInputDelta=4.3793998870889084E-5
Armijo: th(1.077217345015942E-15)=6.055833988253868; dx=-1.263342020650577E22 evalInputDelta=2.190271271285127E-5
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.05581209696771
Fitness changed from 6.055855890966581 to 6.05581209696771
Iteration 242 complete. Error: 6.05581209696771 Total: 0.0054; Orientation: 0.0003; Line Search: 0.0043
th(0)=6.05581209696771;dx=-1.2631342918948574E22
New Minimum: 6.05581209696771 > 6.055768345891934
Armijo: th(2.154434690031884E-15)=6.055768345891934; dx=-1.2629958647400486E22 evalInputDelta=4.375107577558879E-5
Armijo: th(1.077217345015942E-15)=6.055790215725028; dx=-1.263065078317453E22 evalInputDelta=2.1881242681764945E-5
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.055768345891934
Fitness changed from 6.05581209696771 to 6.055768345891934
Iteration 243 complete. Error: 6.055768345891934 Total: 0.0060; Orientation: 0.0004; Line Search: 0.0045
th(0)=6.055768345891934;dx=-1.2628574815907081E22
New Minimum: 6.055768345891934 > 6.05572463766634
Armijo: th(2.154434690031884E-15)=6.05572463766634; dx=-1.2627191424418086E22 evalInputDelta=4.370822559440768E-5
Armijo: th(1.077217345015942E-15)=6.055746486082806; dx=-1.2627883120162582E22 evalInputDelta=2.185980912816632E-5
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.05572463766634
Fitness changed from 6.055768345891934 to 6.05572463766634
Iteration 244 complete. Error: 6.05572463766634 Total: 0.0057; Orientation: 0.0003; Line Search: 0.0046
th(0)=6.05572463766634;dx=-1.2625808472883253E22
New Minimum: 6.05572463766634 > 6.055680972218181
Armijo: th(2.154434690031884E-15)=6.055680972218181; dx=-1.2624425961252356E22 evalInputDelta=4.366544815859186E-5
Armijo: th(1.077217345015942E-15)=6.055702799254373; dx=-1.2625117217067807E22 evalInputDelta=2.183841196679026E-5
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.055680972218181
Fitness changed from 6.05572463766634 to 6.055680972218181
Iteration 245 complete. Error: 6.055680972218181 Total: 0.0050; Orientation: 0.0002; Line Search: 0.0041
th(0)=6.055680972218181;dx=-1.2623043889475176E22
New Minimum: 6.055680972218181 > 6.05563734947488
Armijo: th(2.154434690031884E-15)=6.05563734947488; dx=-1.2621662257501516E22 evalInputDelta=4.362274330116378E-5
Armijo: th(1.077217345015942E-15)=6.055659155167065; dx=-1.2622353073488347E22 evalInputDelta=2.1817051115924357E-5
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.05563734947488
Fitness changed from 6.055680972218181 to 6.05563734947488
Iteration 246 complete. Error: 6.05563734947488 Total: 0.0074; Orientation: 0.0002; Line Search: 0.0064
th(0)=6.05563734947488;dx=-1.2620281065281193E22
New Minimum: 6.05563734947488 > 6.055593769364027
Armijo: th(2.154434690031884E-15)=6.055593769364027; dx=-1.2618900312764048E22 evalInputDelta=4.358011085336955E-5
Armijo: th(1.077217345015942E-15)=6.0556155537483916; dx=-1.261959068902262E22 evalInputDelta=2.1795726488527123E-5
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.055593769364027
Fitness changed from 6.05563734947488 to 6.055593769364027
Iteration 247 complete. Error: 6.055593769364027 Total: 0.0072; Orientation: 0.0004; Line Search: 0.0059
th(0)=6.055593769364027;dx=-1.2617519999899933E22
New Minimum: 6.055593769364027 > 6.055550231813379
Armijo: th(2.154434690031884E-15)=6.055550231813379; dx=-1.2616140126638718E22 evalInputDelta=4.353755064734344E-5
Armijo: th(1.077217345015942E-15)=6.055571994926024; dx=-1.2616830063269328E22 evalInputDelta=2.1774438002886143E-5
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.055550231813379
Fitness changed from 6.055593769364027 to 6.055550231813379
Iteration 248 complete. Error: 6.055550231813379 Total: 0.0074; Orientation: 0.0004; Line Search: 0.0060
th(0)=6.055550231813379;dx=-1.2614760692930293E22
New Minimum: 6.055550231813379 > 6.055506736750862
Armijo: th(2.154434690031884E-15)=6.055506736750862; dx=-1.2613381698724563E22 evalInputDelta=4.3495062516996086E-5
Armijo: th(1.077217345015942E-15)=6.055528478627804; dx=-1.261407119582743E22 evalInputDelta=2.1753185575512646E-5
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.055506736750862
Fitness changed from 6.055550231813379 to 6.055506736750862
Iteration 249 complete. Error: 6.055506736750862 Total: 0.0054; Orientation: 0.0003; Line Search: 0.0041
th(0)=6.055506736750862;dx=-1.2612003143971445E22
New Minimum: 6.055506736750862 > 6.055463284104566
Armijo: th(2.154434690031884E-15)=6.055463284104566; dx=-1.261062502862088E22 evalInputDelta=4.345264629623813E-5
Armijo: th(1.077217345015942E-15)=6.055485004781741; dx=-1.2611314086296162E22 evalInputDelta=2.1731969121141503E-5
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.055463284104566
Fitness changed from 6.055506736750862 to 6.055463284104566
Iteration 250 complete. Error: 6.055463284104566 Total: 0.0049; Orientation: 0.0002; Line Search: 0.0040
Final threshold in iteration 251: 6.055463284104566 (> 0.0) after 2.073s (< 30.000s)

Returns

    6.055463284104566

This training apply resulted in the following configuration:

TrainingTester.java:785 executed in 0.00 seconds (0.000 gc):

    RefList<double[]> state = network.state();
    assert state != null;
    String description = state.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b)
        .orElse("");
    state.freeRef();
    return description;

Returns

    [-0.11621165874799193, 0.6943929766170008, -0.0750275918079723, -0.16447922321118752, 0.527986445883585, 0.1505103227941544, 0.32409469190084483, -0.14118717757200366, -0.7672289777352501]

And regressed input:

TrainingTester.java:797 executed in 0.00 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(data)).flatMap(x -> {
      return RefArrays.stream(x);
    }).limit(1).map(x -> {
      String temp_18_0015 = x.prettyPrint();
      x.freeRef();
      return temp_18_0015;
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [ -0.11172061029230794, 0.14561032921430386, 0.5916839677908636 ]

To produce the following output:

TrainingTester.java:808 executed in 0.00 seconds (0.000 gc):

    Result[] array = ConstantResult.batchResultArray(pop(RefUtil.addRef(data)));
    @Nullable
    Result eval = layer.eval(array);
    assert eval != null;
    TensorList tensorList = Result.getData(eval);
    String temp_18_0016 = tensorList.stream().limit(1).map(x -> {
      String temp_18_0017 = x.prettyPrint();
      x.freeRef();
      return temp_18_0017;
    }).reduce((a, b) -> a + "\n" + b).orElse("");
    tensorList.freeRef();
    return temp_18_0016;

Returns

    [ 0.06970140415297463, 0.1843105443876866, -0.5107234539292786 ]

Conjugate Gradient Descent

First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.

TrainingTester.java:603 executed in 0.00 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging
Reset training subject: 4829895888802
Reset training subject: 4829896779857
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=7.865970331891475}, derivative=-2.832708833538781E22}
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=10.284933613204482}, derivative=2.9255010305801942E23}, evalInputDelta = 2.4189632813130064
10.284933613204482 <= 7.865970331891475
Converged to right
Fitness changed from 7.865970331891475 to 7.865970331891475
Static Iteration Total: 0.0048; Orientation: 0.0002; Line Search: 0.0022
Iteration 1 failed. Error: 7.865970331891475
Previous Error: 0.0 -> 7.865970331891475
Optimization terminated 1
Final threshold in iteration 1: 7.865970331891475 (> 0.0) after 0.005s (< 30.000s)

Returns

    7.865970331891475

This training apply resulted in the following configuration:

TrainingTester.java:785 executed in 0.00 seconds (0.000 gc):

    RefList<double[]> state = network.state();
    assert state != null;
    String description = state.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b)
        .orElse("");
    state.freeRef();
    return description;

Returns

    [-0.11399296020030149, 0.7019740669397789, -0.26574667282054454, -0.16447922321118172, 0.5279864458835533, 0.15051032279415077, 0.3424216801112634, -0.22524476968105545, -0.8935941107689356]

And regressed input:

TrainingTester.java:797 executed in 0.00 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(data)).flatMap(x -> {
      return RefArrays.stream(x);
    }).limit(1).map(x -> {
      String temp_18_0015 = x.prettyPrint();
      x.freeRef();
      return temp_18_0015;
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [ -0.128, 0.08, 0.7 ]

To produce the following output:

TrainingTester.java:808 executed in 0.00 seconds (0.000 gc):

    Result[] array = ConstantResult.batchResultArray(pop(RefUtil.addRef(data)));
    @Nullable
    Result eval = layer.eval(array);
    assert eval != null;
    TensorList tensorList = Result.getData(eval);
    String temp_18_0016 = tensorList.stream().limit(1).map(x -> {
      String temp_18_0017 = x.prettyPrint();
      x.freeRef();
      return temp_18_0017;
    }).reduce((a, b) -> a + "\n" + b).orElse("");
    tensorList.freeRef();
    return temp_18_0016;

Returns

    [ -0.11527364671356025, 0.16864948219762105, -0.687365434166981 ]

Limited-Memory BFGS

Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.

TrainingTester.java:674 executed in 10.03 seconds (0.447 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new LBFGS());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setIterationsPerSample(100);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging
Reset training subject: 4829906061670
Reset training subject: 4829906971020
Adding measurement 1bea8178 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 7.865970331891475 < 7.865970331891475. Total: 1
th(0)=7.865970331891475;dx=-2.8327088335387814E22
Non-optimal measurement 10.284933613204482 < 7.865970331891475. Total: 1
Armijo: th(2.154434690031884)=10.284933613204482; dx=6.923217023643399E33 evalInputDelta=-2.4189632813130064
Non-optimal measurement 10.284933613204482 < 7.865970331891475. Total: 1
Armijo: th(1.077217345015942)=10.284933613204482; dx=3.4616085118073013E33 evalInputDelta=-2.4189632813130064
Non-optimal measurement 10.284933613204482 < 7.865970331891475. Total: 1
Armijo: th(0.3590724483386473)=10.284933613204482; dx=1.1538695039165691E33 evalInputDelta=-2.4189632813130064
Non-optimal measurement 10.284933613204482 < 7.865970331891475. Total: 1
Armijo: th(0.08976811208466183)=10.284933613204482; dx=2.8846737595754438E32 evalInputDelta=-2.4189632813130064
Non-optimal measurement 10.284933613204482 < 7.865970331891475. Total: 1
Armijo: th(0.017953622416932366)=10.284933613204482; dx=5.769347516847115E31 evalInputDelta=-2.4189632813130064
Non-optimal measurement 10.284933613204482 < 7.865970331891475. Total: 1
Armijo: th(0.002992270402822061)=10.284933613204482; dx=9.615579170747554E30 evalInputDelta=-2.4189632813130064
Non-optimal measurement 10.284933613204482 < 7.865970331891475. Total: 1
Armijo: th(4.2746720040315154E-4)=10.284933613204482; dx=1.3736541425663657E30 evalInputDelta=-2.4189632813130064
Non-optimal measurement 10.284933613204482 < 7.865970331891475. Total: 1
Armijo: th(5.343340005039394E-5)=10.284933613204482; dx=1.7170674262327613E29 evalInputDelta=-2.4189632813130064
Non-optimal measurement 10.284933613204482 < 7.865970331891475. Total: 1
Armijo: th(5.9370444500437714E-6)=10.284933613204482; dx=1.907850136066158E28 evalInputDelta=-2.4189632813130064
Non-optimal measurement 10.284933613204482 < 7.865970331891475. Total: 1
Armijo: th(5.937044450043771E-7)=10.284933613204482; dx=1.9078242186174425E27 evalInputDelta=-2.4189632813130064
Non-optimal measurement 10.284933613204482 < 7.865970331891475. Total: 1
Armijo: th(5.397313136403428E-8)=10.284933613204482; dx=1.7341238608773157E26 evalInputDelta=-2.4189632813130064
Non-optimal measurement 10.284933613204482 < 7.865970331891475. Total: 1
Armijo: th(4.4977609470028565E-9)=10.284933613204482; dx=1.442463477250806E25 evalInputDelta=-2.4189632813130064
Non-optimal measurement 10.284933613204482 < 7.865970331891475. Total: 1
Armijo: th(3.4598161130791205E-10)=10.284933613204482; dx=1.0830052915102862E24 evalInputDelta=-2.4189632813130064
Non-optimal measurement 8.146067996599548 < 7.865970331891475. Total: 1
Armijo: th(2.4712972236279432E-11)=8.146067996599548; dx=4.990466972988744E22 evalInputDelta=-0.28009766470807307
Adding measurement 9a535ad to history. Total: 1
New Minimum: 7.865970331891475 > 6.067806210809841
Armijo: th(1.6475314824186289E-12)=6.067806210809841; dx=-1.7420504750935052E22 evalInputDelta=1.7981641210816344
Non-optimal measurement 7.865887857948195 < 6.067806210809841. Total: 2
Armijo: th(1.029707176511643E-13)=7.865887857948195; dx=-2.807935825081462E22 evalInputDelta=8.24739432800925E-5
Non-optimal measurement 7.865964979303593 < 6.067806210809841. Total: 2
Armijo: th(6.057101038303783E-15)=7.865964979303593; dx=-2.8312515977471743E22 evalInputDelta=5.352587882256898E-6
Non-optimal measurement 6.067806210809841 < 6.067806210809841. Total: 2
MIN ALPHA (3.3650561323909904E-16): th(1.6475314824186289E-12)=6.067806210809841
Fitness changed from 7.865970331891475 to 6.067806210809841
Iteration 1 complete. Error: 6.067806210809841 Total: 0.0297; Orientation: 0.0010; Line Search: 0.0263
Non-optimal measurement 6.067806210809841 < 6.067806210809841. Total: 2
LBFGS Accumulation History: 2 points
Non-optimal measurement 6.067806210809841 < 6.067806210809841. Total: 2
th(0)=6.067806210809841;dx=-1.3350768672429327E22
Adding measurement 2f5b3411 to history. Total: 2
New Minimum: 6.067806210809841 > 6.0677495192645114
Armijo: th(2.154434690031884E-15)=6.0677495192645114; dx=-1.3349166119126846E22 evalInputDelta=5.6691545329279336E-5
Non-optimal measurement 6.067777856504459 < 6.0677495192645114. Total: 3
Armijo: th(1.077217345015942E-15)=6.067777856504459; dx=-1.3349967395778086E22 evalInputDelta=2.835430538183914E-5
Non-optimal measurement 6.0677495192645114 < 6.0677495192645114. Total: 3
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.0677495192645114
Fitness changed from 6.067806210809841 to 6.0677495192645114
Iteration 2 complete. Error: 6.0677495192645114 Total: 0.0066; Orientation: 0.0006; Line Search: 0.0050
Non-optimal measurement 6.0677495192645114 < 6.0677495192645114. Total: 3
LBFGS Accumulation History: 3 points
Non-optimal measurement 6.0677495192645114 < 6.0677495192645114. Total: 3
th(0)=6.0677495192645114;dx=-1.3347564032127261E22
Adding measurement 5243e7d4 to history. Total: 3
New Minimum: 6.0677495192645114 > 6.067692894651524
Armijo: th(2.154434690031884E-15)=6.067692894651524; dx=-1.3345962411371866E22 evalInputDelta=5.662461298783228E-5
Non-optimal measurement 6.067721198441342 < 6.067692894651524. Total: 4
Armijo: th(1.077217345015942E-15)=6.067721198441342; dx=-1.3346763221749564E22 evalInputDelta=2.8320823169281084E-5
Non-optimal measurement 6.067692894651524 < 6.067692894651524. Total: 4
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.067692894651524
Fitness changed from 6.0677495192645114 to 6.067692894651524
Iteration 3 complete. Error: 6.067692894651524 Total: 0.0064; Orientation: 0.0005; Line Search: 0.0052
Non-optimal measurement 6.067692894651524 < 6.067692894651524. Total: 4
Rejected: LBFGS Orientation magnitude: 1.479e+00, gradient 1.155e+11, dot -0.972; [9d8ce550-1aaa-4f65-868c-22efc79371e0 = 1.000/1.000e+00, 49acbad7-7b52-4d68-baca-7aef1189df45 = 1.000/1.000e+00, a6829307-2d68-4efb-89c5-735ce8f3f9d7 = 0.000e+00, 2ee3025c-94a2-42f8-9275-afb828c72cf6 = 1.000/1.000e+00, 584a9b50-e12d-4608-b2e2-6b4d589b8b75 = 0.000e+00, 308ee6d4-1259-4cc2-8fd4-bfedef7907a9 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 6.067692894651524, 6.0677495192645114, 6.067806210809841, 7.865970331891475
LBFGS Accumulation History: 3 points
Removed measurement 5243e7d4 to history. Total: 3
Adding measurement 7feb13dc to history. Total: 3
th(0)=6.067692894651524;dx=-1.3344361256801955E22
Adding measurement 5de795a1 to history. Total: 4
New Minimum: 6.067692894651524 > 6.067636336835265
Armijo: th(2.154434690031884E-15)=6.067636336835265; dx=-1.3342760568358858E22 evalInputDelta=5.655781625879541E-5
Non-optimal measurement 6.067664607242714 < 6.067636336835265. Total: 5
Armijo: th(1.077217345015942E-15)=6.067664607242714; dx=-1.3343560912580406E22 evalInputDelta=2.828740881000158E-5
Non-optimal measurement 6.067636336835265 < 6.067636336835265. Total: 5
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.067636336835265
Fitness changed from 6.067692894651524 to 6.067636336835265
Iteration 4 complete. Error: 6.067636336835265 Total: 0.0166; Orientation: 0.0108; Line Search: 0.0049
Non-optimal measurement 6.067636336835265 < 6.067636336835265. Total: 5
Rejected: LBFGS Orientation magnitude: 1.479e+00, gradient 1.155e+11, dot -0.972; [308ee6d4-1259-4cc2-8fd4-bfedef7907a9 = 1.000/1.000e+00, a6829307-2d68-4efb-89c5-735ce8f3f9d7 = 0.000e+00, 584a9b50-e12d-4608-b2e2-6b4d589b8b75 = 0.000e+00, 49acbad7-7b52-4d68-baca-7aef1189df45 = 1.000/1.000e+00, 2ee3025c-94a2-42f8-9275-afb828c72cf6 = 1.000/1.000e+00, 9d8ce550-1aaa-4f65-868c-22efc79371e0 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 6.067636336835265, 6.067692894651524, 6.0677495192645114, 6.067806210809841, 7.865970331891475
Rejected: LBFGS Orientation magnitude: 3.447e+00, gradient 1.155e+11, dot -0.955; [a6829307-2d68-4efb-89c5-735ce8f3f9d7 = 0.000e+00, 9d8ce550-1aaa-4f65-868c-22efc79371e0 = 1.000/1.000e+00, 2ee3025c-94a2-42f8-9275-afb828c72cf6 = 1.000/1.000e+00, 308ee6d4-1259-4cc2-8fd4-bfedef7907a9 = 1.000/1.000e+00, 584a9b50-e12d-4608-b2e2-

...skipping 502592 bytes...

907a9 = 1.000/1.000e+00, 9d8ce550-1aaa-4f65-868c-22efc79371e0 = 1.000/1.000e+00, 49acbad7-7b52-4d68-baca-7aef1189df45 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 6.05563734947488, 6.055680972218181, 6.057598133637887, 6.0576437063573945, 6.05768932514177
Rejected: LBFGS Orientation magnitude: 3.756e+00, gradient 1.123e+11, dot -0.961; [a6829307-2d68-4efb-89c5-735ce8f3f9d7 = 0.000e+00, 9d8ce550-1aaa-4f65-868c-22efc79371e0 = 1.000/1.000e+00, 49acbad7-7b52-4d68-baca-7aef1189df45 = 1.000/1.000e+00, 2ee3025c-94a2-42f8-9275-afb828c72cf6 = 1.000/1.000e+00, 308ee6d4-1259-4cc2-8fd4-bfedef7907a9 = 1.000/1.000e+00, 584a9b50-e12d-4608-b2e2-6b4d589b8b75 = 0.000e+00]
Orientation rejected. Popping history element from 6.05563734947488, 6.055680972218181, 6.057598133637887, 6.0576437063573945
LBFGS Accumulation History: 3 points
Removed measurement 7a10655a to history. Total: 4
Removed measurement 59162ac6 to history. Total: 3
Adding measurement 60fb67fd to history. Total: 3
th(0)=6.05563734947488;dx=-1.262028106528119E22
Adding measurement 47fd1075 to history. Total: 4
New Minimum: 6.05563734947488 > 6.055593769364027
Armijo: th(2.154434690031884E-15)=6.055593769364027; dx=-1.2618900312764048E22 evalInputDelta=4.358011085336955E-5
Non-optimal measurement 6.0556155537483916 < 6.055593769364027. Total: 5
Armijo: th(1.077217345015942E-15)=6.0556155537483916; dx=-1.261959068902262E22 evalInputDelta=2.1795726488527123E-5
Non-optimal measurement 6.055593769364027 < 6.055593769364027. Total: 5
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.055593769364027
Fitness changed from 6.05563734947488 to 6.055593769364027
Iteration 247 complete. Error: 6.055593769364027 Total: 0.0349; Orientation: 0.0276; Line Search: 0.0062
Non-optimal measurement 6.055593769364027 < 6.055593769364027. Total: 5
Rejected: LBFGS Orientation magnitude: 3.755e+00, gradient 1.123e+11, dot -0.961; [2ee3025c-94a2-42f8-9275-afb828c72cf6 = 1.000/1.000e+00, a6829307-2d68-4efb-89c5-735ce8f3f9d7 = 0.000e+00, 9d8ce550-1aaa-4f65-868c-22efc79371e0 = 1.000/1.000e+00, 584a9b50-e12d-4608-b2e2-6b4d589b8b75 = 0.000e+00, 49acbad7-7b52-4d68-baca-7aef1189df45 = 1.000/1.000e+00, 308ee6d4-1259-4cc2-8fd4-bfedef7907a9 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 6.055593769364027, 6.05563734947488, 6.057598133637887, 6.0576437063573945, 6.05768932514177
Rejected: LBFGS Orientation magnitude: 3.757e+00, gradient 1.123e+11, dot -0.961; [a6829307-2d68-4efb-89c5-735ce8f3f9d7 = 0.000e+00, 584a9b50-e12d-4608-b2e2-6b4d589b8b75 = 0.000e+00, 2ee3025c-94a2-42f8-9275-afb828c72cf6 = 1.000/1.000e+00, 308ee6d4-1259-4cc2-8fd4-bfedef7907a9 = 1.000/1.000e+00, 49acbad7-7b52-4d68-baca-7aef1189df45 = 1.000/1.000e+00, 9d8ce550-1aaa-4f65-868c-22efc79371e0 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 6.055593769364027, 6.05563734947488, 6.057598133637887, 6.0576437063573945
LBFGS Accumulation History: 3 points
Removed measurement 47fd1075 to history. Total: 4
Removed measurement 60fb67fd to history. Total: 3
Adding measurement 27bbad8b to history. Total: 3
th(0)=6.055593769364027;dx=-1.2617519999899933E22
Adding measurement 2a14ac79 to history. Total: 4
New Minimum: 6.055593769364027 > 6.055550231813379
Armijo: th(2.154434690031884E-15)=6.055550231813379; dx=-1.2616140126638718E22 evalInputDelta=4.353755064734344E-5
Non-optimal measurement 6.055571994926024 < 6.055550231813379. Total: 5
Armijo: th(1.077217345015942E-15)=6.055571994926024; dx=-1.2616830063269328E22 evalInputDelta=2.1774438002886143E-5
Non-optimal measurement 6.055550231813379 < 6.055550231813379. Total: 5
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.055550231813379
Fitness changed from 6.055593769364027 to 6.055550231813379
Iteration 248 complete. Error: 6.055550231813379 Total: 0.0304; Orientation: 0.0242; Line Search: 0.0054
Non-optimal measurement 6.055550231813379 < 6.055550231813379. Total: 5
Rejected: LBFGS Orientation magnitude: 3.756e+00, gradient 1.123e+11, dot -0.961; [584a9b50-e12d-4608-b2e2-6b4d589b8b75 = 0.000e+00, 308ee6d4-1259-4cc2-8fd4-bfedef7907a9 = 1.000/1.000e+00, a6829307-2d68-4efb-89c5-735ce8f3f9d7 = 0.000e+00, 9d8ce550-1aaa-4f65-868c-22efc79371e0 = 1.000/1.000e+00, 2ee3025c-94a2-42f8-9275-afb828c72cf6 = 1.000/1.000e+00, 49acbad7-7b52-4d68-baca-7aef1189df45 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 6.055550231813379, 6.055593769364027, 6.057598133637887, 6.0576437063573945, 6.05768932514177
Rejected: LBFGS Orientation magnitude: 3.757e+00, gradient 1.123e+11, dot -0.961; [49acbad7-7b52-4d68-baca-7aef1189df45 = 1.000/1.000e+00, 2ee3025c-94a2-42f8-9275-afb828c72cf6 = 1.000/1.000e+00, 308ee6d4-1259-4cc2-8fd4-bfedef7907a9 = 1.000/1.000e+00, a6829307-2d68-4efb-89c5-735ce8f3f9d7 = 0.000e+00, 584a9b50-e12d-4608-b2e2-6b4d589b8b75 = 0.000e+00, 9d8ce550-1aaa-4f65-868c-22efc79371e0 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 6.055550231813379, 6.055593769364027, 6.057598133637887, 6.0576437063573945
LBFGS Accumulation History: 3 points
Removed measurement 2a14ac79 to history. Total: 4
Removed measurement 27bbad8b to history. Total: 3
Adding measurement 3ed005d8 to history. Total: 3
th(0)=6.055550231813379;dx=-1.2614760692930293E22
Adding measurement 66e676c4 to history. Total: 4
New Minimum: 6.055550231813379 > 6.055506736750862
Armijo: th(2.154434690031884E-15)=6.055506736750862; dx=-1.2613381698724563E22 evalInputDelta=4.3495062516996086E-5
Non-optimal measurement 6.055528478627804 < 6.055506736750862. Total: 5
Armijo: th(1.077217345015942E-15)=6.055528478627804; dx=-1.261407119582743E22 evalInputDelta=2.1753185575512646E-5
Non-optimal measurement 6.055506736750862 < 6.055506736750862. Total: 5
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.055506736750862
Fitness changed from 6.055550231813379 to 6.055506736750862
Iteration 249 complete. Error: 6.055506736750862 Total: 0.0293; Orientation: 0.0232; Line Search: 0.0053
Non-optimal measurement 6.055506736750862 < 6.055506736750862. Total: 5
Rejected: LBFGS Orientation magnitude: 3.756e+00, gradient 1.123e+11, dot -0.961; [2ee3025c-94a2-42f8-9275-afb828c72cf6 = 1.000/1.000e+00, 49acbad7-7b52-4d68-baca-7aef1189df45 = 1.000/1.000e+00, 584a9b50-e12d-4608-b2e2-6b4d589b8b75 = 0.000e+00, 308ee6d4-1259-4cc2-8fd4-bfedef7907a9 = 1.000/1.000e+00, 9d8ce550-1aaa-4f65-868c-22efc79371e0 = 1.000/1.000e+00, a6829307-2d68-4efb-89c5-735ce8f3f9d7 = 0.000e+00]
Orientation rejected. Popping history element from 6.055506736750862, 6.055550231813379, 6.057598133637887, 6.0576437063573945, 6.05768932514177
Rejected: LBFGS Orientation magnitude: 3.758e+00, gradient 1.123e+11, dot -0.961; [308ee6d4-1259-4cc2-8fd4-bfedef7907a9 = 1.000/1.000e+00, 9d8ce550-1aaa-4f65-868c-22efc79371e0 = 1.000/1.000e+00, 49acbad7-7b52-4d68-baca-7aef1189df45 = 1.000/1.000e+00, a6829307-2d68-4efb-89c5-735ce8f3f9d7 = 0.000e+00, 584a9b50-e12d-4608-b2e2-6b4d589b8b75 = 0.000e+00, 2ee3025c-94a2-42f8-9275-afb828c72cf6 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 6.055506736750862, 6.055550231813379, 6.057598133637887, 6.0576437063573945
LBFGS Accumulation History: 3 points
Removed measurement 66e676c4 to history. Total: 4
Removed measurement 3ed005d8 to history. Total: 3
Adding measurement 44b56fa8 to history. Total: 3
th(0)=6.055506736750862;dx=-1.2612003143971445E22
Adding measurement 426c611f to history. Total: 4
New Minimum: 6.055506736750862 > 6.055463284104566
Armijo: th(2.154434690031884E-15)=6.055463284104566; dx=-1.261062502862088E22 evalInputDelta=4.345264629623813E-5
Non-optimal measurement 6.055485004781741 < 6.055463284104566. Total: 5
Armijo: th(1.077217345015942E-15)=6.055485004781741; dx=-1.2611314086296162E22 evalInputDelta=2.1731969121141503E-5
Non-optimal measurement 6.055463284104566 < 6.055463284104566. Total: 5
MIN ALPHA (3.5907244833864734E-16): th(2.154434690031884E-15)=6.055463284104566
Fitness changed from 6.055506736750862 to 6.055463284104566
Iteration 250 complete. Error: 6.055463284104566 Total: 0.0846; Orientation: 0.0787; Line Search: 0.0051
Final threshold in iteration 251: 6.055463284104566 (> 0.0) after 10.035s (< 30.000s)

Returns

    6.055463284104566

This training apply resulted in the following configuration:

TrainingTester.java:785 executed in 0.00 seconds (0.000 gc):

    RefList<double[]> state = network.state();
    assert state != null;
    String description = state.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b)
        .orElse("");
    state.freeRef();
    return description;

Returns

    [-0.11621165874799193, 0.6943929766170008, -0.0750275918079723, -0.16447922321118752, 0.527986445883585, 0.1505103227941544, 0.32409469190084483, -0.14118717757200366, -0.7672289777352501]

And regressed input:

TrainingTester.java:797 executed in 0.00 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(data)).flatMap(x -> {
      return RefArrays.stream(x);
    }).limit(1).map(x -> {
      String temp_18_0015 = x.prettyPrint();
      x.freeRef();
      return temp_18_0015;
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [ -0.11172061029230794, 0.14561032921430386, 0.5916839677908636 ]

To produce the following output:

TrainingTester.java:808 executed in 0.00 seconds (0.000 gc):

    Result[] array = ConstantResult.batchResultArray(pop(RefUtil.addRef(data)));
    @Nullable
    Result eval = layer.eval(array);
    assert eval != null;
    TensorList tensorList = Result.getData(eval);
    String temp_18_0016 = tensorList.stream().limit(1).map(x -> {
      String temp_18_0017 = x.prettyPrint();
      x.freeRef();
      return temp_18_0017;
    }).reduce((a, b) -> a + "\n" + b).orElse("");
    tensorList.freeRef();
    return temp_18_0016;

Returns

    [ 0.06970140415297463, 0.1843105443876866, -0.5107234539292786 ]

TrainingTester.java:576 executed in 0.01 seconds (0.000 gc):

    return TestUtil.compare(title + " vs Iteration", runs);
Logging
Plotting range=[1.0, 0.7821473752302387], [250.0, 0.7830317021542661]; valueStats=DoubleSummaryStatistics{count=500, sum=3030.680681, min=6.055463, average=6.061361, max=6.067806}
Plotting 250 points for GD
Plotting 250 points for LBFGS

Returns

Result

TrainingTester.java:579 executed in 0.01 seconds (0.000 gc):

    return TestUtil.compareTime(title + " vs Time", runs);
Logging
Plotting range=[0.0, 0.7821473752302387], [10.005, 0.7830317021542661]; valueStats=DoubleSummaryStatistics{count=500, sum=3030.680681, min=6.055463, average=6.061361, max=6.067806}
Plotting 250 points for GD
Only 0 points for LBFGS

Returns

Result

Results

TrainingTester.java:350 executed in 0.00 seconds (0.000 gc):

    return grid(inputLearning, modelLearning, completeLearning);

Returns

Result

TrainingTester.java:353 executed in 0.00 seconds (0.000 gc):

    return new ComponentResult(null == inputLearning ? null : inputLearning.value,
        null == modelLearning ? null : modelLearning.value, null == completeLearning ? null : completeLearning.value);

Returns

    {"input":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "model":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "complete":{ "LBFGS": { "type": "NonConverged", "value": 6.055463284104566 }, "CjGD": { "type": "NonConverged", "value": NaN }, "GD": { "type": "NonConverged", "value": 6.055463284104566 } }}

LayerTests.java:605 executed in 0.00 seconds (0.000 gc):

    throwException(exceptions.addRef());

Results

detailsresult
{"input":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "model":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "complete":{ "LBFGS": { "type": "NonConverged", "value": 6.055463284104566 }, "CjGD": { "type": "NonConverged", "value": NaN }, "GD": { "type": "NonConverged", "value": 6.055463284104566 } }}OK
  {
    "result": "OK",
    "performance": {
      "execution_time": "14.998",
      "gc_time": "1.725"
    },
    "created_on": 1587006845679,
    "file_name": "trainingTest",
    "report": {
      "simpleName": "Basic",
      "canonicalName": "com.simiacryptus.mindseye.layers.java.FullyConnectedLayerTest.Basic",
      "link": "https://github.com/SimiaCryptus/mindseye-java/tree/c9a1867488dc7e77a975f095285b5882c0486db6/src/test/java/com/simiacryptus/mindseye/layers/java/FullyConnectedLayerTest.java",
      "javaDoc": "The type Basic."
    },
    "training_analysis": {
      "input": {
        "LBFGS": {
          "type": "Converged",
          "value": 0.0
        },
        "CjGD": {
          "type": "Converged",
          "value": 0.0
        },
        "GD": {
          "type": "Converged",
          "value": 0.0
        }
      },
      "model": {
        "LBFGS": {
          "type": "Converged",
          "value": 0.0
        },
        "CjGD": {
          "type": "Converged",
          "value": 0.0
        },
        "GD": {
          "type": "Converged",
          "value": 0.0
        }
      },
      "complete": {
        "LBFGS": {
          "type": "NonConverged",
          "value": 6.055463284104566
        },
        "CjGD": {
          "type": "NonConverged",
          "value": "NaN"
        },
        "GD": {
          "type": "NonConverged",
          "value": 6.055463284104566
        }
      }
    },
    "archive": "s3://code.simiacrypt.us/tests/com/simiacryptus/mindseye/layers/java/FullyConnectedLayer/Basic/trainingTest/202004161405",
    "id": "71a6b5ae-672f-4ff8-a56a-054fd6622425",
    "report_type": "Components",
    "display_name": "Comparative Training",
    "target": {
      "simpleName": "FullyConnectedLayer",
      "canonicalName": "com.simiacryptus.mindseye.layers.java.FullyConnectedLayer",
      "link": "https://github.com/SimiaCryptus/mindseye-java/tree/c9a1867488dc7e77a975f095285b5882c0486db6/src/main/java/com/simiacryptus/mindseye/layers/java/FullyConnectedLayer.java",
      "javaDoc": "The type Fully connected layer."
    }
  }