1. Test Modules
  2. Training Characteristics
    1. Input Learning
      1. Gradient Descent
      2. Conjugate Gradient Descent
      3. Limited-Memory BFGS
    2. Results
  3. Results

Target Description: The type Sub batch layer.

Report Description: The type Sub batch layer test.

Subreport: Logs for com.simiacryptus.ref.lang.ReferenceCountingBase

Test Modules

Using Seed 3995930517763284992

Training Characteristics

Input Learning

In this apply, we use a network to learn this target input, given it's pre-evaluated output:

TrainingTester.java:445 executed in 0.00 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(input_target)).flatMap(RefArrays::stream).map(x -> {
      try {
        return x.prettyPrint();
      } finally {
        x.freeRef();
      }
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [ 0.7, -0.128, 0.08, 0.496, -0.608 ]
    [ -0.608, -0.128, 0.08, 0.7, 0.496 ]
    [ 0.496, -0.128, -0.608, 0.08, 0.7 ]
    [ -0.608, 0.496, -0.128, 0.7, 0.08 ]
    [ 0.08, 0.7, -0.608, 0.496, -0.128 ]

Gradient Descent

First, we train using basic gradient descent method apply weak line search conditions.

TrainingTester.java:638 executed in 0.44 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging
Reset training subject: 4273731195278
BACKPROP_AGG_SIZE = 3
THREADS = 64
SINGLE_THREADED = false
Initialized CoreSettings = {
"backpropAggregationSize" : 3,
"jvmThreads" : 64,
"singleThreaded" : false
}
Reset training subject: 4273764462472
Constructing line search parameters: GD
th(0)=1.7690549327053968;dx=-0.018925780932954404
New Minimum: 1.7690549327053968 > 1.7298743002721217
WOLFE (weak): th(2.154434690031884)=1.7298743002721217; dx=-0.01743856211205328 evalInputDelta=0.039180632433275075
New Minimum: 1.7298743002721217 > 1.6939453863425733
END: th(4.308869380063768)=1.6939453863425733; dx=-0.015908291489866475 evalInputDelta=0.07510954636282352
Fitness changed from 1.7690549327053968 to 1.6939453863425733
Iteration 1 complete. Error: 1.6939453863425733 Total: 0.0824; Orientation: 0.0040; Line Search: 0.0339
th(0)=1.6939453863425733;dx=-0.013409339578050882
New Minimum: 1.6939453863425733 > 1.591455535075515
END: th(9.283177667225559)=1.591455535075515; dx=-0.008617892722741466 evalInputDelta=0.10248985126705823
Fitness changed from 1.6939453863425733 to 1.591455535075515
Iteration 2 complete. Error: 1.591455535075515 Total: 0.0289; Orientation: 0.0011; Line Search: 0.0225
th(0)=1.591455535075515;dx=-0.005615725117688548
New Minimum: 1.591455535075515 > 1.5227236385555087
END: th(20.000000000000004)=1.5227236385555087; dx=-0.0012330258487457948 evalInputDelta=0.06873189652000633
Fitness changed from 1.591455535075515 to 1.5227236385555087
Iteration 3 complete. Error: 1.5227236385555087 Total: 0.0208; Orientation: 0.0010; Line Search: 0.0122
th(0)=1.5227236385555087;dx=-3.9660737483633904E-4
New Minimum: 1.5227236385555087 > 1.5170636843900633
WOLF (strong): th(43.088693800637685)=1.5170636843900633; dx=1.2868579467718478E-4 evalInputDelta=0.005659954165445402
END: th(21.544346900318843)=1.5170639719787817; dx=-1.300531589226398E-4 evalInputDelta=0.005659666576727007
Fitness changed from 1.5227236385555087 to 1.5170636843900633
Iteration 4 complete. Error: 1.5170636843900633 Total: 0.0295; Orientation: 0.0009; Line Search: 0.0244
th(0)=1.5170636843900633;dx=-1.3733635175558617E-4
Armijo: th(46.4158883361278)=1.518322563262323; dx=1.8537396372184046E-4 evalInputDelta=-0.0012588788722596167
New Minimum: 1.5170636843900633 > 1.515819925412146
WOLF (strong): th(23.2079441680639)=1.515819925412146; dx=2.867677541446113E-5 evalInputDelta=0.0012437589779172864
END: th(7.735981389354634)=1.5162196857120098; dx=-8.102145586896903E-5 evalInputDelta=8.439986780535147E-4
Fitness changed from 1.5170636843900633 to 1.515819925412146
Iteration 5 complete. Error: 1.515819925412146 Total: 0.0284; Orientation: 0.0011; Line Search: 0.0232
th(0)=1.515819925412146;dx=-1.3427252528282318E-5
New Minimum: 1.515819925412146 > 1.5157048631371286
END: th(16.66666666666667)=1.5157048631371286; dx=-3.89682944478594E-7 evalInputDelta=1.1506227501745236E-4
Fitness changed from 1.515819925412146 to 1.5157048631371286
Iteration 6 complete. Error: 1.5157048631371286 Total: 0.0211; Orientation: 0.0012; Line Search: 0.0164
th(0)=1.5157048631371286;dx=-1.6586113188833925E-7
New Minimum: 1.5157048631371286 > 1.5157020212023125
WOLF (strong): th(35.90724483386474)=1.5157020212023125; dx=7.643509404848712E-9 evalInputDelta=2.8419348161445868E-6
END: th(17.95362241693237)=1.515702663410641; dx=-7.916543057312457E-8 evalInputDelta=2.1997264876283396E-6
Fitness changed from 1.5157048631371286 to 1.5157020212023125
Iteration 7 complete. Error: 1.5157020212023125 Total: 0.0264; Orientation: 0.0008; Line Search: 0.0221
th(0)=1.5157020212023125;dx=-5.382141237294831E-9
New Minimum: 1.5157020212023125 > 1.5157019312568427
WOLF (strong): th(38.679906946773166)=1.5157019312568427; dx=7.312314147049384E-10 evalInputDelta=8.994546973895012E-8
END: th(19.339953473386583)=1.5157019466714918; dx=-2.3253420664118763E-9 evalInputDelta=7.453082062625072E-8
Fitness changed from 1.5157020212023125 to 1.5157019312568427
Iteration 8 complete. Error: 1.5157019312568427 Total: 0.0149; Orientation: 0.0007; Line Search: 0.0113
th(0)=1.5157019312568427;dx=-1.238061409646982E-9
Armijo: th(41.66666666666668)=1.515701932933708; dx=1.318570017032298E-9 evalInputDelta=-1.6768653132714917E-9
New Minimum: 1.5157019312568427 > 1.5157019187794867
WOLF (strong): th(20.83333333333334)=1.5157019187794867; dx=4.024000198976086E-11 evalInputDelta=1.2477356037265963E-8
END: th(6.944444444444446)=1.5157019241386966; dx=-8.119641205409014E-10 evalInputDelta=7.118146072571108E-9
Fitness changed from 1.5157019312568427 to 1.5157019187794867
Iteration 9 complete. Error: 1.5157019187794867 Total: 0.0209; Orientation: 0.0009; Line Search: 0.0169
th(0)=1.5157019187794867;dx=-1.0589257485911384E-10
New Minimum: 1.5157019187794867 > 1.5157019176438657
END: th(14.96135201411031)=1.5157019176438657; dx=-4.5914753390982526E-11 evalInputDelta=1.1356209306256915E-9
Fitness changed from 1.5157019187794867 to 1.5157019176438657
Iteration 10 complete. Error: 1.5157019176438657 Total: 0.0120; Orientation: 0.0008; Line Search: 0.0081
Low gradient: 5.318656007136528E-6
th(0)=1.5157019176438657;dx=-2.8288101722249475E-11
New Minimum: 1.5157019176438657 > 1.5157019170529942
END: th(32.233255788977644)=1.5157019170529942; dx=-8.374308862617243E-12 evalInputDelta=5.908715738911496E-10
Fitness changed from 1.5157019176438657 to 1.5157019170529942
Iteration 11 complete. Error: 1.5157019170529942 Total: 0.0107; Orientation: 0.0009; Line Search: 0.0065
Low gradient: 1.948174377233266E-6
th(0)=1.5157019170529942;dx=-3.795383404108225E-12
New Minimum: 1.5157019170529942 > 1.5157019170158574
WOLF (strong): th(69.44444444444447)=1.5157019170158574; dx=2.7258036555967287E-12 evalInputDelta=3.713673812910656E-11
New Minimum: 1.5157019170158574 > 1.5157019169778183
END: th(34.722222222222236)=1.5157019169778183; dx=-5.347615863682397E-13 evalInputDelta=7.517586553262845E-11
Fitness changed from 1.5157019170529942 to 1.5157019169778183
Iteration 12 complete. Error: 1.5157019169778183 Total: 0.0181; Orientation: 0.0009; Line Search: 0.0143
Low gradient: 9.169244574347637E-7
th(0)=1.5157019169778183;dx=-8.407504606420359E-13
Armijo: th(74.80676007055155)=1.5157019170180035; dx=1.91512713887265E-12 evalInputDelta=-4.0185188510122316E-11
New Minimum: 1.5157019169778183 > 1.5157019169721413
WOLF (strong): th(37.403380035275774)=1.5157019169721413; dx=5.371881695092593E-13 evalInputDelta=5.6770144141182755E-12
New Minimum: 1.5157019169721413 > 1.5157019169701995
END: th(12.467793345091925)=1.5157019169701995; dx=-3.8143762162219906E-13 evalInputDelta=7.618794484187674E-12
Fitness changed from 1.5157019169778183 to 1.5157019169701995
Iteration 13 complete. Error: 1.5157019169701995 Total: 0.0226; Orientation: 0.0008; Line Search: 0.0186
Low gradient: 4.7081822421068954E-7
th(0)=1.5157019169701995;dx=-2.2166980024890714E-13
New Minimum: 1.5157019169701995 > 1.5157019169663268
END: th(26.861046490814704)=1.5157019169663268; dx=-6.666768671765828E-14 evalInputDelta=3.872679954497471E-12
Fitness changed from 1.5157019169701995 to 1.5157019169663268
Iteration 14 complete. Error: 1.5157019169663268 Total: 0.0131; Orientation: 0.0006; Line Search: 0.0098
Low gradient: 2.2992580804611283E-7
th(0)=1.5157019169663268;dx=-5.286587720565792E-14
New Minimum: 1.5157019169663268 > 1.5157019169658117
WOLF (strong): th(57.870370370370395)=1.5157019169658117; dx=3.506333047592619E-14 evalInputDelta=5.151434834260726E-13
New Minimum: 1.5157019169658117 > 1.515701916965433
END: th(28.935185185185198)=1.515701916965433; dx=-8.901248128046559E-15 evalInputDelta=8.93729534823251E-13
Fitness changed from 1.5157019169663268 to 1.515701916965433
Iteration 15 complete. Error: 1.515701916965433 Total: 0.0118; Orientation: 0.0007; Line Search: 0.0089
Low gradient: 1.1730496350700859E-7
th(0)=1.515701916965433;dx=-1.3760454463380617E-14
Armijo: th(62.33896672545963)=1.5157019169655856; dx=1.865000482949868E-14 evalInputDelta=-1.525446435834965E-13
New Minimum: 1.515701916965433 > 1.515701916965257
WOLF (strong): th(31.169483362729814)=1.515701916965257; dx=2.444777137310339E-15 evalInputDelta=1.7608137170554983E-13
END: th(10.389827787576605)=1.5157019169653183; dx=-8.358710162607828E-15 evalInputDelta=1.1479706074624119E-13
Fitness changed from 1.515701916965433 to 1.515701916965257
Iteration 16 complete. Error: 1.515701916965257 Total: 0.0144; Orientation: 0.0006; Line Search: 0.0119
Low gradient: 7.540534938970734E-8
th(0)=1.515701916965257;dx=-5.6859667165838364E-15
New Minimum: 1.515701916965257 > 1.5157019169652033
WOLF (strong): th(22.384205409012257)=1.5157019169652033; dx=9.006297245578504E-16 evalInputDelta=5.3734794391857577E-14
END: th(11.192102704506128)=1.5157019169652117; dx=-2.3926684600134076E-15 evalInputDelta=4.529709940470639E-14
Fitness changed from 1.515701916965257 to 1.5157019169652033
Iteration 17 complete. Error: 1.5157019169652033 Total: 0.0129; Orientation: 0.0005; Line Search: 0.0101
Low gradient: 2.9229399886352122E-8
th(0)=1.5157019169652033;dx=-8.543578177162814E-16
New Minimum: 1.5157019169652033 > 1.5157019169651935
WOLF (strong): th(24.11265432098767)=1.5157019169651935; dx=3.913434985267034E-17 evalInputDelta=9.769962616701378E-15
END: th(12.056327160493835)=1.5157019169651957; dx=-4.076117330425552E-16 evalInputDelta=7.549516567451064E-15
Fitness changed from 1.5157019169652033 to 1.5157019169651935
Iteration 18 complete. Error: 1.5157019169651935 Total: 0.0102; Orientation: 0.0005; Line Search: 0.0071
Low gradient: 1.384891925627608E-8
th(0)=1.5157019169651935;dx=-1.917925645668544E-16
New Minimum: 1.5157019169651935 > 1.5157019169651913
WOLF (strong): th(25.974569468941517)=1.5157019169651913; dx=6.237577917322152E-18 evalInputDelta=2.220446049250313E-15
END: th(12.987284734470759)=1.5157019169651917; dx=-9.277749303324962E-17 evalInputDelta=1.7763568394002505E-15
Fitness changed from 1.5157019169651935 to 1.5157019169651913
Iteration 19 complete. Error: 1.5157019169651913 Total: 0.0093; Orientation: 0.0005; Line Search: 0.0067
Low gradient: 7.175844307537702E-9
th(0)=1.5157019169651913;dx=-5.1492741526021246E-17
New Minimum: 1.5157019169651913 > 1.5157019169651904
WOLF (strong): th(27.980256761265327)=1.5157019169651904; dx=9.068069649676561E-18 evalInputDelta=8.881784197001252E-16
END: th(13.990128380632664)=1.5157019169651906; dx=-2.1212335865514417E-17 evalInputDelta=6.661338147750939E-16
Fitness changed from 1.5157019169651913 to 1.5157019169651904
Iteration 20 complete. Error: 1.5157019169651904 Total: 0.0099; Orientation: 0.0006; Line Search: 0.0070
Low gradient: 4.164911228243026E-9
th(0)=1.5157019169651904;dx=-1.734648553914483E-17
WOLF (strong): th(30.140817901234595)=1.5157019169651904; dx=8.337797459998672E-18 evalInputDelta=0.0
END: th(15.070408950617297)=1.5157019169651904; dx=-4.5043440394629024E-18 evalInputDelta=0.0
Fitness changed from 1.5157019169651904 to 1.5157019169651904
Static Iteration Total: 0.0088; Orientation: 0.0004; Line Search: 0.0066
Iteration 21 failed. Error: 1.5157019169651904
Previous Error: 0.0 -> 1.5157019169651904
Optimization terminated 21
Final threshold in iteration 21: 1.5157019169651904 (> 0.0) after 0.428s (< 30.000s)

Returns

    1.5157019169651904

This training apply resulted in the following configuration:

TrainingTester.java:785 executed in 0.00 seconds (0.000 gc):

    RefList<double[]> state = network.state();
    assert state != null;
    String description = state.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b)
        .orElse("");
    state.freeRef();
    return description;

Returns

    

And regressed input:

TrainingTester.java:797 executed in 0.00 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(data)).flatMap(x -> {
      return RefArrays.stream(x);
    }).limit(1).map(x -> {
      String temp_18_0015 = x.prettyPrint();
      x.freeRef();
      return temp_18_0015;
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [ 0.7000000128373545, -0.12800002074718903, 0.07999998390203811, 0.49599997466881396, -0.6079999506610229 ]

To produce the following output:

TrainingTester.java:808 executed in 0.00 seconds (0.000 gc):

    Result[] array = ConstantResult.batchResultArray(pop(RefUtil.addRef(data)));
    @Nullable
    Result eval = layer.eval(array);
    assert eval != null;
    TensorList tensorList = Result.getData(eval);
    String temp_18_0016 = tensorList.stream().limit(1).map(x -> {
      String temp_18_0017 = x.prettyPrint();
      x.freeRef();
      return temp_18_0017;
    }).reduce((a, b) -> a + "\n" + b).orElse("");
    tensorList.freeRef();
    return temp_18_0016;

Returns

    [ 0.32672379254322736, 0.14275289224697948, 0.1757592417428162, 0.2664309484213246, 0.08833312504565237 ]

Conjugate Gradient Descent

First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.

TrainingTester.java:603 executed in 0.45 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging
Reset training subject: 4274174855148
Reset training subject: 4274176851742
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=1.7690549327053968}, derivative=-0.018925780932954404}
New Minimum: 1.7690549327053968 > 1.7690549327035043
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=1.7690549327035043}, derivative=-0.018925780932886476}, evalInputDelta = -1.892486167776042E-12
New Minimum: 1.7690549327035043 > 1.7690549326921488
F(7.000000000000001E-10) = LineSearchPoint{point=PointSample{avg=1.7690549326921488}, derivative=-0.018925780932478903}, evalInputDelta = -1.3248069308247068E-11
New Minimum: 1.7690549326921488 > 1.7690549326126603
F(4.900000000000001E-9) = LineSearchPoint{point=PointSample{avg=1.7690549326126603}, derivative=-0.018925780929625866}, evalInputDelta = -9.273648515772948E-11
New Minimum: 1.7690549326126603 > 1.7690549320562425
F(3.430000000000001E-8) = LineSearchPoint{point=PointSample{avg=1.7690549320562425}, derivative=-0.018925780909654605}, evalInputDelta = -6.491542858810817E-10
New Minimum: 1.7690549320562425 > 1.7690549281613168
F(2.4010000000000004E-7) = LineSearchPoint{point=PointSample{avg=1.7690549281613168}, derivative=-0.018925780769855807}, evalInputDelta = -4.544080001167572E-9
New Minimum: 1.7690549281613168 > 1.769054900896838
F(1.6807000000000003E-6) = LineSearchPoint{point=PointSample{avg=1.769054900896838}, derivative=-0.01892577979126421}, evalInputDelta = -3.180855889794998E-8
New Minimum: 1.769054900896838 > 1.7690547100455238
F(1.1764900000000001E-5) = LineSearchPoint{point=PointSample{avg=1.7690547100455238}, derivative=-0.018925772941122423}, evalInputDelta = -2.2265987298375478E-7
New Minimum: 1.7690547100455238 > 1.7690533740882597
F(8.235430000000001E-5) = LineSearchPoint{point=PointSample{avg=1.7690533740882597}, derivative=-0.01892572499009955}, evalInputDelta = -1.5586171371317903E-6
New Minimum: 1.7690533740882597 > 1.769044022482187
F(5.764801000000001E-4) = LineSearchPoint{point=PointSample{avg=1.769044022482187}, derivative=-0.018925389331452165}, evalInputDelta = -1.0910223209759096E-5
New Minimum: 1.769044022482187 > 1.7689785658837813
F(0.004035360700000001) = LineSearchPoint{point=PointSample{avg=1.7689785658837813}, derivative=-0.018923039648050484}, evalInputDelta = -7.636682161549224E-5
New Minimum: 1.7689785658837813 > 1.7685205972894873
F(0.028247524900000005) = LineSearchPoint{point=PointSample{avg=1.7685205972894873}, derivative=-0.018906588295446933}, evalInputDelta = -5.343354159095259E-4
New Minimum: 1.7685205972894873 > 1.7653259807249033
F(0.19773267430000002) = LineSearchPoint{point=PointSample{avg=1.7653259807249033}, derivative=-0.018791254589204155}, evalInputDelta = -0.0037289519804935
New Minimum: 1.7653259807249033 > 1.7435145359620925
F(1.3841287201) = LineSearchPoint{point=PointSample{avg=1.7435145359620925}, derivative=-0.017975604210105016}, evalInputDelta = -0.025540396743304328
New Minimum: 1.7435145359620925 > 1.618938895908847
F(9.688901040700001) = LineSearchPoint{point=PointSample{avg=1.618938895908847}, derivative=-0.011948410130618094}, evalInputDelta = -0.15011603679654972
F(67.8223072849) = LineSearchPoint{point=PointSample{avg=1.984264874650504}, derivative=0.018781260255531847}, evalInputDelta = 0.21520994194510723
F(5.217100560376924) = LineSearchPoint{point=PointSample{avg=1.6797945452276106}, derivative=-0.015252015381786322}, evalInputDelta = -0.0892603874777862
New Minimum: 1.618938895908847 > 1.559636624761981
F(36.51970392263847) = LineSearchPoint{point=PointSample{avg=1.559636624761981}, derivative=0.006650722227460816}, evalInputDelta = -0.20941830794341576
1.559636624761981 <= 1.7690549327053968
New Minimum: 1.559636624761981 > 1.5236955794589995
F(27.023393770495037) = LineSearchPoint{point=PointSample{avg=1.5236955794589995}, derivative=7.420021658774384E-4}, evalInputDelta = -0.24535935324639735
Right bracket at 27.023393770495037
New Minimum: 1.5236955794589995 > 1.5232921505680046
F(26.00388808414968) = LineSearchPoint{point=PointSample{avg=1.5232921505680046}, derivative=4.769317045422314E-5}, evalInputDelta = -0.24576278213739222
Right bracket at 26.00388808414968
Converged to right
Fitness changed from 1.7690549327053968 to 1.5232921505680046
Iteration 1 complete. Error: 1.5232921505680046 Total: 0.0840; Orientation: 0.0004; Line Search: 0.0776
F(0.0) = LineSearchPoint{point=PointSample{avg=1.5232921505680046}, derivative=-6.545855209681081E-4}
New Minimum: 1.5232921505680046 > 1.517257986883629
F(26.00388808414968) = LineSearchPoint{point=PointSample{avg=1.517257986883629}, derivative=1.8947807404814758E-4}, evalInputDelta = -0.006034163684375704
1.517257986883629 <= 1.5232921505680046
New Minimum: 1.517257986883629 > 1.5167021215884229
F(20.166452775909235) = LineSearchPoint{point=PointSample{avg=1.5167021215884229}, derivative=8.070621150138847E-7}, evalInputDelta = -0.006590028979581719
Right bracket at 20.166452775909235
Converged to right
Fitness changed from 1.5232921505680046 to 1.5167021215884229
Iteration 2 complete. Error: 1.5167021215884229 Total: 0.0138; Orientation: 0.0006; Line Search: 0.0112
F(0.0) = LineSearchPoint{point=PointSample{avg=1.5167021215884229}, derivative=-5.795609475193728E-5}
New Minimum: 1.5167021215884229 > 1.5159539106467492
F(20.166452775909235) = LineSearchPoint{point=PointSample{avg=1.5159539106467492}, derivative=-1.6273873195683357E-5}, evalInputDelta = -7.482109416736549E-4
F(141.16516943136463) = LineSearchPoint{point=PointSample{avg=1.5288947616000284}, derivative=2.2815256158921073E-4}, evalInputDelta = 0.012192640011605471
F(10.858859187028049) = LineSearchPoint{point=PointSample{avg=1.5161947900606987}, derivative=-3.549202299437645E-5}, evalInputDelta = -5.073315277241974E-4
F(76.01201430919635) = LineSearchPoint{point=PointSample{avg=1.5182459150613505}, derivative=9.802970917821742E-5}, evalInputDelta = 0.001543793472927657
F(5.847078023784334) = LineSearchPoint{point=PointSample{avg=1.5163986322299332}, derivative=-4.585476337220028E-5}, evalInputDelta = -3.034893584896725E-4
F(40.92954616649034) = LineSearchPoint{point=PointSample{avg=1.51605992111712}, derivative=2.644790487151149E-5}, evalInputDelta = -6.422004713029228E-4
1.51605992111712 <= 1.5167021215884229
New Minimum: 1.5159539106467492 > 1.515889678789464
F(28.104315747614095) = LineSearchPoint{point=PointSample{avg=1.515889678789464}, derivative=8.516916714712466E-8}, evalInputDelta = -8.124427989588678E-4
Right bracket at 28.104315747614095
Converged to right
Fitness changed from 1.5167021215884229 to 1.515889678789464
Iteration 3 complete. Error: 1.515889678789464 Total: 0.0412; Orientation: 0.0006; Line Search: 0.0385
F(0.0) = LineSearchPoint{point=PointSample{avg=1.515889678789464}, derivative=-1.3020473028701213E-5}
New Minimum: 1.515889678789464 > 1.5157540851277234
F(28.104315747614095) = LineSearchPoint{point=PointSample{avg=1.5157540851277234}, derivative=3.359185430316271E-6}, evalInputDelta = -1.3559366174065168E-4
1.5157540851277234 <= 1.515889678789464
New Minimum: 1.5157540851277234 > 1.515744385613359
F(22.34060533664278) = LineSearchPoint{point=PointSample{avg=1.515744385613359}, derivative=5.9851847096142024E-9}, evalInputDelta = -1.4529317610501025E-4
Right bracket at 22.34060533664278
Converged to right
Fitness changed from 1.515889678789464 to 1.515744385613359
Iteration 4 complete. Error: 1.515744385613359 Total: 0.0130; Orientation: 0.0005; Line Search: 0.0104
F(0.0) = LineSearchPoint{point=PointSample{avg=1.515744385613359}, derivative=-2.2844536438888243E-6}
New Minimum: 1.515744385613359 > 1.515713388412051
F(22.34060533664278) = LineSearchPoint{point=PointSample{avg=1.515713388412051}, derivative=-4.912069198589268E-7}, evalInputDelta = -3.0997201307947364E-5
F(156.38423735649945) = LineSearchPoint{point=PointSample{avg=1.5163643487395206}, derivative=1.017727966250003E-5}, evalInputDelta = 6.199631261616378E-4
F(12.029556719730728) = LineSearchPoint{point=PointSample{avg=1.5157227168115015}, derivative=-1.3183407068109265E-6}, evalInputDelta = -2.166880185749953E-5
F(84.2068970381151) = LineSearchPoint{point=PointSample{avg=1.5158360944582847}, deriv

...skipping 14627 bytes...

evalInputDelta = -7.707168236947837E-13
1.5157019169659747 <= 1.5157019169667454
New Minimum: 1.5157019169656203 > 1.515701916965565
F(28.71906449666516) = LineSearchPoint{point=PointSample{avg=1.515701916965565}, derivative=1.4700914616886403E-20}, evalInputDelta = -1.1803891197814664E-12
Right bracket at 28.71906449666516
Converged to right
Fitness changed from 1.5157019169667454 to 1.515701916965565
Iteration 17 complete. Error: 1.515701916965565 Total: 0.0179; Orientation: 0.0003; Line Search: 0.0162
Low gradient: 1.5903790833352016E-7
F(0.0) = LineSearchPoint{point=PointSample{avg=1.515701916965565}, derivative=-2.5293056287101162E-14}
New Minimum: 1.515701916965565 > 1.5157019169653025
F(28.71906449666516) = LineSearchPoint{point=PointSample{avg=1.5157019169653025}, derivative=7.009239298267586E-15}, evalInputDelta = -2.62456723021387E-13
1.5157019169653025 <= 1.515701916965565
New Minimum: 1.5157019169653025 > 1.5157019169652806
F(22.48734653880322) = LineSearchPoint{point=PointSample{avg=1.5157019169652806}, derivative=6.210135295626937E-22}, evalInputDelta = -2.844391389089651E-13
Right bracket at 22.48734653880322
Converged to right
Fitness changed from 1.515701916965565 to 1.5157019169652806
Iteration 18 complete. Error: 1.5157019169652806 Total: 0.0092; Orientation: 0.0005; Line Search: 0.0071
Low gradient: 6.908035285198208E-8
F(0.0) = LineSearchPoint{point=PointSample{avg=1.5157019169652806}, derivative=-4.772095150154349E-15}
New Minimum: 1.5157019169652806 > 1.5157019169652153
F(22.48734653880322) = LineSearchPoint{point=PointSample{avg=1.5157019169652153}, derivative=-1.0354953957359567E-15}, evalInputDelta = -6.52811138479592E-14
F(157.41142577162253) = LineSearchPoint{point=PointSample{avg=1.515701916966588}, derivative=2.1384094136168008E-14}, evalInputDelta = 1.3073986337985843E-12
F(12.108571213201733) = LineSearchPoint{point=PointSample{avg=1.515701916965235}, derivative=-2.7600798452135925E-15}, evalInputDelta = -4.551914400963142E-14
F(84.75999849241214) = LineSearchPoint{point=PointSample{avg=1.515701916965473}, derivative=9.312009379952993E-15}, evalInputDelta = 1.9251267247000214E-13
F(6.519999884031703) = LineSearchPoint{point=PointSample{avg=1.515701916965253}, derivative=-3.688702278219898E-15}, evalInputDelta = -2.7533531010703882E-14
F(45.63999918822192) = LineSearchPoint{point=PointSample{avg=1.5157019169652357}, derivative=2.8116541985704108E-15}, evalInputDelta = -4.4853010194856324E-14
1.5157019169652357 <= 1.5157019169652806
New Minimum: 1.5157019169652153 > 1.515701916965212
F(28.719095102449028) = LineSearchPoint{point=PointSample{avg=1.515701916965212}, derivative=2.0585591093911653E-22}, evalInputDelta = -6.861178292183467E-14
Right bracket at 28.719095102449028
Converged to right
Fitness changed from 1.5157019169652806 to 1.515701916965212
Iteration 19 complete. Error: 1.515701916965212 Total: 0.0184; Orientation: 0.0005; Line Search: 0.0163
Low gradient: 3.832137839569438E-8
F(0.0) = LineSearchPoint{point=PointSample{avg=1.515701916965212}, derivative=-1.468528042145992E-15}
New Minimum: 1.515701916965212 > 1.5157019169651966
F(28.719095102449028) = LineSearchPoint{point=PointSample{avg=1.5157019169651966}, derivative=4.0696191800196606E-16}, evalInputDelta = -1.532107773982716E-14
1.5157019169651966 <= 1.515701916965212
New Minimum: 1.5157019169651966 > 1.5157019169651957
F(22.487348585793992) = LineSearchPoint{point=PointSample{avg=1.5157019169651957}, derivative=8.888501404563988E-24}, evalInputDelta = -1.6209256159527285E-14
Right bracket at 22.487348585793992
Converged to right
Fitness changed from 1.515701916965212 to 1.5157019169651957
Iteration 20 complete. Error: 1.5157019169651957 Total: 0.0085; Orientation: 0.0004; Line Search: 0.0069
Low gradient: 1.6645440749897557E-8
F(0.0) = LineSearchPoint{point=PointSample{avg=1.5157019169651957}, derivative=-2.770706977583502E-16}
New Minimum: 1.5157019169651957 > 1.5157019169651917
F(22.487348585793992) = LineSearchPoint{point=PointSample{avg=1.5157019169651917}, derivative=-6.012152749160349E-17}, evalInputDelta = -3.9968028886505635E-15
F(157.41144010055794) = LineSearchPoint{point=PointSample{avg=1.5157019169652715}, derivative=1.241573368281685E-15}, evalInputDelta = 7.571721027943568E-14
F(12.108572315427534) = LineSearchPoint{point=PointSample{avg=1.5157019169651929}, derivative=-1.6025191300266142E-16}, evalInputDelta = -2.886579864025407E-15
F(84.76000620799273) = LineSearchPoint{point=PointSample{avg=1.5157019169652066}, derivative=5.40660758952917E-16}, evalInputDelta = 1.0880185641326534E-14
F(6.520000477537902) = LineSearchPoint{point=PointSample{avg=1.515701916965194}, derivative=-2.1416827508797106E-16}, evalInputDelta = -1.7763568394002505E-15
F(45.64000334276531) = LineSearchPoint{point=PointSample{avg=1.5157019169651929}, derivative=1.63246251076478E-16}, evalInputDelta = -2.886579864025407E-15
1.5157019169651929 <= 1.5157019169651957
New Minimum: 1.5157019169651917 > 1.5157019169651915
F(28.719102467747625) = LineSearchPoint{point=PointSample{avg=1.5157019169651915}, derivative=2.9314812751856267E-24}, evalInputDelta = -4.218847493575595E-15
Right bracket at 28.719102467747625
Converged to right
Fitness changed from 1.5157019169651957 to 1.5157019169651915
Iteration 21 complete. Error: 1.5157019169651915 Total: 0.0167; Orientation: 0.0003; Line Search: 0.0153
Low gradient: 9.233835259130538E-9
F(0.0) = LineSearchPoint{point=PointSample{avg=1.5157019169651915}, derivative=-8.526371359276233E-17}
New Minimum: 1.5157019169651915 > 1.5157019169651906
F(28.719102467747625) = LineSearchPoint{point=PointSample{avg=1.5157019169651906}, derivative=2.3628504750304806E-17}, evalInputDelta = -8.881784197001252E-16
1.5157019169651906 <= 1.5157019169651915
F(22.4873490935464) = LineSearchPoint{point=PointSample{avg=1.5157019169651906}, derivative=7.804591976167378E-26}, evalInputDelta = -8.881784197001252E-16
Right bracket at 22.4873490935464
Converged to right
Fitness changed from 1.5157019169651915 to 1.5157019169651906
Iteration 22 complete. Error: 1.5157019169651906 Total: 0.0068; Orientation: 0.0003; Line Search: 0.0053
Low gradient: 5.725943515456691E-9
F(0.0) = LineSearchPoint{point=PointSample{avg=1.5157019169651906}, derivative=-3.278642914220052E-17}
New Minimum: 1.5157019169651906 > 1.5157019169651904
F(22.4873490935464) = LineSearchPoint{point=PointSample{avg=1.5157019169651904}, derivative=5.693472679302849E-18}, evalInputDelta = -2.220446049250313E-16
1.5157019169651904 <= 1.5157019169651906
F(19.16012886601178) = LineSearchPoint{point=PointSample{avg=1.5157019169651904}, derivative=-4.657130015640804E-26}, evalInputDelta = -2.220446049250313E-16
Left bracket at 19.16012886601178
Converged to left
Fitness changed from 1.5157019169651906 to 1.5157019169651904
Iteration 23 complete. Error: 1.5157019169651904 Total: 0.0067; Orientation: 0.0003; Line Search: 0.0054
Low gradient: 2.2249651991820935E-9
F(0.0) = LineSearchPoint{point=PointSample{avg=1.5157019169651904}, derivative=-4.950470137571413E-18}
F(19.16012886601178) = LineSearchPoint{point=PointSample{avg=1.5157019169651904}, derivative=-7.324702298687821E-19}, evalInputDelta = 0.0
F(134.12090206208248) = LineSearchPoint{point=PointSample{avg=1.5157019169651917}, derivative=2.4575528893811378E-17}, evalInputDelta = 1.3322676295501878E-15
F(10.316992466314037) = LineSearchPoint{point=PointSample{avg=1.5157019169651904}, derivative=-2.6792394199293065E-18}, evalInputDelta = 0.0
F(72.21894726419826) = LineSearchPoint{point=PointSample{avg=1.5157019169651904}, derivative=1.0948144772959363E-17}, evalInputDelta = 0.0
1.5157019169651904 <= 1.5157019169651904
F(22.487351496352613) = LineSearchPoint{point=PointSample{avg=1.5157019169651904}, derivative=2.05275642752977E-26}, evalInputDelta = 0.0
Right bracket at 22.487351496352613
Converged to right
Fitness changed from 1.5157019169651904 to 1.5157019169651904
Static Iteration Total: 0.0121; Orientation: 0.0003; Line Search: 0.0108
Iteration 24 failed. Error: 1.5157019169651904
Previous Error: 0.0 -> 1.5157019169651904
Optimization terminated 24
Final threshold in iteration 24: 1.5157019169651904 (> 0.0) after 0.449s (< 30.000s)

Returns

    1.5157019169651904

This training apply resulted in the following configuration:

TrainingTester.java:785 executed in 0.00 seconds (0.000 gc):

    RefList<double[]> state = network.state();
    assert state != null;
    String description = state.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b)
        .orElse("");
    state.freeRef();
    return description;

Returns

    

And regressed input:

TrainingTester.java:797 executed in 0.00 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(data)).flatMap(x -> {
      return RefArrays.stream(x);
    }).limit(1).map(x -> {
      String temp_18_0015 = x.prettyPrint();
      x.freeRef();
      return temp_18_0015;
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [ 0.6999999993573813, -0.1280000108546728, 0.079999992076361, 0.4959999918136421, -0.6079999723927151 ]

To produce the following output:

TrainingTester.java:808 executed in 0.00 seconds (0.000 gc):

    Result[] array = ConstantResult.batchResultArray(pop(RefUtil.addRef(data)));
    @Nullable
    Result eval = layer.eval(array);
    assert eval != null;
    TensorList tensorList = Result.getData(eval);
    String temp_18_0016 = tensorList.stream().limit(1).map(x -> {
      String temp_18_0017 = x.prettyPrint();
      x.freeRef();
      return temp_18_0017;
    }).reduce((a, b) -> a + "\n" + b).orElse("");
    tensorList.freeRef();
    return temp_18_0016;

Returns

    [ 0.3267237877819051, 0.1427528935031423, 0.1757592429874321, 0.2664309526980405, 0.08833312302947999 ]

Limited-Memory BFGS

Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.

TrainingTester.java:674 executed in 0.95 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new LBFGS());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setIterationsPerSample(100);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging
Reset training subject: 4274634776492
Reset training subject: 4274635955890
Adding measurement 5d5d897b to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 1.7690549327053968 < 1.7690549327053968. Total: 1
th(0)=1.7690549327053968;dx=-0.018925780932954404
Adding measurement 63e8c94 to history. Total: 1
New Minimum: 1.7690549327053968 > 1.7298743002721217
WOLFE (weak): th(2.154434690031884)=1.7298743002721217; dx=-0.01743856211205328 evalInputDelta=0.039180632433275075
Adding measurement 273780a7 to history. Total: 2
New Minimum: 1.7298743002721217 > 1.6939453863425733
END: th(4.308869380063768)=1.6939453863425733; dx=-0.015908291489866475 evalInputDelta=0.07510954636282352
Fitness changed from 1.7690549327053968 to 1.6939453863425733
Iteration 1 complete. Error: 1.6939453863425733 Total: 0.0135; Orientation: 0.0026; Line Search: 0.0075
Non-optimal measurement 1.6939453863425733 < 1.6939453863425733. Total: 3
LBFGS Accumulation History: 3 points
Non-optimal measurement 1.6939453863425733 < 1.6939453863425733. Total: 3
th(0)=1.6939453863425733;dx=-0.013409339578050882
Adding measurement 3a39321e to history. Total: 3
New Minimum: 1.6939453863425733 > 1.591455535075515
END: th(9.283177667225559)=1.591455535075515; dx=-0.008617892722741466 evalInputDelta=0.10248985126705823
Fitness changed from 1.6939453863425733 to 1.591455535075515
Iteration 2 complete. Error: 1.591455535075515 Total: 0.0065; Orientation: 0.0010; Line Search: 0.0042
Non-optimal measurement 1.591455535075515 < 1.591455535075515. Total: 4
Rejected: LBFGS Orientation magnitude: 2.440e+00, gradient 7.494e-02, dot -0.979; [7ab169aa-afba-47eb-a25e-441e2054c3b1 = 1.000/1.000e+00, 3cc76ce3-c84d-42c5-88ad-e03cb02a6b5a = 1.000/1.000e+00, 0f261389-c51e-4db3-aa8c-c4c1bf6b9bb3 = 1.000/1.000e+00, d35a5a6e-aca7-445b-9831-c8d2927ad795 = 1.000/1.000e+00, 7aef3e29-e506-4c78-82e3-fb2c369313ff = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.591455535075515, 1.6939453863425733, 1.7298743002721217, 1.7690549327053968
LBFGS Accumulation History: 3 points
Removed measurement 3a39321e to history. Total: 3
Adding measurement 1e339f85 to history. Total: 3
th(0)=1.591455535075515;dx=-0.005615725117688548
Adding measurement 7facfbcb to history. Total: 4
New Minimum: 1.591455535075515 > 1.5227236385555087
END: th(20.000000000000004)=1.5227236385555087; dx=-0.0012330258487457948 evalInputDelta=0.06873189652000633
Fitness changed from 1.591455535075515 to 1.5227236385555087
Iteration 3 complete. Error: 1.5227236385555087 Total: 0.0256; Orientation: 0.0180; Line Search: 0.0064
Non-optimal measurement 1.5227236385555087 < 1.5227236385555087. Total: 5
Rejected: LBFGS Orientation magnitude: 6.082e-01, gradient 1.992e-02, dot -0.978; [7ab169aa-afba-47eb-a25e-441e2054c3b1 = 1.000/1.000e+00, 7aef3e29-e506-4c78-82e3-fb2c369313ff = 1.000/1.000e+00, 0f261389-c51e-4db3-aa8c-c4c1bf6b9bb3 = 1.000/1.000e+00, d35a5a6e-aca7-445b-9831-c8d2927ad795 = 1.000/1.000e+00, 3cc76ce3-c84d-42c5-88ad-e03cb02a6b5a = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.5227236385555087, 1.591455535075515, 1.6939453863425733, 1.7298743002721217, 1.7690549327053968
Rejected: LBFGS Orientation magnitude: 5.975e-01, gradient 1.992e-02, dot -0.983; [7aef3e29-e506-4c78-82e3-fb2c369313ff = 1.000/1.000e+00, d35a5a6e-aca7-445b-9831-c8d2927ad795 = 1.000/1.000e+00, 7ab169aa-afba-47eb-a25e-441e2054c3b1 = 1.000/1.000e+00, 0f261389-c51e-4db3-aa8c-c4c1bf6b9bb3 = 1.000/1.000e+00, 3cc76ce3-c84d-42c5-88ad-e03cb02a6b5a = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.5227236385555087, 1.591455535075515, 1.6939453863425733, 1.7298743002721217
LBFGS Accumulation History: 3 points
Removed measurement 7facfbcb to history. Total: 4
Removed measurement 1e339f85 to history. Total: 3
Adding measurement 2cf6dfb1 to history. Total: 3
th(0)=1.5227236385555087;dx=-3.9660737483633904E-4
Adding measurement 5734ca0c to history. Total: 4
New Minimum: 1.5227236385555087 > 1.5170636843900633
WOLF (strong): th(43.088693800637685)=1.5170636843900633; dx=1.2868579467718478E-4 evalInputDelta=0.005659954165445402
Non-optimal measurement 1.5170639719787817 < 1.5170636843900633. Total: 5
END: th(21.544346900318843)=1.5170639719787817; dx=-1.300531589226398E-4 evalInputDelta=0.005659666576727007
Fitness changed from 1.5227236385555087 to 1.5170636843900633
Iteration 4 complete. Error: 1.5170636843900633 Total: 0.0486; Orientation: 0.0374; Line Search: 0.0088
Non-optimal measurement 1.5170636843900633 < 1.5170636843900633. Total: 5
Rejected: LBFGS Orientation magnitude: 3.330e-01, gradient 1.172e-02, dot -0.969; [7ab169aa-afba-47eb-a25e-441e2054c3b1 = 1.000/1.000e+00, 7aef3e29-e506-4c78-82e3-fb2c369313ff = 1.000/1.000e+00, 0f261389-c51e-4db3-aa8c-c4c1bf6b9bb3 = 1.000/1.000e+00, d35a5a6e-aca7-445b-9831-c8d2927ad795 = 1.000/1.000e+00, 3cc76ce3-c84d-42c5-88ad-e03cb02a6b5a = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.5170636843900633, 1.5227236385555087, 1.6939453863425733, 1.7298743002721217, 1.7690549327053968
Rejected: LBFGS Orientation magnitude: 3.219e-01, gradient 1.172e-02, dot -0.974; [7aef3e29-e506-4c78-82e3-fb2c369313ff = 1.000/1.000e+00, 0f261389-c51e-4db3-aa8c-c4c1bf6b9bb3 = 1.000/1.000e+00, 3cc76ce3-c84d-42c5-88ad-e03cb02a6b5a = 1.000/1.000e+00, d35a5a6e-aca7-445b-9831-c8d2927ad795 = 1.000/1.000e+00, 7ab169aa-afba-47eb-a25e-441e2054c3b1 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.5170636843900633, 1.5227236385555087, 1.6939453863425733, 1.7298743002721217
LBFGS Accumulation History: 3 points
Removed measurement 5734ca0c to history. Total: 4
Removed measurement 2cf6dfb1 to history. Total: 3
Adding measurement 70adaa01 to history. Total: 3
th(0)=1.5170636843900633;dx=-1.3733635175558617E-4
Non-optimal measurement 1.518322563262323 < 1.5170636843900633. Total: 4
Armijo: th(46.4158883361278)=1.518322563262323; dx=1.8537396372184046E-4 evalInputDelta=-0.0012588788722596167
Adding measurement 6c4d757a to history. Total: 4
New Minimum: 1.5170636843900633 > 1.515819925412146
WOLF (strong): th(23.2079441680639)=1.515819925412146; dx=2.867677541446113E-5 evalInputDelta=0.0012437589779172864
Non-optimal measurement 1.5162196857120098 < 1.515819925412146. Total: 5
END: th(7.735981389354634)=1.5162196857120098; dx=-8.102145586896903E-5 evalInputDelta=8.439986780535147E-4
Fitness changed from 1.5170636843900633 to 1.515819925412146
Iteration 5 complete. Error: 1.515819925412146 Total: 0.0475; Orientation: 0.0369; Line Search: 0.0091
Non-optimal measurement 1.515819925412146 < 1.515819925412146. Total: 5
Rejected: LBFGS Orientation magnitude: 6.978e-02, gradient 3.664e-03, dot -0.980; [3cc76ce3-c84d-42c5-88ad-e03cb02a6b5a = 1.000/1.000e+00, d35a5a6e-aca7-445b-9831-c8d2927ad795 = 1.000/1.000e+00, 7aef3e29-e506-4c78-82e3-fb2c369313ff = 1.000/1.000e+00, 0f261389-c51e-4db3-aa8c-c4c1bf6b9bb3 = 1.000/1.000e+00, 7ab169aa-afba-47eb-a25e-441e2054c3b1 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.515819925412146, 1.5170636843900633, 1.6939453863425733, 1.7298743002721217, 1.7690549327053968
Rejected: LBFGS Orientation magnitude: 6.922e-02, gradient 3.664e-03, dot -0.981; [3cc76ce3-c84d-42c5-88ad-e03cb02a6b5a = 1.000/1.000e+00, 7ab169aa-afba-47eb-a25e-441e2054c3b1 = 1.000/1.000e+00, 0f261389-c51e-4db3-aa8c-c4c1bf6b9bb3 = 1.000/1.000e+00, 7aef3e29-e506-4c78-82e3-fb2c369313ff = 1.000/1.000e+00, d35a5a6e-aca7-445b-9831-c8d2927ad795 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.515819925412146, 1.5170636843900633, 1.6939453863425733, 1.7298743002721217
LBFGS Accumulation History: 3 points
Removed measurement 6c4d757a to history. Total: 4
Removed measurement 70adaa01 to history. Total: 3
Adding measurement 53178b2d to history. Total: 3
th(0)=1.515819925412146;dx=-1.342725252828232E-5
Adding measurement 3e09c8c4 to history. Total: 4
New Minimum: 1.515819925412146 > 1.5157048631371286
END: th(16.66666666666667)=1.5157048631371286; dx=-3.89682944478594E-7 evalInputDelta=1.1506227501745236E-4
Fitness changed from 1.515819925412146 to 1.5157048631371286
Iteration 6 com

...skipping 22125 bytes...

w Minimum: 1.515701916965257 > 1.5157019169652033
WOLF (strong): th(22.384205409012257)=1.5157019169652033; dx=9.006297245578502E-16 evalInputDelta=5.3734794391857577E-14
Non-optimal measurement 1.5157019169652117 < 1.5157019169652033. Total: 5
END: th(11.192102704506128)=1.5157019169652117; dx=-2.3926684600134076E-15 evalInputDelta=4.529709940470639E-14
Fitness changed from 1.515701916965257 to 1.5157019169652033
Iteration 17 complete. Error: 1.5157019169652033 Total: 0.0469; Orientation: 0.0398; Line Search: 0.0058
Non-optimal measurement 1.5157019169652033 < 1.5157019169652033. Total: 5
Rejected: LBFGS Orientation magnitude: 5.515e-07, gradient 2.923e-08, dot -0.946; [7aef3e29-e506-4c78-82e3-fb2c369313ff = 1.000/1.000e+00, 7ab169aa-afba-47eb-a25e-441e2054c3b1 = 1.000/1.000e+00, 0f261389-c51e-4db3-aa8c-c4c1bf6b9bb3 = 1.000/1.000e+00, d35a5a6e-aca7-445b-9831-c8d2927ad795 = 1.000/1.000e+00, 3cc76ce3-c84d-42c5-88ad-e03cb02a6b5a = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.5157019169652033, 1.515701916965257, 1.6939453863425733, 1.7298743002721217, 1.7690549327053968
Rejected: LBFGS Orientation magnitude: 5.488e-07, gradient 2.923e-08, dot -0.947; [3cc76ce3-c84d-42c5-88ad-e03cb02a6b5a = 1.000/1.000e+00, 7ab169aa-afba-47eb-a25e-441e2054c3b1 = 1.000/1.000e+00, d35a5a6e-aca7-445b-9831-c8d2927ad795 = 1.000/1.000e+00, 7aef3e29-e506-4c78-82e3-fb2c369313ff = 1.000/1.000e+00, 0f261389-c51e-4db3-aa8c-c4c1bf6b9bb3 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.5157019169652033, 1.515701916965257, 1.6939453863425733, 1.7298743002721217
LBFGS Accumulation History: 3 points
Removed measurement a9c73fd to history. Total: 4
Removed measurement 7da7350b to history. Total: 3
Adding measurement 7b26c653 to history. Total: 3
th(0)=1.5157019169652033;dx=-8.543578177162813E-16
Adding measurement 42f04e99 to history. Total: 4
New Minimum: 1.5157019169652033 > 1.5157019169651935
WOLF (strong): th(24.11265432098767)=1.5157019169651935; dx=3.913434985267034E-17 evalInputDelta=9.769962616701378E-15
Non-optimal measurement 1.5157019169651957 < 1.5157019169651935. Total: 5
END: th(12.056327160493835)=1.5157019169651957; dx=-4.076117330425552E-16 evalInputDelta=7.549516567451064E-15
Fitness changed from 1.5157019169652033 to 1.5157019169651935
Iteration 18 complete. Error: 1.5157019169651935 Total: 0.0408; Orientation: 0.0329; Line Search: 0.0063
Non-optimal measurement 1.5157019169651935 < 1.5157019169651935. Total: 5
Rejected: LBFGS Orientation magnitude: 3.425e-07, gradient 1.385e-08, dot -0.941; [7aef3e29-e506-4c78-82e3-fb2c369313ff = 1.000/1.000e+00, d35a5a6e-aca7-445b-9831-c8d2927ad795 = 1.000/1.000e+00, 0f261389-c51e-4db3-aa8c-c4c1bf6b9bb3 = 1.000/1.000e+00, 3cc76ce3-c84d-42c5-88ad-e03cb02a6b5a = 1.000/1.000e+00, 7ab169aa-afba-47eb-a25e-441e2054c3b1 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.5157019169651935, 1.5157019169652033, 1.6939453863425733, 1.7298743002721217, 1.7690549327053968
Rejected: LBFGS Orientation magnitude: 3.408e-07, gradient 1.385e-08, dot -0.940; [0f261389-c51e-4db3-aa8c-c4c1bf6b9bb3 = 1.000/1.000e+00, d35a5a6e-aca7-445b-9831-c8d2927ad795 = 1.000/1.000e+00, 7aef3e29-e506-4c78-82e3-fb2c369313ff = 1.000/1.000e+00, 3cc76ce3-c84d-42c5-88ad-e03cb02a6b5a = 1.000/1.000e+00, 7ab169aa-afba-47eb-a25e-441e2054c3b1 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.5157019169651935, 1.5157019169652033, 1.6939453863425733, 1.7298743002721217
LBFGS Accumulation History: 3 points
Removed measurement 42f04e99 to history. Total: 4
Removed measurement 7b26c653 to history. Total: 3
Adding measurement 5ca453c1 to history. Total: 3
th(0)=1.5157019169651935;dx=-1.917925645668544E-16
Adding measurement 5d9a8866 to history. Total: 4
New Minimum: 1.5157019169651935 > 1.5157019169651913
WOLF (strong): th(25.974569468941517)=1.5157019169651913; dx=6.237577917322151E-18 evalInputDelta=2.220446049250313E-15
Non-optimal measurement 1.5157019169651917 < 1.5157019169651913. Total: 5
END: th(12.987284734470759)=1.5157019169651917; dx=-9.277749303324962E-17 evalInputDelta=1.7763568394002505E-15
Fitness changed from 1.5157019169651935 to 1.5157019169651913
Iteration 19 complete. Error: 1.5157019169651913 Total: 0.0422; Orientation: 0.0352; Line Search: 0.0058
Non-optimal measurement 1.5157019169651913 < 1.5157019169651913. Total: 5
Rejected: LBFGS Orientation magnitude: 1.659e-07, gradient 7.176e-09, dot -0.903; [3cc76ce3-c84d-42c5-88ad-e03cb02a6b5a = 1.000/1.000e+00, 7ab169aa-afba-47eb-a25e-441e2054c3b1 = 1.000/1.000e+00, 7aef3e29-e506-4c78-82e3-fb2c369313ff = 1.000/1.000e+00, 0f261389-c51e-4db3-aa8c-c4c1bf6b9bb3 = 1.000/1.000e+00, d35a5a6e-aca7-445b-9831-c8d2927ad795 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.5157019169651913, 1.5157019169651935, 1.6939453863425733, 1.7298743002721217, 1.7690549327053968
Rejected: LBFGS Orientation magnitude: 1.653e-07, gradient 7.176e-09, dot -0.903; [3cc76ce3-c84d-42c5-88ad-e03cb02a6b5a = 1.000/1.000e+00, 7ab169aa-afba-47eb-a25e-441e2054c3b1 = 1.000/1.000e+00, d35a5a6e-aca7-445b-9831-c8d2927ad795 = 1.000/1.000e+00, 7aef3e29-e506-4c78-82e3-fb2c369313ff = 1.000/1.000e+00, 0f261389-c51e-4db3-aa8c-c4c1bf6b9bb3 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.5157019169651913, 1.5157019169651935, 1.6939453863425733, 1.7298743002721217
LBFGS Accumulation History: 3 points
Removed measurement 5d9a8866 to history. Total: 4
Removed measurement 5ca453c1 to history. Total: 3
Adding measurement 16ef8bdc to history. Total: 3
th(0)=1.5157019169651913;dx=-5.1492741526021246E-17
Adding measurement 3c1b012b to history. Total: 4
New Minimum: 1.5157019169651913 > 1.5157019169651904
WOLF (strong): th(27.980256761265327)=1.5157019169651904; dx=9.068069649676561E-18 evalInputDelta=8.881784197001252E-16
Non-optimal measurement 1.5157019169651906 < 1.5157019169651904. Total: 5
END: th(13.990128380632664)=1.5157019169651906; dx=-2.1212335865514414E-17 evalInputDelta=6.661338147750939E-16
Fitness changed from 1.5157019169651913 to 1.5157019169651904
Iteration 20 complete. Error: 1.5157019169651904 Total: 0.0450; Orientation: 0.0376; Line Search: 0.0061
Non-optimal measurement 1.5157019169651904 < 1.5157019169651904. Total: 5
Rejected: LBFGS Orientation magnitude: 8.758e-08, gradient 4.165e-09, dot -0.939; [d35a5a6e-aca7-445b-9831-c8d2927ad795 = 1.000/1.000e+00, 3cc76ce3-c84d-42c5-88ad-e03cb02a6b5a = 1.000/1.000e+00, 7aef3e29-e506-4c78-82e3-fb2c369313ff = 1.000/1.000e+00, 7ab169aa-afba-47eb-a25e-441e2054c3b1 = 1.000/1.000e+00, 0f261389-c51e-4db3-aa8c-c4c1bf6b9bb3 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.5157019169651904, 1.5157019169651913, 1.6939453863425733, 1.7298743002721217, 1.7690549327053968
Rejected: LBFGS Orientation magnitude: 8.705e-08, gradient 4.165e-09, dot -0.940; [d35a5a6e-aca7-445b-9831-c8d2927ad795 = 1.000/1.000e+00, 0f261389-c51e-4db3-aa8c-c4c1bf6b9bb3 = 1.000/1.000e+00, 7aef3e29-e506-4c78-82e3-fb2c369313ff = 1.000/1.000e+00, 3cc76ce3-c84d-42c5-88ad-e03cb02a6b5a = 1.000/1.000e+00, 7ab169aa-afba-47eb-a25e-441e2054c3b1 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.5157019169651904, 1.5157019169651913, 1.6939453863425733, 1.7298743002721217
LBFGS Accumulation History: 3 points
Removed measurement 3c1b012b to history. Total: 4
Removed measurement 16ef8bdc to history. Total: 3
Adding measurement 29164d43 to history. Total: 3
th(0)=1.5157019169651904;dx=-1.734648553914483E-17
Non-optimal measurement 1.5157019169651904 < 1.5157019169651904. Total: 4
WOLF (strong): th(30.140817901234595)=1.5157019169651904; dx=8.337797459998672E-18 evalInputDelta=0.0
Non-optimal measurement 1.5157019169651904 < 1.5157019169651904. Total: 4
END: th(15.070408950617297)=1.5157019169651904; dx=-4.5043440394629024E-18 evalInputDelta=0.0
Fitness changed from 1.5157019169651904 to 1.5157019169651904
Static Iteration Total: 0.0585; Orientation: 0.0502; Line Search: 0.0070
Iteration 21 failed. Error: 1.5157019169651904
Previous Error: 0.0 -> 1.5157019169651904
Optimization terminated 21
Final threshold in iteration 21: 1.5157019169651904 (> 0.0) after 0.950s (< 30.000s)

Returns

    1.5157019169651904

This training apply resulted in the following configuration:

TrainingTester.java:785 executed in 0.00 seconds (0.000 gc):

    RefList<double[]> state = network.state();
    assert state != null;
    String description = state.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b)
        .orElse("");
    state.freeRef();
    return description;

Returns

    

And regressed input:

TrainingTester.java:797 executed in 0.00 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(data)).flatMap(x -> {
      return RefArrays.stream(x);
    }).limit(1).map(x -> {
      String temp_18_0015 = x.prettyPrint();
      x.freeRef();
      return temp_18_0015;
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [ 0.7000000128373545, -0.12800002074718903, 0.07999998390203811, 0.49599997466881396, -0.6079999506610229 ]

To produce the following output:

TrainingTester.java:808 executed in 0.00 seconds (0.000 gc):

    Result[] array = ConstantResult.batchResultArray(pop(RefUtil.addRef(data)));
    @Nullable
    Result eval = layer.eval(array);
    assert eval != null;
    TensorList tensorList = Result.getData(eval);
    String temp_18_0016 = tensorList.stream().limit(1).map(x -> {
      String temp_18_0017 = x.prettyPrint();
      x.freeRef();
      return temp_18_0017;
    }).reduce((a, b) -> a + "\n" + b).orElse("");
    tensorList.freeRef();
    return temp_18_0016;

Returns

    [ 0.32672379254322736, 0.14275289224697948, 0.1757592417428162, 0.2664309484213246, 0.08833312504565237 ]

TrainingTester.java:576 executed in 0.10 seconds (0.000 gc):

    return TestUtil.compare(title + " vs Iteration", runs);
Logging
Plotting range=[1.0, 0.1806137998805067], [23.0, 0.22889940434628217]; valueStats=DoubleSummaryStatistics{count=63, sum=96.023058, min=1.515702, average=1.524176, max=1.693945}
Plotting 20 points for GD
Plotting 23 points for CjGD
Plotting 20 points for LBFGS

Returns

Result

TrainingTester.java:579 executed in 0.01 seconds (0.000 gc):

    return TestUtil.compareTime(title + " vs Time", runs);
Logging
Plotting range=[0.0, 0.1806137998805067], [0.879, 0.22889940434628217]; valueStats=DoubleSummaryStatistics{count=63, sum=96.023058, min=1.515702, average=1.524176, max=1.693945}
Plotting 20 points for GD
Plotting 23 points for CjGD
Plotting 20 points for LBFGS

Returns

Result

Results

TrainingTester.java:350 executed in 0.00 seconds (0.000 gc):

    return grid(inputLearning, modelLearning, completeLearning);

Returns

Result

TrainingTester.java:353 executed in 0.00 seconds (0.000 gc):

    return new ComponentResult(null == inputLearning ? null : inputLearning.value,
        null == modelLearning ? null : modelLearning.value, null == completeLearning ? null : completeLearning.value);

Returns

    {"input":{ "LBFGS": { "type": "NonConverged", "value": 1.5157019169651904 }, "CjGD": { "type": "NonConverged", "value": 1.5157019169651904 }, "GD": { "type": "NonConverged", "value": 1.5157019169651904 } }, "model":null, "complete":null}

LayerTests.java:605 executed in 0.00 seconds (0.000 gc):

    throwException(exceptions.addRef());

Results

detailsresult
{"input":{ "LBFGS": { "type": "NonConverged", "value": 1.5157019169651904 }, "CjGD": { "type": "NonConverged", "value": 1.5157019169651904 }, "GD": { "type": "NonConverged", "value": 1.5157019169651904 } }, "model":null, "complete":null}OK
  {
    "result": "OK",
    "performance": {
      "execution_time": "2.840",
      "gc_time": "0.513"
    },
    "created_on": 1587006293059,
    "file_name": "trainingTest",
    "report": {
      "simpleName": "SubBatchLayerTest",
      "canonicalName": "com.simiacryptus.mindseye.layers.java.SubBatchLayerTest",
      "link": "https://github.com/SimiaCryptus/mindseye-java/tree/c9a1867488dc7e77a975f095285b5882c0486db6/src/test/java/com/simiacryptus/mindseye/layers/java/SubBatchLayerTest.java",
      "javaDoc": "The type Sub batch layer test."
    },
    "training_analysis": {
      "input": {
        "LBFGS": {
          "type": "NonConverged",
          "value": 1.5157019169651904
        },
        "CjGD": {
          "type": "NonConverged",
          "value": 1.5157019169651904
        },
        "GD": {
          "type": "NonConverged",
          "value": 1.5157019169651904
        }
      }
    },
    "archive": "s3://code.simiacrypt.us/tests/com/simiacryptus/mindseye/layers/java/SubBatchLayer/SubBatchLayerTest/trainingTest/202004160453",
    "id": "ecfef873-c036-4a5e-a992-a750e0d98a36",
    "report_type": "Components",
    "display_name": "Comparative Training",
    "target": {
      "simpleName": "SubBatchLayer",
      "canonicalName": "com.simiacryptus.mindseye.layers.java.SubBatchLayer",
      "link": "https://github.com/SimiaCryptus/mindseye-java/tree/c9a1867488dc7e77a975f095285b5882c0486db6/src/main/java/com/simiacryptus/mindseye/layers/java/SubBatchLayer.java",
      "javaDoc": "The type Sub batch layer."
    }
  }