TF2-HyperparameterSearch-1

2024-01-08 20:31:46

超参数搜索

为什么要超参数搜索

*神经网络有很多训练过程中不变的参数
*网络结构参数:几层, 每层宽度(神经元个数), 每层的激活函数
*训练参数: batch_size , 学习率 , 学习率衰减算法等

手工去试耗费人力

网格搜索

遗传算法

*对自然界的模拟
*A:初始化候选参数结合 -> 训练 -> 得到的模型指标作为生存概率
*B:选择 -> 交叉 -> 变异 -> 产生下一代集合
*C:重新到A

启发式搜索

*Auto-ML
*使用循环神经网络生成参数
*使用强化学习来进行反馈,使用模型来生成参数

import tensorflow as tf
import matplotlib as mlp
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
import pandas as pd
import sklearn
import os
import sys
from tensorflow import keras
print("tf.__version__ = ",tf.__version__)
print(keras.__version__)

tf.__version__ =  2.0.0-alpha0
2.2.4-tf

#Get data
from sklearn.datasets import fetch_california_housing
housing = fetch_california_housing()
data = housing.data
target = housing.target

#split train test valid data
from sklearn.model_selection import train_test_split
x_train_all , x_test , y_train_all , y_test = train_test_split(
    data,target,random_state = 7 , test_size = 0.25
)
print(x_train_all.shape)
x_train , x_valid , y_train , y_valid = train_test_split(
    x_train_all,y_train_all,random_state = 11 , test_size = 0.25
)
print(x_train.shape)

(15480, 8)
(11610, 8)

#Normarlization
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
x_train_scaled = scaler.fit_transform(x_train)
x_valid_scaled = scaler.transform(x_valid)
x_test_scaled = scaler.transform(x_test)

#bulit model


#learning rate [1e-4 , 3e-3, 1e-3 ,3e-3 , 3e-2 , 1e-2 , 3e-2]
#单参数for循环
# w = w + grad * learning_rate
learning_rates = [1e-4 , 3e-4, 1e-3 ,3e-3 , 1e-2 , 3e-2]
historys = []  
for lr in learning_rates:
    
    model = keras.models.Sequential([
        keras.layers.Dense(
            30,
            activation='relu',
            input_shape=x_train.shape[1:]
        ),
        keras.layers.Dense(1)
    ])
    model.summary()
    
    optimizer = keras.optimizers.SGD(lr)
    model.compile(
        loss="mean_squared_error",
        optimizer=optimizer
    )
    callbacks = [keras.callbacks.EarlyStopping(patience=5,min_delta=1e-2)]
    
    
    #Train
    history = model.fit(
        x_train_scaled,
        y_train,
        validation_data = (x_valid_scaled,y_valid),
        epochs = 100,
        callbacks = callbacks
    )
    historys.append(history)

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense (Dense)                (None, 30)                270       
_________________________________________________________________
dense_1 (Dense)              (None, 1)                 31        
=================================================================
Total params: 301
Trainable params: 301
Non-trainable params: 0
_________________________________________________________________
Train on 11610 samples, validate on 3870 samples
Epoch 1/100
11610/11610 [==============================] - 1s 80us/sample - loss: 4.2326 - val_loss: 3.6016
Epoch 2/100
11610/11610 [==============================] - 1s 64us/sample - loss: 3.0963 - val_loss: 2.6659
Epoch 3/100
11610/11610 [==============================] - 1s 64us/sample - loss: 2.3809 - val_loss: 2.0708
Epoch 4/100
11610/11610 [==============================] - 1s 63us/sample - loss: 1.9106 - val_loss: 1.6816
Epoch 5/100
11610/11610 [==============================] - 1s 64us/sample - loss: 1.5842 - val_loss: 1.4146
Epoch 6/100
11610/11610 [==============================] - 1s 63us/sample - loss: 1.3553 - val_loss: 1.2324
Epoch 7/100
11610/11610 [==============================] - 1s 65us/sample - loss: 1.1888 - val_loss: 1.1036
Epoch 8/100
11610/11610 [==============================] - 1s 64us/sample - loss: 1.0666 - val_loss: 1.0115
Epoch 9/100
11610/11610 [==============================] - 1s 65us/sample - loss: 0.9759 - val_loss: 0.9450
Epoch 10/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.9085 - val_loss: 0.8977
Epoch 11/100
11610/11610 [==============================] - 1s 66us/sample - loss: 0.8583 - val_loss: 0.8621
Epoch 12/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.8201 - val_loss: 0.8354
Epoch 13/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.7912 - val_loss: 0.8152
Epoch 14/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.7687 - val_loss: 0.7995
Epoch 15/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.7512 - val_loss: 0.7872
Epoch 16/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.7372 - val_loss: 0.7771
Epoch 17/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.7258 - val_loss: 0.7689
Epoch 18/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.7163 - val_loss: 0.7618
Epoch 19/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.7082 - val_loss: 0.7556
Epoch 20/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.7013 - val_loss: 0.7500
Epoch 21/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.6950 - val_loss: 0.7450
Epoch 22/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.6894 - val_loss: 0.7404
Epoch 23/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.6843 - val_loss: 0.7360
Epoch 24/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.6795 - val_loss: 0.7319
Epoch 25/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.6751 - val_loss: 0.7279
Epoch 26/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.6708 - val_loss: 0.7241
Epoch 27/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.6668 - val_loss: 0.7204
Epoch 28/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.6629 - val_loss: 0.7167
Epoch 29/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.6592 - val_loss: 0.7131
Epoch 30/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.6556 - val_loss: 0.7096
Epoch 31/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.6521 - val_loss: 0.7061
Epoch 32/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.6487 - val_loss: 0.7027
Epoch 33/100
11610/11610 [==============================] - 1s 65us/sample - loss: 0.6454 - val_loss: 0.6994
Epoch 34/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.6422 - val_loss: 0.6961
Epoch 35/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.6390 - val_loss: 0.6928
Epoch 36/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.6359 - val_loss: 0.6896
Epoch 37/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.6329 - val_loss: 0.6865
Epoch 38/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.6299 - val_loss: 0.6834
Epoch 39/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.6270 - val_loss: 0.6804
Epoch 40/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.6242 - val_loss: 0.6773
Epoch 41/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.6214 - val_loss: 0.6744
Epoch 42/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.6187 - val_loss: 0.6715
Epoch 43/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.6160 - val_loss: 0.6686
Epoch 44/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.6134 - val_loss: 0.6657
Epoch 45/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.6108 - val_loss: 0.6630
Epoch 46/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.6083 - val_loss: 0.6602
Epoch 47/100
11610/11610 [==============================] - 1s 65us/sample - loss: 0.6058 - val_loss: 0.6575
Epoch 48/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.6033 - val_loss: 0.6548
Epoch 49/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.6009 - val_loss: 0.6521
Epoch 50/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.5985 - val_loss: 0.6495
Epoch 51/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.5962 - val_loss: 0.6469
Epoch 52/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.5939 - val_loss: 0.6443
Epoch 53/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.5917 - val_loss: 0.6418
Epoch 54/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.5895 - val_loss: 0.6393
Epoch 55/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.5873 - val_loss: 0.6368
Epoch 56/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.5852 - val_loss: 0.6343
Epoch 57/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.5830 - val_loss: 0.6320
Epoch 58/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.5810 - val_loss: 0.6296
Epoch 59/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.5789 - val_loss: 0.6272
Epoch 60/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.5769 - val_loss: 0.6249
Epoch 61/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.5749 - val_loss: 0.6226
Epoch 62/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.5729 - val_loss: 0.6204
Epoch 63/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.5710 - val_loss: 0.6182
Epoch 64/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.5691 - val_loss: 0.6160
Epoch 65/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.5672 - val_loss: 0.6138
Epoch 66/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.5654 - val_loss: 0.6116
Epoch 67/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.5635 - val_loss: 0.6096
Epoch 68/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.5617 - val_loss: 0.6075
Epoch 69/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.5600 - val_loss: 0.6055
Epoch 70/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.5582 - val_loss: 0.6034
Epoch 71/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.5565 - val_loss: 0.6014
Epoch 72/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.5548 - val_loss: 0.5995
Epoch 73/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.5531 - val_loss: 0.5975
Epoch 74/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.5515 - val_loss: 0.5957
Epoch 75/100
11610/11610 [==============================] - 1s 60us/sample - loss: 0.5499 - val_loss: 0.5938
Epoch 76/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.5483 - val_loss: 0.5919
Epoch 77/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.5467 - val_loss: 0.5901
Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_2 (Dense)              (None, 30)                270       
_________________________________________________________________
dense_3 (Dense)              (None, 1)                 31        
=================================================================
Total params: 301
Trainable params: 301
Non-trainable params: 0
_________________________________________________________________
Train on 11610 samples, validate on 3870 samples
Epoch 1/100
11610/11610 [==============================] - 1s 72us/sample - loss: 3.2795 - val_loss: 2.0846
Epoch 2/100
11610/11610 [==============================] - 1s 62us/sample - loss: 1.5364 - val_loss: 1.2477
Epoch 3/100
11610/11610 [==============================] - 1s 62us/sample - loss: 1.0293 - val_loss: 0.9800
Epoch 4/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.8595 - val_loss: 0.8856
Epoch 5/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.7882 - val_loss: 0.8426
Epoch 6/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.7521 - val_loss: 0.8157
Epoch 7/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.7289 - val_loss: 0.7950
Epoch 8/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.7108 - val_loss: 0.7769
Epoch 9/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.6950 - val_loss: 0.7603
Epoch 10/100
11610/11610 [==============================] - 1s 65us/sample - loss: 0.6806 - val_loss: 0.7446
Epoch 11/100
11610/11610 [==============================] - 1s 65us/sample - loss: 0.6672 - val_loss: 0.7298
Epoch 12/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.6546 - val_loss: 0.7156
Epoch 13/100
11610/11610 [==============================] - 1s 65us/sample - loss: 0.6428 - val_loss: 0.7026
Epoch 14/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.6317 - val_loss: 0.6902
Epoch 15/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.6212 - val_loss: 0.6787
Epoch 16/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.6115 - val_loss: 0.6677
Epoch 17/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.6024 - val_loss: 0.6573
Epoch 18/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.5938 - val_loss: 0.6478
Epoch 19/100
11610/11610 [==============================] - 1s 65us/sample - loss: 0.5857 - val_loss: 0.6388
Epoch 20/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.5781 - val_loss: 0.6303
Epoch 21/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.5709 - val_loss: 0.6221
Epoch 22/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.5641 - val_loss: 0.6144
Epoch 23/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.5577 - val_loss: 0.6071
Epoch 24/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.5516 - val_loss: 0.5999
Epoch 25/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.5457 - val_loss: 0.5935
Epoch 26/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.5402 - val_loss: 0.5872
Epoch 27/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.5349 - val_loss: 0.5811
Epoch 28/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.5299 - val_loss: 0.5755
Epoch 29/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.5251 - val_loss: 0.5700
Epoch 30/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.5206 - val_loss: 0.5647
Epoch 31/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.5163 - val_loss: 0.5597
Epoch 32/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.5122 - val_loss: 0.5550
Epoch 33/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.5083 - val_loss: 0.5506
Epoch 34/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.5046 - val_loss: 0.5462
Epoch 35/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.5012 - val_loss: 0.5424
Epoch 36/100
11610/11610 [==============================] - 1s 66us/sample - loss: 0.4979 - val_loss: 0.5385
Epoch 37/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.4947 - val_loss: 0.5349
Epoch 38/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.4917 - val_loss: 0.5315
Epoch 39/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.4889 - val_loss: 0.5281
Epoch 40/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.4862 - val_loss: 0.5248
Epoch 41/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.4836 - val_loss: 0.5217
Epoch 42/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.4812 - val_loss: 0.5190
Epoch 43/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.4788 - val_loss: 0.5162
Epoch 44/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.4766 - val_loss: 0.5133
Epoch 45/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.4746 - val_loss: 0.5105
Epoch 46/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.4726 - val_loss: 0.5084
Epoch 47/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.4707 - val_loss: 0.5062
Epoch 48/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.4688 - val_loss: 0.5039
Epoch 49/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.4671 - val_loss: 0.5017
Epoch 50/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.4654 - val_loss: 0.4999
Epoch 51/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.4637 - val_loss: 0.4973
Epoch 52/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.4622 - val_loss: 0.4958
Epoch 53/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.4607 - val_loss: 0.4942
Epoch 54/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.4592 - val_loss: 0.4925
Model: "sequential_2"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_4 (Dense)              (None, 30)                270       
_________________________________________________________________
dense_5 (Dense)              (None, 1)                 31        
=================================================================
Total params: 301
Trainable params: 301
Non-trainable params: 0
_________________________________________________________________
Train on 11610 samples, validate on 3870 samples
Epoch 1/100
11610/11610 [==============================] - 1s 71us/sample - loss: 1.4974 - val_loss: 0.9273
Epoch 2/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.8025 - val_loss: 0.8286
Epoch 3/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.7442 - val_loss: 0.7829
Epoch 4/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.7062 - val_loss: 0.7466
Epoch 5/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.6750 - val_loss: 0.7142
Epoch 6/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.6476 - val_loss: 0.6851
Epoch 7/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.6235 - val_loss: 0.6601
Epoch 8/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.6021 - val_loss: 0.6388
Epoch 9/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.5835 - val_loss: 0.6196
Epoch 10/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.5666 - val_loss: 0.6022
Epoch 11/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.5517 - val_loss: 0.5859
Epoch 12/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.5385 - val_loss: 0.5733
Epoch 13/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.5269 - val_loss: 0.5612
Epoch 14/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.5168 - val_loss: 0.5501
Epoch 15/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.5081 - val_loss: 0.5408
Epoch 16/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.5001 - val_loss: 0.5324
Epoch 17/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.4930 - val_loss: 0.5250
Epoch 18/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.4868 - val_loss: 0.5181
Epoch 19/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.4810 - val_loss: 0.5113
Epoch 20/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.4756 - val_loss: 0.5055
Epoch 21/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.4707 - val_loss: 0.4999
Epoch 22/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.4664 - val_loss: 0.4953
Epoch 23/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.4622 - val_loss: 0.4900
Epoch 24/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.4586 - val_loss: 0.4859
Epoch 25/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.4550 - val_loss: 0.4821
Epoch 26/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.4516 - val_loss: 0.4783
Epoch 27/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.4484 - val_loss: 0.4748
Epoch 28/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.4455 - val_loss: 0.4719
Epoch 29/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.4428 - val_loss: 0.4685
Epoch 30/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.4399 - val_loss: 0.4653
Epoch 31/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.4376 - val_loss: 0.4628
Epoch 32/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.4348 - val_loss: 0.4605
Epoch 33/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.4328 - val_loss: 0.4573
Epoch 34/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.4303 - val_loss: 0.4543
Epoch 35/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.4283 - val_loss: 0.4524
Epoch 36/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.4264 - val_loss: 0.4504
Epoch 37/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.4244 - val_loss: 0.4474
Epoch 38/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.4225 - val_loss: 0.4450
Epoch 39/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.4207 - val_loss: 0.4426
Epoch 40/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.4191 - val_loss: 0.4410
Epoch 41/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.4175 - val_loss: 0.4389
Epoch 42/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.4160 - val_loss: 0.4372
Epoch 43/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.4145 - val_loss: 0.4355
Epoch 44/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.4129 - val_loss: 0.4342
Epoch 45/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.4117 - val_loss: 0.4334
Epoch 46/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.4103 - val_loss: 0.4303
Model: "sequential_3"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_6 (Dense)              (None, 30)                270       
_________________________________________________________________
dense_7 (Dense)              (None, 1)                 31        
=================================================================
Total params: 301
Trainable params: 301
Non-trainable params: 0
_________________________________________________________________
Train on 11610 samples, validate on 3870 samples
Epoch 1/100
11610/11610 [==============================] - 1s 71us/sample - loss: 1.1967 - val_loss: 0.6288
Epoch 2/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.5528 - val_loss: 0.5565
Epoch 3/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.5040 - val_loss: 0.5199
Epoch 4/100
11610/11610 [==============================] - 1s 65us/sample - loss: 0.4811 - val_loss: 0.5051
Epoch 5/100
11610/11610 [==============================] - 1s 66us/sample - loss: 0.4636 - val_loss: 0.4811
Epoch 6/100
11610/11610 [==============================] - 1s 65us/sample - loss: 0.4530 - val_loss: 0.4710
Epoch 7/100
11610/11610 [==============================] - 1s 66us/sample - loss: 0.4448 - val_loss: 0.4661
Epoch 8/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.4392 - val_loss: 0.4560
Epoch 9/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.4345 - val_loss: 0.4538
Epoch 10/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.4296 - val_loss: 0.4467
Epoch 11/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.4270 - val_loss: 0.4412
Epoch 12/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.4219 - val_loss: 0.4391
Epoch 13/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.4193 - val_loss: 0.4354
Epoch 14/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.4155 - val_loss: 0.4305
Epoch 15/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.4136 - val_loss: 0.4258
Epoch 16/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.4109 - val_loss: 0.4254
Epoch 17/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.4082 - val_loss: 0.4239
Epoch 18/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.4048 - val_loss: 0.4198
Epoch 19/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.4041 - val_loss: 0.4158
Epoch 20/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.4002 - val_loss: 0.4135
Epoch 21/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.3992 - val_loss: 0.4109
Epoch 22/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.3971 - val_loss: 0.4110
Epoch 23/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.3941 - val_loss: 0.4081
Epoch 24/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.3937 - val_loss: 0.4063
Epoch 25/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.3907 - val_loss: 0.4017
Epoch 26/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.3887 - val_loss: 0.4004
Epoch 27/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.3877 - val_loss: 0.3988
Epoch 28/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.3875 - val_loss: 0.3999
Model: "sequential_4"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_8 (Dense)              (None, 30)                270       
_________________________________________________________________
dense_9 (Dense)              (None, 1)                 31        
=================================================================
Total params: 301
Trainable params: 301
Non-trainable params: 0
_________________________________________________________________
Train on 11610 samples, validate on 3870 samples
Epoch 1/100
11610/11610 [==============================] - 1s 72us/sample - loss: 0.9782 - val_loss: 0.7611
Epoch 2/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.6947 - val_loss: 0.4780
Epoch 3/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.4401 - val_loss: 0.4418
Epoch 4/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.4207 - val_loss: 0.4653
Epoch 5/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.4076 - val_loss: 0.4363
Epoch 6/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.3966 - val_loss: 0.4069
Epoch 7/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.3889 - val_loss: 0.4016
Epoch 8/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.3989 - val_loss: 0.3973
Epoch 9/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.3811 - val_loss: 0.3908
Epoch 10/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.3782 - val_loss: 0.3884
Epoch 11/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.3755 - val_loss: 0.3870
Epoch 12/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.3731 - val_loss: 0.3804
Epoch 13/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.3698 - val_loss: 0.3892
Epoch 14/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.3679 - val_loss: 0.3817
Epoch 15/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.3653 - val_loss: 0.3780
Epoch 16/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.3646 - val_loss: 0.3749
Epoch 17/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.4333 - val_loss: 0.3921
Model: "sequential_5"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_10 (Dense)             (None, 30)                270       
_________________________________________________________________
dense_11 (Dense)             (None, 1)                 31        
=================================================================
Total params: 301
Trainable params: 301
Non-trainable params: 0
_________________________________________________________________
Train on 11610 samples, validate on 3870 samples
Epoch 1/100
11610/11610 [==============================] - 1s 72us/sample - loss: 5.8795 - val_loss: 0.8450
Epoch 2/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.5768 - val_loss: 0.4919
Epoch 3/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.4555 - val_loss: 0.4367
Epoch 4/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.4437 - val_loss: 0.4472
Epoch 5/100
11610/11610 [==============================] - 1s 66us/sample - loss: 0.4109 - val_loss: 0.4106
Epoch 6/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.3987 - val_loss: 0.4107
Epoch 7/100
11610/11610 [==============================] - 1s 61us/sample - loss: 0.4086 - val_loss: 0.4026
Epoch 8/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.3899 - val_loss: 0.4080
Epoch 9/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.3892 - val_loss: 0.3873
Epoch 10/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.3782 - val_loss: 0.3947
Epoch 11/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.3792 - val_loss: 0.4038
Epoch 12/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.3800 - val_loss: 0.3878
Epoch 13/100
11610/11610 [==============================] - 1s 64us/sample - loss: 0.3719 - val_loss: 0.3980
Epoch 14/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.3694 - val_loss: 0.3703
Epoch 15/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.3713 - val_loss: 0.3838
Epoch 16/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.3647 - val_loss: 0.3797
Epoch 17/100
11610/11610 [==============================] - 1s 62us/sample - loss: 0.3595 - val_loss: 0.3627
Epoch 18/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.3553 - val_loss: 0.3611
Epoch 19/100
11610/11610 [==============================] - 1s 63us/sample - loss: 0.3542 - val_loss: 0.3694

# history = model.fit(
#     x_train_scaled,
#     y_train,
#     validation_data = (x_valid_scaled,y_valid),
#     epochs = 100,
#     callbacks = callbacks
# )

def plot_learning_curves(history):
    pd.DataFrame(history.history).plot(figsize=(8,5))
    plt.grid(True)
    plt.gca().set_ylim(0,1)
    plt.show()
    
for lr , history in zip(learning_rates,historys):
    print("Learning_rate:",lr)
    plot_learning_curves(history)