Use of hyperthreading in hyperparameter sweeps #10
Description
Hello back,
I aimed to run your code using Python hyperthreading module (I don't know if you are familiar with it) to speed up hyperparameter sweeps, as you may find in the file attached (it is in a .txt extension as I could not directly update a .py file).
However, most cases show how results hardly improve and strongly fluctuate even when using very low learning rates, opposite to when you do so sequentially without hyperthreading (where both loss and val_loss keep gradually decreasing).
Do you know if this is just an incompatibility issue or am I doing something wrong? How do you actually deal with hyperparameter sweeps? Are you using any scikit_tools or you do it manually such as I?
Thank you in advance, and best regards! May you have a nice weekend!
# x_data, y_data generated from a .csv file
SAMPLE = x_data.shape[0]
N_OUTPUTS = x_data.shape[1]
N_INPUTS = y_data.shape[1]
N_EPOCHS = [6000]
N_LAYERS = [1]
N_HIDDEN = [100]
N_MIXES = [8, 12]
DROPOUT = [0]
ACT_FUNCTION = 'tanh'
LR = [0.00005, 0.00001]
BATCH_SIZE = [NSAMPLE]
PTEST = [0.3]
beta1 = [0.9]
beta2 = [0.999]
def MDN(N_MIXES, LR, BATCH_SIZE, N_LAYERS, N_HIDDEN, DROPOUT, PTEST, N_EPOCHS, beta1, beta2):
model = keras.Sequential()
model.add(Dense(N_HIDDEN, batch_input_shape = (None, N_INPUTS), activation = ACT_FUNCTION))
model.add(Dropout(DROPOUT))
for layer in range(N_LAYERS - 1):
model.add(Dense(N_HIDDEN, activation = ACT_FUNCTION))
model.add(Dropout(DROPOUT))
model.add(mdn.MDN(N_OUTPUTS, N_MIXES))
return model
adam = keras.optimizers.Adam(lr=LR, beta_1 = beta1, beta_2 = beta2)
model.compile(loss=mdn.get_mixture_loss_func(N_OUTPUTS,N_MIXES), optimizer=adam)
H = model.fit(x=x_data, y=y_data, verbose=0, batch_size=BATCH_SIZE, epochs=N_EPOCHS, validation_split=PTEST)
return N_MIXES, LR, BATCH_SIZE, N_LAYERS, N_HIDDEN, DROPOUT, beta1, beta2, H.history['loss'], H.history['val_loss']
params = list(itertools.product(*[N_MIXES, LR, BATCH_SIZE, N_LAYERS, N_HIDDEN, DROPOUT, PTEST, N_EPOCHS, beta1, beta2]))
pool = ThreadPool()
results = pool.starmap(MDN, params)
pool.close()
pool.join()