Tkinter RuntimeError with repeated fine-tuning runs #95
Description
Hello,
I'm reprocessing a few dozen 2photon acquisitions and performing fine-tuning on the Ai93 model for each acquisition. We've tested it out and determined that we're going to need to fine-tune a model for each field of view.
We're storing each acquisition in a 3D HDF5 file, then we loop over the parent directories of each acquisition for creating the generator, test generator, and fine tuning objects. We then build the fine-tuning object and run.
This goes well for three to four models before we run into RuntimeErrors caused by Tkinter of all things. I've attached the script we're running. This hasn't always been a problem. We've been able to run a bunch of models overnight before, but that might have been with tiff stacks.
These two resources helped me out. It seems that when matplotlib creates the plots for training and validation losses across fine-tuning, it's invoking Tkinter as the back end. Simplest solution for me is to disable creating these plots for now.
Project-MONAI/tutorials#94
https://stackoverflow.com/questions/27147300/matplotlib-tcl-asyncdelete-async-handler-deleted-by-the-wrong-thread
Are there any other users experiencing this issue? I've also included the output for conda list to show my environment.
import os
import sys
import glob
import h5py
import pathlib
from datetime import datetime
sys.path.append('C:\\Users\\greg\\Documents\\Python\\deepinterpolation-0.1.5')
from deepinterpolation.cli.fine_tuning import FineTuning
def main():
loss_fun = "mean_squared_error"
sessions_to_run = [
"D:\\seagate_data_for_ben\\TSeries-08252022-1332-002\\Registered\\Channel1",
"D:\\seagate_data_for_ben\\TSeries-08252022-1332-003\\Registered\\Channel1"
# More sessions, etc.
]
time_deltas = []
for i in range(0, len(sessions_to_run)):
parent_dir = sessions_to_run[i]
finetuning_params = {}
generator_param = {}
generator_test_param = {}
# Grab some info from datafile
data_file_maybe = glob.glob(parent_dir + "\\registered.h5")
assert len(data_file_maybe) == 1, "Either HDF5 file for registered data was not found, or more than one was found. Check parent directory."
data_file = data_file_maybe[0]
f = h5py.File(data_file, "a")
rename = False
try:
total_frames = f['mov'].shape[0]
rename = True
except:
total_frames = f['data'].shape[0]
if total_frames < 14000:
num_training_samples = total_frames
else:
num_training_samples = 14000
if rename:
f["data"] = f["mov"]
del f["mov"]
f.close()
# Parameters used for the validation step each epoch
generator_test_param["name"] = "OphysGenerator"
generator_test_param["pre_frame"] = 30
generator_test_param["post_frame"] = 30
generator_test_param["data_path"] = data_file
generator_test_param["batch_size"] = 10
generator_test_param["start_frame"] = 1
generator_test_param["end_frame"] = -1
generator_test_param["total_samples"] = 500
generator_test_param["randomize"] = 1
generator_test_param["pre_post_omission"] = 0
#Parameters used for the main data generator
generator_param["name"] = "OphysGenerator"
generator_param["pre_frame"] = 30
generator_param["post_frame"] = 30
generator_param["data_path"] = data_file
generator_param["batch_size"] = 10
generator_param["start_frame"] = 1
generator_param["end_frame"] = -1
generator_param["total_samples"] = num_training_samples
generator_param["randomize"] = 1
generator_param["pre_post_omission"] = 0
# Parameters used for the training process
finetuning_params["name"] = "transfer_trainer"
# Change this path to any model you wish to improve
filename = "2019_09_11_23_32_unet_single_1024_mean_absolute_error_Ai93-0450.h5"
local_path = \
os.path.join(
"C:\\Users\\greg\\Documents\\Python\\deepinterpolation-0.1.5\\sample_data",
filename
)
finetuning_params["model_source"] = {
"local_path": local_path
}
steps_per_epoch = 200
finetuning_params["steps_per_epoch"] = steps_per_epoch
finetuning_params[
"period_save"
] = 5
finetuning_params["learning_rate"] = 0.0001
finetuning_params["loss"] = loss_fun
finetuning_params["output_dir"] = parent_dir
finetuning_params["apply_learning_decay"] = False
finetuning_params["caching_validation"] = False
finetuning_params["use_multiprocessing"] = False
finetuning_params["nb_workers"] = 8
args = {
"finetuning_params": finetuning_params,
"generator_params": generator_param,
"test_generator_params": generator_test_param,
"output_full_args": True
}
t_start = datetime.now()
finetuning_obj = FineTuning(input_data=args, args=[])
finetuning_obj.run()
t_stop = datetime.now()
time_deltas.append(t_stop - t_start)
print(str(t_stop - t_start))
if __name__=='__main__':
main()
INFO:FineTuning:fine tuning job finished - finalizing output model
Saved model to disk
0:54:37.463883
INFO:FineTuning:wrote D:\seagate_data_for_ben\TSeries-11262022-1053-001\Registered\Channel1\2023_04_03_22_32_training_full_args.json
INFO:FineTuning:wrote D:\seagate_data_for_ben\TSeries-11262022-1053-001\Registered\Channel1\2023_04_03_22_32_finetuning.json
INFO:FineTuning:wrote D:\seagate_data_for_ben\TSeries-11262022-1053-001\Registered\Channel1\2023_04_03_22_32_generator.json
INFO:FineTuning:wrote D:\seagate_data_for_ben\TSeries-11262022-1053-001\Registered\Channel1\2023_04_03_22_32_test_generator.json
WARNING:tensorflow:`period` argument is deprecated. Please use `save_freq` to specify the frequency in number of batches seen.
WARNING:tensorflow:`period` argument is deprecated. Please use `save_freq` to specify the frequency in number of batches seen.
Exception ignored in: <function Image.__del__ at 0x000001ECA89DFE50>
Traceback (most recent call last):
File "C:\Users\greg\anaconda3\envs\deepinterpolation\lib\tkinter\__init__.py", line 4017, in __del__
self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x000001ECA89CA550>
Traceback (most recent call last):
File "C:\Users\greg\anaconda3\envs\deepinterpolation\lib\tkinter\__init__.py", line 363, in __del__
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x000001ECA89CA550>
Traceback (most recent call last):
File "C:\Users\greg\anaconda3\envs\deepinterpolation\lib\tkinter\__init__.py", line 363, in __del__
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x000001ECA89CA550>
Traceback (most recent call last):
File "C:\Users\greg\anaconda3\envs\deepinterpolation\lib\tkinter\__init__.py", line 363, in __del__
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x000001ECA89CA550>
Traceback (most recent call last):
File "C:\Users\greg\anaconda3\envs\deepinterpolation\lib\tkinter\__init__.py", line 363, in __del__
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.__del__ at 0x000001ECA89DFE50>
Traceback (most recent call last):
File "C:\Users\greg\anaconda3\envs\deepinterpolation\lib\tkinter\__init__.py", line 4017, in __del__
self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x000001ECA89CA550>
Traceback (most recent call last):
File "C:\Users\greg\anaconda3\envs\deepinterpolation\lib\tkinter\__init__.py", line 363, in __del__
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x000001ECA89CA550>
Traceback (most recent call last):
File "C:\Users\greg\anaconda3\envs\deepinterpolation\lib\tkinter\__init__.py", line 363, in __del__
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x000001ECA89CA550>
Traceback (most recent call last):
File "C:\Users\greg\anaconda3\envs\deepinterpolation\lib\tkinter\__init__.py", line 363, in __del__
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x000001ECA89CA550>
Traceback (most recent call last):
File "C:\Users\greg\anaconda3\envs\deepinterpolation\lib\tkinter\__init__.py", line 363, in __del__
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Tcl_AsyncDelete: async handler deleted by the wrong thread
(deepinterpolation) C:\Users\greg\Documents\Python Scripts>conda list
# packages in environment at C:\Users\greg\anaconda3\envs\deepinterpolation:
#
# Name Version Build Channel
absl-py 1.4.0 pypi_0 pypi
alembic 1.4.1 pypi_0 pypi
argschema 2.0.2 pypi_0 pypi
astunparse 1.6.3 pypi_0 pypi
ca-certificates 2023.01.10 haa95532_0
cachetools 5.3.0 pypi_0 pypi
certifi 2022.12.7 py38haa95532_0
charset-normalizer 3.0.1 pypi_0 pypi
click 8.1.3 pypi_0 pypi
cloudpickle 2.2.1 pypi_0 pypi
colorama 0.4.6 pypi_0 pypi
contourpy 1.0.7 pypi_0 pypi
cycler 0.11.0 pypi_0 pypi
databricks-cli 0.17.4 pypi_0 pypi
docker 6.0.1 pypi_0 pypi
entrypoints 0.4 pypi_0 pypi
flask 2.2.2 pypi_0 pypi
flatbuffers 2.0.7 pypi_0 pypi
fonttools 4.38.0 pypi_0 pypi
gast 0.4.0 pypi_0 pypi
gitdb 4.0.10 pypi_0 pypi
gitpython 3.1.30 pypi_0 pypi
google-auth 2.16.0 pypi_0 pypi
google-auth-oauthlib 0.4.6 pypi_0 pypi
google-pasta 0.2.0 pypi_0 pypi
greenlet 2.0.1 pypi_0 pypi
grpcio 1.51.1 pypi_0 pypi
h5py 3.8.0 pypi_0 pypi
idna 3.4 pypi_0 pypi
importlib-metadata 6.0.0 pypi_0 pypi
itsdangerous 2.1.2 pypi_0 pypi
jinja2 3.1.2 pypi_0 pypi
keras 2.7.0 pypi_0 pypi
keras-preprocessing 1.1.2 pypi_0 pypi
kiwisolver 1.4.4 pypi_0 pypi
libclang 15.0.6.1 pypi_0 pypi
libffi 3.4.2 hd77b12b_6
mako 1.2.4 pypi_0 pypi
markdown 3.4.1 pypi_0 pypi
markupsafe 2.1.2 pypi_0 pypi
marshmallow 3.0.0rc6 pypi_0 pypi
matplotlib 3.6.3 pypi_0 pypi
mlflow 1.14.1 pypi_0 pypi
nibabel 5.0.0 pypi_0 pypi
numpy 1.24.1 pypi_0 pypi
oauthlib 3.2.2 pypi_0 pypi
openssl 1.1.1s h2bbff1b_0
opt-einsum 3.3.0 pypi_0 pypi
packaging 23.0 pypi_0 pypi
pandas 1.5.3 pypi_0 pypi
pillow 9.4.0 pypi_0 pypi
pip 22.3.1 py38haa95532_0
prometheus-client 0.15.0 pypi_0 pypi
prometheus-flask-exporter 0.21.0 pypi_0 pypi
protobuf 3.20.3 pypi_0 pypi
pyasn1 0.4.8 pypi_0 pypi
pyasn1-modules 0.2.8 pypi_0 pypi
pyjwt 2.6.0 pypi_0 pypi
pyparsing 3.0.9 pypi_0 pypi
python 3.8.16 h6244533_2
python-dateutil 2.8.2 pypi_0 pypi
python-editor 1.0.4 pypi_0 pypi
pytz 2022.7.1 pypi_0 pypi
pywin32 305 pypi_0 pypi
pyyaml 6.0 pypi_0 pypi
querystring-parser 1.2.4 pypi_0 pypi
requests 2.28.2 pypi_0 pypi
requests-oauthlib 1.3.1 pypi_0 pypi
rsa 4.9 pypi_0 pypi
scipy 1.10.0 pypi_0 pypi
setuptools 65.6.3 py38haa95532_0
six 1.16.0 pypi_0 pypi
smmap 5.0.0 pypi_0 pypi
sqlalchemy 1.4.46 pypi_0 pypi
sqlite 3.40.1 h2bbff1b_0
sqlparse 0.4.3 pypi_0 pypi
tabulate 0.9.0 pypi_0 pypi
tensorboard 2.11.2 pypi_0 pypi
tensorboard-data-server 0.6.1 pypi_0 pypi
tensorboard-plugin-wit 1.8.1 pypi_0 pypi
tensorflow 2.7.0 pypi_0 pypi
tensorflow-estimator 2.7.0 pypi_0 pypi
tensorflow-io-gcs-filesystem 0.30.0 pypi_0 pypi
termcolor 2.2.0 pypi_0 pypi
tifffile 2023.1.23.1 pypi_0 pypi
typing-extensions 4.4.0 pypi_0 pypi
urllib3 1.26.14 pypi_0 pypi
vc 14.2 h21ff451_1
vs2015_runtime 14.27.29016 h5e58377_2
waitress 2.1.2 pypi_0 pypi
websocket-client 1.4.2 pypi_0 pypi
werkzeug 2.2.2 pypi_0 pypi
wheel 0.37.1 pyhd3eb1b0_0
wincertstore 0.2 py38haa95532_2
wrapt 1.14.1 pypi_0 pypi
zipp 3.11.0 pypi_0 pypi