I am training a Keras (Tensorflow backend, Python, on MacBook) and am getting an error in the early stopping callback in fit_generator function. The error is as follows:
RuntimeWarning: Early stopping conditioned on metric `val_loss` which is not available. Available metrics are:
(self.monitor, ','.join(list(logs.keys()))),
RuntimeWarning: Can save best model only with val_acc available, skipping.
'skipping.' % (self.monitor), RuntimeWarning
[local-dir]/lib/python3.6/site-packages/keras/callbacks.py:497: RuntimeWarning: Early stopping conditioned on metric `val_loss` which is not available. Available metrics are:
(self.monitor, ','.join(list(logs.keys()))), RuntimeWarning
[local-dir]/lib/python3.6/site-packages/keras/callbacks.py:406: RuntimeWarning: Can save best model only with val_acc available, skipping.
'skipping.' % (self.monitor), RuntimeWarning)
Traceback (most recent call last):
:
[my-code]
:
File "[local-dir]/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "[local-dir]/lib/python3.6/site-packages/keras/engine/training.py", line 2213, in fit_generator
callbacks.on_epoch_end(epoch, epoch_logs)
File "[local-dir]/lib/python3.6/site-packages/keras/callbacks.py", line 76, in on_epoch_end
callback.on_epoch_end(epoch, logs)
File "[local-dir]/lib/python3.6/site-packages/keras/callbacks.py", line 310, in on_epoch_end
self.progbar.update(self.seen, self.log_values, force=True)
AttributeError: 'ProgbarLogger' object has no attribute 'log_values'
My code is as follows (which looks OK):
:
ES = EarlyStopping(monitor="val_loss", min_delta=0.001, patience=3, mode="min", verbose=1)
:
self.model.fit_generator(
generator = train_batch,
validation_data = valid_batch,
validation_steps = validation_steps,
steps_per_epoch = steps_per_epoch,
epochs = epochs,
callbacks = [ES],
verbose = 1,
workers = 3,
max_queue_size = 8)
The error message appears to relate to the early stopping callback but the callback looks OK. Also the error states that the val_loss is not appropriate, but I am not sure why... one more unusual thing about this is that the error only occurs when I use smaller data sets.
Any help is appreciated.
If the error only occurs when you use smaller datasets, you're very likely using datasets small enough to not have a single sample in the validation set.
Thus it cannot calculate a validation loss.
The error occurs to us because we forgot to set validation_data in fit() method, while used 'callbacks': [keras.callbacks.EarlyStopping(monitor='val_loss', patience=1)],
Code causing error is:
self.model.fit(
x=x_train,
y=y_train,
callbacks=[keras.callbacks.EarlyStopping(monitor='val_loss', patience=1)],
verbose=True)
Adding validation_data=(self.x_validate, self.y_validate), in fit() fixed:
self.model.fit(
x=x_train,
y=y_train,
callbacks=[keras.callbacks.EarlyStopping(monitor='val_loss', patience=1)],
validation_data=(x_validate, y_validate),
verbose=True)
I up-voted the previous answer as it gave me the insight to verify the data and inputs to the fit_generator function and find out what the root cause of the issue actually was. In summary, in cases where my dataset was small, I calculated validation_steps and steps_per_epoch which turned out to be zero (0) which caused the error.
I suppose the better longer-term answer, perhaps for the Keras team, is to cause an error/exception in fit_generator when these values are zero, which would probably lead to a better understanding about how to address this issue.