I am building a model for classifying whether the person in the image is wearing a mask or not. I used EfficientNetB0 model with custom data augmentation (sequential) layer all stacked with the help of Functional layer. After I saved the model (in h5 format) and loaded the saved model, the accuracy on the test dataset was different.

The accuracy on the test dataset originally was around 98% After loading the saved model, the accuracy on the test dataset plummeted to 91%

I used image_dataset_from_directory to load the images for train, test and validation sets.

train_data = keras.preprocessing.image_dataset_from_directory(
    train_dir, image_size=(224, 224),
    color_mode="rgb", batch_size=64, label_mode="categorical"
)

Then created a prefetch dataset

train_dataset = train_data.shuffle(buffer_size=1000).prefetch(buffer_size=tf.data.AUTOTUNE)

This is the custom data augmentation layer:

data_aug = keras.Sequential([
    keras.layers.RandomRotation(0.3),
    keras.layers.RandomContrast(0.3)
])

The model

inputs = keras.layers.Input(shape=(224, 224, 3))
x = data_aug(inputs)
x = eff_model(x, training=False)
x = keras.layers.GlobalAveragePooling2D()(x)
x = keras.layers.Dense(3, activation="softmax")(x)

model = keras.Model(inputs=inputs, outputs=x)

model.compile(loss=keras.losses.CategoricalCrossentropy(), metrics=["accuracy"], 
              optimizer=keras.optimizers.Adam(0.001))

history = model.fit(train_dataset, steps_per_epoch=len(train_dataset), epochs=20,
                    validation_data=val_dataset, validation_steps=len(val_dataset))

This is the accuracy my model achieved originally on the test dataset.

Saving the model and reloading the saved model:

model.save("face_Detect.h5")
load_model = keras.models.load_model("/content/face_Detect.h5")

This is the accuracy achieved on the loaded model:

I have no idea because the accuracy on the test dataset for the original model and loaded model should be similar but this is a huge difference. Is this a bug in Tensorflow 2.7.0 or am I making some horrendous mistake ?