I have implemented a tied weights Auto-encoder in Keras and have successfully trained it.
My goal is to use only the decoder part of the Auto-encoder as the last layer of another network, to fine tune both the network and the decoder.
Thing is, as you can see below from the summary, the decoder has no parameters with my tied weights implementation, so there is nothing to be fine tuned. (decoder.get_weights() returns [])
My question is: Should I change the implementation of the tied weights, so that the tied layer can still hold weights, that is the transposed weights of the encoder? If yes, how?
Or am I just way off?
Below is the summary of the autoencoder model as well as the class of the tied Dense layer (slightly modified from https://github.com/nanopony/keras-convautoencoder/blob/master/autoencoder_layers.py.)
Layer (type) Output Shape Param # Connected to
====================================================================================================
encoded (Dense) (None, Enc_dim) 33000 dense_input_1[0][0]
____________________________________________________________________________________________________
tieddense_1 (TiedtDense) (None, Out_Dim) 0 encoded[0][0]
====================================================================================================
Total params: 33,000
Trainable params: 33,000
Non-trainable params: 0
________________________________________________________________________
class TiedtDense(Dense):
def __init__(self, output_dim, master_layer, init='glorot_uniform', activation='linear', weights=None,
W_regularizer=None, b_regularizer=None, activity_regularizer=None,
W_constraint=None, b_constraint=None, input_dim=None, **kwargs):
self.master_layer = master_layer
super(TiedtDense, self).__init__(output_dim, **kwargs)
def build(self, input_shape):
assert len(input_shape) >= 2
input_dim = input_shape[-1]
self.input_dim = input_dim
self.W = tf.transpose(self.master_layer.W)
self.b = K.zeros((self.output_dim,))
self.params = [self.b]
self.regularizers = []
if self.W_regularizer:
self.W_regularizer.set_param(self.W)
self.regularizers.append(self.W_regularizer)
if self.b_regularizer:
self.b_regularizer.set_param(self.b)
self.regularizers.append(self.b_regularizer)
if self.activity_regularizer:
self.activity_regularizer.set_layer(self)
self.regularizers.append(self.activity_regularizer)
if self.initial_weights is not None:
self.set_weights(self.initial_weights)
del self.initial_weights
It's been more than 2 years since this question was asked, but this answer might still be relevant for some.
The function Layer.get_weights() retrieves from self.trainable_weights and self.non_trainable_weights (see keras.engine.base_layer.Layer.weights). In your custom layer, your weights self.W and self.b are not being added to any of these collections and that's why the layer has 0 parameters.
You could tweak your implementation as follows:
class TiedtDense(Dense):
def __init__(self, output_dim, master_layer, **kwargs):
self.master_layer = master_layer
super(TiedtDense, self).__init__(output_dim, **kwargs)
def build(self, input_shape):
assert len(input_shape) >= 2
input_dim = input_shape[-1]
self.input_dim = input_dim
self.kernel = tf.transpose(self.master_layer.kernel)
self.bias = K.zeros((self.units,))
self.trainable_weights.append(self.kernel)
self.trainable_weights.append(self.bias)
NOTE: I am excluding the regularizers and constraints for simplicity. If you want those, please refer to keras.engine.base_layer.Layer.add_weight.