You should define it in the compile function :

optimizer = keras.optimizers.Adam(lr=0.01)
model.compile(loss='mse',
              optimizer=optimizer,
              metrics=['categorical_accuracy'])

Looking at your comment, if you want to change the learning rate after the beginning you need to use a scheduler : link

Edit with your code and scheduler:

from keras.models import Sequential
from keras.layers import Dense
import keras
import numpy as np

def lr_scheduler(epoch, lr):
    if epoch > 50:
        lr = 0.001
        return lr
    return lr

model = Sequential()

model.add(Dense(1, input_shape=(10,)))

optimizer = keras.optimizers.Adam(lr=0.01)
model.compile(loss='mse',
              optimizer=optimizer)

callbacks = [keras.callbacks.LearningRateScheduler(lr_scheduler, verbose=1)]

model.fit(np.random.randn(50,10), np.random.randn(50), epochs=100, callbacks=callbacks)

You can change lr during training with

from keras.callbacks import LearningRateScheduler

# This is a sample of a scheduler I used in the past
def lr_scheduler(epoch, lr):
    decay_rate = 0.85
    decay_step = 1
    if epoch % decay_step == 0 and epoch:
        return lr * pow(decay_rate, np.floor(epoch / decay_step))
    return lr

Apply scheduler to your model

callbacks = [LearningRateScheduler(lr_scheduler, verbose=1)]

model = build_model(pretrained_model=ka.InceptionV3, input_shape=(224, 224, 3))
history = model.fit(train, callbacks=callbacks, epochs=EPOCHS, verbose=1)

Suppose that you use Adam optimizer in keras, you'd want to define your optimizer before you compile your model with it.

For example, you can define

myadam = keras.optimizers.Adam(learning_rate=0.1)

Then, you compile your model with this optimizer.

I case you want to change your optimizer (with different type of optimizer or with different learning rate), you can define a new optimizer and compile your existing model with the new optimizer.

Hope this helps!

There is another way, you have to find the variable that holds the learning rate and assign it another value.

optimizer = tf.keras.optimizers.Adam(0.001)
optimizer.learning_rate.assign(0.01)
print(optimizer.learning_rate)

output:

<tf.Variable 'learning_rate:0' shape=() dtype=float32, numpy=0.01>

You can change the learning rate as follows:

from keras import backend as K
K.set_value(model.optimizer.learning_rate, 0.001)

Included into your complete example it looks as follows:

from keras.models import Sequential
from keras.layers import Dense
from keras import backend as K
import keras
import numpy as np

model = Sequential()

model.add(Dense(1, input_shape=(10,)))

optimizer = keras.optimizers.Adam(lr=0.01)
model.compile(loss='mse', optimizer=optimizer)

print("Learning rate before first fit:", model.optimizer.learning_rate.numpy())

model.fit(np.random.randn(50,10), np.random.randn(50), epochs=50, verbose=0)

# Change learning rate to 0.001 and train for 50 more epochs
K.set_value(model.optimizer.learning_rate, 0.001)
print("Learning rate before second fit:", model.optimizer.learning_rate.numpy())

model.fit(np.random.randn(50,10), 
          np.random.randn(50), 
          initial_epoch=50, 
          epochs=50,
          verbose=0)

I've just tested this with keras 2.3.1. Not sure why the approach didn't seem to work for you.

Some time ago I had a project for which I needed something similar. My idea to change the learning rate was to compile a new model with the new rate, then load the parameter weights from de old model to the new one.

For your example:

from keras.models import Sequential
from keras.layers import Dense
import keras
import numpy as np


# Initial model

model = Sequential()
model.add(Dense(1, input_shape=(10,)))

optimizer = keras.optimizers.Adam(lr=0.01)
model.compile(loss='mse', optimizer=optimizer)

model.fit(np.random.randn(50,10), np.random.randn(50), epochs=50)


# Change learning rate to 0.001 and train for 50 more epochs

new_model = Sequential()
new_model.add(Dense(1, input_shape=(10,)))

optimizer = keras.optimizers.Adam(lr=0.001)
new_model.compile(loss='mse', optimizer=optimizer)

new_model.set_weights(model.get_weights())
model = new_model

model.fit(np.random.randn(50,10), np.random.randn(50), initial_epoch=50, epochs=50)

With this you could see a worse fit of your model in the first epochs because ADAM uses previous steps to optimize and you will lose them.

Hope it helps someone!