I have a sequence to sequence learning model which works fine and able to predict some outputs. The problem is I have no idea how to convert the output back to text sequence.
This is my code.
from keras.preprocessing.text import Tokenizer,base_filter
from keras.preprocessing.sequence import pad_sequences
from keras.models import Sequential
from keras.layers import Dense
txt1="""What makes this problem difficult is that the sequences can vary in length,
be comprised of a very large vocabulary of input symbols and may require the model
to learn the long term context or dependencies between symbols in the input sequence."""
#txt1 is used for fitting
tk = Tokenizer(nb_words=2000, filters=base_filter(), lower=True, split=" ")
tk.fit_on_texts(txt1)
#convert text to sequence
t= tk.texts_to_sequences(txt1)
#padding to feed the sequence to keras model
t=pad_sequences(t, maxlen=10)
model = Sequential()
model.add(Dense(10,input_dim=10))
model.add(Dense(10,activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam',metrics=['accuracy'])
#predicting new sequcenc
pred=model.predict(t)
#Convert predicted sequence to text
pred=??
You can use directly the inverse tokenizer.sequences_to_texts function.
text = tokenizer.sequences_to_texts(<list-of-integer-equivalent-encodings>)
I have tested the above and it works as expected.
PS.: Take extra care to make the argument be the list of the integer encodings and not the One Hot ones.
Here is a solution I found:
reverse_word_map = dict(map(reversed, tokenizer.word_index.items()))
I had to resolve the same problem, so here is how I ended up doing it (inspired by @Ben Usemans reversed dictionary).
# Importing library
from keras.preprocessing.text import Tokenizer
# My texts
texts = ['These are two crazy sentences', 'that I want to convert back and forth']
# Creating a tokenizer
tokenizer = Tokenizer(lower=True)
# Building word indices
tokenizer.fit_on_texts(texts)
# Tokenizing sentences
sentences = tokenizer.texts_to_sequences(texts)
>sentences
>[[1, 2, 3, 4, 5], [6, 7, 8, 9, 10, 11, 12, 13]]
# Creating a reverse dictionary
reverse_word_map = dict(map(reversed, tokenizer.word_index.items()))
# Function takes a tokenized sentence and returns the words
def sequence_to_text(list_of_indices):
# Looking up words in dictionary
words = [reverse_word_map.get(letter) for letter in list_of_indices]
return(words)
# Creating texts
my_texts = list(map(sequence_to_text, sentences))
>my_texts
>[['these', 'are', 'two', 'crazy', 'sentences'], ['that', 'i', 'want', 'to', 'convert', 'back', 'and', 'forth']]