I'm using python and I want to get the TFIDF representation for a large corpus of data, I'm using the following code to convert the docs into their TFIDF form.
from sklearn.feature_extraction.text import TfidfVectorizer
tfidf_vectorizer = TfidfVectorizer(
min_df=1, # min count for relevant vocabulary
max_features=4000, # maximum number of features
strip_accents='unicode', # replace all accented unicode char
# by their corresponding ASCII char
analyzer='word', # features made of words
token_pattern=r'\w{1,}', # tokenize only words of 4+ chars
ngram_range=(1, 1), # features made of a single tokens
use_idf=True, # enable inverse-document-frequency reweighting
smooth_idf=True, # prevents zero division for unseen words
sublinear_tf=False)
tfidf_df = tfidf_vectorizer.fit_transform(df['text'])
Here I pass a parameter max_features. The vectorizer will select the best features and return a scipy sparse matrix. Problem is I dont know which features are getting selected and how do I map those feature names back to the scipy matrix I get? Basically for the n selected features from the m number of documents, I want a m x n matrix with the selected features as the column names instead of their integer ids. How do I accomplish this?
You can use tfidf_vectorizer.get_feature_names(). This will print feature names selected (terms selected) from the raw documents.
You can also use tfidf_vectorizer.vocabulary_ attribute to get a dict which will map the feature names to their indices, but will not be sorted. The array from get_feature_names() will be sorted by index.
use tfidf_vectorizer.vocabulary_, this gives a mapping from the features (terms back to the indices)