I am building a NLP App using python. I heard the Spacy is proper to NLP and installed it. How should I use the Japanese engine from Spacy?
pip install -u spacy
or
python -m pip -u Spacy
What shall I install more?
You should download and install the language package.
pip spacy download ja_core_news_lg
or
python -m spacy download ja_core_news_lg
If you face an issue, please try this.
python -m spacy download ja_core_news_sm
Great question!
After installing Spacy, you must download and correctly install the language engine you have in mind or the one that best suits your problem.
In your case I would try:
python -m spacy download ja_core_news_sm
After that you can try to load it from your code with the following lines:
import spacy
nlp = spacy.load("ja_core_news_sm")
Here you have a more complex example of sentence analysis from official doc:
import spacy
from spacy.lang.ja.examples import sentences
nlp = spacy.load("ja_core_news_sm")
doc = nlp(sentences[0])
print(doc.text)
for token in doc:
print(token.text, token.pos_, token.dep_)
Spacy has extensive support for multiple languages (including Japanese) and for most of them it also has model / pipeline alternatives for different types of problems. I strongly recommend you to read here more about spacy language models, different ways to install and handle models dependencies, among other important facts.
Last but not least, Spacy has multiple models for Japanese language (see here):
ja_core_news_sm: Japanese pipeline optimized for CPU. Components: tok2vec, morphologizer, parser, senter, ner, attribute_ruler.ja_core_news_md: Same pipeline but using word embeddings to improve quality (480k keys, 20k unique vectors (300 dimensions))ja_core_news_lg: Same that ja_core_news_md with a bigger vocabulary (480k keys, 480k unique vectors (300 dimensions))ja_core_news_trf: Japanese transformer pipeline (cl-tohoku/bert-base-japanese-char-v2). Components: transformer, morphologizer, parser, ner, attribute_ruler.The appropriate model for you will depend on the needs and restrictions that you have in your problem. For a cutting-edge model if you don't have memory restrictions I would recommend starting with ja_core_news_lg or ja_core_news_trf.
I hope that this complementary information would be useful!
The other answers about installing models will also work, but you can use Japanese in spaCy without a model using pip install spacy[ja], which will pull in the required dependencies.