Is it possible to create new products using scrapy in django ?
Whenever I run scrapy.py I get the following error:
raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: Requested setting INSTALLED_APPS, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.
my app models file
from django.db import models
# Create your models here.
class Product(models.Model):
name = models.CharField(max_length=50)
age = models.IntegerField()
content = models.TextField(max_length=400)
def __str__(self):
return self.name
scrapy.py file
from models import Product
Product.objects.create(name='BMW', age=25, content='cool car')
Yes of course it's possible, you just to import django and configure the DJANGO_SETTINGS_MODULE variable :
import django
import os
# Configure settings for project
# Need to run this before calling models from application!
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'app_name.settings') # Replace app_name with your application name.
# Import settings
django.setup()
# Import the models now
from app_name.models import Product
# Use the imported model
product = Product.objects.create(...)
More infos on this post
For the error you are having above you might be setting up Django incorrectly.
Therefore, this question have two problems to be solved:
**1. Fix Django Setup issue **
For the first question I advise you to setup Django normally creating a project.... Then, create an app as python manage.py startapp product_app ADD the product_app into settings.INSTALLED_APPS as:
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'product_app' <----- without this the models.py won't work.
]
Then, go to your models.py and create the model Product for example. Migrate the model into your database.
Alright your first question might be answered.
2. Add crawled data into a Django model.
The second part is send data to the product table using the Product model (the representation of the product table).
The fields you will create will depend on the data you need. Let say, the data I want to store from the crawled data are: name, description, price, stock and if it is available.
Great, lets create our model:
product_app/
from django.db import models
class Product(models.Model):
name = models.CharField(max_length=250, unique=True)
description = models.TextField(blank=True)
price = models.DecimalField(max_digits=10, decimal_places=2)
stock = models.IntegerField(default =0)
available = models.BooleanField(default=True)
class Meta:
verbose_name = 'product'
verbose_name_plural = 'products'
def __str__(self):
return str(self.name)
Since Django accepts dictionaries as objects to be used for queries you can get this crawled data into a dictionary/json for after be pushed into the Product table.
Alright, now for your part you will need to somehow get the data as dictionary as similar this and push into the Product table:
data = {
"name":"bmw",
"description":"dont waste your money",
"price":25000.00,
"stock":4,
"available":True
}
query example:
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>>
>>> from product_app.models import Product
>>>
>>> data = {"name":"bmw", "description":"dont waste your money", "price":25000.00, "stock":4, "available":True,}
>>>
>>> product = Product(**data)
>>> product.save()
>>>
>>> product
<Product: bmw>
From here you can do whatever you want with the data. Install Django-Rest-Framework and display the data as an REST-API or display the data in a template.
Bear in mind that you can create the queries needed into views.py or wherever you want.