For a django application, I need to process multiple netCDF files each day. Each file contains a 3-dimensional data array (latitude, longitude, time) of one variable with with a shape of (1000, 1000, 24).

In some views of the django application I want to access specific values of these data arrays, i.e. the time series on a specific location (given the latitude and longitude) or the entire map of a variable at a specific point in time.

In data science projects, I typically do this by opening the file with xarray (xarray.open_dataset('file.nc')) and using xarray's .sel() to select the desired values.

In the django application, I currently store the variables in a model that looks like the following:

class Geoid(models.model):
    latitude = models.FloatField()
    longitude = models.FloatField()

class Timestamp(models.model):
    time = models.DateTimeField()

class Variable(models.model):
    geoid = models.ForgeinKey(Geoid)
    time = models.ForgeinKey(Timestamp)
    value = models.FloatField()

I realized that inserting the 1000x1000x24 = 24,000,000 data points of each file into the database takes too much time even if I use django's .bulk_create() method.

This led me to my questions:

Would it actually be better to directly read the data in a django view from the netCDF files using xarray? What would be the pros and cons?

I looked into GeoDjango and the available RasterField but I could not find a method how to lookup pixel values (i.e. given the latitude and the longitude) of a RasterField in a fast and clean way.