I have written pyton code which reads data from mongodb collection and writes it to gcs bucket.
My dataflow pipeline is as below:
p
| ReadFromMongoDB(uri='mongodb+srv://mongo_url',db='db_name',coll='column_name',bucket_auto=True)
| "WriteMyFile" >> beam.io.WriteToText('gs://bucket_name/file123.json')
p.run()
My dataflow pipeline fails with below error:
pymongo.errors.ConfigurationError: The "dnspython" module must be installed to use mongodb+srv:// URIs
As per this document, pymongo (which is required dependency to solve dnspython issue) is already installed in dataflow workers. What else needed to be done?
This is likely because dataflow worker only installs basic requirements of pymongo. And dnspython is included in pymongo extra dependencies so you'll have to use pymongo[srv] as a dependency to resolve dnspython module. You can try to include pymongo[srv]== in you requirements.txt and then submit the pipeline with --requirements_file flag, it will make dataflow worker install the required python dependencies when the container is launched.