I have a simple apache beam pipeline which reads bigquery >> creates a parquet file in GCS >> From the GCS path, I read the metadata information and yield a JSON object with a custom ParDo function >> I write as text back to the GCS.

There is no runtime error but, when I check the file created in the destination GCS path, it has 0 bytes. I checked the whether the Pcollection is empty by using the beam.ParDo(print) function. When I do this it prints the JSON object. I already tried convert the object as str, list and dict. Everything works well with the beam.ParDo(print) statement but fails to write any data to GCS. Please let me know, what is my mistake? Thanks in advance :)

class CreateMetadata(beam.DoFn):
    def process(self, element, *args, **kwargs):
        # element here gets a GCS path as gs://.....
        ### Some internal process ###
        # This function is called only once at the end of the internal 
        # process have a JSON object as entity data
        yield entity_data

with beam.Pipeline(options=pipeline_options) as pipeline:
    table = (
        pipeline | 'ReadTableFromBigQuery' >> beam.io.Read(beam.io.BigQuerySource(query=bigquery.read_sql(query_string=query),
                                                                                  use_standard_sql=True))
        | 'writeAsParquetToGCS' >> pq.WriteToParquet(OUTPUT, schema=bigquery.get_parquet_schema_from_bq(),
                                                     num_shards=1)
        | 'CreateMetadata' >> beam.ParDo(CreateMetadata())
        # | beam.ParDo(print)

        | 'WriteToGCS' >> beam.io.WriteToText(file_path_prefix=OUTPUT.replace('Some GCS path'), file_name_suffix='.json',num_shards=1)
        # Now after the pipeline runs when I check the GCS path there is a file created with 
        # 0 bytes???

    )```