I created a Delta Table in ADLS Gen 1 with the following code in Databricks:

df.write.format("delta").mode("overwrite").saveAsTable("db.my_tbl", path ='adl://organisation.azuredatalakestore.net/folder_name/my_data')

Sometimes, I re-run the code above to generate a new version of the my_tbl table. As usual with delta tables, a history is build and it must regulary be optimized and vaccumed. Now, I am often retraining a ML Model in Azure Machine Learning Studio and am wondering if it possible to register a specific version of the delta table?

Currently, even after vaccuming, all my delta files (including older versions) are registered in Azure ML Studio when reading the parquet files from my_data folder! That is because I can not lower the retention period of the delta table below 168h except turning of spark.databricks.delta.retentionDurationCheck.enabled. I do not want to turn it off.

I register my dataset through the ML Studio Interface as a File Dataset (not Tabular Dataset). This registration looks like this:

Right now I see only the option to create a copy of my_data and reading it instead. Is there another way? Do you know if I can specify something in the path to point to the "right" .parquet files (belonging to a specific delta table version)?