0
376
ViewsNifi: Delete files from S3 when they are X days old
I have Nifi flow which is supposed to delete any files from S3 that are older than 7 days. I have used the following setup to get it done.
My UpdateAttribute processor has a epoch_now attribute that gets current epoch time.
On my RouteOnAttribute I have the following logic to filter out files that are younger than 7 days using this expression: ${epoch_now:minus(${s3.lastModified}):ge(604800000)}
The problem is that ListS3 processor will maintain state and it won't re-list all the files the next time to calculate if any files are expiring and need to be deleted. I looked around but I could not find something like Get* processor which would not maintain state. How do I fix this flow so that it run periodically and keeps deleting files that are 7 days old?
1 answers
Answer question0
You are correct, NiFi does not currently have a processor to query S3 that way.
This might be a better fit for an S3 Lifecycle Rule. You can configure a rule for specific key prefixes, so S3 will automagically delete objects after 7 days. From the S3 console:
- Select your bucket
- Select Properties
- Expand the Lifecycle section
- Click Add rule
There is a wizard-style interface to walk you through the configuration.