I have a very large table with 100,000's of rows with a key and a timestamp.
I currently use batch of AWS servers that are using NodeJS and PG to query for the oldest last_updated timestamp, perform the function, and then update the row with the NOW().
My issue is I am trying to find a way scale with more processes and optimize my row checkouts. I started with each process selecting 1000 at a time and each one having a multiple of 1000 for OFFSET, doing the operation, and then updating
I started to read about SELECT using FOR UPDATE and SKIP LOCK but seems like this could have some performance impacts? Also can't find a clear way to do the SELECT/UPDATE in the same query or do I keep doing single updates at a time similar to this post but seems like this may not be good for larger operations like this?
implementing an UPDATE on SELECT in Postgres
Has anyone approached this type of setup? I also have been debating do I need to build my own middleware that manages a pool of work items and then the workers use that table to select/delete?