Both scan and changing all the data in Dynamodb are not feasible.

You can keep all the dynamodb keys in a cache like redis, There can be a separate job who takes keys from redis and puts to sqs, where the lambda is listening. The redis keys can be kept up to date using dynamodb streams.

Dynamo doesn't offer a good way to trigger a lambda for items that already exist, there are a few ways however you could approach this problem:

Option 1 (Scan in lambda in small batches):

You mentioned that you where concerned with the lambda not having enough resources to scan all of the items in the table you could try operating on the data in smaller chunks to avoid hitting resource limitations. Lambdas have a max execution time of 15 minutes witch should be enough for most jobs. (Please note that in Lambda CPU scales with Memory so depending on the job over provisioning memory could actually save you money by reducing the time the function takes to complete.)

Option 2 (Scheduled ECS Task Fargate):

In ECS using Fargate you can serverlessly create tasks on a cron schedule. If you are worried about resource limits you can provision up to 4 vCPU and 32GB of memory per task, which will make it far less likely you will hit the resource limit. Here is some documentation on how to set that up.

Option 3 (Process items using Dynamo triggers):

You could can configure your dynamo table to trigger a lambda whenever data in the table is Inserted, Modified, or Removed, you can then process items as they come in or as they change.You can even configure it to batch changes up to 10 items to reduce lambda invocations. Here is a link to the documentation.

Note: This Method doesn't trigger for items already in the table. However you can get around this by writing a script to update a arbitrary field on those items.

When you say you want to "trigger" for each item in it it's not 100% clear what you mean. In general I think DynamoDB streams for that, but something has to cause the records to be processed by the stream. That is often done with a simple UpdateItem to each record, setting a field, that likely isn't part of your data, to something like the current time, or something else unique. From there you will get each record to process through a lambda triggered on the stream.

The 100% serverless way to loop through the data is the following:

• Step function that calls a lambda to run through a batch of records.
• Lambda function scans the records, looping through the pages of records.
  • Function should accept a payload that optionally supplies the paging information.
  • Function should check the lambda context to see how much time is remaining before staring each loop and exiting if there is not enough time to process the batch.
  • Function to return the paging information (last evaluated key)
• Step function looks to see if paging is complete.
  • If not, lambda is called again with paging information passed in.
  • If complete end