In Ansible 2.2, I want to loop over a large list of files read from S3.
Here is my role/tasks/main.yml
- name: Simulate variable read from S3
set_fact:
list_of_files_to_import: [
"a.tar.gz",
"b.tar.gz",
"c.tar.gz",
"d.tar.gz",
"e.tar.gz",
"f.tar.gz",
"g.tar.gz",
"h.tar.gz",
...
"zz.tar.gz"
]
- name: Process each file from S3
include: submodule.yml
with_items: list_of_files_to_import
Here is role/tasks/submodule.yml
---
- name: Restore TABLE {{ item }}
debug: var={{ item }}
This crashes because there are too many files.
I have found that I can slice the array and send sections at a time:
- name: Process each file from S3
include: submodule.yml
with_items: "{{ list_of_files_to_import[0:5] }}"
- name: Process each file from S3
include: submodule.yml
with_items: "{{ list_of_files_to_import[5:10] }}"
- name: Process each file from S3
include: submodule.yml
with_items: "{{ list_of_files_to_import[10:15] }}"
- name: Process each file from S3
include: submodule.yml
with_items: "{{ list_of_files_to_import[15:20] }}"
Instead of hardcoding all these little blocks, I wanted to try something like
- name: Process each file from S3
include: submodule.yml
with_items: "{{ list_of_files_to_import[{{start}}:{{end}}] }}"
But we cannot get variable-defined variable names
How can I process a large list of items in Ansible 2.2?
I ended up solving it with a shell script, repeatedly calling the playbook with some --extra-vars to specify which files to process.
This only works because the list of files in S3 have similar filenames. Basically it loops through the filenames and processes each one at a time.
#!/usr/bin/env bash
# return 0(true) if the year/month combination is valid
valid() {
yr=$1
mn=$2
# do not run on months after this month
if [ "$yr" -eq "2017" -a "$mn" -gt "$(date +%m)" ]
then
return 1
fi
return 0
}
# For every year from 2002 to this year
for year in `seq 2002 $(date +%Y)`
do
# for every zero-padded month, 01-12
for month in `seq -f "%02g" 01 12`
do
# for each type of item in inventory
for object in widgets doodads
do
if valid $year $month;
then
ansible-playbook playbook_name.yml --extra-vars "object=$object year=$year month=$month"
else
echo skipping invalid combo $object $year $month
fi
done
done
done