When I spin up a new server I use a cloud-init script to automate the process. I follow progress of this process with:

$ tail -f /var/log/cloud-init-output.log

When it comes to fetching a backup file using scp and piping it through tar -zx I tried the following in the cloud-init script:

$ scp myuser@123.45.6.789:path/to/file.tar.gz /dev/stdout | tar -zx

Whilst this command works, the handy progress indications that scp outputs do not appear in the tail -f output... i.e. I do not see progress like:

file.tar.gz   57%   52MB  25.2MB/s   00:02    

I have also tried bash process substitution like so:

$ scp myuser@123.45.6.789:path/to/file.tar.gz >(tar -zx)

And still the progress indications do not appear in the tail -f output.

How do I preserve the progress indications in the tail -f output? Particularly to see progress when fetching the larger backup files, it would be really handy to preserve these indications.

Note that when I run both of the above in a bash script directly (with set -x at the top), progress does show for the 'bash process substitution' variant but not the 'piping' variant... but when tailing the cloud-init logs, progress shows for neither variant.

It looks like cloud-init just sends both stdout and stderr from all cloud-init stages to /var/log/cloud-init-output.log (see here).

So we can re-create the cloud-init process (for the purposes of this question) with the following call:

$ scp myuser@123.45.6.789:path/to/file.tar.gz >(tar -zx) >output.log 2>&1

and we then follow this log file separately with:

$ tail -f output.log

Strange thing is... no progress status ever appears in this output.log file. And yet progress status is definitely being sent... the only change in the following is to direct to /dev/stdout rather than output.log:

$ scp myuser@123.45.6.789:path/to/file.tar.gz >(tar -zx) >/dev/stdout 2>&1
file.tar.gz   57%   52MB  25.2MB/s   00:02    

So why do we see progress status on /dev/stdout but not in output.log when we direct stdout to output.log?