I want to run multiple time-consuming shell commands from Ruby in a non-blocking (asynchronous) way.
I want to pass options to commands, receive output in Ruby, and (ideally) handle errors.
The script below will naturally take 15 seconds to execute:
test.rb
3.times do |i|
puts `sleep 5; echo #{i} | tail -n 1` # some time-consuming complex command
end
$ /usr/bin/time ruby test.rb
0
1
2
15.29 real 0.13 user 0.09 sys
With Thread, it can apparently be executed in parallel, and it takes only 5 seconds, as expected:
threads = []
3.times do |i|
threads << Thread.new {
puts `sleep 5; echo #{i} | tail -n 1`
}
end
threads.each {|t| t.join() }
$ /usr/bin/time ruby test.rb
2
0
1
5.17 real 0.12 user 0.06 sys
But is this the best approach? Is there any other way?
I have also written using Open3.popen2, but this seems to take 15 seconds to execute as in the first example (unless wrapped in a Thread):
require 'open3'
3.times do |i|
Open3.popen2("sleep 5; echo #{i} | tail -n 1") do |stdin, stdout|
puts stdout.read()
end
end
The documentation describes "block form" and "non-block form", but this "block" refers to anonymous functions, and has nothing to do with concurrency, correct?
Is the Open3 class alone only capable of blocking execution?
The problem with your code is that stdout.read is a blocking call.
You could defer the reading until the command is finished.
At first, create the commands:
commands = Array.new(3) { |i| Open3.popen2("sleep 5; echo hello from #{i}") }
Then, wait for each command to finish:
commands.each { |stdin, stdout, wait_thr| wait_thr.join }
Finally, gather the output and close the IO streams:
commands.each do |stdin, stdout, wait_thr|
puts stdout.read
stdin.close
stdout.close
end
Output: (after 5 seconds)
hello from 0
hello from 1
hello from 2