Question

0

129

Views

linux sort, select last line, pipe to new file

I have a series of multiple files. I would like to sort these files, select the bottom line from each, and pipe those lines into a new single file.

My files look like this:

1,  100,  2.5
2,  100,  3.3
3,  100,  5.1
4,  100,  1.2

These files are all named 51_Sur_extracted_data.csv

The files are housed in parent directories as follows:

Track_0001/output_dfsu/51_Sur_extracted_data.csv Track_0002/output_dfsu/51_Sur_extracted_data.csv

So I would like to sort all of the csv files on the 3rd column, extract the bottom line, and place it into a new summary file. Basically the goal is produce a file that has the maximum values from column three across all of the parent directories. Ideally, I would also like to add a column in the output file that contains the name of the parent directory (Track_0002).

I have the sort figured out, but the rest is proving a bit troublesome for me. For instance, can I sort all of the csv files at the same time and write the output to new files (example 51_Sur_extracted_data_sort.csv)? Then I can grep the last line and pipe it to a new file?

sort -t"," -k3,3g filename

Thanks, K

9 months ago · Santiago Trujillo

2 answers

Answer question

0

Answer question

Find remote jobs

Answer 1 · 2022-04-21T18:07:09.263Z

IF you want the last line of any given command, use tail.

So for you, you would do this:

sort -t"," -k3,3g filename | tail -n1 > newfilename

Answer 2 · 2022-04-21T18:07:09.274Z

I'd do something like this:

for a in */*/*.csv
do 
    dname="$(basename "$(dirname "$a")")"
    echo -e "$dname\t$(sort -t"," -k3,3g "$a" | tail -n 1)"
done

On my test files it returned:

output_abcd 3,  100,  9.1
output_bcde 3,  100,  5.1
output_cdef 3,  100,  5.1
output_abcd 3,  100,  5.1
output_bcde 3,  100,  5.1
output_cdef 3,  100,  5.1
output_abcd 3,  100,  5.1
output_bcde 3,  100,  5.1
output_cdef 1,  100,  7.5
output_abcd 3,  100,  5.1
output_bcde 3,  100,  5.1
output_cdef 3,  100,  5.1
output_abcd 3,  100,  5.1
output_bcde 3,  100,  5.1
output_cdef 3,  100,  5.1
output_abcd 3,  100,  5.1
output_bcde 3,  100,  5.1
output_cdef 3,  100,  5.1
output_abcd 3,  100,  5.1
output_bcde 2,  100,  42.3
output_cdef 3,  100,  5.1
output_abcd 3,  100,  5.1
output_bcde 3,  100,  5.1
output_cdef 3,  100,  5.2
output_abcd 3,  100,  5.1
output_bcde 3,  100,  5.1
output_cdef 3,  100,  5.1

Yeah, my input data is a bit boring. Of course you can sort the end result again, but I'll leave that to your imagination (you already had that step) :)