Business
Jobs
  • About Us
  • Solutions
    • Job Postings
      Post your job and receive qualified candidates in 48h.
    • Candidate Assessments
      500+ technical and psychological tests, plus anti-fraud.
    • Headhunting
      Tailor-made executive search from start to finish.
    • Payroll + EOR
      Payroll dispersal and EOR across 15+ LATAM countries.
  • Pricing
  • Jobs

0

328
Views
How to find the byte position of specific line in a file

What's the fastest way to find the byte position of a specific line in a file, from the command line?

e.g.

$ linepos myfile.txt 13
5283

I'm writing a parser for a CSV that's several GB in size, and in the event the parser is halted, I'd like to be able to resume from the last position. The parser is in Python, but even iterating over file.readlines() takes a long time, since there are millions of rows in the file. I'd like to simply do file.seek(int(command.getoutput("linepos myfile.txt %i" % lastrow))), but I can't find a shell command to efficiently do this.

Edit: Sorry for the confusion, but I'm looking for a non-Python solution. I already know how to do this from Python.

over 4 years ago · Santiago Trujillo
3 answers
Answer question

0

From @chepner's comment on my other answer:

position = 0  # or wherever you left off last time
try:
    with open('myfile.txt') as file:
        file.seek(position)  # zero in base case
        for line in file:
            position = file.tell() # current seek position in file
            # process the line
except:
    print 'exception occurred at position {}'.format(position)
    raise
over 4 years ago · Santiago Trujillo Report

0

Iterating over a file object yields lines with full line endings intact. You should be able to just add the lens to a counter object to get the position. You'll need to multiply based on character encoding (character byte size)

position = 0  # or wherever you left off last time
try:
    with open('myfile.txt') as file:  # don't you go correcting me on naming it file. we don't call file directly anyway!
        file.seek(position)  # zero in base case
        for line in file:
            position += len(line)
            # process the line
except:
    # yes, a naked exception. TWO faux pas in one answer?!?
    print 'exception occurred at position {}'.format(position)
    raise # re-raise to see traceback or what have you
over 4 years ago · Santiago Trujillo Report

0

Well, if your pattern is simple, this would be simple

$ echo -e '#!/bin/bash\necho abracadabra' >/tmp/script
$ pattern=bash
$ sed -rn "0,/$pattern/ {s/^(.*)$pattern.*$/\1/p ;t exit; p; :exit }" /tmp/script \
    | wc -c 
8

As you can see, this will output position of the first character in your pattern, assuming first character in the file has number 1.

NB 1: sed has a habit to add a trailing newline to the last string it parses, thus, when we take a part of the line preceding the pattern, the number of bytes in output should be 7 (count them → #!/bin/), but what wc -c actually counts looks like

$ sed -rn "0,/$pattern/ {s/^(.*)$pattern.*$/\1/p ;t exit; p; :exit }" /tmp/script \
   | hexdump -C
00000000  23 21 2f 62 69 6e 2f 0a                           |#!/bin/.|
00000008

This could be potential source of troubles, if you were looking for EOF, for example. I can’t think of a more appropriate case, I just want to point that out.

NB 2: If pattern will contain special characters, sed will fail. If you could provide an example of what you are looking for, I could escape it.

NB 3: This assumes the pattern is unique. If you will stop reading the file on a second or third instance of pattern, this will not work.


Update. I’ve found a more simple way.

$ grep -bo bash <<< '#!/bin/bash'
7:bash

For GNU grep there are two options:

-b, --byte-offset
    Print the 0-based byte offset within the input file before  each  line  of
    output. If -o (--only-matching)  is specified, print the offset of the
    matching part itself.

I’d suggest to use grep, because if you specify -F key, it will treat pattern as a simple string.

$ grep -F '!@##$@#%%^%&*%^&*(^)((**%%^@#' <<<'!@##$@#%%^%&*%^&*(^)((**%%^@#' 
!@##$@#%%^%&*%^&*(^)((**%%^@#
over 4 years ago · Santiago Trujillo Report
Answer question
Find remote jobs

Discover the new way to find a job!

Top jobs
Top job categories
Business
Post vacancy Pricing Sales
Legal
Terms and conditions Privacy policy
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Show me some job opportunities
There's an error!