[ale] Dealing with really big log files....

[JK]

Michael B. Trausch wrote:
> On Mon, 23 Mar 2009 09:54:35 -0600
> JK <jknapka at kneuro.net> wrote:
> 
>> This is the only good idea so far. It's the only one that
>> doesn't involve reading the entire file up to the data of
>> interest, which is just... not a very smart thing to do,
>> when you have some idea where the data is, in a file
>> approximately the same size as the known universe.
> 
> True enough, but...
> 
> You'd have to know how long each line is to use that well, or
> accurately.  If the lines are variable-length, as I suspect that might
> be, then you've really no way to determine via 'dd' what line you're
> starting on, just the "block" number, which would only really be good
> if you know about what blocks you need from the file.


Yeah but... who cares?  You can just trim any partial lines from
the front and back of the resulting file.  And doing a binary search
(manually if need be) for the interesting chunk is probably quicker
than scanning through 100 GB of junk.


-- 
I do not particularly want to go where the money is -
  it usually does not smell nice there. -- A. Stepanov