[SlugBug] text munging question

[Jonathan]

pault wrote:

> hi all
> 
> i have a number of irc log files which i'd like to tidy up.
> 
> one of the problems is that these cover a period when my connection was flaky
> [take a bow blueyonder]. so i ended up with multiple connections to the same
> channel. as i had logging enabled i have lots of examples of the same line
> being repeated two or three times. i would like to remove all the duplicate
> lines.
> 
> i am open to suggestions as how this may best be accomplished [perl? regex?
> dunno?]

If the lines are duplicated adjacent to each other, then you can just:

cat *.log | uniq > output.file

e.g:

-su-2.05b# cat lines
one
two
two
fize
five
-su-2.05b# cat lines  | uniq
one
two
fize
five

Regards,

Jonathan