Headers in Usenet posts

This is a study base on the messages/posts from the many newsgroups i read. I've exported the data in #! rnews format, and created a small perl program to analyze headers that can be found and establish statistics. The #! rnews format is a simple text file where messages are separated by a single line begining with: #! rnews followed by a numerical reference. The analyse is simple, lines after the separator are headers until the blank line that separates the content/body of the message.

My perl implementation is really basic, it searches for occurences of different headers and stores three different example values. It produces then a table in html style:

$count = 0;
while ($file = glob("news/*")) {
  open (INFILE,"< $file");
  $inhead = 0;
  while ($ligne = <INFILE>) {
    if (index($ligne,'#! rnews')== 0) {
      $count++;
      $inhead = 1;
    } elsif (($inhead == 1) and ($ligne eq "\n")) {
      $inhead = 0;
    } elsif (($inhead == 1) and (index ($ligne, ":")>2)) {
      # process
      ($typehead,$value) = split (/: /, $ligne);
      $dep{$typehead}++;
      chomp($value);
      $value =~ s/</&lt;/gi;
      $value =~ s/>/&gt;/gi;
      unless ($expl{$typehead}[1]) {
        $expl{$typehead}[1] = $value;
      } elsif (!($expl{$typehead}[2]) and ($expl{$typehead}[1] ne $value)) {
        $expl{$typehead}[2] = $value;
      } elsif (!($expl{$typehead}[3]) and ($expl{$typehead}[1] ne $value) and ($expl{$typehead}[2] ne $value)) {
        $expl{$typehead}[3] = $value;
      }
    }
  }
}
print "Process on $count messages\n\n";
print "<table border=1>";
foreach (sort {$dep{$b} <=> $dep{$a}} keys %dep) {
  print "<tr><td>$_</td><td>$dep{$_}</td><td><ul>\n";
  for $i (1..3){
    print "<li>$expl{$_}[$i]</li>\n" if ($expl{$_}[$i]);
  }
  print "</ul></td></tr>\n";
}
print "</table>";
exit(0);

Here are the results i got at the time of writing, sorted by frequency:

Process on 2176 messages

main menu