root/Tags/AudioSegmentation/AudioBook.pm

#! /usr/bin/perl
$VERSION = 0.2;

=head1 NAME

AudioBook - Convert a single transcribed audio file into 15 word audio segments (approximately)  

=cut 

package AudioBook;
use strict;
use diagnostics;
use Carp;
use Getopt::Std;
use File::Basename;
use File::Copy;
use lib '/home/kmaclean/VoxForge-dev/Main/Scripts/Audio_scripts/AudioSegmentation';
use AudioBook::Audio;
use AudioBook::Text;
use AudioBook::Dictionary;

=head1 SYNOPSIS

 $./AudioBook -h                                                                        display help
 $./AudioBook -a speechfile.wav -t text.txt             minimal run configuration

=head1 DESCRIPTION

This is a command line program that segments a speech audio file into 15 word (on average) speech segments.  
It is executable from the command line and uses the following configuration options to help in segmenting speech:

        -a      * audio file name (WAV format only)
        -b      notify if beam width for Forced Alignment exceeds a certain level (default = 250)
                (does not set HVite's beam width parameter)
        -d      pronunciation dictionary  (default = AudioBook/input_files/VoxforgeDict)
        -h      show help
        -l      LICENSE file (default = AudioBook/input_files/LICENCE)
        -m      Target maximum sentence length (default = 20 words)
        -p      Minimum pause for sentence break (default = 2000000 in units of 100ns)
        -q      log words with single quotes (default = yes)
        -r      README file (default = AudioBook/input_files/README)
        -s      Average sentence length (default = 15 words)
        -t      * text file name (containing transcriptions of speech in audio file)
        -u      username or name you want file stats collected by on VoxForge Metrics 
                page:   (http://www.voxforge.org/home/downloads/metrics)
        -v      validate segment audio files to prompt text using forced Aligment
        -w      validate missing word pronunciations to audio recordings
        -x      unique tar file suffix (max 3 characters - remainder is truncated)
        -S      run sanity test
        -T      create gzipped/tar file

                 * required for script to run


=head1 NOTES

=head3 Text Does not Match Audio

If the contents of the text file do not *exactly* match the contents of the speech audio file, the segmentation process necessarily becomes 
a manual, iterative process.

If there are a large divergence in the text from the speech audio, then you will have to manually listen to the speech audio to determine 
where the biggest transcription errors lie, and then modify the original text file to match the speech audio file.  

If the transcription errors are minor, then the first pass forced alignment usually completes successfully.  However, if you see "No tokens survived to final node of network at beam" errors in the 
HVite log (located in interim_files/logs), then using the "-v" verify switch might be helpful in determining where transcription problems 
might exist.

The verify switch performs a forced alignment on the individual segments generated from the first pass forced alignment.  Low scores 
(i.e. the lowest average log likelihood per frame score) indicate that the transcription text might not match the corresponding audio 
file.  Look at the segment text and listen to the corresponding audo file to determine if they match.  If they do not match, then fix the 
text in your original text transcription file, repeat this process (i.e. running the AudioBook program again with the verify switch on) 
until you can get a clean run.

=head3 Segmenting large audio files

For larger files (i.e. greater than 30 minutes of audio), you *may* need to manually segment the audio file into 30 minute segments.

=head3 Automatically Adding Out-of-Vocabulary Words to Pronunciation Dictionary 

The pronunciations generated by the Sequitor G2P scripts need to be manually reviewed before any new pronunciations are added to the
pronunciation dictionary.  Make sure you review the pronunciation before commiting these changes to SVN.    

=head1 REQUIREMENTS

=item 1 - Sequitor G2P trainable Grapheme-to-Phoneme converter (GPL v2; requires Python to be installed)

        http://www-i6.informatik.rwth-aachen.de/web/Software/g2p.html

=item 2 - HTK Hidden Markov Model Toolkit (note: the source is "open", but there are distribution restrictions)

        http://htk.eng.cam.ac.uk/
        
        The HTK toolkit needs to be in your path (see http://www.voxforge.org/home/dev/acousticmodels/linux/create/htkjulius/tutorial/download)

=head1 ALGORITHM

This program tries to segments the speech audio file into 15 word sentences.  However, if the pause following the 15th word relative to the current 
sentence start position is too short, the algorithm looks at the previous word (i.e. word 14) to see if it has a pause of suitable duration.  
If not, it then looks at word following the current start position (i.e. word 16), and so one until a pause of suitable
duration can be found, increasing the number of words to look behind and ahead each time. 

The default pause duration is 2000000 in units of 100ns.  This can be changed (using the "-p" switch") if the speech audio file does segment well 
enough with this default.

=cut 

####################################################################
### Class Variables
####################################################################
our($opt_a,$opt_b,$opt_d,$opt_h,$opt_l,$opt_m,$opt_p,$opt_r,$opt_s,$opt_t,$opt_x,$opt_q,$opt_S,$opt_T,$opt_u,$opt_v,$opt_w); # need to define these because using strict.
my %self;
$self{'debug'} = 0;
$self{'g2p_model'} = "AudioBook/input_files/g2p/models/model-5";
$self{'htk_files'} = "AudioBook/input_files/htk";
$self{'log'} = "AudioBook/output_files/AudioBook_Log";
my $self=\%self;
bless($self,"AudioBook");

my $default_average_sentence_length = 15;
my $default_max_sentence_length = 20;
my $default_min_pause_for_sentence_break = 2000000;
my $command;

####################################################################
### Main
####################################################################
$self->cleanupFiles();
$self->getOptions();
$self->process();
print "completed!\n";

####################################################################
### Methods
####################################################################

=head1 METHODS (not user accessible)

=head2 process

Segement the user designated speech audio file (-a) sing the supplied text file (-t) 

=cut

sub process {
        my ($self)= @_;
        my $debug = $self->{'debug'};
        my $audiofile = $self->{"audiofile"};
        my $textfile = $self->{"textfile"};
        my $username = $self->{"username"};
        my $tarSuffix = $self->{"tarSuffix"};
        my $pronDict = $self->{"pronDict"};
        my $htk_files = $self->{'htk_files'};
        my $log = $self{'log'};
        my $dict = "AudioBook/interim_files/dict";
        my $originalDict = "AudioBook/interim_files/originalDict";
        my $altDict = "AudioBook/interim_files/altDict";        
        my $prompts = "AudioBook/interim_files/prompts";        
        
        my $tempPronDict = "AudioBook/interim_files/pronDict";
        copy($pronDict,$tempPronDict);  

        my $textContents = AudioBook::Text->new($self,$textfile);
        $textContents->createWLISTFile("AudioBook/interim_files/wlist");
        
        my $dictionary = AudioBook::Dictionary->new($self);
        my $missingwordfound = $dictionary->findOutOfVocabularyWords($pronDict,"AudioBook/interim_files/MissingWords");
        if ($missingwordfound) { 
                $dictionary->getRecommendedPronunciations("AudioBook/interim_files/MissingWords_out"); # uses g2p
                $dictionary->updatePronDict($tempPronDict);
                copy($dict,$originalDict); # save dict before suggested pronunications are added - only need these pronunciations for segmentation of audio     
                # need to update dict with missing words
                # can't seem to change default HDMan log file with "-l" parameter
                $command = ("HDMan -A -D -T 1 -g $htk_files/global.ded -m -w AudioBook/interim_files/wlist -i -l AudioBook/interim_files/dlog $dict $tempPronDict"); system($command) == 0 or confess "fullrun $command failed: $?";
                $command = ("mv AudioBook/interim_files/dlog AudioBook/interim_files/logs/dlog2"); print "cmd:$command\n" if $debug; system($command);
                # no longer required$command = ("cp AudioBook/interim_files/MissingWords_out AudioBook/output_files/MissingWords"); print "cmd:$command\n" if $debug; system($command);
        } else {
                open(LOG,">>$log") or confess ("cannot open AudioBook/output_files/MissingWords file");
                print LOG "\nMissing Words that need to be added to Pronunciation Dictionary, with suggested pronunciations::\n";       
                print LOG "------------------------------------------------\n";                         
                print LOG "no missing words\n";
                close LOG
        } 
        # dict may get manually updated; dict only includes suggested prompts, therefore do not copy to output - suggested pronunications are in the log regardless ...
        #... $command = ("cp AudioBook/interim_files/dict AudioBook/output_files"); print "cmd:$command\n" if $debug; system($command);         

        my $audio = AudioBook::Audio->new($self);
        $audio->segment($audiofile,$textContents);
        if ($self->{"verify_segments"}) {
                $audio->verifySegments;
        }       
        if ($missingwordfound) {
                if ($self->{"verify_out_of_vocabulary_pronunciations"}) { 
                        $dictionary->getAlternatePronunciations("AudioBook/interim_files/MissingWords_alt",15); # uses Sequitor g2p to get top N pronunication vairations
                        $dictionary->createAltDict($originalDict,$altDict);     # merge & sort missing_words_alt and originalDict into altDict 
                        $dictionary->validateAlternatePronunciations($originalDict,$altDict,$prompts);
                }
                $dictionary->updatePronDict($pronDict); 
        }       
        
        if (defined($tarSuffix)){
                _createTarFile($self);
        }
}

=head2 cleanupFiles

Removes any old files in the AudioBook/interim_files/ and AudioBook/output_files/ directories, prior to processing.

=cut

sub cleanupFiles {
        my ($self)= @_;
        if (defined(<AudioBook/interim_files/*>)) {
                unlink (<AudioBook/interim_files/*>);
        }
        if (defined(<AudioBook/interim_files/logs/*>)) {
                unlink (<AudioBook/interim_files/logs/*>);      
        }
        if (defined(<AudioBook/interim_files/missingWordsFolder/*>)) {
                unlink (<AudioBook/interim_files/missingWordsFolder/*>);        
        }
        if (defined(<AudioBook/interim_files/wav/*>)) {
                unlink (<AudioBook/interim_files/wav/*>);       
        }       
        if (defined(<AudioBook/output_files/wav/*>)) {
                unlink (<AudioBook/output_files/wav/*>);        
        }
        if (defined(<AudioBook/output_files/*>)) {      
                unlink (<AudioBook/output_files/*>);
        }
}

=head2 _createTarFile 

creates a GZipped Tar file form files contained in AudioBook/output_files

=item * -T      Switch to turn on this functionality

=item * -u      username used in name of the tar file 

=cut

sub _createTarFile { # private
        my ($self)= @_;
        my $debug = $self->{'debug'};
        my $username = $self->{"username"};
        my $tarSuffix = $self->{"tarSuffix"};
        my $readme = $self->{"README"};
        my $license = $self->{"LICENSE"};
        
        my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time);
        $year += 1900;
        $mon = sprintf("%02d", $mon); 
        $mday = sprintf("%02d", $mday);
        print "creating gzipped tar file:$username\-$year$mon$mday\-$tarSuffix\.tgz \n";
        if (defined($readme)) {
                copy("$readme","AudioBook/output_files/README");
        } else {
                print "Warning: no README file to copy\n";
        }
        if (defined($license)) {
                copy("$license","AudioBook/output_files/LICENSE");
        } else {
                print "Warning: no LICENSE file to copy\n";
        }
        copy("AudioBook/interim_files/prompts","AudioBook/output_files/prompts");
        $command = ("cp AudioBook/interim_files/wav/* AudioBook/output_files/wav/"); print "cmd:$command\n" if $debug; system($command);        
        if ($debug) {
                $command = ("tar -zcvf $username\-$year$mon$mday\-$tarSuffix\.tgz AudioBook/output_files --exclude \"\.svn\" "); print "cmd:$command\n" if $debug; system($command);
        } else {
                $command = ("tar -zcf $username\-$year$mon$mday\-$tarSuffix\.tgz AudioBook/output_files --exclude \"\.svn\" "); print "cmd:$command\n" if $debug; system($command);
        }
        print "please submit your tar file to: www.voxforge.org\n";     
}

sub _random_characters {
        my ($length) = @_;      
        my @chars=('a'..'z');
        my $randomString;
        foreach (1..$length){
                $randomString.=$chars[rand @chars];
        }
        return $randomString;
}

=head2 getOptions 

Get the user submitted options ('a:b:d:hl:m:p:r:s:t:u:x:q:vwST')

=cut

sub getOptions {
        my ($self)= @_;
        my $debug = $self->{'debug'};   
        getopts('a:b:d:hl:m:p:r:s:t:u:x:q:vwST');    #  sets $opt_* as a side effect.
        if ($opt_S) { # Sanity test switch
                $self->{"audiofile"}="AudioBook/test/audio.wav";
                #$self->{"textfile"}="AudioBook/test/text-simple.txt";
                $self->{"textfile"}="AudioBook/test/text-original.txt";
                $command = ("cp AudioBook/input_files/VoxForgeDict AudioBook/interim_files/VoxForgeDict"); print "cmd:$command\n"; system($command);
                $self->{"pronDict"}="AudioBook/interim_files/VoxForgeDict";
                $self->{"tarSuffix"}=_random_characters(3);
                $self->{"username"}="test";
                $self->{"average_sentence_length"}= $default_average_sentence_length;
                $self->{"max_sentence_length"}= $default_max_sentence_length;
                $self->{"min_pause_for_sentence_break"}=$default_min_pause_for_sentence_break;
                
                $self->{"log_single_quotes"}= 1;
                $self->{"verify_segments"}=1;   
                $self->{"verify_out_of_vocabulary_pronunciations"}=1;           
                $self->{"README"}="AudioBook/input_files/README";
                $self->{"LICENSE"}="AudioBook/input_files/LICENSE";
        } elsif ($opt_a and $opt_t) {   
                if (-r $opt_a) {
                        $self->{"audiofile"}=$opt_a;
                } else {
                        die "can't open -a" . $self->{"audiofile"} . "\n";              
                }
                if (-r $opt_t) {
                        $self->{"textfile"}=$opt_t;
                } else {
                        die "can't open -t" . $self->{"textfile"} . "\n";               
                }
                if (defined($opt_d)) {
                        if (-r $opt_d) {
                                $self->{"pronDict"}=$opt_d;
                        } else {
                                die "can't open -d" . $self->{"pronDict"} . "\n";       
                        }
                } else {
                        $self->{"pronDict"}="AudioBook/input_files/VoxForgeDict";       
                }
                ### Audio Processing
                if ($opt_s) {
                        $self->{"average_sentence_length"}=$opt_s;
                } else {
                        $self->{"average_sentence_length"}= $default_average_sentence_length;   
                }
                if ($opt_m) {
                        $self->{"max_sentence_length"}=$opt_m;
                } else {
                        $self->{"max_sentence_length"}= $default_max_sentence_length;   
                }
                if ($opt_p) {
                        $self->{"min_pause_for_sentence_break"}=$opt_p;
                } else {
                        $self->{"min_pause_for_sentence_break"}= $default_min_pause_for_sentence_break; 
                }       
                if ($opt_q) {
                        if ($opt_v =~ /^n|no$/i){
                                $self->{"log_single_quotes"}= 0;
                        } else {
                                $self->{"log_single_quotes"}= 1;        
                        }
                } else {
                        $self->{"log_single_quotes"}= 1;        
                }       
                if ($opt_b) {
                        $self->{"beam_width"}=$opt_b;
                } else {
                        $self->{"beam_width"}=250;      
                }
                if ($opt_v) {
                        $self->{"verify_segments"}=1;
                } else {
                        $self->{"verify_segments"}=0;   
                }       
                if ($opt_w) {
                        $self->{"verify_out_of_vocabulary_pronunciations"}=1;
                } else {
                        $self->{"verify_out_of_vocabulary_pronunciations"}=0;   
                }       
                ### Tar file processing
                if (defined($opt_T)) {
                        if ($opt_x) {
                                $self->{"tarSuffix"}=substr($opt_x,0,3); # only use 1st 3 characters.                   
                        }else {
                                $self->{"tarSuffix"}=_random_characters(3);
                        }
                        if ($opt_u) {
                                $self->{"username"}=$opt_u;     
                        }else {
                                $self->{"username"}="anonymous";
                        }       
                        if ($opt_r) {
                                if (-r $opt_r) {
                                        $self->{"README"}=$opt_r;       
                                } else {
                                        die "can't open -r" . $self->{"README"} . "\n";                         
                                }
                        } else {
                                $self->{"README"}="AudioBook/input_files/README";
                        }               
                        if ($opt_l) {
                                if (-r $opt_l) {
                                        $self->{"LICENSE"}=$opt_l;      
                                } else {
                                        die "can't open -l" . $self->{"LICENSE"} . "\n";                                
                                }
                        } else {
                                $self->{"LICENSE"}="AudioBook/input_files/LICENSE";
                        }
                }
        } elsif ($opt_h) {
                print "\nVoxForge Audio Segmentation Script Parameters\n";      
                print   "=============================================\n";      
                print "-a\t* audio file name (WAV format only)\n";
                print "-b\tnotify if beam width for Forced Alignment exceeds a certain level (default = 250)\n";
                print "\t(does not set HVite's beam width parameter)\n";
                print "-d\tpronunciation dictionary  (default = AudioBook/input_files/VoxforgeDict)\n";
                print "-h\tshow help\n";        
                print "-l\tLICENSE file (default = AudioBook/input_files/LICENCE)\n";
                print "-m\tTarget maximum sentence length (default = $default_max_sentence_length words)\n";
                print "-p\tMinimum pause for sentence break (default = $default_min_pause_for_sentence_break in units of 100ns)\n";             
                print "-q\tlog words with single quotes (default = yes)\n";             
                print "-r\tREADME file (default = AudioBook/input_files/README)\n";                             
                print "-s\tAverage sentence length (default = $default_average_sentence_length words)\n";                               
                print "-t\t* text file name (containing transcriptions of speech in audio file)\n";
                
                print "-u\tusername or name you want file stats collected by on VoxForge Metrics \n";
                print "\tpage:\t(http://www.voxforge.org/home/downloads/metrics)\n";    
                
                print "-v\tvalidate segment audio files to prompt text using forced Aligment\n";
                print "-w\tvalidate missing word pronunciations to audio recordings\n";         
                print "-x\tunique tar file suffix (max 3 characters - remainder is truncated)\n";
                print "-S\trun sanity test\n";          
                print "-T\tcreate gzipped/tar file\n";
                print "\n\t* required for script to run\n";     
                print "\n";     
                print "--\n";                   
                print "Free Speech... Recognition\n";
                print "http://www.voxforge.org\n\n";
                exit;
        } else {
                print "\nVoxForge Audio Segmentation Script\n"; 
                print   "==================================\n"; 
                print "Parms -a and -t need to be defined. Use -h parameter for more information\n\n";
                print "--\n";                   
                print "Free Speech... Recognition\n";
                print "http://www.voxforge.org\n\n";
                exit;
        }
        print "audiofile:" . $self->{"audiofile"}. "\n";
        print "textfile:" . $self->{"textfile"}. "\n";
        print "pronDict:" . $self->{"pronDict"} . "\n\n";       
}

=head2 Gettors - Public (used by methods in other sub-classes)

=item * getAverage_sentence_length()

=cut

sub getAverage_sentence_length {
        my $self = shift;
        return $self->{"average_sentence_length"};
}

=item * getMax_sentence_length()

=cut

sub getMax_sentence_length {
        my $self = shift;
        return $self->{"max_sentence_length"};
}

=item * getMin_pause_for_sentence_break()

=cut

sub getMin_pause_for_sentence_break {
        my $self = shift;
        return $self->{"max_sentence_length"};
}
    
=head1 Change Log    

        2008/05/02 - 0.2 - convert to class; major refacture ; renamed fullrun.pl to AudioBook.pm                                                       
        2008/01/31 - 0.1 - created
        
=cut

=head1 AUTHOR
    
    Ken MacLean
    contact@voxforge.org
      
=head1 COPYRIGHT AND LICENSE       
      
Copyright (C) 2008 Ken MacLean
   
This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.
   
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.