voxforge.org
VoxForge Dev
Home · Read · Listen · Forums · Dev · Downloads · About

root/Trunk/Scripts/Testing_scripts/NightlyTest/TestResults

Revision 1560, 6.2 kB (checked in by kmaclean, 2 years ago)

updates to Testing_scripts

Line 
1 Disclaimer: this is a *sanity test* only, and is *not* to be regarded as a valid test of the VoxForge Acoustic Models!
2  * the audio files used for testing include only one voice, the Acoustic Models were also trained using that same voice, so the results will look better;
3  * there are only 50 audio samples in the test database - not enough for a good test;
4  * the VoxForge Acoustic Models are still alpha with respect to Speaker Independent Speech Recognition, so please donate some speech to VoxForge,
5  thanks,
6  Ken
7  
8 Testing Acoustic Models created in: /data/svn-mirror-copy/Nightly_Builds/AcousticModel-2007-01-25
9
10 HTK 16kHz_16bit
11 ---------------
12         word insertion penalty: 0.0
13         grammar scale factor: 1.0
14 ====================== Results Analysis =======================
15   Date: Thu Jan 25 13:48:15 2007
16   Ref : testref.mlf
17   Rec : recout.mlf
18 ------------------------ Overall Results --------------------------
19 SENT: %Correct=22.00 [H=11, S=39, N=50]
20 WORD: %Corr=86.24, Acc=44.44 [H=163, D=1, S=25, I=79, N=189]
21 ===================================================================
22
23 Julian 16kHz_16bit
24 ------------------
25         word insertion penalty
26           first pass (-penalty1):0.5
27           second pass (-penalty2):100.0
28         transition penalty:-55.0 (for short-term inter-word pauses between words (-iwsppenalty))
29 ====================== Results Analysis =======================
30   Date: Thu Jan 25 13:48:19 2007
31   Ref : testref.mlf
32   Rec : julianProcessed
33 ------------------------ Overall Results --------------------------
34 SENT: %Correct=88.00 [H=44, S=6, N=50]
35 WORD: %Corr=97.35, Acc=96.83 [H=184, D=2, S=3, I=1, N=189]
36 ===================================================================
37
38 HTK 8kHz_16bit
39 ---------------
40         word insertion penalty: 10.0
41         grammar scale factor: 5.0
42 ====================== Results Analysis =======================
43   Date: Thu Jan 25 13:48:21 2007
44   Ref : testref.mlf
45   Rec : recout.mlf
46 ------------------------ Overall Results --------------------------
47 SENT: %Correct=52.00 [H=26, S=24, N=50]
48 WORD: %Corr=83.07, Acc=65.08 [H=157, D=1, S=31, I=34, N=189]
49 ===================================================================
50
51 Julian 8kHz_16bit
52 ------------------
53         word insertion penalty
54           first pass (-penalty1):50.0
55           second pass (-penalty2):100.0
56         transition penalty::-55.0 (for short-term inter-word pauses between words (-iwsppenalty))
57 ====================== Results Analysis =======================
58   Date: Thu Jan 25 13:48:23 2007
59   Ref : testref.mlf
60   Rec : julianProcessed
61 ------------------------ Overall Results --------------------------
62 SENT: %Correct=84.00 [H=42, S=8, N=50]
63 WORD: %Corr=95.24, Acc=91.53 [H=180, D=1, S=8, I=7, N=189]
64 ===================================================================
65
66 Notes:
67
68     * the line starting with SENT gives the percentage of sentences that were recognized correctly, out of N sentences in total.
69     * the line starting with WORD gives the percentage of words that were recognized correctly, out of N words in total
70       However, since HTK or Julius erroneously 'added' words that are not in the audio file (i.e. insertion errors) they usually get a lower percentage accuracy rating.
71     * Count definitions:
72           o D - Deletion Error
73           o S - Substitution Error
74           o I - Insertion Error
75
76
77
78 ================================================================================================================
79 For comparison purposes, see below for the same Tests on the most current release of the VoxForge Acoustic Models:
80 (/data/svn-mirror-copy/Tags/Releases/0_1_1-build726)
81 ================================================================================================================
82 HTK 16kHz_16bit
83 ---------------
84         word insertion penalty: 0.0
85         grammar scale factor: 1.0
86 ====================== Results Analysis =======================
87   Date: Thu Jan 25 13:48:30 2007
88   Ref : testref.mlf
89   Rec : recout.mlf
90 ------------------------ Overall Results --------------------------
91 SENT: %Correct=22.00 [H=11, S=39, N=50]
92 WORD: %Corr=88.36, Acc=50.79 [H=167, D=1, S=21, I=71, N=189]
93 ===================================================================
94
95 Julian 16kHz_16bit
96 ------------------
97         word insertion penalty
98           first pass (-penalty1):0.5
99           second pass (-penalty2):100.0
100         transition penalty:-55.0 (for short-term inter-word pauses between words (-iwsppenalty))
101 ====================== Results Analysis =======================
102   Date: Thu Jan 25 13:48:33 2007
103   Ref : testref.mlf
104   Rec : julianProcessed
105 ------------------------ Overall Results --------------------------
106 SENT: %Correct=86.00 [H=43, S=7, N=50]
107 WORD: %Corr=96.83, Acc=96.30 [H=183, D=2, S=4, I=1, N=189]
108 ===================================================================
109
110 HTK 8kHz_16bit
111 ---------------
112         word insertion penalty: 10.0
113         grammar scale factor: 5.0
114 ====================== Results Analysis =======================
115   Date: Thu Jan 25 13:48:40 2007
116   Ref : testref.mlf
117   Rec : recout.mlf
118 ------------------------ Overall Results --------------------------
119 SENT: %Correct=56.00 [H=28, S=22, N=50]
120 WORD: %Corr=82.01, Acc=65.08 [H=155, D=2, S=32, I=32, N=189]
121 ===================================================================
122
123 Julian 8kHz_16bit
124 ------------------
125         word insertion penalty
126           first pass (-penalty1):50.0
127           second pass (-penalty2):100.0
128         transition penalty::-55.0 (for short-term inter-word pauses between words (-iwsppenalty))
129 ====================== Results Analysis =======================
130   Date: Thu Jan 25 13:48:41 2007
131   Ref : testref.mlf
132   Rec : julianProcessed
133 ------------------------ Overall Results --------------------------
134 SENT: %Correct=86.00 [H=43, S=7, N=50]
135 WORD: %Corr=95.77, Acc=92.59 [H=181, D=2, S=6, I=6, N=189]
136 ===================================================================
137
138 Notes:
139
140     * the line starting with SENT gives the percentage of sentences that were recognized correctly, out of N sentences in total.
141     * the line starting with WORD gives the percentage of words that were recognized correctly, out of N words in total
142       However, since HTK or Julius erroneously 'added' words that are not in the audio file (i.e. insertion errors) they usually get a lower percentage accuracy rating.
143     * Count definitions:
144           o D - Deletion Error
145           o S - Substitution Error
146           o I - Insertion Error
Note: See TracBrowser for help on using the browser.
©2005-2009 VoxForge