, I can say about just q just quickly to get through it , that Dave and I submitted this ASRU .  , basically we 're dealing with rever reverberation , and , when we deal with pure reverberation , the technique he 's using works really , really well .  , and when they had the reverberation here , we 'll measure the signal - to - noise ratio and it 's , about nine DB . professor b: phd a: You mean , from the actual , recordings ? professor b: a fair amount of phd d: k phd a: It 's nine DB ? professor b: Yeah .  , I know that when you figured out the filters that we 're using for the Mel scale , there was some experimentation that went on at at ,  at OGI .  , but one of the differences that we found between the two systems that we were using , the the Aurora HTK system baseline system and the system that we were the the  , other system we were using , the  , the SRI system , was that the SRI system had maybe a , hundred hertz high - pass . professor b: For some reason , Dave thought it was twenty , phd d:  So the , center would be somewhere around like hundred professor b: but . professor b: But do you know , for instance , h how far down it would be at twenty hertz ? What the how much rejection would there be at twenty hertz , let 's say ? phd d: At twenty hertz . professor b: Yeah , any idea what the curve looks like ? phd d: Twenty hertz frequency Oh , it 's it 's zero at twenty hertz , right ? The filter ? phd c: Yea - actually , the left edge of the first filter is at sixty - four . professor b: It 's actually set to zero ? What kind of filter is that ? phd c: Yeah . professor b: Is this oh , from the from phd c: It This is the filter bank in the frequency domain that starts at sixty - four . professor b: Oh , so you ,  so you really set it to zero , the FFT ? phd d: Yeah , phd c: Yeah . So I wonder , is it @ @ Was there their experimentation with , say , throwing away that filter or something ? And ,  phd d: throwing away the first ? professor b: Yeah . professor b: Right , but the question is , whether sixty - four hertz is is , too , low . phd d: I t I think I 've tried a hundred and it was more or less the same , or slightly worse . professor b: On what test set ? phd d: On the same , SpeechDat - Car , Aurora . professor b: phd d: So it was professor b: and on and on the , TI - digits also ? phd d: No , no , no . That 'd be something to look at sometime because what , eh , he was looking at was performance in this room . professor b: Would that be more like Well , you 'd think that 'd be more like SpeechDat - Car , I guess , in terms of the noise .  , the signal - to - noise ratio , you know , looks a fair amount better if you if you high - pass filter it from this room . professor b: So , the main the the phd a: So that 's on th that 's on the f the far field ones though , right ? Yeah . phd a: So wha what is ,  what 's causing that ? professor b: Well , we got a a video projector in here , and ,  which we keep on during every every session we record , phd a: Yeah . professor b: which , you know , I I w we were aware of phd a:  - huh . professor b: But but ,  So , those are those are major components , I think , phd a: I see . professor b: but , it ,  I guess , I maybe I said this last week too but it it it really became apparent to us that we need to to take account of noise . And , so I think when when he gets done with his prelim study I think one of the next things we 'd want to do is to take this , noise , processing stuff and and , synthesize some speech from it . phd a: When are his prelims ? professor b: And then  , I think in about , a little less than two weeks .  , let 's see , this is the sixteenth , seventeenth ? Yeah , I don't know if he 's before It might even be in a week . I I guessed that they were gonna do it some time during the semester professor b: week and a half . phd a: but they 'll do it any time , huh ? professor b: They seem to be Well , the semester actually is starting up . phd a: Is it already ? professor b: Yeah , the semester 's late late August they start here .  , that that was sort of one  , the overall results seemed to be first place in in in the case of either , artificial reverberation or a modest sized training set . And But if you had a a really big training set , a recognizer , system that was capable of taking advantage of a really large training set I thought that One thing with the HTK is that is has the as we 're using the configuration we 're using is w s is being bound by the terms of Aurora , we have all those parameters just set as they are . So even if we had a hundred times as much data , we wouldn't go out to , you know , ten or t or a hundred times as many Gaussians or anything .  , so , that ,  that seemed to be So , if you have that that better recognizer that can that can build up more parameters , and if you , have the natural room , which in this case has a p a pretty bad signal - to - noise ratio , then in that case , the right thing to do is just do u use speaker adaptation . But I think that that would not be true if we did some explicit noise - processing as well as , the convolutional kind of things we were doing . phd d: That sample was released only yesterday or the day before , right ? phd a: No Well , I haven't grabbed that one yet . phd d: Oh , there is another short sample set phd a: There was another short one , yeah . phd a: And so I haven't grabbed the latest one that he just , put out yet . professor b: Is there any word yet about the issues about , adjustments for different feature sets or anything ? phd a: No , I I d You asked me to write to him and I think I forgot to ask him about that . phd d: Cuz they have it phd a: Maybe I 'll send it to the list . phd d: Cuz they have , already frozen those in i insertion penalties and all those stuff is what I feel . phd d: And , on that , they have run some experiments using various insertion penalties and all those phd a: And so they 've picked the values . professor b: For r w what test set ? phd d: p the one that they have reported is a NIST evaluation , Wall Street Journal . professor b: But that has nothing to do with what we 're testing on , right ? phd c:   So they 're , like  So they are actually trying to , fix that those values using the clean , training part of the Wall Street Journal . phd d: they want to train it and then this they 're going to run some evaluations . professor b: So they 're set they 're setting it based on that ? phd d: Yeah . So now , we may come back to the situation where we may be looking for a modification of the features to account for the fact that we can't modify these parameters . professor b:  but it 's still worth , I think , just since you know , just chatting with Joe about the issue . Do you think that 's something I should just send to him professor b:  phd a: or do you think I should send it to this there 's an a m a mailing list .  , we 're , you know , certainly willing to talk about it with everybody , but I think I think that , it 's probably best to start talking with him just to phd a: OK . professor b:  @ @ you know , it 's a dialogue between two of you about what you know , what does he think about this and what what you know what could be done about it . professor b: if you get ten people in involved in it there 'll be a lot of perspectives based on , you know , how phd a: Yeah . professor b: but if if if there is any , way to move in a way that would that would , you know , be more open to different kinds of features . But if if ,  if there isn't , and it 's just kind of shut down and and then also there 's probably not worthwhile bringing it into a larger forum where where political issues will come in . phd d: Because he there was some mail r saying that it 's may not be stable for Linux and all those . phd d: SUSI phd a: Yeah , SUSI or whatever it was phd d: yeah . professor b: but , I noticed , just glancing at the , Hopkins workshop , web site that ,  one of the thing I don't know Well , we 'll see how much they accomplish , but one of the things that they were trying to do in the graphical models thing was to put together a a , tool kit for doing ,  r  , arbitrary graphical models for , speech recognition . professor b: So And Jeff ,  the two Jeffs were phd a: Who 's the second Jeff ? professor b:  Oh , do you know Geoff Zweig ? phd a: No . professor b: so he did he did his PHD on dynamic Bayes - nets , for for speech recognition . He had some continuity built into the model , presumably to handle some , inertia in the in the production system , and ,  phd a:  .  , so it 's exactly the same approach , but the features that the VAD neural network use are , MFCC after noise compensation . professor b: What was it using before ? phd c: Before it was just P L phd d:  phd c: So . phd c: Yeah , phd d: Yeah , yeah , yeah , phd c: noisy noisy features . phd c:  This is what we get after This So , actually , we , yeah , here the features are noise compensated and there is also the LDA filter .  , and then it 's a pretty small neural network which use , nine frames of of six features from C - zero to C - fives , plus the first derivatives . phd a: Is that nine frames u s  , centered around the current frame ? Or phd c: Yeah . professor b: S so , I 'm I 'm sorry , there 's there 's there 's how many how many inputs ? phd c: So it 's twelve times nine . So I guess about eleven thousand parameters , which actually shouldn't be a problem , even in in small phones . phd a: So , I 'm I 'm s so what is different between this and and what you phd c: It should be OK . So the previous syst It 's based on the system that has a fifty - three point sixty - six percent improvement . The only thing that changed is the n a p eh a es the estimation of the silence probabilities . phd c:  So it 's it 's not bad , but the problem is still that the latency is too large . professor b: What 's the latency ? phd c: Because  the the latency of the VAD is two hundred and twenty milliseconds . And , the VAD is used  , i for on - line normalization , and it 's used before the delta computation . So if you add these components it goes t to a hundred and seventy , right ? professor b: I I 'm confused . You started off with two - twenty and you ended up with one - seventy ? phd c: With two an two hundred and seventy . phd c: If Yeah , if you add the c delta comp delta computation professor b: Oh . I the is this are these twenty - millisecond frames ? Is that why ? Is it after downsampling ? or phd c: The two - twenty is one hundred milliseconds for the  No , it 's forty milliseconds for t for the , cleaning of the speech . phd c: Yeah , and there is the delta at the input which is , professor b: One hundred milliseconds for smoothing . phd c:  phd d: It 's like forty plus forty plus professor b: And then forty phd c: Mmm . phd c: Yeah , there are twenty that comes from There is ten that comes from the LDA filters also . phd d: If you are using professor b:  phd c: Plus the frame , phd d: t If you are using three frames phd c: so it 's two - twenty . phd d: If you are phrasing f using three frames , it is thirty here for delta . So it 's like s five , six cepstrum plus delta at nine nine frames of professor b: And then ten milliseconds for phd d: Fi - There 's an LDA filter . professor b: ten milliseconds for LDA filter , and t and ten another ten milliseconds you said for the frame ? phd c: For the frame I guess . I computed two - twenty Yeah , well , it 's I guess it 's for the fr the professor b: OK . And then there 's delta besides that ? phd c: So this is the features that are used by our network and then afterwards , you have to compute the delta on the , main feature stream , professor b: OK . phd c: which is  , delta and double - deltas , which is fifty milliseconds . No , the after the noise part , the forty the the other hundred and eighty Well , Wait a minute . Some of this is ,  is ,  is in parallel , isn't it ?  , the LDA Oh , you have the LDA as part of the V D -  , VAD ? Or phd c: The VAD use , LDA filtered features also . professor b: so the delta at the end is how much ? phd c: It 's fifty . So phd c: But well , we could probably put the delta , before on - line normalization . It should not that make a big difference , phd a: What if you used a smaller window for the delta ? phd c: because phd a: Could that help a little bit ?  , I guess there 's a lot of things you could do to phd c: Yeah . phd c: Yeah , professor b: So phd c: but , nnn professor b: Yeah . So if you if you put the delta before the , ana on - line If Yeah phd c:   phd c: Cuz i professor b: And then y then you don't have that additive phd c: Yeah , phd d: Yep . phd c: cuz the time constant of the on - line normalization is pretty long compared to the delta window , professor b: OK . And you ought to be able to shove tw ,  sh  pull off twenty milliseconds from somewhere else to get it under two hundred , right ?  phd a: Is two hundred the d professor b: The hundred milla phd c:   It could be eighty and and probably do @ @ phd c: Yeah , phd a: i a hun phd c: yeah . phd a:  Wh - what 's the baseline you need to be under ? Two hundred ? professor b: Well , we don't know . professor b: if it 's two if if it 's ,  if it 's two - fifty , then we could keep the delta where it is if we shaved off twenty . If it 's two hundred , if we shaved off twenty , we could we could , meet it by moving the delta back . phd a: So , how do you know that what you have is too much if they 're still deciding ? professor b: we don't , but it 's just  , the main thing is that since that we got burned last time , and you know , by not worrying about it very much , we 're just staying conscious of it . professor b: And so , th  , if if if a week before we have to be done someone says , " Well , you have to have fifty milliseconds less than you have now " , it would be pretty frantic around here . professor b:  phd a: But still , that 's that 's a pretty big , win . And it doesn't seem like you 're in terms of your delay , you 're , that professor b: He added a bit on , I guess , because before we were we were had were able to have the noise , stuff , and the LVA be in parallel . phd c: Well , but I think the main thing , maybe , is the cleaning of the speech , which takes forty milliseconds or so . phd d: Yeah , the LDA LDA we don't know , is , like is it very crucial for the features , right ? phd c: No . professor b: so you could start pulling back , phd d: S s h professor b: but phd d: Yeah , professor b: But I think you have phd d: l professor b: you have twenty for delta computation which y now you 're sort of doing twice , right ? But yo w were you doing that before ? phd c: Mmm . Well , in the proposal , the input of the VAD network were just three frames , I think . professor b: So , what you have now is fort  , forty for the the noise , twenty for the delta , and ten for the LDA . That 's seventy milliseconds of stuff which was formerly in parallel , phd c:  professor b: right ? So I think , phd c:   professor b: you know , that 's that 's the difference as far as the timing , right ? phd c: Yeah . professor b: and you could experiment with cutting various pieces of these back a bit , but  , we 're s we 're not we 're not in terrible shape . professor b: It 's it 's not like it 's adding up to four hundred milliseconds or something . phd a: Where where is this where is this fifty - seven point O two in in comparison to the last evaluation ? professor b: Well , it 's I think it 's better than anything , anybody got . Yeah , and r and phd c: It would phd d: Yeah , so this is this is like the first proposal .  , I 'm sure other people working on this are not sitting still either , but phd a: Yeah . professor b: but but , the important thing is that we learn how to do this better , and , you know . So , our ,  Yeah , you can see the kind of kind of numbers that we 're having , say , on SpeechDat - Car which is a hard task , cuz it 's really ,  I think it 's just sort of sort of reasonable numbers , starting to be . Yeah , even for a well - matched case it 's sixty percent error rate reduction , professor b: Yeah . So actually , this is in between what we had with the previous VAD and what Sunil did with an IDL VAD . Which gave sixty - two percent improvement , right ? phd d: Yeah , it 's almost that . phd a: What was that ? Say that last part again ? phd c: So , if you use , like , an IDL VAD , for dropping the frames , phd d: o o Or the best we can get . phd c: the best that we can get i That means that we estimate the silence probability on the clean version of the utterances . phd a: So that would be even That wouldn't change this number down here to sixty - two ? phd c: Yeah . So you you were get phd c: If you add a g good v very good VAD , that works as well as a VAD working on clean speech , phd a: Yeah . phd c: then you wou you would go phd a: So that 's sort of the best you could hope for . professor b: And ,  and sixty - two with the the , you know , quote , unquote , cheating VAD . phd c: yeah , the next thing is , I started to play Well , I don't want to worry too much about the delay , no . And the other stream is the output of a neural network , using as input , also , these , cleaned MFCC .  phd a: Those are th those are th what is going into the tandem net ? phd c: I don't have the comp Mmm ? phd a: Those two ? phd c: So there is just this feature stream , the fifteen MFCC plus delta and double - delta . phd a: Yeah ? phd c: so it 's makes forty - five features that are used as input to the HTK . professor b: cuz then it has one part that 's discriminative , phd c: Yeah . Right now it seems that i I just tested on SpeechDat - Car while the experiment are running on your on TI - digits . Well , it improves on the well - matched and the mismatched conditions , but it get worse on the highly mismatched .  , professor b: y phd c: like , on the well - match and medium mismatch , the gain is around five percent relative , but it goes down a lot more , like fifteen percent on the HM case . professor b: You 're just using the full ninety features ? phd c:  The professor b: Y you have ninety features ? phd c: i I have ,  From the networks , it 's twenty - eight . It 's i i i It 's because it 's what we did for the first proposal . phd c: But we have to for sure , we have to go down , because the limit is now sixty features .  phd a: So , it seems funny that I don't know , maybe I don't u quite understand everything , but that adding features I guess I guess if you 're keeping the back - end fixed . But I guess if you 're keeping the number of Gaussians fixed in the recognizer , then professor b: Well , yeah . professor b: But , just in general , adding information Suppose the information you added , well , was a really terrible feature and all it brought in was noise . professor b: Right ? So so ,  Or or suppose it wasn't completely terrible , but it was completely equivalent to another one feature that you had , except it was noisier . professor b: Right ? In that case you wouldn't necessarily expect it to be better at all . I 'm just surprised that you 're getting fifteen percent relative worse on the wel professor b:  - huh . professor b: So , " highly mismatched condition " means that in fact your training is a bad estimate of your test . professor b: So having having , a g a l a greater number of features , if they aren't maybe the right features that you use , certainly can e can easily , make things worse . If you have if you have , lots and lots of data , and you have and your your your training is representative of your test , then getting more sources of information should just help . professor b: So I wonder , Well , what 's your what 's your thought about what to do next with it ? phd c: I don't know . I 'm surprised , because I expected the neural net to help more when there is more mismatch , as it was the case for the professor b:   phd c: Yeah , it 's the same training set , so it 's TIMIT with the TI - digits ' , noises , added . phd c:  professor b: Well , we might  , we might have to experiment with ,  better training sets . professor b: I The other thing is , before you found that was the best configuration , but you might have to retest those things now that we have different The rest of it is different , right ? So , For instance , what 's the effect of just putting the neural net on without the o other other path ? phd c:   professor b: You don't necessarily know what phd a: What if you did the Would it make sense to do the KLT on the full set of combined features ? Instead of just on the phd c: Yeah . The reason I did it this ways is that in February , it we we tested different things like that , so , having two KLT , having just a KLT for a network , or having a global KLT . phd c: And phd a: So you tried the global KLT before phd c: Well phd a: and it didn't really phd c: Yeah . phd c: The differences between these configurations were not huge , but it was marginally better with this configuration . professor b: And I guess if the These are all so all of these seventy - three features are going into , the ,  the  . professor b: And is are i i are are any deltas being computed of tha of them ? phd c: Of the straight features , yeah . phd c: So , yeah , maybe we can add some context from these features also as Dan did in in his last work . i Yeah , but the other thing I was thinking was , now I lost track of what I was thinking . phd a: What is the You said there was a limit of sixty features or something ? phd c:   phd a: What 's the relation between that limit and the , forty - eight  , forty eight hundred bits per second ? professor b: Oh , I know what I was gonna say . phd a: So I I I don't understand , phd c: The f the forty - eight hundred bits is for transmission of some features . phd a: because i  , if you 're only using h phd c: And generally , i it s allows you to transmit like , fifteen , cepstrum . professor b: The issue was that , this is supposed to be a standard that 's then gonna be fed to somebody 's recognizer somewhere which might be , you know , it it might be a concern how many parameters are use u used and so forth . What I was going to say is that , maybe maybe with the noise removal , these things are now more correlated . So you have two sets of things that are kind of uncorrelated , within themselves , but they 're pretty correlated with one another . professor b: And , they 're being fed into these , variants , only Gaussians and so forth , and and , phd c:   professor b: so maybe it would be a better idea now than it was before to , have , one KLT over everything , to de - correlate it . phd d: What are the S N Rs in the training set , TIMIT ? phd c: It 's , ranging from zero to clean ? Yeah . So we found this this ,  this Macrophone data , and so forth , that we were using for these other experiments , to be pretty good . professor b: So that 's i after you explore these other alternatives , that might be another way to start looking , is is just improving the training set . professor b: we were getting , lots better recognition using that , than Of course , you do have the problem that , u i we are not able to increase the number of Gaussians , or anything to , to match anything . So we 're only improving the training of our feature set , but that 's still probably something . phd a: So you 're saying , add the Macrophone data to the training of the neural net ? The tandem net ? professor b: Yeah , that 's the only place that we can train . professor b: We can't train the other stuff with anything other than the standard amount , phd a: Right .  ,  phd a: What what was it trained on again ? The one that you used ? phd c: It 's TIMIT with noise . phd c: So , yeah , it 's rather a small professor b: How big is the net , by the way ? phd c: it 's , five hundred hidden units . And professor b: And again , you did experiments back then where you made it bigger and it and that was that was sort of the threshold point . @ @ ? phd d: So is it is it though the performance , big relation in the high ma high mismatch has something to do with the , cleaning up that you that is done on the TIMIT after adding noise ? phd c:  phd d: So it 's i All the noises are from the TI - digits , phd c: Yeah . phd d: right ? So you i phd c:  They k  phd d: Well , it it 's like the high mismatch of the SpeechDat - Car after cleaning up , maybe having more noise than the the training set of TIMIT after clean s after you do the noise clean - up . phd d: So it had like all these different conditions of S N Rs , actually in their training set of neural net . phd d: But after cleaning up you have now a different set of S N Rs , right ? phd c: Yeah . phd d: And is it something to do with the mismatch that that 's created after the cleaning up , like the high mismatch phd c: You mean the the most noisy occurrences on SpeechDat - Car might be a lot more noisy than phd d:   professor b: but , you 're saying Yeah , the noisier ones are still going to be , even after our noise compensation , are still gonna be pretty noisy . phd d: Yeah , so now the after - noise compensation the neural net is seeing a different set of S N Rs than that was originally there in the training set . phd d: And that SNR may not be , like , com covering the whole set of S N Rs that you 're getting in the SpeechDat - Car . professor b: Right , but the SpeechDat - Car data that you 're seeing is also reduced in noise by the noise compensation . phd c: Well , if the initial range of SNR is different , we the problem was already there before . phd c: Because Mmm professor b: Yeah , it depends on whether you believe that the noise compensation is equally reducing the noise on the test set and the training set .  professor b: Right ?  , you 're saying there 's a mismatch in noise that wasn't there before , phd d:  . professor b: but if they were both the same before , then if they were both reduic reduced equally , then , there would not be a mismatch . professor b: So , this may be Heaven forbid , this noise compensation process may be imperfect , but . I I just that could be seen from the TI - digits , testing condition because , the noises are from the TI - digits , right ? Noise phd c: Yeah . So phd d: So cleaning up the TI - digits and if the performance goes down in the TI - digits mismatch high mismatch like this phd c: Clean training , yeah . professor b: the Macrophone data , I think , you know , it was recorded over many different telephones . I don't think there 's anybody recording over a car from a car , but I think it 's it 's varied enough that if if doing this adjustments , and playing around with it doesn't , make it better , the most  , it seems like the most obvious thing to do is to improve the training set .   , what we were  the condition It it gave us an enormous amount of improvement in what we were doing with Meeting Recorder digits , even though there , again , these m Macrophone digits were very , very different from , what we were going on here . But it was just I think just having a a nice variation in acoustic conditions was just a good thing . phd c: Yeah , actually to s eh , what I observed in the HM case is that the number of deletion dramatically increases . phd c: When I added the num the neural network it doubles the number of deletions . Yeah , so I don't you know how to interpret that , but , mmm professor b: Yeah . phd c: t phd a: And and did an other numbers stay the same ? Insertion substitutions stay the same ? phd c: They p stayed the same , phd a: Roughly ? phd c: they maybe they are a little bit  , lower . But professor b: Did they increase the number of deletions even for the cases that got better ? phd c:   professor b: And it Remind me again , the " highly mismatched " means that the phd c: Clean training and professor b: sorry ? phd c: It 's clean training Well , close microphone training and distant microphone , high speed , I think . professor b: Close mike training phd c: Well The most noisy cases are the distant microphone for testing . phd c: The feature are the same except that professor b: that 's right , that 's right .  phd a: Well that that says that , you know , the ,  the models in in , the recognizer are really paying attention to the neural net features . professor b: But , yeah , actually the TIMIT noises are sort of a range of noises and they 're not so much the stationary driving kind of noises , right ? It 's it 's pretty different . " phd c: " Subway " , right ? and phd d: " Street " or " Airport " or something . phd c: So it 's mostly Well , " Car " is stationary , professor b:   phd c: " Babble " , it 's a stationary background plus some voices , professor b:   professor b: Well , I I think that if you run it Actually , you maybe you remember this . When you in in the old experiments when you ran with the neural net only , and didn't have this side path , with the the pure features as well , did it make things better to have the neural net ? phd c:   professor b: Was it about the same ?  , w i phd c: It was b a little bit worse . professor b: So , until you put the second path in with the pure features , the neural net wasn't helping at all . as soon as we added LDA on - line normalization , and all these things , then professor b: They were doing similar enough things . Well , I still think it would be k sort of interesting to see what would happen if you just had the neural net without the side thing . phd c: Yeah , professor b: And and the thing I I have in mind is , maybe you 'll see that the results are not just a little bit worse . You know ? And ,  But if on the ha other hand , it 's , say , somewhere in between what you 're seeing now and and and , what you 'd have with just the pure features , then maybe there is some problem of a of a , combination of these things , or correlation between them somehow . professor b: If it really is that the net is hurting you at the moment , then I think the issue is to focus on on , improving the the net . professor b: So what 's the overall effe  , you haven't done all the experiments but you said it was i somewhat better , say , five percent better , for the first two conditions , and fifteen percent worse for the other one ? But it 's but of course that one 's weighted lower , phd c: Y yeah , oh . That 's not that bad , but it was l like two percent relative worse on SpeechDat - Car . phd d: Well , it will overall it will be still better even if it is fifteen percent worse , because the fifteen percent worse is given like f w twenty - five point two five eight . So the so the worst it could be , if the others were exactly the same , is four , phd d: Is it like professor b: and and , in fact since the others are somewhat better phd d: Yeah , so it 's four . Is i So either it 'll get cancelled out , or you 'll get , like , almost the same . phd a: In the ,  a lot of the ,  the Hub - five systems , recently have been using LDA . So there 's the the LDA is is right there before the H M phd d: Yeah . phd a: So , you guys are using LDA but it seems like it 's pretty far back in the process . The LDA that you saying is , like , you take a block of features , like nine frames or something , and then do an LDA on it , phd a: Yeah . phd a: it 's you know , you 're just basically i phd d: Yeah , so this is like a two d two dimensional tile . And the LDA that we are f applying is only in time , not in frequency high cost frequency . So it 's like more like a filtering in time , rather than doing a r phd a: Ah . So what i what about ,  i u what i w  , I don't know if this is a good idea or not , but what if you put ran the other kind of LDA , on your features right before they go into the  ? phd d: it phd c:   phd a: But  , w but the other features that you have , th the non - tandem ones , phd c:  . Well , in the proposal , they were transformed u using PCA , but phd a:  - huh . professor b: The a the argument i is kind of i in and it 's not like we really know , but the argument anyway is that , we always have the prob  , discriminative things are good . professor b: they 're good because you you you learn to distinguish between these categories that you want to be good at distinguishing between . It PAC - PCA low - order PCA throws away pieces that are  , maybe not not gonna be helpful just because they 're small , basically . professor b: But , the problem is , training sets aren't perfect and testing sets are different . So you f you you face the potential problem with discriminative stuff , be it LDA or neural nets , that you are training to discriminate between categories in one space but what you 're really gonna be g getting is is something else . professor b: And so , Stephane 's idea was , let 's feed , both this discriminatively trained thing and something that 's not . So you have a good set of features that everybody 's worked really hard to make , phd a: Yeah . professor b: and then , you you discriminately train it , but you also take the path that that doesn't have that , phd a:  - huh . And that that seem So it 's kind of like a combination of the  , what , Dan has been calling , you know , a feature  , you know , a feature combination versus posterior combination or something . It 's it 's , you know , you have the posterior combination but then you get the features from that and use them as a feature combination with these these other things . And that seemed , at least in the last one , as he was just saying , he he when he only did discriminative stuff , i it actually was was it didn't help at all in this particular case . But by having them both there The fact is some of the time , the discriminative stuff is gonna help you . professor b: And some of the time it 's going to hurt you , phd a: Right . professor b: and by combining two information sources if , you know if if phd a: So you wouldn't necessarily then want to do LDA on the non - tandem features because now you 're doing something to them that professor b: That i i I think that 's counter to that idea . But if that 's the hypothesis , at least it would be counter to that hypothesis to do that . professor b: and in principle you would think that the neural net would do better at the discriminant part than LDA .  , we ,  we were getting ready to do the tandem , stuff for the Hub - five system , and , Andreas and I talked about it , and the idea w the thought was , " Well , yeah , that i you know th the neural net should be better , but we should at least have  , a number , you know , to show that we did try the LDA in place of the neural net , so that we can you know , show a clear path . phd a: You know , that you have it without it , then you have the LDA , then you have the neural net , and you can see , theoretically . professor b: or tha that 's a phd a: That 's what that 's what we 're gonna do next as soon as I finish this other thing . phd a: it everybody believes it , professor b: Oh , no it 's a g phd a: but you know , we just professor b: No , no , but it might not not even be true . professor b: it 's it 's it 's it 's it 's a great idea .  , one of the things that always disturbed me , in the the resurgence of neural nets that happened in the eighties was that , a lot of people Because neural nets were pretty easy to to use a lot of people were just using them for all sorts of things without , looking at all into the linear , versions of them . professor b: And , people were doing recurrent nets but not looking at IIR filters , and You know , so I think , yeah , it 's definitely a good idea to try it . phd a: Yeah , and everybody 's putting that on their systems now , and so , I that 's what made me wonder about this , professor b: Well , they 've been putting them in their systems off and on for ten years , phd a: but . professor b: but but but , phd a: Yeah , what  is it 's it 's like in the Hub - five evaluations , you know , and you read the system descriptions and everybody 's got , you know , LDA on their features . phd c: It 's the transformation they 're estimating on Well , they are trained on the same data as the final  are . Cuz they don't have these , you know , mismatches that that you guys have . phd a: So that 's why I was wondering if maybe it 's not even a good idea . professor b: part of why I I think part of why you were getting into the KLT Y you were describing to me at one point that you wanted to see if , you know , getting good orthogonal features was and combining the the different temporal ranges was the key thing that was happening or whether it was this discriminant thing , right ? So you were just trying I think you r  , this is it doesn't have the LDA aspect but th as far as the orthogonalizing transformation , you were trying that at one point , right ? phd c:   phd d: So , yeah , I 've been exploring a parallel VAD without neural network with , like , less latency using SNR and energy , after the cleaning up . So what I 'd been trying was ,  After the b after the noise compensation , n I was trying t to f find a f feature based on the ratio of the energies , that is , cl after clean and before clean . So that if if they are , like , pretty c close to one , which means it 's speech . And if it is n if it is close to zero , which is So it 's like a scale @ @ probability value . So I was trying , with full band and multiple bands , m ps  separating them to different frequency bands and deriving separate decisions on each bands , and trying to combine them .  , the advantage being like it doesn't have the latency of the neural net if it if it can professor b:   phd d: just like Which means that it 's it 's doing a slightly better job than the previous VAD , professor b:   phd d: so ,  professor b: But i d I 'm sorry , phd d: so u professor b: does it still have the median filter stuff ? phd d: It still has the median filter . professor b: So it still has most of the delay , phd d: So professor b: it just doesn't phd d: Yeah , so d with the delay , that 's gone is the input , which is the sixty millisecond . professor b: Well , w i phd d: At the input of the neural net you have this , f nine frames of context plus the delta . phd d: so the delay is only the forty millisecond of the noise cleaning , plus the hundred millisecond smoothing at the output . So the the di the biggest The problem f for me was to find a consistent threshold that works well across the different databases , because I t I try to make it work on tr SpeechDat - Car professor b:   phd d: and it fails on TI - digits , or if I try to make it work on that it 's just the Italian or something , it doesn't work on the Finnish . So there are there was , like , some problem in balancing the deletions and insertions when I try different thresholds . phd d: So The I 'm still trying to make it better by using some other features from the after the p clean up maybe , some , correlation auto - correlation or some s additional features of to mainly the improvement of the VAD . professor b: Now this this this , " before and after clean " , it sounds like you think that 's a good feature . That that , it you th think that the ,  the i it appears to be a good feature , right ? phd d:   phd c: Yeah , eventually we could could just phd d: Yeah , so Yeah , so that 's the Yeah . phd d: Because they did that itself phd c: Then you don't have to worry about the thresholds and phd d: There 's a threshold and Yeah . So if we if we can live with the latency or cut the latencies elsewhere , then then that would be a , good thing . professor b: anybody has anybody you guys or or Naren , somebody , tried the , second th second stream thing ?  . phd d: Oh , I just I just h put the second stream in place and ,  ran one experiment , but just like just to know that everything is fine . phd d: So it was like , forty - five cepstrum plus twenty - three mel log mel . phd d: And and , just , like , it gave me the baseline performance of the Aurora , which is like zero improvement . phd d: So I just tried it on Italian just to know that everything is But I I didn't export anything out of it because it was , like , a weird feature set . Well , what I think , you know , would be more what you 'd want to do is is is , put it into another neural net . phd d: The  , other thing I was wondering was , if the neural net , has any because of the different noise con unseen noise conditions for the neural net , where , like , you train it on those four noise conditions , while you are feeding it with , like , a additional some four plus some f few more conditions which it hasn't seen , actually , phd c:   phd d:  instead of just h having c  , those cleaned up t cepstrum , sh should we feed some additional information , like The the We have the VAD flag .  , should we f feed the VAD flag , also , at the input so that it it has some additional discriminating information at the input ? phd c:  -  !  professor b: Wh -  , the the VAD what ? phd d: We have the VAD information also available at the back - end . phd d: So if it is something the neural net is not able to discriminate the classes professor b: Yeah . phd d:  Because most of it is sil  , we have dropped some silence f We have dropped so silence frames ? professor b:   So , by having an additional , feature which says " this is speech and this is nonspeech " , it certainly helps in some unseen noise conditions for the neural net . phd a: What Do y do you have that feature available for the test data ? phd d: Well , we have we are transferring the VAD to the back - end feature to the back - end . phd d: Which is which is certainly giving a phd a: So you 're saying , feed that , also , into the neural net . The other thing you could do is just , p modify the , output probabilities of the of the , neural net , tandem neural net , based on the fact that you have a silence probability . professor b: So you have an independent estimator of what the silence probability is , and you could multiply the two things , and renormalize . professor b: But but ,  phd c: Yeah , so maybe , yeah , when phd a: But in principle wouldn't it be better to feed it in ? And let the net do that ? professor b: Well , u Not sure . " Or you can say , " It 's silence ! Go away ! "  , i Doesn't ? I think I think the second one sounds a lot more direct . So , what if you then ,  since you know this , what if you only use the neural net on the speech portions ? professor b: Well , phd c: That 's what phd a: Well , I guess that 's the same . professor b: Yeah , y you 'd have to actually run it continuously , phd a: But   , train the net only on professor b: but it 's @ @ Well , no , you want to train on on the nonspeech also , because that 's part of what you 're learning in it , to to to generate , that it 's it has to distinguish between . phd a: But  , if you 're gonna if you 're going to multiply the output of the net by this other decision , would then you don't care about whether the net makes that distinction , right ? professor b: Well , yeah . Now the only thing that that bothers me about all this is that I I I The the fact i i It 's sort of bothersome that you 're getting more deletions . But So I might maybe look at , is it due to the fact that  , the probability of the silence at the output of the network , is , professor b: Is too high . So maybe So phd c: If it 's the case , then multiplying it again by i by something ? phd d: It may not be it professor b: Yeah . phd d: Yeah , it it may be too it 's too high in a sense , like , everything is more like a , flat probability . phd d: So , like , it 's not really doing any distinction between speech and nonspeech phd c: yeah . phd a: Be interesting to look at the Yeah , for the I wonder if you could do this . But if you look at the , highly mism high mismat the output of the net on the high mismatch case and just look at , you know , the distribution versus the the other ones , do you do you see more peaks or something ? phd c: Yeah . phd c: or professor b: But I bu phd c: It it seems that the VAD network doesn't Well , it doesn't drop , too many frames because the dele the number of deletion is reasonable . But it 's just when we add the tandem , the final MLP , and then professor b: Yeah . Now the only problem is you don't want to ta I guess wait for the output of the VAD before you can put something into the other system , phd c: u professor b: cuz that 'll shoot up the latency a lot , right ? Am I missing something here ? phd c: But phd d:   But but I I guess phd a: But if you were gonna put it in as a feature it means you already have it by the time you get to the tandem net , right ? phd d: well . professor b: It 's kind of done in  , some of the things are , not in parallel , but certainly , it would be in parallel with the with a tandem net . So maybe , if that doesn't work ,  But it would be interesting to see if that was the problem , anyway . And and and then I guess another alternative would be to take the feature that you 're feeding into the VAD , and feeding it into the other one as well . professor b:  But that 's Yeah , that 's an interesting thing to try to see , if what 's going on is that in the highly mismatched condition , it 's , causing deletions by having this silence probability up up too high , phd c:   phd c: So , m professor b: Cuz Well , the V A if the VAD said since the VAD is is is right a lot ,  phd c: Yeah . Yeah , and the other thing Well , there are other issues maybe for the tandem , like , well , do we want to , w  n Do we want to work on the targets ? Or , like , instead of using phonemes , using more context dependent units ? phd a: For the tandem net you mean ? phd c: Well , I 'm Yeah . phd c: I 'm thinking , also , a w about Dan 's work where he he trained a network , not on phoneme targets but on the  state targets . professor b: Problem is , if you are going to run this on different m test sets , including large vocabulary , phd c: Yeah . I was just thinking maybe about , like , generalized diphones , and come up with a a reasonable , not too large , set of context dependent units , and and Yeah . But I d I d it it i it 's all worth looking at , phd c:   professor b: but it sounds to me like , looking at the relationship between this and the speech noise stuff is is is probably a key thing . phd a: So if ,  if the , high mismatch case had been more like the , the other two cases in terms of giving you just a better performance , how would this number have changed ? phd c:   If if i phd a: y Like sixty ? professor b: Well , we don't know what 's it 's gonna be the TI - digits yet . If you extrapolate the SpeechDat - Car well - matched and medium - mismatch , it 's around , yeah , maybe five . phd c: Well , it 's around five percent , because it 's s Right ? If everything is five percent . phd c: I d I d I just have the SpeechDat - Car right now , so phd a: Yeah . phd c: It 's running it shou we should have the results today during the afternoon , phd a:  . Well  So I won't be here for phd a: When When do you leave ? professor b: I 'm leaving next Wednesday .  , phd a: But you 're professor b: so I phd a: are you you 're not gonna be around this afternoon ? professor b: Yeah . professor b: This afternoon  Oh , right , for the Meeting meeting ? Yeah , that 's just cuz of something on campus . But , yeah , so next week I won't , and the week after I won't , cuz I 'll be in Finland .  , phd a: What 's September sixth ? professor b: and  , that 's during Eurospeech .  Right ? So it 'll be a few weeks , really , before we have a meeting of the same cast of characters . And and then  , we 'll start up again with Dave and Dave and Barry and Stephane and us on the , twentieth . Thirteenth ? About a month ? phd a: So , you 're gonna be gone for the next three weeks or something ? professor b: I 'm gone for two and a half weeks starting starting next Wed - late next Wednesday . Is that right ? professor b: I won't it 's probably four because of is it three ? Let 's see , twenty - third , thirtieth , sixth . And the the third one won't probably won't be a meeting , cuz cuz , Su - Sunil , Stephane , and I will all not be here . So it 's just , the next two where there will be there , you know , may as well be meetings , phd a: OK . And then starting up on the thirteenth , we 'll have meetings again but we 'll have to do without Sunil here somehow . phd a: When is the evaluation ? November , or something ? professor b: Yeah , it was supposed to be November fifteenth


professor b: So ne next week we 'll have , both Birger and , Mike Michael Michael Kleinschmidt and Birger Kollmeier will join us . professor b: and you 're you 're probably gonna go up in a couple three weeks or so ? When d when are you thinking of going up to , OGI ? phd d: Yeah , like , not next week but maybe the week after . So at least we 'll have one meeting with yo with you still around , and and phd d:  - huh . So there was this conference call this morning , and the only topic on the agenda was just to discuss a and to come at  , to get a decision about this latency problem . professor b: No , this I 'm sorry , this is a conference call between different Aurora people or just ? phd d: yeah . There were like two hours of discussions , and then suddenly , people were tired , I guess , and they decided on a number , two hundred and twenty , included e including everything . W It 's it 's p d primary primarily determined by the VAD at this point , phd d:  . We probably should do that pretty soon so that we don't get used to it being a certain way . We have , like , a system that gives sixty - two percent improvement , but if you want to stick to the this latency Well , it has a latency of two thirty , but if you want also to stick to the number of features that limit it to sixty , then we go a little bit down but it 's still sixty - one percent . And i is the tandem network , small enough that it will fit on the terminal size in terms of ? phd d: no , I don't think so . phd d: It 's still in terms of computation , if we use , like , their way of computing the the maps the the MIPs , I think it fits , professor b:   phd d: and I don't know how much this can be discussed or not , because it 's it could be in ROM , so it 's maybe not that expensive . But professor b: Ho - how much memory d ? H how many ? phd d: I d I d  , I I don't kn remember exactly , but  . I 'd like to see that , cuz maybe I could think a little bit about it , cuz we maybe we could make it a little smaller or  , it 'd be it 'd be neat if we could fit it all . professor b: But I guess it 's still within their rules to have have it on the , t  , server side .  , mmm , and one of the trick was to , use some kind of hierarchical structure where the silence probability is not computed by the final tandem network but by the VAD network .  , so apparently it looks better when , we use the silence probability from the VAD network professor b: Huh . So it 's some kind of hierarchical thing , that Sunil also tried , on SPINE and apparently it helps a little bit also . Yeah , the reason w why why we did that with the silence probability was that ,  professor b: Could ?  , I 'm I 'm really sorry . So there is the tandem network that e e e estimates the phone probabilities professor b: Yeah . phd d: And things get better when , instead of using the silence probability computed by the tandem network , we use the silence probability , given by the VAD network , professor b: Oh . phd d: professor b: The VAD network is ? phd d: Which is smaller , but maybe ,  So we have a network for the VAD which has one hundred hidden units , and the tandem network has five hundred . phd d: Maybe it 's has something to do to the fact that we don't have infinite training data and professor b: We don't ? phd d: Well ! And so Well , things are not optimal professor b: Yeah . phd d: and Mmm grad e: Are you you were going to say why what made you wh what led you to do that .  , there was a p problem that we observed , that there was there were , like , many insertions in the in the system . phd d: Actually plugging in the tandem network was increasing , I I I think , the number of insertions . professor b: So , you know , in a way what it might i it 's it 's a little bit like combining knowledge sources . phd d:  professor b: Right ? Because the fact that you have these two nets that are different sizes means they behave a little differently , phd d:   And , if you have ,  f the distribution that you have from , f speech sounds is w sort of one source of knowledge . professor b: And this is and rather than just taking one minus that to get the other , which is essentially what 's happening , you have this other source of knowledge that you 're putting in there . Well , there are other things that we should do but , it requires time and We have ideas , like so , these things are like hav having a better VAD . Of course we have ideas on this also , but w we need to try different things and  , but their noise estimation ,   professor b: back on the second stream , that 's something we 've talked about for a while . professor b: so we have this this default idea about just using some sort of purely spectral thing ? phd d: yeah . professor b: for a second stream ? phd d: But , we we did a first try with this , and it it clearly hurts . So , if you just had a second stream that was just spectral and had another neural net and combined there , that that , might be good .  Yeah , and the other thing , that noise estimation and th  , maybe try to train  , the training data for the t tandem network , right now , is like i is using the noises from the Aurora task and I think that people might , try to argue about that because then in some cases we have the same noises in for training the network than the noises that are used for testing , professor b: Right . Well , that 's something I would like to investigate further , but , I did , like ,  I did , listen to the m most noisy utterances of the SpeechDat - Car Italian and tried to transcribe them . phd d: but but still , what happens is is that , the digit error rate on this is around one percent , professor b: Yeah .  , but what happens also is that if I listen to the ,  a re - synthesized version of the speech and I re - synthesized this using a white noise that 's filtered by a LPC , filter professor b: Yeah . phd d: well , you can argue , that ,  that this is not speech , professor b: Yeah . But s actually it sound like whispering , so we are professor b: Well , it 's phd d: eh professor b: There 's two problems there .   , so so the first is that by doing LPC - twelve with synthesized speech w like you 're saying , it 's i i you 're you 're adding other degradation . professor b: Right ? So it 's not just the noise but you 're adding in fact some degradation because it 's only an approximation .  , and the second thing is which is m maybe more interesting is that , if you do it with whispered speech , you get this number . What if you had done analysis re - synthesis and taken the pitch as well ? Alright ? So now you put the pitch in . professor b: What would the percentage be then ? phd d:  professor b: See , that 's the question . So , you see , if it 's if it 's if it 's ,  Let 's say it 's back down to one percent again . professor b: That would say at least for people , having the pitch is really , really important , which would be interesting in itself . But professor b: if i on the other hand , if it stayed up near five percent , then I 'd say " boy , LPC n twelve is pretty crummy " . professor b: So I I I 'm not sure I 'm not sure how we can conclude from this anything about that our system is close to the human performance . Well , the point is that eh l ey the point is that , what I what I listened to when I re - synthesized the LP - the LPC - twelve spectrum is in a way what the system , is hearing , cuz @ @ all the all the , excitation all the well , the excitation is is not taken into account . And professor b: Well , you 're not doing the LPC phd d: in this case professor b: so so what if you did a phd d: Well , it 's not LPC , sure , professor b: What if you did LPC - twenty ? phd d: but LPC ? professor b: Twenty . professor b: So , all I 'm saying is that you have in addition to the w the , removal of pitch , you also are doing , a particular parameterization , phd d:   professor b: so , let 's see , how would you do ? So , fo phd d: But that 's that 's what we do with our systems . Actually , we d we we don't , because we do we do , mel filter bank , for instance . Right ? phd d: Yeah , but is it that is it that different ,  ? professor b: I don't know what mel , based synthesis would sound like , phd d: I professor b: but certainly the spectra are quite different . phd a: Couldn't you t couldn't you , test the human performance on just the original audio ? phd d:   So , y  , your performance was one percent , and then when you re - synthesize with LPC - twelve it went to five . professor b:  We were we were j It it it 's a little bit still apples and oranges because we are choosing these features in order to be the best for recognition . professor b: And , i if you listen to them they still might not be very Even if you made something closer to what we 're gonna i it might not sound very good . professor b: and i the degradation from that might might actually make it even harder , to understand than the LPC - twelve . So all I 'm saying is that the LPC - twelve puts in synthesis puts in some degradation that 's not what we 're used to hearing , phd d:  - huh . professor b: and is ,  It 's not it 's not just a question of how much information is there , as if you will always take maximum advantage of any information that 's presented to you . And so it it isn't phd a: But professor b: But , I agree that it says that , the kind of information that we 're feeding it is probably , a little bit , minimal . And that 's why I was saying it might be interesting if you an interesting test of this would be if you if you actually put the pitch back in . So , you just extract it from the actual speech and put it back in , and see does that is that does that make the difference ? If that if that takes it down to one percent again , then you 'd say " OK , it 's it 's in fact having , not just the spectral envelope but also the also the the pitch that , @ @ has the information that people can use , anyway . phd a: But from this it 's pretty safe to say that the system is with either two to seven percent away from the performance of a human . professor b: Well , or it 's it 's phd a: Two two to six percent . professor b: Yeah , so It 's it 's one point four times , to , seven times the error , phd d: To f seven times , yeah . phd d: But but professor b: But that 's that 's what that 's the first thing that I would be curious about , is , you know , i i when you we phd d: But the signal itself is like a mix of  , of a a periodic sound and , @ @  , unvoiced sound , and the noise professor b:   So , what what do you mean exactly by putting back the pitch in ? Because phd a: In the LPC synthesis ? I think professor b: Yeah . professor b: So ,  and you did it with a noise source , rather than with with a s periodic source . professor b: Right ? So if you actually did real re - synthesis like you do in an LPC synthesizer , where it 's unvoiced you use noise , where it 's voiced you use , periodic pulses . professor b: Right ? phd d: Yeah , but it 's neither purely voiced or purely unvoiced . professor b: Well , it might be hard to do it phd d: So professor b: but it but but the thing is that if you  , if you detect that there 's periodic s strong periodic components , then you can use a voiced voice thing . professor b: But I 'm I 'm just saying , at least as a thought experiment , that 's what I would wanna test . professor b: I wan would wanna drive it with a a a two - source system rather than a than a one - source system . professor b: And then that would tell you whether in fact it 's Cuz we 've talked about , like , this harmonic tunneling or other things that people have done based on pitch , maybe that 's really a key element . Maybe maybe , without that , it 's it 's not possible to do a whole lot better than we 're doing . Evi professor b: But , other than that , I don't think it 's  , other than the pitch de information , it 's hard to imagine that there 's a whole lot more in the signal that that ,  that we 're throwing away that 's important . professor b: Right ?  , we 're using a fair number of filters in the filter bank and  phd d:   If somebody was paying really close attention , you might get I would actually think that if , you looked at people on various times of the day and different amounts of attention , you might actually get up to three or four percent error on digits . professor b: So it 's you know , we 're not we 're not incredibly far off . On the other hand , with any of these numbers except maybe the one percent , it 's st it 's not actually usable in a commercial system with a full telephone number or something .  , while we 're still on Aurora stuff maybe you can talk a little about the status with the , Wall Street Journal things for it . They wrote some scripts that sort of make it easy to run the system on the Wall Street Journal , data .  , I 'm waiting there was one problem with part of it and I wrote a note to Joe asking him about it .  , they on their web site they , did this little table of where their system performs relative to other systems that have done this this task . So they 're professor b: This is on clean test set ? phd a: This is on clean on clean stuff . They they 've started a table where they 're showing their results on various different noise conditions but they they don't have a whole lot of it filled in and and I didn't notice until after I 'd printed it out that , they don't say here what these different testing conditions are . professor b:  phd a: You actually have to click on it on the web site to see them . professor b: What kind of numbers are they getting on these on the test conditions ? phd a: Well , see , I was a little confused because on this table , I 'm the they 're showing word error rate . But on this one , I I don't know if these are word error rates because they 're really big . professor b:  phd a: So m I guess maybe they 're error rates but they 're ,  they 're really high . professor b: I I I don't find that surpri phd a: So professor b: we W what 's what 's some of the lower error rates on on on  , some of the higher error rates on , some of these w  , highly mismatched difficult conditions ? What 's a ? phd d:  . phd a: Correct ? phd d: And the baseline , eh phd a: Accuracy ? phd d: error rate . professor b: So if you 're doing so if you 're doing , phd d: and phd a: Yeah . professor b: Yeah , and if you 're saying sixty - thousand word recognition , getting sixty percent error on some of these noise condition not at all surprising . phd d: The baseline is sixty percent also on digits , phd a: Oh , is it ? phd d: on the m more mismatched conditions . professor b: It 's a bad sign when you looking at the numbers , you can't tell whether it 's accuracy or error rate .  , they 're I I 'm still waiting for them to release the , multi - CPU version of their scripts , cuz right now their script only handles processing on a single CPU , which will take a really long time to run . But their s professor b: This is for the training ? phd a:  I beli Yes , for the training also . So , as soon as they get that , then I 'll I 'll grab those too professor b: OK . I 'll go ahead and try to run it though with just the single CPU one , professor b: if the phd a: and I they they , released like a smaller data set that you can use that only takes like sixteen hours to train and stuff . So I can I can run it on that just to make sure that the the thing works and everything . Is that about right you think ? phd d: we don't know yet , I I think . Did they say anything on the conference call about , how the Wall Street Journal part of the test was going to be run ? Because I I thought I remembered hearing that some sites were saying that they didn't have the compute to be able to run the Wall Street Journal stuff at their place , phd d: No . Well , this first , this was not the point at all of this the meeting today phd a: Oh , OK . phd d: and , professor b: Some phd d: frankly , I don't know because I d didn't read also the most recent mails about the large - vocabulary task . But , did you do you still , get the mails ? You 're not on the mailing list or what ? phd a:   I I don't get any mail about professor b: I have to say , there 's  something funny - sounding about saying that one of these big companies doesn't have enough cup compute power do that , so they 're having to have it done by Mississippi State . phd a: because there 's this whole issue about , you know , simple tuning parameters , like word insertion penalties . If you change your front - end , you know , the scale is completely can be completely different , so . But phd d: You didn't get any answer from Joe ? phd a: I did , but Joe said , you know , " what you 're saying makes sense phd d:  - huh . phd a: that 's th We had this back and forth a little bit about , you know , are sites gonna are you gonna run this data for different sites ? And , well , if if Mississippi State runs it , then maybe they 'll do a little optimization on that parameter , and ,  But then he wasn't asked to run it for anybody . phd a: he 's been putting this stuff out on their web site and for people to grab but I haven't heard too much about what 's happening . professor b: So it could be  , Chuck and I had actually talked about this a couple times , and and over some lunches , I think , that , one thing that we might wanna do The - there 's this question about , you know , what do you wanna scale ? Suppose y you can't adjust these word insertion penalties and so forth , so you have to do everything at the level of the features . What could you do ? And , one thing I had suggested at an earlier time was maybe some sort of scaling , some sort of root or or something of the , features . But the problem with that is that isn't quite the same , it occurred to me later , because what you really want to do is scale the , @ @ the range of the likelihoods rather than phd d: Nnn , the dist Yeah . professor b: But , what might get at something similar , it just occurred to me , is kind of an intermediate thing is because we do this strange thing that we do with the tandem system , at least in that system what you could do is take the , values that come out of the net , which are something like log probabilities , and scale those . But but , you know So because what we 're doing is pretty strange and complicated , we don't really know what the effect is at the other end . professor b: So , my thought was maybe  , they 're not used as probabilities , but the log probabilities we 're taking advantage of the fact that something like log probabilities has more of a Gaussian shape than Gaus - than probabilities , and so we can model them better . So , in a way we 're taking advantage of the fact that they 're probabilities , because they 're this quantity that looks kind of Gaussian when you take it 's log . And we may end up being in a situation where we just you know really can't change the word insertion penalty . But the other thing we could do is also we could  , this this may not help us , in the evaluation but it might help us in our understanding at least . We might , just run it with different insper insertion penalties , and show that , " well , OK , not changing it , playing the rules the way you wanted , we did this . " phd a: I wonder if it it might be possible to , simulate the back - end with some other system . So we we get our f front - end features , and then , as part of the process of figuring out the scaling of these features , you know , if we 're gonna take it to a root or to a power or something , we have some back - end that we attach onto our features that sort of simulates what would be happening . phd a: professor b: And just adjust it until it 's the best number ? phd a: and just adjust it until that our l version of the back - end , decides that that professor b: Well , we can probably use the real thing , can't we ? And then jus just , use it on a reduced test set or something . So  , I I think that that 's a reasonable thing to do and the only question is what 's the actual knob that we use ? phd a:   professor b: And the knob that we use should  , unfortunately , like I say , I don't know the analytic solution to this cuz what we really want to do is change the scale of the likelihoods , phd a:   grad e: Out of curiosity , what what kind of recognizer is the one from Mississippi State ? phd a: w what do you mean when you say " what kind " ? grad e: Is it ?  , is it like a Gaussian mixture model ? phd a: Yeah . phd a: It 's the same system that they use when they participate in the Hub - five evals .  , they started off with  , when they were building their system they were always comparing to HTK to make sure they were getting similar results . And so , it 's a Gaussian mixture system ,  professor b: Do they have the same sort of mix - down sort of procedure , where they start off with a small number of some things phd a: I don't know . professor b: D Do you know what kind of tying they use ? Are they they sort of some sort of a bunch of Gaussians that they share across everything ? Or or if it 's ? phd a: Yeah , th I have I I I don't have it up here but I have a the whole system description , that describes exactly what their system is professor b: OK . phd a: It 's some kind of a mixture of Gaussians and , clustering and ,  They 're they 're trying to put in sort of all of the standard features that people use nowadays . professor b: So the other , Aurora thing maybe is I I dunno if any of this is gonna come in in time to be relevant , but , we had talked about , Guenter playing around , over in Germany phd d:   professor b: and and , @ @  , possibly coming up with something that would , fit in later .  , I saw that other mail where he said that he  , it wasn't going to work for him to do CVS . professor b: So if he 'll he might work on improving the noise estimate or on some histogram things , or phd d: Yeah . I just saw the Eurospeech We we didn't talk about it at our meeting but I just saw the just read the paper . Someone , I forget the name , and and Ney , about histogram equalization ? Did you see that one ? phd d: it was a poster .  It was something similar to n on - line normalization finally  , in the idea of of normalizing professor b: Yeah . But it 's a little more it it 's a little finer , right ? So they had like ten quantiles phd d: Yeah . professor b: So you you have the distributions from the training set , phd d: N professor b: and then ,  So this is just a a histogram of of the amplitudes , I guess . And and and then , when you get a new new thing that you you want to adjust to be better in some way , you adjust it so that the histogram of the new data looks like the old data . professor b: You do this kind of piece - wise linear or , some kind of piece - wise approximation . They did a  one version that was piece - wise linear and another that had a power law thing between them between the points . And , they said they s they sort of see it in a way as s for the speech case as being kind of a generalization of spectral subtraction in a way , because , you know , in spectral subtraction you 're trying to get rid of this excess energy . And then they have s they have some kind of , a floor or something , grad e:  . phd a: So is this a histogram across different frequency bins ? professor b: and phd a: Or ? professor b: I think this i You know , I don't remember that . I  Something like one per frequency band , professor b: One phd a: So , one histogram per frequency bin . phd a: So th professor b: And I don't remember whether it was filter bank things phd a: Oh . phd d: and I didn't professor b: or whether it was FFT bins phd a: Huh . professor b: or phd a: And and that that , histogram represents the different energy levels that have been seen at that frequency ? professor b: I don't remember that . And they do they said that they could do it for the test So you don't have to change the training . phd a: So they ,  Is the idea that you you run a test utterance through some histogram generation thing and then you compare the histograms and that tells you what to do to the utterance to make it more like ? professor b: I guess in pri Yeah . professor b: whether it was some , on - line thing , or whether it was a second pass , or what . We 're sort of curious about , what are some things that are , u u  , @ @ conceptually quite different from what we 've done . professor b: Cuz we you know , one thing that w that , Stephane and Sunil seemed to find , was , you know , they could actually make a unified piece of software that handled a range of different things that people were talking about , and it was really just sort of setting of different constants . So , we 're not making any use of pitch , which again , might might be important , because the stuff between the harmonics is probably a schmutz . And and ,  And we there 's this overall idea of really sort of matching the the hi distributions somehow . So I guess , Guenter 's gonna play around with some of these things now over this next period , phd d: I dunno . professor b: or ? phd d: I don't have feedback from him , but professor b: Yeah . phd d: I guess he 's gonna , maybe professor b: Well , he 's got it anyway , so he can . professor b: So potentially if he came up with something that was useful , like a diff a better noise estimation module or something , he could ship it to you guys u up there phd d:  - huh . So , why don't we just ,  I think starting starting a w couple weeks from now , especially if you 're not gonna be around for a while , we 'll we 'll be shifting more over to some other other territory . But , n not not so much in this meeting about Aurora , but but , maybe just , quickly today about maybe you could just say a little bit about what you 've been talking about with Michael . So Michael Kleinschmidt , who 's a PHD student from Germany , showed up this week . And he 's done some work using an auditory model of , human hearing , and using that f  , to generate speech recognition features . And he did work back in Germany with , a toy recognition system using , isolated digit recognition as the task .  , and he tried that on I think on some Aurora data and got results that he thought seemed respectable . And he w he 's coming here to u u use it on a  , a real speech recognition system . And , maybe I should say a little more about these features , although I don't understand them that well . And I 'm - I 'm not sure what we have @ @ in there that isn't already modeled in something like , PLP . And then the second stage is , the most different thing , I think , from what we usually do . It 's ,  it computes features which are , based on sort of like based on diffe different w  , wavelet basis functions used to analyze the input .  So th he uses analysis functions called Gabor functions , which have a certain extent , in time and in frequency . And the idea is these are used to sample , the signal in a represented as a time - frequency representation . And , that , is is interesting , cuz , @ @ for for one thing , you could use it , in a a multi - scale way . You could have these instead of having everything like we use a twenty - five millisecond or so analysis window , typically , and that 's our time scale for features , but you could using this , basis function idea , you could have some basis functions which have a lot longer time scale and , some which have a lot shorter , and so it would be like a set of multi - scale features . So he 's interested in ,  Th - this is because it 's ,  there are these different parameters for the shape of these basis functions ,  there are a lot of different possible basis functions . And so he he actually does an optimization procedure to choose an an optimal set of basis functions out of all the possible ones . H What does he do to choose those ? grad c: The method he uses is kind of funny is , he starts with he has a set of M of them .  , he and then he uses that to classify  , he t he tries , using just M minus one of them . grad c: Whichever sub - vector , works the the best , I guess , he says the the fe feature that didn't use was the most useless feature , professor b: Y yeah . grad c: so we 'll throw it out and we 're gonna randomly select another feature from the set of possible basis functions . phd a: So it 's a professor b: So i so it 's actuall phd a: it 's a little bit like a genetic algorithm or something in a way . grad e: It 's like a greedy professor b: But it 's but it 's  , it 's there 's a lot number of things I like about it , let me just say .  , i i in truth , both pieces of this are have their analogies in stuff we already do . But it 's a different take at how to approach it and potentially one that 's m maybe a bit more systematic than what we 've done , and a b a bit more inspiration from from auditory things . The primary features , are in fact Yeah , essentially , it 's it 's , you know , PLP or or mel cepstrum , or something like that . We always have some you know , the the the kind of filter bank with a kind of quasi - log scaling .  , if you put in if you also include the RASTA in it i RASTA the filtering being done in the log domain has an AGC - like , characteristic , which , you know , people typi typically put in these kind of , auditory front - ends .  , I would agree that the second one is is somewhat more different but , it 's mainly different in that the things that we have been doing like that have been  , had a different kind of motivation and have ended up with different kinds of constraints . So , for instance , if you look at the LDA RASTA stuff , you know , basically what they do is they they look at the different eigenvectors out of the LDA and they form filters out of it . But , they 're not sort of systematically multi - scale , like " let 's start here and go to there , and go to there , and go to there " , and so forth . It 's more like , you run it on this , you do discriminant analysis , and you find out what 's helpful .  grad c: I it 's multi - scale because you use several of these in parallel , professor b: Yeah .  , but it 's also ,   Hyn - when Hynek 's had people do this kind of LDA analysis , they 've done it on frequency direction and they 've done it on the time direction . I think he may have had people sometimes doing it on both simultaneously some two - D and that would be the closest to these Gabor function kind of things . And , the other thing that 's interesting the the ,  the feature selection thing , it 's a simple method , but I kinda like it .  , eh , I remember people referring to it as old when I was playing with it twenty years ago , so I know it 's pretty old , called Stepwise Linear Discriminant Analysis in which you which I think it 's used in social sciences a lot . And what what Michael 's describing seems to me much , much better , because the problem with the stepwise discriminant analysis is that you don't know that you know , if you 've picked the right set of features . So , here at least you 're starting off with all of them , and you 're throwing out useless features .  , so the only thing is , of course , there 's this this artificial question of of , exactly how you how you a how you assess it and if if your order had been different in throwing them out .  , it still isn't necessarily really optimal , but it seems like a pretty good heuristic . professor b: And and and , the thing that I wanted to to add to it also was to have us use this in a multi - stream way . professor b: so so that , when you come up with these different things , and these different functions , you don't necessarily just put them all into one huge vector , but perhaps you have some of them in one stream and some of them in another stream , and so forth . And ,  And we 've also talked a little bit about , Shihab Shamma 's stuff , in which you the way you look at it is that there 's these different mappings and some of them emphasize , upward moving , energy and fre and frequency . But , I think we 're sorta gonna start off with what he , came here with and branch out branch out from there . grad e: As as we were talking about this I was thinking , whether there 's a relationship between  , between Michael 's approach to , some some sort of optimal brain damage or optimal brain surgeon on the neural nets . grad e: So , like , if we have ,  we have our we have our RASTA features and and presumably the neural nets are are learning some sort of a nonlinear mapping , from the the the features to to this this probability posterior space . grad e: Right ? And ,  and each of the hidden units is learning some sort of some sort of some sort of pattern . And then when you 're looking at the the , the best features , you know , you can take out you can do the do this , brain surgery by taking out , hidden units that don't really help at all . grad e: And this is k sorta like professor b: Right ? grad e: Yeah . professor b: y actually , you make me think a a very important point here is that , if we a again try to look at how is this different from what we 're already doing , there 's a a ,  a nasty argument that could be made th that it 's it 's not different at at all , because ,  if you ignore the the selection part because we are going into a a very powerful , nonlinearity that , in fact is combining over time and frequency , and is coming up with its own you know , better than Gabor functions its , you know , neural net functions , grad e:   grad c:  professor b: so you could argue that in fact it But I I don't actually believe that argument because I know that , you can ,  computing features is useful , even though in principle you haven't added anything in fact , you subtracted something , from the original waveform You know , if you 've you 've processed it in some way you 've typically lost something some information . And so , you 've lost information and yet it does better with with features than it does with the waveform . So that 's why it really seems like the constraint in in all this stuff it 's the constraints that are actually what matters . Because if it wasn't the constraints that mattered , then we would 've completely solved this problem long ago , because long ago we already knew how to put waveforms into powerful statistical mechanisms . Well , if we had infinite processing power and data , I guess , using the waveform could grad e: Right . But but , i it 's With finite of those things  , we we have done experiments where we literally have put waveforms in and and and , phd d:   professor b: we kept the number of parameters the same and so forth , and it used a lot of training data . And it and it it ,  not infinite but a lot , and then compared to the number parameters and it it ,  it just doesn't do nearly as well . professor b: it 's not just having the maximum information , you want to suppress , the aspects of the input signal that are not helpful for for the discrimination you 're trying to make . So maybe just briefly ,  grad e: Well , that sort of segues into what what I 'm doing . grad e: so , the big picture is k  , come up with a set of , intermediate categories , then build intermediate category classifiers , then do recognition , and , improve speech recognition in that way .  , so right now I 'm in in the phase where I 'm looking at at , deciding on a initial set of intermediate categories . And I 'm looking for data data - driven methods that can help me find , a set of intermediate categories of speech that , will help me to discriminate later down the line . And one of the ideas , that was to take a take a neural net train train an ordinary neural net to  , to learn the posterior probabilities of phones . And so , at the end of the day you have this neural net and it has hidden hidden units .  , and I 'm gonna to try to to look at those patterns to to see , from those patterns  , presumably those are important patterns for discriminating between phone classes . And maybe maybe some , intermediate categories can come from just looking at the patterns of  , that the neural net learns . professor b: Be - before you get on the next part l let me just point out that s there 's there 's a a pretty nice relationship between what you 're talking about doing and what you 're talking about doing there . professor b: So , it seems to me that , you know , if you take away the the the difference of this primary features , and , say , you use as we had talked about maybe doing you use P - RASTA - PLP or something for the the primary features , then this feature discovery , thing is just what he 's talking about doing , too , except that he 's talking about doing them in order to discover intermediate categories that correspond to these  , what these sub - features are are are are showing you . And , the other difference is that , he 's doing this in a in a multi - band setting , which means that he 's constraining himself to look across time in some f relatively limited , spectral extent . Right ? And whereas in in this case you 're saying " let 's just do it unconstrained " . So they 're they 're really pretty related and maybe they 'll be at some point where we 'll see the the connections a little better and connect them . Yeah , so so that 's the that 's the first part  , one one of the ideas to get at some some patterns of intermediate categories .  , the other one was , to , come up with a a a model  , a graphical model , that treats the intermediate categories as hidden hidden variables , latent variables , that we don't know anything about , but that through , s statistical training and the EM algorithm , at the end of the day , we have ,  we have learned something about these these latent ,  latent variables which happen to correspond to intermediate categories . Yeah , and so those are the the two directions that I 'm I 'm looking into right now . Should we do our digits and get ou get our treats ? grad e: Oh , tea time ? professor b: Yeah . It 's kind of like , you know , the little rats with the little thing dropping down to them . phd a: That 's ri professor b: We do the digits and then we get our treats