phd a: And what was that , Morgan ? What project ? professor b: the , TORRENT chip . professor b: we went through it Jim and I went through old emails at one point and and for two years there was this thing saying , yeah , we 're we 're two months away from being done . phd a: So , should we just do the same kind of deal where we go around and do , status report kind of things ? OK . phd a: Any objection ? Do y OK , M professor b: All in favor phd a: Do you want to start , Morgan ? Do you have anything , or ? professor b: I don't do anything . I No , I I 'm involved in discussions with with people about what they 're doing , but I think they 're since they 're here , they can talk about it themselves . Why don't you go ahead , Barry ? grad f: you 're gonna talk about Aurora stuff , per se ? phd a: OK . Well , this past week I 've just been , getting down and dirty into writing my my proposal . I just finished a section on ,  on talking about these intermediate categories that I want to classify , as a as a middle step . And , I hope to hope to get this ,  a full rough draft done by , Monday so I can give it to Morgan . phd a: So , is the idea you 're going to do this paper and then you pass it out to everybody ahead of time and ? grad f: Right , right . So , y you write up a proposal , and give it to people ahead of time , and you have a short presentation . phd a: Have you d ? I was just gonna ask , do you want to say any a little bit about it , grad f: Y s phd a: or ? Mmm .  , a little bit about ? phd a: Wh - what you 're what you 're gonna You said you were talking about the , particular features that you were looking at , grad f: Oh , the the phd a: or grad f: Right . Well , I was , I think one of the perplexing problems is , for a while I was thinking that I had to come up with a complete set of intermediate features in intermediate categories to to classify right away .  , build a system that , classifies these ,  these feat  , these intermediate categories using , multi - band techniques . Look at then I would look at the errors produced in the phoneme recognition and say , OK , well , I could probably reduce the errors if I included this extra feature or this extra intermediate category . phd a: So you 're gonna use TIMIT ? grad f: for that for that part of the the process , yeah , I would use TIMIT . grad f: that 's that 's ,  that 's just the ph the phone recognition task . So , I would mov I would then shift the focus to , something like Schw - Switchboard , where I 'd I would be able to ,  to model , intermediate categories that span across phonemes , phd a:   grad f: not just within the phonemes , themselves , and then do the same process there , on on a large vocabulary task like Switchboard .  , and for that for that part I would I 'd use the SRI recognizer since it 's already set up for for Switchboard . phd a: It 's about a month from now ? grad f: It 's a it 's a month and and a week . phd a: So , you want to go next , Dave ? And we 'll do grad e: Oh . And , we we got an improvement , in word error rate , training on the TI - digits data set and testing on Meeting Recorder digits of , six percent to four point five percent , on the n on the far - mike data using PZM F , but , the near - mike performance worsened , from one point two percent to two point four percent . And , wh why would that be , considering that we actually got an improvement in near - mike performance using HTK ? And so , with some input from , Andreas , I have a theory in two parts .  , first of all HTK sorry , SR - the SRI system is doing channel adaptation , and so HTK wasn't .  , so this ,  This mean subtraction approach will do a kind of channel normalization and so that might have given the HTK use of it a boost that wouldn't have been applied in the SRI case . So those finer - grained acoustic models could be more sensitive to the artifacts in the re - synthesized audio . And me and Barry were listening to the re - synthesized audio and sometimes it seems like you get of a bit of an echo of speech in the background . And so that seems like it could be difficult for training , cuz you could have different phones lined up with a different foreground phone , depending on the timing of the echo . So , I 'm gonna try training on a larger data set , and then , eh , the system will have seen more examples o of these artifacts and hopefully will be more robust to them . professor b: I had another thought just now , which is , remember we were talking before about we were talking in our meeting about , this stuff that some of the other stuff that Avendano did , where they were , getting rid of low - energy sections ?  , if you if you did a high - pass filtering , as Hirsch did in late eighties to reduce some of the effects of reverberation , Avendano and Hermansky were arguing that , perhaps one of the reasons for that working was ma may not have even been the filtering so much but the fact that when you filter a an all - positive power spectrum you get some negative values , and you gotta figure out what to do with them if you 're gonna continue treating this as a power spectrum . So if you imagine a a waveform that 's all positive , which is the time trajectory of energy , and , shifting it downwards , and then getting rid of the negative parts , that 's essentially throwing away the low - energy things . And it 's the low - energy parts of the speech where the reverberation is most audible . You know , you have the reverberation from higher - energy things showing up in So in this case you have some artificially imposed reverberation - like thing .  , you 're getting rid of some of the other effects of reverberation , but because you have these non - causal windows , you 're getting these funny things coming in , at n And , what if you did ?  , there 's nothing to say that the the processing for this re - synthesis has to be restricted to trying to get it back to the original , according to some equation . professor b: And one of the things you could do is , you could do some sort of VAD - like thing grad e:   professor b: and you actually could take very low - energy sections and set them to some some , very low or or near zero value .  , I 'm just saying if in fact it turns out that that these echoes that you 're hearing are ,  grad e:  - huh . professor b: or pre - echoes , whichever they are are are , part of what 's causing the problem , you actually could get rid of them . professor b: so that if you made a mistake you were more likely to keep in an echo than to throw out speech . phd g: what is the reverberation time like there ? grad e: In thi in this room ?  phd g: On , the the one what the s in the speech that you are you are using like ? grad e: Y Yeah . professor b: so it 's these are just microphone this micro close microphone and a distant microphone , he 's doing these different tests on . I think it 's I would guess , point seven , point eight seconds f  , R T grad f:  ! professor b: something like that ? But it 's you know , it 's this room . But the other thing is , he 's putting in w I was using the word " reverberation " in two ways . He 's also putting in , a he 's taking out some reverberation , but he 's putting in something , because he has averages over multiple windows stretching out to twelve seconds , which are then being subtracted from the speech . And since , you know , what you subtract , sometimes you 'll be you 'll be subtracting from some larger number and sometimes you won't . professor b: So you can end up with some components in it that are affected by things that are seconds away .  , and if it 's a low energy compo portion , you might actually hear some funny things . grad e: O o one thing , I noticed is that , the mean subtraction seems to make the PZM signals louder after they 've been re - synthesized . So I was wondering , is it possible that one reason it helped with the Aurora baseline system is just as a kind of gain control ? Cuz some of the PZM signals sound pretty quiet if you don't amplify them . I don't see why why your signal is louder after processing , because yo grad e: Yeah . professor b: I don't think just multiplying the signal by two would have any effect .  , I think if you really have louder signals , what you mean is that you have better signal - to - noise ratio . phd c: Well , well professor b: So if what you 're doing is improving the signal - to - noise ratio , then it would be better . professor b: But just it being bigger if with the same signal - to - noise ratio grad e: It w i i it wouldn't affect things . phd c: Well , the system is use the absolute energy , so it 's a little bit dependent on on the signal level . professor b: So if the if the if you change in both training and test , the absolute level by a factor of two , it will n have no effect . phd a: Did you add this data to the training set , for the Aurora ? Or you just tested on this ? grad e:   . Did I w what ? phd a: Well , Morgan was just saying that , as long as you do it in both training and testing , it shouldn't have any effect . But I didn't I 'm not sure if it made the clean ti TI - digits any louder . If it 's if it 's like , if it 's trying to find a a reverberation filter , it could be that this reverberation filter is making things quieter .  , there 's there 's nothing inherent about removing if you 're really removing , grad e: Nuh - huh . phd a: I wonder if there could be something like ,  for s for the PZM data , phd c: Eh phd a: you know , if occasionally , somebody hits the table or something , you could get a spike . I 'm just wondering if there 's something about the ,  you know , doing the mean normalization where , it it could cause you to have better signal - to - noise ratio . It it i maybe i If ,  Subtracting the the mean log spectrum is is is like dividing by the spectrum . So , depending what you divide by , if your if s your estimate is off and sometimes you 're you 're you 're getting a small number , you could make it bigger . professor b: So , it 's it 's just a a question of there 's It it could be that there 's some normalization that 's missing , or something to make it grad e:   professor b: y you 'd think it shouldn't be larger , but maybe in practice it is . So , you trained it on TI - digits ? But except this , it 's exactly the same system as the one that was tested before and that was trained on Macrophone . Right ? So on TI - digits it gives you one point two percent error rate and on Macrophone it 's still O point eight . grad e: If you 're talking about the Macrophone results that Andreas had about , a week and a half ago , I think it 's the same system . So you use VTL -  , vocal tract length normalization and , like MLLR transformations also , grad e:   phd c: and professor b: I 'm sorry , was his point eight percent , er , a a result on testing on Macrophone or or training ? phd c: all that stuff . grad e: That 's phd c: It was training on Macrophone and testing yeah , on on meeting digits . I I 've just been text testing the new Aurora front - end with well , Aurora system actually so front - end and HTK , acoustic models on the meeting digits and it 's a little bit better than the previous system . professor b: So , what w ? phd c: And phd g: With the with the HTK back - end ? What we have for Aurora ? phd c: Yeah . phd g: I know in the meeting , like phd c: On the meeting we have two point seven . phd c: we have the new LDA filters , and I think , maybe I didn't look , but one thing that makes a difference is this DC offset compensation .  , eh Do y did you have a look at at the meet  , meeting digits , if they have a DC component , or ? grad e: I I didn't .  , any all of the mikes have the DC removal some capacitor sitting right in that bias it . professor b: typi you know , unless Actually , there are instrumentation mikes that that do pass go down to DC . And you can get , I think it was I think it was in the Wall Street Journal data that that I can't remember , one of the DARPA things . professor b: we didn't we didn't know about for a while , while we were messing with it . And , the interesting thing that I tried was , Adam and Morgan had this idea , since my original attempts to , take the mean of the phase spectra over time and normalize using that , by subtracting that off , didn't work .  , so , well , that we thought that might be due to , problems with , the arithmetic of phases . They they add in this modulo two pi way and , there 's reason to believe that that approach of taking the mean of the phase spectrum wasn't really mathematically correct . So , what I did instead is I took the mean of the FFT spectrum without taking the log or anything , and then I took the phase of that , and I subtracted that phase off to normalize . professor b: You see , all he has to do is go back and reverse what he did before , and he 's really got something . phd a: Well , could you take what was left over and then subtract that ? professor b: Ex - exactly . phd g: Oh , it 's professor b: Just listen very carefully to what I say and do the opposite . I 'm more interested in trying to figure out what 's still the difference between the SRI system and the Aurora system . So , I think I will maybe train , like , gender - dependent models , because this is also one big difference between the two systems .  , the other differences were the fact that maybe the acoustic models of the SRI are more SRI system are more complex . But , Chuck , you did some experiments with this and phd a: It didn't seem to help in the HTK system . professor b: Well , it sounds like they also have he he 's saying they have all these , different kinds of adaptation . phd a: Like they do ,  I 'm not sure how they would do it when they 're working with the digits , phd c: The vocal tr phd a: but , like , in the Switchboard data , there 's ,  conversation - side normalization for the non - C - zero components , phd c: Yeah . phd c: So , it might be it might be better with it might be worse if the channel is constant , phd a: Yeah . phd g: And the acoustic models are like - k triphone models or or is it the whole word ? phd c: SRI it 's it 's tr grad f: SRI . So there 's there 's these kind of , pooled models and and they can go out to all sorts of dependencies . professor b: They have tied states and I think I I I don't real I 'm talk I 'm just guessing here . And maybe see with Andreas if Well , I I don't know how much it helps , what 's the model . phd a: So so the n stuff on the numbers you got , the two point seven , is that using the same training data that the SRI system used and got one point two ? phd c: That 's right . grad e: You know , the the Aurora baseline is set up with these ,  this version of the clean training set that 's been filtered with this G - seven - one - two filter , and , to train the SRI system on digits S - Andreas used the original TI - digits , under U doctor - speech data TI - digits , which don't have this filter . professor b: So is that ?  , are are these results comparable ? So you you were getting with the , Aurora baseline something like two point four percent on clean TI - digits , when , training the SRI system with clean TR digits TI - digits . And , so , is your two point seven comparable , where you 're , using , the submitted system ? phd c: Yeah . grad e: W w it was one one point two phd c: Ye grad e: with the SRI system , professor b: I 'm sorry . professor b: OK , so the comparable number then ,  for what you were talking about then , since it was HTK , would be the  , two point f phd c: It was four point something . Right ? The HTK system with , b grad e: D d professor b: Oh , right , right , right , right . phd c: MFCC features grad e: Do you mean the b ? The baseline Aurora - two system , trained on TI - digits , tested on Meeting Recorder near , I think we saw in it today , and it was about six point six percent . And another thing I I maybe would like to do is to just test the SRI system that 's trained on Macrophone test it on , the noisy TI - digits , professor b: Yeah . But I wonder if it 's just because maybe Macrophone is acoustically closer to the meeting digits than than TI - digit is , which is TI - digits are very clean recorded digits professor b:   phd c: and phd a: You know , it would also be interesting to see ,  to do the regular Aurora test , phd c: f s phd a: but use the SRI system instead of HTK . phd c: So we don't professor b: You 'd have to train the SRI system with with all the different languages . phd a: So , like , comple professor b: It 'd be a lot of work . phd a: I guess the work would be into getting the the files in the right formats , or something . phd a: Because when you train up the Aurora system , you 're ,  you 're also training on all the data . professor b: That 's true , but I think that also when we 've had these meetings week after week , oftentimes people have not done the full arrange of things phd a:   professor b: because on on whatever it is they 're trying , because it 's a lot of work , even just with the HTK . professor b: So , it 's it 's a good idea , but it seems like it makes sense to do some pruning phd a:   professor b: first with a a test or two that makes sense for you , phd a: Yeah . But , just testing on TI - digits would already give us some information about what 's going on .  , the next thing is this this VAD problem that ,  So , I 'm just talking about the the curves that I I sent I sent you so , whi that shows that when the SNR decrease , the current VAD approach doesn't drop much frames for some particular noises , which might be then noises that are closer to speech , acoustically . I They were supp Supposedly , in the next evaluation , they 're going to be supplying us with boundaries . First of all , the boundaries might be ,  like we would have t two hundred milliseconds or before and after speech . professor b: Do we ?  , is there some reason that we think that 's the case ? phd c: And No . professor b: But maybe we 'll get some insight on that when when , the gang gets back from Crete . professor b: And then the thing is if if they really are going to have some means of giving us fairly tight , boundaries , then that won't be so much the issue . phd g: Because w we were wondering whether that VAD is going to be , like , a realistic one or is it going to be some manual segmentation . And then , like , if if that VAD is going to be a realistic one , then we can actually use their markers to shift the point around , the way we want professor b:   phd g: to find a  , rather than keeping the twenty frames , we can actually move the marker to a point which we find more suitable for us . phd g: But if that is going to be something like a manual , segmenter , then we can't use that information anymore , phd c:   phd g: because that 's not going to be the one that is used in the final evaluation . There 's an  , I think it 's still for even for the evaluation , it might still be interesting to work on this because the boundaries apparently that they would provide is just , starting of speech and end of speech  , at the utterance level . phd c: So phd g: with some pauses in the center , provided they meet that whatever the hang - over time which they are talking . professor b: So if you could get at some of that ,  phd c: So professor b: although that 'd be hard . It might be useful for , like , noise estimation , and a lot of other things that we want to work on . So I did I just started to test putting together two VAD which was was not much work actually .  , I im re - implemented a VAD that 's very close to the , energy - based VAD that , the other Aurora guys use . phd c: and , detect detecting the first group of four frames that have a energy that 's above this threshold , and , from this point , tagging the frames there as speech . And it really removes it , still o on the noises where our MLP VAD doesn't work a lot . phd c: professor b: Cuz I would have thought that having some kind of spectral information , phd c: and professor b: you know , in the old days people would use energy and zero crossings , for instance  , would give you some better performance . Right ? Cuz you might have low - energy fricatives or or ,  stop consonants , or something like that . So , your point is will be to u use whatever professor b: Oh , that if you d if you use purely energy and don't look at anything spectral , then you don't have a good way of distinguishing between low - energy speech components and nonspeech . professor b: just as a gross generalization , most nonsp many nonspeech noises have a low - pass kind of characteristic , some sort of slope . And and most , low - energy speech components that are unvoiced have a a high - pass kind of characteristic phd c:   professor b: you know , at the beginning of a of a of an S sound for instance , just starting in , it might be pretty low - energy , phd c:   Whereas , a a lot of rumble , and background noises , and so forth will be predominantly low - frequency .  , you know , by itself it 's not enough to tell you , but it plus energy is sort of phd c: Yeah . professor b: if you look up in Rabiner and Schafer from like twenty - five years ago or something , that 's sort of what they were using then . It it might be that what I did is so , removes like low ,  low - energy , speech frames . Because the way I do it is I just I just combine the two decisions so , the one from the MLP and the one from the energy - based with the with the and operator . So if the energy - based dropped dropped low - energy speech , mmm , they they are they are lost . phd c: But s still , the way it 's done right now it it helps on on the noises where it seems to help on the noises where our VAD was not very good . But but , I guess what you 're saying is that the the MLP - based one has the spectral information . professor b: Well , you can imagine phd c: The way I use a an a " AND " operator is So , it I ,  professor b: Is ? phd c: The frames that are dropped by the energy - based system are are , dropped , even if the , MLP decides to keep them . phd a: No professor b: but  , I guess in principle what you 'd want to do is have a  , a probability estimated by each one and and put them together . phd a: Something that that I 've used in the past is ,  when just looking at the energy , is to look at the derivative . phd a: But , I 'm I 'm trying to remember if that requires that you keep some amount of speech in a buffer . phd c: Well , actually if I don't maybe don't want to work too much of on it right now . I just wanted to to see if it 's what I observed was the re was caused by this this VAD problem .  , which I 've just started yesterday to launch a bunch of , twenty - five experiments , with different , values for the parameters that are used . So , it 's the Makhoul - type spectral subtraction which use an over - estimation factor . So , we substr I subtract more , noise than the noise spectra that is estimated on the noise portion of the s  , the utterances . And after subtraction , I also add a constant noise , and I also try different , noise , values and we 'll see what happen . But st still when we look at the ,  Well , it depends on the parameters that you use , but for moderate over - estimation factors and moderate noise level that you add , you st have a lot of musical noise . On the other hand , when you subtract more and when you add more noise , you get rid of this musical noise but maybe you distort a lot of speech . So the next thing , maybe I what I will try to to do is just to try to smooth mmm , the ,  to smooth the d the result of the subtraction , to get rid of the musical noise , using some kind of filter , or phd g: Can smooth the SNR estimate , also . So , to get something that 's would be closer to what you tried to do with Wiener filtering . phd c: It phd g: And it 's phd c: Maybe you can phd g: go ahead . And there are there were some bugs in the program , so I was p initially trying to clear them up . Because one of the bug was I was assuming that always the VAD  , the initial frames were silence . So the it wasn't estimating the noise initially , and then it never estimated , because I assumed that it was always silence . phd c: So , in some cases s there are also phd g: SpeechDat - Car Italian . And , so once it was cleared , I ran a few experiments with different ways of smoothing the estimated clean speech and how t estimated the noise and , eh , smoothing the SNR also . And so the the trend seems to be like , smoothing the current estimate of the clean speech for deriving the SNR , which is like deriving the Wiener filter , seems to be helping . So we 'll have , like , a few results where the estimating the the More smoothing is helping . And , so I 'm I 'm trying a few more experiments with different time constants for smoothing the noise spectrum , and smoothing the clean speech , and smoothing SNR . So , one is fixed in the line , like Smoothing the clean speech is is helping , so I 'm not going to change it that much . But , the way I 'm estimating the noise and the way I 'm estimating the SNR , I 'm just trying trying a little bit . So , that h And the other thing is , like , putting a floor on the , SNR , because that if some In some cases the clean speech is , like when it 's estimated , it goes to very low values , so the SNR is , like , very low . So , I 'm thinking of , like , putting a floor also for the SNR so that it doesn't vary a lot in the low - energy regions . The results are , like So far I 've been testing only with the baseline , which is which doesn't have any LDA filtering and on - line normalization . So it 's just VAD , plus the Wiener filter , plus the baseline system , which is , just the spectral  , the mel sp mel , frequency coefficients . And the other thing that I tried was but I just took of those , Carlos filters , which Hynek had , to see whether it really h helps or not . And it 's it seems to be like it 's not hurting a lot by just blindly picking up one filter which is nothing but a four hertz a band - pass m m filter on the cubic root of the power spectrum . So there must be something that I can that can be done with that type of noise compensation also , which I guess I would ask Carlos about that .  , how how he derived those filters and and where d if he has any filters which are derived on OGI stories , added with some type of noise which what we are using currently , or something like that . So maybe I 'll professor b: This is cubic root of power spectra ? phd g: Yeah . professor b: So , if you have this band - pass filter , you probably get n you get negative values . phd g: So it has , like the spectrogram has , like  , it actually , enhances the onset and offset of  , the the begin and the end of the speech . So it 's there seems to be , like , deep valleys in the begin and the end of , like , high - energy regions , professor b:   phd g: So , those are the regions where there are , like when I look at the spectrogram , there are those deep valleys on the begin and the end of the speech . There are a few very not a lot of because the filter doesn't have a really a deep negative portion , so that it 's not really creating a lot of negative values in the cubic root . So , I 'll I 'll s may continue with that for some w I 'll I 'll Maybe I 'll ask Carlos a little more about how to play with those filters , and but while making this Wiener filter better . professor b: last week you were also talking about building up the subspace stuff ? phd g: Yeah . I I I would actually m m didn't get enough time to work on the subspace last week . phd a: How about you , Carmen ? phd d: Well , I am still working with , eh , VTS . And , one of the things that last week , eh , say here is that maybe the problem was with the diff because the signal have different level of energy . professor b:  ? phd d: And , maybe , talking with Stephane and with Sunil , we decide that maybe it was interesting to to apply on - line normalization before applying VTS . But then we decided that that 's it doesn't work absolutely , because we modified also the noise . I don't hav I don't this is I didn't do the experiment yet to apply VTS in cepstral domain . professor b: The other thing is So so , in i i and Not and C - zero would be a different So you could do a different normalization for C - zero than for other things anyway .  , the other thing I was gonna suggest is that you could have two kinds of normalization with with , different time constants . So , you could do some normalization s  , before the VTS , and then do some other normalization after . phd d: Well , we s decide to m to to obtain the new expression if we work in the cepstral domain . It 's k it 's k It 's quite a lot It 's a lot of work . phd d: And I want to know if if we have some feeling that the result I I would like to know if I don't have any feeling if this will work better than apply VTS aft in cepstral domain will work better than apply in m mel in filter bank domain . Well , you 're I think you 're the first one here to work with VTS , so , maybe we could call someone else up who has , ask them their opinion . phd c: Actually , the VTS that you tested before was in the log domain and so the codebook is e e kind of dependent on the level of the speech signal . phd d: Yeah ? phd c: And So I expect it If if you have something that 's independent of this , I expect it to it to , be a better model of speech . And then you have one number which is very dependent on the level cuz it is the level , phd d:   phd d: Ye phd c: Because it 's like first doing general normalization phd d: Yea phd c: and then noise removal , which is phd d: Yeah . We I was thinking to to to estimate the noise with the first frames and then apply the VAD , professor b:   phd d: We we see Well , I am thinking about that and working about that , professor b: Yeah .  , one of the things we 've talked about maybe it might be star time to start thinking about pretty soon , is as we look at the pros and cons of these different methods , how do they fit in with one another ? Because we 've talked about potentially doing some combination of a couple of them . Maybe maybe pretty soon we 'll have some sense of what their characteristics are , phd d:  -


professor c: And  Hans -  , Hans - Guenter will be here , I think by next next Tuesday or so . professor c: So he 's he 's going to be here for about three weeks , phd b: Oh ! That 's nice . professor c: So he 's gonna look in on everything we 're doing phd d:   grad e: Th - that 's his spectral subtraction group ? professor c: Yeah , grad e: Is that right ? professor c: yeah . He 's , very , very , easygoing , easy to talk to , and , very interested in everything . professor c: he 's he 's he 's he 's phd a: Wh - Back when I was a grad student he was here for a ,  a year or n six months .  , I went around and talked to everybody , and it seemed like they they had some new results but rather than them coming up and telling me I figured we should just wait a week and they can tell both you know , all of us . So , why don't we why don't we start with you , Dave , and then , we can go on . grad e: So , since we 're looking at putting this ,  mean log m magnitude spectral subtraction , into the SmartKom system , I I did a test seeing if , it would work using past only and plus the present to calculate the mean . So , I did a test , where I used twelve seconds from the past and the present frame to , calculate the mean . And phd a: Twelve seconds Twelve twelve seconds back from the current frame , is that what you mean ? grad e:  Twelve seconds , counting back from the end of the current frame , phd a: OK , OK . So it was , twen I think it was twenty - one frames and that worked out to about twelve seconds . grad e: And compared to , do using a twelve second centered window , I think there was a drop in performance but it was just a slight drop . grad e: Is is that right ? professor c: yeah , it was pretty it was pretty tiny . And , that that  , that 's encouraging for for the idea of using it in an interactive system like And , another issue I 'm I 'm thinking about is in the SmartKom system . So say twe twelve seconds in the earlier test seemed like a good length of time , but what happens if you have less than twelve seconds ? And ,  So I w bef before ,  Back in May , I did some experiments using , say , two seconds , or four seconds , or six seconds . In those I trained the models using mean subtraction with the means calculated over two seconds , or four seconds , or six seconds . And , here , I was curious , what if I trained the models using twelve seconds but I f I gave it a situation where the test set I was subtracted using two seconds , or four seconds , or six seconds . And , I th I think it was , four se I think I think it was , something like four seconds and , six seconds , and eight seconds . And it seems like it it it hurts compared to if you actually train the models using th that same length of time but it it doesn't hurt that much .  , u usually less than point five percent , although I think I did see one where it was a point eight percent or so rise in word error rate . But this is , w where , even if I train on the , model , and mean subtracted it with the same length of time as in the test , it the word error rate is around , ten percent or nine percent . professor c: But it but looking at it the other way , isn't it what you 're saying that it didn't help you to have the longer time for training , if you were going to have a short time for grad e: That that 's true .  , professor c: why would you do it , if you knew that you were going to have short windows in testing . grad e: Wa phd a: Yeah , it seems like for your  , in normal situations you would never get twelve seconds of speech , right ? I 'm not e u phd b: You need twelve seconds in the past to estimate , right ? Or l or you 're looking at six sec seconds in future and six in professor c: Yeah . phd a: Is this twelve seconds of  , regardless of speech or silence ? Or twelve seconds of speech ? grad e: Of of speech . professor c: The other thing , which maybe relates a little bit to something else we 've talked about in terms of windowing and so on is , that , I wonder if you trained with twelve seconds , and then when you were two seconds in you used two seconds , and when you were four seconds in , you used four seconds , and when you were six and you basically build up to the twelve seconds . grad e: But s so I g So I guess the que the question I was trying to get at with those experiments is , " does it matter what models you use ? Does it matter how much time y you use to calculate the mean when you were , tra doing the training data ? " professor c: Right . But  the other thing is that that 's  , the other way of looking at this , going back to , mean cepstral subtraction versus RASTA kind of things , is that you could look at mean cepstral subtraction , especially the way you 're doing it , as being a kind of filter . You know , basically you 're you 're you 're doing a high - pass filter or a band - pass filter of some sort and and just design a filter . And then , you know , a filter will have a certain behavior and you loo can look at the start up behavior when you start up with nothing . professor c: And and , you know , it will , if you have an IIR filter for instance , it will , not behave in the steady - state way that you would like it to behave until you get a long enough period , but , by just constraining yourself to have your filter be only a subtraction of the mean , you 're kind of , you know , tying your hands behind your back because there 's filters have all sorts of be temporal and spectral behaviors . professor c: And the only thing , you know , consistent that we know about is that you want to get rid of the very low frequency component . phd b: But do you really want to calculate the mean ? And you neglect all the silence regions or you just use everything that 's twelve seconds , and grad e: you do you mean in my tests so far ? phd b: Ye - yeah . grad e: th  , yeah , I guess it 's something I need to play with more to decide how to set that up for the SmartKom system . Like , may maybe if I trained on six seconds it would work better when I only had two seconds or four seconds , and professor c: Yeah . professor c: Yeah , and again , if you take this filtering perspective and if you essentially have it build up over time .  , if you computed means over two and then over four , and over six , essentially what you 're getting at is a kind of , ramp up of a filter anyway . But , if you do that , then , in practice somebody using the SmartKom system , one would think if they 're using it for a while , it means that their first utterance , instead of , you know , getting , a forty percent error rate reduction , they 'll get a  , over what , you 'd get without this , policy , you get thirty percent . And then the second utterance that you give , they get the full you know , full benefit of it if it 's this ongoing thing . phd a: Oh , so you you cache the utterances ? That 's how you get your ,  professor c: Well , I 'm saying in practice , yeah , grad e: M phd a: Ah . professor c: that 's If somebody 's using a system to ask for directions or something , phd a: OK . And and to begin with if it doesn't get them quite right , ma m maybe they 'll come back and say , " excuse me ? " phd a:   And so , there 's a higher probability of it making an error , in the first utterance . phd a: What would be really cool is if you could have  , this probably users would never like this but if you had could have a system where , before they began to use it they had to introduce themselves , verbally . professor c: Oh , the other thing I guess which which , I don't know much about as much as I should about the rest of the system but but , couldn't you , if you if you sort of did a first pass I don't know what kind of , capability we have at the moment for for doing second passes on on , some kind of little small lattice , or a graph , or confusion network , or something . But if you did first pass with , the with either without the mean sub subtraction or with a a very short time one , and then , once you , actually had the whole utterance in , if you did , the , longer time version then , based on everything that you had , and then at that point only used it to distinguish between , you know , top N , possible utterances or something , you you might it might not take very much time .  , I know in the large vocabulary stu  , systems , people were evaluating on in the past , some people really pushed everything in to make it in one pass but other people didn't and had multiple passes . And , the argument , against multiple passes was u u has often been " but we want to this to be r you know have a nice interactive response " . And the counterargument to that which , say , BBN I think had , was " yeah , but our second responses are second , passes and third passes are really , really fast " . grad e: S so , the the idea of the second pass would be waiting till you have more recorded speech ? Or ? professor c: Yeah , so if it turned out to be a problem , that you didn't have enough speech because you need a longer longer window to do this processing , then , one tactic is you know , looking at the larger system and not just at the front - end stuff is to take in , the speech with some simpler mechanism or shorter time mechanism , grad e:   professor c: do the best you can , and come up with some al possible alternates of what might have been said . And , either in the form of an N - best list or in the form of a lattice , or or confusion network , or whatever . professor c: And then the decoding of that is much , much faster or can be much , much faster if it isn't a big bushy network . And you can decode that now with speech that you 've actually processed using this longer time , subtraction . professor c: So  , it 's it 's common that people do this sort of thing where they do more things that are more complex or require looking over more time , whatever , in some kind of second pass . professor c: and again , if the second pass is really , really fast  , another one I 've heard of is is in in connected digit stuff , going back and l and through backtrace and finding regions that are considered to be a d a digit , but , which have very low energy . professor c: So , there 's lots of things you can do in second passes , at all sorts of levels . phd a: So is that ,  that it ? grad e: I guess that 's it .  , so , the last two weeks was , like So I 've been working on that Wiener filtering . And , found that , s single like , I just do a s normal Wiener filtering , like the standard method of Wiener filtering . And that doesn't actually give me any improvement over like  , b it actually improves over the baseline but it 's not like it doesn't meet something like fifty percent or something . So , I 've been playing with the v phd a: Improves over the base line MFCC system ? Yeah . professor c: Is that using in combination with something else ? phd b: No , just just one stage Wiener filter professor c: With with a phd b: which is a standard Wiener filter . professor c: No , no , but  in combination with our on - line normalization or with the LDA ? phd b: Yeah , yeah , yeah , yeah . phd b: So , I di i di professor c: So , does it g does that mean it gets worse ? Or ? phd b: No . Like I have an LDA f LDA plus on - line normalization , and then I plug in the Wiener filter in that , professor c: Yeah ? phd b: so it improves over not having the Wiener filter . So it improves but it it doesn't take it like be beyond like thirty percent over the baseline . So professor c: But that 's what I 'm confused about , cuz I think I thought that our system was more like forty percent without the Wiener filtering . phd a: Is this with the v new VAD ? phd b: well , these are not No , it 's the old VAD . So my baseline was , nine This is like w the baseline is ninety - five point six eight , and eighty - nine , and professor c: So  , if you can do all these in word errors it 's a lot a lot easier actually . phd b: What was that ? Sorry ? professor c: If you do all these in word error rates it 's a lot easier , right ? phd b: Oh , OK , OK , OK . phd d: The baseline is something similar to a w  , the t the the baseline that you are talking about is the MFCC baseline , right ? phd b: The t yeah , there are two baselines . So the baseline One baseline is MFCC baseline that When I said thirty percent improvement it 's like MFCC baseline . professor c: So so so what 's it start on ? The MFCC baseline is is what ? Is at what level ? phd b: It 's the it 's just the mel frequency and that 's it . professor c: No , what 's what 's the number ? phd b: so I I don't have that number here . What 's ten point seven ? phd b: It 's a medium misma OK , sorry . phd b: So professor c: OK , four point three , ten point seven , phd b: And forty forty . professor c: And what were you just describing ? phd b: Oh , the one is this one is just the baseline plus the , Wiener filter plugged into it . professor c: But where 's the , on - line normalization and so on ? phd b: Oh , OK . So , with the with the on - line normalization , the performance was , ten OK , so it 's like four point three . So the h well matched has like literally not changed by adding on - line or LDA on it . professor c: OK , and what kind of number an and what are we talking about here ? phd b: It 's the It - it 's Italian . professor c: Is this TI - digits phd b: I 'm talking about Italian , professor c: or Italian ? phd b: yeah . professor c: And what did So , what was the , corresponding number , say , for , the Alcatel system for instance ? phd b: Mmm .  professor c: Do you know ? phd d: Yeah , so it looks to be ,  phd b: You have it ? phd d: Yep , it 's three point four , eight point , seven , and , thirteen point seven . phd b: So , this is the single stage Wiener filter , with The noise estimation was based on first ten frames . phd b: Actually I started with using the VAD to estimate the noise and then I found that it works it doesn't work for Finnish and Spanish because the VAD endpoints are not good to estimate the noise because it cuts into the speech sometimes , so I end up overestimating the noise and getting a worse result . phd b: So ,  so this was ,  And so this was giving  , this this was like not improving a lot on this baseline of not having the Wiener filter on it . And , so , I ran this stuff with one more stage of Wiener filtering on it but the second time , what I did was I estimated the new Wiener filter based on the cleaned up speech , and did , smoothing in the frequency to to reduce the variance professor c:   phd b: I have I 've I 've observed there are , like , a lot of bumps in the frequency when I do this Wiener filtering which is more like a musical noise or something . And so by adding another stage of Wiener filtering , the results on the SpeechDat - Car was like ,  So , I still don't have the word error rate . And the rest is like the LDA plu and the on - line normalization all remaining the same .  , so this was , like , compared to ,  Fifty - seven is what you got by using the French Telecom system , right ? phd d: No , I don't think so . phd b: Y i phd d: Is it on Italian ? phd b: No , this is over the whole SpeechDat - Car . phd b: Yeah , so the new the new Wiener filtering schema is like some fifty - six point four six which is like one percent still less than what you got using the French Telecom system . But again , you 're you 're more or less doing what they were doing , right ? phd b: It 's it 's different in a sense like I 'm actually cleaning up the cleaned up spectrum which they 're not doing . They 're d what they 're doing is , they have two stage stages of estimating the Wiener filter , but the final filter , what they do is they they take it to their time domain by doing an inverse Fourier transform . phd b: And they filter the original signal using that fil filter , professor c:  - huh . phd b: which is like final filter is acting on the input noisy speech rather than on the cleaned up . So this is more like I 'm doing Wiener filter twice , but the only thing is that the second time I 'm actually smoothing the filter and then cleaning up the cleaned up spectrum first level . phd b: And actually I tried it on s the original clean  , the original spectrum where , like , I the second time I estimate the filter but actually clean up the noisy speech rather the c s first output of the first stage and that doesn't seems to be a giving , that much improvement . And and what I t what I tried was , by using the same thing but  , so we actually found that the VAD is very , like , crucial . phd b: by instead of using the current VAD , if you just take up the VAD output from the channel zero , when instead of using channel zero and channel one , because that was the p that was the reason why I was not getting a lot of improvement for estimating the noise . So I just used the channel zero VAD to estimate the noise so that it gives me some reliable mar markers for this noise estimation . professor c: What 's a channel zero VAD ? phd b: professor c: I 'm I 'm confused about that . phd b: so , it 's like phd d: So it 's the close - talking microphone . phd b: Yeah , the close - talking without professor c: Oh , oh , oh , oh . phd b: So because the channel zero and channel one are like the same speech , but only w  , the same endpoints . professor c:  phd b: But the only thing is that the speech is very noisy for channel one , so you can actually use the output of the channel zero for channel one for the VAD .  , so a are they going to pro What are they doing to do , do we know yet ? about as far as what they 're what the rules are going to be and what we can use ? phd d: Yeah , so actually I received a a new document , describing this . phd b: Yeah , that 's phd d: And what they did finally is to , mmm , not to align the utterances but to perform recognition , only on the close - talking microphone , phd b: Which is the channel zero . professor c: So it 's not like that 's being done in one place or one time . phd d: And professor c: That 's that 's just a rule and we 'd you you were permitted to do that . Is is that it ? phd d: I think they will send , files but we we don't Well , apparently professor c: Oh , so they will send files so everybody will have the same boundaries to work with ? phd d: Yeah . phd d: Oh , i Yeah , so what happened here is that , the overall improvement that they have with this method So Well , to be more precise , what they have is , they have these alignments and then they drop the beginning silence and and the end silence but they keep , two hundred milliseconds before speech and two hundred after speech .  , and the overall improvement over the MFCC baseline So , when they just , add this frame dropping in addition it 's r  , forty percent , right ? professor c:   phd b: Yeah , which is phd d: which is , t which is the overall improvement . phd b: It gives like negative Well , in in like some Italian and TI - digits , phd d: Yeah , some @ @ . So by using the endpointed speech , actually it 's worse than the baseline in some instances , which could be due to the word pattern . professor c: Yeah , phd d: And Yeah , the other thing also is that fourteen percent is less than what you obtain using a real VAD . phd d: So  So I think this shows that there is still work  , well , working on the VAD is still still important I think . professor c: Yeah , c phd d:  phd a: Can I ask just a a high level question ? Can you just say like one or two sentences about Wiener filtering and why why are people doing that ? phd b:  . phd a: What 's what 's the deal with that ? phd b: OK , so the Wiener filter , it 's it 's like it 's like you try to minimize  , so the basic principle of Wiener filter is like you try to minimize the , d  , difference between the noisy signal and the clean signal if you have two channels . Like let 's say you have a clean t signal and you have an additional channel where you know what is the noisy signal . And you get you can do that  , if if you have only a c noisy signal , at a level which you , you w try to estimate the noise from the w assuming that the first few frames are noise or if you have a w voice activity detector , you estimate the noise spectrum . phd b: And then you phd a: Do you assume the noise is the same ? phd b: Yeah . phd b: So but that 's not the case in , many many of our cases but it works reasonably well . And then you do this  , this is the transfer function of the Wiener filter , so " SF " is a clean speech spectrum , power spectrum phd a:   professor c: Right phd b: And , professor c: actually , I guess phd b: Yeah . phd b: So but the thing is that you have to estimate the SF from the noisy spectrum , what you have . So you estimate the NF from the initial noise portions and then you subtract that from the current noisy spectrum to get an estimate of the SF . phd a: So is this , basically s  , similar to just regular spectral subtraction ? phd b: It professor c: It 's all pretty related , phd b: Yeah . It 's it 's there 's a di there 's a whole class of techniques where you try in some sense to minimize the noise . phd a: Do people use the Wiener filtering in combination with the spectral subtraction typically , or is i are they sort of competing techniques ? phd b: Not seen . phd b: So it 's like I haven't seen anybody using s Wiener filter with spectral subtraction . professor c: but y but there you make different approximations , and in spectral subtraction , for instance , there 's a a an estimation factor . professor c: You sometimes will figure out what the noise is and you 'll multiply that noise spectrum times some constant and subtract that rather than and sometimes people even though this really should be in the power domain , sometimes people s work in the magnitude domain because it it it works better . phd a: So why did you choose , Wiener filtering over some other one of these other techniques ? phd b: the reason was , like , we had this choice of using spectral subtraction , Wiener filtering , and there was one more thing which I which I 'm trying , is this sub space approach . phd b: So I picked up phd a: So you 're sort of trying @ @ them all . phd b: Y Yeah , phd a: Ah , phd b: we just wanted to have a few noise production compensation techniques phd a: I see . professor c: I m  yeah , there 's Car - Carmen 's working on another , on the vector Taylor series . professor c: So they were just kind of trying to cover a bunch of different things with this task and see , you know , what are what are the issues for each of them . phd b: So so one of one of the things that I tried , like I said , was to remove those zeros in the fri filter by doing some smoothing of the filter . phd b: Like , you estimate the edge of square and then you do a f smoothing across the frequency so that those zeros get , like , flattened out . So what I did was like I p did this and then you I plugged in the one more the same thing but with the smoothed filter the second time . phd b: So that 's where I got like fifty - six point five percent improvement on SpeechDat - Car with that . And So the other thing what I tried was I used still the ten frames of noise estimate but I used this channel zero VAD to drop the frames . And that has taken the performance to like sixty - seven percent in SpeechDat - Car , which is which which like sort of shows that by using a proper VAD you can just take it to further , better levels . phd a: So that 's sort of like , you know , best - case performance ? phd b: Yeah , so far I 've seen sixty - seven  , no , I haven't seen s like sixty - seven percent . And , using the channel zero VAD to estimate the noise also seems to be improving but I don't have the results for all the cases with that . So I used channel zero VAD to estimate noise as a lesser 2 x frame , which is like , everywhere I use the channel zero VAD . And that seems to be the best combination , rather than using a few frames to estimate and then drop a channel . phd b: so we c so  , which which means , like , by using this technique what we improve just the VAD professor c: Yes . And , w  Yeah , but this all these things , I have to still try it on the TI - digits , which is like I 'm just running . And there seems to be not improving a a lot on the TI - digits , so I 'm like investigating that , why it 's not . So ,  so the other the other thing is like I 've been I 'm doing all this stuff on the power spectrum . So Tried this stuff on the mel as well mel and the magnitude , and mel magnitude , and all those things . So , one of one of reasons I thought like doing the averaging , after the filtering using the mel filter bank , that seems to be maybe helping rather than trying it on the mel filter ba filtered outputs . phd b: Yeah , th that 's that 's the only thing that I could think of why why it 's giving improvement on the mel . professor c: how about the subspace stuff ? phd b: Subspace , I 'm I 'm like that 's still in a little bit in the back burner because I 've been p putting a lot effort on this to make it work , on tuning things and other stuff .  , So to r to remind you a little bit of of what I did before , is just to apply some spectral subtraction with an overestimation factor also to get , an estimate of the noise , spectrum , and subtract this estimation of the noise spectrum from the , signal spectrum , but subtracting more when the SNR is is , low , which is a technique that it 's often used . phd a: " Subtracting more " , meaning ? phd d: So you overestimate the noise spectrum . phd d: So , above twenty DB , it 's one , so you just subtract the noise . phd d: And then it 's b Generally Well , I use , actually , a linear , function of the SNR , phd a:   phd d: which is bounded to , like , two or three , when the SNR is below zero DB . phd d: doing just this , either on the FFT bins or on the mel bands , t doesn't yield any improvement professor c: Oh !  , what are you doing with negative , powers ? phd d: o Yeah . So there is also a threshold , of course , because after subtraction you can have negative energies , phd a:   phd d: and So what I I just do is to put ,  to to add to put the threshold first and then to add a small amount of noise , which right now is speech - shaped .  phd a: Speech - shaped ? phd d: Yeah , so it 's a it has the overall overall energy ,  pow it has the overall power spectrum of speech . phd a: So when y when you talk about there being something less than zero after subtracting the noise , is that at a particular frequency bin ? phd d: i  - huh . phd a: And so when you say you 're adding something that has the overall shape of speech , is that in a in a particular frequency bin ? Or you 're adding something across all the frequencies when you get these negatives ? phd d: For each frequencies I a I 'm adding some , noise , but the a the amount of the amount of noise I add is not the same for all the frequency bins . Right now I don't think if it makes sense to add something that 's speech - shaped , because then you have silence portion that have some spectra similar to the sp the overall speech spectra . So this is something I can still work on , phd a: So what does that mean ? phd d: but  . phd a: I 'm trying to understand what it means when you do the spectral subtraction and you get a negative . It means that at that particular frequency range you subtracted more energy than there was actually phd d: That means that   So so yeah , you have an an estimation of the noise spectrum , but sometimes , of course , it 's as the noise is not perfectly stationary , sometimes this estimation can be , too small , so you don't subtract enough . phd a: So in in an ideal word i world if the noise were always the same , then , when you subtracted it the worst that i you would get would be a zero .  , the lowest you would get would be a zero , cuz i if there was no other energy there you 're just subtracting exactly the noise . phd d:  -  , professor c: Yep , there 's all there 's all sorts of , deviations from the ideal here . professor c: for instance , you 're you 're talking about the signal and noise , at a particular point . And even if something is sort of stationary in ster terms of statistics , there 's no guarantee that any particular instantiation or piece of it is exactly a particular number or bounded by a particular range . professor c: So , you 're figuring out from some chunk of of of the signal what you think the noise is . professor c: and there 's absolutely no reason to think that you 'd know that it wouldn't , be negative in some places . professor c: on the other hand that just means that in some sense you 've made a mistake because you certainly have stra subtracted a bigger number than is due to the noise . professor c:  Also , we speak the whole where all this stuff comes from is from an assumption that signal and noise are uncorrelated . And that certainly makes sense in s in in a statistical interpretation , that , you know , over , all possible realizations that they 're uncorrelated phd a:   professor c: or assuming , ergodicity that i that i  , across time , it 's uncorrelated . But if you just look at a quarter second , and you cross - multiply the two things , you could very well , end up with something that sums to something that 's not zero . But if down the road you 're making use of something as if it is a power spectrum , then it can be bad to have something negative . Now , the other thing I wonder about actually is , what if you left it negative ? What happens ? phd b: Is that the log ? professor c: because  , are you taking the log before you add them up to the mel ? phd b: After that . So the thing is , I wonder how if you put your thresholds after that , I wonder how often you would end up with ,  with negative values . phd b: But you will But you end up reducing some neighboring frequency bins @ @ in the average , right ? When you add the negative to the positive value which is the true estimate . But nonetheless , you know , these are it 's another f kind of smoothing , right ? that you 're doing . So , you 've done your best shot at figuring out what the noise should be , and now i then you 've subtracted it off . And then after that , instead of instead of , leaving it as is and adding things adding up some neighbors , you artificially push it up . professor c: Which is , you know , it 's there 's no particular reason that that 's the right thing to do either , right ? phd b: Yeah , yeah . professor c: So , i in fact , what you 'd be doing is saying , " well , we 're d we 're we 're going to definitely diminish the effect of this frequency in this little frequency bin in the in the overall mel summation " . I d I don't know if it would be phd a: Sort of the opposite of that would be if if you find out you 're going to get a negative number , you don't do the subtraction for that bin . professor c: although phd a: That would be almost the opposite , right ? Instead of leaving it negative , you don't do it . If your if your subtraction 's going to result in a negative number , you you don't do subtraction in that . professor c: Yeah , but that means that in a situation where you thought that that the bin was almost entirely noise , you left it . phd d: And , yeah , some people also if it 's a negative value they , re - compute it using inter interpolation from the edges and bins . professor c: People can also , reflect it back up and essentially do a full wave rectification instead of a instead of half wave . Well , actually I tried , something else based on this , is to to put some smoothing , because it seems to to help or it seems to help the Wiener filtering professor c:   Actually , when you do spectral subtraction you can , find this this equivalent in the s in the spectral domain . You can  compute , y you can say that d your spectral subtraction is a filter , and the gain of this filter is the , signal energy minus what you subtract , divided by the signal energy . And this is a gain that varies over time , and , you know , of course , depending on the s on the noise spectrum and on the speech spectrum . And what happen actually is that during low SNR values , the gain is close to zero but it varies a lot . Mmm , and this this is the cause of musical noise and all these the the fact you we go below zero one frame and then you can have an energy that 's above zero . But it 's the smoothing is nonlinear in the sense that I tried to not smooth if the gain is high , because in this case we know that , the estimate of the gain is correct because we we are not close to to to zero , and to do more smoothing if the gain is low . So , well , basically that 's this idea , and it seems to give pretty good results , although I 've just just tested on Italian and Finnish . And on Italian it seems my result seems to be a little bit better than the Wiener filtering , phd b:   phd d: I don't know if you have these improvement the detailed improvements for Italian , Finnish , and Spanish there phd b: Fff . No , I don't have , for each , phd d: or you have just have your own . professor c: So these numbers he was giving before with the four point three , and the ten point one , and so forth , those were Italian , right ? phd b: Yeah , yeah , yeah . phd d:  phd b: I actually didn't give you the number which is the final one , phd d: no , we 've phd b: which is , after two stages of Wiener filtering . phd d:  professor c: But do you have numbers in terms of word error rates on on Italian ? So just so you have some sense of reference ? phd d: Yeah . professor c: And this is , spectral subtraction plus what ? phd d: Plus plus nonlinear smoothing . Well , it 's the system it 's exactly the sys the same system as Sunil tried , professor c: On - line normalization and LDA ? phd d: but professor c: Yeah . But instead of double stage Wiener filtering , it 's it 's this smoothed spectral subtraction . phd a: for Do they use spectral subtraction , or Wiener filtering , or ? phd b: They use spectral subtraction , right . phd d: It it 's Wiener filtering , phd b: Oh , it 's it 's Wiener filtering . professor c: Yeah , plus , I guess they have some sort of cepstral normalization , as well . phd b: s They have like yeah , th the just noise compensation technique is a variant of Wiener filtering , phd d:   phd d: One in the time domain and one in the frequency domain by just taking the first , coefficients of the impulse response .  , what you did , it 's similar phd b: It 's similar in the smoothing and phd d: because you have also two two kind of smoothing . phd d: One in the time domain , and one in the frequency domain , phd b: Yeah . phd a: Does the smoothing in the time domain help phd d:  phd a: Well , do you get this musical noise stuff with Wiener filtering or is that only with , spectral subtraction ? phd b: No , you get it with Wiener filtering also . phd a: Does the smoothing in the time domain help with that ? Or some other smoothing ? phd b: Oh , no , you still end up with zeros in the s spectrum . phd b: the phd d: Well , I cannot you cannot hear beca well , actually what I d did not say is that this is not in the FFT bins .  So , it could be seen as a f a a smoothing in the frequency domain because I used , in ad mel bands in addition and then the other phase of smoothing in the time domain . But , when you look at the spectrogram , if you don't have an any smoothing , you clearly see , like in silence portions , and at the beginning and end of speech , you see spots of high energy randomly distributed over the the spectrogram . phd d:  phd a: That 's the musical noise ? phd d: Which is musical noise , phd a:   phd d: yeah , if if it If you listen to it  , if you do this in the FFT bins , then you have spots of energy randomly distributing . And if you f if you re - synthesize these spot sounds as , like , sounds , phd a:   phd d:  professor c: Well , none of these systems , by the way , have  , y you both are are working with , our system that does not have the neural net , phd d: And phd b: Yep . So one would hope , presumably , that the neural net part of it would would improve things further as as they did before .  Yeah , although if if we , look at the result from the proposals , one of the reason , the n system with the neural net was , more than well , around five percent better , is that it was much better on highly mismatched condition . I 'm thinking , for instance , on the TI - digits trained on clean speech and tested on noisy speech . phd d: And if we have no , spectral subtraction or Wiener filtering , i the system is  , we thought the neural neural network is much better than before , even in these cases of high mismatch . phd a: Could you train a neural net to do spectral subtraction ? professor c: Yeah , it could do a nonlinear spectral subtraction phd d:   professor c: but I don't know if it  , you have to figure out what your targets are . phd a: Yeah , I was thinking if you had a clean version of the signal and and a noisy version , and your targets were the M F -  , you know , whatever , frequency bins phd d:   professor c: Yeah , well , that 's not so much spectral subtraction then , phd d:   professor c: but but but it 's but at any rate , yeah , people ,  phd a: People do that ? professor c: y yeah , in fact , we had visitors here who did that I think when you were here ba way back when . professor c: The objection everyone always raises , which has some truth to it is that , it 's good for mapping from a particular noise to clean but then you get a different noise . professor c: And the experiments we saw that visitors did here showed that it there was at least some , gentleness to the degradation when you switched to different noises . phd a: How did it compare on  , for for good cases where it it  , stuff that it was trained on ? Did it do pretty well ? professor c: Oh , yeah , it did very well .  , we 're not doing exactly that , we 're not trying to generate good examples but by trying to do the best classifier you possibly can , for these little phonetic categories , phd a: it we 'll just have to see whether it helps more or less the same , but I would imagine it would help some . professor c: So in any event , all of this I was just confirming that all of this was with a simpler system .  , Yeah , so this is th the ,  Well , actually , this was kind of the first try with this spectral subtraction plus smoothing , professor c:   And it seems that the one that I chose for the first experiment was the optimal one , so  , professor c: It 's amazing how often that happens .  Yeah , another thing that I it 's important to mention is , that this has a this has some additional latency . Because when I do the smoothing , it 's a recursion that estimated the means , so of the g of the gain curve . So , instead o of using the current estimated mean to , subtract the current frame , it 's better to use an estimate that 's some somewhere in the future . phd b: You mean , the m the mean is computed o based on some frames in the future also ? professor c:   phd b: Or or no ? phd d: It 's the recursion , so it 's it 's the center recursion , right ? phd b:   professor c: One five ? phd d:  professor c: One five ? Five zero ? phd d: Five zero , professor c: Five zero . phd b: why why is that delay coming ? Like , you estimate the mean ? phd d: Yeah , the mean estimation has some delay , right ? phd b: Oh , yeah . phd b: It isn't OK , so it 's like it looks into the future also . professor c: What if you just look into the past ? phd d: It 's , not as good . professor c: How m by how much ? phd d: it helps a lot over the ba the baseline but , mmm professor c: By how much ? phd d: it It 's around three percent , relative . phd d: mmm So ,  professor c: It 's depending on how all this stuff comes out we may or may not be able to add any latency . Yeah , b but I don't think we have to worry too much on that right now while you kno . professor c: s Yeah , I think the only thing is that phd d: So professor c: I would worry about it a little . professor c: Because if we completely ignore latency , and then we discover that we really have to do something about it , we 're going to be find ourselves in a bind . professor c: Yeah , just , you know , just be be a little conservative phd d: Oh yes . professor c: because we may end up with this crunch where all of a sudden we have to cut the latency in half or something . So , yeah , there are other things in the , algorithm that I didn't , @ @ a lot yet , phd a: Oh ! phd d: which phd a: Sorry . If if there 's another part of the system that causes a latency of a hundred milliseconds , is this an additive thing ? Or c or is yours hidden in that ? phd d:   We can do something in parallel also , in some like some cases like , if you wanted to do voice activity detection . phd b: So you can make a decision on that voice activity detection and then you decide whether you want to filter or not . phd a: couldn't ,  I Couldn't you just also  , i if you know that the l the largest latency in the system is two hundred milliseconds , don't you couldn't you just buffer up that number of frames and then everything uses that buffer ? phd b: Yeah . phd a: And that way it 's not additive ? professor c: Well , in fact , everything is sent over in buffers cuz of isn't it the TCP buffer some ? phd b: You mean , the the data , the super frame or something ? phd d:   phd b: Yeah , but that has a variable latency because the last frame doesn't have any latency phd d:   phd b: So But the only thing is that the first frame in that twenty - four frame buffer has a twenty - four frame latency . phd b: Because it just goes as phd a: Yeah , I wasn't thinking of that one in particular phd b: Yeah . phd a: but more of , you know , if if there is some part of your system that has to buffer twenty frames , can't the other parts of the system draw out of that buffer and therefore not add to the latency ? professor c: Yeah . And and that 's sort of one of the all of that sort of stuff is things that they 're debating in their standards committee . Like , I played a little bit with this overestimation factor , but I still have to to look more at this , at the level of noise I add after .  , I know that adding noise helped , the system just using spectral subtraction without smoothing , but I don't know right now if it 's still important or not , and if the level I choose before is still the right one .  , and another thing is to Yeah , for this I just use as noise estimate the mean , spectrum of the first twenty frames of each utterance . I don't remember for this experiment what did you use for these two stage phd b: I used ten just ten frames . Yeah , because phd d: The ten frames ? phd b: the reason was like in TI - digits I don't have a lot . But , so what 's this result you told me about , the fact that if you use more than ten frames you can improve by t phd b: Well , that 's that 's using the channel zero . phd b: Which phd d: But this is ten frames plus plus phd b: Channel zero dropping . phd d: no , these results with two stage Wiener filtering is ten frames phd b: t Oh , this phd d: but possibly more . The second thing I was working on is to , try to look at noise estimation , mmm , and using some technique that doesn't need voice activity detection .  , and for this I u simply used some code that , I had from from Belgium , which is technique that , takes a bunch of frame , and for each frequency bands of this frame , takes a look at the minima of the energy . And then average these minima and take this as an an energy estimate of the noise for this particular frequency band . So , I compute an FFT based on the long , signal frame which is sixty - four millisecond phd a: So you have one minimum for each frequency ? phd d: What what I what I d  , I do actually , is to take a bunch of to take a tile on the spectrogram and this tile is five hundred milliseconds long and two hundred hertz wide . phd d: And this tile  , in this tile appears , like , the harmonics if you have a voiced sound , because it 's it 's the FTT bins . And when you take the m the minima of of these this tile , when you don't have speech , these minima will give you some noise level estimate , If you have voiced speech , these minima will still give you some noise estimate because the minima are between the harmonics . And If you have other other kind of speech sounds then it 's not the case , but if the time frame is long enough , like s five hundred milliseconds seems to be long enough , you still have portions which , are very close whi which minima are very close to the noise energy . You said five hundred milliseconds phd d: Mmm ? professor c: but you said sixty - four milliseconds . Which is which ? What ? phd d: Sixty - four milliseconds is to compute the FFT , bins . phd d: actually it 's better to use sixty - four milliseconds because , if you use thirty milliseconds , then , because of the this short windowing and at low pitch , sounds , the harmonics are not , wha  , correctly separated . phd d: So if you take these minima , it b they will overestimate the noise a lot . professor c: So you take sixty - four millisecond F F Ts and then you average them over five hundred ? Or ?  , what do you do over five hundred ? phd d: So I take to I take a bunch of these sixty - four millisecond frame to cover five hundred milliseconds , professor c: Ah . phd d: on the on on the bunch of  fifty frames , right ? professor c: I see . So the interest of this is that , as y with this technique you can estimate u some reasonable noise spectra with only five hundred milliseconds of of signal , so if the the n the noise varies a lot , you can track better track the noise , professor c:   The only requirement is that you must have , in these five hundred milliseconds segment , you must have voiced sound at least . So what I did is just to simply replace the VAD - based , noise estimate by this estimate , first on SpeechDat - Car Well , only on SpeechDat - Car actually . And it 's , slightly worse , like one percent relative compared to the VAD - based estimates .  , I think the reason why it 's not better , is that the SpeechDat - Car noises are all stationary . So , u y y there really is no need to have something that 's adaptive professor c:   But , I expect s maybe some improvement on TI - digits because , nnn , in this case the noises are all sometimes very variable . professor c: But are you comparing with something e I 'm I 'm p s a little confused again , i it  , when you compare it with the V A D - based , phd d:   professor c: VAD - Is this is this the ? phd d: It 's It 's the France - Telecom - based spectra , s  , Wiener filtering and VAD . professor c: Oh , you 're not doing this with our system ? phd d: In i I 'm not No , no . Actually , th the best system that we still have is , our system but with their noise compensation scheme , right ? professor c: Right . But phd d: So I 'm trying to improve on this , and by by replacing their noise estimate by , something that might be better . professor c: Couldn't you try this for that ? phd d: But I di professor c: Do you think it might help ? phd d: Not yet , because I did this in parallel , professor c: I see , phd d: and I was working on one and the other . phd b: So I 'm also using that n new noise estimate technique on this Wiener filtering what I 'm trying . phd b: So I I have , like , some experiments running , I don't have the results .  , the idea is just to  , flatten the log , spectrum , and to flatten it more if the the probability of silence is higher . So in this way , you can also reduce somewhat reduce the musical noise and you reduce the variability if you have different noise shapes , because the the spectrum becomes more flat in the silence portions . With this , no improvement , but there are a lot of parameters that we can play with and ,  Actually , this this could be seen as a soft version of the frame dropping because , you could just put the threshold and say that " below the threshold , I will flatten comp completely flatten the the spectrum " . So it would be like frame dropping , because during the silence portions which are below the threshold of voice activity probability , w you would have some kind of dummy frame which is a perfectly flat spectrum . And this , whitening is something that 's more soft because , you whiten you just , have a function the whitening is a function of the speech probability , so it 's not a hard decision . phd d: so I think maybe it can be used together with frame dropping and when we are not sure about if it 's speech or silence , well , maybe it has something do with this .  , you know , in in JRASTA we were essentially adding in , white  , white noise dependent on our estimate of the noise . professor c: You could imagine one that that that made use of where where the amount that you added in was , a function of the probability of it being s speech or noise . Yeah , w Yeah , right now it 's a constant that just depending on the the noise spectrum . professor c: Cuz that that brings in sort of powers of classifiers that we don't really have in , this other estimate . professor c: What what what point does the , system stop recording ? How much phd a: It 'll keep going till I guess when they run out of disk space , professor c: It went a little long ?  , disk phd a: but I think we 're OK .  Yeah , so there are with this technique there are some I just did something exactly the same as as the Ericsson proposal but , the probability of speech is not computed the same way . And I think , i for yeah , for a lot of things , actually a g a good speech probability is important . Like for frame dropping you improve , like you can improve from ten percent as Sunil showed , if you use the channel zero speech probabilities .  , so yeah , the next thing I started to do is to , try to develop a better voice activity detector . And ,  I d  yeah , for this I think we can maybe try to train the neural network for voice activity detection on all the data that we have , including all the SpeechDat - Car data .  , and the way I mi I do that is that I just use the HTK system but I train it only on the close - talking microphone .  It seems to be ,  i Actually what I observed is that for Italian it doesn't seem Th - there seems to be a problem . Because What ? phd b: you mean their d the frame dropping , right ? Yeah , it doesn't phd d: Yeah . phd d: so  , the c the current VAD that we have was trained on , t SPINE , right ? phd b: TI - digits . And actually we observed we listened to some of the utterances and sometimes for Finnish there is music in the recordings and strange things , right ? phd b: Yeah . phd d:  Yeah , so the idea was to train all the databases and obtain an alignment to train on these databases , and , also to , try different kind of features , as input to the VAD network . And we came up with a bunch of features that we want to try like , the spectral slope , the , the degree o degree of voicing with the features that , we started to develop with Carmen , e with , the correlation between bands and different kind of features , phd b: Yeah . Well , Hans - Guenter will be here next week so I think he 'll be interested in all all of these things