phd e: So it 's well , it 's spectral subtraction or Wiener filtering , depending on if we put if we square the transfer function or not . phd e: And then with over - estimation of the noise , depending on the ,  the SNR , with smoothing along time , smoothing along frequency . phd e: And , the best result is when we apply this procedure on FFT bins , with a Wiener filter . phd e: So it 's good because it 's difficult when we have to add noise to to to find the right level . It 's the same , idea but it 's working on mel bands , and it 's a spectral subtraction instead of Wiener filter , and there is also a noise addition after , cleaning up the mel bins . professor b: if you look at databases , the , one that has the smallest smaller overall number is actually better on the Finnish and Spanish , but it is , worse on the , Aurora phd e: It 's worse on professor b:  on the , TI - TI - digits , phd e: on the multi - condition in TI - digits . But , when you say u  , unified do you mean , it 's one piece of software now , or ? phd e: So now we are , yeah , setting up the software .  , and we phd a: So what 's what 's happened ? I think I 've missed something . So a week ago maybe you weren't around when when when Hynek and Guenther and I ? phd c: Hynek was here .  And then if I summarize somebody can tell me if I 'm wrong , which will also be possibly helpful . phd e: p - p - p professor b: We ,  we looked at ,  anyway we after coming back from QualComm we had , you know , very strong feedback and , I think it was Hynek and Guenter 's and my opinion also that , you know , we sort of spread out to look at a number of different ways of doing noise suppression . professor b: we had a long discussion about how they were the same and how they were d  , completely different . professor b: And , fundamentally they 're the same sort of thing but the math is a little different so that there 's a a there 's an exponent difference in the index you know , what 's the ideal filtering , and depending on how you construct the problem . professor b: And , I guess it 's sort you know , after after that meeting it sort of made more sense to me because  , if you 're dealing with power spectra then how are you gonna choose your error ? And typically you 'll do choose something like a variance . Whereas when you 're when you 're doing the the , looking at it the other way , you 're gonna be dealing with signals phd c:   professor b: and you 're gonna end up looking at power  , noise power that you 're trying to reduce . And so , eh so there should be a difference of you know , conceptually of of , a factor of two in the exponent . professor b: But there 're so many different little factors that you adjust in terms of of , over - subtraction and and and and and so forth , that arguably , you 're c and and and the choice of do you do you operate on the mel bands or do you operate on the FFT beforehand . There 're so many other choices to make that are are almost well , if not independent , certainly in addition to the choice of whether you , do spectral subtraction or Wiener filtering , that , @ @ again we sort of felt the gang should just sort of figure out which it is they wanna do and then let 's pick it , go forward with it . And and , we said , take a week , go arm wrestle , you know , grad d: Oh . professor b: And and so they so instead they went to Yosemite and bonded , and and they came out with a single single piece of software . phd a: So so you guys have combined or you 're going to be combining the software ? professor b:  . phd c: Well , the piece of software has , like , plenty of options , phd e: Oh boy . professor b: Well , that 's fine , but the thing is the important thing is that there is a piece of software that you that we all will be using now . phd e: Well , if we want to , like , optimize different parameters of phd c: Parameters . But , still so , there will be a piece of software with , will give this system , the fifty - three point sixty - six , by default and professor b:   phd a: I I I don't have a sense of phd e: It 's just one percent off of the best proposal . professor b: So it so , it 's it it 's not using our full bal bag of tricks , if you will . professor b: And , and it it is , very close in performance to the best thing that was there before .  , but , you know , looking at it another way , maybe more importantly , we didn't have any explicit noise , handling stationary dealing with e e we didn't explicitly have anything to deal with stationary noise . phd a: So will the neural net operate on the output from either the Wiener filtering or the spectral subtraction ? Or will it operate on the original ? professor b: Well , so so so argu arguably , what we should do  , I gather you have it sounds like you have a few more days of of nailing things down with the software and so on . But and then but , arguably what we should do is , even though the software can do many things , we should for now pick a set of things , th these things I would guess , and not change that . And I think , you know , that our goal should be by next week , when Hynek comes back , to  , really just to have a firm path , for the you know , for the time he 's gone , of of , what things will be attacked . But I would I would I would thought think that what we would wanna do is not futz with this stuff for a while because what 'll happen is we 'll change many other things in the system , phd a:   professor b: and then we 'll probably wanna come back to this and possibly make some other choices . phd a: But just conceptually , where does the neural net go ? Do do you wanna h run it on the output of the spectrally subtracted ? phd e: Mmm . professor b: Well , depending on its size Well , one question is , is it on the , server side or is it on the terminal side ?  , if it 's on the server side , it you probably don't have to worry too much about size . So the issue is is , for instance , could we have a neural net that only looked at the past ? phd a: Right . professor b: what we 've done in  in the past is to use the neural net , to transform , all of the features that we use . This is essentially ,  I guess it 's it 's more or less like a spee a speech enhancement technique here phd a:   professor b: right ? where we 're just kind of creating new if not new speech at least new new FFT 's that that have you know , which could be turned into speech  , that that have some of the noise removed . professor b: after that we still do a mess of other things to to produce a bunch of features . And then the the way that we had it in our proposal - two before , we had the neural net transformed features and we had the untransformed features , which I guess you you actually did linearly transform with the KLT , phd e: Yeah . professor b: but but but  , to orthogonalize them but but they were not , processed through a neural net . And Stephane 's idea with that , as I recall , was that you 'd have one part of the feature vector that was very discriminant and another part that wasn't , phd a:   professor b: which would smooth things a bit for those occasions when , the testing set was quite different than what you 'd trained your discriminant features for .  , y you know , that 's still being debated by the by people in Europe but , no matter how they end up there , it 's not going to be unlimited amounts , phd a: Yeah . And I think those that we last time we agreed that those are the three things that have to get , focused on . phd a: What was the issue with the VAD ? professor b: Well , better ones are good . phd a: And so the w the default , boundaries that they provide are they 're OK , but they 're not all that great ? professor b: I guess they still allow two hundred milliseconds on either side or some ? Is that what the deal is ? phd e:   phd e: And all the speech pauses , which is Sometimes on the SpeechDat - Car you have pauses that are more than one or two seconds . And , yeah , it seems to us that this way of just dropping the beginning and end is not We cou we can do better , I think , phd a:   phd e: because , with this way of dropping the frames they improve over the baseline by fourteen percent and Sunil already showed that with our current VAD we can improve by more than twenty percent . phd e: So , our current VAD is is more than twenty percent , while their is fourteen . And another thing that we did also is that we have all this training data for let 's say , for SpeechDat - Car . And if we just take only the , VAD probabilities computed on the clean signal and apply them on the far - field , test utterances , then results are much better . phd e: So it means that there are stim still phd a: How how much latency does the ,  does our VAD add ? phd e: If if we can have a good VAD , well , it would be great . phd a: Is it significant , phd e: right now it 's , a neural net with nine frames . phd a: or ? phd e: So it 's forty milliseconds plus , the rank ordering , which , should be phd c: Like another ten frames . professor b: So what 's the ? If you ignore  , the VAD is sort of in in parallel , isn't i isn't it , with with the ?  , it isn't additive with the the , LDA and the Wiener filtering , and so forth . phd c: So we  , if so if we if so which is like if we reduce the delay of VA So , the f the final delay 's now ba is f determined by the delay of the VAD , because the LDA doesn't have any delay . So if we re if we reduce the delay of the VAD , it 's like effectively reducing the delay . phd a: How how much , delay was there on the LDA ? phd c: So the LDA and the VAD both had a hundred millisecond delay . So and they were in parallel , so which means you pick either one of them phd a: Mmm . professor b: And there didn't seem to be any , penalty for that ? There didn't seem to be any penalty for making it causal ? phd c: Pardon ? Oh , no . phd c: Or something like that professor b: And he says Wiener filter is is forty milliseconds delay . phd a: What amount of latency are you thinking about when you say that ? professor b:  . professor b: You know , they 're saying ,  one group is saying a hundred and thirty milliseconds and another group is saying two hundred and fifty milliseconds . phd a: Were you thinking of the two - fifty or the one - thirty when you said we should have enough for the neural net ? professor b: Well , it just it when we find that out it might change exactly how we do it , is all . professor b: how much effort do we put into making it causal ?  , I think the neural net will probably do better if it looks at a little bit of the future . And , you know , how how much time should we put into into that ? So it 'd be helpful if we find out from the the standards folks whether , you know , they 're gonna restrict that or not . But I think , you know , at this point our major concern is making the performance better and and , if , something has to take a little longer in latency in order to do it that 's you know , a secondary issue . professor b: But if we get told otherwise then , you know , we may have to c clamp down a bit more . phd c: So , the one one one difference is that was there is like we tried computing the delta and then doing the frame - dropping . phd c: The earlier system was do the frame - dropping and then compute the delta on the professor b:  - huh . So , yeah , what we do is we compute the silence probability , convert it to that binary flag , professor b:  - huh . phd c: and then in the end you c up upsample it to match the final features number of phd e:   phd a: Did that help then ? phd c: It seems to be helping on the well - matched condition . And it actually r reduced a little bit on the high mismatch , so in the final weightage it 's b b better because the well - matched is still weighted more than professor b: So , @ @  , you were doing a lot of changes . Did you happen to notice how much , the change was due to just this frame - dropping problem ? What about this ? phd c: y you had something on it . Sometime we we change two two things together and But it 's around maybe it 's less than one percent . But like we 're saying , if there 's four or five things like that then pretty sho soon you 're talking real improvement . Well , we can do the frame - dropping on the server side or we can just be careful at the terminal side to send a couple of more frames before and after , and So . phd a: You have ,  So when you  , maybe I don't quite understand how this works , but , couldn't you just send all of the frames , but mark the ones that are supposed to be dropped ? Cuz you have a bunch more bandwidth .  , it it always seemed to us that it would be kind of nice to in addition to , reducing insertions , actually use up less bandwidth . phd a: If the net 's on the server side then it could use all of the frames . It 's , like , you mean you just transferred everything and then finally drop the frames after the neural net . Right now we are  , ri Right now what wha what we did is , like , we just mark we just have this additional bit which goes around the features , saying it 's currently a it 's a speech or a nonspeech . phd c: So there is no frame - dropping till the final features , like , including the deltas are computed . phd c: And after the deltas are computed , you just pick up the ones that are marked silence and then drop them . professor b: So it would be more or less the same thing with the neural net , I guess , actually . Yeah , that 's what that 's what that 's what , this is doing right now . So , what 's ,  ? That 's that 's a good set of work that that ,  phd c: Just one more thing . Like , should we do something f more for the noise estimation , because we still ? professor b: Yeah . I tried just plugging the , Guenter noise estimation on this system , and it  , it got worse . professor b: it does seem like , you know , i i i i some compromise between always depending on the first fifteen frames and a a always depending on a a pause is is is a good idea .  , maybe you have to weight the estimate from the first - teen fifteen frames more heavily than than was done in your first attempt . No , do you have any way of assessing how well or how poorly the noise estimation is currently doing ? phd e: Mmm . phd e: We don't have nothing that phd c: Is there was there any experiment with ? Well , I I did The only experiment where I tried was I used the channel zero VAD for the noise estimation and frame - dropping . So I don't have a I don't have a split , like which one helped more . phd c: So , that 's the professor b: So that 's something you could do with , this final system . professor b: If it 's , you know , essentially not better , then it 's probably not worth phd e: Yeah . It 's , like , ev even even if I use a channel zero VAD , I 'm just averaging the the s power spectrum . But the Guenter 's argument is , like , if it is a non - stationary segment , then he doesn't update the noise spectrum . So , th the Guenter was arguing that , even if you have a very good VAD , averaging it , like , over the whole thing is not a good idea . phd c: Because you 're averaging the stationary and the non - stationary , and finally you end up getting something which is not really the s because , you anyway , you can't remove the stationary part fr  , non - stationary part from the signal . So , that 's so that 's still a slight difference from what Guenter is trying  professor b: Well , yeah . And and also there 's just the fact that , eh , although we 're trying to do very well on this evaluation , we actually would like to have something that worked well in general . And , relying on having fifteen frames at the front or something is is pretty phd c: Yeah , yeah .  , it 'd certainly be more robust to different kinds of input if you had at least some updates . What what do you ,  what do you guys see as as being what you would be doing in the next week , given wha what 's happened ? phd c: Cure the VAD ? phd e: Yeah . phd e: So , should we keep the same ? I think we might try to keep the same idea of having a neural network , but training it on more data and adding better features , I think , but because the current network is just PLP features . phd e: There 's no RASTA , no phd a: So , I I don't remember what you said the answer to my , question earlier . Will you will you train the net on after you 've done the spectral subtraction or the Wiener filtering ? professor b: This is a different net . phd c: So we have a VAD which is like neur that 's a neural net . phd c: So , right now we have , like ,  we have the cleaned - up features , so we can have a better VAD by training the net on the cleaned - up speech . So it 's , like , where do we want to put the VAD ?  , it 's like phd a: Can you use the same net to do both , or ? phd c: For phd a: Can you use the same net that you that I was talking about to do the VAD ? phd c:   phd c: So the net the final net  , which is the feature net so that actually comes after a chain of , like , LDA plus everything . And and you can actually do it for final frame - dropping , but not for the VA - f noise estimation . professor b: You see , the idea is that the , initial decision to that that you 're in silence or speech happens pretty quickly . phd a: Cuz that 's used by some of these other ? professor b: And that Yeah . And that 's sort of fed forward , and and you say " well , flush everything , it 's not speech anymore " . professor b: it is used ,  Yeah , it 's only used f Well , it 's used for frame - dropping . professor b: because , you know , there 's if you have more than five hundred milliseconds of of of nonspeech then you figure it 's end of utterance or something like that .  , keeping the same method but but , seeing if you cou but ,  noise estimation could be improved . And then , later on in the month I think we wanna start including the neural net at the end . phd a: So , Hynek is coming back next week , you said ? professor b: Yeah , that 's the plan . professor b: I guess the week after he 'll be , going back to Europe , and so we wanna phd a: Is he in Europe right now or is he up at ? professor b: No , no .  , the idea was that , we 'd we 'd sort out where we were going next with this with this work before he , left on this next trip .  , Barry , you just got through your quals , so I don't know if you have much to say . No , just , looking into some some of the things that , John Ohala and Hynek , gave as feedback , as as a starting point for the project . In in my proposal , I I was thinking about starting from a set of , phonological features , or a subset of them . grad d: He said , these these phonological features are are sort of figments of imagination also . grad d: Ye professor b: But we don't have too much trouble recognizing synthetic speech since we create it in the first place . grad d: just looking at the data and seeing what 's similar and what 's not similar . grad d: So , I 'm I 'm , taking a look at some of , Sangita 's work on on TRAPS . She did something where ,  w where the TRAPS learn She clustered the the temporal patterns of , certain certain phonemes in in m averaged over many , many contexts . grad d: And , so , those are interesting things to phd a: So you 're now you 're sort of looking to try to gather a set of these types of features ? grad d: Right . grad d: you know ? A a a set of small features and continue to iterate and find , a better set


professor b: OK , so We we had a meeting with ,  with Hynek , in in which , Sunil and Stephane ,  summarized where they were and and , talked about where we were gonna go . phd e: D did did you guys get your code pushed together ? phd d: Oh , yeah . professor b: What was the update ? phd a: What was the update ? So there is th then the all the new features that go in . These are the phd e: Is the ,  the CVS mechanism working well ? phd a: Yeah . phd e: Are are people , up at OGI grabbing code  , via that ? phd d: I don't think I don't think phd e: Or ? phd a: I don't know if they use it , but . phd d: Yeah , I I don't think anybody up there is like working on it right now . professor b: I think it more likely that what it means is that when Sunil is up there he will grab it . professor b: But what 'll happen is is he 'll go back up there and , Pratibha will come back from from , the east coast . professor b: And ,  and and I guess actually , after Eurospeech for a little bit , he 'll go up there too . So , actually everybody who 's working on it will be up there for at least a little while . phd e: So has Has anybody tried remotely accessing the CVS using , SSH ? professor b: Yeah .  phd a: I don't know if Hari did that or You d phd d: I can actually do it today .  , I can just log into phd e: Have you tried it yet ? phd d: No , I didn't . phd a: Actually I I tried wh while when I installed the repository , I tried from Belgium . phd a: I logged in there and I tried to import phd e: Yeah ? It worked good ? phd a: Yeah , it works . phd e: Oh , good ! phd a: But it 's So , right now it 's the mechanism with SSH . phd e: Great ! phd a: I don't s I didn't set up You can also set up a CVS server on a new port . phd e: right ? phd a: But I didn't do that because I was not sure about security problems . I I would have to phd e: So w when you came in from Belgian Belgium , using SSH , was it asking you for your own password into ICSI ? So if yo you can only do that if you have an account at ICSI ? phd a: Right . phd e: Cuz there is an a way to set up anonymous CVS right ? phd a: Yeah , you ha in this way you ca you have to set up a CVS server but then , yeah , you can access it . phd e: So the anonymous mechanism phd a: You can access them and mostly if you if y the set the server is set up like this . Because a lot of the open source stuff works with anonymous CVS and I 'm just wondering  , for our transcripts we may want to do that . professor b: Yeah , for this stuff I don't think we 're quite up to that . So ,  phd e: What 's new ? professor b: Well , I think maybe the thing to me might be I me I 'm sure you 've just been working on on , details of that since the meeting , right ? And so phd a: Mmm , since the meeting , well , I I 've been I 've been train training a new VAD and a new feature net . professor b: But I guess maybe the thing since you weren't yo you guys weren't at that that meeting , might be just just to , sort of recap , the the conclusions of the meeting . Cuz that was sort of ,  we we 'd sort of been working up to that , that that , he would come here this week and and we would sort of phd e:  - huh . professor b: Since he 's going out of town like now , and I 'm going out town in a couple weeks , and time is marching , sort of , given all the mu many wonderful things we could be working on , what what will we actually focus on ? phd e:   professor b: And ,  and what do we freeze ? And , you know , what do we ? So ,  . So then there 's something central and there aren't at least a bunch of different versions going off in in ways that differ trivially . professor b: and then within that , I guess the idea was to freeze a certain set of options for now , to run it , a particular way , and decide on what things are gonna be experimented with , as opposed to just experimenting with everything . So we 've been working like six weeks on on the noise compensation and we end up with something that seems reasonable . phd e: Are you gonna use which of the two techniques ? phd a: So finally it 's it 's , Wiener filtering on FFT bins . And it uses , two steps , smoothing of the transfer function , the first step , that 's along time , which use recursion . And after this step there is a further smoothing along frequency , which use a sliding window of twenty FFT bins . And ,  phd e: So this is on the  , before any mel scaling has been done ? phd a: Yeah , yeah . phd e: This is phd a: It was professor b: This this smoothing is done on the estimate , of what you 're going to subtract ? Or on the thing that has already had something subtracted ? phd a: Yeah . So we are going to fix this for the moment and work on the other aspects of the whole system . phd a: So professor b: Actually , let me int eh , Dave isn't here to talk about it , but let me just interject . This module , in principle , i  , you would know whether it 's true in fact , is somewhat independent from the rest of it .  , well you don't I guess you don't re - synthesize speech , but you could phd a: We we do not fo professor b: but you could . phd a: Well well , we do , but we don't don't re - synthesize . In in the program we don't re - synthesize and then re - analyze once again . professor b: But you have a re - synthesized thing that you that 's an an option here . professor b: Yeah , I gu I guess my point is that , i in some of the work he 's doing in reverberation , one of the things that we 're finding is that , it 's it 's for the for an artificial situation , we can just deal with the reverberation and his techniques work really well . But for the real situation  , problem is , is that you don't just have reverberation , you have reverberation in noise . So in fact it might be a very nice thing to do , to just take the noise removal part of it and put that in front of what he 's looking at . And , generate new files or whatever , and and ,  and then do the reverberation part . He 's  , e phd e: I guess he 's busy with professor b: Yeah , prelims , right . professor b: but but , you know , that 'll  , it 's clear that we ,  we are not with the real case that we 're looking at , we can't just look at reverberation in isolation because the interaction between that and noise is is considerable . And that 's  , in the past we 've looked at , and this is hard enough , the interaction between channel effects and and ,  and additive noise , so convolutional effects and and additive effects . And we have , the , LDA stuff that in principle is doing something about convolutional effects . i i There 's all these interactions between these two and that 's part of why these guys had to work so hard on on juggling everything around . But now when you throw in the reverberation , it 's even worse , because not only do you have these effects , but you also have some long time effects . And , so Dave has something which , is doing some nice things under some conditions with with long time effects but when it 's when there 's noise there too , it 's it 's it 's pretty hard . So we have to start Since any almost any real situation is gonna have  , where you have the microphone distant , is going to have both things , we we actually have to think about both at the same time . professor b: So ,  So there 's this noise suppression thing , which is sort of worked out and then , maybe you should just continue telling what what else is in the the form we have . phd a: Yeah , well , the , the other parts of the system are the the blocks that were already present before and that we did not modify a lot . professor b: So that 's again , that that 's the Wiener filtering , followed by , that 's done at the FFT level . And then the mel and then the log , and then the phd a: Then the LDA filter , professor b: LDA filter . phd a: mmm , then the downsampling , professor b: And then  downsample , phd a: DCT , professor b: DCT , phd a: then , on - line normalization , professor b: on - line norm , phd a: followed by upsampling . phd a: And finally frame dropping , which  , would be a neural network also , used for estimated silence probabilities . And the input of this neural network would be somewhere between log mel bands or one of the earlier stages of the processing . So that 's sort of most of this stuff is yeah , is operating parallel with this other stuff . So the things that we , I guess we sort of  , There 's there 's some , neat ideas for V A So , in I think there 's sort of like There 's a bunch of tuning things to improve stuff . There 's questions about various places where there 's an exponent , if it 's the right exponent , or ways that we 're estimating noise , that we can improve estimating noise . But structurally it seemed like the things the main things that that we brought up that , are are gonna need to get worked on seriously are , a a significantly better VAD , putting the neural net on , which , you know , we haven't been doing anything with , the , neural net at the end there , and , the , opening up the second front . phd e: The other half of the channel ? professor b: Yeah , yeah , cuz we we have we have , half the the , data rate that they allow . phd e: That what you mean ? professor b: And , so the initial thing which came from , the meeting that we had down south was , that , we 'll initially just put in a mel spectrum as the second one . We probably will go to something better later , but the initial thing is that cepstra and spectra behave differently , phd e:   phd e: So if you took the system the way it is now , the way it 's fro you 're gonna freeze it , and it ran it on the last evaluation , where it would it be ? phd a:   It , phd e: In terms of ranking ? phd a: Ri - right now it 's second . professor b: Although you you know , you haven't tested it actually on the German and Danish , have you ? phd a: No , we didn't . phd e: So on the ones that you did test it on it would have been second ? professor b: Yeah . Would it  But When you 're saying second , you 're comparing to the numbers that the ,  that the best system before got on ,  also without German and Danish ? phd a: Yeah , yeah . professor b: Well ranking didn't before , but I 'm just asking where this is to where theirs was without the German and Danish , phd a: Yeah . phd e: Where where where were we actually on the last test ? professor b: Oh , we were also esp essentially second , although there were there were  , we had a couple systems and they had a couple systems . And so , I guess by that we were third , but  , there were two systems that were pretty close , that came from the same place . phd e: We 're so this second that you 're saying now is system - wide second ? professor b: See  , no I think it 's also institutional , isn't it ? phd e: Still institutionally second ? professor b: Right ?  , I think both of their systems probably phd a: we are between their two systems . professor b: And and , you know , in some sense we 're all doing fairly similar things .  , one could argue about the LDA and so forth but I I think , you know , in a lot of ways we 're doing very similar things . But what what phd e: So how did they fill up this all these these bits ?  , if we 're u professor b: why are we using half ? Well , so you could you c phd e: Yeah . Or how are they using more than half , I guess maybe is what I professor b: Yeah , so I I think  , you guys are closer to it than me , so correct me if I 'm wrong , but I I think that what 's going on is that in in both cases , some kind of normalization is done to deal with convola convolutional effects . And they seem to comple complement each other enough and be different enough that they both seem to help help us . phd e: So th So professor b: So that if you throw away high modulation frequencies , then you can downsample . professor b: So phd e: So what if you didn't So do you explicitly downsample then ? Do we explicitly downsample ? professor b: Yeah . phd e: And what if we didn't do that ? Would we get worse performance ? phd a:  Yeah , not better , not worse . professor b: I think it doesn't affect it , does it ? phd e: I see . So I think the thing is , since we 're not evidently throwing away useful information , let 's try to put in some useful information . professor b: And , so I you know , we we 've found in a lot of ways for quite a while that having a second stream  , helps a lot . So that 's that 's put in , and you know , it may even end up with mel spectrum even though I 'm saying I think we could do much better , just because it 's simple . And you know , in the long run having something everybody will look at and say , " oh , yeah , I understand " , is is very helpful . phd e: So you would you 're You 're thinking to put the , mel spectrum in before any of the noise removal stuff ? or after ? professor b: Well , that 's a question . It looks like it 'd be straightforward to to , remove the noise , and , phd e: Cuz that happens before the mel conversion , right ? professor b: Yeah . There 's even a question in my mind anyhow of whether th you should take the log or not . phd d: Well , it it it it so it actually makes it dependent on the overall energy of the  , the frame . professor b: If you do or don't normalize ? phd d: If yo if you don't normalize and if if you don't normalize . And then if if normalization helps , then y you have something to compare against , and say , " OK , this much effect "  , you don't want to change six things and then see what happens . And then saying , oh  particularly because we 've found in the past there 's all these these these different results you get with slight modifications of how you do normalization . So , I would think you would wanna have some baseline that says , " OK , we don't normalize , this is what we get " , when we do this normalization , when we do that normalization . phd e: So this second stream , will it add latency to the system professor b: No , it 's in parallel . phd e: or ? grad c: Para professor b: We 're not talking about computation time here . grad c: So with this , new stream would you train up a VAD on both both features , somehow ? phd d: No , I guess the VAD has its own set of features . that 's phd d: which could be this one of these streams , or it can be something derived from these streams . phd a: And there is also the idea of using TRAPS , maybe , for the VAD , which ,  phd d: Yeah , that 's also phd a: Well , Pratibha apparently showed , when , she was at IBM , that it 's a good idea . grad c: Would would that fit on the handset , or ? Oh ! phd a: I have no idea . phd d: Well , it has t  the th phd a: It would have to fit but Yeah . And so I guess the issue there is , are we are we using neural - net - based TRAPS , and and how big are they ? So that 'll that 'll be , you know , an issue . grad c: Cuz she also does the ,  the correlation - based , TRAPS , with without the neural net , just looking at the correlation between professor b: Right . professor b: Or a simple neural net , right ?  , the thing is , if you 're doing correlation , you 're just doing a simple  , dot product , you know , with some weights which you happened to learn from this learn from the data . professor b: And so , putting a nonlinearity on it is , you know , not that big a deal . professor b: So , the question is , how complex a function do you need ? Do you need to have an added layer or something ? In which case , potentially , you know , it could be big . phd e: So the meeting with Hynek that you guys just had was to decide exactly what you were gonna freeze in this system ? Is that ? Or was there ? Were you talking about what t new stuff , or ? professor b: What to freeze and then what to do after we froze . And like I was saying , I think the you know , the basic directions are , there 's lots of little things , such as improve the noise estimator but the bigger things are adding on the neural net and , the second stream . phd d: So , I 'll ,  I 'll actually after the meeting I 'll add the second stream to the VAD and maybe I 'll start with the feature net in that case . It 's like , you 're looking at the VAD , right ? phd a: yeah . phd d: OK , so just figure how to take the features from the final phd a: Yeah . But , yeah , I think there are plenty of issues to work on for the feature net @ @ . phd e: What about the , the new part of the evaluation , the , Wall Street Journal part ? professor b: Right . Well you you may be called upon to help , on account of , all the work in this stuff here has been , with small vocabulary . So what how is the , interaction supposed to happen ?  , I remember the last time we talked about this , it was sort of up in the air whether they were going to be taking , people 's features and then running them or they were gonna give the system out or phd d: Yeah . professor b: Do we already have it ? phd d: Yeah , th I I guess it 's almost ready . phd e: So we 'll have to grab this over CVS or something ? phd d: It - no , it 's just downloadable from their from their web site . professor b: Cuz one of the things that might be helpful , if you 've if you 've got time in all of this is , is if if these guys are really focusing on improving , all the digit stuff , maybe and you got the front - end from them , maybe you could do the runs for the phd e: OK . professor b: and and , you know , iron out hassles that that you have to , tweak Joe about or whatever , phd e: Sure . professor b: S phd d: So I 'll point you to the web site and the mails corresponding . So I phd e: And it but it 's not ready yet , the system ? phd d: I I think they are still , tuning something on that . So they 're like , d they 're varying different parameters like the insertion penalty and other stuff , and then seeing what 's the performance . phd e: Are those going to be parameters that are frozen , nobody can change ? Or ? phd d: w I guess there is , time during which people are gonna make suggestions . phd d: So these sugges these this , period during which people are gonna make suggestions is to know whether it is actually biased towards any set of features or professor b: Yeah , so I th th certainly the thing that I would want to know about is whether we get really hurt , on in insertion penalty , language model , scaling , sorts of things . professor b: in which case , H Hari or Hynek will need to , you know , push the case more about about this . phd e: And we may be able to revisit this idea about , you know , somehow modifying our features to work with professor b: Yes .  , some of that may be , a last minute rush thing because if the if our features are changing  . Yeah , the other thing is that even though it 's months away , it 's starting to seem to me now like November fifteenth is right around the corner . And , if they haven't decided things like this , like what the parameters are gonna be for this , when " deciding " is not just somebody deciding .  , in fact there should be some understanding behind the , deciding , which means some experiments and and so forth . phd e: So wha what 's the significance of November fifteenth ? professor b: That 's when the evaluation is . So , yeah , so after But , you know , they may even decide in the end to push it off . But , due to other reasons , like some people are going away , I 'm I 'm hoping it 's not pushed off for a l a long while . There 's there 's not anybody OGI currently who 's who 's , working with this and and phd e: Is is this part of the evaluation just a small part , or ho how important is this to the overall ? professor b: I I think it 's it 's ,  it depends how badly you do . phd d: b phd e: This is one of those things that will be debated afterwards ? professor b: Yeah . Well , it 's it 's Conceptually , it my impression , again , you guys correct me if I 'm wrong , but my impression is that , they want it as a double check . That you haven't come across you haven't invented features which are actually gonna do badly for a a significantly different task , particularly one with larger vocabulary . professor b: the truth is , most of the applications they 're looking at are pretty small vocabulary . phd e: Seems to me that if it 's a double check , they should give you a one or a zero . Y you passed the threshold or you didn't pass the threshold , and they shouldn't even care about what the score is .  , but in in the current thing , for instance , where you have this well - matched , moderately - matched , and and mis highly - mismatched , the emphasis is somewhat on the on the well - matched , but it 's only a a marginal , phd e: Yeah . professor b: right ? It 's like forty , thirty - five , twenty - five , or something like that . So you still if you were way , way off on the highly - mismatched , it would have a big effect . So again , if you 're if you get If it doesn't help you much , for noisy versions of this of large vocabulary data , then , you know , it may not hurt you that much . professor b: But if it if you don't if it doesn't help you much at all , or to put it another way , if it helps some people a lot more than it helps other people , if their strategies do , then phd e:   So is this ,  ?  , Guenter was putting a bunch of Wall Street Journal data on our disks . phd e: So that 's the data that we 'll be running on ? professor b: Yeah . professor b: well there 's training and test , right ? phd e: I I guess , I 'm not sure . professor b: No , if it 's like the other things , there 's there 's data for training the H M Ms and and data for testing it . phd e: I just professor b: So I wouldn't So it it 's phd e: OK . But I think it 's trained on clean and Is it trained on clean and and test on ? phd d: The Wall Street ? professor b: Yeah . It 's training on a range between ten and twenty DB , I think , and testing between five and fifteen . phd d: It 's ,  It 's like a medium medium - mismatch condition , sort of . phd a: and So the noise is There is a range of different noises also  which are selected randomly and added randomly , to the files . When did they estimate that they would have that system available for download ? phd d: I guess I guess one some preliminary version is already there . phd e: Oh , so there 's w something you can download to just learn ? phd d: Yeah , it 's already there . phd e: OK , phd d: But they 're actually parallel - y doing some modifications also , I think . phd d: So I guess the f final system will be frozen by middle of , like , one more week maybe . grad c: Is this their , SVM recognizer ? phd d: No , it 's just a straightforward  . professor b: You know , their their They have a lot of options in their recognizer and and the SVM is one of the things they 've done with it , but it 's not their more standard thing . phd d: Yeah , this is a g yeah , this i professor b: what ? phd d: yeah . phd e: So , just so that I understand , they 're providing scripts and everything so that basically , you you push a button and it does training , and then it does test , and everything ? Is that the idea ? phd d: I I I think yeah , I I guess something like that . phd d: is what Do they provide all the scripts , everything , and then Just , phd e: I see . Somehow yo there 's hooks to put your features in and phd d: ju Yeah , I th I think . In fact , if you look into it a little bit , it might be reasonable You know Joe , right ? Yeah . professor b: Just to sort of ask him about the issue of , different features having different kinds of , scaling characteristics and so on . So that , you know , w w possibly having entirely different optimal values for for the usual twiddle factors and what 's what 's the plan about that ? phd e: OK . phd d: So sh shall we , like , add Chuck also to the mailing lists ? It may be better , in that case if he 's going to professor b: Yeah . phd d: Yeah , I guess maybe Hari or Hynek , one of them , has to send a mail to Joe . phd d: Well , yeah , to add or maybe wh phd e: I I know him really well . professor b: Do you have Hari 's ,  ? phd e: I have Hari 's professor b: Yeah , so maybe just CC Hari and say that you 've just been asked to handle the large vocabulary part here , and , you know , phd e: OK . Why don't you just ask Joe but CC Hari , and then in the note say , " Hari , hopefully this is OK with you " . professor b: That way you can get started asking Joe quickly while he 's while he 's maybe still , you know , putting in nails and screws and  Yeah . phd d: And there is an , archive of all the mails that has been gon that has gone , between these people among these people . phd d: So , like like , it 's , like professor b: Have you thought about how long would be  , most useful for you to go up to OGI ? phd a: I don't know ,  . professor b: Oh , so you 're you 're imagining more that you would come back here first for a while and then and then go up there ? phd a: I professor b: it 's to you . professor b: I ju you guys are Well , y anyway , you don't have to decide this second but thi think about it about what what you would think would be the the best way to work it . And , I was looking at some of the work that , Sangita was doing on these TRAPS things . So she has ,  she has temporal patterns for , a certain set of phonemes , from from TIMIT , right ? the most common phonemes .  , so she has ,  a TRAP for each one of the phonemes , times fifteen , for each of the fifteen critical bands . And , she does this agglomerative hierarchical clustering which which basically , is a clustering algorithm that , starts with many , many , many different points many different clusters  , corresponding to the number of data , patterns that you have in the data . phd e: And then you can pick , values anywhere along that tree to fix your set of clusters . grad c: Right , usually it 's when ,  when the sol similarity measures , don't go down as much . And what she found was , sh  , was there were five broad ,  broad categories , corresponding to , things like , fricatives and , vocalic , and , stops .  , and , I was thinking about ways to to generalize this because w you 're it 's sort of like a it 's not a completely automatic way of clustering , because yo beforehand you have these these TRAPS and you 're saying that that these frames correspond to this particular phoneme .  , and that 's that 's constraining your your clustering to to the set of phonemes that you already have .  , whereas maybe we want to just take take a look at , arbitrary windows in time , of varying length , and cluster those . grad c: And I 'm thinking if we if we do that , then we would probably , at some point in the clustering algorithm find that we 've clustered things like , OK , thi this is a transition , this is a relatively stable stable point . grad c: and I 'm hoping to find other things of of similarity and maybe use these things as the intermediate ,  intermediate categories that , I 'll later classify . F  , I 'm professor b: Cuz that 's what you 're gonna be using , right ? grad c: Yeah , yeah . I I haven't exactly figured out , the exact details for that but , the the representation of the data that I was thinking of , was using , critical band , energies , over different lengths of time . professor b: Yeah , it seems somehow that needs th  , there 's a couple things that I wonder about with this . professor b: if you 're going for this sort of thing where you have  , little detectors that are looking at narrow bands , then what you 're going to be looking for should be some category that you can find with the narrow bands .  , and then the other thing , is that I wonder about with it , and and don't take this in the wrong way , like I I know what I 'm doing or anything , grad c: Right . professor b: the sort of standard answer about this sort of thing is that if you 're trying to find the right system in some sense , whether you 're trying by categories or or parameters  , and your goal is discrimination , then having choices based on discrimination as opposed to , unsupervised nearness of things , is actually better . professor b: and I don't know if that  , since you 're dealing with issues of robustness , you know , maybe maybe this isn't right , but it 'd be something I 'd be concerned about . Because , for instance , you can imagine , i i if you remember from from ,  from your your quals , John Ohala saying that , " buh " and " puh " differed , not really cuz of voicing but because of aspiration . professor b: So , if you looked if you were doing some coarse clustering , you probably would put those two sounds together . And yet , I would gue I would guess that many of your recognition errors were coming from , pfft , screwing up on this distinction . professor b: So , in fact , it 's a little hard because recognizers , to first order , sort of work . And the reasons we 're doing the things we 're doing is because they don't work as well as we 'd like . And since they sort of work , it means that they are already doing if you go and take any recognizer that 's already out there and you say , " how well is it distinguishing between schwas and stops ? " grad c:   professor b: Boy , I bet they 're all doing nearly perfectly on this , right ? grad c:   professor b: So these these big categories that differ in huge obvious ways , we already know how to do . So , what are we bringing to the party ?  , in fact what we wanna do is have something that , particularly in the presence of noise , is better at distinguishing between , categories that are actually close to one another , and hence , would probably be clustered together .  , I understand that there 's this other constraint that you 're considering , is that you wanna have categories that ,  that would be straightforward for , say , a human being to mark if you had manual annotation . But I think it 's also essential that you wanna look at what are the confusions that you 're making and how can you come up with , categories that , can clarify these confusions . professor b: So , the standard sort of way of doing that is take a look at the algorithms you 're looking at , but then throw in some discriminative aspect to it .  Y y this is more like , you know , how does LDA differ from PCA ?  , they 're the same sort of thing . professor b: But , you know and and , this is a little harder because you 're not just trying to find parameters .  , well W actually , you stopped thinking about it for a long time , but you used to think about it a lot . phd e: I guess I don't I don't  , it 's not clear to me how to reconcile , you know , what you 're saying , which I think is right , with the way I 've been looking at it . But it seems to me that the desire the desirable feature to have is something that , is bottom - up . phd e: And and so I guess what I don't understand is how to do that and still be discriminative , because to be discriminative you have to have categories and the only categories that we know of to use are sort of these human human sig significant categories that are significant to humans , like phonemes , things like that . professor b: Well , here 's a here 's a ,  Here 's a generic and possibly useless thought , which is , what do you really  , in a sense the only s s systems that make sense , are ones that that have something from top - down in th in them . Right ? Because if e even the smallest organism that 's trying to learn to do anything , if it doesn't have any kind of reward for doing or penal penalty for doing anything , then it 's just going to behave randomly . professor b: So whether you 're talking about something being learned through evolution or being learned through experience , it 's gotta have something come down to it that gives its reward or , you know , at least some reinforcement learning , phd e: Right . professor b: right ? phd e: So the question is , how far down ? professor b: And phd e: We could stop at words , but we don't , right ? We go all the way down to phonemes . professor b: Right , but I me I I think that maybe in some ways part of the difficulty is is trying to deal with the with these phonemes . You know , and and and i it 's almost like you want categories if if our if our , metric of of goodness , i if our phd e:   professor b: correction if our metric of badness is word error rate then , maybe we should be looking at words . professor b: for for for very nice , reasons we 've looked for a while at syllables , and they have a lot of good properties , but i i i if you go all the way to words , that 's really  , d w In many applications you wanna go further . You wanna go to concepts or something , or have have have concepts , actions , this sort of thing . And and phd e: Yeah , so the common right , the common wisdom is you can't do words because there 's too many of them , right ? So you have to have some smaller set that you can use , and and so everybody goes to phonemes . But the problem is that we we build models of words in terms of phonemes and these models are are really cartoon - ish , right ? So when you look at conversational speech , for example , you don't see the phonemes that you that you have in your word models . See , so her here 's maybe where If the issue is that we 're trying to come up with , some sort of intermediate categories which will then be useful for later stuff , then maybe it doesn't matter that we can't have enough phd e:   professor b: what you wanna do is is build up these categories that are that are best for word recognition . professor b: And and somehow if that 's built into the loop of what the categories  , we do this every day in this very gross way of of running o a thousand experiments phd e:  Right . In some ways it 's really not a bad bad thing to do because it tells you in fact how your adjustments at the very low level affect the the final goal . professor b: so maybe there 's a way to even put that in in a much more automatic way , phd e: Right . professor b: where you take , you know , something about the error at the level of the word or some other it could be syllable but in some large unit , phd e:  - huh . professor b: and  yeah , you may not have word models , you have phone models , whatever , but you sort of don't worry about that , and just somehow feed it back through . professor b: You know , so that 's , wh what I called a useless comments because I 'm not really telling you how to do it . But  , it 's a it 's it 's , you know it phd e: No , but I think the important part in there is that , you know , if you want to be discriminative , you have to have  , you know , categories . If you can put the words in to the loop somehow for determining goodness of your sets of clusters  professor b: Now , that being said , I think that that if you have something that is ,  i Once you start dealing with spontaneous speech , all the things you 're saying are are really true . professor b: If you have read speech that 's been manually annotated , like TIMIT , then , you know , i i you the phones are gonna be right , actually , for the most part . Yeah , professor b: So so , it doesn't really hurt them to to do that , to put in discrimination at that level . professor b: if you go to spontaneous speech then it 's it 's trickier and and and , the phones are  , you know , it 's gonna be based on bad pronunciation models that you have of phd e:  professor b: and ,  And it won't allow for the overlapping phenomenon phd e: Mmm . So it 's almost like there 's this mechanism that we have that , you know , when when we 're hearing read speech and all the phonemes are there you know , we we deal with that , but but when we go to conversational , and then all of a sudden not all the phonemes are there , it doesn't really matter that much to us as humans because we have some kind of mechanism that allows for these word models , whatever those models are , to be munged , you know , and and it doesn't really hurt , and I 'm not sure how how to build that in . professor b: Yeah , I guess the other thing i is is to think of a little bit  , we when y when you start looking at these kind of results I think it usually is is pretty intuitive , but start looking at  , what are the kinds of confusions that you do make , you know , between words if you want or or or , even phones in in in in read speech , say , when there is noise . You know , so is it more across place or more across manner ? Or is it cor you know , is it ? grad c:   professor b: I know one thing that happens is that you you you , you lose the , low energy phones . And if that if that is if it  , if that turns it into another word or or different you know , or another pair of words or something , then it 's more likely to happen . But , I don't know , I w I would I would guess that you 'd grad c:   Anyway , that 's phd e: I think part of the difficulty is that a l a lot of the robustness that we have is probably coming from a much higher level . phd e: You know , we understand the context of the situation when we 're having a conversation . And so if there 's noise in there , you know , our brain fills in and imagines what what should be there . professor b: but  , even if you do  , diagnostic rhyme test kind of things , you know , where there really isn't an any information like that , people are still better in noise than they than they are in in , than the machines are . If we 're not working on that then we should work on something else and improve it , but especially if it looks like the potential is there