three two three four seven six five five three one six two four one six seven seven eight nine zero nine four zero zero three zero one five eight one seven three five three two six eight zero three six two four three zero seven four five zero six nine four seven four eight five seven nine six one five O seven eight O two zero nine six zero four zero zero one two  Yeah , you don't actually n need to say the name . grad c: OK , this is Barry Chen and I am reading transcript professor f: That 'll probably be bleeped out . professor f:   not that there 's anything defamatory about  eight five seven or or anything , but grad c: OK .  so here 's what I have for I I was just jotting down things I think th w that we should do today .  This is what I have for an agenda so far  , We should talk a little bit about the plans for the  the field trip next week .  a number of us are doing a field trip to   OGI And  mostly  First though about the logistics for it . professor f: in and kind of go around see what people have been doing talk about that , a r progress report . If we find some holes in some things that that someone could use some help with , he 's he 's volunteering to help . So , and  , then  , talk a little bit about about disks and resource resource issues that that 's starting to get worked out . And then , anything else anybody has that isn't in that list ?  grad d: I was just wondering , does this mean the battery 's dying and I should change it ? professor f:  I think that means the battery 's O K . professor f: d do you grad d: Oh OK , so th phd a: Yeah , that 's good .   I I it was it was kind of my bright idea to have us take a plane that leaves at seven twenty in the morning .  this is  The reason I did it  was because otherwise for those of us who have to come back the same day it is really not much of a of a visit .  So  the issue is how how how would we ever accomplish that ?  what what what part of town do you live in ? grad c: I live in , the corner of campus . OK , so would it be easier those of you who are not , you know , used to this area , it can be very tricky to get to the airport at at  , you know , six thirty . Would it be easier for you if you came here and I drove you ? Yeah ? Yeah , yeah , OK . It 'll take it it it won't be bad traffic that time of day and and  phd a: I guess once you get past the bridge that that would be the worst . professor f: And then Martin Luther King to nine - eighty to eight - eighty , phd a: Yeah . phd a: Oh , I professor f: So that leaves us fifty minutes before the plane it 'll just yeah . So Great , OK so that 'll It 's  , it 's still not going to be really easy but well Particularly for for  for Barry and me , we 're not we 're not staying overnight so we don't need to bring anything particularly except for  a pad of paper and So , and ,  you , two have to bring a little bit grad c: OK . professor f: but  you know , don't don't bring a footlocker and we 'll be OK So .  I 'll I 'll I 'll I 'll give you my phone number , If I 'm not here for a few m after a few minutes then grad c: Wake you up . I just ,  it for me it just means getting up a half an hour earlier than I usually do .  , I I I figured maybe wait on the potential goals for the meeting  until we talk about wh what 's been going on . phd g: So , it means that  , well , it is , a digit French database of microphone speech , downsampled to eight kilohertz and I 've added noise to one part , with the actually the Aurora - two noises . professor f: OK  , So the HTK base lines so this is using mel cepstra and so on , or ? Yeah . professor f: And again , I guess the p the plan is , to  then given this What 's the plan again ? phd g: The plan with these data ? professor f: With So So Does i Just remind me of what what you were going to do with the what what what what 's y You just described what you 've been doing . phd g:  we actually we want to , mmm , analyze three dimensions , the feature dimension , the training data dimension , and the test data dimension . So we have the  , TI - digit task , the Italian task , the French task and the Finnish task . professor f: Yeah ? phd g: So we have numbers with  systems   neural networks trained on the task data . And then to have systems with neural networks trained on , data from the same language , if possible , with , well , using a more generic database , which is phonetically phonetically balanced , and . professor f: So - so we had talked I guess we had talked at one point about maybe , the language ID corpus ? phd g: Yeah . professor f: Is that a possibility for that ? phd g: Ye -  Yeah , but ,  these corpus , w w there is a CallHome and a CallFriend also , The CallFriend is for language ind identification . This could be a a problem for Why ? Because  , the the SpeechDat databases are not telephone speech . That 's really funny isn't it ?  cuz th this whole thing is for developing new standards for the telephone . phd g: Yeah , but the the idea is to compute the feature before the before sending them to the Well , you don't do not send speech , you send features , computed on th the the device , professor f:   professor f: Oh I see , so your point is that it 's it 's it 's  the features are computed locally , and so they aren't necessarily telephone bandwidth ,  or telephone distortions . phd a: Did you happen to find out anything about the OGI multilingual database ? professor f: Yeah , that 's wh that 's wh that 's what I meant . phd g: Yeah , it 's professor f: I said @ @ , there 's there 's there 's an OGI language ID , not the not the ,  the CallFriend is a is a , LDC w thing , right ? phd g: Yea - Yeah , there are also two other databases . One they call the multi - language database , and another one is a twenty - two language , something like that . professor f: But I 'm not sure phd g: So professor f: we ' r e e The bandwidth shouldn't be such an issue right ? Because e e this is downsampled and and filtered , right ? So it 's just the fact that it 's not telephone .  some of this stuff 's recorded in the car , and some of it 's  there 's there 's many different acoustic differences .  , unless we 're going to include a bunch of car recordings in the in the training database , I 'm not sure if it 's completely rules it out phd g: Yeah . professor f: if our if we if our major goal is to have phonetic context and you figure that there 's gonna be a mismatch in acoustic conditions does it make it much worse f to sort of add another mismatch , if you will . professor f: i i I I guess the question is how important is it to for us to get multiple languages  , in there . Well , actually , for the moment if we w do not want to use these phone databases , we we already have  English , Spanish and French  , with microphone speech . professor f: So that 's what you 're thinking of using is sort of the multi the equivalent of the multiple ? phd g: Well . phd g: So w f for for  Italian , which is close to Spanish , French and , i i  , TI - digits we have both  , digits training data and also more general training data . professor f: Well , we also have this Broadcast News that we were talking about taking off the disk , which is is microphone data for for English . phd g: Yeah , and perhaps ,  We were thinking that perhaps the cross - language issue is not , so big of a issue . u So that th Well , the the guy who has to develop an application with one language can use the net trained o on that language , or a generic net , professor f: depen it depen it depends how you mean " using the net " . phd g: but not trained on a professor f: So , if you 're talking about for producing these discriminative features that we 're talking about you can't do that . But if we say , " No , you have to have a different feature set for each language , " I think this is ver gonna be very bad .  , in principle ,  conceptually , it 's sort of like they want a re @ @ well , they want a replacement for mel cepstra . professor f: So , we say " OK , this is the year two thousand , we 've got something much better than mel cepstra . " OK ? And so we give them these gobbledy - gook features but these gobbledy - gook features are supposed to be good for any language . professor f: Cuz you don't know who 's gonna call , and you know ,  so it 's it 's it 's ,  how do you know what language it is ? Somebody picks up the phone . Someone picks up the phone , right ? phd g: Well , I chh professor f: And and he he picks up the ph phd g: Yeah , but the the application is there is a target language for the application . professor f: you talk on the phone , phd g: Yeah ? professor f: and it sends features out . If it 's th in the phone , but professor f: But that 's the image that they have . phd g: well , it that that could be th at the server 's side , professor f: It could be , phd g: and , well . professor f: but that 's the image they have , right ? So that 's that 's  , one could argue all over the place about how things really will be in ten years . But the particular image that the cellular industry has right now is that it 's distributed speech recognition , where the , probabilistic part , and and s semantics and so forth are all on the servers , and you compute features of the  , on the phone . We might might or might not agree that that 's the way it will be in ten years , but that 's that 's that 's what they 're asking for . Now , it 's the OGI , folks ' perspective right now that probably that 's not the biggest deal . And they may very well be right , but I I was hoping we could just do a test and determine if that was true . Maybe maybe we have a couple languages in the training set and that gives us enough breadth  , that that that the rest doesn't matter .  , the other thing is , this notion of training to  which I I guess they 're starting to look at up there , training to something more like articulatory features .  , and if you have something that 's just good for distinguishing different articulatory features that should just be good across , you know , a wide range of languages . professor f: but Yeah , so I don't th I know unfortunately I don't I see what you 're comi where you 're coming from , I think , but I don't think we can ignore it .  , tr for instance training on English and testing on Italian , or Or we can train or else , can we train a net on , a range of languages and which can include the test the test @ @ the target language , grad c: Test on an unseen . phd g: or professor f: Yeah , so , there 's there 's ,  This is complex . So , ultimately , as I was saying , I think it doesn't fit within their image that you switch nets based on language . professor f: from a purist 's standpoint it 'd be nice not to because then you can say when because surely someone is going to say at some point , " OK , so you put in the German and the Finnish . professor f: now , what do you do , when somebody has Portuguese ? " you know ?  , and  , however , you aren't it isn't actually a constraint in this evaluation . So I would say if it looks like there 's a big difference to put it in , then we 'd make note of it , and then we probably put in the other , because we have so many other problems in trying to get things to work well here that that , you know , it 's not so bad as long as we we note it and say , " Look , we did do this " . phd g: Mmm ? phd a: And so , ideally , what you 'd wanna do is you 'd wanna run it with and without the target language and the training set for a wide range of languages . phd a: And that way you can say , " Well , " you know , " we 're gonna build it for what we think are the most common ones " , professor f: Yeah . phd a: but if that somebody uses it with a different language , you know , " here 's what 's you 're l here 's what 's likely to happen . " professor f: Yeah , cuz the truth is , is that it 's it 's not like there are  , al although there are thousands of languages , from  , the point of view of cellular companies , there aren't . professor f: There 's you know , there 's fifty or something , you know ? So , an and they aren't you know , with the exception of Finnish , which I guess it 's pretty different from most most things . I guess Finnish is a is is a little bit like Hungarian , supposedly , right ? phd a: I don't know anything about Finnish . professor f: Or is I think well , I kn oh , well I know that H  , H  , I 'm not a linguist , but I guess Hungarian and Finnish and one of the one of the languages from the former Soviet Union are in this sort of same family . professor f: But they 're just these , you know ,  countries that are pretty far apart from one another , have I guess , people rode in on horses and brought their phd a:  . I re - installed , HTK , the free version , so , everybody 's now using three point O , which is the same version that , OGI is using . And , so we 've been talking about this this , cube thing , and it 's beginning more and more looking like the , the Borge cube thing .  , but I I 'm Am I professor f: So are are you going to be assimilated ? phd a: Resistance is futile .  , the the stuff that we 've been working on with TIMIT , trying to get a ,  a labels file so we can , train up a train up a net on TIMIT and test , the difference between this net trained on TIMIT and a net trained on digits alone . professor f: And again , when y just to clarify , when you 're talking about training up a net , you 're talking about training up a net for a tandem approach ? grad c: Yeah , yeah . professor f: And and the inputs are PLP and delta and that sort of thing , grad c: Well , the inputs are one dimension of the cube , professor f: or ? grad c: which , we 've talked about it being , PLP , M F C Cs , J - JRASTA , JRASTA - LDA phd g:  . professor f: Yeah , but your initial things you 're making one choice there , grad c: Yeah , professor f: right ? grad c: right . professor f: Which is PLP , or something ? grad c: I I haven't I haven't decided on on the initial thing .  , so so you take PLP and you you , do it  , you you , use HTK with it with the transformed features using a neural net that 's trained . professor f: And that 's the and , and th and then the testing would be these other things which which which might be foreign language .  , those listening to this will not have a picture either , so , I guess I 'm I 'm not any worse off . It sounds s I I get I think I get the general idea of it , grad c: Yeah , yeah , professor f: yeah . phd a: So , when you said that you were getting the labels for TIMIT , are y what do you mean by that ? grad c: b May   Oh , I 'm just I 'm just , transforming them from the , the standard TIMIT transcriptions into into a nice long huge P - file to do training . Were the digits , hand - labeled for phones ? grad c: the the digits phd a: Or were they those labels automatically derived ? grad c: Oh yeah , those were those were automatically derived by by Dan using , embedded embedded training and alignment . phd a: I was just wondering because that test you 're t grad c:  - huh . phd a: I I think you 're doing this test because you want to determine whether or not , having s general speech performs as well as having specific speech . professor f: Well , especially when you go over the different languages again , because you 'd the different languages have different words for the different digits , phd a:   And I was professor f: so it 's phd a: yeah , so I was just wondering if the fact that TIMIT you 're using the hand - labeled stuff from TIMIT might be confuse the results that you get . phd a: Right , but if it 's better , it may be better because it was hand - labeled . professor f: you know , I I I guess I 'm sounding cavalier , but  , I think the point is you have , a bunch of labels and and they 're han hand  hand - marked . It would be another interesting scientific question to ask , " Is it because it 's a broad source or because it was , you know , carefully ? " phd a:   And that 's something you could ask , but given limited time , I think the main thing is if it 's a better thing for going across languages on this training tandem system , phd a: Yeah . professor f: then it 's probably phd a: What about the differences in the phone sets ? grad c: between languages ? phd a: No , between TIMIT and the the digits . Well , there 's a mapping from the sixty - one phonemes in TIMIT to to fifty - six , the ICSI fifty - six . grad c: And then the digits phonemes , there 's about twenty twenty - two or twenty - four of them ? Is that right ? phd a: Out of that fifty - six ? phd g: Yep . phd g: But , actually , the issue of phoneti phon  phone phoneme mappings will arise when we will do severa use several languages phd e: Yeah . phd g: because you Well , some phonemes are not , in every languages , and So we plan to develop a subset of the phonemes , that includes , all the phonemes of our training languages , phd a:   phd e: SAMPA phone ? For English  American English , and the the the language who have more phone are the English . But n for example , in Spain , the Spanish have several phone that d doesn't appear in the E English and we thought to complete . But for that , it needs we must r h do a lot of work because we need to generate new tran transcription for the database that we have . phd b: Other than the language , is there a reason not to use the TIMIT phone set ? Cuz it 's larger ? As opposed to the ICSI phone set ? grad c: Oh , you mean why map the sixty - one to the fifty - six ? phd b: Yeah . I have professor f: I forget if that happened starting with you , or was it o or if it was Eric , afterwards who did that . phd a: Yeah , and I think some of them , they were making distinctions between silence at the end and silence at the beginning , when really they 're both silence . phd a: I th I think it was things like that that got it mapped down to fifty - six . professor f: Yeah , especially in a system like ours , which is a discriminative system . And the ones that are gone , I think are I think there was they also in TIMIT had like a glottal stop , which was basically a short period of silence , phd b:   phd b: Well , we have that now , too , right ? phd a: I don't know . professor f: i It 's actually pretty common that a lot of the recognition systems people use have things like like , say thirty - nine , phone symbols , right ?  , and then they get the variety by by bringing in the context , the phonetic context . What there 's Can you describe what what 's on the cube ? grad c: Yeah , w I th I think that 's a good idea professor f:  grad c: to to talk about the whole cube professor f: Yeah , yeah .  , do you wanna do it ? professor f: OK , so even even though the meeting recorder doesn't doesn't ,  and since you 're not running a video camera we won't get this , but if you use a board it 'll help us anyway . professor f: but you 've got the wireless on , grad c: Yeah , I have the wireless . OK , well , professor f: he can't , actually , but grad c: s basically , the the cube will have three dimensions . So the the training for HTK is always that 's always set up for the individual test , right ? That there 's some training data and some test data . And , yeah , the training for the HTK models is always , fixed for whatever language you 're testing on . So , then I think it 's probably instructive to go and and and show you the features that we were talking about . grad c: Yeah , just the multi - band features , right ? phd g: And grad c: Yeah . phd a: What about mel cepstrum ? Or is that grad c: Oh ,  phd a: you don't include that because it 's part of the base or something ? phd e: Yeah databases . professor f: Well , y you do have a baseline system that 's m that 's mel cepstra , phd e: Yeah .  at least at least conceptually , you know , it doesn't meant you actually have to do it , phd g: Yeah . phd a: It 'd be an interesting test just to have just to do MFCC with the neural net phd e: Without the phd a: and everything else the same . D Because I think that for a bunch of their experiments they used , mel cepstra , actually . phd a: Is that Was that distributed with Aurora , or ? grad c: One L or two L 's ? phd a: Where did that ? professor f: The newer one . And , oh yeah , and professor f: Is it French French or Belgian French ? There 's a phd g: It 's , French French . Yeah , Herve always insists that Belgian is i is absolutely pure French , has nothing to do with but he says those those those Parisians talk funny . grad c: right ? Spanish Oh , Spanish stories ? phd e: Albayzin is the name . phd a: What about TI - digits ? grad c: TI - digits  all these Aurora f d data p data is from is derived from TI - digits . grad c: basically , they they corrupted it with , different kinds of noises at different SNR levels . professor f: y And I think Stephane was saying there 's there 's some broader s material in the French also ? phd g: Yeah , we cou we could use grad c: OK . phd b: Did the Aurora people actually corrupt it themselves , or just specify the signal and the signal - t grad c: They they corrupted it , themselves , phd b: OK . grad c: but they also included the the noise files for us , right ? Or phd g: Yeah . professor f: I 'm just curious , Carmen  , I couldn't tell if you were joking or i Is it is it Mexican Spanish , phd e: No no no no . phd g: Yeah , the No , the French is f yeah , from , Paris , grad c: Oh , from Paris , OK . And , with within the training corporas  , we 're , thinking about , training with noise . So , incorporating the same kinds of noises that , Aurora is in incorporating in their ,  in their training corpus .  , I don't think we we 're given the ,  the unseen noise conditions , though , right ? professor f: I think what they were saying was that , for this next test there 's gonna be some of the cases where they have the same type of noise as you were given before hand and some cases where you 're not . professor f: So , presumably , that 'll be part of the topic of analysis of the the test results , is how well you do when it 's matching noise and how well you do where it 's not . grad c: So , I guess we can't train on on the the unseen noise conditions .  , i i i i it does seem to me that a lot of times when you train with something that 's at least a little bit noisy it can it can help you out in other kinds of noise even if it 's not matching just because there 's some more variance that you 've built into things . professor f: exactly how well it will work will depend on how near it is to what you had ahead of time . professor f: and then your testing corpus ? grad c: the testing corporas are , just , the same ones as Aurora testing . grad c: we ' r we 're gonna get German , right ? Ge - At the final test will have German . professor f: Well , so , yeah , the final test , on a guess , is supposed to be German and Danish , phd g: yeah . professor f: Oh , there 's a there 's Spanish testing in the Aurora ? phd g: not yet , but , yeah , e phd e: Yeah , it 's preparing . phd g: and , well , according to Hynek it will be we will have this at the end of November , or  . phd g: Yeah professor f: So that 's , three hundred and forty - three , different systems that are going to be developed . grad d: What a what about noise conditions ? professor f: What ? grad d: w Don't we need to put in the column for noise conditions ? professor f: Are you just trying to be difficult ? grad d: No , I just don't understand . grad c: Well , th  , when when I put these testings on there , I 'm assumi professor f: I 'm just kidding . And they 're all they 're all gonna be test tested , with one training of the HTK system . grad d: And do we do all our training on clean data ? grad c: no , no , phd e: Also , we can clean that . grad c: we 're we 're gonna be , training on the noise files that we do have . professor f: So ,  Yeah , so I guess the question is how long does it take to do a a training ?  , it 's not totally crazy t  , these are a lot of these are built - in things and we know we have programs that compute PLP , we have MSG , we have JRA you know , a lot of these things will just kind of happen , won't take  a huge amount of development , it 's just trying it out . professor f: But how how long does it take , do we think , for one of these trainings ? grad c: That 's a good question .  , cuz , so , for instance , I think the major advantage of MSG grad c: Oh ! professor f: Yeah , grad c: Och ! professor f: good point . A major advantage of MSG , I see , th that we 've seen in the past is combined with PLP . grad c: Now , this is turning into a four - dimensional cube ? phd a: Well , you just select multiple things on the one dimension . professor f: Yeah , so , you don't wanna ,  Let 's see , seven choose two would be , twenty - one different combinations . phd b: It 's not a complete set of combinations , though , professor f: Probably phd b: right ? It 's not a complete set of combinations , though , professor f: What ? phd b: right ? grad c: No . Yeah , there 's grad c: That would be professor f: yeah , so PLP and MSG I think we definitely wanna try cuz we 've had a lot of good experience with putting those together . phd a: When you do that , you 're increasing the size of the inputs to the net . Do you have to reduce the hidden layer , or something ? professor f: Well , so  , so i it doesn't increase the number of trainings . phd a: No , no , I 'm I 'm just wondering about number of parameters in the net . Do you have to worry about keeping that the same , or ? professor f: I don't think so . phd b: There 's a computation limit , though , isn't there ? professor f: Yeah , it 's just more compu Excuse me ? phd b: Isn't there like a limit on the computation load , or d latency , or something like that for Aurora task ? professor f: Oh yeah , we haven't talked about any of that at all , have we ? grad c: No . What it is is that there 's there 's ,  it 's just penalty , you know ? That that if you 're using , a megabyte , then they 'll say that 's very nice , but , of course , it will never go on a cheap cell phone .  , and , expensive cell phones , exa expensive hand - helds , and so forth , are gonna have lots of memory . So it 's just that , these people see the the cheap cell phones as being still the biggest market , so . But , yeah , I was just realizing that , actually , it doesn't explode out ,  It 's not really two to the seventh . But it 's but but i i it doesn't really explode out the number of trainings cuz these were all trained individually . Right ? So , if you have all of these nets trained some place , then , you can combine their outputs and do the KL transformation and so forth grad c:   phd a: But wh what about a net that 's trained on multiple languages , though ? professor f: Well , you gotta do the KL transformation , phd g: Eight y professor f: but phd a: Is that just separate nets for each language then combined , or is that actually one net trained on ? phd e: Necessary to put in . Right ? phd g: So , in the broader training corpus we can we can use , the three , or , a combination of of two two languages . professor f: Yeah , so , I guess the first thing is if w if we know how much a how long a a training takes , if we can train up all these these combinations , then we can start working on testing of them individually , and in combination . professor f: Because the putting them in combination , I think , is not as much computationally as the r training of the nets in the first place . But there is the testing also , which implies training , the HTK models phd e: The the model the HTK model . professor f: How long does it take for an , HTK training ? phd g: It 's around six hours , I think . professor f: No , I 'm sorry , ru running on what machine ? phd e: Ravioli . phd g: Yeah , I I think it 's - it 's - it 's not so long because , well , the TI - digits test data is about ,  how many hours ?  , th  , thirty hours of speech , I think , professor f: It 's a few hours . professor f: so , clearly , there there 's no way we can even begin to do an any significant amount here unless we use multiple machines . professor f: Right ? So so w we  there 's plenty of machines here and they 're n they 're often not in in a great great deal of use . So , I think it 's it 's key that that the that you look at , you know , what machines are fast , what machines are used a lot  , are we still using P - make ? Is that ? grad c: Oh , I don't know how w how we would P - make this , though . professor f: Well , you have a  , once you get the basic thing set up , you have just all the  , a all these combinations , grad c: Yeah . It 's it 's let 's say it 's six hours or eight hours , or something for the training of HTK . How long is it for training of of , the neural net ? grad c: The neural net ?  . phd b: How big is the net ? phd e: For Albayzin I trained on neural network , was , one day also . professor f: And I think there there there 's I think you folks are probably go the ones using them right now . phd a: Is it faster to do it on the SPERT , or ? professor f: don't know . grad c: It 's it 's still a little faster on the professor f: Used to be . Or either Adam or or Dan did some testing and they found that the SPERT board 's still still faster . grad c: And the benefits is that , you know , you run out of SPERT and then you can do other things on your your computer , professor f:   You could set up , you know , ten different jobs , or something , to run on SPERT different SPERT boards and and have ten other jobs running on different computers . So , it 's got to take that sort of thing , or or we 're not going to get through any significant number of these . professor f:  So this is Yeah , I kind of like this because what it No grad c: OK . professor f: no , what I like about it is we we we do have a problem that we have very limited time . You know , so , with very limited time , we actually have really quite a quite a bit of computational resource available if you , you know , get a look across the institute and how little things are being used . And  , on the other hand , almost anything that really i you know , is is new , where we 're saying , " Well , let 's look at , like we were talking before about , voiced - unvoiced - silence detection features and all those sort " that 's phd e: Yeah . But if it 's new , then we have this development and and and learning process t to to go through on top of just the the all the all the work . So what I like about this is you basically have listed all the things that we already know how to do . And , you 're just saying let 's look at the outer product of all of these things and see if we can calculate them . a a Am I am I interpreting this correctly ? Is this sort of what what you 're thinking of doing in the short term ? phd g: Mmm . professor f: So so then I think it 's just the the missing piece is that you need to , you know you know , talk to talk to , Chuck , talk to , Adam , sort out about , what 's the best way to really , you know , attack this as a as a as a mass problem in terms of using many machines .  , and  , then , you know , set it up in terms of scripts and so forth , and  , in in kind o some kind of structured way .  , and , you know , when we go to , OGI next week , we can then present to them , you know , what it is that we 're doing . And , we can pull things out of this list that we think they are doing sufficiently , grad c: Mmm . professor f: that , you know , we 're not we won't be contributing that much . phd b: How big are the nets you 're using ? grad c: for the for nets trained on digits , we have been using , four hundred order hidden units . And , for the broader class nets we 're we 're going to increase that because the , the digits nets only correspond to about twenty phonemes .  , w we 're gonna professor f: Oh , it 's not actually broader class , it 's actually finer class , but you mean y You mean more classes . professor f: Carmen , did you do you have something else to add ? We you haven't talked too much , and phd e: D I begin to work with the Italian database to nnn , to with the f front - end and with the HTK program and the @ @ . And I trained eh , with the Spanish two neural network with PLP and with LogRASTA PLP . professor f: Well , JRASTA has the potential to do better , but it doesn't always . It 's it 's ,  instead of doing RASTA with a log , you 're doing RASTA with a log - like function that varies depending on a J parameter , which is supposed to be sensitive to the amount of noise there is . So , it 's sort of like the right transformation to do the filtering in , is dependent on how much noise there is . It 's a little complicated because once you do that , you end up in some funny domain and you end up having to do a transformation afterwards , which requires some tables . professor f: so it 's it 's it 's a little messier , there 's more ways that it can go wrong , but if if if you 're careful with it , it can do better . phd e: and I think to to to recognize the Italian digits with the neural netw Spanish neural network , and also to train another neural network with the Spanish digits , the database of Spanish digits . Was for me , n it was a difficult work last week with the labels because the the program with the label obtained that I have , the Albayzin , is different w to the label to train the neural network . professor f: I 'm sorry , phd e: I 'm sorry , professor f: I have a p I had a problem with the pronunciation . phd e: Oh , also that professor f: So , OK , so let 's start over . professor f: So , TI TIMI TIMIT 's hand - labeled , and and you 're saying about the Spanish ? phd e: The Spanish labels ? That was in different format , that the format for the em the program to train the neural network . Yeah , but n yes , because they have one program , Feacalc , but no , l LabeCut , l LabeCut , but don't doesn't , eh , include the HTK format to convert . I ask e even I ask to Dan Ellis what I can do that , and h they he say me that h he does doesn't any any s any form to to do that . And at the end , I think that with LabeCut I can transfer to ASCII format , and HTK is an ASCII format . And I m do another , one program to put ASCII format of HTK to ase ay ac ASCII format to Exceed professor f:   phd e: Actually that was complicated , professor f: So you phd e: but well , I know how we can did that do that . So it 's just usual kind of  sometimes say housekeeping , right ? To get these get these things sorted out . professor f: So it seems like there 's there 's some peculiarities of the ,  of each of these dimensions that are getting sorted out . And then , if if you work on getting the , assembly lines together , and then the the pieces sort of get ready to go into the assembly line and gradually can start , you know , start turning the crank , more or less . And , we have a lot more computational capability here than they do at OGI , so I think that i if What 's what 's great about this is it sets it up in a very systematic way , so that , once these all of these , you know , mundane but real problems get sorted out , we can just start turning the crank phd e:   professor f: and and push all of us through , and then finally figure out what 's best .  , the first thing was ,  we we actually had thought of this as sort of like ,  not not in stages , but more along the the time axis . grad c: je - je - je - je - je check out the results and and go that way . No , I 'm just saying , I 'm just thinking of it like loops , grad c:  - huh . professor f: right ? And so , y y y if you had three nested loops , that you have a choice for this , a choice for this , and a choice for that , grad c: Yeah . professor f: And , the thing is that once you get a better handle on how much you can realistically do , concurrently on different machines , different SPERTs , and so forth , and you see how long it takes on what machine and so forth , you can stand back from it and say , " OK , if we look at all these combinations we 're talking about , and combinations of combinations , and so forth , " you 'll probably find you can't do it all . professor f: OK , so then at that point , we should sort out which ones do we throw away . professor f: Which of the combinations across you know , what are the most likely ones , and And , I still think we could do a lot of them . But , probably when you include all the combinations , you 're actually talking about a thousand of them or something , and that 's probably more than we can do . And I know that , Stephane 's working from an NT machine , so his his home directory exists somewhere else . We 're over the next year or two , we 're gonna be upgrading the networks in this place , grad c:   So , it actually has reached the point where it 's a significant drag on the time for something to move the data from one place to another . professor f: So , you you don't w especially in something with repetitive computation where you 're going over it multiple times , you do don't want to have the the data that you 're working on distant from where it 's being where the computation 's being done if you can help it . Now , we are getting more disk for the central file server , which , since it 's not a computational server , would seem to be a contradiction to what I just said . But the idea is that , suppose you 're working with , this big bunch of multi multilingual databases . professor f: Then , when you 're working with something and accessing it many times , you copy the piece of it that you 're working with over to some place that 's close to where the computation is and then do all the work there . And then that way you you won't have the the network you won't be clogging the network for yourself and others . So , it 's gonna take us It may be too late for this , p precise crunch we 're in now , but , we 're ,  It 's gonna take us a couple weeks at least to get the , the amount of disk we 're gonna be getting . We 're actually gonna get , I think four more , thirty - six gigabyte drives and , put them on another another disk rack . We ran out of space on the disk rack that we had , so we 're getting another disk rack and four more drives to share between ,  primarily between this project and the Meetings Meetings Project . But , we 've put another I guess there 's another eighteen gigabytes that 's that 's in there now to help us with the immediate crunch . But , are you saying So I don't know where you 're Stephane , where you 're doing your computations . If i so , you 're on an NT machine , so you 're using some external machine phd g: Yeah , it ,  Well , to It 's Nutmeg and Mustard , I think ,  professor f: Do you know these yet ? phd g: I don't know what kind .  , are these are these , computational servers , or something ? I 'm I 've been kind of out of it . professor f: Unfortunately , these days my idea of running comput of computa doing computation is running a spread sheet . Yeah , I 'm not sure what 's available on is it you said Nutmeg and what was the other one ? phd g: Mustard . Yeah , so basically , Chuck will be the one who will be sorting out what disk needs to be where , and so on , and I 'll be the one who says , " OK , spend the money . Which , n these days , if you 're talking about scratch space , it doesn't increase the , need for backup , and , I think it 's not that big a d and the the disks themselves are not that expensive . Right now it 's phd a: What you can do , when you 're on that machine , is , just go to the slash - scratch directory , and do a DF minus K , and it 'll tell you if there 's space available . phd a: and if there is then ,  professor f: But wasn't it ,  I think Dave was saying that he preferred that people didn't put stuff in slash - scratch . It 's more putting in d s XA or XB or , phd a: Well , there 's different there , there 's professor f: right ? phd a: Right . So there 's the slash - X - whatever disks , and then there 's slash - scratch . And if it 's called " slash - scratch " , it means it 's probably an internal disk to the machine . And so that 's the kind of thing where , like if  , OK , if you don't have an NT , but you have a a a Unix workstation , and they attach an external disk , it 'll be called " slash - X - something "  , if it 's not backed up and it 'll be " slash - D - something " if it is backed up . And if it 's inside the machine on the desk , it 's called " slash - scratch " . It 's easy to unhook the external disks , put them back on the new machine , but then your slash - scratch is gone . So , you don't wanna put anything in slash - scratch that you wanna keep around for a long period of time . But if it 's a copy of , say , some data that 's on a server , you can put it on slash - scratch because , first of all it 's not backed up , and second it doesn't matter if that machine disappears and you get a new machine because you just recopy it to slash - scratch . So tha that 's why I was saying you could check slash - scratch on those on on , Mustard and and Nutmeg to see if if there 's space that you could use there . Yeah , and we do have  , yeah , so so you yeah , it 's better to have things local if you 're gonna run over them lots of times so you don't have to go to the network . professor f: Right , so es so especially if you 're right , if you 're if you 're taking some piece of the training corpus , which usually resides in where Chuck is putting it all on the on the , file server , then , yeah , it 's fine if it 's not backed up because if it g g gets wiped out or something , y  it is backed up on the other disk . phd a: Yeah , so , one of the things that I need to I 've started looking at  , is this the appropriate time to talk about the disk space stuff ? professor f: Sure . Dan David , put a new , drive onto Abbott , that 's an X disk , which means it 's not backed up . So , I 've been going through and copying data that is , you know , some kind of corpus stuff usually , that that we 've got on a CD - ROM or something , onto that new disk to free up space on other disks . We haven't deleted them off of the slash - DC disk that they 're on right now in Abbott , but we I would like to go through sit down with you about some of these other ones and see if we can move them onto , this new disk also . So , anything that that you don't need backed up , we can put on this new disk .  , but if it 's experiments and you 're creating files and things that you 're gonna need , you probably wanna have those on a disk that 's backed up , just in case something goes wrong .  So far I 've I 've copied a couple of things , but I haven't deleted anything off of the old disk to make room yet . So I I guess I 'll need to get together with you and see what data we can move onto the new disk . professor f: yeah , I I just an another question occurred to me is is what were you folks planning to do about normalization ? phd g:  . So that this could be another dimension , but we think perhaps we can use the the best , normalization scheme as OGI is using , so , with parameters that they use there ,  professor f: Yeah , I think that 's a good idea . phd g: u u professor f:  it 's i i we we seem to have enough dimensions as it is . professor f: probably the on - line line normalization because then it it 's if we do anything else , we 're gonna end up having to do on - line normalization too , so we may as well just do on - line normalization . So , I guess , yeah , th the other topic I maybe we 're already there , or almost there , is goals for the for next week 's meeting . i i i it seems to me that we wanna do is flush out what you put on the board here . Like a s like a slide ? professor f: so w we can say what we 're doing , grad c: OK . And , also , if you have sorted out , this information about how long i roughly how long it takes to do on what and , you know , what we can how many of these trainings , and testings and so forth that we can realistically do , then one of the big goals of going there next week would be to to actually settle on which of them we 're gonna do . Anything else that I a a Actually started out this this field trip started off with with , Stephane talking to Hynek , so you may have you may have had other goals , for going up , and any anything else you can think of would be we should think about accomplishing ?  , I 'm just saying this because maybe there 's things we need to do in preparation . And  and the other the the last topic I had here was ,  d Dave 's fine offer to to , do something on this .  he 's doing he 's working on other things , but to to do something on this project . So the question is , " Where where could we , most use Dave 's help ? " phd g: yeah , I was thinking perhaps if , additionally to all these experiments , which is not really research , well  it 's , running programs professor f: Yeah . phd g: and , trying to have a closer look at the perhaps the , speech , noise detection or , voiced - sound - unvoiced - sound detection and Which could be important in i for noise noise phd a: I think that would be a I think that 's a big big deal . Because the you know , the thing that Sunil was talking about , with the labels , labeling the database when it got to the noisy stuff ? The That that really throws things off . You know , having the noise all of a sudden , your your , speech detector ,  the the ,  What was it ? What was happening with his thing ? professor f:  phd a: He was running through these models very quickly . professor f: The only problem  , maybe that 's the right thing the only problem I have with it is exactly the same reason why you thought it 'd be a good thing to do . But I think the first responsibility is sort of to figure out if there 's something that , an an additional  , that 's a good thing you remove the mike . professor f: over years , if he 's if he 's interested in , you know , voiced - unvoiced - silence , he could do a lot . But if there if in fact there 's something else that he could be doing , that would help us when we 're we 're sort of  strapped for time We have we we 've , you know , only , another another month or two to you know , with the holidays in the middle of it , to to get a lot done . If we can think of something some piece of this that 's going to be The very fact that it is sort of just work , and i and it 's running programs and so forth , is exactly why it 's possible that it some piece of could be handed to someone to do , because it 's not  , yeah , so that that 's the question . And we don't have to solve it right this s second , but if we could think of some some piece that 's that 's well defined , that he could help with , he 's expressing a will willingness to do that . phd e: Yes , maybe to , mmm , put together the the label the labels between TIMIT and Spanish or something like that . professor f: So what we were just saying is that that ,  I was arguing for , if possible , coming up with something that that really was development and wasn't research because we we 're we have a time crunch . And so , if there 's something that would would save some time that someone else could do on some other piece , then we should think of that first . See the thing with voiced - unvoiced - silence is I really think that that it 's to do to do a a a a poor job is is pretty quick , or , you know , a so - so job . You can you can you can throw in a couple fea we know what what kinds of features help with it . But I remember , in fact , when you were working on that , and you worked on for few months , as I recall , and you got to , say ninety - three percent , and getting to ninety - four really really hard . So ,  And th th the other tricky thing is , since we are , even though we 're not we don't have a strict prohibition on memory size , and and computational complexity , clearly there 's some limitation to it . So if we have to if we say we have to have a pitch detector , say , if we if we 're trying to incorporate pitch information , or at least some kind of harmonic harmonicity , or something , this is another whole thing , take a while to develop .  , one I think one of the a lot of people would say , and I think Dan would also , that one of the things wrong with current speech recognition is that we we really do throw away all the harmonicity information . Reason for doing that is that most of the information about the phonetic identity is in the spectral envelopes are not in the harmonic detail . So wh that so the the other suggestion that just came up was , well what about having him work on the , multilingual super f superset kind of thing .  , coming up with that and then , you know , training it training a net on that , say , from from ,  from TIMIT or something . What what would you what would you think it would wh what would this task consist of ? phd g: Yeah , it would consist in , well , creating the the superset , and , modifying the lab labels for matching the superset . professor f: creating a superset from looking at the multiple languages , phd g: Well , creating the mappings , actually . phd g: Yeah , yeah , with the @ @ three languages , phd e: Maybe for the other language because TIMIT have more phone . grad c: There 's ,  Carmen was talking about this SAMPA thing , and it 's , it 's an effort by linguists to come up with , a machine readable IPA , sort of thing , right ? And , they they have a web site that Stephane was showing us that has ,  has all the English phonemes and their SAMPA correspondent , phoneme , professor f: Yeah . grad c: and then , they have Spanish , they have German , they have all all sorts of languages , mapping mapping to the SAMPA phonemes , which phd e: Yeah , the tr the transcription , though , for Albayzin is n the transcription are of SAMPA the same , how you say , symbol that SAMPA appear . phd b: I was gonna say , does that mean IPA is not really international ? grad c: No , it 's it 's saying phd a: It uses special diacritics and stuff , which you can't do with ASCII characters . professor f: What ,  Has OGI done anything about this issue ? Do they have Do they have any kind of superset that they already have ? phd g: I don't think so . Well , they they they 're going actually the the other way , defining  , phoneme clusters , apparently . phd a: So they just throw the speech from all different languages together , then cluster it into sixty or fifty or whatever clusters ? phd g: I think they 've not done it , doing , multiple language yet , but what they did is to training , English nets with all the phonemes , and then training it in English nets with , kind of seventeen , I think it was seventeen , broad classes . But Hynek didn't add didn't have all the results when he showed me that , so , well . phd g: But professor f: Is there 's some way that we should tie into that with this . Right ?  , if if in fact that is a better thing to do , should we leverage that , rather than doing , our own . Right ? So , if i if if they s  , we have i we have the the trainings with our own categories . And now we 're saying , " Well , how do we handle cross - language ? " And one way is to come up with a superset , but they are als they 're trying coming up with clustered , and do we think there 's something wrong with that ? phd g: I think that there 's something wrong professor f: OK . What w phd g: or Well , because Well , for the moment we are testing on digits , and e i perhaps u using broad phoneme classes , it 's it 's OK for  ,  classifying the digits , but as soon as you will have more words , well , words can differ with only a single phoneme , and which could be the same , class . Although , you are not using this for the phd g: So , I 'm professor f: You 're using this for the feature generation , though , not the phd g: Yeah , but you will ask the net to put one for th th the phoneme class professor f: Yeah . phd a: So you 're saying that there may not be enough information coming out of the net to help you discriminate the words ? professor f: Yeah . phd b: Fact , most confusions are within the phone phone classes , right ? I think , Larry was saying like obstruents are only confused with other obstruents , et cetera , et cetera . grad c: So so , maybe we could look at articulatory type stuff , professor f: But that 's what I thought they were gonna grad c: right ? professor f: Did they not do that , or ? phd g: I don't think so . Well , professor f: So phd g: they were talking about , perhaps , but they d professor f: They 're talking about it , phd g: I d professor f: but that 's sort of a question whether they did phd g: w Yeah . professor f: Instead of the the the the superclass thing , which is to take So suppose y you don't really mark arti To really mark articulatory features , you really wanna look at the acoustics and and see where everything is , and we 're not gonna do that . So , the second class way of doing it is to look at the , phones that are labeled and translate them into acoustic  ,  articulatory , features . You won't really have these overlapping things and so forth , phd a: So the targets of the net are these ? professor f: but phd a: Articulatory features . phd a: But that implies that you can have more than one on at a time ? professor f: Right . And ,  I don't know if our software this if the qu versions of the Quicknet that we 're using allows for that . Do you know ? grad c: Allows for ? professor f: Multiple targets being one ? grad c: Oh , we have gotten soft targets to to work . professor f: is that we could we could , just translate instead of translating to a superset , just translate to articulatory features , some set of articulatory features and train with that . Now the fact even though it 's a smaller number , it 's still fine because you have the the , combinations . So , in fact , it has every , you know it had has has every distinction in it that you would have the other way . We could I don't know , if you had  the phone labels , you could replace them by their articulatory features and then feed in a vector with those  , things turned on based on what they 're supposed to be for each phone to see if it if you get a big win . phd a: So , if your net is gonna be outputting , a vector of basically of well , it 's gonna have probabilities , but let 's say that they were ones and zeros , then y and you know for each , I don't know if you know this for your testing data , but if you know for your test data , you know , what the string of phones is and and you have them aligned , then you can just instead of going through the net , just create the vector for each phone and feed that in to see if that data helps . Eh , eh , what made me think about this is , I was talking with Hynek and he said that there was a guy at A T - andT who spent eighteen months working on a single feature . And because they had done some cheating experiments professor f: This was the guy that we were just talking a that we saw on campus . phd a: Well , Hynek said that that , I guess before they had him work on this , they had done some experiment where if they could get that one feature right , it dramatically improved the result . phd a: So I was thinking , you know it made me think about this , that if it 'd be an interesting experiment just to see , you know , if you did get all of those right . So that 's that 's equivalent to saying that you 've got got all the phones right . professor f: Although , yeah , it would be make an interesting cheating experiment because we are using it in this funny way , phd a: Yeah . phd a: And then you also don't know what error they 've got on the HTK side . phd b: The soft training of the nets still requires the vector to sum to one , though , right ? grad c: To sum up to one . phd b: So you can't really feed it , like , two articulatory features that are on at the same time with ones cuz it 'll kind of normalize them down to one half or something like that , for instance . Is it always softmax grad c: it 's sig No , it 's actually sigmoid - X phd g: or ? Yeah . grad c: for the phd g: So if you choose sigmoid it 's o it 's OK ? grad c: You ,  professor f: Did we just run out of disk , grad c: I think I think apparently , the ,  professor f: or ? phd b: Why don't you just choose linear ? Right ? grad c: What 's that ? phd b: Linear outputs ? grad c: Linear outputs ? phd b: Isn't that what you 'll want ? grad c:  . Right , but during the training , we would train on sigmoid - X phd b: Oh , you Yeah ? grad c: and then at the end just chop off the final nonlinearity . professor f: So , we 're we 're we 're off the air , or ? About to be off the air .


phd e: Ah grad f: Wh - what causes the crash ? phd a: Did you fix something ? phd c: Hello . grad f: Oh , maybe it 's the turning turning off and turning on of the mike , right ? professor b: you think that 's you ? Oh . phd e: Mmm , channel five ? Doesn't work ? professor b: Yeah , that 's the mike number there , mike number five , and channel channel four . So  I also copied  the results that we all got in the mail I think from  from OGI and we 'll go go through them also . phd d: The PLP , the PLP with JRASTA , the MSG , and the MFCC from the baseline Aurora . We 've trained  several neural networks on so on the TI - digits English and on the Italian data and also on the broad  English  French and  Spanish databases . Mmm , so there 's our result tables here , for the tandem approach , and  , actually what we we @ @ observed is that if the network is trained on the task data it works pretty well . Our our  There 's a We 're pausing for a photo phd c: Chicken on the grill . phd a: How about over th from the front of the room ? phd c: Yeah , it 's longer . And  actually we have  , results are similar Only on , phd a: Do you mean if it 's trained only on On data from just that task , phd d: yeah . But actually we didn't train network on  both types of data   phonetically ba phonetically balanced  data and task data . So , professor b: So how  clearly it 's gonna be good then phd a: So what 's th professor b: but the question is how much worse is it if you have broad data ?  , my assump From what I saw from the earlier results ,  I guess last week , was that  , if you trained on one language and tested on another , say , that the results were were relatively poor . professor b: But but the question is if you train on one language but you have a broad coverage and then test in another , does that is that improve things i c in comparison ? phd d: If we use the same language ? professor b: No , no , no . Different lang So  If you train on TI - digits and test on Italian digits , you do poorly , let 's say . professor b: I don't have the numbers in front of me , phd d: But Yeah but I did not  do that . E So , you didn't train on TIMIT and test on on Italian digits , say ? phd d: We No , we did four four kind of of testing , actually . The second test is trained on a single language  with broad database , but the same language as the t task data . The third test is by using ,  the three language database professor b: W which in phd d: and the fourth is professor b: It has three languages . That 's including the w the the phd d: This includes professor b: the one that it 's phd d: Yeah . phd d: Yeah And the fourth test is  excluding from these three languages the language that is the task language . So  for  TI - digits for ins example  when we go from TI - digits training to TIMIT training  we lose  around ten percent ,  . And then when we jump to the multilingual data it 's  it become worse and , well Around  , let 's say , twenty perc twenty percent further . phd d: But the first step is al already removing the task s specific from from phd a: Already , right right right . phd a: So they were sort of building here ? phd d: And we lose phd a: OK ? phd d: Yeah .  So , basically when it 's trained on the the multilingual broad data  or number so , the the ratio of our error rates  with the baseline error rate is around  one point one . professor b: I i if you compare everything to the first case at the baseline , you get something like one point one for the for the using the same language but a different task , and something like one point three for three three languages broad stuff . phd d: Tas - task data professor b: I I I meant something different by baseline phd d: we are u Yeah . professor b: So if we call a factor of w just one , just normalized to one , the word error rate that you have for using TI - digits as as training and TI - digits as test , phd d: Mmm . professor b: If we call that " one " , then what you 're saying is that the word error rate for the same language but using  different training data than you 're testing on , say TIMIT and so forth , it 's one point one . professor b: you do go to three languages including the English , it 's something like one point three . professor b: And if you exclude English , from this combination , what 's that ? phd d: If we exclude English ,  there is not much difference with the data with English . So what what it 's saying here is just that " yes , there is a reduction in performance , when you don't  have the s when you don't have  phd a: Task data . So it 's So when you go to a different task , there 's actually not so different . It 's when you went to these So what 's the difference between two and three ? Between the one point one case and the one point four case ? I 'm confused . The only difference it 's is that it 's multilingual  professor b: Cuz in both in both both of those cases , you don't have the same task . professor b: So is is the training data for the for this one point four case does it include the training data for the one point one case ? phd d:  yeah . professor b: How m how much bigger is it ? phd d:  It 's two times , grad f: Yeah ,  . professor b: So it 's two times , but it includes the but it includes the broad English data . professor b: So you have band - limited TIMIT , gave you  almost as good as a result as using TI - digits on a TI - digits test . OK ? phd d:  ? professor b:  and  But , when you add in more training data but keep the neural net the same size , it  performs worse on the TI - digits . OK , now all of this is This is noisy TI - digits , I assume ? Both training and test ? phd d:  professor b: Yeah . We we we may just need to  So  it 's interesting that h going to a different different task didn't seem to hurt us that much , and going to a different language  It doesn't seem to matter The difference between three and four is not particularly great , so that means that whether you have the language in or not is not such a big deal . professor b: It sounds like   we may need to have more of  things that are similar to a target language or  . You have the same number of parameters in the neural net , you haven't increased the size of the neural net , and maybe there 's just just not enough complexity to it to represent the variab increased variability in the in the training set .  So , what about So these are results with  th that you 're describing now , that they are pretty similar for the different features or or  phd d: let me check . For the PLP with JRASTA the the we This is quite the same tendency , with a slight increase of the error rate ,  if we go to to TIMIT . There there is a difference actually with b between PLP and JRASTA is that JRASTA seems to perform better with the highly mismatched condition but slightly slightly worse for the well matched condition . professor b: I have a suggestion , actually , even though it 'll delay us slightly , would would you mind running into the other room and making copies of this ? Cuz we 're all sort of If we c if we could look at it , while we 're talking , I think it 'd be phd d: Yeah , yeah . professor b: I 'll I 'll sing a song or dance or something while you do it , too . phd a: The  What was Was this number forty or It was roughly the same as this one , he said ? When you had the two language versus the three language ? professor b:  . That 's interesting because it it seems like what it 's saying is not so much that you got hurt  because you  didn't have so much representation of English , because in the other case you don't get hurt any more , at least when it seemed like  it it might simply be a case that you have something that is just much more diverse , phd a:   I wonder were  all three of these nets using the same output ? This multi - language  labelling ? grad f: He was using  sixty - four phonemes from SAMPA . phd a: So this would From this you would say , " well , it doesn't really matter if we put Finnish into the training of the neural net , if there 's gonna be , you know , Finnish in the test data . " Right ? professor b: Well , it 's it sounds  , we have to be careful , cuz we haven't gotten a good result yet . professor b: But I I I I think it does suggest that it 's not so much   cross language as cross type of speech . professor b: It 's it 's  But we did Oh yeah , the other thing I was asking him , though , is that I think that in the case Yeah , you you do have to be careful because of com compounded results . I think we got some earlier results in which you trained on one language and tested on another and you didn't have three , but you just had one language . Didn - Wasn't there something of that ? Where you , say , trained on Spanish and tested on on TI - digits , or the other way around ? Something like that ? phd e: No . professor b: This may have been what I was asking before , Stephane , but but , wasn't there something that you did , where you trained on one language and tested on another ?  no no mixture but just grad f: I 'll get it for you . professor b: We 've never just trained on one lang phd d: Training on a single language , you mean , and testing on the other one ? professor b: Yeah . phd d: So the only task that 's similar to this is the training on two languages , and that professor b: But we 've done a bunch of things where we just trained on one language . Either thi this is test with  the same language but from the broad data , or it 's test with  different languages also from the broad data , excluding the So , it 's it 's three or three and four . phd e: The early experiment that phd a: Did you do different languages from digits ? phd d:  . You mean training digits on one language and using the net to recognize on the other ? phd a: Digits on another language ? phd d: No . professor b:  What phd c: These numbers are  ratio to baseline ? professor b: So ,  wha what 's the phd d: So . professor b: This this chart this table that we 're looking at is  , show is all testing for TI - digits , or ? grad f: Bigger is worse . phd d: The upper part is for TI - digits grad f: Yeah , yeah , yeah . phd d: And the first four rows is well - matched , then the s the second group of four rows is mismatched , and finally highly mismatched . professor b: Well , What was is that i What was it that you had done last week when you showed Do you remember ? Wh - when you showed me the your table last week ? phd d: It - It was part of these results . phd a: So where is the baseline for the TI - digits located in here ? phd d: You mean the HTK Aurora baseline ? phd a: Yeah . professor b:   let 's see PLP  with on - line normalization and delta - del so that 's this thing you have circled here in the second column , phd d: Yeah . Actually I I  forgot to say that the multilingual net are trained on  features without the s derivatives  but with increased frame numbers . And we can we can see on the first line of the table that it it it 's slightly slightly worse when we don't use delta but it 's not not that much . What 's MF , MS and ME ? phd a: Multi - French , Multi - Spanish phd d: So . professor b: OK so I think what I 'm what I saw in your smaller chart that I was thinking of was was there were some numbers I saw , I think , that included these multiple languages and it and I was seeing that it got worse . In fact it might have been just this last category , having two languages broad that were where where English was removed . What I we hadn't seen yet was that if you added in the English , it 's still poor . professor b:   now , what 's the noise condition  of the training data phd d: Still poor . The noise condition is the same It 's the same  Aurora noises  , in all these cases for the training . professor b: So there 's not a statistical sta a strong st statistically different noise characteristic between  the training and test phd d: No these are the s s s same noises , professor b: and yet we 're seeing some kind of effect phd d: yeah . At least at least for the first for the well - matched , grad f: Well matched condition . professor b: So there 's some kind of a a an effect from having these  this broader coverage  Now I guess what we should try doing with this is try testing these on u this same sort of thing on you probably must have this lined up to do . So what appears is that perhaps Spanish is not very close to Italian because  , well , when using the the network trained only on Spanish it 's the error rate is almost  twice the baseline error rate . Is there any difference in So it 's in the  So you 're saying that when you train on English and  and and test on phd d: Yeah . professor b: No , you don't have training on English testing phd d: There there is another difference , is that the noise the noises are different . professor b: In in what ? phd d: Well , For for the Italian part  the  the  networks are trained with noise from Aurora TI - digits , phd e: Aurora - two . professor b: Do we have any  test sets  in any other language that  have the same noise as in the Aurora ? phd d: And phd e: Mmm , no . phd a: Can I ask something real quick ? In in the upper part in the English stuff , it looks like the very best number is sixty point nine ? and that 's in the  the third section in the upper part under PLP JRASTA , sort of the middle column ? phd d: Yeah . phd a: So that 's matched training ? Is that what that is ? phd d: It 's no , the third part , so it 's  highly mismatched . phd a: So why do you get your best number in Wouldn't you get your best number in the clean case ? phd c: Well , it 's relative to the  baseline mismatching phd d: Yeah . And then so , in the in the  in the non - mismatched clean case , your best one was under MFCC ? That sixty - one point four ? phd d: Yeah . But , what is  what is currently running , that 's  , i that just filling in the holes here or or ? pretty much ? phd d: no we don't plan to fill the holes professor b: OK . phd d: but actually there is something important , is that  we made a lot of assumption concerning the on - line normalization and we just noticed  recently that  the approach that we were using was not  leading to very good results when we used the straight features to HTK . So basically d if you look at the at the left of the table , the first  row , with eighty - six , one hundred , and forty - three and seventy - five , these are the results we obtained for Italian  with straight mmm , PLP features using on - line normalization . And the , mmm what 's in the table , just at the left of the PLP twelve on - line normalization column , so , the numbers seventy - nine , fifty - four and  forty - two are the results obtained by  Pratibha with  his on - line normalization  her on - line normalization approach . phd a: Where is that ? seventy - nine , fifty professor b: it 's just sort of sitting right on the  the column line . So these are the results of OGI with on - line normalization and straight features to HTK . And the previous result , eighty - six and so on , are with our features straight to HTK . phd d: So what we see that is there is that   the way we were doing this was not correct , but still the networks are very good . professor b: So , do you know what was wrong with the on - line normalization , or ? phd d: Yeah . There were diff there were different things and basically , the first thing is the mmm , alpha  value . I assume that this was not important because  previous results from from Dan and show that basically the both both values g give the same same  results . Actually ,  what we were doing is to start the recursion from the beginning of the utterance . phd d: And Pratibha did something different is that he  she initialed the  values of the mean and variance by computing this on the twenty - five first frames of each utterance . There were other minor differences , the fact that she used fifteen dissities instead s instead of thirteen , and that she used C - zero instead of log energy .  , I changed the code  and now we have a baseline that 's similar to the OGI baseline . phd d: We It it 's slightly  different because I don't exactly initialize the same way she does . Actually I start , mmm , I don't wait to a fifteen twenty - five twenty - five frames before computing a mean and the variance to e to to start the recursion . phd d: I I use the on - line scheme and only start the re recursion after the twenty - five twenty - fifth frame . So  I retrained the networks with these well , the the the networks are retaining with these new features . phd d: So basically what I expect is that these numbers will a little bit go down but perhaps not not so much professor b: Right . It it will learn how to normalize and professor b: OK , but I think that given the pressure of time we probably want to draw because of that especially , we wanna draw some conclusions from this , do some reductions in what we 're looking at , phd d: Yeah . professor b: and make some strong decisions for what we 're gonna do testing on before next week . So do you are you w did you have something going on , on the side , with  multi - band or on on this , phd d: Yeah I professor b: or ? phd d: No , I we plan to start this  so , act actually we have discussed  @ @  , these what we could do more as a as a research and and we were thinking perhaps that  the way we use the tandem is not  , well , there is basically perhaps a flaw in the in the the stuff because we trained the networks If we trained the networks on the on a language and a t or a specific task , professor b:   phd d: what we ask is to the network is to put the bound the decision boundaries somewhere in the space . phd d: And  mmm and ask the network to put one , at one side of the for for a particular phoneme at one side of the boundary decision boundary and one for another phoneme at the other side . And so there is kind of reduction of the information there that 's not correct because if we change task and if the phonemes are not in the same context in the new task , obviously the decision boundaries are not should not be at the same place . professor b: I di phd d: But the way the feature gives The the way the network gives the features is that it reduce completely the it removes completely the information a lot of information from the the features by   placing the decision boundaries at optimal places for one kind of data but this is not the case for another kind of data . professor b: It 's a trade - off , phd d: So professor b: right ? Any - anyway go ahead . So  what we were thinking about is perhaps  one way to solve this problem is increase the number of outputs of the neural networks .  , I I think you could make the same argument , it 'd be just as legitimate , for hybrid systems as well . phd d: Yeah but , we know that professor b: And in fact , th things get better with context dependent versions . professor b: Yeah , but it 's still true that what you 're doing is you 're ignoring you 're you 're coming up with something to represent , whether it 's a distribution , probability distribution or features , you 're coming up with a set of variables that are representing  , things that vary w over context . So , for that reason , when you in in in a hybrid system , when you incorporate context one way or another , you do get better scores . I I 'm I 'm sort of And once you the other thing is that once you represent start representing more and more context it is  much more  specific to a particular task in language . So   , the the acoustics associated with  a particular context , for instance you may have some kinds of contexts that will never occur in one language and will occur frequently in the other , so the qu the issue of getting enough training for a particular kind of context becomes harder . We already actually don't have a huge amount of training data  phd d: Yeah , but mmm , the the way we we do it now is that we have a neural network and basically the net network is trained almost to give binary decisions . professor b: It 's and and it is true that if there 's two phones that are very similar , that  the i it may prefer one but it will give a reasonably high value to the other , too . Yeah , sure but  So basically it 's almost binary decisions and  the idea of using more classes is to get something that 's less binary decisions . Because then you would say that in that this phone in this context is a one , but the same phone in a slightly different context is a zero . phd d: But yeah , but professor b: That would be even even more distinct of a binary decision . phd d: Yeah , but if professor b:  for instance , the the thing I was arguing for before , but again which I don't think we have time to try , is something in which you would modify the code so you could train to have several outputs on and use articulatory features phd d: Mmm . professor b: cuz then that would that would go that would be much broader and cover many different situations . Yeah , but I think Yeah , perhaps you 're right , but you have more classes so you you have more information in your features . phd d: because it 's it 's information that helps to discriminate , professor b:   phd d: if it 's possible to be able to discriminate among the phonemes in context . phd d: But the professor b:  we we could disagree about it at length phd d: Mmm . professor b: but the the real thing is if you 're interested in it you 'll probably try it phd d: Mmm . But but what I 'm more concerned with now , as an operational level , is  , you know , phd d: Mmm . professor b: what do we do in four or five days ?  , and so we have to be concerned with Are we gonna look at any combinations of things , you know once the nets get retrained so you have this problem out of it . professor b: are we going to look at multi - band ? Are we gonna look at combinations of things ?  , what questions are we gonna ask ,  now that , we should probably turn shortly to this O G I note .  , how are we going to combine with what they 've been focusing on ?  ,  we haven't been doing any of the L D A RASTA sort of thing . professor b: And they , although they don't talk about it in this note , there 's  , the issue of the  Mu law business  versus the logarithm , so . professor b: So what i what is going on right now ? What 's right you 've got nets retraining , Are there is there are there any H T K trainings testings going on ? phd d: N phd e: I I I 'm trying the HTK with eh , PLP twelve on - line delta - delta and MSG filter together .  professor b: And is this with the revised on - line normalization ? phd e: Ye -  , with the old older , phd d: Yeah . professor b: but again we have the hope that it We have the hope that it maybe it 's not making too much difference , phd e: Yeah . Well , something using place of articulation which which leads to nine , I think , broad classes . And then , something that combine both , and we have twenty f twenty - five ? grad f: Twenty - seven . professor b: So what you do  I just wanna understand phd d:  For the moments we do not don't have nets , professor b: so You have two net or three nets ? Was this ? How many how many nets do you have ? No nets . phd d: It 's just Were we just changing the labels to retrain nets with fewer out outputs . So you 're having multiple nets and combining them , or ?  , how are you how are you coming up with If you say  If you have a place characteristic and a manner characteristic , how do you phd d: It - It 's the single net , phd a: I think they have one output . grad f:  -  phd d: if we have twenty - seven classes , professor b: I see . professor b: So you 're sort of going the other way of what you were saying a bit ago instead of yeah . I think it will get worse because Well , I believe the effect that of of too reducing too much the information is basically basically what happens professor b:  - huh . phd d: and professor b: But you think if you include that plus the other features , phd d: but Yeah , because there is perhaps one important thing that the net brings , and OGI show showed that , is the distinction between sp speech and silence Because these nets are trained on well - controlled condition . So this is one thing And But perhaps , something intermediary using also some broad classes could could bring so much more information . professor b: And then  , just to remind me , all of that goes into  , that all of that is transformed by  , K - KL or something , or ? phd d:   phd d: or  , phd e: No transform the PLP phd d: per phd e: and only transform the other I 'm not sure . professor b: Well no , phd d: This is still something that professor b: I think I see . phd d: yeah , we don't know professor b: So there 's a question of whether you would phd e: Two e @ @ it 's one . So that 's something that you 're you haven't trained yet but are preparing to train , and phd d: Yeah . professor b: So I think , you know , we need to choose the choose the experiments carefully , so we can get  key key questions answered  before then phd d:   professor b: and leave other ones aside even if it leaves incomplete tables someplace , it 's it 's really time to time to choose .  professor b: OK , so  , Something I asked So they 're they 're doing the the VAD I guess they mean voice activity detection So again , it 's the silence So they 've just trained up a net which has two outputs , I believe .  I asked  Hynek whether I haven't talked to Sunil I asked Hynek whether they compared that to just taking the nets we already had and summing up the probabilities . To get the speech voice activity detection , or else just using the silence , if there 's only one silence output . But on the other hand , maybe they can get by with a smaller net and maybe sometimes you don't run the other , maybe there 's a computational advantage to having a separate net , anyway . professor b: there 's a an example or two that you can find , where it made it slightly worse , but  in in all but a couple examples .  how are trained the the LDA filter ? How obtained the LDA filter ? phd d: Mmm . phd e: Yeah ,  for example , LDA filter need a set of a set of training to obtain the filter . phd e: And maybe for the Italian , for the TD TE on for Finnish , these filter are are obtained with their own training set . That 's that 's so that 's a that 's a very good question , then now that it I understand it . It 's " yeah , where does the LDA come from ? " In the In earlier experiments , they had taken LDA from a completely different database , right ? phd e: Yeah . Yeah , because maybe it the same situation that the neural network training with their own phd d: Mmm .  , but  to tell you the truth , I wasn't actually looking at the LDA so much when I I was looking at it I was mostly thinking about the the VAD . And  , it ap it ap Oh what does what does ASP ? Oh that 's phd d: The features , yeah . phd e: what is what is the difference between ASP and  baseline over ? phd c: ASP . professor b: And it 's This is mostly better than baseline , although in some cases it 's a little worse , in a couple cases . professor b: I think this was I think this is the same point we were at when when we were up in Oregon . phd d: I think I think it 's the C - zero using C - zero instead of log energy . phd a: They s they say in here that the VAD is not used as an additional feature . professor b: Shouldn't it be phd d: Because phd a: Does does anybody know how they 're using it ? professor b: Yeah . professor b: if you look down at the block diagram , they estimate they get a they get an estimate of whether it 's speech or silence , phd a: But that professor b: and then they have a median filter of it . professor b: You find stretches where the combination of the frame wise VAD and the the median filter say that there 's a stretch of silence . professor b: Right ? So  phd a: So it 's it 's I don't understand . You mean it 's throwing out frames ? Before professor b: It 's throwing out chunks of frames , yeah . There 's the the median filter is enforcing that it 's not gonna be single cases of frames , or isolated frames . professor b: So it 's throwing out frames and the thing is  , what I don't understand is how they 're doing this with H T phd a: Yeah , that 's what I was just gonna ask . professor b: This is phd a: How can you just throw out frames ? professor b: Yeah . Well , you you can , phd d: i professor b: right ?  y you you phd d: Yeah . Yeah , so  in the i i in the in the decoding , you 're saying that we 're gonna decode from here to here . professor b: I think they 're they 're they 're treating it , you know , like  well , it 's not isolated word , but but connected , you know , the the phd a: In the text they say that this this is a tentative block diagram of a possible configuration we could think of . No they they have numbers though , right ? So I think they 're they 're doing something like that . I think that they 're they 're I think what  by tha that is they 're trying to come up with a block diagram that 's plausible for the standard . In other words , it 's   from the point of view of of  reducing the number of bits you have to transmit it 's not a bad idea to detect silence anyway . I 'm just wondering what exactly did they do up in this table if it wasn't this . But it 's the thing is it 's that that that 's that 's I I Certainly it would be tricky about it intrans in transmitting voice ,   for listening to , is that these kinds of things  cut speech off a lot . professor b: It does introduce delays but they 're claiming that it 's it 's within the the boundaries of it . professor b: And the LDA introduces delays , and b what he 's suggesting this here is a parallel path so that it doesn't introduce  , any more delay . I it introduces two hundred milliseconds of delay but at the same time the LDA down here I don't know Wh what 's the difference between TLDA and SLDA ? phd c: Temporal and spectral . The temporal LDA does in fact include the same so that I think he well , by by saying this is a b a tentative block di diagram I think means if you construct it this way , this this delay would work in that way phd a: Ah . So  I think that it 's it 's nice to do that in this because in fact , it 's gonna give a better word error result and therefore will help within an evaluation .  , as you know , part of the problem with evaluation right now is that the word models are pretty bad and nobody wants has has approached improving them . So it 's possible that a lot of the problems with so many insertions and so forth would go away if they were better word models to begin with . So  The question we 're gonna wanna go through next week when Hynek shows up I guess is given that we 've been if you look at what we 've been trying , we 're  looking at  , by then I guess , combinations of features and multi - band  , and we 've been looking at cross - language , cross task issues . And I guess when he comes here we 're gonna have to start deciding about  what do we choose from what we 've looked at to  blend with some group of things in what they 've looked at And once we choose that , how do we split up the effort ?  , because we still have even once we choose , we 've still got  another month or so ,  there 's holidays in the way , but but  I think the evaluation data comes January thirty - first so there 's still a fair amount of time to do things together it 's just that they probably should be somewhat more coherent between the two sites in that that amount of time . phd a: When they removed the silence frames , did they insert some kind of a marker so that the recognizer knows it 's knows when it 's time to back trace or something ? professor b: Well , see they , I I think they 're  . They 're they 're getting around the way the recognizer works because they 're not allowed to  , change the scripts for the recognizer , I believe . professor b:  that 's sort of what the way I had imagined would happen is that on the other side , yeah you p put some low level noise or something . professor b: Most recognizers don't like zeros but but you know , put some epsilon in or some rand phd a: Yeah .  i w Or something professor b: Maybe not a constant but it doesn't ,  don't like to divide by the variance of that , but  it 's phd a: That 's right . or else ,   maybe there is some indicator to tell it to start and stop , I don't know . Otherwise , if it 's just a bunch of speech , stuck together professor b: No they 're phd a: Yeah . And   I think  , I wanna look at these numbers off - line a little bit and think about it and and talk with everybody  , outside of this meeting .  , but  No  it sounds like  there there there are the usual number of of little little problems and bugs and so forth but it sounds like they 're getting ironed out . And now we 're seem to be kind of in a position to actually  , look at stuff and and and compare things .  I don't know what the One of the things I wonder about , coming back to the first results you talked about , is is how much ,  things could be helped by more parameters . Because anyway when we go to twice as much data and have the same number of parameters , particularly when it 's twice as much data and it 's quite diverse , I wonder if having twice as many parameters would help . How are we doing on the resources ? Disk , and phd d: I think we 're alright , professor b: OK . professor b: Are were you folks using Gin ? That 's a that just died , you know ? phd d: Mmm , no . professor b: That 'll be It 's a seven hundred fifty megahertz  SUN phd d:  . grad g: Do we Do we have that big new IBM machine the , I think in th professor b: We have the little tiny IBM machine that might someday grow up to be a big IBM machine . It 's got s slots for eight ,  IBM was donating five , I think we only got two so far , processors . So instead of having eight processors that were eight hundred megahertz , we ended up with two that are five hundred and fifty megahertz . So I don't think anybody has been sufficiently excited by it to spend much time  with it , but  Hopefully , they 'll get us some more parts , soon and  , yeah , I think that 'll be once we get it populated , that 'll be a nice machine . grad g: And if we can do things on Linux , some of the machines we have going already , like Swede ? professor b:   And it does have two processors , you know and Somebody could do you know , check out  the multi - threading libraries . And  i it 's possible that the  , I guess the prudent thing to do would be for somebody to do the work on on getting our code running on that machine with two processors even though there aren't five or eight . There 's there 's there 's gonna be debugging hassles and then we 'd be set for when we did have five or eight , to have it really be useful . You don't get the don't get the visuals but grad g: I is it  mostly  the neural network trainings that are  slowing us down or the HTK runs that are slowing us down ? professor b: I think yes .  , Isn't that right ?  I think you 're you 're sort of held up by both , right ? If the if the neural net trainings were a hundred times faster you still wouldn't be anything running through these a hundred times faster because you 'd be stuck by the HTK trainings , phd d: Mmm . professor b: But if the HTK  I think they 're both It sounded like they were roughly equal ? Is that about right ? phd d: Yeah . grad g: Because ,  I think that 'll be running Linux , and Sw - Swede and Fudge are already running Linux so ,  I could try to get  the train the neural network trainings or the HTK stuff running under Linux , and to start with I 'm wondering which one I should pick first . professor b: probably the neural net cuz it 's probably it it 's it 's  Well , I I don't know . They both HTK we use for  this Aurora stuff   , I think It 's not clear yet what we 're gonna use for trainings  Well , there 's the trainings  is it the training that takes the time , or the decoding ?  , is it about equal between the two ? For for Aurora ? phd d: For HTK ? professor b: For Yeah . Well , I don't know how we can I don't know how to Do we have HTK source ? Is that Yeah . professor b: You would think that would fairly trivially the training would , anyway , th the testing  I don't I don't think would parallelize all that well . But I think that you could certainly do d  , distributed , sort of Ah , no , it 's the each individual sentence is pretty tricky to parallelize . phd a: They have a they have a thing for doing that and th they have for awhile , in H T And you can parallelize the training . professor b: Yeah ? phd a: And run it on several machines professor b: Aha ! phd a: and it just basically keeps counts . phd a: I don't what their scripts are set up to do for the Aurora stuff , but phd d: Yeah . professor b: Something that we haven't really settled on yet is other than this Aurora stuff ,  what do we do , large vocabulary training slash testing for  tandem systems . So I I think the the advantage of going with the neural net thing is that we 're gonna use the neural net trainings , no matter what , for a lot of the things we 're doing , grad g: OK . professor b: whereas , w exactly which  Gaussian - mixture - based  thing we use is gonna depend  So with that , maybe we should  go to our digit recitation task . I think we can @ @ You know Herve 's coming tomorrow , right ? Herve will be giving a talk , yeah , talk at eleven . Did  , did everybody sign these consent Er everybody Has everyone signed a consent form before , on previous meetings ? You don't have to do it again each time Yes