professor f: We can say the word " zero " all we want , phd g: I 'm doing some professor f: but just phd g: square brackets , coffee sipping , square brackets . phd d: There 's gonna be some zeros from this morning 's meeting because I noticed that professor f: u phd d: Barry , I think maybe you turned your mike off before the digits were Oh , was it during digits ? Oh , so it doesn't matter . phd b: So it 's not it 's not that bad if it 's at the end , but it 's in the beginning , it 's bad . grad a: Yeah , you wanna you wanna keep them on so you get good noise noise floors , through the whole meeting . Yeah I did have to run , but grad e: Is there any way to change that in the software ? grad a: Change what in the software ? grad e: Where like you just don't like if you if it starts catching zeros , like in the driver or something in the card , or somewhere in the hardware Where if you start seeing zeros on w across one channel , you just add some random , @ @ noise floor like a small noise floor . grad a:  certainly we could do that , but I don't think that 's a good idea . professor f: Well , I u I actually don't know what the default is anymore as to how we 're using the the front - end stuff but for for when we use the ICSI front - end , grad a: As an argument . professor f: but  , there is an there is an o an option in in RASTA , which , in when I first put it in , back in the days when I actually wrote things , I did actually put in a random bit or so that was in it , grad e: OK . professor f: but then I realized that putting in a random bit was equivalent to adding  adding flat spectrum , grad e: Right . Gee ! Here we all are ! grad a: so the only agenda items were Jane was Jane wanted to talk about some of the IBM transcription process . professor f: There 's an agenda ? grad a: I sort of condensed the three things you said into that . And then just I only have like , this afternoon and maybe tomorrow morning to get anything done before I go to Japan for ten days . So if there 's anything that n absolutely , desperately needs to be done , you should let me know now . professor f: Yeah , I guess you first you have to do the first one , grad a: Yeah . professor f: Oh they they had some extension that they announced or something ? phd g: Well yeah . Liz had sent them a note saying " could we please have another " I don't know , " three days " or something , and they said yes . phd d: And then she said " Did I say three ? grad a: Oh , phd d: I meant four . " grad a: that was the other thing  , phd g: But u grad a: Dave Gelbart sent me email , I think he sent it to you too , that  , there 's a special topic , section in si in Eurospeech on new , corp corpors corpora . grad e: Huh ! grad a: And  , professor f: Oh ! phd b: I got this mail from grad a: I s forwarded it to Jane as I thought being the most relevant person .  So , I thought it was highly relevant postdoc c: Yeah I 'm professor f: That 's grad a: have you did you look at the URL ? postdoc c: Yeah . phd b: Was this SmartKom message ? I think Christoph Draxler sent this , postdoc c: Yeah . grad a: but obviously I can't , really do , most of it , postdoc c: Yeah . For instance that Morgan  , accounted for fifty - six percent of the Robustness meetings in terms of number of words . postdoc c: because is it partly , eh , c correctly identified words ? Or is it or just overall volume ? phd g: No . I think it 's he 's he 's in all of them , postdoc c: Oh . phd g: we didn't mention Morgan by name grad a: and he talks a lot . professor f: Well we have now , but phd g: We we we something about grad a: Did you identify him as a senior member ? phd g: No , we as identify him as the person dominating the conversation . professor f:  I get these AARP things , but I 'm not se really senior yet , but phd g: Right professor f: phd g:  . professor f: but  , other than that delightful result , what was the rest of the paper about ? phd g: well it was about it had three sections professor f: You sent it to me but I haven't seen it yet .  , the one was that the just the the amount of overlap grad a: The good , the bad , and the ugly . phd g: s in terms of in terms of number of words and also we computed something called a " spurt " , which is essentially a stretch of speech with  , no pauses exceeding five hundred milliseconds .  , and we computed how many overlapped i  spurts there were and how many overlapped words there were .  , for four different corpora , the Meeting Recorder meetings , the Robustness meetings Switchboard and CallHome , and , found and sort of compared the numbers .  , and found that the , you know , as you might expect the Meeting Recorder meetings had the most overlap  , but next were Switchboard and CallHome , which both had roughly the same , almost identical in fact , and the Robustness meetings were had the least , so One sort of unexpected result there is that  two - party telephone conversations have about the same amount of overlap , grad a: I 'm surprised . phd g: sort of in gen you know order of magnitude - wise as ,  as face - to - face meetings with multiple grad a: I have I had better start changing all my slides ! phd g: Yeah . Also , I in the Levinson , the pragmatics book , in you know , textbook , there 's I found this great quote where he says you know you know , how people it talks about how  how how people are so good at turn taking , postdoc c:   phd g: and so they 're so good that generally , u the overlapped speech does not is less than five percent . grad e: Did he mean face like face - to - face ? Or ? phd g: Well , in real conversations , grad e:  . phd b: But postdoc c: Well , of course , no , it doesn't necessarily go against what he said , cuz he said " generally speaking " . phd b: And in f phd g: Well , he he made a claim grad a: Well phd g: Well grad a:  phd b: But professor f: Yeah , we we have pretty limited sample here . phd b: Five percent of time or five percent of what ? grad a: Yeah , I was gonna ask that too . phd b: Yeah , so postdoc c: It 's i it 's not against his conclusion , phd g: So but still but still u postdoc c: it just says that it 's a bi bell curve , and that , you have something that has a nice range , in your sampling . So there are slight There are differences in how you measure it , but still it 's You know , the difference between  between that number and what we have in meetings , which is more like , you know , close to in meetings like these ,  you know , close to twenty percent . professor f: But what was it like , say , in the Robustness meeting , for instance ? phd g: That grad a: But phd g: Robustness meeting ? It was about half of the r So , in terms of number of words , it 's like seventeen or eigh eighteen percent for the Meeting Recorder meetings and about half that for , the Robustness . professor f: Maybe ten percent ? grad a: But I don't know if that 's really a fair way of comparing between , multi - party , conversations and two - party conversations . phd b: Then then then you have to grad a:  that 's just something phd d: Yeah , I just wonder if you have to normalize by the numbers of speakers or something . phd b: Then Yeah , then normalize by by something like that , postdoc c: Yeah , that 's a good point . phd g: but this obvious thing to see if if there 's a dependence on the number of  participants . grad a: You have a lot of a lot of two - party , subsets within the meeting . phd g: And and and then and we also d computed this both with and without backchannels , postdoc c:   phd g: so you might think that backchannels have a special status because they 're essentially just grad a:  - huh . So , did we all said "  - huh " and nodded at the same time , phd g: R right . phd g: But , even if you take out all the backchannels so basically you treat backchannels l as nonspeech , as pauses , grad a:   You know , it goes down from maybe For Switchboard it goes down from I don't know f  I don't know f fourteen percent of the words to maybe  I don't know , eleven percent or something it 's it 's not a dramatic change , grad a:   phd g: so it 's Anyway , so it 's  That was that was one set of results , and then the second one was just basically the the stuff we had in the in the HLT paper on how overlaps effect the recognition performance .  , but mostly we added one one number , which was what if you  , basically score ignoring all So so the the conjecture from the HLT results was that most of the added recognition error is from insertions due to background speech . So , we scored all the recognition results , in such a way that the  grad a: Oh by the way , who 's on channel four ? You 're getting a lot of breath . phd g: OK , so so if you have the foreground speaker speaking here , and then there 's some background speech , may be overlapping it somehow , and this is the time bin that we used , then of course you 're gonna get insertion errors here and here . phd g: Right ? So we scored everything , and I must say the NIST scoring tools are pretty nice for this , where you just basically ignore everything outside of the , region that was deemed to be foreground speech . And where that was we had to use the t forced alignment , results from s for so That 's somewhat that 's somewhat subject to error , but still we we  , Don did some ha hand - checking and and we think that based on that , we think that the results are you know , valid , although of course , some error is gonna be in there . But basically what we found is after we take out these regions so we only score the regions that were certified as foreground speech , the recognition error went down to almost  , the level of the non - overlapped speech . So that means that even if you do have background speech , if you can somehow separate out or find where it is , the recognizer does a good job , grad a: That 's great . phd g: even though there is this back grad a: Yeah , I guess that doesn't surprise me , because , with the close - talking mikes , the the signal will be so much stronger .  , grad a: What what sort of normalization do you do ? phd g: so  , well , we just @ @ we do u you know , vit grad a:  in you recognizer , in the SRI recognizer . phd g: Well , we do  , VTL vocal tract length normalization , w and we  you know , we we  , make all the features have zero mean and unit variance . grad a: Over an entire utterance ? professor f: And grad a: Or windowed ? phd g: Over over the entire c over the entire channel . We just took the old So this is actually a sub - optimal way of doing it , grad a: Right . So the recognizer didn't have the benefit of knowing where the foreground speech a start professor f: Were you including the the lapel in this ? phd g: Yes . professor f: And did the did did the la did the the problems with the lapel go away also ? Or phd g: it Yeah . professor f: fray for for insertions ? phd g: It u not per  , not completely , but yes , professor f: Less so . So we have to  professor f: you still phd g: Well I should bring the should bring the table with results . professor f: I would presume that you still would have somewhat higher error with the lapel for insertions than phd g: Yes . professor f: Cuz again , looking forward to the non - close miked case , I think that we s still phd g:   And then , the third thing was , we looked at , what we call " interrupts " , although that 's that may be a misnomer , but basically we looked at cases where  , so we we used the punctuation from the original transcripts and we inferred the beginnings and ends of sentences . So , you know postdoc c: Di - did you use upper - lower case also , or not ? phd g:  postdoc c: U upper lower case or no ? phd g:  ? postdoc c: OK . phd g: No , we only used , you know ,  periods , question marks and exclamation . And we know that there 's th that 's not a very g  , we miss a lot of them , postdoc c: Yeah . That 's OK but phd g: but but it 's f i i postdoc c: Comma also or not ? phd g: No commas . And then we looked at locations where , if you have overlapping speech and someone else starts a sentence , you know , where do these where do other people start their turns not turns really , but you know , sentences , phd b: Ah . phd g:  So we only looked at cases where there was a foreground speaker and then at the to at the so the the foreground speaker started into their sentence and then someone else started later . phd b: Somewhere in between the start and the end ? phd g: OK ? And so what phd b: OK . phd g: Sorry ? phd b: Somewhere in between the start and the end of the foreground ? phd g: Yes . phd g: So , the the question was how can we what can we say about the places where the second or or actually , several second speakers ,  start their " interrupts " , as we call them . phd g: w And we looked at this in terms of  grad a: On T - closures , only . phd g: So so we had we had  u to for for the purposes of this analysis , we tagged the word sequences , and and we time - aligned them .  , and we considered it interrupt if it occurred in the middle of a word , we basically you know , considered that to be a interrupt as if it were at at the beginning of the word . phd g: And then we looked at the the locatio the , you know , the features that the tags because we had tagged these word strings , that that occurred right before these these  , interrupt locations . phd b: Tag by  phd g: And the tags we looked at are the spurt tag , which basically says or actually Sorry . So whether there was a pause essentially here , because spurts are a defined as being you know , five hundred milliseconds or longer pauses , and then we had things like discourse markers , backchannels , disfluencies .  , filled pauses So disfluen the D 's are for , the interruption points of a disfluency , so , where you hesitate , or where you start the repair there .  , repeated you know , repeated words is another of that kind of disfluencies and so forth . So we had both the beginnings and ends of these  so , the end of a filled pause and the end of a discourse marker . We just looked at the distribution of words , and so every " so yeah " , and " OK " , and "  - huh " were were the were deemed to be backchannels and " wow " and " so " and  " right " ,  were  Not " right " . But so , we sort of just based on the lexical  , identity of the words , we we tagged them as one of these things . So , and then we looked at the disti so we looked at the distribution of these different kinds of tags , overall  , and and and particularly at the interruption points . And  , we found that there is a marked difference so that for instance after so at the end after a discourse marker or after backchannel or after filled pause , you 're much more likely to be interrupted than before . So pauses are always an opportunity for So we have this little histogram which shows these distributions and , phd d: I wonder phd g: you know , it 's it 's it 's not No big surprises , but it is sort of interesting from grad a: It 's nice to actually measure it though . In other words  if you weren't going to pause you you will because you 're g being interrupted . professor f: But he yeah , he 's he 's right , y  maybe you weren't intending to pause at all , but You were intending to stop for fifty - seven milliseconds , phd g: Right . And and we so we wrote this and then , we found we were at six pages , and then we started cutting furiously phd b: Oops . phd g: and threw out half of the material again , and  played with the LaTeX stuff and grad a: Made the font smaller and the narrows longer . phd g: but we s we put Oh , I I grad a: Took out white space . phd g: you know the the gap between the two columns is like ten millimeters , phd b: Yeah . phd d: Wasn't there wasn't there some result , Andreas professor f: Yeah phd d: I I thought maybe Liz presented this at some conference a while ago about  , backchannels phd g:   Well phd d:  - i i do you rem phd g: y We didn't talk about , prosodic , properties at all , phd d: Right . But phd g: although that 's I I take it that 's something that  Don will will look at grad e: Yeah , we 're gonna be looking at that . postdoc c: Well , I didn't know about Liz 's finding on that , phd d: About postdoc c: but I know of another paper that talks about something phd d:  - huh . phd d: It made me think about a cool little device that could be built to  to handle those people that call you on the phone and just like to talk and talk and talk . And you just have this little detector that listens for these drops in pitch and gives them the backchannel . phd g: There 's actually  there 's this a former student of here from Berkeley , Nigel Nigel Ward . phd g: He did a system  , in he he lives in Japan now , and he did this backchanneling , automatic backchanneling system . phd g: It 's a very phd d: Oh ! phd g: So , exactly what you describe , phd d: Huh . And it 's apparently for Japa - in Japanese it 's really important that you backchannel . Actually for a lot of these people I think you could just sort of backchannel continuously and it would pretty much be fine . Where the barber who was afraid of scissors was playing a a tape of clipping sounds , and saying "  - huh " , " yeah " , " how about them sports teams ? " phd g: Anyway . So the paper 's on - line and y I I think I  I CC ' ed a message to Meeting Recorder with the URL so you can get it . So I I 'm actually about to send Brian Kingbury an email saying where he can find the the s the m the material he wanted for the s for the speech recognition experiment , so but I haven't sent it out yet because actually my desktop locked up , like I can't type anything .  b so if there 's any suggestions you have for that I was just gonna send him the phd d: Is it the same directory that you had suggested ? phd g: I made a directory . phd g: He does ? postdoc c: And he and he 's phd g: Yeah but but but he has to postdoc c: I 'd hafta add him to Meeting Recorder , I guess , phd g: he prefe he said he would prefer FTP postdoc c: but OK . phd g: and also , the other person that wants it There is one person at SRI who wants to look at the  , you know , the  the data we have so far , postdoc c: OK . So what I did is I  @ @ I made a n new directory after Chuck said that would c that was gonna be a good thing . Right ? The same the same as the mailing list , professor f: Yeah , phd g: and professor f: the No vowels .  , professor f: Yeah phd g: and then under there  actually Oh and this directory , is not readable . So , in other words , to access anything under there , you have to be told what the name is . phd g: So  , and the directory for this I call it I " ASR zero point one " because it 's sort of meant for recognition . professor f: So anyone who hears this meeting now knows the grad a: Beta ? phd g: And then then in there I have a file that lists all the other files , so that someone can get that file and then know the file names and therefore download them . If you don't know the file names you can't professor f: Is that a dash or a dot in there ? phd g:  you can grad a: Don't don't don't say . So all I all I was gonna do there was stick the the transcripts after we the way that we munged them for scoring , because that 's what he cares about , and  , and also and then the the waveforms that Don segmented .  , just basically tar them all up f  w for each meeting I tar them all into one tar file and G - zip them and stick them there . grad a: I  , put digits in my own home directory home FTP directory , phd g: And so . phd d: So we could point Mari to this also for her March O - one request ? phd g: OK . phd d: Or phd g: Oh ! phd d: You n Remember she was phd g: Oh she wanted that also ? phd d: Well she was saying that it would be nice if we had they had a Or was she talking Yeah . She was saying it would be nice if they had eh the same set , so that when they did experiments they could compare . grad e:  I phd g: But yeah , we can send I can CC Mari on this so that she knows phd d: Yeah . phd d: so I was gonna probably put it grad a: We can put it in the same place .  , so either we should regenerate the original versions , or  , we should just make a note of it . And so I but OK so but for the other meetings it 's the downsampled version that you have . Oh that 's th important to know , OK so we should probably  give them the non - downsampled versions . Alright , then I 'll hold off on that and I 'll wait for you  grad e: Probably by tomorrow phd g: gen grad e: I can I 'll send you an email . Yeah , definitely they should have the full bandwidth version , grad e: Yeah , because  I I think Liz decided to go ahead with the downsampled versions cuz we can There was no s like , r significant difference . grad e: It does take up less disk space , and apparently it did even better than the original than the original versions , phd g: Yeah . Good that Well , it 's a good thing that grad a: OK , I think we 're losing , Don and Andreas at three - thirty , right ? OK . professor f: So , that 's why it was good to have Andreas , say these things but So , we should probably talk about the IBM transcription process stuff that postdoc c: OK . So ,  you know that Adam created  , a b a script to generate the beep file ? professor f:  . But but you were gonna to use the originally transcribed file because I tightened the time bins and that 's also the one that they had already in trying to debug the first stage of this . And  , my understanding was that ,  I haven't I haven't listened to it yet , grad a:   postdoc c: but it sounded very good and and I understand that you guys were going to have a meeting today , before this meeting .  , just so that while I 'm gone , you can regenerate it if you decide to do it a different way . So  , Chuck and Thilo should , now more or less know how to generate the file postdoc c: Excellent . grad a: and , the other thing Chuck pointed out is that , since this one is hand - marked , there are discourse boundaries . So what what we 're probably gonna do is just write a script , that if two , chunks are very close to each other on the same channel we 'll just merge them . grad a: So , and that will get around the problem of , the , you know " one word beep , one word beep , one word beep , one word beep " . phd d: Yeah , in fact after our meeting  , this morning Thilo came in and said that  , there could be other differences between the  already transcribed meeting with the beeps in it and one that has just r been run through his process . phd d: So tomorrow , when we go to make the   , chunked file for IBM , we 're going to actually compare the two . So he 's gonna run his process on that same meeting , postdoc c: Great idea ! phd d: and then we 're gonna do the beep - ify on both , and listen to them and see if we notice any real differences . phd g: Beep - ify ! postdoc c: OK , now one thing that prevented us from apply you you from applying Exactly . Wel -  we just wanna if if there 're any major differences between doing it on the hand postdoc c:  - huh . phd g: So this training meeting ,  w un is that  some data where we have  very  , you know , accurate time marks ? for postdoc c: I went back and hand - marked the ba the bins , I ment I mentioned that last week . phd d: But the but there 's yeah , but there is this one issue with them in that there 're there are time boundaries in there that occur in the middle of speech . phd g: Because phd d: So Like when we went t to  When I was listening to the original file that Adam had , it 's like you you hear a word then you hear a beep and then you hear the continuation of what is the same sentence . grad a: It 's i phd d: So there are these chunks that look like  that have  grad a:  that 's not gonna be true of the foreground speaker . So you 'll you 'll have a chunk of , you know , channel A which starts at zero and ends at ten , and then the same channel starting at eleven , ending at fifteen , and then again , starting at sixteen , ending at twenty . Right , so that 's three chunks where actually we w can just make one chunk out of that which is A , zero , twenty . postdoc c: Yeah , I thought that was phd d: So if you were to use these , you have to be careful not to pull out these individual postdoc c: Yeah . phd g: Oh !  it Right ,  w  what I would I was interested in is having a se having time marks for the beginnings and ends of speech by each speaker . phd g: So  , it I don't care that you know , there 's actually abutting segments that we have to join together . phd g: But what we do care about is that the beginnings and ends  are actually close to the speech inside of that phd d: Yeah , I think Jane tightened these up by hand . phd g: OK , so what is the sort of how tight are they ? professor f: it looks much better . I just wanted to get it so tha So that if you have like " yeah " in a swimming in a big bin , then it 's phd g: No , no ! I don grad a: Let me make a note on yours . phd g: I it 's f That 's fine because we don't want to th that 's perfectly fine . You always want to have a little bit of pause or nonspeech around the speech , say for recognition purposes .  , but just just u w you know get an id I just wanted to have an idea of the of how much extra you allowed  so that I can interpret the numbers if I compared that with a forced alignment segmentation . postdoc c: but but my main goal was  , in these areas where you have a three - way overlap and one of the overlaps involves " yeah " , and it 's swimming in this huge bin , I wanted to get it so that it was clo more closely localized . But are we talking about , I don't know , a tenth of a second ? a ? You know ? How how much how much extra would you allow at most postdoc c: I I wanted to I wanted it to be able to l he be heard normally , phd g:   postdoc c: so that if you if you play back that bin and have it in the mode where it stops at the boundary , it sounds like a normal word . postdoc c: Now sometimes you know , it 's these are involved in places where there was no time . postdoc c:  some cases , there 're some people  , who who have very long segments of discourse where , you know , they 'll they 'll breath and then I put a break . postdoc c: But other than that , it 's really pretty continuous and this includes things like going from one sentence into the u one utterance into the next , one sentence into the next , w without really stopping .  i they , i you know in writing you have this two spaces and a big gap phd g:   postdoc c: But but  i some people are planning and , you know , a lot we always are planning what we 're going to say next . postdoc c: But  , in which case , the gap between these two complete syntactic units , which of course n spoken things are not always complete syntactically , but but it would be a shorter p shorter break than maybe you might like . postdoc c: But the goal there was to not have the text be so so crudely parsed in a time bin .  , because from a discourse m purpose it 's it 's more it 's more useful to be able to see and also you know , from a speech recognition purpose my impression is that if you have too long a unit , it 's it doesn't help you very much either , cuz of the memory . postdoc c: So , that means that the amount of time after something is variable depending partly on context , but my general goal when there was sufficient space , room , pause after it to have it be kind of a natural feeling gap . postdoc c: Which I c I don't know what it would be quantified as . You know , Wally Chafe says that  , in producing narratives , the spurts that people use tend to be , that the the what would be a pause might be something like two two seconds . postdoc c: I was interested that you chose  , you know  , the you know that you use cuz I think that 's a unit that would be more consistent with sociolinguistics . phd g: Well we chose  , you know , half a second because if if you go much larger , you have a y you know , your your statement about how much overlap there is becomes less , precise , postdoc c:   Yeah , I also used I think something around zero point five seconds for the speech - nonspeech detector phd g: and it 's also based  Liz suggested that value based on the distribution of pause times that you see in Switchboard and and other corpora . postdoc c: In any case , this this  , meeting that I hand I I hand - adjusted two of them I mentioned before , phd g:   postdoc c: and I sent I sent email , phd g: OK , postdoc c: so phd g: So so at some point we will try to fine - tune our forced alignment postdoc c: And I sent the path . phd g: maybe using those as references because you know , what you would do is you would play with different parameters . And to get an object You need an objective measure of how closely you can align the models to the actual speech . So , I will  phd b: Yeah and hopefully the new meetings which will start from the channelized version will will have better time boundaries and alignments . postdoc c: But I like this idea of  , for our purposes for the for the IBM preparation , n having these joined together , phd b: Yeah . phd b: And and in in the in the previous version where in the n which is used now , there , the backchannel would would be in - between there somewhere , so . Well , phd b: but postdoc c: that 's that 's right , but you know , thi this brings me to the other f stage of this which I discussed with you earlier today , phd b: Yeah . postdoc c: which is the second stage is  , w what to do in terms of the transcribers adjustment of these data .  , the tr so the idea initially was , we would get  , for the new meetings , so the e EDU meetings , that Thilo ha has now presegmented all of them for us , on a channel by channel basis . And  , so , I 've assigned I 've I 've assigned them to our transcribers and  , so far I 've discussed it with one , with  And I had a about an hour discussion with her about this yesterday , we went through  EDU - one , at some extent . And it occurred to me that  that basically what we have in this kind of a format is you could consider it as a staggered mixed file , we had some discussion over the weekend a about at at this other meeting that we were all a at  , about whether the tran the IBM transcribers should hear a single channel audio , or a mixed channel audio . And  , in in a way , by by having this this chunk and then the backchannel after it , it 's like a stagal staggered mixed channel . And  , it occurred to me in my discussion with her yesterday that  , the the the maximal gain , it 's from the IBM people , may be in long stretches of connected speech . So it 's basically a whole bunch of words which they can really do , because of the continuity within that person 's turn . So , what I 'm thinking , and it may be that not all meetings will be good for this , but but what I 'm thinking is that in the EDU meetings , they tend to be driven by a couple of dominant speakers . And , if the chunked files focused on the dominant speakers , then , when when it got s patched together when it comes back from IBM , we can add the backchannels . It seems to me that  , you know , the backchannels per - se wouldn't be so hard , but then there 's this question of the time @ @  , marking , and whether the beeps would be  y y y And I 'm not exactly sure how that how that would work with the with the backchannels . And , so  And certainly things that are intrusions of multiple words , taken out of context and displaced in time from where they occurred , that would be hard . So , m my thought is i I 'm having this transcriber go through the EDU - one meeting , and indicate a start time f for each dominant speaker , endpoi end time for each dominant speaker , and the idea that these units would be generated for the dominant speakers , and maybe not for the other channels . grad a: Yeah the only , disadvantage of that is , then it 's hard to use an automatic method to do that . The advantage is that it 's probably faster to do that than it is to use the automated method and correct it . I think I I think  , you know , the original plan was that the transcriber would adjust the t the boundaries , and all that for all the channels but , you know , that is so time - consuming , and since we have a bottleneck here , we want to get IBM things that are usable s as soon as possible , then this seemed to me it 'd be a way of gett to get them a flood of data , which would be useful when it comes back to us . postdoc c: Oh also , at the same time she when she goes through this , she 'll be  If there 's anything that was encoded as a pause , but really has something transcribable in it , then she 's going to  , make a mark w  , so you know , so that that bin would be marked as it as double dots and she 'll just add an S . And in the other in the other case , if it 's marked as speech , and really there 's nothing transcribable in it , then she 's going to put a s dash , and I 'll go through and it and  , you know , with a with a substitution command , get it so that it 's clear that those are the other category . But  , the transcribable events that  , I 'm considering in this , continue to be laugh , as well as speech , and cough and things like that , so I 'm not stripping out anything , just just you know , being very lenient in what 's considered speech . Yeah ? phd d: Jane ? In terms of the this new procedure you 're suggesting , u what is the grad a: It 's not that different . phd d: So I 'm a little confused , because how do we know where to put beeps ? Is it i d y is it postdoc c: Oh , OK . postdoc c: So what it what it what it involves is is really a s  , the original pr procedure , but only applied to  , a certain strategically chosen s aspect of the data . grad a: We pick the easy parts of the data basically , postdoc c: So grad a: and transcriber marks it by hand . grad a: And because phd d: But after we 've done Thilo 's thing . Oh , OK , postdoc c: Yes ! grad a: I didn't I didn't understand that . phd g: So ,  grad a: OK , leave the mikes on , and just put them on the table . postdoc c: We start with the presegmented version grad a: Let me mark you as no digits . phd b: You start with the presegmentation , r yeah ? postdoc c: Yeah . And then  , the transcriber , instead of going painstakingly through all the channels and moving the boundaries around , and deciding if it 's speech or not , but not transcribing anything . OK ? Instead of doing that , which was our original plan , the tra They focus on the dominant speaker phd d:   So what they do is they identify who 's the di dominant speaker , and when the speaker starts . postdoc c: So  , you 're still gonna phd b: And you just postdoc c: So we 're It 's based on your se presegmentation , that 's the basic thing . phd b: and you just use the s the segments of the dominant speaker then ? For for sending to to IBM or ? postdoc c: Yeah . phd d: So , now Jane , my question is when they 're all done adjusting the w time boundaries for the dominant speaker , have they then also erased the time boundaries for the other ones ? postdoc c:   postdoc c: That 's that 's why she 's notating the start and end points of the dominant speakers . So , on a you know , so i in EDU - one , i as far as I listened to it , you start off with a a s section by Jerry . So Jerry starts at minute so - and - so , and goes until minute so - and - so . And he starts at minute such - and - such , and goes on till minute so - and - so . And then meanwhile , she 's listening to both of these guys ' channels , determining if there 're any cases of misclassification of speech as nothing , and nothing as speech , phd d:   phd d: So she does the adjustments on those guys ? postdoc c: But you know , I wanted to say , his segmentation is so good , that  , the part that I listened to with her yesterday didn't need any adjustments of the bins . So this is not gonna be a major part of the process , at least least not in not on ones that that really phd d: So if you don't have to adjust the bins , why not just do what it for all the channels ? postdoc c:  -  ? phd d: Why not just throw all the channels to IBM ? postdoc c: Well there 's the question o of whether Well , OK . She i It 's a question of how much time we want our transcriber to invest here when she 's gonna have to invest that when it comes back from IBM anyway . postdoc c: So if it 's only inserting "  -  "s here and there , then , wouldn't that be something that would be just as efficient to do at this end , instead of having it go through I B M , then be patched together , then be double checked here . But But then we could just use the the output of the detector , and do the beeping on it , and send it to I B phd d: Without having her check anything . postdoc c: Well , I guess grad a: I think we just we just have to listen to it and see how good they are . phd b: For some meetings , I 'm I 'm sure it i n postdoc c: I 'm I 'm open to that , it was professor f: Yeah , if it 's working well , phd b: That 's And some on some meetings it 's good . professor f: that sounds like a good idea since as you say you have to do stuff with the other end anyway .  the detector , this phd d: Yeah ,  we have to fix it when it comes back anyhow . postdoc c: Now , you were saying that they they differ in how well they work depending on channel s sys systems and stuff . So we should perhaps just select meetings on which the speech - nonspeech detection works well , postdoc c: But EDU is great . phd b: and just use , those meetings to to to send to IBM and , do the other ones .  , my my my impression is that it 's better for meetings with fewer speakers , and it 's better for for meetings where nobody is breathing . phd d: So in fact this might suggest an alternative sort of a a c a hybrid between these two things . Yeah ? phd d: So the the one suggestion is you know we we run Thilo 's thing and then we have somebody go and adjust all the time boundaries phd b: Yeah . phd d: There 's a a another possibility if we find that there are some problems , phd b: Yeah . phd d: and that is if we go ahead and we just run his , and we generate the beeps file , then we have somebody listen beeps file . And erase phd d: And they listen to each section and say " yes , no " whether that section is phd b: Yeah . And it just You know , there 's a little interface which will for all the " yes " - es it then that will be the final beep file . postdoc c: That 's interesting ! Cuz that 's that 's directly related to the e end task .  it wouldn't be that much fun for a transcriber to sit there , hear it , beep , yes or no . I don't know , I I think I 'm I 'm really tending towards grad a: One and a half times real time . professor f: what 's the worst that happens ? Do the transcribers  as long as th on the other end they can say there 's there 's something conventions so that they say " huh ? " phd d: Yeah . professor f: i i It i phd d: We can just catch it at the catch everything at this side . phd d: Well maybe that 's the best way to go , postdoc c: How interesting ! phd d: just grad a:  it just depends on how postdoc c: Well EDU phd b: Yeah , grad a: Sorry , go ahead . phd b: u u u postdoc c: So I was gonna say , EDU - one is good enough , phd b: Yeah . postdoc c: maybe we could include it in this in this set of  , this stuff we send . phd b: Yeah there 's I I think there are some meetings where it would would It 's possible like this . grad a: Yeah I I think , we won't know until we generate a bunch of beep files automatically , listen to them and see how bad they are . phd d: We won't be able to s include it with this first thing , grad a: If postdoc c:  . phd d: because there 's a part of the process of the beep file which requires knowing the normalization coefficients . phd d: Right , except I don't think that the c the instructions for doing that was in that directory , right ? I I didn't see where you had gener grad a: No , but it 's easy enough to do . phd b: What professor f: But I but I have a phd b: Doing the gain ? It 's no problem . professor f: But but but I I I have another suggestion on that , which is , since , really what this is , is is is trying to in the large , send the right thing to them and there is gonna be this this post - processing step , why don't we check through a bunch of things by sampling it ? phd d:   professor f: Right ? In other words , rather than , saying we 're gonna listen to everything grad a: I didn't mean listen to everything , I meant , just see if they 're any good . So y you do a bunch of meetings , you listen to to a little bit here and there , phd d: Yeah . professor f: if it sounds like it 's almost always right and there 's not any big problem you send it to them . professor f: And , you know , then they 'll send us back what we w what what they send back to us , postdoc c: Oh , that 'd be great . professor f: and we 'll we 'll fix things up and some meetings will cost more time to fix up than others . grad a: And we should just double - check with Brian on a few simple conventions on how they should mark things . grad a: Yeah , cuz @ @  what I had originally said to Brian was well they 'll have to mark , when they can't distinguish between the foreground and background , professor f: Yeah . But if we send them without editing , then we 're also gonna hafta have m  , notations for words that are cut off , phd d:   phd d: And they may just guess at what those cut - off words are , postdoc c: Yeah . phd d: but w  we 're gonna adjust everything when we come back grad a: But what what we would like them to do is be conservative so that they should only write down the transcript if they 're sure . postdoc c: which professor f: i Can I maybe have have an order of it 's probably in your paper that I haven't looked at lately , but postdoc c: Certainty . professor f: an order of magnitude notion of of how on a good meeting , how often  , do you get segments that come in the middle of words and so forth , and  in a bad meeting how often ? phd b:  . postdoc c: Was is it in a in a what what is the t professor f: Well he 's saying , you know , that the the EDU meeting was a good good meeting , postdoc c: In a good meeting , what ? phd b: Yeah . professor f: right ? postdoc c: Oh I see , professor f: and so so so it was almost it was almost always doing the right thing . And then ,  in a bad meeting , or p some meetings where he said oh he 's had some problems , what does that mean ? postdoc c:  - huh . professor f: So  does one of the does it mean one percent and ten percent ? Or does it mean five percent and fifty percent ? postdoc c: OK . professor f:  phd b: So professor f: Or Maybe percentage isn't the right word , postdoc c: Just phd b: Yeah th professor f: but you know how many how many per minute , or You know . phd b: Yeah , the the problem is that , nnn , the numbers Ian gave in the paper is just  , some frame error rate . So that 's that 's not really What will be effective for for the transcribers , is They have to yeah , in in they have to insure that that 's a real s spurt or something . So the speech the amount of speech that is missed by the detector , for a good meeting , I th is around or under one percent , I would say . For yeah , but there can be more There 's There 's more amount speech  , more amount of Yeah well , the detector says there is speech , but there is none . Now what about in a meeting that you said we 've you 've had some more trouble with ? phd b: I can't really hhh , Tsk . That 's really I I did this on on four meetings and only five minutes of of every meet of of these meetings so , it 's not not that representative , but , it 's perhaps , Fff .  Yeah , it 's perhaps then it 's perhaps five percent of something , which s  the the frames speech frames which are which are missed , but  , I can't can't really tell . So I So i Sometime , we might wanna go back and look at it more in terms of how many times is there a spurt that 's that 's  , interrupted ? phd b: Yeah . professor f: Something like that ? postdoc c: The other problem is , that when it when it  d i on the breathy ones , where you get breathing , inti indicated as speech . professor f: And phd b: So postdoc c: And I guess we could just indicate to the transcribers not to encode that if they We could still do the beep file . professor f: Yeah again I I think that that is probably less of a problem because if you 're if there 's If if a if a word is is split , then they might have to listen to it a few times to really understand that they can't quite get it . professor f: Whereas if they listen to it and there 's don't hear any speech I think they 'd probably just listen to it once . professor f: So there 'd you 'd think there 'd be a a factor of three or four in in , cost function , postdoc c: OK . phd b: Yeah , so but I think that 's n that really doesn't happen very often that that that a word is cut in the middle or something . professor f: So so what you 're saying is that nearly always what happens when there 's a problem is that is that  , there 's some  ,  nonspeech that  that is b interpreted as speech . phd b: Yeah , it 's professor f: You know , if they they hear you know , a dog bark and they say what was the word , they you know , they phd b: Yeah , I als I professor f: Ruff ruff ! phd b: Yeah I also thought of there there are really some channels where it is almost  , only bre breathing in it . I 've got a a P - a method with loops into the cross - correlation with the PZM mike , and then to reject everything which which seems to be breath . phd b: So , I could run this on those breathy channels , and perhaps throw out grad a: That 's a good idea . But I think I th Again , I think that sort of that that would be good , phd b: Yeah . professor f: But I think none of this is stuff that really needs somebody doing these these  , explicit markings . Oh , I 'd be delighted with that , I I was very impressed with the with the result . professor f: Yeah , cuz the other thing that was concerning me about it was that it seemed kind of specialized to the EDU meeting , and and that then when you get a meeting like this or something , phd b: Yeah . professor f: and and you have a b a bunch of different dominant speakers postdoc c: Oh yeah , interesting . professor f: Whereas this sounds like a more general solution postdoc c: Oh yeah , I pr I much prefer this , professor f: is postdoc c: I was just trying to find a way Cuz I I don't think the staggered mixed channel is awfully good as a way of handling overlaps . phd d: And we can just , you know , get the meeting , process it , put the beeps file , send it off to IBM . I would phd d: Do what ? phd b: listen to it , and then grad a: Or at least sample it . professor f: make sure you don't send them three hours of " bzzz " or something . phd b: And there 's there 's one point which I  yeah , which which I r we covered when I when I r listened to one of the EDU meetings , professor f: Great . grad a:  - huh phd b: And i the speech - nonspeech detector just assigns randomly the speech to to one of the channels , so .  - I haven't - I didn't think of of s of this before , grad a: What can you do ? phd b: but what what shall we do about s things like this ? postdoc c: Well you were suggesting You suggested maybe just not sending that part of the meeting . postdoc c: But phd b: But , sometimes the the the laptop is in the background and some somebody is is talking , and , that 's really a little bit confusing , but grad a: It 's a little bit confusing . phd b: that 's that 's a second question , " what what will different transcribers do with with the laptop sound ? " postdoc c: Would you would professor f: What was the l what was the laptop sound ? postdoc c: Yeah , go ahead . postdoc c: Well , so  So my standard approach has been if it 's not someone close - miked , then , they don't end up on one of the close - miked channels . phd b: when thi when this is sent to to the I M - eh , I B M transcribers , I don't know if if they can tell that 's really postdoc c: Yeah , that 's right . grad a:  postdoc c: Well , they have a convention , in their own procedures , which is for a background sound . grad a: Right , but , in general I don't think we want them transcribing the background , cuz that would be too much work . grad a: Right ? For it because in the overlap sections , then they 'll phd d: Well I don't think Jane 's saying they 're gonna transcribe it , but they 'll just mark it as being there 's some background stuff there , grad a: But that 's gonna be all over the place . phd d: right ? grad a: How w how will they tell the difference between that sort of background and the dormal normal background of two people talking at once ? phd b: Yeah . postdoc c: Oh , I think I think it 'd be easy to to say " background laptop " . grad a: How would they know that ? phd d: But wait a minute , why would they treat them differently ? phd b: Yeah . postdoc c: Well because one of them grad a: Because otherwise it 's gonna be too much work for them to mark it . postdoc c: Oh , I s background laptop or , background LT wouldn't take any time . grad a: Sure , but how are they gonna tell bet the difference between that and two people just talking at the same time ? postdoc c: And phd b: Yeah . Acoustically , can't you tell ? phd b: It 's really good sound , so postdoc c: Oh is it ? Oh ! professor f: Well , isn't there a category something like  , " sounds for someone for whom there is no i close mike " ? phd b: Yeah that would be very important , grad a: But how do we d how do we do that for the I B M folks ? postdoc c: Yeah . grad a: How can they tell that ? phd d: Well we may just have to do it when it gets back here . grad a: And they 'll just mark it however they mark it , postdoc c: That sounds good . postdoc c: Well , as it comes back , we have a  when we can use the channelized interface for encoding it , then it 'll be easy for us to handle . postdoc c: But but if if out of context , they can't tell if it 's a channeled speak  , you know , a close - miked speaker or not , then that would be confusing to them . postdoc c: I don't know , I it doesn't I don't Either way would be fine with me , I don't really care . Do you think we should send the  that whole meeting to them and not worry about pre - processing it ? professor f: Yes ma ' postdoc c: Or  , what  is we we should leave the part with the audio in the  , beep file that we send to IBM for that one , or should we start after the that part of the meeting is over in what we send . professor f: Which part ? phd b: With postdoc c: So , the part where they 're using sounds from their from their laptops . phd b: with the laptop sound , or ? just postdoc c: w If we have speech from the laptop should we just  , excise that from what we send to IBM , or should we i give it to them and let them do with it what they can ? phd d: I think we should just it it 's gonna be too much work if we hafta worry about that I think . postdoc c: And give them freedom to to indicate if it 's just not workable . professor f: Cuz , I wouldn't don't think we would mind having that transcribed , if they did it . grad a: I think phd d: Yeah , e grad a: As I say , we 'll just have to listen to it and see how horrible it is . grad a:  -  , and and they 're very it 's very audible ? on the close - talking channels ? phd b: What what I would Yeah . postdoc c: OK , so we read the transcript number first , right ? grad a: Are we gonna do it altogether or separately ? phd b: So What time is it ? professor f: why don't we do it together , postdoc c: quarter to four . professor f: One , two , three , go ! postdoc c: It 's kind of interesting if there 're any more errors in these , than we had the first set . phd d: Do you guys plug your ears when you do it ? grad a: I do . professor f: I haven't been , phd d: How can you do that ? professor f: no . phd b: Perhaps there are lots of errors in it phd d: Gah ! grad a: Total concentration . Are you guys ready ? phd d: You hate to have your ears plugged ? professor f: Yeah


phd e: What am I ? professor f: Little low ? phd e: Channel four ? professor f: Channel five . phd g: The gai the gain 's up at it what it usually is , professor f: Is it ? phd g: but if you think it 's Yeah . phd d: I think phd g: She can just walk in , I guess , or phd d: Yeah . I was gonna ask Adam to , say if he thought anymore about the demo stuff because it occurred to me that this is late May and the DARPA meeting is in mid July .  , but I don't remember w what we I know that we were gonna do something with the transcriber interface is one thing , but I thought there was a second thing . Anybody remember ? phd g: Well , we were gonna do a mock - up , like , question answering or something , I thought , that was totally separate from the interface . Do you remember ? Remember , like , asking questions and retrieving , but in a pre - stored fashion . phd g: That was the thing we talked about , I think , before the transcriber professor f: Yeah . So anyway , you have to sort out that out and get somebody going on it cuz we 're got a got a month left basically . phd g: I was writing the digits and then I realized I could xerox them , professor f: Oh , oh . professor f: So , the choice is , which which do we want more , the the the comparison , of everybody saying them at the same time or the comparison of people saying the same digits at different times that ? phd g: It 's just cuz I didn't have any more digit sheets .  , we could use them for normalizing or something , but it of course goes more quickly doing them in unison . phd e: But anyway , they won't be identical as somebody is saying zero in some sometimes , you know , saying O , and so , it 's not i not identical .  ju what what might those be ? phd d: IBM stuff and , just getting  , meeting information organized . phd c: Are you implying that it 's currently disorganized ? phd d: In my mind . professor f: Is there stuff that 's happened about , the SRI recognizer et cetera , tho those things that were happening before with ? phd c: Well . professor f: Y y you guys were doing a bunch of experiments with different front - ends and then with Is is that still sort of where it was , the other day ? phd c: We 're improving . phd d: Now the the You saw the note that the PLP now is getting basically the same as the MFCC .  , it seems It looks l I haven't The It 's The experiment is still not complete , but , it looks like the vocal tract length normalization is working beautifully , actually , w using the warp factors that we computed for the SRI system and just applying them to the ICSI front - end . phd c: Just had to take the reciprocal of the number because they have different meanings in the two systems .  phd c: But one issue actually that just came up in discussion with Liz and and Don was , as far as meeting recognition is concerned , we would really like to , move , to , doing the recognition on automatic segmentations . phd c: Because in all our previous experiments , we had the  , you know , we were essentially cheating by having the , you know , the h the hand - segmentations as the basis of the recognition . phd c: And so now with Thilo 's segmenter working so well , I think we should consider doing a phd e: Mmm . phd g: And even The good thing is that since you , have high recall , even if you have low precision cuz you 're over - generating , that 's good because we could train noise models in the recognizer for these kinds of , transients and things that come from the microphones , phd c: Right . phd g: but I know that if we run recognition unconstrained on a whole waveform , we do very poorly because we 're we 're getting insertions in places what that you may well be cutting out . phd c: We should we should consider doing some extra things , like , you know , retraining or adapting the the models for background noise to the to this environment , for instance . phd g: And , yeah , using Thilo 's , you know , posteriors or some kind of or phd c: So . So , talked with Brian and gave him the alternatives to the single beep at the end of each utterance that we had generated before . And so he talked it over with the transcriber and the transcriber thought that the easiest thing for them would be if there was a beep and then the nu a number , a digit , and then a beep , at the beginning of each one professor f: Yeah . And , so Adam wrote a little script to generate those style , beeps phd c: Where 'd you get the digits from ? phd d: and so we 're I came up here and just recorded the numbers one through ten . phd c: And do you splice them into the waveform ? Or ? phd d: Yeah . He then he d I recorded Actually , I recorded one through ten three times at three different speeds and then he picked . phd d: He liked the fastest one , so he just cut those out and spliced them in between , two beeps . phd d: Does it ? phd e: It will be funny when you 're really reading digits , and then there are the chunks with with your digits in ? phd d: Yeah . postdoc a: Now actually , phd d: That 'll throw them , postdoc a: we 're Are we handling ? phd d: huh ? professor f: maybe we should have you record A , B , C for those or something . And she said it wasn't gonna the transcriber said it wouldn't be a problem cuz they can actually make a template , that has beep , number , beep . phd d: So , we We 're gonna send them one more sample meeting , and Thilo has run his segmentation . And when we get that back we 'll see if that sort of fixes the problem we had with , too many beeps in the last transcription . Do w do what Do you have any idea of the turn - around on on those steps you just said ? phd g: Great . professor f: e e u u The reason I 'm asking is because , Jane and I have just been talking , and she 's just been doing . professor f: And so we don't sort of really know exactly what they 'll be doing , how long they 'll be doing it , and so forth , because right now she has no choice but to operate in the mode that we already have working . professor f: And , so it 'd be It 'd be good to sort of get that resolved , soon as we could , phd d: Yeah . I Yeah , I I hope @ @ we can get a better estimate from this one that we send them .   in particular I would I would really hope that when we do this DARPA meeting in July that we sort of have we 're we 're into production mode , somehow phd d:   professor f: You know , that we we actually have a stream going and we know how how well it does and how and how it operates . Maybe before we do the meeting info organize thing , maybe you could say relevant stuff about where we are in transcriptions . So , we  , the transcribers have continued to work past what I 'm calling " set one " , which was the s the set that I 've been ,  OK , talking about up to this point , but , they 've gotten five meetings done in that set . I 'm thinking of hiring another one , which will because we 've had a lot of attrition . And that will bring our total to professor f: They die off after they do this for a while .  , you know , one of them really w wasn't planning phd c: Oh , that was an unfor unforeseen side effect of postdoc a: Eh , one of them , had never planned to work past January .  , it 's th all these various things , cuz we , you know , we presented it as possibly a month project back in January and and and and  , so it makes sense .  , through attrition we we 've we 're down to to two , but they 're really solid . We had a good core phd g: Well , they won't hear this since they 're going .  , I d it 's just a matter of we w we 're we 've got , professor f: No backs . postdoc a: two of the ones who who , ha had been putting in a lot of hours up to this point and they 're continuing to put in a a lot of hours , which is wonderful , and excellent work . And so , then , in addition , I hired two more today and I 'm planning to h hire a third one with this within this coming week , but but the plan is just as , Morgan was saying we discussed this , and the plan right now is to keep the staff on the on the leaner side , you know , rather than hiring , like , eight to ten right now , professor f:   postdoc a: because if the IBM thing comes through really quickly , then , we wouldn't wanna have to , you know , lay people off and stuff . And this way it 'll  , I got really a lot of response for for my notice and I think I could hire additional people if I wish to . An - and the other thing is , in the unlikely event and since we 're so far from this , it 's a little hard to plan this way in the unlikely event that we actually find that we have , transcribers on staff who are twiddling their thumbs because , you know , there 's , you know , all the stuff that that was sitting there has been transcribed and they 're and they 're faster the the pipeline is faster than  , than the generation , eh , i in in the day e event that that day actually dawns , I I bet we could find some other stuff for them to do . professor f: So I I think that , eh , eh , a as we were talking , if we if we hire twelve , then we could , you know , run into a problem later . But but ,  for all sorts of reasons but if we hire f you know , f we have five on staff five or six on staff at any given time , then it 's a small enough number so we can be flexible either way . phd g: It 'd be great , too , if , we can we might need some help again getting the tighter boundaries or some hand to experiment with ,  you know , to have a ground truth for this segmentation work , which I guess you have some already that was really helpful , and we could probably use more . That was a thing I I planned working on , is , to use the the transcriptions which are done by now , and to to use them as ,  phd g: Oh . To to create some speech - nonspeech labels out of them , and Yeah , but that that 's a thing w was w what I 'm just looking into . postdoc a: The the the pre - segmentations are so much are s so extremely helpful . Now there was , I g guess So , a couple weeks ago I needed some new ones and it happened to be during the time that he was on vacation f for just very few days you were away . postdoc a: so I so I started them on the non - pre - segmented and then switched them over to yours and , they ,  you know , they always appreciate that when they have that available . postdoc a: And and and she was And so , I asked her  , They 're very perceptive . I haven't done it yet , but I wanna do that and she 's out of town , for a couple of weeks , but I wanna do that when she returns .  , cuz she was saying , you know , in a in a span of very short period we asked It seems like the ones that need to be adjusted are these these these things , and she was saying the short utterances , the ,  phd g:  . But but actually i it 's so correct for so much of the time , that it 's an enormous time saver phd e: Yeah . phd g: Is there actually a record of where they change ?  , you can compare , do a diff on the just so that we knew postdoc a: You could do it . It 's it 's complicated in that  , hhh , i hhh , i phd e: Yeah . Actually , when when they create new yeah , new segments or something , it will be , not that easy but  . phd g: just so that if we run it we know whether we 're which ones were cheating phd e: Yeah . phd g: and postdoc a: There is a there is one problem with that and that is when they start part way through then what I do is I merge what they 've done with the pre - segmented version . Wha - what you 'd really like is that they started with pre - segmented and were pre - segmented all the way through . postdoc a: And ,  @ @ I ,  the it wasn't possible for about four of the recent ones . As long as we have a record , I guess , of the original automatic one , we can always find out how well we would do fr from the recognition side by using those boundaries . phd g: Also if you need someone to record this meeting , I 'm happy to for the transcribers I could do it , or Chuck or Adam . So , Jane and Adam and I had a meeting where we talked about the reorganization of the directory structure for all of the meeting professor f: Did you record it ? phd d: No . And then , Jane also s prepared a  , started getting all of the the meetings organized , so she prepared a a spreadsheet , which I spent the last couple of days adding to . So I went through all of the data that we have collected so far , and have been putting it into , a spreadsheet with start time , the date , the old meeting name , the new meeting name , the number of speakers , the duration of the meeting , comments , you know , what its transcription status is , all that kind of stuff . And so , the idea is that we can take this and then export it as HTML and put it on the Meeting Recorder web page so we can keep people updated about what 's going on . phd d: I 've gotta get some more information from Jane cuz I have some some gaps here that I need to get her to fill in , but so far , as of Monday , the fourteenth , we 've had a total number of meeting sixty - two hours of meetings that we have collected .  , and I 'm gonna have on here the total amount that 's been transcribed so far , but I 've got a bunch of  , that 's what I have to talk to Jane about , figuring out exactly which ones have have been completed and so forth . But , this 'll be a nice thing that we can put up on the the web site and people can be informed of the status of various different ones . And it 'll also list , like under the status , if it 's at IBM or if it 's at ICSI , or if it 's completed or which ones we 're excluding and and there 's a place for comments , so we can , say why we 're excluding things and so forth . professor f: Now would the ones that , are already transcribed we h we have enough there that c you know , we 've already done some studies and so forth and  , shouldn't we go through and do the business - es u of of having the , participants approve it , for approve the transcriptions for distribution and so forth ? postdoc a: interesting idea . In principle , I I would say yes , although I still am doing some the final - pass editing , trying to convert it over to the master file as the being the channelized version and it 's Yeah , it seems like I get into that a certain way and then something else intervenes and I have to stop . Cleaning up the things like the , places where the transcriber was uncertain , and and doing spot - checking here and there . So , I guess it would make sense to wait until th that 's done , but but professor f: Well , le let me put in another sort of a milestone kind of as as I did with the ,  the the pipeline . professor f: we are gonna have this DARPA meeting in the middle of July , postdoc a: Yes . professor f: and I think it w it 'd be given that we 've been we 've given a couple public talks about it already , spaced by months and months , I think it 'd be pretty bad if we continued to say none of this is available . So we can s we we wanna be able to say " here is a subset that is available right now " postdoc a:   phd c: And they don't have to approve , you know , th an edited version , they can just give their approval to whatever version postdoc a: Well , maybe professor f: Well , in principle , yes . But , i if if if somebody actually did get into some legal issue with it then we phd c: Bu Yeah . Presumably if if s errors are found , they will be fixed , but they won't change the the content of the meetings . phd g: Well , i if Jane is clarifying question question , then , you know , how can they agree to it before they know her final version ? postdoc a: The other thing , too , is there can be subtleties where a person uses this word instead of that word , which @ @ could 've been transcribed in the other way . phd g: Thing postdoc a: And no and they wouldn't have been slanderous if it had been this other word . You know ? professor f: I it you know , there there is a point at which I agree it becomes ridiculous because , you know , you could do this final thing and then a year from now somebody could say , you know , that should be a period and not a question mark . Right ? And you don't you there 's no way that we 're gonna go back and ask everybody " do you approve this , you know this document now ? " So So I think what it is is that the the the the thing that they sign I I haven't looked at it in a while , but it has to be open enough that it sort of says " OK , from now on you know , now that I 've read this , you can use do anything you want with these data . professor f: And ,  But , i I think we wanna So , assuming that it 's in that kind of wording , which I don't remember , I think i we just wanna have enough confidence ourselves that it 's so close to the final form it 's gonna be in , a year from now that they 're postdoc a:   It 's just , a question of , if if the person is using the transcript as the way of them judging what they said and whether it was slanderous , then it seems like it 's it 's i it needs to be more correct than if we could count on them re - listening to the meeting . postdoc a: Because it becomes , eh , in a way a a f  , a legal document i if they 've agreed to that . I forget how we ended up on this , but I remember my taking the position of not making it so so easy for everybody to observe everything and Adam was taking the position of of having it be really straightforward for people to check every aspect of it including the audio . And I don't remember who won , Adam or me , but postdoc a: Well , if it 's only the transcript , though  , th this this is my point , that that professor f: the  , that that 's why I 'm bringing this up again , because I can't remember how we ended up . postdoc a: then it becomes professor f: That it was the transcrip He wanted to do a web interface that would make it postdoc a: Well , if it 's just the audio Well . phd g: with the web interface it 's interesting , because you could allow the person who signs to be informed when their transcript changes , or something like that . Like , I don't wanna know , but some people might be really interested and then y In other words , they would be informed if there was some significant change other than typos and things like that . professor f: You decided you were whispering Satanic incantations under your breath when you were phd g: Well , I don't know what happened to the small heads thing , but I j  , I 'm just saying that , like , you know , you can sort of say that any things that are deemed professor f: They disappeared from view .  , I agree that at some point people probably won't care about typos but they would care about significant meaning changes and then they could be asked for their consent , I guess , if if those change . Cuz assumi assuming we we don't really distribute things that have any significant changes from what they sign anyway . phd c: Tha That 's How about having them approve the audio and not the transcripts ? phd g: Oh , my God . grad b: That 's phd c: We just have to give them a chance to listen to it , and if they don't , that 's their problem . phd g: You you d That 's like postdoc a: Unfortunately , in in the sign thing that they signed , it says " transcripts " . postdoc a: " You 'll be you 'll be provided the transcripts when they 're available . phd g: that 's a lot to ask for people that have been in a lot of meetings . professor f: W anyway , haven't we we 've gone down this path a number of times . I know this can lead to extended conversations and and not really get anywhere , so let let me just suggest that  , off - line that , the people involved figure it out and take care of it before it 's July . So so that in July we can tell people " yes , we have this and you can use it " . phd g: it 's I guess one thing we 're learning is that the amount We have eight meetings there because we couldn't use the non - native all non - native meetings and it 's , well , probably below threshold on enough data for us for the things we 're looking at because the prosodic features are very noisy and so you you need a lot of data in order to model them .  , so we 're starting to see some patterns and we 're hoping that maybe with , I don't know , double or triple the data with twenty meetings or so , that we would start to get better results . But we did find that some of the features that , I gue Jane would know about , that are expressing sort of the distance of , boundaries from peaks in the utterance and some local , range pitch range effects , like how close people are to their floor , are showing up in these classifiers , which are also being given some word features that are cheating , cuz they 're true words . Right ? phd g: spurts is not cheating except that of course you know the real words , grad b: Right . phd c: No phd g: Not exactly , but i grad b: But ra somewhat ? professor f: On the average . Well , we don't know and actually that 's one of the things we 're interested in doing , is a sort of professor f:  - huh . phd c: Have you tried using just time , as opposed to number of words ? phd g: So . grad b: I think ti  Just p time position , like when the word starts ? phd c: Yeah . grad b: I don't know if that was in the phd c: Well , no ,  t time time position relative to the beginning of the spurt . phd g: we didn't try it , but it 's s grad b: Like , there 's a lot of different features you could just pull out . professor f: How about time position normalized by speak phd g: And it depends on speaking rate professor f: Yeah . phd g: But we one of the interesting things was I guess you reported on some te punctuation type grad b: Yeah . phd g: finding sentence boundaries , finding disfluency boundaries , and then I had done some work on finding from the foreground speech whether or not someone was likely to interrupt , so where you know , if I 'm talking now and someone and and Andreas is about to interrupt me , is he gonna choose a certain place in my speech , either prosodically or word - based . And there the prosodic features actually showed up and a neat thing even though the word features were available . And a neat thing there too is I tried some putting the speaker So , I gave everybody a short version of their name . So that means that overall , it wasn't just modeling Morgan , or it wasn't just modeling a single person , professor f:   phd g: but was sort of trying to , get a general idea the model the tree classifier was trying to find general locations that were applicable to different speakers , even though there are huge speaker effects . The but the main limitation now is I because we 're only looking at things that happen every ten words or every twenty words , we need more more data and more data per speaker . It 'd also be interesting to look at the EDU meetings because we did include meeting type as a feature , so whether you were in a r Meeting Recorder meeting or a Robustness meeting did matter to interrupts because there are just fewer interrupts in the Robustness meetings . phd g: And so the classifier learns more about Morgan than it does about sort of the average person , professor f:   phd g: So it 's And I think Don ,  Well , we have a long list of things he 's starting to look at now over the summer , where we can And he 'll be able to report on more things in the future . But it was great that we could at least go from the you know , Jane 's transcripts and the , recognizer output and get it to this point . And I think it 's something Mari can probably use in her preliminary report like , " yeah , we 're at the point where we 're training these classifiers and we 're just reporting very preliminary but suggestive results that some features , both word and pro prosodic , work . " The other thing that was interesting to me is that the pitch features are better than in Switchboard . And I think that really is from the close - talking mikes , cuz the pitch processing that was done has much cleaner behavior than than the Switchboard telephone bandwidth . Well , first of all , the pitch tracks are m have less , halvings and doublings than than Switchboard and there 's a lot less dropout , so if you ask how many regions where you would normally expect some vowels to be occurring are completely devoid of pitch information , professor f:    phd g: in other words the pitch tracker just didn't get a high enough probability of voicing for words for for , you know , five word professor f:  . So the missing We had a big missing data problem in Switchboard and , so the features weren't as reliable cuz they were often just not available . phd d: Could it have to do with the the lower frequency cut - off on the Switchboard ? phd g: So that 's actually good .  , the tele we had telephone bandwidth for Switchboard and we had the an annoying sort of telephone handset movement problem that I think may also affect it . phd g: Anyway , Don 's been doing a great job and we hope to continue with , Andreas 's help and also some of Thilo 's help on this , professor f: Great . phd e: Y phd g: to to try to get a non - cheating version of how all this would work . professor f: Has has ,  ? We just I think , just talked about this the other day , but h has has anybody had a chance to try changing , insertion penalty sort of things with the with the , using the tandem system input for the ? phd c: Oh , yeah . phd c: There were a little the relative number of I think there were a higher number of deletions , actually . phd c: So , you ,  So , actually it it preferred to have a positive er , negative insertion penalty , phd g: Deletions ? phd c: which means that ,  professor f:  - huh . phd c: But , you know , it didn't change th the by adjusting that the ,  professor f: OK . But , you know , given that that word error rate is so high , that 's not a professor f: OK . phd c: But , we s just ,  you know , Chuck and I talked and the @ @ next thing to do is probably to tune the  , the size of the Gaussian system , @ @ to to this to this feature vector , which we haven't done at all . phd c: And , for instance , Dan @ @ Dan just sent me a message saying that CMU used , something like ten Gaussians per cluster You know , each each mixture has ten Gaussians phd d:   We 're using sixty - four , phd c: and and we 're using sixty - four , phd d: right ? phd c: so that 's obviously a big difference phd d: Yeah . phd c: and it might be way off and give very poorly trained , you know , Gaussians that way , professor f:  . So so , we have The turn - around time on the training when we train only the a male system with , you know , our small training set , is less than twenty - four hours , so we can run lots of  , basically just brute force , try a whole bunch of different  , settings . professor f: huh ? phd c: But the PLP features work  , you know , continue to improve the , professor f: OK . phd c:  As I said before , the  using Dan 's , vocal tract normalization option works very well . So , @ @ I ran one experiment where we 're just did the vocal tract le normalization only in the test data , professor f:   phd c: so I didn't bother to retrain the models at all , and it improved by one percent , which is about what we get with  , with , you know , just @ @ actually doing both training and test normalization , with , the , with the standard system . So , in a few hours we 'll have the numbers for the for retraining everything with vocal tract length normalization and So , that might even improve it further . phd c: So , it looks like the P L - fea P features do very well now with after having figured out all these little tricks to to get it to work . So you mean you improve one percent over a system that doesn't have any V T L in it already ? phd c: Exactly . And and what that suggests also is of course that the current Switchboard MLP isn't trained on very good features . phd c: because it was trained on whatever , you know , was used , last time you did Hub - five stuff , which didn't have any of the professor f: Right . professor f: Right ?  , y the phd c: Well , but if you add them all up you have , almost five percent difference now . phd c: actually , and it 's , What 's actually qu interesting is that with  , well , you m prob maybe another half percent if you do the VTL in training , and then interestingly , if you optimize you get more of a win out of rescoring the , the N best lists , and optimizing the weights ,  than phd d: Than you do with the standard ? phd c: Yeah . But the part that 's actually adjustment of the front - end per se as opposed to doing putting VTLN in or something is it was a couple percent . professor f: Right ? It was it was there was there was one thing that was one and a half percent and one that was point eight . One of them was , the change to ,  because it did it all at once , to  , from bark scale to mel scale , phd c:   professor f: which I really feel like saying in quotes , because @ @ they 're essentially the same scale but the but but but any i individual particular implementation of those things puts things in a particular place . professor f: So that 's why I wanted to look I still haven't looked at it yet . professor f: and it it 's probably something like there 's one fewer or one more filter in the sub one kilohertz band phd c:   professor f: it could be there 's something more fundamental but it you know , I I don't know it yet . And the other and the other that was like one and a half or something , and then there was point eight percent , which was what was the other thing ? phd d: Well , that was combined with the triangular . professor f: We d weren't able to separate them out cuz it was just done in one thing . So that was that was ,  that one I can claim credit for , i in terms of screwing it up in the first place . So that someone e until someone else fixed it , which is that , I never put when I u We had some problems before with offsets . professor f: So we we had ,  ea everybody else who was doing Wall Street Journal knew that there were big DC offsets in th in these data in those data and and and nobody happened to mention it to us , phd c:  . professor f: and we were getting these , like , really terrible results , like two , three times the error everybody else was getting . And then in casual conversation someone ment mentioned "  , well , I guess , you know , of course you 're taking care of the offsets . professor f: And at that point , you know , we were pretty new to the data and we 'd never really , like , looked at it on a screen and then when we just put it on the screen and wroop ! phd c:   So , in PLP phd g: There was a like a hum or some or when they recorded it ? professor f: No . It 's just , it it 's it 's not at all uncommon for for recorded electronics to have different , DC offsets . It 's you know , you could have ten , twenty , maybe thirty millivolts , whatever , and it 's consistently in there . The thing is , most people 's front - ends have pre - emphasis with it , with zero at zero frequency , so that it 's irrelevant . We had we had the equivalent of pre - emphasis in a a , Fletcher - Munson style weighting that occurs in the middle of P L but it doesn't actually have a zero at zero frequency , phd g:  . So at that point I reali " oh sh we better have a have a high - pass filter " just , you know just take care of the problem . So I put in a high - pass filter at , I think ninety ninety hertz or so  , for a sixteen kilohertz sampling rate . And so well , so , you know , the code doesn't know anything about that and so this is all at eight kilohertz and so it was at forty - five hertz instead of at instead of at ninety . professor f: So , I don't know if Dan fixed it or or , what he phd c: Well , he made it a parameter . Yeah , I guess if he did it right , he did fix it and then and then it 's taking care of sampling rate , which is great . phd d: What what is the parameter ? professor f: He had a phd d: Is it , just the f lower cut - off that you want ? phd c: It 's called , H - HPF . phd c: u And but HPF , you know , when you put a number after it , uses that as the hertz value of the cut - off . professor f: frankly , we never did that with the RASTA filter either , phd c:   professor f: so the RASTA filter is actually doing a different thing in the modulation spectral domain depending on what sampling rate you 're doing , which is another old old bug of mine . So that that was the problem there was th we we we had always intended to cut off below a hundred hertz phd c:   Well , but ,  Well , again , after completing the current experiments , we 'll we can add up all the  differences professor f: Oh , yeah . phd c: and and an professor f: But but , I guess my my point was that that , the hybrid system thing that we did was , primitive in many ways . professor f: And I think I agree with you that if we fixed lots of different things and they would all add up , we would probably have a a a competitive system . professor f: unless you call well , if you call VTL the front - en front - end , that 's , a little more . phd d: One experiment we should we 'll probably need to do though when  , at some point , is , since we 're using that same the net that was trained on PLP without all these things in it , for the tandem system , we may wanna go back and retrain , professor f: Right ? But . phd c: eh At this point I 'm as  , you know e I 'm wondering is it Can we expect , a tandem system to do better than a properly trained you know , a Gaussian system trained directly on the features with , you know , the right ch choice of parameters ? professor f: Well , that 's what we 're seeing in other areas . Right ? So , it 's so ,  phd d: So , we But but we may not . professor f: the reason to think it should is because you 're putting in the same information and you 're transforming it to be more discriminative . Now the thing is , in some databases I wouldn't expect it to necessarily give you much and and part of what I view as the real power of it is that it gives you a transformational capability for taking all sorts of different wild things that we do , not just th the standard front - end , but other things , like with multiple streams and so forth , phd c:   professor f: If it 's , you know , like way way worse then , you know phd c: Right . phd d: So , Morgan , an another thing that Andreas and I were talking about was , so @ @ in the first experiment that he did we just took the whole fifty - six , outputs and that 's , basically compared to a thirty - nine input feature vector from either MFCC or PLP . phd d: But one thing we could do is professor f: Let let me let me just ask you something . When you say take the fifty - six outputs , these are the pre final nonlinearity outputs phd d: Yeah . And so so then you u Do you use all fifty - six of the KLT phd d: That 's what we did . professor f: or ? phd d: Right ? So one thing we were wondering is , if we did principal components and , say , took out just thirteen , and then did deltas and double - deltas on that professor f: OK . professor f:  these were all different databases and different you know , in HTK and all that , phd d: Yeah . But my recollection of it was that it didn't make it better but it didn't make it worse . professor f: But , again , given all these differences , maybe it 's more important in your case that you not take a lot of these low - variance , components . phd d: Cuz in a sense , the net 's already got quite a bit of context in those features , professor f: Yeah . phd d: so if we did deltas and double - deltas on top of those , we 're getting sort of even more . phd c: But there the main point is that , you know , it took us a while but we have the procedure for coupling the two systems debugged now and  , there 's still conceivably some bug somewhere in the way we 're feeding the tandem features  , either generating them or feeding them to this to the SRI system , phd d:   phd c: but it 's professor f: There might be , cuz that 's a pretty big difference . phd c: I 'm actually f quite sure that the feeding the features into the system and training it up , professor f: What if ? phd c: that that I think that 's this that 's essentially the same as we use with the ce with the P L P fe features . phd c: There we could the another degree of freedom is how do you generate the K L T transform ? phd d:   professor f: These nets are trained with particular normalization and when that gets screwed up it it can really hurt it . phd d: I 'm doing what Eric E Eric coached me through then that part of it , so I 'm pretty confident in that . phd d: the only slight difference is that I use normalization values that , Andreas calculated from the original PLP , phd c: Right . So , I u I do Oh , we actually don't do that normalization for the PLP , do we ? For the st just the straight PLP features ? phd c: No . Well , you might e e phd c: So , there 's there is there is room for bugs that we might not have discovered , phd d: So that 's that 's another Yeah . I I would actually double check with Stephane at this point , phd c: but professor f: cuz he 's probably the one here  , he and Dan are the ones who are at this point most experienced with the tandem phd d:   professor f: thing and there may there may be some little bit here and there that is not not being handled right .  , you can't just , like , print the the values out in ASCII and , you know , look at them , see if they 're professor f: Not unless you had a lot of time phd g: Well professor f: and phd g: eh , and also they 're not  , as I understand it , you you don't have a way to optimize the features for the final word error . phd g: these are just discriminative , but they 're not , optimized for the final phd c: They 're optimized for phone discrimination , not for phd g: Right . So it there 's always this question of whether you might do better with those features if there was a way to train it for the word error metric that you 're actually that you 're actually professor f: That 's right . It 's indirect , so you don't know professor f: So wha w what an and you may not be in this case , come to think of it , because , you 're just taking something that 's trained up elsewhere . So in fact you the the net is trained on , a , Viterbi alignment of the training data that comes from your full system . And so that 's where the feedback comes all around , so that it is actually discriminant . You can prove that it 's it 's a ,  If you believe in the Viterbi assumption that , getting the best path , is almost equivalent to getting the best , total probability , then you actually do improve that by ,  by training up on local local ,  local frames . But , we aren't actually doing that here , because we did we did that for a hybrid system , and now we 're plugging it into another system and so it isn't i i i it wouldn't quite apply here . phd c: Do y phd d: So another huge experiment we could do would be to take the tandem features , do SRI forced alignments using those features , and then re - do the net with those . Another thing is since you 're not using the net for recognition per se but just for this transformation , it 's probably bigger than it needs to be . So , you 're using a l a long a larger phone set than what phd d: Mmm . professor f: The other thing , just to mention that Stephane this was an innovation of Stephane 's , which was a pretty neat one , and might particularly apply here , given all these things we 're mentioning . Even the local ones , given , you know , these potential outer loops which , you know , you can convince yourself turn into the global ones .  , when something about the test set is different enough from the training set that that , the discrimination that you 're learning is is is not a good one . professor f: So , his idea was to take as the input feature vector to the , Gaussian mixture system , a concatenation of the neural net outputs and the regular features . No , but we we when when we when I first started corresponding with Dan about how to go about this , I think that was one of the things that we definitely went there .  , I 'm sure that Stephane wasn't the first to think of it , phd c: Yeah . So that 's that that 's our current best best system in the , in the Aurora thing . phd c: And do you do a KLT transform on the con on the combined feature vector ? professor f: Yeah . phd c: Do you d you do a KLT transform on the combined feature vector ? professor f: Yeah . professor f: Well , actually , I ,  you should check with him , because he tried several different combinations . phd c: Because you end up with this huge feature vector , so that might be a problem , a unless you do some form of dimensionality reduction . So he did one where he put o put e the whole thing into one KLT , and another one , since the the PLP things are already orthogonalized , he left them alone and and just did a KLT on the on the on the net outputs phd c:   phd d: Did he did he try to ? So he always ended up with a feature vector that was twice as long as either one of the ? professor f: No . phd g: We need to close up cuz I need to save the data and , get a call . phd g: or , do them together ? professor f: I I g I think , given that we 're in a hurry for snacks , maybe we should do them together . professor f: Well , it 's it 's it 's not You know , it 's not gonna work out phd g: Adam 's not here , so he 's not here to tell me no . professor f: but we could we could just , see if we find a rhythm , you know , what phd g: Sure . professor f: O 's or zeroes , we wanna agree on that ? phd g: Maybe just whatever people would naturally do ? I don't know . professor f: Why don't we do zer i Anyone have a problem with saying zero ? Is zero OK ? phd g: No