Tails Tell Tales: Chapter-Wide Manga Transcriptions with Character Names
Enabling engagement of manga by visually impaired individuals presents a
significant challenge due to its inherently visual nature. With the goal of
fostering accessibility, this paper aims to generate a dialogue transcript of a
complete manga chapter, entirely automatically, with a particular emphasis on
ensuring narrative consistency. This entails identifying (i) what is being
said, i.e., detecting the texts on each page and classifying them into
essential vs non-essential, and (ii) who is saying it, i.e., attributing each
dialogue to its speaker, while ensuring the same characters are named
consistently throughout the chapter.
  To this end, we introduce: (i) Magiv2, a model that is capable of generating
high-quality chapter-wide manga transcripts with named characters and
significantly higher precision in speaker diarisation over prior works; (ii) an
extension of the PopManga evaluation dataset, which now includes annotations
for speech-bubble tail boxes, associations of text to corresponding tails,
classifications of text as essential or non-essential, and the identity for
each character box; and (iii) a new character bank dataset, which comprises
over 11K characters from 76 manga series, featuring 11.5K exemplar character
images in total, as well as a list of chapters in which they appear. The code,
trained model, and both datasets can be found at:
https://github.com/ragavsachdeva/magi
66ac3dbd4b7cf96c1c8f43ae
Ragav Sachdeva
admin_assigned
2024-08-02T09:12:59.043Z
630852d2f0dc38fb47c347a4
https://cdn-avatars.huggingface.co/v1/production/uploads/630852d2f0dc38fb47c347a4/NqkR5EUT45x-Ucx79XOXY.png
Ragav Sachdeva
user
ragavsachdeva
66ac3dbd4b7cf96c1c8f43af
Gyungin Shin
admin_assigned
2024-08-02T09:13:04.673Z
6238c507254d95e12bd1952b
/avatars/8cea1c761c7b2e996277727143bfbea2.svg
Gyungin Shin
user
noelshin
66ac3dbd4b7cf96c1c8f43b0
Andrew Zisserman
Tails Tell Tales: Chapter-Wide Manga Transcriptions with Character Names
Enabling engagement of manga by visually impaired individuals presents a
significant challenge due to its inherently visual nature. With the goal of
fostering accessibility, this paper aims to generate a dialogue transcript of a
complete manga chapter, entirely automatically, with a particular emphasis on
ensuring narrative consistency. This entails identifying (i) what is being
said, i.e., detecting the texts on each page and classifying them into
essential vs non-essential, and (ii) who is saying it, i.e., attributing each
dialogue to its speaker, while ensuring the same characters are named
consistently throughout the chapter.
  To this end, we introduce: (i) Magiv2, a model that is capable of generating
high-quality chapter-wide manga transcripts with named characters and
significantly higher precision in speaker diarisation over prior works; (ii) an
extension of the PopManga evaluation dataset, which now includes annotations
for speech-bubble tail boxes, associations of text to corresponding tails,
classifications of text as essential or non-essential, and the identity for
each character box; and (iii) a new character bank dataset, which comprises
over 11K characters from 76 manga series, featuring 11.5K exemplar character
images in total, as well as a list of chapters in which they appear. The code,
trained model, and both datasets can be found at:
https://github.com/ragavsachdeva/magi
A model named Magiv2 is introduced to automatically generate dialogue transcripts for manga, ensuring narrative consistency and high precision in speaker identification.
Magiv2 speaker diarisation PopManga dataset character bank dataset