text:
September 14 - October 6, 2019	, Allen Theatre
Into the Breeches!
by George Brant
directed by Laura Kepley
Run time: 2 hours and 15 minutes, including a 15 minute intermission
"We Can Do It!" So said Rosie the Riveter, and so say the women of Cleveland, Ohio, in this vibrant new comedy! It's 1942, and the leading men of the Oberon Play House are fighting overseas. With "The Show Must Go On!" as their battle cry, a band of passionate, yet inexperienced, performers rally together to produce the Oberon's season opener, an all-female version of Shakespeare's ambitious Henry IV and Henry V. Will their production be a victory on the home front or a target for rotten tomatoes? A hilarious and moving story about what happens when we're all in it together.
Nisi Sturgis as Maggie Dalton. Photo Credit: Roger Mastroianni
From left to right: Elisabeth A. Yancey (Grace Richards) and Comfort Dolo (Ida Green). Photo Credit: Roger Mastroianni
From left to right: Nisi Sturgis (Maggie Dalton), Tina Stafford (Celeste Fielding). Photo Credit: Roger Mastroianni
From left to right: Elisabeth A. Yancey (Grace Richards), Nisi Sturgis (Maggie Dalton). Photo Credit: Roger Mastroianni
The cast of Into the Breeches! Photo Credit: Roger Mastroianni
From left to right: Brian Sills (Stuart Lasker), Peggy Roeder (Winifred Snow). Photo Credit: Roger Mastroianni
From left to right: Peggy Roeder (Winifred Snow), Comfort Dolo (Ida Green), Courtney Stennett (June Bennett), Nisi Sturgis (Maggie Dalton), Elisabeth Yancey (Grace Richards). Photo Credit: Roger Mastroianni
From left to right: Nisi Sturgis (Maggie Dalton), Jeff Talbott (Ellsworth Snow). Photo Credit: Roger Mastroianni
From left to right: Comfort Dolo (Ida Green), Elisabeth A. Yancey (Grace Richards). Photo Credit: Roger Mastroianni
From left to right: Courtney Stennett (June Bennett), Nisi Sturgis (Maggie Dalton). Photo Credit: Roger Mastroianni
From left to right: Elisabeth A. Yancey (Grace Richards), Courtney Stennett (June Bennett), Tina Stafford (Celeste Fielding), Peggy Roeder (Winifred Snow), Nisi Sturgis (Maggie Dalton). Photo Credit: Roger Mastroianni
Peggy Roeder as Winifred Snow. Photo Credit: Roger Mastroianni
From left to right: Nisi Sturgis (Maggie Dalton), Peggy Roeder (Winifred Snow), and Jeff Talbott (Ellsworth Snow). Photo Credit: Roger Mastroianni
From left to right: Tina Stafford (Celeste Fielding) and Nisi Sturgis (Maggie Dalton). Photo Credit: Roger Mastroianni
From left to right: Comfort Dolo (Ida Green), Brian Sills (Stuart Lasker). Photo Credit: Roger Mastroianni
From left to right: Elisabeth A. Yancey (Grace Richards), Peggy Roeder (Winifred Snow), Nisi Sturgis (Maggie Dalton), Tina Stafford (Celeste Fielding), Courtney Stennett (June Bennett). Photo Credit: Roger Mastroianni
Behind-the-Scenes: Into the Breeches!
September 30, 2019 @ 6:30pm - 8:00pm
Made in Cleveland
George Brant
(Playwright) locally produced plays include Grounded (CPH, Dobama), Marie and Rosetta (CPH), Grizzly Mama (Dobama), Salvage (none too fragile), and Elephant's Graveyard (Parma Senior High School). Other plays include Tender Age, The Mourners' Bench, Three Voyages of the Lobotomobile, One Hand Clapping, Any Other Name, Night of the more(Playwright) locally produced plays include Grounded (CPH, Dobama), Marie and Rosetta (CPH), Grizzly Mama (Dobama), Salvage (none too fragile), and Elephant's Graveyard (Parma Senior High School). Other plays include Tender Age, The Mourners' Bench, Three Voyages of the Lobotomobile, One Hand Clapping, Any Other Name, Night of the Mime, and Dark Room. His work has been produced internationally by such companies as the Public Theater, the Atlantic Theater Company, Trinity Repertory Company, Alley Theatre, City Theatre, London's Gate Theatre, Page 73, Studio Theatre, and the Traverse Theatre, among others. His scripts have been awarded a Lucille Lortel Award, an Edgerton Foundation New Play Award, the Smith Prize, the Keene Prize for Literature, an NNPN Rolling World Premiere, a Scotsman Fringe First Award, an Off-West End Theatre Award, three Ohio Arts Council Individual Excellence Awards, a Creative Workforce Fellowship and the Theatre Netto Festival Grand Prize. Brant has received writing fellowships from the O'Neill National Playwrights Conference, the McCarter Theatre Center, the MacDowell Colony, the Hermitage Artist Retreat, the Djerassi Resident Artists Program and the Blue Mountain Center, as well as commissions from the Metropolitan Opera, Trinity Repertory and Dobama Theatre. George received his MFA in Writing from the James A. Michener Center for Writers at UT-Austin and is a member of the Dramatists Guild and is published by Samuel French, Oberon Books, and Smith & Kraus. He is proud to call Cleveland home.
Alex Brightwell
(Understudy: Stuart Lasker) was born in West Virginia and raised in North Carolina. He earned his BFA from the University of Minnesota/Guthrie Theater Actor Training Program. He was a Conservatory member for the 2018 and 2019 seasons at the Chautauqua Theater Company, where he played Nick Bottom in A Midsummer more(Understudy: Stuart Lasker) was born in West Virginia and raised in North Carolina. He earned his BFA from the University of Minnesota/Guthrie Theater Actor Training Program. He was a Conservatory member for the 2018 and 2019 seasons at the Chautauqua Theater Company, where he played Nick Bottom in A Midsummer Night's Dream; Alfie in One Man, Two Guvnors; Silvius in As You Like It; and Br'er Rabbit/Ratts in An Octoroon. Cleveland credits include the New Ground Theater Festival reading of Tiny Houses and the CWRU/CPH MFA productions of The Merchant of Venice, Hay Fever, The Seagull, and Fifth of July. He has also been seen at Yellow Tree Theater, Minnesota Jewish Theatre, Walking Shadow Theatre Company, Public Theater of Minnesota, and the Guthrie Theater.
Kasey Connolly
(Understudy: Maggie Dalton) came to the CWRU/CPH MFA Acting Program from New York City where she attended the Gallatin School of Individualized Study at NYU. While there, she focused primarily on classical texts and performed in Hamlet; Mother Courage; and an all-female, democratically directed production of Julius Caesar. Cleveland credits more(Understudy: Maggie Dalton) came to the CWRU/CPH MFA Acting Program from New York City where she attended the Gallatin School of Individualized Study at NYU. While there, she focused primarily on classical texts and performed in Hamlet; Mother Courage; and an all-female, democratically directed production of Julius Caesar. Cleveland credits include understudy for Maggie Dalton in Into the Breeches! and the New Ground Theatre Festival readings of Christina, Tiny Houses, and Cicada. Film credits include the upcoming independent shorts Apprehending Edmund Montrell and A Way Home. kaseyconnolly.com
Comfort Dolo
(Ida Green) can't believe she's nearing the end of her CWRU/CPH adventure! She was born in Chattanooga, Tennessee and raised in friendly Fridley, Minnesota. She graduated from Gustavus Adolphus College in St. Peter, MN with double majors in Honors Theatre and Communication Studies. Favorite credits include: To Kill a Mockingbird more(Ida Green) can't believe she's nearing the end of her CWRU/CPH adventure! She was born in Chattanooga, Tennessee and raised in friendly Fridley, Minnesota. She graduated from Gustavus Adolphus College in St. Peter, MN with double majors in Honors Theatre and Communication Studies. Favorite credits include: To Kill a Mockingbird (The Guthrie Theater), TOUCH, a remake of the 1984 Emmy Award winning film (Illusion Theater), and various commercials ranging from the Minnesota State Lottery to hand modeling for KeyBank. She recently made her Cleveland Play House debut in Into the Breeches! (Ida Green) while also understudying the role of Jasmine in Pipeline. CWRU/CPH Credits include: Sorel Bliss in Hay Fever, Sorin in The Seagull, and Shirley in Fifth of July. Naomi in Sugar (Phoenix Theater), Carlos Sirah's The Utterances (The Playwright's Center). She is a big fan of theatre for social justice, the German language, and gummy bears!
Peggy Roeder
(Winifred Snow) earned her Equity Union Card at the Cleveland Play House four decades ago. She appeared at CPH in Butterflies Are Free, The Removalists, Hamlet, In Fashion, and other shows. Since then, she has worked at Steppenwolf (The Dresser), Goodman (Landscape of the Body), Mark Taper Forum (The Cider more(Winifred Snow) earned her Equity Union Card at the Cleveland Play House four decades ago. She appeared at CPH in Butterflies Are Free, The Removalists, Hamlet, In Fashion, and other shows. Since then, she has worked at Steppenwolf (The Dresser), Goodman (Landscape of the Body), Mark Taper Forum (The Cider House Rules), Asolo Rep (Grapes of Wrath), Atlantic Theatre Co. (The Cider House Rules), Milwaukee Rep (Clean House), Chicago Shakespeare (Romeo and Juliet), The Muny of St Louis (Wizard of Oz), Peninsula Players (Outside Mullingar), Marriott Lincolnshire (Gypsy), and many other theatres. Film and TV credits include Groundhog Day, Road To Perdition, Were the World Mine, Stranger Than Fiction, Star Trek:DS9, and Law & Order. Her work has garnered five Jeff Awards, three Artisan Awards, and two After Dark Awards.
Brian Sills
(Stuart Lasker) previous Cleveland Play House credits include Eric Coble's Fairfield and Around The World In 80 Days. Brian also hosted the Copacabana, the annual CPH gala this year. Off-Broadway credits include: The Machine (Park Avenue Armory), Prurience (The Guggenheim), The First Annual Trump Family Special (Triad Theatre). London: The more(Stuart Lasker) previous Cleveland Play House credits include Eric Coble's Fairfield and Around The World In 80 Days. Brian also hosted the Copacabana, the annual CPH gala this year. Off-Broadway credits include: The Machine (Park Avenue Armory), Prurience (The Guggenheim), The First Annual Trump Family Special (Triad Theatre). London: The Machine (Donmar Warehouse). National US Tour: The Lion King (original cast). Select Regional Theatre credits include: Chicago Shakespeare Theater, Bay Street Theatre, Chautauqua Theatre Company, Kansas City Repertory Theatre, Capital Repertory Theatre, Asolo Repertory Theatre, Milwaukee Repertory Theatre, Clarence Brown Theatre. Television credits include: The Marvelous Mrs. Maisel, Elementary, Blue Bloods, Crashing, Unbreakable Kimmy Schmidt, Mr. Robot, and the upcoming Lifetime holiday TV movie Twinkle All The Way. Film credits include: Bikini Moon, Baby On Board, Nothing Like The Holidays, Rocky Marciano. A graduate of The Ryerson Theatre School at Ryerson University in Toronto, Brian currently resides in New York City.
Tina Stafford
(Celeste Fielding) is thrilled to be back at CPH! She was last seen as Queen Elizabeth and the Nurse in Shakespeare in Love, toured through Cleveland with Once First National Tour, and Jolson, and appeared at Willoughby Fine Arts with Liz Swados' Missionaries. She has played leading roles at Arena more(Celeste Fielding) is thrilled to be back at CPH! She was last seen as Queen Elizabeth and the Nurse in Shakespeare in Love, toured through Cleveland with Once First National Tour, and Jolson, and appeared at Willoughby Fine Arts with Liz Swados' Missionaries. She has played leading roles at Arena Stage, Cincinnati Playhouse in the Park, Milwaukee Rep, Paper Mill Playhouse, Kansas City Rep, Goodspeed Musicals, La Jolla Playhouse, Denver Center, Westport Country Playhouse, Bucks County Playhouse, Cape Playhouse, Shakespeare Theatre of NJ, the Maine, Texas and Utah Shakespeare Festivals, and she's done many interesting Off-Broadway shows you've never heard of. All My Children fans may remember Ms. Stafford as Erica Kane's gruff but loveable prison guard, Slim. Proud Equity member.
Courtney Stennett
(June Bennett) is a native of Eagle, Idaho. She graduated with her BFA in Performance from Baylor University. Prior to making the move to Cleveland, Courtney enjoyed living and performing in Chicago. She most recently appeared onstage at Cleveland Play House in Into the Breeches! She spent the summer more(June Bennett) is a native of Eagle, Idaho. She graduated with her BFA in Performance from Baylor University. Prior to making the move to Cleveland, Courtney enjoyed living and performing in Chicago. She most recently appeared onstage at Cleveland Play House in Into the Breeches! She spent the summer at Chautauqua Theatre Company in One Man, Two Guvnors; A Midsummer Night's Dream; and in the New Play Workshop production of Agent 355. She has been seen in the CWRU/CPH productions of Hay Fever as Judith Bliss, The Seagull as Masha, and Fifth of July as Sally Friedman. Summer of 2018 she appeared internationally at the Edinburgh Fringe Festival in Call of the Void. Additional recent professional credits include Agnes of God (Aleatoric Theatre), A Little Princess (Windy City Music Theatre), and Seminar (Spartan Theatre Company). Courtney is delighted and humbled to be working toward her MFA in the CWRU/CPH Acting program and thrilled to be part of this ensemble!
Nisi Sturgis
(Maggie Dalton) is delighted to be back in Cleveland! CPH first welcomed her as Mrs. Giving in In the Next Room, or the vibrator play. In 2017, she returned to play 14 characters in Ken Ludwig's Baskerville: A Sherlock Holmes Mystery. Her extensive regional credits include, Doubt, Pride more(Maggie Dalton) is delighted to be back in Cleveland! CPH first welcomed her as Mrs. Giving in In the Next Room, or the vibrator play. In 2017, she returned to play 14 characters in Ken Ludwig's Baskerville: A Sherlock Holmes Mystery. Her extensive regional credits include, Doubt, Pride and Prejudice, You Can't Take It With You, Mrs. Warren's Profession (Denver Center); A Doll's House, Who's Afraid of Virginia Woolf?, Life of Riley, Pentecost, Much Ado About Nothing, Pericles (The Old Globe); A Streetcar Named Desire, Arms and the Man, Our Town, Trelawny of the Wells (Shakespeare Theatre NJ); Failure: A Love Story, Macbeth, Cymbeline, As You Like It, Ceasar, (ISF); Disgraced tour at the Goodman Theatre, Berkeley Rep., and Seattle Rep. (BATA award). On Broadway, she appeared as The Woman in 39 Steps. Some Off-Broadway credits include Intimate Apparel and The Pretty Trap. She loves Shakespeare, roles across the country include: Hermione, Titania, Lady Macbeth, Viola, the Nurse, Mark Antony and Jaques. MFA from The Old Globe Theatre. Recurring role as June Thompson for five seasons in HBO's Boardwalk Empire. She is an Assistant Professor of Acting at the University of Illinois Urbana-Champaign.
Jeff Talbott
(Ellsworth Snow) is full of Christmas cheer and happy to be back sharing it with his family at CPH after opening the season in Into the Breeches! He has been on Broadway (Sly Fox, Fortune's Fool); Off-Broadway (Abundance, Such Good Friends, Home of the Brave); and has extensive regional credits, more(Ellsworth Snow) is full of Christmas cheer and happy to be back sharing it with his family at CPH after opening the season in Into the Breeches! He has been on Broadway (Sly Fox, Fortune's Fool); Off-Broadway (Abundance, Such Good Friends, Home of the Brave); and has extensive regional credits, including his most recent: the regional premiere of Oslo. Other regional credits: The 25th Annual Putnam County Spelling Bee, The Odd Couple, Race, circle mirror transformation, Frost/Nixon, and Doubt at theatres including Repertory Theatre of St. Louis, Pioneer Theatre Company, Paper Mill Playhouse, Goodspeed Opera House, Actors' Theatre of Louisville, and Yale Repertory Theatre, among many others. He has been on film (Julie & Julia) and TV (The Blacklist, Orange is the New Black, Unforgettable, Boardwalk Empire, The Sopranos, all the Law & Orders, One Life to Live, and As the World Turns). As a writer, his plays include The Submission (Laurents/Hatcher Award, Outer Critics Circle's John Gassner Award; off-Broadway at MCC Theater); The Gravedigger's Lullaby (Off-Broadway at TACT); i (Pioneer Theatre Company) and Civics and Humanities for Non-Majors (commissioned and produced by Montclair State University). He writes musicals with composer Will Van Dyke. They have written Imagine Harry, Wintersong, Seven Broken Hearts, and The Circle and the Wheel. They have also released an EP of their songs called A View of the River (available on iTunes). Jeff is a graduate of the Yale School of Drama. jefftalbott.com
Elisabeth Yancey
(Grace Richards)
recently made her Cleveland Play House debut as Grace Richards in Into the Breeches!. Other credits with CWRU/CPH MFA Acting Program include The Merchant of Venice, Hay Fever, The Seagull, and Fifth of July. Internationally, Elisabeth performed at The Edinburgh Fringe Festival in the world premiere of
more(Grace Richards)
recently made her Cleveland Play House debut as Grace Richards in Into the Breeches!. Other credits with CWRU/CPH MFA Acting Program include The Merchant of Venice, Hay Fever, The Seagull, and Fifth of July. Internationally, Elisabeth performed at The Edinburgh Fringe Festival in the world premiere of Call of the Void. Other favorite credits include CPH's New Ground Theatre Festival in Cicada and Christina, and the Oberlin Summer Theater Festival in Romeo and Juliet and Little Women, as well as work with Wellesley Repertory Theatre. Elisabeth holds a BA in Theatre Studies from Wellesley College, a certificate of classical acting from the London Academy of Music and Dramatic Arts, and a certificate of literature from Cambridge University. She is so grateful to be a part of such a beautiful Cleveland tradition! Instagram: @elisabethyancey
Kyra Button
(Assistant Stage Manager) is excited to be in her third season at Cleveland Play House, having worked on Pipeline, Into the Breeches!, An Iliad, Tiny Houses, The Seagull, Shakespeare in Love, Macbeth, The Invisible Hand, and The 25th Annual Putnam County Spelling Bee, just to name a few. Her previous more(Assistant Stage Manager) is excited to be in her third season at Cleveland Play House, having worked on Pipeline, Into the Breeches!, An Iliad, Tiny Houses, The Seagull, Shakespeare in Love, Macbeth, The Invisible Hand, and The 25th Annual Putnam County Spelling Bee, just to name a few. Her previous credits include The Heart's Impatience (Shufflefoot Theatre Company), Resistance (Semicolon Theatre), A Streetcar Named Desire (St. Ann's Warehouse), Medea and Dreamgirls (Red House Arts Center), This Day Forward (The Vineyard Theatre), and The Intergenerational Project (Rose Bruford, London). She is overjoyed to be continuing her work with CPH and is proud to hold a BFA in Stage Management from Syracuse University's Department of Drama.
Angela Calin
(Costume Designer) is very pleased to be back on the creative team for Into the Breeches and to make her debut at the Cleveland Play House. She is a costume and set designer based in Los Angeles with over 120 productions in local and regional theatres. Her recent credits include more(Costume Designer) is very pleased to be back on the creative team for Into the Breeches and to make her debut at the Cleveland Play House. She is a costume and set designer based in Los Angeles with over 120 productions in local and regional theatres. Her recent credits include Shakespeare in love at The Alliance Theatre and Orlando Shakespeare; Sweet Charity at UCLA/Freud Playhouse; Mamma Mia at the Hollywood Bowl; Two Degrees and To Kill a Mockingbird at Denver Center Theatre Company; The Monster Builder and The Roommate at South Coast Repertory; Into the Breeches At Chautauqua Theatre Company, Othello, Man Of La Mancha, and Noises Off at A Noise Within Theatre where she's a resident artist. Her work has been seen at the South Coast Rep., Old Globe, Milwaukee Rep, Denver Center Theatre, Alliance Theatre, Pasadena Playhouse, Orlando Shakespeare, Georgia Shakespeare, Antaeus and various other theatres in Los Angeles. Angela's designs garnered L.A. Drama Critic's Circle, Ovation, Backstage and Drama Logue Awards. She has 16 film production credits in the US and her native Romania and she holds an MFA in set and costume design from the Academy of Arts "N. Grigorescu" in Bucharest- Romania.
Jean E. Compton
(Assistant Stage Manager) is thrilled to return to Cleveland Play House, where she previously spent the 2015-2016 centennial season as the Stage Management Apprentice. She is now based in Chicago, where credits include The Death of Gaia Divine (Momentary Theatre), Spirits to Enforce (The Passage Theatre), Candide and Mudpie 2018 more(Assistant Stage Manager) is thrilled to return to Cleveland Play House, where she previously spent the 2015-2016 centennial season as the Stage Management Apprentice. She is now based in Chicago, where credits include The Death of Gaia Divine (Momentary Theatre), Spirits to Enforce (The Passage Theatre), Candide and Mudpie 2018 (Mudlark Theater Company), You for Me for You (Sideshow Theatre Company), Lizzie (Firebrand Theatre), and Shining Lives (Northlight Theatre). Her regional credits include A Midsummer Night's Dream, An Octoroon, and Into the Breeches! (Chautauqua Theater Company); Venus in Fur and The Whale (Rep Stage); and the 2016-2017 season at Two River Theater. She is a graduate of The Theatre School at DePaul University. jeanecompton.com
John Godbout
(Stage Manager) was Cleveland Play House's resident stage manager from 2001 to 2008 and returned to CPH in the fall of 2011. This past summer, he worked at Porthouse Theatre where he stage managed Man of La Mancha and The Music Man. John has also stage managed at North more(Stage Manager) was Cleveland Play House's resident stage manager from 2001 to 2008 and returned to CPH in the fall of 2011. This past summer, he worked at Porthouse Theatre where he stage managed Man of La Mancha and The Music Man. John has also stage managed at North Shore Music Theatre, Capital Repertory Theatre, The Weston Playhouse, Berkshire Theatre Festival, Arkansas Repertory Theatre, Northern Stage, and Seaside Music Theatre.
Kevin Inouye
(Fight Choreographer) is a Certified Teacher and Theatrical Firearms Instructor with the Society of American Fight Director, a SAG-AFTRA stunt performer, and has choreographed theatrical violence for over 70 live shows and over 25 screen productions. He joined Case Western Reserve University and the CWRU/CPH MFA Acting program in fall more(Fight Choreographer) is a Certified Teacher and Theatrical Firearms Instructor with the Society of American Fight Director, a SAG-AFTRA stunt performer, and has choreographed theatrical violence for over 70 live shows and over 25 screen productions. He joined Case Western Reserve University and the CWRU/CPH MFA Acting program in fall of 2018 as Assistant Professor of Movement, Acting, & Stage Combat. He is a published author, actor, director, theatrical armorer, and teaches workshops regularly around the country.
Ora Jewell-Busche
(Wig Designer) is a Minnesota based wig and makeup designer with over a decade of working in the field of opera, theater, and film. She has worked extensively with companies ranging from The Lyric Opera of Chicago, Chicago Shakespeare, Lookingglass Theater, The Steppenwolf, The Goodman Theater, and many more. Recent more(Wig Designer) is a Minnesota based wig and makeup designer with over a decade of working in the field of opera, theater, and film. She has worked extensively with companies ranging from The Lyric Opera of Chicago, Chicago Shakespeare, Lookingglass Theater, The Steppenwolf, The Goodman Theater, and many more. Recent credits include: Aida, Turandot, Carmen, The Magic Flute (Pacific Symphony); Les Uncomfortables, The Barber of Seville, Don Pasquale, Rigoletto (Lyric Opera of the North); Rusalka, Hansel and Gretel (Little LOON); Il Trovatore, A Streetcar Named Desire, The Tales of Hoffman, The Daughter of the Regiment (Hawaii Opera Theater). Film and television credits include Were the World Mine, Chicago Overcoat, Prison Break and Batman v Superman. Ora also teaches Theatrical Makeup at Makeup First School of Makeup artistry.
Thom Jones
(Voice, Speech and Text Coach) is the head of Voice and Speech for Brown University/Trinity Repertory Company's Master of Fine Arts in Acting and Directing and has coached voice, dialects, and text extensively in theatre, film, and television. Theatres where he has coached include: The Public Theater, Shakespeare in more(Voice, Speech and Text Coach) is the head of Voice and Speech for Brown University/Trinity Repertory Company's Master of Fine Arts in Acting and Directing and has coached voice, dialects, and text extensively in theatre, film, and television. Theatres where he has coached include: The Public Theater, Shakespeare in the Park, Noel Coward Theatre (London's West End), Yale Repertory Company, Dallas Theater Center, McCarter Theatre, Cleveland Play House, The Huntington Theatre, Trinity Repertory Company, Hartford Stage, Actor's Theatre of Louisville, Syracuse Stage, Cincinnati Playhouse, Long Wharf Theatre, Alabama Shakespeare Festival, Hangar Theatre. For Film and Television, he has voice, dialect and text coached numerous actors, including: Nicole Kidman, Ben Mendelsohn, Sissy Spacek, Alan Rickman, Vanessa Redgrave, Robin Williams, Emma Watson, Andra Day, Miles Teller, Jennifer Hudson, Bill Skarsgard, Lashana Lynch, Lakeith Stanfield, Aaron Eckhart, Katey Segal, Ciaran Hinds, David Oyelowo, Amy Adams, Lily Rabe, Mel Gibson, Olivia Cooke, Tye Sheridan, Simon Peg, Mark Rylance, Sherie Rene Scott, Sienna Miller, Brendan Gleeson, Melissa Leo, Emily Van Camp, Blythe Danner, Logan Lerman, Linda Emond, Ryan Reynolds, and others, as well as for directors Steven Spielberg, Martin Scorcese, Lee Daniels, Ridley Scott, Ben Younger, and others.
Laura Kepley
(Director) became Artistic Director of Cleveland Play House in 2013 and has directed numerous CPH mainstage productions including Every Brilliant Thing; Into the Breeches!; Tiny Houses (world premiere, also at Cincinnati Playhouse in the Park); Sweat; The Diary of Anne Frank; Shakespeare in Love; The Crucible; Steel Magnolias; The Good more(Director) became Artistic Director of Cleveland Play House in 2013 and has directed numerous CPH mainstage productions including Every Brilliant Thing; Into the Breeches!; Tiny Houses (world premiere, also at Cincinnati Playhouse in the Park); Sweat; The Diary of Anne Frank; Shakespeare in Love; The Crucible; Steel Magnolias; The Good Peaches (world premiere); Fairfield (world premiere); How I Learned to Drive (also at Syracuse Stage); The Little Foxes; Venus in Fur; Good People (also at Syracuse Stage); A Carol for Cleveland (world premiere); In the Next Room, or the vibrator play; My Name is Asher Lev and CPH readings of Roe Green Award-winning plays Tiny Houses; The Chinese Lady; Soups, Stews and Casseroles: 1976; Marjorie Prime and Daphne's Dive. She joined CPH in 2010, having arrived from Trinity Repertory Company in Providence, Rhode Island where she was Resident Director and Artistic Associate for four seasons and Interim Director of the Brown/Trinity Rep M.F.A. in Directing Program for one. She has also directed for The Alliance Theatre, Asolo Repertory Theatre, Chautauqua Theater Company, Cincinnati Playhouse in the Park, Contemporary American Theatre Festival, and The John F. Kennedy Center for the Performing Arts, among others. A native Ohioan, Laura received her undergraduate degree from Northwestern University and her Master of Fine Arts from Brown University/Trinity Rep. She is a Drama League Fellow and a recipient of the 2009-2011 National Endowment for the Arts/Theatre Communications Group Career Development Program for Directors.
Rebecca Marzalek-Kelly
(Assistant Director/Drama League Directing Fellow) is a Brooklyn-based director and choreographer. Her work has been seen in NYC at BAM (Fisher), The Public Theater, PS122, BRIC House, The Bushwick Starr, and Terminal 5. She's directed multiple productions with the experimental company Fovea Floods, highlights include Fasten and The Turn of more(Assistant Director/Drama League Directing Fellow) is a Brooklyn-based director and choreographer. Her work has been seen in NYC at BAM (Fisher), The Public Theater, PS122, BRIC House, The Bushwick Starr, and Terminal 5. She's directed multiple productions with the experimental company Fovea Floods, highlights include Fasten and The Turn of the Screw. She assistant directed with LAByrinth Theater while in residence at The Public Theater. As a guest artist at Skidmore College, she directed Who Will Carry the Word and Let the Right One In. Most recently she was a directing observer on Hadestown at the Walter Kerr Theatre observing director, Rachel Chavkin. Through TCG and the Andrew W. Mellon Foundation, she's a three-time Leadership U(niversity) Serving the Field grant recipient. Her training includes Williamstown Theatre Festival, SITI Company Summer Intensive, SDCF Observership Class, and Skidmore College. She's a member of Lincoln Center Theater Directors Lab and a 2019 Drama League Directing Fellow.
(Managing Director) is in his thirteenth season at Cleveland Play House, recipient of the 2015 Regional Theatre Tony Award. During his tenure, CPH made its historic relocation to the Allen Theatre in partnership with Playhouse Square and Cleveland State University. He serves on the Board of Directors of Theatre Forward, more(Managing Director) is in his thirteenth season at Cleveland Play House, recipient of the 2015 Regional Theatre Tony Award. During his tenure, CPH made its historic relocation to the Allen Theatre in partnership with Playhouse Square and Cleveland State University. He serves on the Board of Directors of Theatre Forward, Arts Cleveland, and the Commission on Economic Inclusion. From 1998 to 2007, Kevin was Managing Director of Woolly Mammoth Theatre Company in Washington, D.C. where he helped lead the funding and construction of an award-winning new theatre facility. Previous positions include Director of Marketing at La Jolla Playhouse in San Diego, California. He is a former president of both the National New Play Network and the League of Washington Theatres. Kevin teaches theatre management at Cleveland State University and is an adjunct professor in the Department of Theater at Case Western Reserve University. He is a graduate of Moorhead (MN) State University, holds a Master of Arts in Arts Administration from Indiana University, and is a member of the Leadership Cleveland Class of 2009.
(Scenic Design) designed Into the Breeches!, Sweat, The Diary of Anne Frank, All the Way, A Christmas Story, The Whipping Man, The Glass Menagerie, Emma, and Bill W. and Dr. Bob for Cleveland Play House. Other credits include Wizard of Oz, Fiddler on the Roof, Les Misérables, West Side Story, more(Scenic Design) designed Into the Breeches!, Sweat, The Diary of Anne Frank, All the Way, A Christmas Story, The Whipping Man, The Glass Menagerie, Emma, and Bill W. and Dr. Bob for Cleveland Play House. Other credits include Wizard of Oz, Fiddler on the Roof, Les Misérables, West Side Story, Billy Elliot, and Seussical The Musical (The Muny in St. Louis); exhibit designer for Avatar the Exhibition (Experience Music Project in Seattle); Fallen Angels (Asolo Repertory Theatre); Joshua's Boots (Opera Theatre of St. Louis and Amarillo Opera); The Rainmaker, A Moon for the Misbegotten (2007 Prague Design Quadrennial); The Dazzle (A.C.T. San Francisco); The Giver, The Diary of Anne Frank, Jesus Hates Me, and House of the Spirits (Denver Center for the Performing Arts); Saint Joan, Jekyll & Hyde, Ordinary Nation, Rabbit Hole, Fall of Heaven, and 4000 Miles (Repertory Theatre of St. Louis); The Subject Tonight is Love (Alliance Theatre Company); Just Stopped by to See the Man and The Whipping Man (The Old Globe); and numerous Utah Shakespeare Festival productions in the 2012/13 season. Robert is a Teaching Professor at Washington University in St. Louis and Director of the Beyond Boundaries Program in interdisciplinary education. morgansetdesign.com
Jane Shaw
(Sound Designer) for Cleveland Play House: Sweat; The Royale; Shakespeare in Love (original music); All the Way; Steel Magnolias; The Crucible; The Little Foxes; Grounded; A Carol for Cleveland; and In the Next Room, or the vibrator play. Her credits include Engagement Party (Hartford Stage), Actually (MTC), Annie Get Your more(Sound Designer) for Cleveland Play House: Sweat; The Royale; Shakespeare in Love (original music); All the Way; Steel Magnolias; The Crucible; The Little Foxes; Grounded; A Carol for Cleveland; and In the Next Room, or the vibrator play. Her credits include Engagement Party (Hartford Stage), Actually (MTC), Annie Get Your Gun (Bay Street), They Promised Her the Moon (Old Globe), Oo Bla Dee (Two River), and productions at Theatre for a New Audience, New York Theatre Workshop, the Mint Theater, Asolo, Northern Stage, Dorset Theater Festival, and American Repertory Theater. Ms. Shaw's sound designs have earned a Drama Desk, Bessie Award, Henry Award, Connecticut Critics Circle Awards, Premios ACE 2012, Meet the Composer grant, Theatre Communication Group's Career Development Program and nominations for two Lortels, Henry Hewes and Elliot Norton awards. She was born in Kansas and lives in Brooklyn.


text:
The Persian Blog of Kings
Iran Analysis and Global Politics
It's been a while since I last updated, so I suppose I should start with a brief explanation of why that is. Way back when, in what now seems like decades ago but was only January, I deliberately stepped away from blogging to take care of a few other side projects I needed to finish. Among those was a paper on populist securitization that I was presenting at the International Studies Association (ISA) annual conference in Hawaii, a course on Iranian foreign policy I was set to teach at Georgetown over the summer, and making progress toward getting my thesis published as a standalone book. By the end of February, all three seemed to be going pretty well. I was nearly done with the paper (and getting excited about Hawaii in the process), the course was getting decent registration numbers, and the book was garnering some initial attention from publishers.
Then, everything stopped.
You already know the reason. The novel coronavirus has totally upended life as we know it. As I write this, I'm on my thirty-fifth day of devout social distancing practice. My wife and I only leave our apartment for neighborhood walks, food, and little else. By the middle of March, the ISA conference had been canceled, and preparations for the summer term at Georgetown had shifted entirely to virtual instruction (something I didn't want to do for a new seminar course). I'm hoping that the book publication is still in the works, although I'm sympathetic to editors and readers who might be preoccupied with concerns other than a manuscript about Israel-Iran relations.
With all my near term projects suspended for the foreseeable future, I considered a return to blogging. There was, however, one small problem: international politics isn't really happening right now either. Sure, there are issues related to the coronavirus that are relevant to international relations, such as the competition for PPE, the race to find an effective treatment (or, in the case of the United States, shady attempts to purchase one and keep it for yourself), and the great power blame game taking place between the United States and China. But international politics, at least in its former, universally understood sense, is on lockdown with the rest of us.
Iran, the nominal topic of this blog, was among the hardest and earliest affected nations in facing this new crisis. The virus was quick to infiltrate not only the country but also many of its most valued institutions. It took root in the holy city of Qom, the seat of the Iranian clergy, before making its way into the upper echelons of government. The deputy health minister, Iraj Harirchi, took ill on national television (he eventually recovered), while a shocking number of parliamentarians became infected. Several political officials have already died from the virus.
While the United States and China attempt to cast the blame for the outbreak on each other, the sources of widespread contagion and medical system duress are less ambiguous in Iran. The Iranian government was, like virtually every other nation in the world, too slow to take aggressive action in response to the virus's spread, including shutting down travel, commerce, and social gatherings, leading to an explosion in cases across the country.
Yet unlike the majority of other countries, Iran is also contending with crippling American economic sanctions, which the Trump administration is still aggressively enforcing despite widespread international calls for relief. These measures are, unsurprisingly, significantly inhibiting Iran's ability to deal with the crisis and care for its citizens. As horrifying as the numbers coming out of Iran are right now, the actual figures are undoubtedly worse than what's being officially reported. Nearly every Iran-watcher has written some version of the same article, cataloging how the Trump administration's relentless pursuit of regime collapse via economic starvation is killing thousands of the very people the United States claims it wants to help. Some of them continue, credulously, to make overtures to the Trump administration to seize this opportunity as the basis for a diplomatic opening to Iran. (Meanwhile, right-wing "analysts" are busy pumping out content cheering on the sanctions, hoping, as ever, that continued economic misery combined with the horrors of the pandemic will hasten the downfall of the regime.)
If there is even the slightest hint of a silver lining here, it lies in the Iranian people's fortitude in the face of danger. In an emotional and historical sense, Iran may be better prepared for what lies ahead than many of its global peers. Iran has known more than its fair share of national tragedy and struggle in the recent past. The terrible price, both human* and financial, of the Iran-Iraq war left an indelible image on the country and its people. The dead literally overlook the nation as images of the martyrs stare out from billboards scattered throughout the country, stark reminders of too many lives lost too soon. There are living reminders, as well. Many Iranians who were exposed to Saddam Hussein's chemical weapons attacks are still alive today. The lingering effects of those attacks on their health, and their respiratory system, in particular, puts them at increased risk for severe complications from COVID-19.
I don't know what the future holds for Iran — or any of the rest of us, for that matter. We're in uncharted territory now, and I, like many others, can only speculate. For Iran, I suspect that this crisis, when it ends, will do little to soften the country's outlook on the world, and the West in particular. Perhaps in the early days of the crisis, there was a brief window of opportunity for Europe to make an overture to Iran, to demonstrate that they genuinely were on the side of the Iranian people despite whatever disagreements they might have had with the regime and its nuclear activities. But that moment is likely gone. Europe has its own problems to deal with now, and thousands of Iranians will be left dead where it doesn't count.
*I'm aware of the controversy surrounding the official death toll of the war and whether or not it comes anywhere close to the oft-cited one million lives figure frequently quoted by many scholars, journalists, and other prominent figures. Regardless of the actual number, the narrative of the conflict and the collective price paid by the nation for defeating the Iraqis — including withstanding Saddam Hussein's chemical weapons attacks — remains a salient narrative in Iranian political and historical thought.
Author Jonathan LesliePosted on April 19, 2020 Leave a comment on The End of Politics
Sign up here to follow the blog by email. I only post when I find something important, so I promise I won't overload your inbox with needless messages.
Jonathan Leslie
Don't Swim in the Threat Stream
Iran hasn't changed, even if the United States has
The Assassination of Mohsen Fakhrizadeh
Why is the New York Times still taking Mahmoud Ahmadinejad seriously?
Flammable Iran
The Persian Blog of Kings Blog at WordPress.com.

redpajama_set_name: RedPajamaCommonCrawl


Answered Any Black Ops 1 Zombies (ONLINE) mod menu's?
Any Black Ops 1 Zombies (ONLINE) mod menu's? Running Solo zombies is great but playing with friends is even better. Any menu's available for teh online zombie use?
Hey Insanedevo have you checked out the BO1 fourm check it out and see if you find anything.
As UnsignedGlitch said, the BO1 section is the place to go. Most menus are built to not go online, but some are. Usually tools are able to go online.
@DF_AUS I'm pretty sure this thread has been answered, and user is no longer responding.


George has been a coach since the 1970's He is a level three (Old System) Sprints Relays Hurdles Distance and Walks coach. He has been an Athletics New Zealand coach lecturer since 1983 and an IAAF coach lecturer since 1990. George is qualified to coach all events and has coached athletes to world championship level. He has coached athletes to 127 New Zealand titles and many more masters' titles.
Anne is an Athletics NZ level two coach in Jumps Sprints Relays and Hurdles and an IAAF level 3 in Jumps. Anne teaches mathematics at PNGHS.
Malcolm is an experienced sprints coach having coached Bevan Smith who was one of New Zealand's leading sprinters.
Brian coaches primarily High Jumpers and Pole-vaulters. He has a Level 2 qualification in High Jump, Sprints, Hurdles, Relays and Distance Running.
Katrina qualified as a Run Jump Throw and Level 1 coach while still a student at Freyberg High School. She is a former multiple national champion: Youth 800m Champion 1995/96/97, Youth 400m 1996, Junior 400m & 800m 1998. Katrina attended Tulane University, New Orleans, on a four year athletics scholarship and now works for Sport Manawatu.
Tara has level 2 qualification in Sprints, Hurdles and Relays. Tara has been HOD of PE at St Peters and in 1992 was the National Junior 200m Champion. Prominent athletes Tara has coached include Veronica Torr and the Signal twins.
Vanessa has Run Jump Throw and level 2 qualifications and is currently developing her own programme. Vanessa has won numerous national masters titles as well as finishing fourth in the 400m at the NZ and World Masters Track and Field Championships.
Callum has Run Jump Throw and a Full level 2 qualification. A silver medalist in the senior decathlon and fourth placing at the world double decathlon (icosathlon) means Callum can coach all events with a primary interest in Throws. Callum teaches at Roslyn.
Viliami was a former pole vaulter and has an IAAF Level I coaching course, which he completed in the Islands. As an athlete, he represented Tonga at the South Pacific Games and won a silver medal.


text:
\section{Introduction}
\label{sect:overview}

The architecture of HPC platforms has shown significant changes in the
last three decades, from vector-parallel architecture to
distributed-memory scalar processors, and then many-cores with SIMD
units, accelerators, and most recently heterogeneous many-cores.

The ideas behind the accelerators and heterogeneous many-cores are
very similar. In both cases, a number of relatively simple and thus
energy- and area-efficient processors are combined with relatively
small number of complex, high-performance processors. The difference
is that in the case of the accelerator systems, the complex,
high-performance cores are usually commodity processors (the host CPU)
in one die, and accelerator cores are in a separate die, and they are
connected by general-purpose link such as the PCI express. The host
CPU is usually of x86 architecture.  GPGPUs used with x86 processors
are the currently the most widely used accelerator systems. This
architecture has many advantages, but the most important one is that
all of the hardware and software for the host CPU is already
there. The developer of the accelerator hardware can concentrate on
the accelerator hardware itself and software to make use of it.

The disadvantage of the accelerator system is the existence of the
communication link between the host and accelerators. Usually, both
the bandwidth and latency of the communication between the host and
accelerators is limited by that of the standard PCIe specification.
The theoretical peak throughput of the PCIe 3.0 standard (with 16
lanes) is only 16GB/s. On the other hand, some of latest GPGPUs have
multiple channels of the HBM memory, with the total bandwidth
approaching to 1TB/s.  If we divide the calculations of an application
into that on the host and that on accelerators, communication between
them would become necessary, and in many cases that would limit the
performance. Thus, either we have to port all of application to the
accelerator side, or we have to live with relatively low performance.
In addition, in many cases size of the on-board memories of the
accelerator boards is small, less than 16GB, and thus either the
problem size is limited or we need to store the data in the main
memory of the host CPU.

In principle, heterogeneous many-cores can solve these limitations of
accelerator systems, since what correspond to the host CPUs of
accelerator systems are now integrated to the same LSI chip as the
accelerators, and they share the same physical memory.
Examples of such architecture include  Sunway SW26010 and PEZY-SC2.
The former is used in the TaihuLight system, which was ranked \#1 in
the Top 500 list four times in years 2016 and 2017. The latter was
used in the Gyoukou system, which was ranked \# 4 in the Top 500 list
of November 2017. We can see that though the heterogeneous many-core
architecture has clear advantages, they are yet to be widely used.

Currently, to port applications to these systems and achieve good
performance requires quite a lot of efforts. One reason is that
``automatic'' parallelization through, for example, OpenACC results in
rather poor performances in the case of TaihuLight, and currently only
a subset of OpenCL is available on PEZY-SC2. In the case of PEZY-SC2,
right now the host CPU on chip is disabled and thus the current system
an accelerator system with a Xeon-D CPU. However, since the memory on
PEZY-SC2 is larger than that on Xeon-D, all data can (and should) be
on the side of PEZY-SC2.

Another reason is that the performance ratio between the complex cores
and simple cores tend to be very large. In the case of TaihuLight, one
complex core and one simple core have the same peak performance, but
there is only one complex core for every 64 simple cores. Thus, even
though it is not easy to write codes for simple cores, almost all
codes should be moved to the simple core side to obtain decent
performance.

One way to make the application development on these machines easier
is to provide DSLs or frameworks, in which the users express the
problem to be solved or numerical method to be used in high-level,
machine-independent way. The machine-specific part of DSL runtime
library must be optimized to each architecture, but since one DSL can
be used to implement many applications, if such a DSL is possible a
lot of works by many researchers can be used for more productive
researches.

We have been developing FDPS, Framework for Developing particle
simulators \citep{Iwasawaetal2016}.  The basic idea of FDPS is to
provide a set of library functions necessary for high-performance,
highly scalable particle simulation codes. FDPS receives from user
applications the definition of particle data (originally in C++ class)
and a function to evaluate particle-particle interactions as the
source code. FDPS itself is written as a template library which is
compiled with these user-defined class and interaction function. Thus,
a user's application can call functions defined in FDPS to process
particles they defined and to calculate the interactions they
defined. We have extended the API so that FDPS can accept the particle
data class (or struct) and the interaction function written in both
Fortran \citep{2018PASJ...70...70N} and C. The pure-C language
interface makes it possible for programs written in any language with
reasonable FFI to C language to use FDPS functions.

FDPS relies on parallel Barnes-Hut tree algorithm with domain
decomposition by multisection algorithm and local essential tree
method \citep{Makino2004} for interaction calculation. Currently, FDPS
supports usual multicore architectures and also accelerator
architectures. In the case of the support of accelerator
architectures, the multiwalk algorithm \citep{Hamadaetal2009} is
used. This means that everything other than the interaction
calculation using the interaction list is done on the host side, and
the user-supplied interaction function need to take care of the data
transfer between the host and accelerators.

Thus, currently FDPS does not support heterogeneous many-cores very
well. One could  use the  accelerator support, but the performance gain
would be rather limited because of the reasons described above.
In order to improve the efficiency, it is necessary to move operations
other than the interaction calculation such as the tree construction
and the construction of the interaction list to the simple cores.

In this paper, we report the result of porting relatively simple
N-body simulation code for planetary ring systems, developed based on
our FDPS framework, to two heterogeneous many-core processors: Sunway
SW26010 and PEZY-SC2. The code is not yet the complete port of FDPS to
these processors, but more like a production code based on FDPS for a
specific problem. The reason why full FDPS port is not yet done is
simply that we made this porting partly to evaluate the architecture
and partly to try large-scale calculations which were not practical on
other architectures.

In the rest of this paper, we
first give an overview of the current state of the arts for the
large-scale simulations of planetary rings in section \ref{sect:rings}, and then
short description of the architecture of the two systems
in section \ref{sect:hardware}.  In section \ref{sect:innovation}, we
describe in detail the new algorithms we developed to achieve high
performance on extreme-scale heterogeneous many-core architectures.
In section \ref{sect:performance} we describe how the performance was
measured and achieved performance.  Section \ref{sect:summary} is for
discussion and summary.

\section{Simulation of planetary rings}
\label{sect:rings}

Saturn's ring was first observed by Galileo Galilei in 1610. For more
than three centuries, it had been the only known ring system within
our solar system. In 1977, rings of Uranus were found through
occultation observations from an aircraft, and then in 1979 rings of
Jupiter by Voyager 1 and in 1989 those of Neptune by Voyager 2.  Very
recently, it turned out that some of minor planets also have
rings. The first distinctive example is 10199 Chariklo, whose orbit is
between those of Saturn and Uranus (and thus one of Centaurs). There
are probably more Centaurs with rings.

Thus, quite recently, a wide variety of ring systems have been found.
How these rings were formed and have evolved is an important question
in planetary science, and large-scale, global simulation, if possible,
would help greatly to advance our understanding.

Planetary rings are usually at the radii around the Roche limit. Thus,
mutual gravity between particles does not easily lead to the formation
of new satellites, but is important enough to form spiral waves
(``wakes'') in very small scales, which increase the effective
viscosity and should enhance the radial transport of the angular
momentum. On the other hand, the actual ring system seems to consist
of very large number of narrow rings, separated with distinct gaps. It
is believed that these gaps are maintained by high-order resonances
with small embedded satellites (so-called moonlets), but whether or
not such gaps can be formed and maintained by resonances has not been
fully understood.

Up to now, most of simulations of ring structures have been local
ones, in which a small patch was cut out from the ring and simulated
under the assumption of the local Hill approximation and periodic
boundary condition \citep{WisdomTremaine1988}. Rein and Latter
\citep{ReinLatter2013} performed ``Large-scale'' simulation of viscous
overstability in Saturn's rings, using up to 204,178 particles and up
to 10,000 orbits using this local approach.  Because very long
simulations are necessary, the number of particles has been
small. They used {\tt REBOUND} \citep{ReinLiu2012}, an MPI-parallel
$N$-body simulation code.

Michikoshi and Kokubo \citep{MichikoshiKokubo2017} performed global
simulations of rings with the largest number of particles reported so
far. They used 300M particles to model two narrow rings of
Chariklo. They have developed their parallel code using the framework
we developed, FDPS \citep{Iwasawaetal2016, 2018PASJ...70...70N}.

Almost all previous studies of planetary rings adopted so-called
``local'' approximation, in which only a small patch of a ring is
simulated assuming periodic boundary condition in both radial and
azimuthal directions.

Michikoshi and Kokubo \citep{MichikoshiKokubo2017} performed global
simulations of rings with 300M particles, using FDPS
\citep{Iwasawaetal2016, 2018PASJ...70...70N}. They so far followed the system only for 10
orbital periods.

The total calculation cost is roughly proportional to number of
particles multiplied by the number of orbital periods followed, since
the calculation cost per timestep is $O(N \log N)$ when Barnes and Hut
tree algorithm is used and the number of timestep required for ring
simulations is essentially independent of the number of
particles. Thus, we can conclude that the size of state-of-the-art
simulations of planetary rings is around $10^9$ particle-orbits, or
around $10^{12}$ particle-steps.

We should note that even though the simulations so far done in this
field is relatively small, that does not mean there is no need or
possibilities for larger scale simulations. If we want to model the
global structures of rings, we cannot rely on local treatment. For
example, the effect of resonances with small satellites can only be
studied using global simulations. On the other hand, the number of
particles one need for global simulations, even for a very narrow
radial range, is very large. For example, consider the A ring of
Saturn with the radius of around $1.3\times 10^5\rm km$. The typical
radius of ring particles is 6~m \citep{ZEBKER1985531}, and the optical
depth of the ring is around unity. Thus, we need $10^4$ particles per
square km or around $10^{12}$ particles for the radial range of 100
km. With this radial range, we can model many of fine features
observed by Cassini directly.

If we could use particles with larger size, we could reduce the number
of particles required significantly. However, that would change the
viscous diffusion timescale of the ring, and thus what would be
observed. It is necessary to perform
simulations with particles of real physical radius, which would require
at least $10^{16}$ and ideally $10^{19}$ particle steps.

In other fields of astrophysics, very large simulations have been
performed. For example, Ishiyama \citep{Ishiyama2014} used $4096^3$
particles to follow the formation and growth of dark matter halos of
smallest scales. This simulation corresponds to $10^{16}$ particle
steps. Part of this calculation was performed on K computer. The
performance of K computer is $4.0\times 10^{10}$ particle steps per
second on the entire K computer, or 60,000 particle step per second
per core for a processor core with the theoretical peak performance of
16 Gflops \citep{Ishiyamaetal2012}. The efficiency they achieved is
55\% of the theoretical peak.

The algorithms used in large-scale $N$-body simulations are rather
similar, and that means they are well studied and close to optimal.
All of them use domain decomposition and Barnes and Hut tree
algorithm. For domain decomposition, several variations have been
used, such as Orthogonal Recursive Bisection \citep{Salmon1990},
Hashed Oct Tree \citep{WarrenSalmon1992}, Multisection
\citep{Makino2004}.

Efficient implementations on large-scale GPGPU clusters exist
\citep{Hamadaetal2009, PortegiesZwartetal2014, Bedorfetal2014}.
\citet{Bedorfetal2014} performed the simulation of Milky Way Galaxy
using $2.42 \times 10^{11}$ particles. The achieved performance is
24.77 PF on ORNL Titan, and one timestep took 5.5 seconds. Thus they
have achieved the performance of $4.4 \times 10^{10}$ particle steps
per seconds. The theoretical peak performance of Titan is 73.2 PF in
single precision. Thus, the achieved efficiency is 33.8\%.

\section{Sunway TaihuLight and PEZY-SC2 systems}
\label{sect:hardware}

In this section, we briefly describe the features of the Sunway
TaihuLight system and two systems with PEZY-SC2 processors: Gyoukou
and Shoubu System B. For more details of TaihuLight system see
\citet{Fu2016}. TaihuLight consists of 40960 Sunway 26010 processors,
and Gyoukou and Shoubu System B 13312 and 512 PEZY-SC2 processors,
respectively. Unfortunately, Gyoukou was turned off by March 31, 2018,
and thus our performance measurement on Gyoukou system was based on a
preliminary version of the simulation code, and the efficiency
measured on Gyoukou is lower than that measured on Shoubu System B.

One SW26010 processor consists of four  CGs (core groups), each with one
MPE (management processing element) and 64 CPEs (computing processing
elements). Both MPE and CPE are 64-bit RISC cores. MPE has L1 cache
memories for  both instructions and data, and also L2 data cache. On
the other hand, each CPE has L1 instruction cache and 64KB of local
data memory. CPEs can communicate with the main memory through DMA.
Each CPE can initiate multiple asynchronous DMA operations.

Each core group is connected to 8GB DDR3 DRAM memory with the theoretical
peak transfer rate of 34GB/s. The processor runs at the clock
frequency of 1.45GHz, and each core (both MPE and CPE) can perform
four double precision FMA operations. Thus, the theoretical peak
performance of one processor is 3016 Gflops and that of one CG is 754
Gflops. Thus, even when we use the nominal number for DRAM bandwidth,
the B/F ratio is only 0.045. This is less than 1/10 of the number for
K computer.

Compared to that of K computer, the network is also  weak,
with the total bandwidth of around 10GB/s per node. This is about the
same as the performance of a single link of 6D torus network of K
computer. Since the SW processor is around 25 times faster than the
SPARC64 processor of K computer, the relative network bandwidth is
different by  more than two orders of magnitudes.

One PEZY-SC2 processor chip consists of  2048 processors (64
of them are disabled and the available number of processors is
1984). Each processor can perform 1, 2 and 4 multiply-and-add
operation for FP64, FP32, and FP16 data. For FP32 and FP16, 2- and
4-way SIMD operations are performed.  With the clock speed of 700MHz,
the theoretical peak speed is 2.8, 5.6 and 11.1TF, for FP64, FP32 and
FP16, respectively. At present, each SC2 processor chip have 4 channels of
DDR4 memory, for the peak throughput of 76.8GB/s. Thus B/F is 0.027.

They have three levels of shared cache, but without
coherency. Instead, they have explicit cache flush instructions to each
levels. Two processors share L1D, and 16 processors L2D, and all
processors LLC. Each processor runs either four or eight threads
simultaneously. Thus, it is relatively easy to hide the  latency of
the arithmetic units and L1D.

In the original design, each SC2 processor chip had six MIPS64 cores,
which were supposed to run the operating system and main body of the
application programs. Unfortunately, currently they are disabled, and
operating system and application programs run on the frontend  Xeon~D-1571
processor. Each Xeon~D  hosts eight SC2 processors. Thus, the
performance ratio between Xeon~D and SC2 is close to 100.
Moreover, these eight SC2 are connected to Xeon~D through single PCIe
Gen3 16-lane channel. Thus, the peak data transfer speed between one
SC2 and Xeon~D is 2 GB/s, for the peak speed of 2.8TF.

In summary, TaihuLight and two systems based on PEZY-SC2 processors  share the following
characteristics:

\begin{enumerate}

\item Very large performance ratio between ``general-purpose''
  and ``computing'' cores, close to 1:100.
\item  Very small memory  B/F numbers, around 0.03.
\item  Even smaller network B/F numbers, 0.006 or 0.001.
\item Very large number of MPI processes, 160k  or 10k.
\item Very large number of ``computing'' cores per MPI process, 64 or 1984.
\end{enumerate}

Just one of these characteristics makes it very difficult to achieve
reasonable performance for particle-based simulations using previously
known parallelization algorithms. In the next section, we describe the
new algorithms we implemented to achieve good performance on these systems.

\section{New algorithms for extreme-scale simulations}
\label{sect:innovation}

  \subsection{Overview of new algorithms}

In this section, we describe the new algorithms we made in order to
utilize TaihuLight and PEZY-SC2 based systems for the simulations of self-gravitating
planetary rings. The following is the list of new algorithms.

\begin{enumerate}
\item The reuse of the interaction list over multiple timesteps.
\item Elimination of  the global all-to-all communication.
\item ``Semi-dynamic'' load balance between computing cores
\item Optimizations specific to the ring geometry.
\end{enumerate}

For TaihuLight and PEZY-SC2 based systems, we have modified our FDPS
framework in architecture-specific way so that we implement the
algorithms and run the code under the limited available time and
software environment. However, many of these algorithms are ported
back to the original FDPS so that anybody who uses FDPS can take
advantage of these new algorithms.

In the rest of this section, we briefly describe these new innovations.

\subsection{Reuse of the interaction list}
\label{subsec:list}

The following gives the usual steps for highly parallel code for
self-gravitating particle system:

\begin{enumerate}

  \item Perform domain decomposition.
  \item Exchange particles so that particles belong to appropriate domains.
  \item Perform interaction calculation using fast algorithm such as
    Barnes-Hut tree.
  \item Integrate the orbits of particles.
  \item Go back to step 1.

\end{enumerate}

In the case of approaches with local essential tree, step (3) consists
of the following substeps:

\begin{description}

\item{(3a)} Construct the ``local'' tree structure from particles in
  the domain.
\item{(3b)} Collect the information necessary for the calculation of
  interaction from other processes (so called local essential tree).
\item{(3c)} Construct the ``global'' tree from the collected information.
\item{(3d)} For small groups of particles, traverse the tree and
  calculate the interaction. Repeat this for all groups.
\end{description}
In the original algorithm \citep{BarnesHut1986}, the traversal of the
tree is done for each particle, and force calculation is done during
the traversal. However, on almost all modern implementation, following
the idea of Barnes \citep{Barnes1990}, tree traversal are done for
groups of neighboring particles, which are constructed using the tree
structure itself. During the traversal for a group, the list of
particles and tree nodes which exert the force on this group of
particles is constructed, and actual force calculation is done through
the double loop over particles in the group and those in the
list. This structure makes it possible to use vector pipelines, scalar
SIMD units, and even special-purpose computers \citep{Makino1991c}
with high efficiency. For GPGPUs, the extension of this algorithm, in
which multiple lists are constructed and then sent to GPGPU, is used
\citep{Hamadaetal2009}.

This approach does not work well on TaihuLight or PEZY-SC2 based systems, because of
the low performance of general-purpose core and limited memory bandwidth.
The
performance we can achieve with either approach for ring simulation on
these machines is less than 1\%.
Thus, it is necessary to reduce the cost of tree construction and tree
traversal, and we achieved this by using the same interaction lists
over multiple timesteps. We call this method the persistent
interaction list method.

The idea behind this method is essentially the same as that for
the  neighbor-list method used in many simulation codes for particles
with short-range  interactions.

By using this persistent interaction list, we can reduce the
calculation cost of the part other than the interaction calculation
drastically. While we are using the same interaction lists, we skip the
domain decomposition, exchange of particles, construction of the local
tree. We still need to update the physical quantities of the nodes of
the tree, since particles move at each timestep.
We first update the information of the nodes of
the local tree. Then,
using the list of nodes for the local essential tree, the
communication between the nodes is performed. Finally, the physical
quantities of the global tree are updated, and the force calculation is
performed using this updated global tree and the persistent
interaction list.

The most time-consuming part of the tree construction is the
sorting. In the case of TaihuLight, we implemented the parallel sample
sort \citep{cmsort} on CPEs. In the case of Gyoukou, the sorting was
performed on Xeon~D host processor. In the case of Shoubu System B, it
was performed on the side of PEZY-SC2 processors. Also, some other
operations are moved from Xeon-D to PEZY-SC2. Thus, the overall
performance is significantly better for Shoubu System B. As stated
earlier, Gyoukou was turned off on March 31, 2018, and we could not
measure the performance of our improved code on systems with more than
512 PEZY-SC2 processors.

We have ported all operations in timesteps in which the interaction
list is used (list-reusing steps), except for MPI functions for
communication, to CPEs (TaihuLight) or SC2 processors (PEZY-SC2 based
systems).  For the timestep in which the interaction list is
constructed (list-constructing step), some of operations are still
done on Xeon~D in the case of PEZY-SC2 based systems.

\subsection{Tree and Domain structures on Cylindrical Coordinate}
\label{subsec:cylcoord}

We want to model a relatively narrow ring, and this means the usual
domain decomposition in Cartesian coordinates can cause serious
problems. Figure~\ref{fig:domain_cart} illustrates the problem. We can
see the domains near the $y$ axis are very elongated. This irregular
shape of domains results in the increase of communication between
processes, and thus serious degradation in the efficiency.

\begin{figure}
  \centering
  \includegraphics[width=8cm,clip]{./fig/domain_cart.eps}
  \caption{Schematic figure of domain decomposition by the multisection
    method in $x$-$y$ coordinate. Domains are divided by $16 \times 16$.}
  \label{fig:domain_cart}
\end{figure}

\begin{figure}
  \centering
    \includegraphics[width=8cm,clip]{./fig/domain_cyl.eps}
  \caption{Schematic figure of domain decomposition by the multisection
    method in cylindrical coordinate. Domains are divided by $2 \times 128$.}
  \label{fig:domain_cyl}
\end{figure}

We can avoid this problem, if we apply the domain decomposition
in the  cylindrical coordinates (figure  \ref{fig:domain_cyl}).
Note that we can also use the  cylindrical coordinates for the
construction of the tree.  Since
the ring is narrow, the local distance $s$ in the Cartesian coordinates
$(x, y, z)$ can be approximated by that in cylindrical coordinates
($r$, $\phi$, $z$).
\begin{equation}
  \label{eq:metric}
  ds^2 = dx^2 + dy^2 + dz^2 \sim d\phi ^2 + dr^2 + dz^2,
\end{equation}
when $r \sim 1$. Thus,
we can use the cylindrical coordinate
for domain decomposition and tree
construction and even for the tree traversal,  without any modification of the algorithm or program
itself. The actual interaction calculation is faster in Cartesian
coordinates and thus Cartesian coordinates is used.

\subsection{Coordinate rotation}
\label{subsec:exptcl}

The simulation of ring with very large number of processes poses
new challenges. As we increase the number of processes, the size of
the domains becomes smaller. On the other hand, the timestep does not
become much smaller even when we increase the total number of
particles, since the random velocities of ring particles become
smaller when we increase the number of particles. Thus, the distance
that particles move can be comparable or even larger than the domain
size, resulting in the increase in the amount of  communication.

We can ``solve'' this problem by the rotation of  the coordinates and domain
structure, so that particles do not move much. If we rotate the
coordinates at the speed of Kepler rotation at the center of the ring,
particles at the center of the ring do not move much. Particles at
other radial positions still move, but the speed becomes much smaller
than that of the Kepler rotation. Thus, communication due to Kepler
rotation can be almost eliminated.

\subsection{Elimination of all-to-all communication}
\label{subsec:exlet}

In FDPS, the exchange of LET (local essential tree) data is done
though a single call to the {\tt MPI\_Alltoallv} function.  This
implementation works fine even for full-node runs on K computer, but
becomes problematic on systems with relatively weak network like
TaihuLight and PEZY-SC2 based systems. We can eliminate this all-to-all communication, by
constructing the ``tree of domains'' locally and let only higher-level
information be sent to distant processes.

In the current implementation specialized to narrow rings, we
implemented a very simple two-level tree, in which the second-level
tree nodes have all processes in the radial direction. For example, if we
have a process grid of (1000, 10), where 1000 in angular and 10 in
radial direction, 10 domains in the radial direction are combined to
one tree node, resulting in 1000 second-level nodes. Only these 1000
nodes exchange their center-of-mass information. All LET information
other than these center-of-mass data of second-level nodes are sent
either to other second-level nodes (and then broadcast to
lower-level nodes) or sent directly to lower-level nodes.

In this implementation, there is still one global communication in the
angular direction, but we can use {\tt MPI\_Allgather} since only the
top-level data are sent. Thus the reduction in the communication was
quite significant.

\subsection{Load Balance among computing cores}
\label{subsec:force}

In our current implementation, interaction lists are created at the
list-construction step, and are reused for several steps. The total
number of lists in one MPI process is around $10^5$, and we need to
use 64 or 1984 computing cores efficiently for them.
If we just assign a fixed number of lists to cores, random variation
of the list length can result in large load imbalance. Therefore, some
load balance strategy is necessary. We applied the following simple algorithm.

\begin{enumerate}

\item Sort the interaction lists by their length.

\item Assign the longest 64 lists on 64 CPEs (in case of TaihuLight).

\item For each remaining list, assign it to the the CPE with the
  shortest total calculation cost.

\end{enumerate}

Since the calculation time of cores is quite predictable, this
algorithm works very well.

In the case of PEZY-SC2 based systems, we further improved the load
balance by using multiple cores which share the cache for one
interaction list.

\subsection{Interaction Kernel}

In the case of TaihuLight, we found the compiler-generated code for
the interaction kernel, even when SIMD operations are used, does not
give very good performance. We rewrite the interaction kernel fully in
the assembly language, with hand-unroll and careful manual scheduling. As
a result, we achieved more than 50\% of the theoretical peak
performance for the kernel.

We have applied similar optimization also on PEZY-SC2 based
systems. In addition, on PEZY-SC2 based systems we used
single-precision calculation for the interaction kernel. In order to
avoid the large roundoff at the first subtraction of the position
vectors, both positions and velocities are shifted with the new origin
at the position of one of the particles which share the interaction
list.  After this shifting, positions and velocities are converted to
single precision, and actual interaction calculation is done using
single-precision SIMD operations.

\section{Measured performance}
\label{sect:performance}
\subsection{How the performance is measured}

To measure the performance, we measure the time for 64 timesteps,
including the time for  diagnostics.
The
execution time is measured by the MPI wallclock timer, and operation
count is from the counted number of interactions
calculated. Equation~\ref{eq:interation}  gives the definition of the
particle-particle interaction.

\begin{equation}
  \bm F_{ij} = \begin{cases} G \dfrac{m_i m_j}
    {r_{ij}^3} \bm r_{ij} & \left(r_{ij} > r_\text{coll} \right)
    \\
    \Biggl[  G \dfrac{m_i m_j} {r_\text{coll}^3}  + \dfrac{m_j}{m_i
        + m_j}  \times \\ \qquad  \left(      \kappa \dfrac{r_{ij} -
        r_\text{coll}}{r_{ij}}    + \eta \dfrac{\bm r_{ij} \cdot \bm
        v_{ij}}{r_{ij}^2}    \right) \Biggr] \bm r_{ij} & \left(
    r_{ij} \le r_\text{coll} \right) \end{cases}
  \label{eq:interation}
\end{equation}
}
with
$\bm r_{ij} = \bm r_j - \bm r_i$, $\bm v_{ij} = \bm v_j - \bm v_i$,
$r_{ij} = \| \bm r_{ij} \|$

Here, ${\bm F_{ij}}$ is the acceleration of particle $i$ due to
particle $j$, ${\bm r_{ij}}$ and ${\bm v_{ij}}$ are the
relative position and velocity vectors, $G$ is the gravitational
constant (taken to be unity in this paper), $m_i$ is the mass of
particle $i$,  $r_\text{coll}$ is the distance at which
two particles collide, and $\eta$ and $\kappa$ are parameters which
determine the coefficient of restitution. We chose these parameters
so that the coefficient of restitution in radial direction is 0.5.

We used this form to calculate all particle-particle interaction. For
particle-tree-node interaction, we used center-of-mass
approximation. Particle-particle interaction consists of 9
multiplications, 8 additions, and one square root and one division
operations. Instruction set of Sunway 26010 processor does not include
fast approximation for neither square root or reciprocal square
root. So we implemented fast initial guess and high-order convergence
iteration in software. The number of operations in this part is 7
multiplications, 5 additions and two integer operations. Therefore,
for particle-cell interactions the number of floating-point operations
is 31, and for particle-particle interactions, which include the
repulsive force during physical collisions, is 49.  The total number
of floating-point operations is obtained by counting the number of
interactions calculated and multiply them with these number of
floating-point operations per interaction. We ignore all operations
other than the interaction calculation, since as far as the number of
floating-point operations is concerned, that for interaction
calculation is more than 99\% of total operation count.

For PEZY-SC2 based systems we used the same operation count as we used
for TaihuLight, in order to make the direct comparison possible, even
though the details of the implementation of the force kernels are
different.

For the weak-scaling measurement, we have performed runs with 10M
particles per MPI process on TaihuLight and PEZY-SC2 based
systems. Initial condition is such that the ring width and ring radius
is unchanged. Table \ref{tab:initialcoditions} summarizes the initial
condition.

\begin{table}
\centering
 \caption{Initial condition for weak scaling runs}
 \label{tab:initialcoditions}
 \begin{tabular}{lc}
\hline
   Central planet & Saturn\\
   Ring inner radius & $10^5$ km\\
   Ring width        & $100$ km\\
   Number of MPI processes & 1024 -- 160,000 \\
   Number of particles per process & $10^7$  \\
   particle radius & 3.5 -- 500 m\\
\hline
\end{tabular}
\end{table}

\subsection{Performance Results}
\begin{figure}
  \includegraphics[width=8cm]{./fig/weak_scaling2.eps}
  \caption{Time per timestep for weak-scaling test. The number of
    particles per process is 10M. Triangles, crosses and squares show the
    results on TaihuLight, Gyoukou and Shoubu System B, respectively.
  }
  \label{fig:weak}
\end{figure}

\begin{figure}
  \centering
  \includegraphics[width=8cm, clip]{./fig/weak_scaling_speed.eps}
  \caption{Performance in petaflops for weak-scaling test.  The number of
    particles per process is 10M. Triangles, crosses and squares show the
    results on TaihuLight, Gyoukou and Shoubu System B, respectively.}
  \label{fig:weakpf}
\end{figure}

\begin{figure}
  \centering
  \includegraphics[width=8cm, clip]{./fig/strong_scaling.eps}
  \caption{Time per timestep for strong-scaling test.  Triangles and crosses  show the
  results on TaihuLight and PEZY-SC2 based systems, respectively. The total number of
  particles is $10^{11}$ on TaihuLight and $10^{10}$ on PEZY-SC2 based systems.}
  \label{fig:strong}
\end{figure}

Figures \ref{fig:weak} and \ref{fig:weakpf} shows the time per one
timestep and the performance, for the weak scaling measurements. We
can see that the weak scaling performance is quite good on both of
TaihuLight and PEZY-SC2 based systems. The peak performance of single
MPI process is roughly four times faster for PEZY-SC2 based systems,
and that's the reason why they are three to four times faster in this
weak-scaling measurement. We can see that Shoubu System B is about
50\% faster than Gyoukou. As we have already discussed, this is not
due to any hardware difference but the difference in the software
used.

Figures \ref{fig:strong} shows the strong scaling result.  The total
number of particles is $10^{11}$ and $10^{10}$, for TaihuLight and
Gyoukou. We do not show the strong-scaling result for Shoubu System B,
since it is rather small system and strong-scaling result is not so
meaningful.
We can see that speedup is almost linear.

\begin{table}
\centering
  \caption{Breakdown of calculation time for weak-scaling runs}
  \label{tab:timeweak}
  \begin{tabular}{ccccc}
    \hline
 System  & \# processes & interaction & comm. & others\\
    \hline
&10000 &   2.07& 0.041 & 0.466\\
&20000 &  1.63& 0.040  & 0.590 \\
TaihuLight &40000  &  2.13 & 0.064 & 0.478 \\
&80000 &   1.71 & 0.053 & 0.630\\
&160000 &  2.31& 0.090& 0.476\\
\hline
&1024&  0.332 &  0.114 & 0.281\\
Gyoukou &2048&   0.392 & 0.121 & 0.235\\
&4096&  0.355 & 0.143& 0.289\\
&8192   & 0.453& 0.147&     0.222\\
\hline
        &8&  0.327 &  0.018 & 0.132\\
Shoubu B &32&   0.344 & 0.020 & 0.181\\
        &128&  0.348 & 0.027& 0.132\\
        &512& 0.360& 0.030&     0.135\\
\hline
\end{tabular}
\end{table}

\begin{table}
\centering
  \caption{Breakdown of calculation time for strong-scaling runs}
  \label{tab:timestrong}
  \begin{tabular}{ccccc}
    \hline
 System  & \# processes & interaction & comm. & others\\
    \hline
& 10000& 2.0738 & 0.0410 & 0.4658 \\
& 20000& 1.0499 & 0.0253 & 0.2426 \\
TaihuLight & 40000& 0.5565 & 0.0298 & 0.1125 \\
& 80000& 0.2991 & 0.0233 & 0.0652 \\
& 160000& 0.1765 & 0.0322 & 0.0356 \\
\hline
& 1024& 0.3323 & 0.1140 & 0.2808 \\
Gyoukou & 2048& 0.1512 & 0.0668 & 0.1658 \\
& 4096& 0.0854 & 0.0417 & 0.0923 \\
& 8192& 0.0538 & 0.0357 & 0.0582 \\
    \hline
\end{tabular}
\end{table}

Tables \ref{tab:timeweak} and \ref{tab:timestrong} show the breakdown
of the calculation time per one timestep, again for both the weak and
strong scaling runs. As expected, in the case of strong-scaling runs,
the calculation time for communication does not decrease
significantly, and eventually limits the performance.  As already
stated, our main interest is to use very large number of
particles. Therefore, for actual scientific runs, the communication
time would not become the limiting factor.

If we compare the calculation times on Gyoukou and Shoubu System B, we
can see that the times for the interaction calculation are
similar. but for both communications and ``others'', Shoubu System B is
much faster. Again, this is not due to hardware difference but due to
software difference.

The performance of run for $1.6\times 10^{12}$ particles on 160k
processes (40000 nodes) of TaihuLight is 47.9 PF, or 39.7\% of the
theoretical peak performance of the Sunway TaihuLight system.  On
PEZY-SC2 based systems, we achieved 10.6PF for $8\times 10^{9}$
particles on 8K SC2 chips, or efficiency of 23.3\% of the theoretical
peak performance. On 512-chip Shoubu System B, we achieved the speed
of 1.01 PF, or 35.5\%

The overall efficiency we achieved on PEZY-SC2 based systems is a bit lower compared
to that on TaihuLight. This difference is not due to any fundamental
difference in the architecture but purely due to the limitation on the
available time for program development and performance measurement. As
we stated, the calculation in the list-construction step, such as
the constructions of the tree and the interaction lists are currently
done on Xeon~D, and around 40\% of the total time is consumed in this
step at the time of measurement on Gyoukou. Most of these are now done
on SC2 side, and that is why the performance of Shoubu System B is
better than that of Gyoukou.

In terms of the number of particles integrated per second, we have
achieved $5.5\times 10^{11}$ particles per second, which is more than
10 times faster than the results of previous works on K computer
\citep{Ishiyamaetal2012} or ORNL Titan \citep{Bedorfetal2014}.

\section{Discussion and summary}
\label{sect:summary}

\subsection{Performance Portability}

We have reported the measured performance of two rather different HPC
systems, Sunway TaihuLight and PEZY-SC2 based systems, for the same
large-scale simulation of self-gravitating planetary rings.  In both
cases, we have achieved fairly high efficiency, more than 30\% of the
theoretical peak. The parallel algorithm used is essentially the same
for the two systems. However, the actual codes are rather different,
simply because of the difference in the architecture and the software
development environment.

Sunway TaihuLight has a heterogeneous many-core architecture
integrated in one chip. Thus, the CPU (MPE in their terms) and
accelerators (CPE in their terms) share the same physical memory, but
CPEs lack the data cache and need to rely on DMA controller to access
the main memory efficiently.

On TaihuLight, one can use OpenACC compiler. However, in order to
achieve high performance, one is practically forced to use the Athread
call, which makes the 64 CPEs and their local memories visible to
programmers.

On the other hand, PEZY-SC2 systems, at least at present, have a
rather classical accelerator-based architecture, in which CPU (a
Xeon-D processor) and accelerators (PEZY-SC2 processors) are connected
through PCI Express interface. This means that they have separate
physical memories. Within one chip, however, processing elements of
PEZY-SC2 processor have three levels of data caches. Currently PZCL, a
dialect of OpenCL, is supported on PEZY-SC2 based systems.

Because of these differences (shared and separate memory, DMA and
cache, thread-based and OpenCL-like), the actual programs for two
machines have become quite different, even though the algorithms used
are the same and the problem to be solved is the same.

Both codes, however, are based on our framework, FDPS
\citep{Iwasawaetal2016, 2018PASJ...70...70N} and follow its
structure. The basic idea of FDPS is to separate the implementation of
parallel algorithms and description of the physical problem. FDPS
provides the former and the application programmers provides the
latter, in the form of the data type definition of particles and
functional form of particle-particle interaction. Users of FDPS can
write their programs by specifying the data structure of particles
they use, and calling necessary FDPS functions for domain
decomposition, particle migration between processes, and interaction
calculation. Currently, users should provide optimized function for
particle-particle interaction calculation.

Many of the parallel algorithms we newly implemented are not specific
to planetary rings but can be applied to any other particle-based
simulations. Using FDPS, users can write their programs in their
favorite language (currently C++, Fortran and C are supported)
\citep{2018PASJ...70...70N}, and let FDPS do complex parallelization.

Thus, it seems that one way to achieve performance and program
portability on new machines with rather exotic architecture such as
the machines evaluated in this paper is to develop the framework with
a common API and internal implementations specialized and optimized to
specific architectures. This is fairly straightforward in the case of
TaihuLight, in which the CPE and MPEs share the single physical
memory, since the data structure that FDPS handles can still in the
shared main memory. The basic data structure of FDPS is just an array of
particles, and both the user-developed  application program and the FDPS
side can access that particle array in usual way.

On the other hand, how the separate memory spaces of PEZY-SC2 should
be handled within FDPS requires a bit more consideration. One
possibility would be to add the interface in which the user-side
programs, for example the function to perform time integration, is
passed to FDPS, instead of directly called within the user-side
program. Here, the function to be passed applies to single particle
(or some small array or particles), and applying it to all particles
in the system will be the responsibility of FDPS.  This approach will
probably make the software development and performance improvement
easier on other machines, since parallelization in both MPI and OpenMP
level can be taken care of within FDPS.

This problem of portability is of course not limited to FDPS. It
occurs in practically any application in any field of
computational science. We clearly need a new and systematic approach
to solve this problem, and we think the use of frameworks such as
FDPS may be an efficient and practical way.

Our view of frameworks is that they should allow users to express
their problems in simple and (ideally) machine-independent way. In the
case of particle-based simulations, we have designed FDPS to meet this
goal, and it actually works pretty well on large HPC systems, both
with and without GPGPUs. Our experience on TaihuLight and PEZY-SC2
indicates that it is also possible to extend FDPS to cover these
systems. We believe similar approaches will be used in other fields.

\subsection{Summary}

In this paper, we described the implementation and performance of
a highly efficient simulation code for self-gravitating planetary
rings on  Sunway TaihuLight and PEZY-SC2 based systems.

The measured performance is 47.9 PF, or 39.7\% of the theoretical
peak, for simulation of $1.6\times 10^{12}$ particles on 40,000 nodes
of TaihuLight, 10.6PF, or 23.3\% of the theoretical peak, for
simulation of $8\times 10^{10}$ particles on 8192 nodes of Gyoukou,
and 1.01PF, or 35.5\% of the theoretical peak for $5\times 10^{9}$
particles on 512 nodes of Shoubu System B. As noted earlier, Gyoukou
and Shoubu System B use the same PEZY-SC2 processor. The difference in
the efficiency is purely due to the fact that Gyoukou was turned off
on March 31, 2018. The software at that time was still under
development.

Compared to previous achievements on K computer or ORNL Titan, the
achieved efficiency is similar or higher, and the speed in terms of
the number of particles integrated per second is higher, for both
TaihuLight and PEZY-SC2 based systems. As we stated earlier, this
level of performance would not be achieved without the new algorithms
described in this paper.

Compared to other multi-core processors for modern  HPC systems such
as Fujitsu SPARC64 VIIIfx and IXfx or Intel Xeon Phi processors,
both SW26010 processor  of TaihuLight and PEZY-SC2 processor of
PEZY-SC2 based systems have several unique features which allow very high peak
performance but at the same time make it much harder to achieve high
efficiency on real applications. These are:

\begin{itemize}

  \item Heterogeneous architecture with rather extreme performance
    ratio of 1:64 in the case of SW26010 and even larger in the case
    of SC2.
  \item The lack of cache hierarchy (SW26010) or cache coherency (SC2).
  \item Very limited main memory bandwidth, with B/F values around
    0.02--0.04.  This is about 1/10 of the numbers of Fujitsu or Intel HPC processors.

\end{itemize}

On the other hand, SW26010 comes with very well-thought features which
allows the programmers to optimize the performance of code on
CPE. These features include:
\begin{itemize}

\item Low-latency DMA controller which can be initiated by any CPE.
\item Low-latency, high-bandwidth communication between CPEs.

\end{itemize}

These two features allow very efficient use of the main memory
bandwidth. The two-dimensional structure of the network within CG seem
to be optimized for highly efficient implementation of matrix-matrix
multiplications, but it is actually quite useful for other real
applications, whenever fast inter-core communication is necessary.

It is certainly true that the need to use DMAs for data transfer
between CPE and main memory complicates the use of CPE. However, it is
also true that it makes quite optimized access to main memory
possible, since the application programmer can (or have to) control
all main memory accesses. In the case of our code, in several places
we have ``vectorizable'' loops, which perform the same operation on all
particles in the system. The number of operations per particle is
relatively small, of the order of ten, and the data size of one
particle is 32 bytes. In the case of manycore architecture with
hierarchical cache memory, to achieve high efficiency on simple vector
operations like

{\tt a[i] =   b[i]+c[i]}

is actually quite complicated. In modern processors, load address
would be predicted and hardware prefetch is generated. The hardware
prefetch would probably work for a very simple loop like the above
example, but would fail if many vectors are loaded. Then the
programmer need to experiment with software prefetch, to find the way
to get  the best performance.

In the case of SW26010, currently it is rather tedious and error-prone
to write the equivalent operation using the combination of Athread and
DMA, and sometimes inner kernel in assembly language, but once we do
so, we can get a performance close to the theoretical limit
relatively easily.

The existence of low-latency (less than 10 clock cycles) communication
path between CPEs is quite important for using CPEs for fine-grain
parallelism such as loop-level parallelization. Such low-latency
communication is difficult to implement on shared memory processors
with hierarchical cache.

The SC2 processor supports the cache flush and synchronization at each
level of the cache hierarchy, making the relatively low-latency
communications between processors possible.  However, it is clearly
desirable to have more direct control of interprocessor communication.

One common problem of SW26010 or SC2 is that writing high-performance
kernel for them means writing the inner kernel in the assembly
language. This is purely the software limitation, and probably not so
difficult to fix. In this aspect, SC2 is somewhat easier to deal with,
since it supports 8-way multithreaded execution, which effectively
hides the latencies of L1 cache and arithmetic unit from compiler.

In conclusion, we have implemented parallel particle simulation code
on Sunway SW26010 and PEZY-SC2 processors, and found that it is not
impossible to achieve high performance on their rather extreme
architectures with carefully designed algorithms. Even though the B/F
number are less than 0.1 and the network bandwidth is similarly low,
the efficiency we have achieved is comparable to that on K computer,
with 15 times more memory and network bandwidth. We feel that
architecture evolution in this direction will help the HPC community
to continue improving the performance. We also believe that high-level
software framework such as our FDPS will help many researchers to run
their own applications efficiently on new architectures.

\section*{Acknowledgment}

This work was supported by The Large Scale Computational Sciences with
Heterogeneous Many-Core Computers in grant-in-aid for High Performance
Computing with General Purpose Computers in MEXT (Ministry of
Education, Culture, Sports, Science and Technology-Japan), by JSPS
KAKENHI Grant Number JP18K11334 and JP18H04596 and also by JAMSTEC,
RIKEN, and PEZY Computing, K.K./ExaScaler Inc. In this research
computational resources of the PEZY-SC2 based systems supercomputer,
developed under the Large-scale energy-efficient supercomputer with
inductive coupling DRAM interface project of NexTEP program of Japan
Science and Technology Agency, has been used.

\bibliographystyle{SageH}

redpajama_set_name: RedPajamaArXiv


question BSA Pare bike, Green or brown ?
what was this box used for?
question Ammunition: can anyone identify the era of these rifle cartridges?
Browning .30 Cal and GPMG turret mounts.
question are these bren gun magazines?


text:
Canadian Dairy Giant to Acquire Dairy Crest For £975 Million
Canada-based Saputo Group, which is one of the top ten dairy processors in the world, has made a recommended all cash offer of £975 million to acquire Dairy Crest, the UK dairy business. The acquisition will allow Saputo Group to increase its international presence and enter the UK market by acquiring and investing in a well-established and successful industry player with a solid asset base and an experienced management team. Saputo Group does not currently have any operations in Europe and sees Dairy Crest as an attractive platform for growth in the UK.
Headquartered in Montreal, Saputo Group is the largest cheese manufacturer and the leading fluid milk and cream processor in Canada, the top dairy processor in Australia and the second largest in Argentina. In the United States, the Saputo Group ranks among the top three cheese producers and is one of the largest producers of extended shelf-life and cultured dairy products.
Employing approximately 15,000 people, Saputo Group operates 62 plants and sells its products in more than 40 countries. For the twelve months ended 31 March 2018, Saputo's consolidated revenues amounted to C$11.543 billion (£6.706 billion) and its net earnings amounted to C$853 million (£495 million).
Dairy Crest has transformed itself in recent years from a supply-driven commoditised business to a leading UK manufacturer of foods and added-value ingredients. Pivotal to this transformation was the sale, executed by Dairy Crest's current management team, of its Dairies operation to Müller in 2015.
Dairy Crest manufactures and markets a portfolio of market-leading brands including Cathedral City cheese, Clover dairy spread, Country Life butter and Frylight cooking spray. It also produces ingredients for the high growth global infant formula market – demineralised whey powder, a by-product from the cheese-making process, and galacto-oligosaccharides (GOS), a lactose-based prebiotic which helps to improve gut health.
During the last financial year, Dairy Crest raised just under £70 million through a share placement to increase cheese production capacity at its creamery in Davidstow, Cornwall. The proceeds are also being used to enable the site to become self-sufficient in the supply of water and to improve energy resilience, thereby reducing its environmental impact.
Dairy Crest employs approximately 1,100 people in seven locations in the United Kingdom – five operating sites, an Innovation Centre on the campus of Harper Adams University and a Head Office situated in the outskirts of London.
Lino A. Saputo, Jr, chairman and chief executive of Saputo.
For the twelve months ended 31 March 2018, Dairy Crest reported revenue of approximately £457 million, profit before tax (including exceptional items) of approximately £179 million and adjusted profit before tax of approximately £62 million.
Lino A. Saputo, Jr, chairman and chief executive of Saputo Inc, says: "Dairy Crest is an attractive platform for Saputo and fits well within our growth strategy. We believe that under Saputo ownership, Dairy Crest will be able to accelerate its long-term growth and business development potential and provide benefits to Dairy Crest's employees and stakeholders. This recommended offer represents a compelling opportunity for Dairy Crest Shareholders, providing immediate value certainty."
Stephen Alexander, chairman of Dairy Crest, comments: "The Acquisition should enable Dairy Crest to benefit from Saputo's global expertise and strong financial position to fulfil and accelerate its growth ambitions. The businesses have strong shared values and the board is confident that Saputo's plans to invest in and grow the Dairy Crest business mean the proposed transaction is positive for all its stakeholders.
"The price premium Saputo is offering reflects Dairy Crest's portfolio of market-leading brands, its manufacturing and innovation capabilities as well as its experienced management team, who will continue to run the business under Saputo's ownership. It is for these reasons that the board is unanimously recommending this transaction to Dairy Crest shareholders."
Glanbia Performs Well in First Half
Coca-Cola Company Adds Two More Billion-dollar Brands
$5.8 Billion Pet Food Acquisition