text:
#BookSpotlight #Reviews: The Day He Came Back by Penelope Ward
THE DAY HE CAME BACK IS LIVE!
A Standalone Contemporary Romance
New York Times, USA Today, and #1 Wall Street Journal Bestselling Author
It was the summer of my life.
I'd met the guy of my dreams.
Unfortunately, he was the son of my uppity employer—and very much off-limits.
But Gavin was a rebel. He knew his mother would disown him if she found out about us; in his eyes, we just had to be careful.
He never treated me as his mother did—like hired help.
Instead, Gavin put me on a pedestal and loved me harder than I'd ever been loved in my life.
What a summer it was.
Until it all ended—badly.
I was never supposed to see Gavin again.
That didn't stop me from thinking about him every day for ten years.
I knew little about his life now, just that he was an entrepreneur living an ocean away.
When a twist of fate had me working again in the very place our love affair started a decade earlier, I knew it was only a matter of time before I might see him again.
But I wasn't prepared.
What if he hated me?
What if he loved someone else now?
I wasn't prepared for all the unknowns.
And most of all, I wasn't prepared for today to be the day he came back.
Amazon: https://fave.co/2NcLeGh
Apple Books: https://fave.co/2ScFey5
Nook: https://fave.co/2MKredQ
Kobo: https://fave.co/2YEXXbb
Google Play: https://fave.co/2YIBLN9
Audio: https://adbl.co/2YaupCK
Paperback: https://fave.co/2xzNIFX
Goodreads: https://fave.co/2S5uxgp
By Janett
This book was simply beautiful and heartbreaking.
When Penelope Ward writes a story she gets your feelings one hundred percent involved with her characters and the storyline. While some people might say that one of the subject matters of this story are "old fashioned," this is something that still happens in a lot of countries.
Raven and Gavin loved each other at a very young age, they kept their relationship hidden from his family, but unfortunately destiny is sometimes very cruel and they ended up living separate lives. Years later life brings them together again, but would they be able to get over all the heartbreak from the past?
Gavin comes back after all those years gone and see the girl who broke his heart taking care of his father, while he doesn't agree with it, he lets it go because he see the affection these two have for each other. Raven was such a strong, beautiful soul, the love for her mom is admirable, and the devotion to Gavin's dad made me cry. I'm not going to lie, this book affected me a lot and that's probably I related to Raven so much, because I'm going through something similar to what is described in the book. Penelope did a fantastic job treating a sensitive and sad subject with respect and compassion.
This is an excellent story that will stay in your heart for a long time.
by Darlene
The Day He Came Back by Penelope Ward reminds me of a slower time. A time of old fashion values, ideologies, and prestigious devisions.
As you read this book, you'll feel as though you've stepped back into the 1950's but with a modern twist.
I enjoyed the pace of this book and the way Penelope unfolds her characters and the dramas they have had to faced over the past decade.
Because at the heart of it ... lost love is a tragedy in its own right.
Penelope handles sensitive subjects with care, while still giving her readers a raw honesty.
"The moment we met, you knocked me on my ass. I knew there was something there. You're not just some girl. You're the girl."
Raven and Gavin are worth the read and worth all the feels.
Penelope Ward is a New York Times, USA Today and #1 Wall Street Journal bestselling author of contemporary romance.
She grew up in Boston with five older brothers and spent most of her twenties as a television news anchor. Penelope resides in Rhode Island with her husband, son, and beautiful daughter with autism.
With over two million books sold, she is a 21-time New York Times bestseller and the author of over twenty novels. Her books have been translated into over a dozen languages and can be found in bookstores around the world.
https://www.facebook.com/penelopewardauthor
Facebook Private Fan Group:
https://www.facebook.com/groups/PenelopesPeeps/
@penelopewardauthor
http://instagram.com/PenelopeWardAuthor/
https://twitter.com/PenelopeAuthor
BOOKS BY PENELOPE WARD
When August Ends
Amazon: https://fave.co/2WOsEWr
Amazon Print: https://fave.co/2x37hWQ
Apple Books: https://fave.co/2KpxcAO
Nook: https://fave.co/2x0gqiL
Kobo: https://fave.co/2Rq3PyP
Google Play: https://fave.co/2xbd8tn
Audio: https://fave.co/2WO5DTi
Love Online
Amazon: https://fave.co/2x2Hi1A
Apple Books: https://fave.co/2Kq72Oo
B&N: https://fave.co/2RsbAo4
Kobo: https://fave.co/2Rj0JMU
Google Play: https://fave.co/2KpYVBs
Audio: https://fave.co/2WLLw8t
Gentleman Nine
Amazon: https://fave.co/2KlJREE
Apple Books: https://fave.co/2RlJ5bl
Nook: https://fave.co/2RmoaVI
Kobo: https://fave.co/2WOnAkL
Google Play: https://fave.co/2Rm84LS
Audio: https://fave.co/2x0Nwiz
Drunk Dial
Amazon: https://fave.co/2Rq4vnR
Apple Books: https://fave.co/2Kop8jK
Nook: https://fave.co/2WK3s3a
Kobo: https://fave.co/2RkOgZf
Google Play: https://fave.co/2KoX3Zs
Audio: https://fave.co/2Rvfexz
Mack Daddy
Amazon: https://fave.co/2wZadUi
Apple Books: https://fave.co/2KsSAFc
B&N: https://fave.co/2Rm8LEY
Kobo: https://fave.co/2x3Fc1r
Google Play: https://fave.co/2WO6ovC
Audio: https://fave.co/2WSn2KR
RoomHate
Amazon: https://fave.co/2x3n6wM
Apple Books: https://fave.co/2x3FdCJ
B&N: https://fave.co/2WKZrLP
kobo: https://fave.co/2x0Wtsf
Google Play: https://fave.co/2WO8rQn
Stepbrother Dearest
Amazon: https://fave.co/2KqyP0Q
Apple Books: https://fave.co/2x2Iimm
B&N: https://fave.co/2Rmtbxv
kobo: https://fave.co/2WOuUNp
Google Play: https://fave.co/2WO6Zxm
Neighbor Dearest
Amazon: https://fave.co/2WP3C9p
Apple Books: https://fave.co/2RqgBxs
B&N: https://fave.co/2wZbbQq
Kobo: https://fave.co/2WPNVyG
Google Play: https://fave.co/2Rq5jJl
Jaded and Tyed
(A novelette)
Amazon Print only: https://fave.co/2wZo9h7
Audio: https://fave.co/2WPO0lY
Sins of Sevin
Amazon: https://fave.co/2KqUczk
Apple Books: https://fave.co/2RjW2Tg
B&N: https://fave.co/2Rq5sMX
kobo: https://fave.co/2x2UaVA
Google Play: https://fave.co/2Rn1fJY
Jake Undone
(Jake #1)
Amazon: https://fave.co/2KqA1RU
Apple Books: https://fave.co/2KmcmSV
B&N: https://fave.co/2x2DU6U
kobo: https://fave.co/2RlliIR
Google Play: https://fave.co/2Kq8Rec
Jake Understood
Amazon: https://fave.co/2RopKXd
Apple Books: https://fave.co/2Korlvy
B&N: https://fave.co/2WORPrU
kobo: https://fave.co/2Rn2ngu
Google Play: https://fave.co/2Rn2F74
My Skylar
Amazon: https://fave.co/2Rn2WH8
Apple Books: https://fave.co/2x7t3Zj
B&N: https://fave.co/2KrBaZF
kobo: https://fave.co/2WSoooV
Google Play: https://fave.co/2WR0wSu
Amazon: https://fave.co/2Rqh82q
Apple Books: https://fave.co/2x3ouQ0
B&N: https://fave.co/2KoOBJH
kobo: https://fave.co/2wYndcQ
Google Play: https://fave.co/2KpA8gO
BOOKS BY PENELOPE WARD AND VI KEELAND
Hate Notes
Amazon: https://fave.co/2WO8WtG
Amazon UK: https://fave.co/2RocfGN
Audio ➜ https://fave.co/2RocykV
THE RUSH DUET
Rebel Heir
(Rush, Book 1)
Amazon: https://fave.co/2KrzwHd
Apple Books: https://fave.co/2KpkOR3
B&N: https://fave.co/2WO9sb6
Kobo: https://fave.co/2WR23bc
Google Play: https://fave.co/2x3GCJv
Amazon: https://fave.co/2WPOCbg
Apple Books: https://fave.co/2WSoGfv
B&N: https://fave.co/2WP4Xgr
Kobo: https://fave.co/2WOSedU
Google Play: https://fave.co/2RvgO2t
Cocky Bastard
Amazon: https://fave.co/2RmKjmE
Apple Books: https://fave.co/2WLOkCx
B&N: https://fave.co/2RjX4yC
Kobo: https://fave.co/2WPnMju
Google Play: https://fave.co/2Koctxa
Stuck-Up Suit
Amazon: https://fave.co/2Kpo5jq
Apple Books: https://fave.co/2RorBv9
B&N: https://fave.co/2Kme32J
Kobo: https://fave.co/2x17MRl
Google Play: https://fave.co/2WOwNtt
Playboy Pilot
Amazon: https://fave.co/2KoirhF
Apple Books: https://fave.co/2WL1bol
Nook: https://fave.co/2RonOxL
Kobo: https://fave.co/2Kw0kGH
Google Play: https://fave.co/2Kpp3fy
Mister Moneybags
Amazon: https://fave.co/2RjeCuy
Apple Books: https://fave.co/2x7u0AR
B&N: https://fave.co/2Rn5k0y
Kobo https://fave.co/2x0k9wL
Google Play: https://fave.co/2WOjAkd
British Bedmate
Amazon: https://fave.co/2x4yrwk
Apple Books: https://fave.co/2x0ZSai
B&N: https://fave.co/2RjL4Nu
Kobo: https://fave.co/2x3oTlw
Google Play: https://fave.co/2Rq6XL1
#BlogTour #Review: Already Gone by Kristen Proby and K.L. Grayson
Already Gone by Kristen Proby and K.L. Grayson
Already Gone, an all-new standalone contemporary romance by New York Times bestselling author Kristen Proby and USA Today bestselling author K.L. Grayson
Falling in love is easy…
New Hope, South Carolina is my home. It's where I grew up, got into trouble, and fell in love for the first time. Scarlett Kincaid was more than the girl next door, she was my best friend, until she decided that small town life wasn't for her. One minute she was here, and the next she was gone.
The girl I used to fish with down at the creek is now the biggest name in country music. She headlines world tours, has won four Grammys, and I haven't seen her since. Until today when she sped through town in her fancy car. One look at her big brown eyes was all it took to stir up a whole slew of emotions. Emotions I'd long ago buried and sure as hell don't have time for.
It's the aftermath that's hard…
There are two things in my life that matter. My music, and my dad. Twelve years ago, I packed a bag and chased my dream. Leaving New Hope and escaping the gossip mill was the easiest decision I ever made. I never planned to return, but my father needs me, and he always comes first. So, I did what I had to do. I cut my tour short and came home, despite having a sister who hates me, and a community that doesn't trust me.
And then there's Tucker Andrews.
When he propped an arm on the roof of my car, pulled down his sunglasses and flashed his police badge, I nearly swallowed my tongue. Gone is the lanky boy who used to throw rocks at me and pull my pigtails. Tucker is now a six-foot package of brawny, sexy man wrapped in more muscle than I have hit singles. Did I mention he's a cop and a single dad?
My goal was to help Dad, and get back to my life. But what am I supposed to do when the life that used to strangle me suddenly fits like a glove, and makes me dream of things I never thought I'd have? What happens when the boy I walked away from years ago becomes one of the most important people in my life? I don't have room for a man much less love.
Because it's impossible to hold onto someone who's already gone.
Amazon US: https://amzn.to/2YXGs20
Amazon Worldwide: http://mybook.to/AlreadyGoneKLGKP
Apple Books: http://bit.ly/KLKPAGAB
Nook: http://bit.ly/AGKLGKPK
Kobo: http://bit.ly/AGKLGKPKobo
Add to Goodreads: http://bit.ly/AGKLGKPGR
Already Gone Landing Page: https://www.kristenprobyauthor.com/already-gone
New Hope, South Carolina. Population 6,129. I know every soul represented in that number. Not a single one of them drives the shiny red Mercedes that just went speeding by. Seventy-five in a forty-five. I flick on my lights and press on the gas, sending my cruiser flying past the city limit sign and the godawful billboard that sits directly behind it; the one declaring New Hope home to country music superstar, Scarlett Kincaid. It wouldn't be a big deal if this were actually her home. It's not. Scarlett may have been born here, but her fancy boots haven't landed on this soil in over a decade. All it took was one call from a hotshot music executive to send her packing before the ink was dry on her high school diploma. Scarlett flew from this town fast enough to leave our heads spinning. Before any of us could process what had happened, little Scarlett Kincaid—the same girl who used to build forts with me in my living room while my mama made us mac 'n' cheese—had a hit single sitting at number one on the Billboard charts. She went from homecoming queen and most likely to marry a rich spouse in our senior yearbook, to the queen of country music. The country loves her. Hell, the whole world loves her. New Hope…not so much. And it's high time that fucking sign comes down. But first, I have to deal with this speed demon in the sexy red car. I sound the sirens, and the car pulls to the side of the road and waits while I walk to the driver's side window. It's still up, the heavy tint preventing me from seeing inside. With a hand on my holster—because you never know what you're going to walk up on—I knock on the window. The dark glass lowers. My first thought: this woman is absolutely gorgeous. Long, dark hair. Pouty lips. And a tiny pink dress. Her eyes are covered by oversized aviators, but I'm sure they're as pretty as the rest of her. My second thought: what crazy excuse is she going to come up with to try and get out of this ticket? It never ceases to amaze me the things women are willing to do to keep from getting into trouble. I've been offered everything from a blowjob to a pay-off to marriage. "Do you know why I pulled you over today, ma'am?" "Tucker?" The woman smiles, then pushes her sunglasses to the top of her head. And that's when I see the brown eyes I've spent more than a decade trying to forget. "Tucker Andrews, is that you?" I step back and square my shoulders. "You can call me Officer Andrews. Do you know why I pulled you over today, ma'am?" "Tucker." The woman laughs and shakes her head. "It's me, Scarlett." At the mention of her name, I'm met with an onslaught of flashbacks. Running hand and hand through the neighborhood with her, laughing and playing, only to have her ignore me the second we got to school. The popular crowd versus the nerds who desperately tried to fit in. She the former, me the latter, and the pain it caused every time she acted as though she didn't know my name. For years, I pretended it didn't bother me because I knew that when I got home, Scarlett would meet me at the fence, and the awkwardness from the day would dissipate as though it never happened. But it did. Day after day after day. I was a glutton for punishment. There was nothing in the world I loved more than Scarlett Kincaid, and it didn't matter how badly she hurt me, I was always willing to forgive her. Her smile and laugh might've gotten to me in middle school and high school, but I refuse to let it affect me now. "I know who you are." Her smile falters.
First of all, let me say that you wouldn't know that this book was written by two authors. The flow of the story and the beauty of the characters is a testament of the good paring and talent of these two ladies.
Scarlett and Tucker were good friends growing up, the thing is, they wanted different things in life. He wanted to stay in his small town and create a life there and she wanted bigger and better things, like being a big time singer. They both got what they wanted, but from those decisions came resentment and a sense of abandonment. When an unexpected situation brings Scarlett back, can they forgive and forget and give a relationship a go?
I found this story to be one with a lot of heart. Not only are Scarlett and Tucker great characters, but Chloe and Scarlett's dad were fantastic supporting characters who I adored. The chemistry between Scarlett and Tucker is off the charts hot and you can feel it every time they're together.
I loved this book because in a way I believe it show us that sometimes what we think is the most important thing in life is not, and also because sometimes you have to compromise in order to get that person you really love. Great read!!!
About Kristen
Kristen was born and raised in a small resort town in her beloved Montana. In her mid-twenties, she decided to stretch her wings and move to the Pacific Northwest, where she made her home for more than a dozen years.
During that time, Kristen wrote many romance novels and joined organizations such as RWA and other small writing groups. She spent countless hours in workshops, and more mornings than she can count up before the dawn so she could write before going to work. She submitted many manuscripts to agents and editors alike, but was always told no. In the summer of 2012, the self-publishing scene was new and thriving, and Kristen had one goal: to publish just one book. It was something she longed to cross off of her bucket list.
Not only did she publish one book, she's since published close to thirty titles, many of which have hit the USA Today, New York Times and Wall Street Journal Bestsellers lists. She continues to self publish, best known for her With Me In Seattle and Boudreaux series, and is also proud to work with William Morrow, a division of HarperCollins, with the Fusion Series.
Kristen and her husband, John, make their home in her hometown of Whitefish, Montana with their two pugs and two cats.
Facebook: https://www.facebook.com/BooksByKristenProby/
Twitter: https://twitter.com/Handbagjunkie
Instagram: https://www.instagram.com/kristenproby/
BookBub: https://www.bookbub.com/authors/kristen-proby
Goodreads: http://bit.ly/2kBRdpj
Amazon: http://amzn.to/2BD4vfq
Pinterest: @handbagjunkie Website: https://www.kristenprobyauthor.com/
Newsletter: https://mailchi.mp/kristenproby/newsletter-sign-up
About K.L. Grayson
K.L. Grayson resides in a small town outside of St. Louis, MO. She is entertained
daily by her extraordinary husband, who will forever inspire every good quality she
writes in a man. Her entire life rests in the palms of six dirty little hands, and when the
day is over and those pint-sized cherubs have been washed and tucked into bed, you can find her typing away furiously on her computer. She has a love for alpha-males,
brownies, reading, tattoos, sunglasses, and happy endings…and not particularly in that
Connect with K.L. Grayson
Facebook: http://bit.ly/KLGraysonFB
Instagram: http://bit.ly/KLGraysonIG
Pinterest: http://bit.ly/KLGraysonPin
Bookbub: http://bit.ly/KLGraysonBB
Amazon: https://amzn.to/2MVYFfe
Twitter: http://bit.ly/KLGraysonTW
Goodreads: http://bit.ly/KLGraysonGR
Website: www.klgrayson.com
#ReleaseBlitz: On The Corner of Love and Hate by Nina Bocci
On The Corner of Love and Hate
by Nina Bocci
Release Day: August 20, 2019
For fans of Lauren Layne and Kristan Higgins comes a delightfully "fun bit of fluffy entertainment" (Publishers Weekly) in which a young woman is forced to help her old friend revamp his image for the upcoming mayoral elections...and discovers that she might not be as immune to his charms as she once thought.
When Emmanuelle Peroni's father—and current mayor of Hope Lake, Pennsylvania—suggests she help with Cooper Endicott's campaign, she's horrified. Cooper, one of her (former) oldest friends, drives her crazy in every way possible. But he's also her father's protégé, so Emma reluctantly launches her plan to help him win the local election.
It's not as easy as it looks. Cooper's colorful love life is the sticking point for many voters, and his opponent is digging up everything he can from his past. It seems that every time Emma puts out the flames from one scandal, another one flares up. Emma knows that if Cooper wants to win, he needs to keep his nose clean. The only problem? She might just be falling in love with the one person she promised never to pursue: the mayoral candidate himself.
#Buy Links
B&N http://bit.ly/OtCoLaHBN
iTunes http://bit.ly/OtCoLaHZon
BaM http://bit.ly/2wyetKo
Amazon http://bit.ly/OtCoLaHZon
Add to Goodreads: https://www.goodreads.com/book/show/42587433-on-the-corner-of-love-and-hate
Thud. Whoosh. Slap. Thud. Whoosh. Slap. The trio of irksome sounds repeated another half-dozen times. My eyes darted upward, a silent prayer falling from my lips. Dear God, please give me the strength not to shove that tennis ball somewhere that would require surgery. Amen. My coworker casually leaned back in his chair, his long legs outstretched and crossed at the ankles on the shiny surface of the conference room table. Beneath his brown leather loafers sat a report. His unfinished-yet-due-tomorrow report. I marveled at his ability to multitask. It would have been more appropriate if he had been, say, working. Instead, he was tossing a ball against the conference room wall with one hand while texting with the other. Even though he didn't take his eyes off his phone screen, he caught the ball every single time. If I hadn't been so annoyed, I would have actually been impressed. The clock ticked against the pale yellow wall above his head. With each passing tick, the ball struck with a thwack to its right. "Cooper, could you please stop?" I finally said, rubbing my temples to ease the headache that was forming. Thud. Whoosh. Slap.
"Cooper," I repeated, glancing up from my laptop. "Hello?" Thud, whoosh, slap was the only response I got. Sliding back my chair, I stood up and walked around the long maple conference table. It was only when I got close enough to see the scantily clad woman in his text window that I noticed the wireless earbuds that were blasting music into his ears. As the ball left his hand, I touched his shoulder. Startled, he lost his grip on the ball, sending it sailing behind him. "What's up?" he sputtered, quickly pulling his earbuds out. I didn't miss his hand sliding his phone into his pocket. He looked every bit like a teenager caught red-handed by the principal. "Are you kidding me?" I exclaimed. "You've had music on this entire time? I read nearly two pages of the brewery expansion proposal out loud to you twenty minutes ago!" At least he had the decency to look remorseful. "I thought you were talking to yourself, so I"—he motioned to the black Beats— "figured I'd give you your privacy while I caught up on work." My eyebrows must have reached my hairline, because with a mildly guilty expression he pulled his legs down from the table. I snorted. "Yes, I start all sentences with, 'Cooper, what do you think about' when I'm talking to myself. Were you just smiling and nodding for my health?" Shifting in his seat, he straightened. I huffed. The small laugh lines around his mouth became more pronounced, an indication that he was fighting back a smile. "Emmanuelle," he purred smoothly. "Don't Emmanuelle me," I clapped back. "That tone may work on your fan club, but not me." He held his arms up in a defensive position. "Okay, okay, I'm sorry. What did I miss?" He grabbed for the papers in my hand. Holding them back against my chest, I scowled. "Hope Lake Brewing Company. Expansion. Asking for input before it goes to the town council for approval." He whistled and rocked back in his chair. "Council is going to reject anything that comes across their desk from them. They hate the 'vibe' the brew house brings, and the addition would make the council's heads explode." I nodded. "Yep, which is why the guys asked us for help. To try and edit the proposal to appeal to them. It's also why I booked us the conference room for this meeting that you just Tindered your way through." "That's not a word, and I wasn't—" he began, patting his pocket absently. Probably making sure the evidence was tucked away safely. I held up my hand. "Save it. I don't care what or who you're doing. Just that you're not paying attention. Again." When the owners of HLBC, Drew and Luke Griffin, first came to our department, Cooper and I had championed their proposal to build a brewing company, tasting room, and outdoor entertainment space just along the lakefront. It was one of the first projects Cooper and I had worked on together, and it was just what we'd needed in town back then—a fun, innovative business that catered to every age. Now, six years later, HLBC was one of Hope Lake's most popular spots, and the brothers were looking to expand their space to include rooms for private events and a small restaurant. Cooper and I were supposed to be discussing how to approach the town council about it. Looked like I'd just been talking to myself instead. "I'm going back to my office, where I can work in peace," I said. Exasperated, I started gathering up my stuff. After a few seconds of awkward silence, he cleared his throat. "You're right. I'm sorry. Let's go over it. Again." I stacked my files, feeling my blood starting to boil. Having to repeat myself irked me, but I needed his input whether I liked it or not. Glancing up, I noticed Cooper readying to say something else when our shared assistant, Nancy, hurried in with the main office calendar and a fistful of Sharpies clutched in her hand.
"I've been searching for you two everywhere," she said, looking wide-eyed at each of us in turn. The conference table, at least on my side, was covered in charts, graphs, and photos of the lakefront. On Cooper's side—well, there was a lot of polished maple visible. "Did you discuss the project?" she asked hopefully, her face falling when I shook my head. "Okay, well, I guess you'll handle that, uh, later. I'm sure." She gave me a look. "I hope," she mouthed, then cleared her throat and pulled out the head chair of the conference table and sat down with the main office calendar in front of her. "It's time for the afternoon rundown—are you ready?" Cooper groaned. Not at Nancy but at the calendar she had opened. It had been on my desk this morning when I'd filled it with upcoming appointments and meetings. By the looks of it, Nancy had managed to fill almost every empty space that remained. We kept it old school at our office. Instead of using Google calendar or iCal, we used a large paper desk calendar with a color-coded legend, labels, and tabs to keep our government office running like clockwork. It's not as though we hadn't tried to modernize, but some of the, ahem, older department staff were frosty toward change. Nancy, Cooper, and I worked at the Hope Lake Community Development Office on the top floor of Borough Building. In a small town like Hope Lake, my department was sort of the home base for everything. From simple things such as parade permits to more detailed ventures—for example, helping to secure funding for business owners like HLBC—the CDO, as we tended to call it, had its hand in pretty much everything. It wasn't big, but what we lacked in size and staff we made up for in energy and results. "The upcoming week is brutal," Nancy apologized, looking at Cooper, who, not surprisingly, was on his phone again. "Emma, I'm afraid you're a bit overscheduled." She tapped a Sharpie on the table. I waved a dismissive hand. "It can't be any worse than that week the staff came down with the flu." I had practically run the office that week even though I was heavily medicated myself. "It's close." She held up two fingers barely an inch apart. "You're back-to-back Monday. There is a pocket of time during the event this weekend with the future Mr. Mayor here and his opponent." Cooper perked up then. He knocked twice on the wooden table. "Don't jinx me." Oh, sure, you're paying attention now. "You're a shoo-in. People love you, Cooper. And with the mayor already behind you, how can you not be?" Nancy assured him. Nancy wasn't blowing smoke. Cooper had decided to run for office this year, and his magnetic personality made him the perfect political candidate. He was brilliant, liked by the majority of the town, and had confidence to spare because he knew he was the best choice for the job. Even I could admit that, and we were often at odds. "Emma, I know you wanted to have a sit-down with Drew and Luke from the brewing company about the proposed expansion before they go to the council, but I don't see how it's going to happen." Nancy jotted a note onto the calendar. Over the years, we'd gotten our system down to a science: orange for me, blue for Cooper, hot pink for our department administrative assistant, green for Nancy, and red for the mayor, because red was my dad's favorite color. Blue, not surprisingly, was the color least visible on the entire calendar. It was sporadically used, even from my vantage point, which meant that Cooper had a light schedule this week. Shocking. I chewed the pen cap, irritated. Nancy continued reading off meeting after meeting throughout the week. "These two on Thursday—I can probably sit in on them to give you a break, Emma," she offered.
Looking over Nancy's shoulder, I marveled at the Technicolor scheduling system. It might have been old-fashioned, but at least it looked good. Shaking my head, I pointed at the partially torn yellow Post-it stuck on the edge of the frame. That was how my father added mayoral meetings to the calendar. Stickies. He was nothing if not professional. "No can do, my friend. You're going to be at a ribbon cutting with Mayor Dad." She looked up, her lips a thin, flat line. "I am? He didn't tell me." Sighing, she jotted the information down. "I wish he'd told me I was supposed to go, too!" She took her calendar duties very seriously. I for one appreciated it, and I knew my father did, too, even if he did use his own odd system to add to it. It kept all of us in line. Together, Nancy and I figured out the rest of the week, Cooper staying silent and, surprise surprise, on his phone. We looked over the days, pointing and crossing out, trying in vain to find somewhere to squeeze in a last sit-down. "It's not going to work," I lamented, sinking into the chair beside her. "Well, someone from the department needs to at least show their face at the city events meeting," she urged, looking pointedly at Cooper. A notebook was now on his lap, his hand moving swiftly over the page. He didn't look up when she said his name or when she repeated it a few seconds later. He was too deeply invested in whatever he was doing. At least he's off the phone. Tearing the Post-it off the calendar and balling it up in her fist, Nancy lobbed it at him. "Cooper!" she shouted, snapping her fingers as if she were telling a dog to sit. Fitting. He smiled at her. "I'm listening." "Uh-huh, we need you to take a meeting or two on Thursday so Emma can head down to the lake to meet Drew and Luke. Unless you'd rather take the HLBC meeting."
"Thursday?" he repeated, sliding his phone out from behind the notebook. When did he take that out? He was stealthy like a teen texting in class. With a shrug, he shook his head. "Sorry, I'm booked all day and I've got a campaign publicity debrief at noon. That's taking up most of the afternoon." "Doesn't that just mean you and Henry are meeting at the diner to play on Facebook and Twitter together?" I scoffed, feeling the blood rushing to my face. Henry was one of my and Cooper's oldest friends. As a teacher, he had limited time to meet up with Cooper, so I understood Cooper's reticence to reschedule, but— Then it hit me. "Wait . . . why are you having mayoral meetings during work and school? How's Henry getting out of class to meet you?" Setting his phone down, he stood and straightened his tie. "I'll have you know, I'm meeting him at the high school. I wish I could help, but alas—" "You can't," I finished, sliding out of my chair to stand myself. With Cooper running for mayor of Hope Lake, the brunt of his work at the CDO was taking a backseat. I noticed, the staff noticed, and the mayor noticed. If it had been anyone else, they probably would have been fired, but Cooper was Hope Lake's golden boy. Once he was elected, we could hire someone new to replace him. But until that happened, it fell to us to pick up his slack. Cooper walked toward the door, leaving his phone—aka his most prized possession—on the conference table. Surely he would be back in for it the second he realized it wasn't attached to his hand. "Wait, you can't leave!" Nancy called after him. "I need the theater proposal paperwork. You guys have that meeting with the council on Monday and the mayor wants the weekend to review the specs.
Cooper, it has to be before end of day since you have the debate tomorrow! Everything is done, right? Please tell me it's done." "It's handled," Cooper said smoothly over his shoulder, tapping his temple. "And it's not a debate. It's a photo op, remember? Pose, smile, shake hands. You know, the usual." "Thank God. I don't have time today to do it if you didn't," she said, pretend wiping her brow. Smiling broadly, he clapped his hands together. "Oh, come on, Nance. Have I ever left you hanging?" Her silence spoke volumes. If she'd had the time, and the inclination, she could have created a depressing list of how often that had happened. Looking uncomfortable at Nancy's lack of response, Cooper disappeared through the door, only to reappear two seconds later. "That would have been bad!" he said with a tight smile, jogging in to grab the iPhone. "Cooper, are you sure you can't reschedule your Thursday plans with Henry until after work so Emma isn't pulled in nineteen directions?" Nancy said quickly. "It's just about the local sports participation in the Thanksgiving parade. They're looking for guidance with the floats and theming—it won't exactly take up all your brain space. The other is an initial meeting to see if the CDO can finally purchase the old bank." Nancy already had a blue Sharpie at the ready, clutched between her fingers. "Or if you wanted to switch with Emma, you could meet with Drew and Luke and Emma could handle the parade instead. You'd probably get some free beer out of it." For a moment, he looked like he was going to agree. His jawline ticked anxiously, a habit he'd had since we were kids. It appeared whenever he struggled with a decision. Reluctantly, I admitted to myself that it was happening more often than not. "I'm really sorry, I can't," he finally said. "You know how important these meetings are for the core of my campaign. I've got to run. I'm late."
I glanced at the clock. "It's barely four." "I have a thing." "You came in at ten because of a 'thing.' " I air-quoted it because although he said those things were for the mayoral campaign, I didn't believe him. Call it years of experience or just a gut feeling. "Cooper, I need you to focus. You're all over the place, and things are going to start falling through the cracks here. We can't afford any missteps. Not when we're under a microscope. The council is looking for any reason to put the screws in this department." Cooper's opponent, Kirby Rogers, had been on the town council for the past few years. He had made it his mission to strip the CDO—funding, staff, all of it gone. With nothing but a grimace, Cooper left, leaving no opening for discussion. I shook my head at his retreating form. "Forget him, I'll figure it out," I said, glancing between the calendar with the work appointments and my nearly empty personal calendar. "I can pop over to the brewery and see Drew and Luke on my way home Tuesday or Friday night. They owe me dinner, anyway," I said with a weak laugh, an attempt at loosening the anxiety-ridden ball in my stomach. How am I going to accomplish all of this? "Just see when they're free." I tapped away on my phone. Making a note, I double-checked my iPhone's calendar as Nancy read off the rest of the upcoming schedule. "Emma," she said with a heavy sigh, "I don't want you to overwork yourself." "I'm fine. It's an adjustment we're going to have to get used to since we're going to be picking up all the Cooper slack," I insisted, knowing that she was always worried about me in a big-sisterly sort of way. "Promise," I said after seeing her frown. Months ago, before he had decided to run for mayor and before he had become so distracted by the election, Cooper had been an asset. I longed for those days. He had a gift, an ability to coax the very best of ideas out of you, and he transformed them into solid plans that we then presented to Mayor Dad and the town council. His undivided input would have been valuable here. That part of Cooper I respected and enjoyed working with. Pre-candidate Cooper. Except lately, so much had changed. I missed the focused Cooper. The guy who would pull together a presentation in just a few hours. The guy I could count on to bring the best ideas out of me when I thought I had hit a wall. Or even the guy who got his work done on time. I hated myself a little bit because I was missing that coworking partnership. We did make a good team when we weren't arguing. "Not for anything, but you'd think he'd want to head over to Hope Lake Brewing Company to see the guys." "His head was so buried in his phone, he probably didn't hear you mention them." Nancy nodded. "What do you think? Is this going to get better or worse as the campaign progresses?" She packed up her Sharpies and hoisted the large calendar off the table, mindful not to drop any of the Post-its and papers tacked to it. I slung my arm over her shoulder. "Worse. So much worse."
Nina Bocci is a USA Today Bestselling novelist, publicist, eternal optimist, unabashed lipgloss enthusiast, constant apologist and a hopeless romanticist. She has too many college degrees that she's not using and a Lego addiction that she blames on her son.
Follow Nina
Facebook: https://www.facebook.com/ninabocci/
Amazon: https://www.amazon.com/Nina-Bocci/
Goodreads: https://www.goodreads.com/author/show/14164319.Nina_Bocci
Twitter: @ninabocci
Website: http://www.ninabocci.com
#BookSpotlight #Reviews: The Day He Came Back by P...
#BlogTour #Review: Already Gone by Kristen Proby a...
#ReleaseBlitz: On The Corner of Love and Hate by N...
#ExcerptReveal: The Day He Came Back by Penelope W...

redpajama_set_name: RedPajamaCommonCrawl


text:
Art, Design, Miscellaneous, People
Thinking Big
May 21, 2013 Pat Webster	2 Comments
Recently I saw a photo of a giant yellow ducky floating in Hong Kong harbour. Called Spreading Joy Around the World, it's by the Dutch artist Florentijn Hofman. And it is BIG: 54 ft, or 16.5 metres, tall.
The artist said it was intended to make people feel happy. It worked. It made me smile.
It also set me thinking about the impact of size in a landscape.
At Glen Villa, the Big Chair always brings a smile. From a distance, it's hard to appreciate the scale. But once someone sits in the chair, its giant size becomes apparent.
The Big Chair at Glen Villa
The chair was created by NIP Paysage, a firm of young and inventive landscape architects based in Montreal. I first saw the chair at Canada Blooms, in an installation designed by the Reford Gardens in Metis, Quebec. As soon as I spotted it, I knew it was the chair I'd been looking for. Based on the traditional Adirondack chair, the Big Chair is a surefire success. It's a rare visitor to Clen Villa who doesn't climb up to have their photo taken. And everyone who does feels like a child again, and smiles.
Oversized objects carry a strong message. They disturb our sense of balance, confuse our impressions of distance and perspective. Small objects can do the same. A smaller-than-normal object set in the distance lengthens a view. Close at hand, something smaller than expected can make us laugh.
This tiger was tucked into a planter on an outdoor mantlepiece at Chanticleer, a very fine public garden near Philadelphia. I love the touch of colour it adds, and how it transforms a group of tiny shade-loving plants into a pint-sized jungle.
A tiger in the jungle, at Chanticleer
Playing with scale and perspective in a garden isn't a new trick. André Le Noôtre worked his magic in the 1600's at Vaux-le-Vicomte, south of Paris. At the end of a kilometre-long expanse, he placed a statue of Hercules silhouetted on the horizon. The statue was so large that it made that one kilometre appear much shorter. To add to the trickery, he contoured the land, entirely concealing a canal and the grand staircase and statuary that led to it.
Despite looking fairly close, Hercules is still about half a kilometre away.
Standing on top of that stone wall and looking back towards the chateau, you get an idea of the distance. And you see what previously was hidden from view: an impressive canal.
Looking back towards the chateau at Vaux-le-Vicomte
The British landscape designer Russell Page liked to play with perspective, narrowing a path as it receded to increase the sense of distance. Other designers have planted trees closer together the farther away they are, or even have planted shorter trees, all to accomplish the same visual trick, to make the distance seem greater than it is.
This photo came with a (probably apocryphal) story about a woman who couldn't get permission to build a shelter for her horses. Could she put up a table and chairs? No problem, came the response.
Ah, the joys of photoshop!
Confounding expectations adds fun to a garden. The rubber ducky doesn't even have to be big.
A light-hearted touch in a Montreal garden:
a ducky swimming in a birdbath
blogCanada BloomsChanticleerFlorentijn HofmanGlen VillaNip PaysageperspectiveReford Gardensrubber duckRussell PagescaleThe Big ChairVaux-le-Vicomte
Previous PostAn Artist's Garden in SpringNext PostSnow: can this be May?


This new research identifies web design firms with outstanding records of client satisfaction.
WASHINGTON, April 21st, 2016 — Clutch has published its first pages of research on web design companies located in Denver, Miami, and Phoenix. The research leverages Clutch's proprietary methodology, mapping each firm's focus on web design services against their ability to deliver world-class results for their clients.
The featured Denver firms are: Unleaded Group, 601am, Axial, Pyxl, Elevated Third, Fruition, Arrae Creative, Cymax Media, Xcite Media Group, 10 Pound Gorilla, Planet Media, Slice of Lime.
The featured Miami firms are: Crea7ive Interactive Advertising, Rand Internet Marketing, Disartmedia Digital Agency, Paperstreet Web Design, Pacific 54, Sparxoo, South Florida Web Studio, Ambition Insight, CBIL360, Lucid Site Designs, Zakkour Technology Group.
The featured Phoenix firms are: eCreations, Tempo Creative, Pyxl, Nuanced Media, Epic Web Solutions, Tension Design, Avenue 25, Dreamztech USA, Miss Details Design, Coplex.
The research is based on over a dozen quantitative and qualitative factors, including company experience, client list, industry recognition, client reviews, and market presence. Profiles of each company detailing the companies' expertise and displaying full-length client reference interviews are published on Clutch.co.
Clutch's effort to identify leading web design agencies is ongoing. Clutch encourages established companies to apply to be included in future research updates.
Clutch is a Washington, DC-based B2B research firm that identifies top service providers and solutions firms that deliver results for their clients. The Clutch methodology is an innovative research process melding the best of traditional B2B research and newer consumer review services. To date, Clutch has researched and reviewed 1000+ companies spanning 50+ market segments. Contact: Ayrald Hubert / 202-683-6095 / ayrald@clutch.co.


How do you obtain the true count?
Say in the first hand of a 4 deck game the dealer comes out with eight cards and the running count is +4. The true count will be +1, because it's +4 divided 4. But in the second hand do you start the count with the running count of +4 or with the true count of +1? I thought you would continue with with running count until I read a book which calculated the true count by dividing the #half decks remaining.
Running count is +5, ten cards have been played out, with 8 half-decks in a 4deck game we divide +5 by 7 which equals a total of less than 1. You keep playing as if the deck were neutral, since the true count is less than +1, even though the running count is positive.
So do you continue on with the running count after the first round of play if you divide by full decks, and with the true count if you divide by half decks?
to find the true count, take the rc and divide that by the number of decks remaining to be played. the "true edge" method of obtaing the count would be to divide the rc by the number of 1/2 decks remaining. this, if done correctly,will give you an approximation of the edge you currently have, or dont have over the house. for every number the true count goes up, your edge goes up by about .5%. and of course when the tc goes down, so does your edge. but if you use the true edge method you must take into account the % that the house already has before a single card is dealt.
Running count is +5, ten cards have been played out, with 8 half-decks in a 4deck game we divide +5 by 7 which equals a total of less than 1 (this is refering to the true edge, the tc would be a little above 1). You keep playing as if the deck were neutral, since the true count (true edge) is less than +1, even though the running count is positive."
I believe Snyder coined the true edge method, and I dont remember the above text in BB in BJ or in his online forum article. and why would you divide by 7? only 10 cards have been played, a half deck of cards is 26, its more like 7.5. your true edge in this situation is just above 0, so i guess it is neutral. (true edge in this situation is 5 divided by 7.5 minus the house edge) somewhere around a +.15%, nothing to get excited about.


WinRAR puts you ahead of the crowd when it comes to compression. By consistently creating smaller archives, WinRAR is often faster than the competition. This will save you disc space, transmission costs AND valuable working time as well. WinRAR is ideal for multimedia files. WinRAR automatically recognizes and selects the best compression method. The special compression algorithm compresses multimedia files, executables and object libraries.
If Windows User Account Control prevents extracting or archiving commands to create files in system protected folders, WinRAR attempts to start its another copy with administrator privileges to complete the operation. It is necessary to confirm privileges elevation in response to User Account Control prompt to allow such second WinRAR copy to run.
You can drag files in Explorer with right mouse button, drop them to some folder and select WinRAR archiving commands in context menu to create an archive with these files in the destination folder.
It is possible to use WinRAR.ini file instead of Registry to store WinRAR settings. You may prefer such approach if you plan to install WinRAR to removable media, such as USB flash drive, and then run it on different computers.
See "WinRAR.ini file" topic in "Configuration settings" chapter of WinRAR help for details.
Added extraction support for ZIP and ZIPX archives using XZ compression algorithm.
If ZIP or ZIPX archive contains any compression algorithms except usual "Store" or "Deflate", algorithm names are displayed in "Version to extract" field of archive information dialog. Algorithms with names unknown to WinRAR will be assigned "m" name, where "num" is a compression algorithm numeric identifier.
"Open with WinRAR" command is available in Windows context menus for archive formats supported by WinRAR. It can be turned off with "Open with WinRAR (for usual archives)" option in WinRAR "Settings/Integration/Context menu items..." dialog.
Command "ch" supports switches -tl, -cu, -cl also for ZIP archives. Previously "ch" allowed these switches only for RAR archive format.
For archive formats lacking file time information, such as .bz2, .xz and .Z, WinRAR sets container archive modification time to extracted files. It does not apply to tar.bz2, tar.xz and tar.Z, which use file time stored in tar headers.
WinRAR icon in Windows context menus is scaled up correctly in high DPI screen modes such as 150% or 200% of normal text size.
It is possible to disable WinRAR "Benchmark" command with "Benchmark" variable of HKEY_CURRENT_USER\Software\WinRAR\Policy Registry key. In multi-user environment "Benchmark" command might abuse shared computational resources. See "Registry variables" topic in "Configuration settings" chapter of WinRAR help for more details.
Additionally to "sfxcmd" variable containing the entire command line, SFX module sets "sfxpar" variable containing command line parameters only, without leading SFX module name. These variables are set before starting a program specified in "Setup" command.
File overwrite prompt in console RAR displays the size and modification time for existing and new files.
When archiving from stdin with -si switch, RAR sets the current system time to modification time of archived file. Previous version did not set this time at all.
It is possible to use -si and -v switches together. Previous versions did not allow to create volumes when archiving from stdin.
Warning is issued when starting unpacking 4 GB or larger file from RAR or ZIP archive to FAT32 patition, so user may cancel the operation. FAT32 does not support files of such size.
Also this warning is issued when starting to create RAR archive with "Store" (-m0) method if estimated archive size is 4 GB or more.
"Select all" button in "Settings/Integration" dialog is renamed to "Toggle all". Now it deselects archive formats if all they are already selected.
"Delete archive" extraction option correctly removes all .zip and .7z volumes. Previously it removed only the first volume in the set for these archive formats.
2) After Installation, DO NOT Run it!


text:
\section{Introduction}
In the setting of {\it multi-agent online learning} (\cite{shalev-shwartz_online_2011,cesa-bianchi_prediction_2006}), $K$ players interact with each other over time. At each time step $t$, each player $k \in \{1, \ldots, K\}$ chooses an {\it action} $\bz\^t\!k$; $\bz\^t\!k$ may represent, for instance, the bidding strategy of an advertiser at time $t$. Player $k$ then suffers a {\it loss} $\ell_t(\bz\^t\!k)$ that depends on both player $k$'s action $\bz\^t\!k$ and the actions of all other players at time $t$ (which are absorbed into the loss function $\ell_t(\cdot)$). Finally, player $k$ receives some {\it feedback} informing them of how to improve their actions in future iterations. In this paper we study gradient-based feedback, meaning that the feedback is the vector $\bg\^t\!k = \grad_{\bz\!k} \ell_t(\bz\^t\!k)$. %

A fundamental quantity used to measure the performance of an online learning algorithm is the {\it regret} of player $k$, which is the difference between the total loss of player $k$ over $T$ time steps and the loss of the best possible action in hindsight: formally, the regret at time $T$ is $\sum_{t=1}^T \ell_t(\bz\^t\!k) - \min_{\bz\!k} \sum_{t=1}^T \ell_t(\bz\!k)$. An algorithm is said to be {\it no-regret} if its regret at time $T$ grows sub-linearly with $T$ for an adversarial choice of the loss functions $\ell_t$. If all agents playing a game follow no-regret learning algorithms to choose their actions, then it is well-known that the empirical frequency of their actions converges to a {\it coarse correlated equilibrium (CCE)} (\cite{moulin_strategically_1978,cesa-bianchi_prediction_2006}). In turn, a substantial body of work (e.g., \cite{cesa-bianchi_prediction_2006,daskalakis2009network,even2009convergence,cai2011minmax,viossat_no-regret_2013,krichene_convergence_2015,bloembergen_evolutionary_2015,monnot_limits_2017,mertikopoulos_learning_2018,krichene_learning_2018}) has focused on establishing for which classes of games or learning algorithms this convergence to a CCE can be strengthened, such as to convergence to a {\it Nash equilibrium (NE)}. %

However, the type of convergence guaranteed in these works generally either applies only to the time-average of the joint action profiles, or else requires the sequence of learning rates to converge to 0. Such guarantees leave substantial room for improvement: a statement about the average of the joint action profiles fails to capture the game dynamics over time (\cite{mertikopoulos_cycles_2017}), and both types of guarantees use newly acquired information with decreasing weight, which, as remarked by \cite{lin_finite-time_2020}, is very unnatural from an economic perspective.\footnote{In fact, even in the adversarial setting, standard no-regret algorithms such as FTRL  (\cite{shalev-shwartz_online_2011}) need to be applied with decreasing step-size in order to achieve sublinear regret.} Therefore, the following question is of particular interest (\cite{mertikopoulos_learning_2018,lin_finite-time_2020,mertikopoulos_cycles_2017,daskalakis_training_2017}):
\begin{equation}
  \tag{$\star$}\label{eq:main-question}
  \parbox{10cm}{\centering\text{\it Can we establish last-iterate rates if all players act according to }\\ \text{\it a no-regret learning algorithm with constant step size?}}
\end{equation}
We measure the proximity of an action profile $\bz = (\bz\!1, \ldots, \bz\!K)$ to equilibrium in terms of the {\it total gap function} at $\bz$ (Definition \ref{def:total-gap}): it is defined to be the sum over all players $k$ of the maximum decrease in cost player $k$ could achieve by deviating from its action $\bz\!k$. \cite{lin_finite-time_2020} took initial steps toward addressing (\ref{eq:main-question}), showing that if all agents follow the {\it online gradient descent} algorithm, then for all {\it $\lambda$-cocoercive games}, the action profiles $\bz\^t = (\bz\^t\!1, \ldots, \bz\^t\!K)$ will converge to equilibrium in terms of the total gap function at a rate of $O(1/\sqrt{T})$. Moreover, linear last-iterate rates have been long known for smooth {\it strongly-monotone} games (\cite{tseng_linear_1995,gidel_variational_2018,liang_interaction_2018,mokhtari_unified_2019,azizian_tight_2019,zhou_robust_2020}), a sub-class of $\lambda$-cocoercive games. Unfortunately, even $\lambda$-cocoercive games exclude many important classes of games, such as bilinear games, which are the adaptation of matrix games to the unconstrained setting. Moreover, this shortcoming is not merely an artifact of the analysis of \cite{lin_finite-time_2020}: it has been observed (e.g.~\cite{daskalakis_training_2017, gidel_variational_2018}) that in bilinear games, the players' actions in online gradient descent not only fail to converge, but diverge to infinity. Prior work on last-iterate convergence rates for these various subclasses of monotone games is summarized in Table \ref{tab:last-iterate} for the case of perfect gradient feedback; the setting for noisy feedback is summarized in Table \ref{tab:last-iterate-stoch} in Appendix  \ref{sec:stoch-background}.

\begin{table}
  \centering
  \caption{
\neurips{Known last-iterate convergence rates for learning in smooth monotone games with perfect gradient feedback (i.e., {\it deterministic} algorithms). We specialize to the 2-player 0-sum case in presenting prior work, since some papers in the literature only consider this setting. Recall that a game $\MG$ has a {\it $\gamma$-singular value lower bound} if for all $\bz$, all singular values of $\partial F_\MG(\bz)$ are $\geq \gamma$. %
    $\ell, \Lambda$ are the Lipschitz constants of $F_\MG, \partial F_\MG$, respectively, and $c, C > 0$ are absolute constants where $c$ is sufficiently small and $C$ is sufficiently large. %
    Upper bounds in the left-hand column are for the \ExG algorithm, and lower bounds are for a general form of 1-SCLI methods which include \ExG. Upper bounds in the right-hand column are for algorithms which are implementable as online no-regret learning algorithms (e.g., \OG or online gradient descent), and lower bounds are shown for two classes of algorithms containing \OG and online gradient descent, namely $p$-SCLI algorithms for general $p \geq 1$ (recall for \OG, $p = 2$) as well as those satisfying a 2-step linear span assumption (see \cite{ibrahim_linear_2019}). The reported upper and lower bounds are stated for the total gap function (Definition \ref{def:total-gap}); leading constants and factors depending on distance between initialization and optimum are omitted.}
  \arxiv{{\small Known last-iterate convergence rates for learning in smooth monotone games with perfect gradient feedback (i.e., {\it deterministic} algorithms). We specialize to the 2-player 0-sum case in presenting prior work, since some papers in the literature only consider this setting. Recall that a game $\MG$ has a {\it $\gamma$-singular value lower bound} if for all $\bz$, all singular values of $\partial F_\MG(\bz)$ are $\geq \gamma$. %
    $\ell, \Lambda$ are the Lipschitz constants of $F_\MG, \partial F_\MG$, respectively, and $c, C > 0$ are absolute constants where $c$ is sufficiently small and $C$ is sufficiently large. %
    Upper bounds in the left-hand column are for the \ExG algorithm, and lower bounds are for a general form of 1-SCLI methods which include \ExG. Upper bounds in the right-hand column are for algorithms which are implementable as online no-regret learning algorithms (e.g., \OG or online gradient descent), and lower bounds are shown for two classes of algorithms containing \OG and online gradient descent, namely $p$-SCLI algorithms for general $p \geq 1$ (recall for \OG, $p = 2$) as well as those satisfying a 2-step linear span assumption (see \cite{ibrahim_linear_2019}). The reported upper and lower bounds are stated for the total gap function (Definition \ref{def:total-gap}); leading constants and factors depending on distance between initialization and optimum are omitted.
    }}}
  \label{tab:last-iterate}
  \centering
 \begin{adjustbox}{center}
  \begin{tabular}{lll}
    \toprule
     & \multicolumn{2}{c}{Deterministic} \\
    Game class & \multicolumn{1}{c}{Extra gradient} & \multicolumn{1}{c}{Implementable as no-regret} \\
    \midrule
    \makecell[l]{$\mu$-strongly \\ monotone}& \makecell[l]{{\it Upper:} $\ell\left(1 - \frac{c\mu}{\ell}\right)^T$ \cite[EG]{mokhtari_unified_2019}\\ {\it Lower: } $\mu \left(1 - \frac{C\mu}{\ell} \right)^T$ \cite[$1$-SCLI]{azizian_tight_2019}}& \makecell[l]{{\it Upper:} $\ell  \left(1 - \frac{c\mu}{\ell} \right)^T$ \cite[OG]{mokhtari_unified_2019} \\ {\it Lower:} $\mu \left( 1 - \frac{C\mu}{\ell}\right)^T$ \cite[2-step lin.~span]{ibrahim_linear_2019} \\ {\it Lower:} $\mu \left(1 - \sqrt[p]{\frac{C\mu}{\ell}}\right)^T$ \cite[$p$-SCLI]{arjevani_lower_2015,ibrahim_linear_2019}}  \\
    \cmidrule(rl){1-3}
    \makecell[l]{Monotone, \\ $\gamma$-sing.~val.\\ low. bnd.} & \makecell[l]{{\it Upper:} $\ell  \left(1 - \frac{c \gamma^2}{\ell^2} \right)^T$ \cite[EG]{azizian_tight_2019} \\ {\it Lower:} $\gamma \left( 1 - \frac{C\gamma^2}{\ell^2} \right)^T$ \cite[1-SCLI]{azizian_tight_2019}} & \makecell[l]{{\it Upper:} $\ell  \left(1 - \frac{c \gamma^2}{\ell^2} \right)^T$ \cite[OG]{azizian_tight_2019} \\ {\it Lower:} $\gamma \left(1 - \frac{C\gamma}{\ell}\right)^T$ \cite[2-step lin.~span]{ibrahim_linear_2019}\\ {\it Lower:} $\gamma \left(1 - \sqrt[p]{\frac{C\gamma}{\ell}}\right)^T$ \cite[$p$-SCLI]{arjevani_lower_2015,ibrahim_linear_2019}}  \\
    \cmidrule(rl){1-3}
    $\lambda$-cocoercive & \multicolumn{1}{c}{Open}& {\it Upper:} $\frac{1}{\lambda\sqrt{T}}$ \cite[Online grad.~descent]{lin_finite-time_2020}   \\
    \cmidrule(rl){1-3}
    Monotone &\makecell[l]{{\it Upper:} $ \frac{\ell + \Lambda}{\sqrt{T}}$ \cite[EG]{golowich_last_2020}\\ {\it Lower: } $\frac{\ell}{\sqrt{T}}$ \cite[1-SCLI]{golowich_last_2020}} & \makecell[l]{{\it Upper:} $ \frac{\ell + \Lambda}{\sqrt{T}}$ ({\bf Theorem \ref{thm:ogda-last}}, OG) \\ {\it Lower: } $\frac{\ell}{\sqrt{T}}$ ({\bf Theorem \ref{thm:mm-spectral}}, $p$-SCLI, lin.~coeff.~matrices)}\\
    \bottomrule
  \end{tabular}
\if 0
  \begin{tablenotes}
    \item[$\dagger$] The 1-SCLI methods for which all lower bounds in the left-hand column are proved are slightly more general than taking $p=1$ in Definition \ref{def:pcli}: the vector $\bz\^{t-1}$ in (\ref{eq:scli-update}) is allowed to be multiplied by a matrix which is a {\it polynomial} in $\partial F$ (as opposed to an affine function of $\partial F$).
    \end{tablenotes}
    \fi
\end{adjustbox}
\vspace{-0.5cm}
\end{table}

\subsection{Our contributions}
In this paper we answer (\ref{eq:main-question}) in the affirmative for all {\it monotone games} (Definition \ref{def:monotone}) satisfying a mild smoothness condition, which includes smooth $\lambda$-cocoercive games and bilinear games. Many common and well-studied classes of games, such as zero-sum polymatrix games (\cite{bregman1987methods,daskalakis2009network,cai_zero-sum_2016}) and its generalization zero-sum socially-concave games (\cite{even2009convergence}) are monotone but are not in general $\lambda$-cocoercive. Hence our paper is the first to prove last-iterate convergence in the sense of (\ref{eq:main-question}) for the unconstrained version of these games as well. In more detail, we establish the following: %
\begin{itemize}
\item We show in Theorem \ref{thm:ogda-last} and Corollary \ref{cor:tgap} that the actions taken by learners following the {\it optimistic gradient (\OG)} algorithm, which is no-regret, exhibit last-iterate convergence to a Nash equilibrium in smooth, monotone games at a rate of $O(1/\sqrt{T})$ in terms of the global gap function. The proof uses a new technique which we call {\it adaptive potential functions} (Section \ref{sec:ada-potential}) which may be of independent interest. %
\item We show in Theorem \ref{thm:mm-spectral} that the rate $O(1/\sqrt{T})$ cannot be improved for any algorithm belonging to the class of $p$-SCLI algorithms (Definition \ref{def:pcli}), which includes \OG. %
\end{itemize}
The \OG algorithm is closely related to the {\it extra-gradient (\ExG)} algorithm (\cite{korpelevich_extragradient_1976,nemirovski_prox-method_2004}),\footnote{\ExG is also known as {\it mirror-prox}, which specifically refers to its generalization to general Bregman divergences.} which, at each time step $t$, assumes each player $k$ has an oracle $\MO_k$ which provides them with an additional gradient at a slightly different action than the action $\bz\^t\!k$ played at step $t$. Hence \ExG does not naturally fit into the standard setting of multi-agent learning. One could try to ``force'' \ExG into the setting of multi-agent learning by taking actions at odd-numbered time steps $t$ to simulate the oracle $\MO_k$, and using the even-numbered time steps to simulate the actions $\bz\^t\!k$ that \ExG actually takes. Although this algorithm exhibits last-iterate convergence at a rate of $O(1/\sqrt{T})$ in smooth monotone games when all players play according to it \cite{golowich_last_2020}, it is straightforward to see that it is {\it not} a no-regret learning algorithm, i.e., for an adversarial loss function the regret can be linear in $T$ (see Proposition \ref{prop:eg-regret} in Appendix \ref{sec:eg-linear-reg}).

Nevertheless, due to the success of \ExG at solving monotone variational inequalities, \cite{mertikopoulos_learning_2018} asked whether similar techniques to \ExG could be used to speed up last-iterate convergence to Nash equilibria. Our upper bound for \OG answers this question in the affirmative: various papers (\cite{chiang_online_2012,rakhlin_online_2012,rakhlin_optimization_2013,hsieh_convergence_2019}) have observed that \OG may be viewed as an approximation of \ExG, in which the previous iteration's gradient is used to simulate the oracle $\MO_k$. Moreover, our upper bound of $O(1/\sqrt{T})$ applies in many games for which the approach used in \cite{mertikopoulos_learning_2018}, namely Nesterov's dual averaging (\cite{nesterov_primal-dual_2009}), either fails to converge (such as bilinear games) or only yields asymptotic rates with decreasing learning rate (such as smooth strictly monotone games). Proving last-iterate rates for \OG has also been noted as an important open question in \cite[Table 1]{hsieh_convergence_2019}. At a technical level, the proof of our upper  bound (Theorem \ref{thm:ogda-last}) uses the proof technique in \cite{golowich_last_2020} for the last-iterate convergence of \ExG as a starting point. In particular, similar to \cite{golowich_last_2020}, our proof proceeds by first noting that some iterate $\bz\^{t^*}$ of \OG will have gradient gap $O(1/\sqrt{T})$ (see Definition \ref{def:grad-gap}; this is essentially a known result) and then showing that for all $t \geq t^*$ the gradient gap only increases by at most a constant factor. The latter step is the bulk of the proof, as was the case in \cite{golowich_last_2020}; however, since each iterate of \OG depends on the previous two iterates and gradients, the proof for \OG is significantly more involved than that for \ExG. We refer the reader to Section \ref{sec:ada-potential} and Appendix \ref{sec:ogda-proofs} for further details.

The proof of our lower bound for $p$-SCLI algorithms, Theorem \ref{thm:mm-spectral}, reduces to a question about the spectral radius of a family of polynomials. In the course of our analysis we prove a conjecture by \cite{arjevani_lower_2015} about such polynomials; though the validity of this conjecture is implied by each of several independent results in the literature (e.g., \cite{arjevani_iteration_2016,nevanlinna_convergence_1993}), our proof is more direct than previous ones. %

Lastly, we mention that our focus in this paper is on the unconstrained setting, meaning that the players' losses are defined on all of Euclidean space. We leave the constrained setting, in which the players must project their actions onto a convex constraint set, to future work. %

\subsection{Related work}
\paragraph{Multi-agent learning in games.} In the constrained setting, many papers have studied conditions under which the action profile of no-regret learning algorithms, often variants of Follow-The-Regularized-Leader (\FTRL), converges to equilibrium. However, these works all assume either a learning rate that decreases over time (\cite{mertikopoulos_learning_2018,zhou_countering_2017,zhou_learning_2018,zhou_mirror_2017}), or else only apply to specific types of {\it potential games} (\cite{krichene_convergence_2015,krichene_learning_2018,palaiopanos_multiplicative_2017,kleinberg_multiplicative_2009,chen_generalized_2016,blum_routing_2006,panageas_average_2014}), which significantly facilitates the analysis of last-iterate convergence.\footnote{In {\it potential games}, there is a canonical choice of potential function whose local  minima are equivalent to being at a Nash equilibrium. The lack of existence of a natural potential function in general monotone games is a significant challenge in establishing last-iterate convergence.} %

Such potential games are in general incomparable with monotone games, and do not even include finite-state two-player zero sum games (i.e., {\it matrix games}). In fact,
\cite{bailey_multiplicative_2018} showed that the actions of players following \FTRL in two-player zero-sum matrix games {\it diverge} from interior Nash equilibria. Many other works (\cite{hart_uncoupled_2003,mertikopoulos_cycles_2017,kleinberg_beyond_2011,daskalakis_learning_2010,balcan_weighted_2012,papadimitriou_nash_2016}) establish similar non-convergence results in both discrete and continuous time for various types of monotone games, including zero-sum polymatrix games. Such non-convergence includes chaotic behavior such as Poincar\'{e} recurrence, which showcases the insufficiency of on-average convergence (which holds in such settings) and so is additional motivation for the question (\ref{eq:main-question}).

\paragraph{Monotone variational inequalities \& \OG.} The problem of finding a Nash equilibrium of a monotone game is exactly that of finding a solution to a monotone variational inequality (VI). \OG was originally introduced by \cite{popov_modification_1980}, who showed that its iterates converge to solutions of monotone VIs, without proving explicit rates.\footnote{Technically, the result of \cite{popov_modification_1980} only applies to two-player zero-sum monotone games (i.e., finding the saddle point of a convex-concave function). The proof readily extends to general monotone VIs (\cite{hsieh_convergence_2019}).} It is also well-known that the {\it averaged} iterate of \OG converges to the solution of a monotone VI at a rate of $O(1/T)$ (\cite{hsieh_convergence_2019,mokhtari_convergence_2019,rakhlin_optimization_2013}), which is known to be optimal (\cite{nemirovski_prox-method_2004,ouyang_lower_2019,azizian_accelerating_2020}). Recently it has been shown (\cite{daskalakis_last-iterate_2018,lei_last_2020}) that a modification of \OG known as optimistic multiplicative-weights update exhibits last-iterate convergence to Nash equilibria in two-player zero-sum monotone games, but as with the unconstrained case (\cite{mokhtari_convergence_2019}) non-asymptotic rates are unknown. To the best of our knowledge, the only work proving last-iterate convergence rates for general smooth monotone VIs was \cite{golowich_last_2020}, which only treated the EG algorithm, which is not no-regret. There is a vast literature on solving VIs, and we refer the reader to \cite{facchinei_finite-dimensional_2003} for further references.

\section{Preliminaries}
Throughout this paper we use the following notational conventions. For a vector $\bv \in \BR^n$, let $\| \bv \|$ denote the Euclidean norm of $\bv$. For $\bv \in \BR^n$, set $\MB(\bv, R) := \{ \bz \in \BR^n : \| \bv - \bz \| \leq R\}$; when we wish to make the dimension explicit we write $\MB_{\BR^n}(\bv, R)$. For a matrix $\bA \in \BR^{n \times n}$ let  $\| \bA \|_\sigma$ denote the spectral norm of $\bA$. %

We let the set of $K$ players be denoted by $\MK := \{ 1, 2, \ldots K \}$. Each player $k$'s actions $\bz\!k$ belong to their {\it action set}, denoted $\MZ_k$, where $\MZ_k \subseteq \BR^{n_k}$ is a convex subset of Euclidean space. Let $\MZ = \prod_{k=1}^K \MZ_k \subseteq \BR^n$, where $n = n_1 + \cdots + n_K$. In this paper we study the setting where the action sets are unconstrained (as in \cite{lin_finite-time_2020}), meaning that $\MZ_k = \BR^{n_k}$, and $\MZ = \BR^n$, where $n = n_1 + \cdots + n_K$.
The {\it action profile} is the vector $\bz := (\bz\!1, \ldots, \bz\!K) \in \MZ$. For any player $k \in \MK$, let $\bz\!{-k} \in \prod_{k' \neq k} \MZ_{k'}$ be the vector of actions of all the other players. Each player $k \in \MK$ wishes to minimize its {\it cost function} $f_k : \MZ \ra \BR$, which is assumed to be twice continuously differentiable. %
The tuple $\MG := (\MK, (\MZ_k)_{k=1}^K, (f_k)_{k=1}^K)$ is known as a {\it continuous game}.

At each time step $t$, each player $k$ plays an action $\bz\^t\!k$; we assume the feedback to player $k$ is given in the form of the gradient $\grad_{\bz\!k} f_k(\bz\^t\!k, \bz\^t\!{-k})$ of their cost function with respect to their action $\bz\^t\!k$, given the actions $\bz\^t\!{-k}$ of the other players at time $t$. We denote the concatenation of these gradients by $F_\MG(\bz) := (\grad_{\bz\!1} f_1(\bz), \ldots, \grad_{\bz\!K}f_K(\bz)) \in \BR^n$. When the game $\MG$ is clear, we will sometimes drop the subscript and write $F : \MZ \ra \BR^n$. %

\paragraph{Equilibria \& monotone games.} A {\it Nash equilibrium} in the game $\MG$ is an action profile $\bz^* \in \MZ$ so that for each player $k$, it holds that $f_k(\bz^*\!k, \bz\!{-k}^*) \leq f_k(\bz\!k', \bz\!{-k}^*)$ for any $\bz\!{k}' \in \MZ_k$. %
Throughout this paper we study {\it monotone} games:
\begin{definition}[Monotonicity; \cite{rosen_existence_1965}]
  \label{def:monotone}
The game $\MG = (\MK, (\MZ_k)_{k=1}^K, (f_k)_{k=1}^K)$ is {\it monotone} if for all $\bz, \bz' \in \MZ$, it holds that $\lng F_\MG(\bz') - F_\MG(\bz), \bz' - \bz \rng \geq 0$. In such a case, we say also that $F_\MG$ is a monotone operator.
\end{definition}
The following classical result characterizes the Nash equilibria in monotone games:
\begin{proposition}[\cite{facchinei_finite-dimensional_2003}]
  \label{prop:nash-charac}
In the unconstrained setting, if the game $\MG$ is monotone, any Nash equilibrium $\bz^*$ satisfies $F_\MG(\bz^*) = {\ensuremath{\mathbf 0}}$. Conversely, if $F_\MG(\bz) = {\ensuremath{\mathbf 0}}$, then $\bz$ is a Nash equilibrium. %
\end{proposition}
In accordance with Proposition \ref{prop:nash-charac}, one measure of the proximity to equilibrium of some $\bz \in \MZ$ is the norm of $F_\MG(\bz)$:
\begin{defn}[Gradient gap function]
  \label{def:grad-gap}
Given a monotone game $\MG$ with its associated operator $F_\MG$, the {\it gradient gap function} evaluated at $\bz$ is defined to be $\| F_\MG(\bz) \|$.
\end{defn}
It is also common (\cite{mokhtari_convergence_2019,nemirovski_prox-method_2004}) to measure the distance from equilibrium of some $\bz \in \MZ$ by adding the maximum decrease in cost that each player could achieve by deviating from their current action $\bz\!k$: %
\begin{defn}[Total gap function]
  \label{def:total-gap}
  Given a monotone game $\MG = (\MK, (\MZ_k)_{k=1}^K, (f_k)_{k=1}^K)$, compact subsets $\MZ_k' \subseteq \MZ_k$ for each $k \in \MK$, and a point $\bz \in \MZ$, define the {\it total gap function} at $\bz$ with respect to the set $\MZ' := \prod_{k=1}^K \MZ_k'$ by
  $
\Gap_\MG^{\MZ'}(\bz) := \sum_{k=1}^K \left(f_k(\bz) - \min_{\bz\!{k}' \in \MZ_k'} f_k(\bz\!{k}', \bz\!{-k})\right).
$
At times we will slightly abuse notation, and for $F := F_\MG$, write $\Gap_F^{\MZ'}$ in place of $\Gap_\MG^{\MZ'}$.
\end{defn}
As discussed in \cite{golowich_last_2020}, it is in general impossible to obtain meaningful guarantees on the total gap function by allowing each player to deviate to an action in their entire space $\MZ_k$, which necessitates defining the total gap function in Definition \ref{def:total-gap} with respect to the compact subsets $\MZ_k'$. We discuss in Remark \ref{rmk:bounded} how, in our setting, it is without loss of generality to shrink $\MZ_k$ so that $\MZ_k = \MZ_k'$ for each $k$. Proposition \ref{prop:gradient-total} below shows that in monotone games, the gradient gap function upper bounds the total gap function:
\begin{proposition}
  \label{prop:gradient-total}
  Suppose $\MG = (\MK, (\MZ_k)_{k=1}^K, (f_k)_{k=1}^K)$ is a monotone game, and compact subsets $\MZ_k' \subset \MZ_k$ are given, where the diameter of each $\MZ_k'$ is upper bounded by $D > 0$. Then
  $$
\Gap_\MG^{\MZ'}(\bz) \leq D \sqrt{K} \cdot \| F_\MG(\bz) \|.
  $$
\end{proposition}
For completeness, a proof of Proposition \ref{prop:gradient-total} is presented in Appendix~\ref{sec:basic-proofs}.

\paragraph{Special case: convex-concave min-max optimization.} %
Since in a two-player zero-sum game $\MG = (\{1,2\}, (\MZ_1, \MZ_2), (f_1, f_2))$ we must have $f_1 = -f_2$, it is straightforward to show that $f_1(\bz\!1, \bz\!2)$ is convex in $\bz\!1$ and concave in $\bz\!2$. Moreover, it is immediate that Nash equilibria of the game $\MG$ correspond to saddle points of $f_1$; thus a special case of our setting is that of finding saddle points of convex-concave functions (\cite{facchinei_finite-dimensional_2003}). Such saddle point problems have received much attention recently since they can be viewed as a simplified model of generative adversarial networks (e.g., \cite{gidel_variational_2018,daskalakis_training_2017,chavdarova_reducing_2019,gidel_negative_2018,yadav_stabilizing_2017}). %

\paragraph{Optimistic gradient (\OG) algorithm.}
In the {\it optimistic gradient (\OG)} algorithm, each player $k$ performs the following update:
\begin{equation}
  \tag{\OG}\label{eq:opgd}
  \bz\^{t+1}\!k := \bz\^t\!k - 2 \eta_t \bg\^{t}\!k + \eta_t \bg\^{t-1} \!k, %
\end{equation}
where $\bg\^t\!k = \grad_{\bz\!k}f_k(\bz\^t\!k, \bz\^{t}\!{-k})$ for $t \geq 0$.
The following essentially optimal regret bound is well-known for the \OG algorithm, when the actions of the other players $\bz\^t\!{-k}$ (often referred to as the {\it environment}'s actions) are adversarial:%
\begin{proposition}
  \label{prop:op-noregret}
Assume that for all $\bz\!{-k}$ the function $\bz\!k \mapsto f_k(\bz\!k,\bz\!{-k})$ is convex. Then the regret of \OG with learning rate $\eta_t = O(D/L\sqrt{t})$ is $O(DL\sqrt{T})$, where $L = \max_t \| \bg\^t\!k\|$ and $D = \max \{  \|\bz\!k^*\|, \max_t \|\bz\^t\!k\| \}$.
\end{proposition}
In Proposition \ref{prop:op-noregret}, $\bz\!k^*$ is defined by $\bz^*\!k \in \argmin_{\bz\!k \in \MZ_k} \sum_{t'=0}^t f_k(\bz\!k, \bz\^{t'}\!{-k})$.
The assumption in the proposition that $\| \bz\^t\!k \| \leq D$ may be satisfied in the unconstrained setting by projecting the iterates onto the region $\MB(0, D) \subset \BR^{n_k}$, for some $D \geq \| \bz\!k^*\|$, without changing the regret bound. The implications of this modification to (\ref{eq:opgd}) are discussed further in Remark \ref{rmk:bounded}. %

\section{Last-iterate rates for \OG via adaptive potential functions}
\label{sec:ogda-lir}
In this section we show that in the unconstrained setting (namely, that where $\MZ_k = \BR^{n_k}$ for all $k \in \MK$), when all players act according to \OG, their iterates exhibit last-iterate convergence to a Nash equilibrium. %
Our convergence result holds for games $\MG$ for which the operator $F_\MG$ satisfies the following smoothness assumption:
\begin{assumption}[Smoothness]
  \label{asm:smoothness}
  For a monotone operator $F : \MZ \ra \BR^n$, assume that the following first and second-order Lipschitzness conditions hold, for some $\ell, \Lambda > 0$:
  \begin{align}
    \label{eq:1o-smooth} \forall \bz, \bz' \in \MZ, \qquad \| F(\bz) - F(\bz') \| & \leq \ell \cdot \| \bz - \bz' \| \\
    \label{eq:2o-smooth} \forall \bz, \bz' \in \MZ, \qquad \| \partial F(\bz) - \partial F (\bz') \|_\sigma & \leq \Lambda \cdot \| \bz - \bz' \|.
  \end{align}
  Here $\partial F: \MZ \ra \BR^{n \times n}$ denotes the Jacobian of $F$.
\end{assumption}
Condition (\ref{eq:1o-smooth}) is entirely standard in the setting of solving monotone variational inequalities (\cite{nemirovski_prox-method_2004}); condition (\ref{eq:2o-smooth}) is also very mild, being made for essentially all second-order methods (e.g., \cite{abernethy_last-iterate_2019,nesterov_cubic_2006}).

By the definition of $F_\MG(\cdot)$, when all players in a game $\MG$ act according to (\ref{eq:opgd}) with constant step size $\eta$, then the action profile $\bz\^t$ takes the form
\begin{equation}
  \label{eq:ogda}
\bz\^{-1}, \bz\^0 \in \BR^n, \qquad \bz\^{t+1} = \bz\^t - 2 \eta F_\MG(\bz\^t) + \eta F_\MG(\bz\^{t-1}) \ \ \forall t \geq 0.
\end{equation}
The main theorem of this section, Theorem \ref{thm:ogda-last}, shows that under the \OG updates (\ref{eq:ogda}), the iterates converge at a rate of $O(1/\sqrt{T})$ to a Nash equilibrium with respect to the gradient gap function:
\begin{theorem}[Last-iterate convergence of \OG]
  \label{thm:ogda-last}
Suppose $\MG$ is a monotone game so that $F_\MG$ satisfies Assumption \ref{asm:smoothness}. For some $\bz\^{-1}, \bz\^0 \in \BR^n$, suppose there is $\bz^* \in \BR^n$ so that $F_\MG(\bz^*) = 0$ and $\| \bz^* - \bz\^{-1} \| \leq D, \| \bz^* - \bz\^0 \| \leq D$. Then the iterates $\bz\^T$ of \OG (\ref{eq:ogda}) for any $\eta \leq \min\left\{ \frac{1}{150 \ell}, \frac{1}{1711 D \Lambda} \right\}$ satisfy:
  \begin{equation}
    \label{eq:last-iterate}
 \| F_\MG (\bz\^T) \| \leq \frac{60D}{\eta \sqrt{T}}
\end{equation}
\end{theorem}
By Proposition \ref{prop:gradient-total}, we immediately get a bound on the total gap function at each time $T$:
\begin{corollary}[Total gap function for last iterate of \OG]
  \label{cor:tgap}
In the setting of Theorem \ref{thm:ogda-last}, let $\MZ_k' := \MB(\bz\!k\^0, 3D)$ for each $k \in \MK$. Then, with $\MZ' = \prod_{k \in \MK} \MZ_k'$,
\begin{equation}
  \label{eq:tgap-ogda}
\Gap_\MG^{\MZ'}(\bz\^T) \leq \frac{180 KD^2}{\eta \sqrt{T}}.
  \end{equation}
\end{corollary}
We made no attempt to optimize the consants in Theorem \ref{thm:ogda-last} and Corollary \ref{cor:tgap}, and they can almost certainly be improved.

\begin{remark}[Bounded iterates]
  \label{rmk:bounded}
  Recall from the discussion following Proposition \ref{prop:op-noregret} that it is necessary to project the iterates of \OG onto a compact ball to achieve the no-regret property. As our guiding question (\ref{eq:main-question}) asks for last-iterate rates achieved by a no-regret algorithm, we should ensure that such projections are compatible with the guarantees in Theorem \ref{thm:ogda-last} and Corollary \ref{cor:tgap}. For this we note that \cite[Lemma 4(b)]{mokhtari_convergence_2019} showed that for the dynamics (\ref{eq:ogda}) without constraints, for all $t \geq 0$, $\| \bz\^t - \bz^* \| \leq 2 \| \bz\^0 - \bz^* \|$.
  Therefore, as long as we make the very mild assumption of a known a priori upper bound $\| \bz^* \| \leq D/2$ (as well as $\| \bz\^{-1}\!k \| \leq D/2$, $ \| \bz\^0\!k \| \leq D/2$), if all players act according to (\ref{eq:ogda}), then the updates (\ref{eq:ogda}) remain unchanged if we project onto the constraint sets $\MZ_k := \MB({\ensuremath{\mathbf 0}}, 3D)$ at each time step $t$. %
  This observation also serves as motivation for the compact sets $\MZ_k'$ used in Corollary \ref{cor:tgap}: the natural choice for $\MZ_k'$ is $\MZ_k$ itself, and by restricting $\MZ_k$ to be compact, this choice becomes possible.
\end{remark}

\subsection{Proof overview: adaptive potential functions}
\label{sec:ada-potential}
In this section we sketch the idea of the proof of Theorem \ref{thm:ogda-last}; full details of the proof may be found in Appendix~\ref{sec:ogda-proofs}. First we note that it follows easily from results of \cite{hsieh_convergence_2019} that \OG exhibits {\it best-iterate} convergence, i.e., in the setting of Theorem \ref{thm:ogda-last} we have, for each $T > 0$, $\min_{1 \leq t \leq T} \| F_\MG(\bz\^t) \| \leq O(1/\sqrt{T})$.\footnote{In this discussion we view $\eta, D$ as constants.} The main contribution of our proof is then to show the following: if we choose $t^*$ so that $\| F_\MG(\bz\^{t^*}) \| \leq O(1/\sqrt{T})$, then for all $t' \geq t^*$, we have $\| F_\MG(\bz\^{t'}) \| \leq O(1) \cdot \| F_\MG(\bz\^{t^*})\|$. This was the same general approach taken in \cite{golowich_last_2020} to prove that the extragradient (EG) algorithm has last-iterate convergence. In particular, they showed the stronger statement that $\| F_\MG(\bz\^t) \|$ may be used as an approximate potential function in the sense that it only increases by a small amount each step:
\begin{equation}
  \label{eq:grow-slow}
\| F_\MG(\bz\^{t'+1}) \| \underbrace{\leq}_{t' \geq 0} (1 + \| F(\bz\^{t'}) \|^2) \cdot \| F_\MG(\bz\^{t'}) \| \underbrace{\leq}_{t' \geq t^*} (1 + O(1/T)) \cdot \| F_\MG(\bz\^{t'}) \|.
\end{equation}
However, their approach relies crucially on the fact that for the EG algorithm, $\bz\^{t+1}$ depends only on $\bz\^t$. For the \OG algorithm, it is possible that (\ref{eq:grow-slow}) fails to hold, even when $F_\MG(\bz\^t)$ is replaced by the more natural choice of $(F_\MG(\bz\^{t}), F_\MG(\bz\^{t-1}))$.\footnote{For a trivial example, suppose that $n = 1$, $F_\MG(\bz) = \bz$, $\bz\^{t'} = \delta > 0$, and $\bz\^{t'-1} = 0$. Then $\|(F_\MG(\bz\^{t'}), F_\MG(\bz\^{t'-1}))\| = \delta$ but $\|(F_\MG(\bz\^{t'+1}), F_\MG(\bz\^{t'}))\| > \delta\sqrt{2 - 4\eta}$.\label{foot:potential-grow}} %

Instead of using $\| F_\MG(\bz\^t) \|$ as a potential function in the sense of (\ref{eq:grow-slow}), we propose instead to track the behavior of $\| \tilde F\^t \|$, where %
\begin{equation}
  \label{eq:Ct}
\tilde F\^t := F_\MG(\bz\^t + \eta F_\MG(\bz\^{t-1})) + \bC\^{t-1} \cdot F_\MG(\bz\^{t-1}) \in \BR^n,
\end{equation}
and the matrices $\bC\^{t-1} \in \BR^{n \times n}$ are defined recursively {\it backwards}, i.e., $\bC\^{t-1}$ depends directly on $\bC\^t$, which depends directly on $\bC\^{t+1}$, and so on. For an appropriate choice of the matrices $\bC\^t$, we show that $\tilde F\^{t+1} = (I - \eta \bA\^t + \bC\^t) \cdot \tilde F\^t$, for some matrix $\bA\^t \approx \partial F_\MG(\bz\^t)$. We then show that for $t \geq t^*$, it holds that $\| I - \eta \bA\^t + \bC\^t \|_\sigma \leq 1 + O(1/T)$, from which it follows that $\| \tilde F\^{t+1} \| \leq (1 + O(1/T)) \cdot \| \tilde F\^t \|$. This modification of (\ref{eq:grow-slow}) is enough to show the desired upper bound of $\| F_\MG(\bz\^T) \| \leq O(1/\sqrt{T})$.

To motivate the choice of $\tilde F\^t$ in (\ref{eq:Ct}) it is helpful to consider the simple case where $F(\bz) = \bA \bz$ for some $\bA \in \BR^{n \times n}$, which was studied by \cite{liang_interaction_2018}. Simple algebraic manipulations using (\ref{eq:ogda}) (detailed in Appendix~\ref{sec:ogda-proofs}) show that, for the matrix $\bC := \frac{(I + (2\eta \bA)^2)^{1/2} - I}{2}$, we have $\tilde F\^{t+1} = (I - \eta \bA + \bC) \tilde F\^t$ for all $t$. It may be verified that we indeed have $\bA\^t = \bA$ and $\bC\^t = \bC$ for all $t$ in this case, and thus (\ref{eq:Ct}) may be viewed as a generalization of these calculations to the nonlinear case.

\paragraph{Adaptive potential functions.}
In general, a {\it potential function} $\Phi(F_\MG, \bz)$ depends on the problem instance, here taken to be $F_\MG$, and an element $\bz$ representing the current state of the algorithm. Many convergence analyses from optimization (e.g., \cite{bansal_potential-function_2017,wilson_lyapunov_2018}, and references therein) have as a crucial element in their proofs a statement of the form $\Phi(F_\MG, \bz\^{t+1}) \lesssim \Phi(F_\MG, \bz\^t)$. For example, for the iterates $\bz\^t$ of the EG algorithm, \cite{golowich_last_2020} (see (\ref{eq:grow-slow})) used the potential function $\Phi(F_\MG, \bz\^{t}) := \| F_\MG(\bz\^t) \|$.

Our approach of controlling the the norm of the vectors $\tilde F\^t$ defined in (\ref{eq:Ct}) can also be viewed as an instantion of the potential function approach: since each iterate of \OG depends on the previous two iterates, the state is now given by $\bv\^t := (\bz\^{t-1}, \bz\^t)$. The potential function is given by $\Phi_{\OGmath}(F_\MG, \bv\^t) := \| \tilde F\^t\|$, where $\tilde F^\t$ is defined in (\ref{eq:Ct}) and indeed only depends on $\bv\^t$ once $F_\MG$ is fixed since $\bv\^t$ determines $\bz\^{t'}$ for all $t' \geq t$ (as \OG is deterministic), which in turn determine $\bC\^{t-1}$. However, the potential function $\Phi_{\OGmath}$ is quite unlike most other choices of potential functions in optimization (e.g., \cite{bansal_potential-function_2017}) in the sense that it depends {\it globally} on $F_\MG$: For any $t' > t$, a local change in $F_\MG$ in the neighborhood of $\bv\^{t'}$ may cause a change in $\Phi_{\OGmath}(F_\MG, \bv\^t)$, {\it even if $\| \bv\^t - \bv\^{t'} \|$ is arbitrarily large}. Because $\Phi_{\OGmath}(F_\MG, \bv\^t)$ adapts to the behavior of $F_\MG$ at iterates later on in the optimization sequence, we call it an
{\it adaptive potential function}. We are not aware of any prior works using such adaptive potential functions to prove last-iterate convergence results, and we believe this technique may find additional applications.

\section{Lower bound for convergence of $p$-SCLIs}
\label{sec:scli-lb}
The main result of this section is Theorem \ref{thm:mm-spectral}, stating that the bounds on last-iterate convergence in Theorem \ref{thm:ogda-last} and Corollary \ref{cor:tgap} are tight when we require the iterates $\bz\^T$ to be produced by an optimization algorithm satisfying a particular formal definition of ``last-iterate convergence''. %
Notice that that we cannot hope to prove that they are tight for {\it all} first-order algorithms, since the averaged iterates $\bar \bz\^T := \frac 1T \sum_{t=1}^T \bz\^t$ of \OG satisfy $\Gap_\MG^{\MZ'}(\bar \bz\^T) \leq O\left( \frac{D^2}{\eta T} \right)$ \cite[Theorem 2]{mokhtari_convergence_2019}. Similar to \cite{golowich_last_2020}, we use {\it $p$-stationary canonical linear iterative methods ($p$-SCLIs)} to formalize the notion of ``last-iterate convergence''. \cite{golowich_last_2020} only considered the special case $p=1$ to establish a similar lower bound to Theorem \ref{thm:mm-spectral} for a family of last-iterate algorithms including the extragradient algorithm. The case $p > 1$ leads to new difficulties in our proof since even for $p = 2$ we must rule out algorithms such as Nesterov's accelerated gradient descent (\cite{nesterov_introductory_1975}) and P\'olya's heavy-ball method (\cite{polyak_introduction_1987}), a situation that did not arise for $p=1$.

\if 0
$\MFbil_{n,\ell,D}$, defined to be the set of $\ell$-Lipschitz operators $F : \BR^n \ra \BR^n$ of the form
\begin{equation}
  \label{eq:linear-F}
  F(\bz) = \bA \bz + \bb, \qquad \bz = \matx{\bx \\ \by}, \quad \bA = \matx{{\ensuremath{\mathbf 0}} & \bM \\ -\bM^\t & {\ensuremath{\mathbf 0}}}, \quad \bb = \matx{\bb_1 \\ -\bb_2},
\end{equation}
for which $\bA$ is of full-rank and $-\bA^{-1} \bb \in \MB_{\BR^{n/2}}(0,D) \times \MB_{\BR^{n/2}}(0,D)$. For $F$ in (\ref{eq:linear-F}), we have $F = F_\MG$ for the two-player zero-sum game $\MG$ with objective function $f(\bx, \by) = \bx^\t \bM \by + \bb_1^\t \bx + \bb_2^\t \by$. The unique Nash eqilibrium of $\MG$ is given by $\bz^* = -\bA^{-1} \bb$.
\fi
\begin{defn}[$p$-SCLIs \cite{arjevani_lower_2015,azizian_accelerating_2020}]
  \label{def:pcli}
  An algorithm $\MA$ is a {\it first-order $p$-stationary canonical linear iterative algorithm ($p$-SCLI)} if, given a monotone operator $F$, and an arbitrary set of $p$ initialization points $\bz\^0, \bz\^{-1}, \ldots, \bz\^{-p+1} \in \BR^n$, it generates iterates $\bz\^t$, $t \geq 1$, for which
  \begin{equation}
    \bz\^t = \sum_{j=0}^{p-1} \alpha_j \cdot F(\bz\^{t-p+j}) + \beta_j \cdot \bz\^{t-p+j},
    \label{eq:scli-update}
  \end{equation}
  for $t = 1,2,\ldots$, where $\alpha_j, \beta_j \in \BR$ are any scalars.\footnote{We use slightly different terminology from \cite{arjevani_lower_2015}; technically, the $p$-SCLIs considered in this paper are those in \cite{arjevani_lower_2015} with {\it linear coefficient matrices}.} %
\end{defn}
From (\ref{eq:ogda}) it is evident that OG with constant step size $\eta$ is a 2-SCLI with $\beta_1 = 1, \beta_0 = 0, \alpha_1 = -2\eta, \alpha_0 = \eta$. Many standard algorithms for convex function minimization, including gradient descent, Nesterov's accelerated gradient descent (AGD), and P\'olya's Heavy Ball method, %
are of the form (\ref{eq:scli-update}) as well. We additionally remark that several variants of SCLIs (and their non-stationary counterpart, CLIs) have been considered in recent papers proving lower bounds for min-max optimization (\cite{azizian_tight_2019,ibrahim_linear_2019,azizian_accelerating_2020}).

For simplicity, we restrict our attention to monotone operators $F$ arising as $F = F_\MG : \BR^n \ra \BR^n$ for a two-player zero-sum game $\MG$ (i.e., the setting of min-max optimization). For simplicity suppose that $n$ is even and for $\bz \in \BR^n$ write $\bz = (\bx, \by)$ where $\bx, \by \in \BR^{n/2}$.
Define $\MFbil_{n,\ell,D}$ to be the set of $\ell$-Lipschitz operators $F : \BR^n \ra \BR^n$ of the form $F(\bx, \by) = (\grad_\bx f(\bx, \by), -\grad_\by f(\bx, \by))^\t$ for some bilinear function $f : \BR^{n/2} \times \BR^{n/2} \ra \BR$, with a unique equilibrium point $\bz^* = (\bx^*, \by^*)$%
, which satisfies $\bz^* \in \MD_D := \MB_{\BR^{n/2}}({\ensuremath{\mathbf 0}},D) \times \MB_{\BR^{n/2}}({\ensuremath{\mathbf 0}},D)$. The following Theorem \ref{thm:mm-spectral} uses functions in $\MFbil_{n,\ell,D}$ as ``hard instances'' to show that the $O(1/\sqrt{T})$ rate of Corollary \ref{eq:tgap-ogda} cannot be improved by more than an {\it algorithm-dependent} constant factor.
\begin{theorem}[Algorithm-dependent lower bound for $p$-SCLIs]
  \label{thm:mm-spectral}
  Fix $\ell, D > 0$, let $\MA$ be a $p$-SCLI, and let $\bz\^t$ denote the $t$th iterate of $\MA$. Then there are constants $c_\MA, T_\MA > 0$ so that the following holds: For all $T \geq T_\MA$, there is some $F \in \MFbil_{n,\ell,D}$ so that for some initialization $\bz\^{0}, \ldots, \bz\^{-p+1} \in \MD_D$ and $T' \in \{ T, T+1, \ldots, T+p-1 \}$, it holds that $\Gap_F^{\MD_{2D}}(\bz\^{T'}) \geq  \frac{c_\MA \ell D^2}{\sqrt{T}}$.
\end{theorem}
We remark that the order of quantifiers in Theorem \ref{thm:mm-spectral} is important: if instead we first fix a monotone operator $F \in \MFbil_{n,\ell,D}$ corresponding to some bilinear function $f(\bx, \by) = \bx^\t \bM \by$, then as shown in \cite[Theorem 3]{liang_interaction_2018}, the iterates $\bz\^T = (\bx\^T, \by\^T)$ of the \OG algorithm will converge at a rate of $e^{-O\left( \frac{\sigma_{\min}(\bM)^2}{\sigma_{\max}(\bM)^2} \cdot T \right)}$, which eventually becomes smaller than the sublinear rate of $1/\sqrt{T}$.\footnote{$\sigma_{\min}(\bM)$ and $\sigma_{\max}(\bM)$ denote the minimum and maximum singular values of $\bM$, respectively. The matrix $\bM$ is assumed in \cite{liang_interaction_2018} to be a square matrix of full rank (which holds for the construction used to prove Theorem \ref{thm:mm-spectral}).} Such ``instance-specific'' bounds are complementary to the minimax perspective taken in this paper.

We briefly discuss the proof of Theorem \ref{thm:mm-spectral}; the full proof is deferred to Appendix~\ref{sec:lb-proofs}. As in prior work proving lower bounds for $p$-SCLIs (\cite{arjevani_lower_2015,ibrahim_linear_2019}), we reduce the problem of proving a lower bound on $\Gap_\MG^{\MD_D}(\bz\^t)$ to the problem of proving a lower bound on the supremum of the spectral norms of a family of polynomials (which depends on $\MA$). Recall that for a polynomial $p(z)$, its {\it spectral norm} $\rho(p(z))$ is the maximum norm of any root. We show: %
\begin{proposition}
\label{prop:local-linear}
Suppose $q(z)$ is a degree-$p$ monic real polynomial such that $q(1) = 0$, $r(z)$ is a polynomial of degree $p-1$, and $\ell > 0$. Then there is a constant $C_0 > 0$, depending only on $q(z), r(z)$ and $\ell$, and some $\mu_0 \in (0,\ell)$, so that for any $\mu \in (0, \mu_0)$,
$$
\sup_{\nu \in [\mu, \ell]} \rho(q(z) - \nu \cdot r(z)) \geq 1 - C_0 \cdot \frac{\mu}{\ell}.
$$
\end{proposition}
The proof of Proposition \ref{prop:local-linear} uses elementary tools from complex analysis. The fact that the constant $C_0$ in Proposition \ref{prop:local-linear} depends on $q(z), r(z)$ leads to the fact that the constants $c_\MA, T_\MA$ in Theorem \ref{thm:mm-spectral} depend on $\MA$. Moreover, we remark that this dependence cannot be improved from Proposition \ref{prop:local-linear}, so removing it from Theorem \ref{thm:mm-spectral} will require new techniques:
\begin{proposition}[Tightness of Proposition \ref{prop:local-linear}]
  \label{prop:local-linear-tight}
  For any constant $C_0 > 0$ and $\mu_0 \in (0,\ell)$, there is some $\mu \in (0,\mu_0)$ and polynomials $q(z), r(z)$ so that $\sup_{\nu \in [\mu, \ell]} \rho(q(z) - \nu \cdot r(z)) < 1 - C_0 \cdot \mu$. Moreover, the choice of the polynomials is given by
  \begin{equation}
    \label{eq:qr}
    q(z) = \ell ( z -\alpha)(z-1), \qquad r(z) = -(1+\alpha)z + \alpha \qquad \text{for} \qquad \alpha := \frac{\sqrt{\ell} - \sqrt{\mu}}{\sqrt{\ell} + \sqrt{\mu}}.
  \end{equation}
\end{proposition}
The choice of polynomials $q(z), r(z)$ in (\ref{eq:qr}) are exactly the polynomials that arise in the $p$-SCLI analysis of Nesterov's AGD \cite{arjevani_lower_2015}; as we discuss further in Appendix~\ref{sec:lb-proofs}, Proposition \ref{prop:local-linear} is tight, then, even for $p=2$, because acceleration is possible with a $2$-SCLI.
As byproducts of our lower bound analysis, we additionally obtain the following:
\begin{itemize}
\item Using Proposition \ref{prop:local-linear}, we show that any $p$-SCLI algorithm must have a rate of at least $\Omega_\MA(1/T)$ for smooth convex function minimization (again, with an algorithm-dependent constant).\footnote{\cite{arjevani_iteration_2016} claimed to prove a similar lower bound for stationary algorithms in the setting of smooth convex function minimization; however, as we discuss in Appendix~\ref{sec:lb-proofs}, their results only apply to the strongly convex case, where they show a linear lower bound.%
  } This is slower than the $O(1/T^2)$ error achievable with Nesterov's AGD with a time-varying learning rate. %
\item We give a direct proof of the following statement, which was conjectured by \cite{arjevani_lower_2015}: for polynomials $q,r$ in the setting of Proposition \ref{prop:local-linear}, for any $0 < \mu < \ell$, there exists $\nu \in [\mu,\ell]$ so that $\rho(q(z) - \nu \cdot r(z)) \geq \frac{\sqrt{\ell/\mu} - 1}{\sqrt{\ell/\mu} + 1}$. Using this statement, for the setting of Theorem \ref{thm:mm-spectral}, we give a proof of an {\it algorithm-independent} lower bound $\Gap_F^{\MD_D} (\bz\^t) \geq \Omega(\ell D^2 / T)$. Though the algorithm-independent lower bound of $\Omega(\ell D^2/T)$ has already been established in the literature, even for non-stationary CLIs (e.g., \cite[Proposition 5]{azizian_accelerating_2020}), we give an alternative proof from existing approaches.
\end{itemize}

\vspace{-0.4cm}
\section{Discussion}
In this paper we proved tight last-iterate convergence rates for smooth monotone games when all players act according to the optimistic gradient algorithm, which is no-regret. We believe that there are many fruitful directions for future research. First, it would be interesting to obtain last-iterate rates in the case that each player's actions is constrained to the simplex and they use the {\it optimistic multiplicative weights update (OMWU)} algorithm.  \cite{daskalakis_last-iterate_2018,lei_last_2020} showed that OMWU exhibits last-iterate convergence, but non-asymptotic rates remain unknown even for the case that $F_\MG(\cdot)$ is linear, which includes finite-action polymatrix games. %
Next, it would be interesting to determine whether Theorem \ref{thm:ogda-last} holds if (\ref{eq:2o-smooth}) is removed from Assumption \ref{asm:smoothness}; this problem is open even for the EG algorithm (\cite{golowich_last_2020}). Finally, it would be interesting to extend our results to the setting where players receive noisy gradients (i.e., the stochastic case). %
As for lower bounds, it would be interesting to determine whether an algorithm-independent lower bound of $\Omega(1/\sqrt{T})$ in the context of Theorem \ref{thm:mm-spectral} could be proven for stationary $p$-SCLIs. %
As far as we are aware, this question is open even for convex minimization (where the rate would be $\Omega(1/T)$).

\neurips{
\newpage
\section*{Broader impact}
As this is a theoretical paper, we expect that the direct ethical and societal impacts  of this work will be limited. As the setting of multi-agent learning in games describes many systems with potential for practical impact, such as GANs, we believe that the insights developed in this paper may eventually aid the improvement of such technologies. If not deployed and regulated carefully, technologies such as GANs could lead to harmful outcomes, such as through the proliferation of false media (``deepfakes''). We hope that, through a combination of legal and technological measures, such negative impacts of GANs can be limited and the positive applications, such as drug discovery and image analysis in the medical field, may be realized.
}

\neurips{
\begin{ack}
  We thank Yossi Arjevani for a helpful conversation.

  N.G.~is supported by a Fannie \& John Hertz Foundation Fellowship and an NSF Graduate Fellowship. C.D.~is supported by NSF Awards IIS-1741137, CCF-1617730 and CCF-1901292, by a Simons Investigator Award, and by the DOE PhILMs project (No. DE-AC05-76RL01830).

\end{ack}}

\arxiv{
  \section*{Acknowledgements}
    We thank Yossi Arjevani for a helpful conversation.
}

\neurips{{\small

redpajama_set_name: RedPajamaArXiv