text:
Welcome from Head of School
Overview and Board Governance
Members' Bios
Casady's History
Campus Virtual Tours and Map
The Episcopal Spirit at Casady School
New Faculty in 2021-22
Strategic Academic Plan (PK-12)
Primary Division (PK-K)
Lower Division (1-4)
Lower Division | New Schedule
Middle Division (5-8)
Upper Division (9-12)
Extended Programs
Dean's Fly-In Program
Athletic Conference Affiliation
Fitness and PE
Athletics: Middle Division
Today's Games (MD)
Teams and Coaches (MD)
Athletics: Upper Division
Today's Games (UD)
Teams and Coaches (UD)
Collegiate Athletes by Decade
Giving to Casady
Make an Annual Fund Gift Online
Make an Endowment Gift Online
Campaign for Casady School
A Message from the Head of School
Volunteer Leadership
Contact Us and Support the Campaign
Casady Golf Tournament
Alumni Christmas Party
Casady Parents' Organization
One Night Alone, Together
Cyclone Booster Club
Join the Booster Club Online
Purchase Parking Passes Online
Sponsor the Sports Guide Online
Casady News Headlines
About Casady
Committed to a Deeper Level of Learning
At Casady, all students are provided a foundation of excellence, which instills them with the educational and social skills to triumph in their future endeavors.
Your Journey Begins Today
Selecting a school is one of the most important decisions a parent can make. At Casady, we take that investment seriously and work to provide our students a life of balance, purpose, and joy.
Create an Inquiry
The Courage To Explore
Casady's academic program provides our students the challenges and opportunities necessary to develop critical and creative thinking skills, the ability to plan, evaluate and implement actions, and most importantly, the support to nurture their passions.
About Athletics
Forever A Cyclone
Competitive athletics and physical education are central to the development of leadership and teamwork as well as self-confidence and respect for others.
Pursuing Passions
Through the arts, students can learn valuable skills and talents that provide universal benefits and gain an understanding of a world deeper than they had previously believed. Fine arts programs are imperative in fostering creativity and inspiring the mind, as well as the imagination.
About Advancement
Supporting the Mission
Casady's advancement initiatives help further the Mission of the School by supporting programs and activities that take place in the classroom, on the stage, or on the field.
MyCASADY
Casady Podcast Network
Faculty Staff Directory
Casady focuses its financial aid dollars on students entering grades 5 - 12 to help offset the higher tuition in the Middle and Upper Divisions.
Ledbetter Scholars
This scholarship fund was established with an estate gift from Bud Ledbetter '53 in April 2007 and was established to honor his memory and that of his wife Pauline Morrison Ledbetter '55. The scholarship distributions are directed to students who show great academic promise and who demonstrate financial need. Scholarships are renewable each year until graduation, providing the student maintains an excellent academic record and demonstrates good citizenship. If you are interested in the Ledbetter scholarship, please contact the Admission Office at (405) 749-3185.
Malone Scholars
Recognizing its academic leadership, the Malone Family Foundation awarded Casady School with a $2 million scholarship endowment in May 2006. Created for incoming students in grades seven through 12 who demonstrate outstanding abilities as well as financial need, the scholarship enables qualified young adults to benefit from an independent education. Scholarships are renewable each year until graduation, providing the student maintains an excellent academic record and demonstrates good citizenship. If you are interested in the Malone scholarship, please contact the Admission Office.
University Scholars Program
Through its University Scholars Program, Casady School offers a tuition discount to children of full-time teaching faculty at participating Oklahoma City area colleges and universities. Families who qualify and enroll at Casady receive a 10 percent discount. Additionally, students entering grades five through 12 may receive up to 100 percent of need as demonstrated by Facts Tuition Aid. The deadline for University Scholars applications is May 1. After that date, qualified families can apply for the following school year. Click here for the application. Applications are also available in the Admission Office.
Episcopal Clergy Discount
Casady School offers a tuition discount to children of full-time clergy of the Episcopalian Diocese of Oklahoma and local parishes. Families who qualify and enroll at Casady receive a 20 percent tuition discount. Additionally, students entering grades 5 through 12 may receive 100 percent of need as demonstrated by Facts Tuition Aid. The deadline for Clergy Scholarship applications is May 1. After that date, qualified families can apply for the following school year.
We recognize the financial investment in a Casady education may be a significant expense for many families. Qualified students should not be deterred from applying to Casady for financial reasons. Through scholarships, payment plans and need-based financial aid, we work toward making a Casady education feasible for highly motivated children throughout the Oklahoma City area.
Financial aid awards typically meet 80 percent of the difference between what a family can afford and the cost of tuition. Financial aid consists of grants that need not be repaid.
Casady respects the freedom and responsibility families have to make choices regarding their financial status. However, if only one parent works outside the home, the Financial Aid Committee must take this into consideration. Casady's Financial Aid Committee calculates an income for the second parent at the level of minimum wage minus taxes. An exception to this policy might occur if the parent is caring for a pre-school-aged child, a child with a disability, a seriously ill child, parent or grandparent or if there are other special circumstances.
This action will ensure the most effective use of the School's financial aid budget and help more qualified families afford the cost of a Casady education.
Casady annually awards more than $1.5 million in financial aid to approximately 20 percent of our student population.
Oct. 15: Open Enrollment for Financial Aid
Nov. 30: Current Families - Applications Complete*
Feb. 1: Prospective Families - Applications Complete*
* Please use 2019 Federal Income Tax Return
*Financial Aid applications will be reviewed on a rolling basis after the above noted deadlines, subject to available funding.
Financial aid awards are determined by a Financial Aid Committee. The awards are based primarily on need, although a few merit-based awards also are available. Need-based financial aid awards are determined through a computation provided by FACTS Grant and Aid Assessment, based on figures submitted by the family. All financial information is held strictly confidential.
To apply, go online to https://online.factsmgt.com/signin/44D31, complete the online application, and pay the $35 FACTS application fee.
Submit the appropriate tax forms required by FACTS and complete the rest of the application checklist.
FACTS will provide Casady with the information needed to award any financial aid for which you may be eligible. Your financial information stays strictly confidential.
Casady will not grant financial aid awards until the student has been accepted or reenrolled and we have received:
A signed enrollment/reenrollment contract;
The FACTS Grant and Aid Assessment verified application.
In order to ensure timely processing, we recommend completing the process online as early as November. Casady will not accept paper applications for FACTS. Contact Melissa Jenkins in Casady's Business Office at (405) 749-3165 or jenkinsm@casady.org with questions about this process.
Financial aid is limited and will be distributed to the best qualified candidates who have a verified application on file with FACTS.
9500 N. Pennsylvania Ave., Oklahoma City, OK 73120 | Phone: (405) 749-3100
Casady School is an independent, co-educational, college preparatory, Episcopal day school serving students in pre-k-12. Educating Mind, Body, and Spirit.
Casady School is a PreK-12, independent, college preparatory Episcopal day school committed to deeper-level learning. Casady School welcomes a student body that reflects the diversity of the world around us and therefore does not discriminate on the basis of race, color, religion, gender identity, sexual orientation, socioeconomic status, nationality, or ethnic origin in administration of its educational policies, admissions policies, scholarship and loan programs, athletics, and other school-administered programs generally accorded or made available to students at the School.


Preparation of the results of the Mitigation Study as supporting evidence for the Sentencing Memorandum.
The prosecution has the resources of the State and all of its cooperative agreements with related agencies to gather data to use as evidence of aggravation.
The Sentencing Mitigation Study is prepared to give the Defender evidence to argue for mitigation of the sentence.


text:
Cagnac-les-Mines (; ) is a commune in the Tarn department in southern France.

It is a former coalmining town for workers of the Compagnie minière de Carmaux. It has a mining museum, whereupon is standing a huge miners safety lamp.

Geography
The Vère forms part of the commune's north-western border.

See also
Communes of the Tarn department

References

Communes of Tarn (department)

redpajama_set_name: RedPajamaWikipedia


JoyFest Artists to Sing at Stellar Awards This Weekend!
Home / Blog / Artist News / JoyFest Artists to Sing at Stellar Awards This Weekend!
JoyFest artists Jonathan McReynolds, Koryn Hawthorne and Erica Campbell of Mary Mary will be among the artists who are scheduled to perform at the 34th Annual Stellar Awards this weekend. The Stellar Awards will air nationwide later this spring. You can find a local date and time HERE.
Campbell is the host of the highly-rated nationally syndicated radio show "Get Up! Mornings with Erica Campbell" that can be heard weekdays 6am-10am EST in over 40 markets, while ministering internationally.


The invasion of Normandy was one of the most important events in the 20th century. Soon, you'll be able to recreate those pivotal days on your tabletops with Ultracombat Normandy. Dishdash Games is bringing you a new set of miniatures game rules, and you can get the hardcover book, as well as three decks of cards for one special price.


text:
Outgoing trustee says township is in 'good hands'
Posted by Joe Pinchot | Jan 12, 2018 | Politics | 0 |
Although Brookfield voters ousted Trustee Dion Magestro, he is not going out on a sour note.
"I will tell you that I believe that, as it stands right now, the township is in good hands," Magestro said at a Dec. 20 meeting.
The good hands are those of Trustees Ron Haun and Gary Lees, who was re-elected, and Fiscal Administrator Dena McMullin.
"I commend you both for having amicable discussions where you're not throwing chairs and calling each other names," Magestro told Haun and Lees.
"I hope that you guys continue that the township heads in the right direction by being able to look in the mirror every night and saying, 'I did the right thing. I didn't necessarily do the popular thing – I did the right thing,'" Magestro said.
He called McMullin's financial advice an "integral part in the machine."
"Take that little part out and I think the machine could fall apart," he said.
Magestro offered his services in "an official or an unofficial capacity."
"I will be here to help," he said. "I hope I can lend some insight as to doing things in a reasonable manner."
Lees said he has enjoyed working with Magestro for four years.
"His business abilities have certainly been a big plus," Lees said at a Dec. 4 meeting. "He's certainly been a good right hand for the last four years. I certainly will miss him and I certainly know the township will miss him, also."
"It's been an honor and a pleasure to serve with you," Haun said. "What you've brought to the table was very valuable to this township. I've been in office 10 years and you were a breath of fresh air because you brought a mindset of trying to get every department in this township to operate the way it should. I'm very grateful for what you did. You became my friend the last four years."
PreviousSchool board to hear plan for new stadium
NextIce queen takes first step on road to Olympics
Trustees set new meetings
Trustees, chiefs to set parameters for search
Lees: Suttles won't accept decisions
Trustees' 2nd meeting draws twice as many

redpajama_set_name: RedPajamaCommonCrawl


To analyze and update the existing protocols and create protocols which do not currently exist such as protocols related to harassment against students.
To approve a protocol that includes sexist violence, transphobia and homophobia by means of a participative process that involves all staff and students. The final outcome will be that everyone in the faculties know what to do in case of suffering sexual harassment.
A session was held at MU level in December 2017 to make the collective aware of the importance of having an adequate protocol and that the approval process should be participatory.
Once the participatory approach was accepted, a consultancy specialized in designing protocols was hired and the process was initiated during October 2018.
The first step was to carry out a questionnaire among the members of the Faculty of Humanities and Education to know the real situation in the faculty. However, fewer responses than expected were obtained which affected the development of the process.
The second step is to create working groups to go into the answers obtained from the questionnaire in more depth with the aim of designing a protocol adapted to the real situation of the faculty. Unfortunately, very few people volunteered for the working groups and we were forced to designate people to work in the groups.
The third step will be to design the protocol taking into account the diagnosis and the protocols that are currently in force in other universities.
The protocol will then be revised by various committees and passed to the Governing Board for approval.
The greatest resistance is the lack of participation of the university members (staff and students), both when filling in the questionnaire and when participating in the working groups.
We must continue to insist on the importance of the entire community being involved in order to obtain a protocol that is known by everyone. Participation in the design and approval of the protocol will lead to an appropriate use of the support structures in case of sexual harassment.
Once the protocol is approved, we must continue to ensure that everyone is informed, for example, in the welcoming scheme for new workers.


Always try to play with other children and friends but always let your parents or carers know where you are.
Find the safest place to cross then stop on the pavement at the curb or the edge if there is no curb.
Have a good look around, make sure you can see if anything is coming and that drivers can see you.
Listen carefully, never cross whilst chatting, listening to your MP3 player or talking on your mobile phone.
Remember when it is safe to cross walk straight across the road, keep looking and listening while you cross.
Bullying can happen to anyone, anytime, anywhere. No one has the right to upset or hurt you.
Racism is treating someone differently or unfairly because they belong to a different race, religion or culture.
It is against the law to treat people differently or unfairly because of their race.


The solution is easy: www.istaprivate.com. You do not need to download anything, we don't ask for personal information or passwords. All you need to complete is go to the main page, enter the name of the profile you wish to see, then click View Profile. The profile picture of the instagram account can look, check if it's the right profile, or even, write the profile name again! If the instagram profile is correct, click View Private Photos & Videos. That instagram private profile is going to be revealed, enjoy!
That instagram video she posted... you wish to notice it, I know. All of us want to see a private instagram video in 2019, and the best way to take action is and is going to be istaprivate. Look at a private instagram video without that individual understanding that you did! We're not associated with Instagram and this site was designed for entertainment purposes only! It's not illegal to see private photos on Instagram and it's not harmful in virtually any way. The profile you wish to see might be accompanied by 1000s of people and they can see those pictures and videos any time. The only difference between you and those people is that you will be not actually following that profile. In the event that you carefully followed our short and simple tutorial, you will be able to see any profile that is private. Nobody will know what you are doing or what pictures you're downloading. This is the best part concerning this private Instagram method. So if you wish to see a private instagram video in 2019, go ahead, it's free!
Maybe you've seen on reddit and other websites method to see a private instagram story or to consider private instagram stories and profiles generally, but truth to be told, the majority of the methods show you to make a new account and hope for the best. We don't want that, nobody wants that, nobody has time and energy to wait to be accepted, you wish to look at a private picture on instagram or a private story and you want it with no that individual realize that you did. This is the bad thing about instagram stories in 2018, the one who posts them can see that you viewed her or his story. This is exactly why you employ istaprivate.com to see private instagram stories, because when you look at a private profile with our website, that you do not need to input your profile, you don even need to have a profile! View any private story for free, with no an account yourself!
Istaprivate.com is Private Instagram Viewer 2018 Updated that can help you look at any profile for free, 100% anonymous and without having to download anything. Our online private profile viewer is very easy to use, you can use it for unlimited accounts and it gets daily updates! Solutions when you wish to look at the pictures of a vintage friend, relative, or you simply need to know what your crush is doing. The difficulties arrive when that profile is private and you can't see any picture or Instagram story. What can you do in this case? We've managed this dilemma too, so we created an incredible method for you really to view private Instagram profiles and photos without after the profile you need to see. People may claim that that is impossible, but don't trust them. Viewing private profiles on Instagram is currently possible thanks to the website! There isn't to download any application and we won't look for all of your personal information or credit card, so don't worry. Here is the newest and safest method in 2018 to unlock private profiles on Instagram. Go ahead and view private instagram pictures in 2018 for free! Take realize that this is the only website where you are able to view private profiles on Instagram. Other websites or apps will make an effort to harm your device as well as steal your individual information. We advise you to prevent those websites at any cost. As I said before, that you do not need to download anything to use our Instagram profile viewer. All that's necessary is really a device (iPhone, Android, Computer, Laptop, Mac) with internet connection. You can also put it to use with any browser you've on your phone or computer.


1 January, 2013 26 January, 2013 RemyComments are off for this post.
Tired of being logged out from your administration session in your browser?
If you would like to change the default inactivity timeout for your sessions on the BlackBerry Administration Service (BAS) you can do this by modifying a setting in a configuration file.
You just have to edit <.. your installation directory ..> \BAS\jboss\ejb\server\default\deploy\jboss-web.deployer\conf\web.xml.
Just below Default Session Configuration in the web.xml file you can find the relevant lines.
To edit the timeout you change the line with session-timeout to the desired timeout in minutes. Configuring a value of 0 disables the timeout altogether.
Restart the BlackBerry Administration service after making the change.
Refer to KB20570 for more information.


The woman who is accused of drop-kicking a birthday cake at a local Kroger allegedly assaulted an ice cream store worker last summer.
Tricia Kortes has been charged with one count of assault and battery for an incident that took place at Ray's Ice Cream in Royal Oak on July 9, 2015.
Kortes is also facing disorderly conduct charges for allegedly drop-kicking a "Superman v. Batman" cake at a Kroger in West Bloomfield.
We're told she was was infuriated and took the cake to the front of the counter to "drop kick" it, according to police who have issued a misdemeanor citation to Kortes for "Tumultuous Conduct."
7 Action News reached Kortes on her cell phone in June . The woman who answered quickly claimed that we had the wrong phone number and hung up. We got a similar response when a man answered her husband's cell phone.


For the community in Tennessee, see Rogersville, Tennessee.
Location of Rogersville in Lauderdale County, Alabama.
Rogersville is located at 34°49′24″N 87°17′8″W / 34.82333°N 87.28556°W / 34.82333; -87.28556 (34.823444, -87.285693). According to the U.S. Census Bureau, the town has a total area of 3.1 square miles (8.0 km2), all land.
As of the census of 2000, there were 1,199 people, 536 households, and 361 families residing in the town. The population density was 389.6 people per square mile (150.3/km²). There were 604 housing units at an average density of 196.3 per square mile (75.7/km²). The racial makeup of the town was 93.08% White, 5.67% Black or African American, 0.17% Native American, 0.42% Asian, 0.08% from other races, and 0.58% from two or more races. 0.42% of the population were Hispanic or Latino of any race.
There were 536 households out of which 26.1% had children under the age of 18 living with them, 51.1% were married couples living together, 12.9% had a female householder with no husband present, and 32.5% were non-families. 31.2% of all households were made up of individuals and 19.0% had someone living alone who was 65 years of age or older. The average household size was 2.24 and the average family size was 2.79.
In the town, the population was spread out with 22.2% under the age of 18, 7.3% from 18 to 24, 24.7% from 25 to 44, 24.4% from 45 to 64, and 21.4% who were 65 years of age or older. The median age was 42 years. For every 100 females, there were 83.6 males. For every 100 females age 18 and over, there were 79.8 males.
The median income for a household in the town was $29,779, and the median income for a family was $37,639. Males had a median income of $30,852 versus $18,571 for females. The per capita income for the town was $16,435. About 10.2% of families and 14.3% of the population were below the poverty line, including 19.2% of those under age 18 and 15.7% of those age 65 or over.
As of the census of 2010, there were 1,257 people, 555 households, and 342 families residing in the town. The population density was 405.5 people per square mile (157.1/km²). There were 639 housing units at an average density of 206.1 per square mile (79.9/km²). The racial makeup of the town was 91.2% White, 6.1% Black or African American, 1.0% Native American, 0.6% Asian, 0.2% from other races, and 0.8% from two or more races. 1.7% of the population were Hispanic or Latino of any race.
There were 555 households out of which 24.7% had children under the age of 18 living with them, 44.3% were married couples living together, 13.2% had a female householder with no husband present, and 38.4% were non-families. 34.6% of all households were made up of individuals and 18.9% had someone living alone who was 65 years of age or older. The average household size was 2.26 and the average family size was 2.94.
In the town, the population was spread out with 22.1% under the age of 18, 7.5% from 18 to 24, 23.4% from 25 to 44, 25.1% from 45 to 64, and 22.0% who were 65 years of age or older. The median age was 43 years. For every 100 females, there were 87.9 males. For every 100 females age 18 and over, there were 92.6 males.
The median income for a household in the town was $32,727, and the median income for a family was $44,803. Males had a median income of $44,219 versus $38,229 for females. The per capita income for the town was $19,605. About 12.2% of families and 12.8% of the population were below the poverty line, including 12.5% of those under age 18 and 10.8% of those age 65 or over.
Norman Tellous Cooper - (1913 - 1994)- AL Sports Hall of Fame - Coached football at Howard College, Vanderbilt, Kansas and LSU - former professional football player - 1933 graduated from Lauderdale County High School, Rogersville, AL; 1937 graduated with B.S. from Howard College, Birmingham, AL; 1941 graduated with M.A. from Peabody College for Teachers, Nashville, TN - brother of Ken Cooper—Norman's nicknames "Archie", "Slim" , "Shorty"
^ "American FactFinder". United States Census Bureau. Retrieved 2015-07-30.
This page was last edited on 1 April 2019, at 23:54 (UTC).


text:
Evidence That Your Emoticon Addiction May Actually Make People Like You More
By Alice Robb
The brain is a funny organ. It controls consciousness and thought but, it turns out, it can also be tricked into responding to a few punctuation marks as if they were a human face: The brain perceives emoticons the same way as it does real faces, according to a new paper in the journal Social Neuroscience.
Australian psychologist Dr. Owen Churches and his coauthors at Flinders University and the University of South Australia recruited 28 participants and monitored their neural activity as they were presented with different stimuli: smiling emoticons, random punctuation marks, or pictures of smiling male or female faces. If the punctuation marks were rotated—for instance, (-: instead of :-)—the brain didn't respond the same way.
"Emoticons are a new form of language that we're producing," Churches told ABC News. "And to decode that language, we've produced a new pattern of brain activity."
Though they've only come into popular use in the last few years, emoticons have become the subject of a growing literature within computer science and psychology. Here's what some other researchers have discovered about emoticons and their impact on meaning.
Using smiley faces can make you seem more likeable
For a 2007 paper in the journal International Journal of Business Communication, Kristin Byron of Syracuse University recruited 300 college students and had them take a personality survey, which was graded for emotional stability. The researchers then had the students read a series of banal emails from strangers—high school students asking for information on the university or professors requesting copies of academic papers. Some of the emails included smiley faces, while others consisted only of text. The students then had to try to assess the personality of the high schoolers or professors, based only on their emails. It turned out that the students who were higher in emotional stability tended to rate the senders as more "likeable" if they used emoticons, while less stable students weren't swayed by the smiley faces.
elicits a stronger response than :-)
For a 2012 paper in the journal Cyberpsychology Behavior And Social Networking, Tina Ganster and her colleagues at Germany's University of Duisburg-Essen compared the psychological impact of sending and receiving old-school smilies— the kind made up of punctuation marks, ":-)", with pictographic smilies (). Ganster recruited 130 subjects online and had them read the transcript of an IM conversation that either included punctuation smilies, pictographic smilies or neither—and found that the pictographic smilies had the strongest impact on subjects' mood.
Emoticons make IMing more fun
In a 2008 paper in the journal Information & Management, researchers led by Albert Huang at the University of the Pacific looked at how using emoticons in IM conversations affected emotions in 216 people. "IM messages are less formal and individuality is enhanced by a large variety of emoticons that allow users to express emotions easily," wrote Huang. As they expected, they found a positive correlation between enjoyment and emoticon use:
An emoticon speeds up communication and eliminates some difficulty in expressing feeling using words; the process is easier, more interactive, and more fun. Also many emoticons are aesthetically pleasant and look amusing and many users apply emoticons sarcastically.
Smile, The Plank, Culture, emoticons, communication


Es gibt 65862 Eintr�ge in das G�stebuch.
My spouse and i ended up being really fortunate Ervin could round up his survey using the precious recommendations he grabbed from your very own web site.
It's not at all simplistic to just be giving for free hints that many others might have been trying to sell. Therefore we realize we need the writer to give thanks to because of that. The main illustrations you've made, the straightforward blog navigation, the relationships you can assist to engender - it is many wonderful, and it is aiding our son and the family feel that this idea is thrilling, which is really pressing.
Rathеr enlightening....lօoking forward to visiting again.
I appreciate the information on your websites.
Thanks a ton for sharing this ԝell put togetheг internet site.
Just what I was searching for, thank you for posting.
Namely, the ethereal, aerial, gaseous, liquid and solid states through binding of vibrational elements from the subtle towards the crude.
You realize a whole lot its almost tough to argue with you (not that I actually would want to...HaHa). You definitely put a brand new spin on a subject which has been discussed for decades. Excellent stuff, just excellent!
I don�t know why I can't join it. Is there anyone else having the same RSS problems? Anyone who knows the solution will you kindly respond? Thanx!!


text:
package com.link_intersystems.net.http.test;

import java.io.*;
import java.net.*;
import java.nio.charset.StandardCharsets;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.concurrent.CountDownLatch;
import java.util.stream.IntStream;

import static java.util.concurrent.TimeUnit.MILLISECONDS;

/**
 * @author René Link {@literal <rene.link@link-intersystems.com>}
 */
public class HttpMockServer implements Closeable, Runnable {

    private ReceivedRequest latestRequest;

    private Map<String, RequestMocking> requestMockings = new LinkedHashMap<>();

    public OngoingRequestMocking whenRequestPath(String requestPath) {
        return new OngoingRequestMocking(requestPath, requestMockings);
    }

    public static class ReceivedRequest {
        private String method;
        private String path;
        private String version;

        private Map<String, String> headers = new LinkedHashMap<>();
        private byte[] body;

        public ReceivedRequest(String method, String path, String version) {
            this.method = method;
            this.path = path;
            this.version = version;
        }

        public String getMethod() {
            return method;
        }

        public String getPath() {
            return path;
        }

        public String getVersion() {
            return version;
        }

        public void setBody(byte[] body) {
            this.body = body;
        }

        void addHeader(String name, String value) {
            headers.put(name, value);
        }

        public Map<String, String> getHeaders() {
            return headers;
        }

        public byte[] getBody() {
            return body;
        }
    }

    private ServerSocket serverSocket;
    private Thread socketHandlerThread;
    private CountDownLatch countDownLatch = new CountDownLatch(1);

    public void start() throws IOException {
        serverSocket = new ServerSocket(findAvailablePort());

        socketHandlerThread = new Thread(this);
        socketHandlerThread.start();
    }

    public int getPort() {
        if (serverSocket == null) {
            throw new IllegalStateException("HttpMockServer not started yet");
        }
        return serverSocket.getLocalPort();
    }

    public void stop() {
        if (socketHandlerThread != null) {
            socketHandlerThread.interrupt();
            Thread.yield();
            try {
                countDownLatch.await(250, MILLISECONDS);
            } catch (InterruptedException e) {
                socketHandlerThread.stop();
            }
        }
    }

    @Override
    public void run() {
        try {
            Socket accept = serverSocket.accept();
            OutputStream out = accept.getOutputStream();
            InputStream in = accept.getInputStream();

            latestRequest = receiveRequest(in);
            writeResponse(out, latestRequest);
            out.flush();
            out.close();
            in.close();
        } catch (IOException e) {
            throw new IllegalStateException(e);
        } finally {
            countDownLatch.countDown();
        }
    }

    private void writeResponse(OutputStream out, ReceivedRequest receivedRequest) throws IOException {
        RequestMocking requestMocking = requestMockings.get(receivedRequest.getPath());
        if (requestMocking == null) {
            writeResponse(out, 400);
        } else {
            writeResponse(out, requestMocking);
        }
    }

    private void writeResponse(OutputStream out, int responseCode) throws IOException {
        out.write("HTTP/1.1".getBytes(StandardCharsets.UTF_8));
        out.write(" ".getBytes(StandardCharsets.UTF_8));
        out.write(Integer.toString(responseCode).getBytes(StandardCharsets.UTF_8));
        out.write(" OK\n".getBytes(StandardCharsets.UTF_8));
    }

    private void writeResponse(OutputStream out, RequestMocking requestMocking) throws IOException {
        int responseCode = requestMocking.getResponseCode();
        writeResponse(out, responseCode);

        byte[] body = requestMocking.getBody();
        if (body != null) {
            out.write("\n".getBytes(StandardCharsets.UTF_8));
            out.write(body);
        }
    }

    private ReceivedRequest receiveRequest(InputStream in) throws IOException {
        String request = readLine(in);
        String[] parts = request.split("\\s");
        ReceivedRequest receivedRequest = new ReceivedRequest(parts[0], parts[1], parts[2]);
        readHeaders(in, receivedRequest);
        readBody(in, receivedRequest);
        return receivedRequest;
    }

    private void readBody(InputStream in, ReceivedRequest receivedRequest) throws IOException {
        int contentLength = Integer.MAX_VALUE;

        Map<String, String> headers = receivedRequest.getHeaders();
        String contentLengthValue = headers.get("Content-Length");
        if (contentLengthValue != null) {
            contentLength = Integer.valueOf(contentLengthValue);
        }

        ByteArrayOutputStream bout = new ByteArrayOutputStream();
        int readCount = 0;

        int read;
        while (readCount < contentLength) {
            int available = in.available();
            if (available < 1) {
                break;
            }
            read = in.read();
            if (read == -1) {
                break;
            }
            readCount++;
            bout.write(read);
        }

        receivedRequest.setBody(bout.toByteArray());
    }

    private String readLine(InputStream in) throws IOException {
        StringBuilder sb = new StringBuilder();

        int read;

        while ((read = in.read()) != -1) {
            char c = (char) read;
            if (c == '\r') {
                continue;
            }
            if (c == '\n') {
                break;
            }
            sb.append(c);
        }

        return sb.toString();
    }

    private void readHeaders(InputStream in, ReceivedRequest receivedRequest) throws IOException {
        String line;

        while ((line = readLine(in)) != null) {
            if (line.isEmpty()) {
                break;
            }

            String[] header = line.split(": ");
            receivedRequest.addHeader(header[0], header[1]);
        }
    }

    @Override
    public void close() throws IOException {
        try {
            socketHandlerThread.stop();
        } catch (Exception e) {
        }
        serverSocket.close();
    }

    public static int findAvailablePort() {
        return IntStream.range(50000, 65535).filter(HttpMockServer::available).findFirst().orElseThrow(() -> new IllegalStateException("no port available"));
    }

    public static boolean available(int port) {
        ServerSocket ss = null;
        DatagramSocket ds = null;
        try {
            ss = new ServerSocket(port);
            ss.setReuseAddress(true);
            ds = new DatagramSocket(port);
            ds.setReuseAddress(true);
            return true;
        } catch (IOException e) {
        } finally {
            if (ds != null) {
                ds.close();
            }

            if (ss != null) {
                try {
                    ss.close();
                } catch (IOException e) {
                    /* should not be thrown */
                }
            }
        }

        return false;
    }

    public URL getBaseUrl() {
        try {
            return new URL("http://localhost:" + serverSocket.getLocalPort());
        } catch (MalformedURLException e) {
            throw new IllegalStateException(e);
        }
    }

    public ReceivedRequest getLatestRequest() {
        return latestRequest;
    }

}

redpajama_set_name: RedPajamaGithub


Work has begun on the project located at 278 Bunnerong Road Hillsdale. Meso Solutions are currently inserting sheet piles and bulk excavation works are set to take place shortly.
Great design by Nick Krikis and the architects at KTA. Bradley and the team at Jebeko have got things well and truly under control. ACSES Engineers are proud to be involved in this terrific development and honoured to be the Project Structural Engineer.
But you're forming up Level 1?
Yes, but you're in the process of forming the deck for Level 1?
Suddenly we found ourselves needing to support an additional two levels of a structure that was not designed to carry an additional two levels.
This was going to be interesting, to say the least.
Using Finite Element Modelling (FEM), we were able to highlight weak areas of the existing structure that needed reinforcement as well as confirm areas that had sufficient strength to resist the new loads.
From the foundations to the supporting columns and walls to the suspended slabs, ACSES Engineers developed a total solution that enabled the extra levels to be added, while still maintaining the overall safety of the structure.
Below are some images of the carbon fibre reinforcement that were retrofitted to the soffit of the transfer slab. A terrific technology that gave us the extra strength we needed. Other strengthening works included widening and underpinning specific foundations as well as 'beefing up' specific columns and walls.
Shane and his team at First Class Building did an amazing job pulling it all together and the end result is a building that is safe, two storeys taller, and just about to be handed over to the various new owners.
ACSES Engineers are honoured to have been appointed as the Project Structural Engineer for the project located at 43-45 Beane Street Gosford. We are incredibly excited to be part of the design team that will help deliver this exceptional development on the Central Coast.
For more information please contact us anytime.
An absolute joy to have been involved in this project. The client knew exactly what he wanted (and what he didn't want). The builder was a genius. The architect – PL Projects – was focussed, firm, yet fair. And the team at ACSES Engineers just loved working on it.
ACSES Engineers designed both the bulk excavation solution and the various building structures. Utilising our finite element modelling (FEM) capabilities, we were able to eliminate the need for shoring works by taking advantage of the geotechnical parameters of the site. The FEM analysis verified that our vertical cut solution could safely be carried out without adversely impacting the neighbouring structures.
Again using our FEM expertise, ACSES Engineers designed the superstructure as a concrete framed solution made up of off-form reinforced concrete columns, walls and stairs. To achieve various parameters set by the architect, we designed the suspended slabs as post-tensioned solutions.
The level of detailing required for this master piece was 'unbelievable'. From the concrete pool rising out of the ground with the infinity edges. To 'the box' suspended two levels over the Lower Ground Floor and cantilevering out 5m past its supports. To the off-form concrete walls almost 8m tall that needed to be poured in one go. To the 'zig-zag' stairs that could only be 100mm thick or the cantilevered stairs that could be no thicker than 110mm – the end results make all that work well worth the effort.
ACSES Engineers are incredibly proud to have been part of the Design Team and honoured to have been the Project Structural Engineer for this development.
The structural frame at 1 Bede Street Strathfield has been completed, and the builder has done an amazing job in an incredibly short time.
ACSES Engineers designed both the shoring and bulk excavation solutions, as well as the building superstructure.
Utilising our finite element modelling capabilities, we were able to minimise the shoring works required to support the two level excavation face. Our modelling took advantage of the geotechnical parameters present at the site and ACSES Engineers were able to design a cantilevered pile wall solution with shotcrete infills.
Again using our FEM expertise, ACSES Engineers designed the superstructures as a concrete framed solutions made up of columns and walls using the Rediwall product wherever possible. This eliminated the need for traditional vertical formwork and resulted in further savings in time and cost. The slabs were all designed as conventionally reinforced slab solutions and the individual units were partitioned using lightweight non-load bearing walls made from fibre cement (Hebel) and framed Gyproc.
The project was architecturally designed by Ghazi Al Ali Architects and ACSES Engineers are proud to be part of the Design Team and honoured to be the Project Structural Engineer for this development.
Works are well underway at 1 Villawood Place Villawood and the team from Grandviewco have gotten off to an excellent start.
Utilising our finite element modelling capabilities, we were able to minimise the shoring works required to support the three level excavation face. All the basement levels were designed as conventionally reinforced slabs, while Ground Floor and Level 1 were designed as post tensioned solutions. All these levels were supported on traditionally formed and reinforced concrete walls and columns. Working with our design partners from Ultrafloor and AFS Systems, ACSES Engineers designed all the upper levels as a load bearing wall – concrete framed solution. From Level 1 upwards, AFS Logic Wall was used to support the Ultrafloor slab system, from Level 2 onwards. This construction process was adopted by Grandviewco to eliminate the need for traditional formwork as much as possible. To partition the units, lightweight non-load bearing walls made from fibre cement (Hebel) and framed Gyproc were used.
The project was architecturally designed by Tony Owen Partners and ACSES Engineers are proud to be part of the Design Team and honoured to be the Project Structural Engineer for this development.
Here is the 3D Perspective.
ACSES Engineers designed a shoring and bulk excavation solution that extended four levels below 5-7 Telegraph Road Pymble as well as the superstructures.
Utilising our finite element modelling capabilities, we significantly reduced the required shoring works at 5-7 Telegraph Road Pymble by designing an anchored pile and shotcrete wall to support the excavated face above the Class IV material, and then vertically cut the rock strata down to the bulk excavation level.
We also designed the crane base, without requiring piles, to bear directly on the Class IV material located higher than the final bulk excavation level. This shoring and bulk excavation solution designed by ACSES Engineers resulted in massive cost and time savings for South West Builders, who must be congratulated on successfully carrying out the detailed works.
Again using our FEM expertise, ACSES Engineers designed the superstructures as a concrete framed solutions made up of columns and walls using the Dincel Wall product wherever possible. This eliminated the need for traditional vertical formwork and resulted in further savings in time and cost for the builder. To partition the units, lightweight non-load bearing walls made from fibre cement (Hebel) and framed Gyproc were used.
The project was architecturally designed by Aleksander Design Group and ACSES Engineers are proud to be part of the Design Team and honoured to be the Project Structural Engineer for this development.
Thanks for all the fantastic feedback.
Shoring and bulk excavation works at 2A-8 Burwood Road Burwood have been completed and foundation works have begun. It's a tight, detailed site, but the COPLEX Constructions team have everything running smoothly like a well-oiled machine. ACSES Engineers are proud to be part of the Design Team and honoured to be the Project Structural Engineer.
Sheet Piles are in and bulk excavation works at 75-83 Second Avenue Campsie have now begun. Sam and the team at SWEDA have got it all under control and congratulations to Balintore Developments on yet another great project. ACSES Engineers are proud to be part of the Design Team and honoured to be the Project Structural Engineer for this development.
Here are the 3D Perspectives.
Will Fossil Fuels Run Out?


LIMITED EDITIONS: ARE YOU INTERESTED IN DOING BUSINESS IN MEKONG & BANGLADESH?
The Swedish Chamber of Commerce India in collaboration with Business Sweden cordially invites you to two interactive sessions with Mr. Johan Frisell, Ambassador of Sweden to Bangladesh, and Mr. Peder Cederholm, Trade Commissioner to Thailand, to understand and explore the business opportunities in Mekong and Bangladesh.
For both events, please RSVP the latest by October 10 to dolly.berry@swedishchamber.in, +91 11 46067123/4.


text:
The Hidden Gem Behind Flawless Diamonds: The Story of Nicholas Flathau
by Richard Thompson July 9, 2020 0662
Success doesn't come easy to most of us in life. However, when perseverance does make success knock on your door, all you need to do is learn how to preserve it. That is exactly what Nicholas Flathau has done over the last ten years to reach where he is today. The young entrepreneur was under pressure from a young age as he started living on his own when he was just seventeen. This pressure kept him going for a long time, and finally, like your everyday carbon, the man, under immense pressure turned into a gem of a person himself. Owning Flawless Diamonds Co. blessed Nicholas with enough expertise in the jewelry making and marketing domain, and eventually began his entrepreneurial rise.
Nicholas Flathau's story with Flawless Diamonds is a very interesting one. Early in his life, Nicholas didn't have a lot planned and decided to get his degree and be an anesthesiologist. Soon enough, the young varsity football star inside the man rose again and his entrepreneurial spirit flared to life. He started his entrepreneurial career by opening an AT&T company but soon decided to switch directions in life. The young boy, adept at the craftsmanship, looked into the specifics of crafting jewelry as a hobby. His skills were undoubtedly great and this was ascertained by a friend who called him to show interest in buying his fine specimens of craftsmanship. Right then, Nicholas knew that he could build on this and create a jewelry brand of his own.
Today, Flawless Diamonds Co. is a renowned name in the States and is growing every day. Based in Chicagoland, the company deals with global customers and extends its services to as many jewelry lovers as possible. The rise was meteoric and Nicholas went ahead to earn a million in just a year, and at the young age of 26, Nicholas Flathau was a renowned millionaire.
The reason Flawless Diamonds shines so bright in American eyes is that the company understands the needs of its customers. Today, Flawless Diamonds brings the best custom made jewelry to its clients by entirely cutting out the middleman. By doing this, Nicholas and his team can offer competitive rates for clients, and these are comparatively better than the ones offered by the company's competitors.
Nicholas's handcrafting journey took a major turn when he met a gentleman in Texas in 2019, where he learned a lot about the art of jewelry making. Today, that gentleman is Nicholas's mentor and this promises Flawless Diamonds Co. a bright future.
In addition to exceptional quality and high-grade work, Flawless Diamonds Co. has also used the social media game to its benefit. With his immense social media knowledge and a one in a million company behind him, Nicholas has generated not just money, but love and support for his brand. Flawless Diamonds Co. is straight forward with its clients and maintains a respectful and professional relationship with them admirable transparency. The company keeps getting better every day, especially after having established a source of manufacturing very close to its HQ. Today, Nicholas guides Flawless Diamonds and two more companies to unprecedented success by striving and working hard every day.
CEO of Nova Mentis, Dr. Aylia Mohammadi Exploring the Gut-Brain Connection in Depression Related to Obesity and Diabetes
Why You Need a Digital Strategy To Succeed in These Unstable Economic Times with Tiana Burse
After being a professional journalist for 5 years and understanding the ups and downs of health care sector all over the world, Richard shifted his focus to the digital world. Today, he works as a contributor for eNEWS 2.0 with a knack for covering health news in the best possible format.
Increasing Laboratories and Chemical Industries are Driving the Global Demand for Carboys
Richard Thompson October 11, 2019


text:
A new company, Neo-Bionica, designs and manufactures prototypes of implantable medical devices. It will speed up their development in Australia.
Neo-Bionica builds electronics for medical devices and packages them so they can be implanted safely. It aims to reduce the time between initial concept and clinical application.
The company, which opened in July 2021, is a joint venture between the Bionics Institute and the University of Melbourne. Researchers at both institutions design devices to understand, diagnose and treat medical conditions. These conditions include epilepsy, Parkinson's disease, hearing loss, stroke and diabetes. Only a small number of high-quality devices are needed for early clinical studies. But no manufacturing facilities in Australia could make this type of device. Ordering from international suppliers is costly and can take months.
Neo-Bionica fills this gap for academic and industry researchers. The company will collaborate with centres such as the Melbourne Centre for Nanofabrication.
Neo-Bionica staff can also help clients navigate the approval process for first-in-human studies of medical devices. They are experts in Australia's ethical and regulatory requirements.
The Bionics Institute and the University of Melbourne each invested AU$2.5 million in Neo-Bionica. The company is part of the capabilities available to the Aikenhead Centre for Medical Discovery, of which both institutions are partners. Neo-Bionica is based at St Vincent's Hospital Melbourne.
Clients work with Neo-Bionica on a fee-for-service basis. Using funding from the Higher Education State Investment Fund, the company has created a voucher program. It will subsidise access costs and seed projects that have the potential to become ongoing collaborations. The company will issue calls for applications several times a year.
Image: NIH/Flickr (CC BY-NC 2.0)
Detecting epileptic seizures with an implantable device
Epi-Minder has raised $A10 million to develop a device for seizure detection and prediction, which is now undergoing a Phase I clinical trial.
Brain implants to predict and prevent seizures
A Carbon Cybernetics implant that stimulates and records information from single brain cells is being developed to help manage epilepsy.
Melbourne Biomedical Precinct
Explore the Melbourne Biomedical Precinct, where health researchers and industry partners collaborate to deliver lifesaving drugs and treatments.


In snow areas, several inches of snow can accumulate on bridge structures forming a bank, or an iced bank in lower temperatures. These snow banks or iced banks may cause traffic accidents when they fall onto the bridge roadway. HIREC paint applied to bridge frames is an effective countermeasure against the formation of such snow banks.
In winter, snow fallen onto bridges sometimes forms banks or icicles on bridge frames. Wet-snow easily forms thick snow banks on the bridge frames, and may become icy-hard banks during the night. These banks or icicles may cause car damage or traffic accidents when they fall on to the bride roadway. Against this problem, some countermeasures have been applied, including a snow melting method using heaters. Such countermeasures, however, may have some challenges such as increasing maintenance cost.
Painting HIREC onto bridge construction may reduce the snow accretion problems.
Note *: The performance of the snow accretion using HIREC may vary according to the design conditions and designs of the bridge beams and snow separators, etc. It may include the face angle of the beam surface, shape, size and positions of the separators, etc. Prior investigation is crucial before applying the HIREC.


OPEC, which uses secondary sources to monitor its output, said production from all 12 members fell to 29.633 million b/d in November from 29.827 million b/d in October, a drop of 194,000 b/d.
The estimates showed Libyan output falling to 371,000 b/d in November from 512,000 b/d in October. Nigerian production fell to 1.821 million b/d from 1.925 million b/d. Saudi Arabia scaled back output to 9.626 million b/d in November from 9.714 million b/d in October and 10.037 million b/d in September. Algeria, Kuwait and the UAE had smaller falls.
The production of oil has taken a swift rise though amidst the downfall in production within several other members of OPEC. Iraqi production climbed to 3.107 million b/d in November from 2.915 million b/d in October.
A table of output figures has been also published by OPEC and it has been submitted directly to the country members. These can differ markedly from the estimates from secondary sources. Iran, for example, told OPEC it had pumped an average 3.3 million b/d in November, some 600,000 b/d more than the secondary source figure of 2.716 million b/d.
Venezuela's direct submission of 2.854 million b/d was 490,000 b/d higher than the 2.364 million b/d derived from secondary sources. Libya's direct submission of 226,000 b/d was considerably below the secondary source estimate, as was Nigeria's -- 1.672 million b/d compared with 1.821 million b/d.
Saudi Arabia has stated OPEC that it pumped an average 9.746 million b/d, little changed from the 9.753 million b/d it submitted for October but 120,000 b/d higher than the secondary source estimate.
According to the estimations by the experts in OPEC, average demand for its crude next year, plus movements in and out of stocks, at 29.57 million b/d, unchanged from its previous monthly report. But it expected the call on OPEC to fall from 30.4 million b/d in the fourth quarter of this year to 29.16 million b/d in the first quarter of 2014.
In spite of the low demand project in the coming year, especially in 2014, OPEC ministers meeting in Vienna last week said they would maintain the current 30 million b/d overall output ceiling until their next meeting in June, in the interest of maintaining equilibrium in world oil markets.
According to the monthly report, "Member countries re-confirmed their readiness to promptly respond to unforeseen developments that could have an adverse impact on an orderly and balanced oil market."
It has been also speculated by the experts that the potential development will alter OPEC's view of the market balance include Libyan output recovering to around 1.4-1.5 million b/d and a full lifting of sanctions on Iran.
There is skepticism about a full recovery in Libyan output given the continuing unrest in the country. In case of Iran, analysts do not expect high rise in production and exports until at least the second half of next year.
The clock has yet to start ticking officially on last month's ground-breaking six-month agreement that, in return for concessions on the nuclear program, will give Iran some sanctions relief but keep core oil and banking sanctions in place while negotiators work toward a comprehensive agreement that would see sanctions lifted.


text:
Community News & Infos Exhibition
Rebecca Unz: Head Studies
This year's August Sander Award goes to the young Austrian artist Rebecca Unz (* 2000) for her portrait series "Sensibelchen," 2018-2019. The digital photographs focus on the faces of young men and women from the musical environment of hip hop and rap. They are head studies, modeled by a special light guide that assigns each portrait its place in the series of images: impressively, the series progresses image by image like a musical scale from a warm to a cold hue, thus reinforcing the range of variation in the moments and sensitivities observed by Rebecca Unz.
In this exhibition, conceived on the occasion of the award ceremony, other works will be shown in addition to the award-winning series, which deal with the various facets of portraiture, up to and including an abstract analysis of form. For Rebecca Unz, the genre holds a multitude of possibilities for artistically reflecting on a person as a complex individual. Here, the person is also understood as a projection surface, and the use of various creative means and their specific possibilities each emphasize their own aspects. In her diploma thesis at the Ortweinschule in Graz, where she attended the class for Photography & Multimedia Art, she used the example of the rapper "Eloquent" to explore multi-layered imaginary worlds about his person and image.
Rebecca Unz not only uses the medium of photography, but also drawing, executed in pencil or in digital implementation, is one of her artistic means, with whose potential she experiments - whether in a naturalistic or in an abstract manner - with confidence. She deliberately blurs the boundaries of recognizability of the techniques she employs, challenging the viewer's attentive perception.
The August Sander Prize was offered for the first time in 2018 by the Photographische Sammlung/SK Stiftung Kultur, donated by Ulla Bartenbach and Prof. Dr. Kurt Bartenbach. The prize is associated with the idea of promoting young contemporary artistic approaches in response to factual-conceptual photography, with a particular focus on the human portrait.
Rebecca Unz: Benni / Sensibelchen, 24.03.2019, © Rebecca Unz
till: 8. August 2021
Die Photographische Sammlung/SK Stiftung Kultur
Im Mediapark 7 |50670 Köln
← Previous To mark the 80th birthday of photographer Timm Rautert, the Museum Folkwang is hosting a comprehensive retrospective
Helmut Newton Foundation: America 1970s/80s Next →
India Art Fair announces 2022 edition in cooperation with Chennai Photo Biennale among others
India Art Fair, the leading platform to discover and celebrate modern and contemporary art from South Asia, announces the full list of participating galleries and programme highlights for its forthcoming 13th edition. Taking place from
Portraits of an LGBTI+ Generation
Portraits of an LGBTI+ Generation, a photography and video series featuring members of the dance clubs photographed and interviewed in places that are meaningful to them. Since 2019, LGBTI+ Elders Social Dance Clubs, part of
A great journal with more than 200 articles about Vietnamese Contemporary Photography and more
Matca is an independent initiative dedicated to photography as a form of visual art. Founded in 2016 in Vietnam, Matca has evolved across many platforms, including but not limited to an online journal, a physical
Uwa Iduozee: Blind Spot(s)
The exhibition showcases photographs taken by Uwa Iduozee in the United States in 2020. The year was marked, on the one hand, by the crisis caused by the pandemic, and, on the other hand, by
Diemut von Funck: Let the sunshine in…
Summer, sun, light – like a breeze, Diemut von Funck's fine art photographs invite us to dive in, awakening an atmosphere of lightness and transparency, tracing "what could be", far from our reality. Moments of
Martin Parr: We Love Sports
CAMERA – Centro Italiano per la fotografia announces 'Martin Parr. We Love Sports', the great show that will feature an outright legend of contemporary photography. The exhibition will be staged in collaboration with Lavazza Group, institutional


After a disappointment like this, I think I would be motivated to go back and do it better the next year. If nothing else to prove it to myself that I can.
Don't beat yourself up too badly VT is a tough course with a lot of light rollers and a couple tough hills that can really take a surprising toll. I think you are right that a shorter cycle may have hurt you a bit. Overall I think your training was really quite good. The three small things I noticed were that, one, your last specific block was very close to the marathon. I wouldn't do one inside the last three weeks. I know you had a short cycle but I would suggest one of canova's 30k to 40k marathon specific progression or alternating pace workouts in the the last few weeks instead. Second your training loop for marathon specific stuff is very flat, which in terms of overall fitness is IDEAL, but in getting ready for VT where there is a ton of light rollers the first 20 milers this may have left you getting beat up more than you expected. If your regular training runs are very hilly ignore this one because that should be enough to counter the flat workouts. Finally a few of your specific workouts drifted towards being more half marathon than marathon. For example doing the reps at sub 5:00 pace in the PM session of specific blocks. In a shorter cycle that could have an impact. Anyway take it for what it is worth. Keep up the good work and keep tweaking things and you'll really nail one sooner or later.
Thanks, Nate! Makes sense…appreciate you taking the time to respond.
I'm not a nutritionist or anything close to a health nut and don't try to act like one. With that being said, I've got some practice on fueling for 2+ hour races so I'll give my recommendation. Lately, both in training and racing, I've been eating those Powerbar energy blasts. I think they go down easier without water so they are more convenient than a gel. They are ~25 calories per gummy. I probably had 10 between miles 8-22 at Boston plus a gel somewhere in there. Gu chomps are the same sort of stuff. I'm a coffee addict so I tell myself I can handle the caffeinated versions but it is a fine line between getting a little extra boost and looking for a porta-pot stop!
Got any plans for the fall? Your workouts across the board are significantly better than mine but the shorter the distance, the bigger that gap is. I might try your hand in a 1:04:59 half for a goal race.. then move back to a marathon the next season. But hey, life's too short to not do what you want. Keep grinding at 26.2 if that's what your heart desires!
I speak for alot of folks when I say that reading your training blog is humbling and beneficial for us!
It happens – you'll be that much better next time out!
As a person who has had numerous, numerous "bad days" (see: just about every Ironman I've entered), my two cents: don't overthink it.
Bad days happen, like you said. Have a bad 5k, do another next Sunday. Bad days are magnified in the marathon, because in a given year you have two, maybe three in you. Particularly when you're trying to compete at your level. To Nate's point, you put in a seriously hard workout day recently. 29 miles in two runs, that may have been your race right there. Separately you were able to recover and battle on. Even 20, 22 mile training runs – they're not the marathon.
There's something different about the race week prep, the travel, the sleeping in a hotel, the food, the morning of, the pinning of a bib – all that stuff takes a different toll than just going out and cranking on your own.
Like you, I have a tendency to not eat during long training efforts. On a hundred mile ride I may only take in 500-800 calories, which I can survive the ride on, but it's the recovering afterwards. I'm certainly not saying the Canova-esque glycogen depletion runs are wrong, but I think it's important to consider the effect it has on YOU. What works for Graham doesn't work for me, and vice versa. If you need to consume calories during marathons, there's no shame in that game. If you need it to come from liquid instead of solid, do it.
Just my outside perspective, but you do a LOT of your training on the same routes, week in and week out. Everything is very controlled, you seem driven by what the data is telling you. What about the qualitative data? How are you feeling? I think some of the guys that use the NCR Trail for their long runs EVERY week are making a mistake, because it doesn't have the feel of the road. And unless they're racing on NCR, it's too controlled.
Again, ask 10 different people about training and you'll get 10 different answers. I'm not saying I have the answer, because I can't even run one mile as fast as your marathon pace right now. But, I do know that when things are stagnant, or results aren't what I'm looking for, something has to change. We invest too much time and energy into it to not give ourselves the best chance at success. Change things up! Do workouts in some different places, challenge yourself with different workouts. Race a couple of low key events, shorter stuff – whatever.
To quote Eminem, you've got the beat, all you need is the words.


text:
\section{Method: Low-Rank + Sparse Tensor Decomposition}

At a high level, our algorithm takes a trained neural network $f_t$ as input and then outputs a low-rank plus sparse tensor-compressed neural network $f_s$ by {\bf Phase 1:} low-rank plus sparse tensor decomposition and {\bf Phase 2:} iterative pruning and fine-tuning. This is outlined in Figure \ref{fig: algo flow}. The technical description is given in Algorithm \ref{alg: outer}.

\begin{algorithm}
\caption{Low-Rank + Sparse Compression and Fine-Tuning}
\begin{algorithmic}
\label{alg: outer}
\REQUIRE Trained NN $f_t$, tensor ranks $\{\mat{r}_i\}_{i=1}^L$, sparsity target $s_f$, number of epochs $e$
\FOR{$i$ in $[1,2,3,\dots]$}
\STATE Compress layer $i$ weight $\ten{A}_i$ with target rank $\mat{r}_i$ using \eqref{eq: tensor residual minimization} or \eqref{eq: tensor masking residual minimization}.
\ENDFOR
\FOR{$t$ in $[1,2,3,\dots,e]$}
\STATE Global magnitude pruning on sparse residuals $\{\ten{S}_l\}_{i=1}^L$ with target sparsity $s_t$ from \eqref{eq: pruning}.
\STATE Train for one epoch using loss function $\mathcal{L}$ from \eqref{eq: KD loss}.
\ENDFOR
\end{algorithmic}
\end{algorithm}

\subsection{Reconstruction}

We consider two approaches to represent the tensor $\ten{A}$. We assume that the reconstruction $\ten{A}=h(\ten{L},\ten{S})$ combines the low-rank and sparse components and is performed before any forward propagation. Depending on the specific form of $h$, which will be specified below, direct contraction of the layer activation with $\ten{S}$ and $\ten{L}$ respectively may be more efficient.

\paragraph{Additive:} Given weight tensor $\ten{A}$ we reconstruct
\begin{equation}
    \label{eq: additive reconstruction}
    \ten{A} = \ten{L}+\ten{M}\odot\ten{S}
\end{equation}
where $\ten{L}$ is a low-rank tensor whose weights are maintained in low-rank format and $\ten{S}$ is a sparse residual whose weights will be pruned. The binary mask $\ten{M}$ controls the sparsity of $\ten{S}$. After decomposition, each forward pass of the neural network will reconstruct the weight $\ten{A}\leftarrow \ten{L}+\ten{M}\odot\ten{S}$, permitting backpropagation directly to the low-rank tensor factors comprising $\ten{L}$ and the sparse unpruned entries of $\ten{S}$. For example, when the low-rank tensor $\ten{L}$ is in CP format we store the factor matrices $\{\mat{U}^{(i)}\}$ and not the full tensor $\ten{L}$.

\paragraph{Masking:} We identify a drawback of the reconstruction specified in \eqref{eq: additive reconstruction}. First, representational capacity is wasted when the low-rank tensor $\ten{L}$ and the sparse tensor $\ten{S}$ both contribute to an entry in $\ten{A}$. Each nonzero entry in $\ten{S}$ has no constraint, and so the given entry could be represented by $\ten{S}$ alone. This would permit the low-rank factors of $\ten{L}$ to fit other entries in $\ten{A}$ where the sparse pruned $\ten{S}$ is zero and cannot contribute. Therefore we propose an alternate ``masking" reconstruction method in which we reconstruct $\ten{A}$ as
\begin{equation}
    \label{eq: masking reconstruction}
    \ten{A} = (1-\ten{M})\odot\ten{L}+\ten{M}\odot\ten{S}.
\end{equation}
Under this reconstruction each entry of $\ten{A}$ is represented by either $\ten{L}$ or $\ten{S}$, but not both.

\subsection{Initialization}

The sparse weight $\ten{S}$ will be iteratively pruned. Therefore our goal is to capture any possible coarse-grained structure in $\ten{A}$ using $\ten{L}$, and retain fine structure in $\ten{S}$ that will be well-suited to sparse pruning. We describe two methods of decomposition. The tensor $\ten{A}$ may be reshaped before performing factorization.

\paragraph{Residual:}
A simple approach is to obtain $\ten{L}$ by minimizing the residual to $\ten{A}$. We focus on the Tensor-Train format for notational convenience and use
\begin{equation}
    \label{eq: reconstruction}
    \ten{L}=\llbracket \ten{G}^{(1)},\dots,\ten{G}^{(d)}\rrbracket
\end{equation}
to represent the reconstruction of the low-rank TT-format tensor $\ten{L}$ from its factors $\{\ten{G}^{(i)}\}_{i=1}^d$.
\begin{equation}
    \label{eq: tensor residual minimization}
    \begin{split}
    \ten{G}^{(1)},\dots,\ten{G}^{(d)} &= \argmin_{\ten{G}^{(1)},\dots,\ten{G}^{(d)}}\left\|\llbracket \ten{G}^{(1)},\dots,\ten{G}^{(d)}\rrbracket-\ten{A}\right\|_2\\
    \ten{L} &= \llbracket\ten{G}^{(1)},\dots,\ten{G}^{(d)}\rrbracket\\
    \ten{S} &= \ten{A}-\ten{L}.
    \end{split}
\end{equation}
We obtain an approximate solution to the minimization problem through standard decomposition methods implemented in Tensorly \citep{kossaifi2016tensorly}. For the TT and Tucker formats the decomposition to obtain the factors of $\ten{L}$ requires $d-1$ tensor unfoldings and singular value decompositions. The CP format uses alternating least squares (ALS) iterations. In general, the decomposition time for the networks we consider requires less than five minutes and is inconsequential compared to the resources used during the fine-tuning phase which takes approximately 12hrs.

The decomposition given in \eqref{eq: tensor residual minimization} provides an exact reconstruction of the the original tensor $\ten{A}$ before the sparse component is pruned at the first iteration. We note that this naive decomposition has one clear weakness: it does not take into account the fact that many entries of $\ten{S}$ will be pruned. A natural approach would be to consider robust PCA or other decomposition methods that explicitly target a low-rank plus sparse decomposition. We attempted this approach but did not observe any clear gains. We use the residual initialization approach with the additive approach.

\paragraph{Masking:} In order to directly encourage the low-rank factors to capture only the low-magnitude entries (which will be pruned) we consider an alternate decomposition. We mask the top-$K\%$ of entries using the operator $P_\Omega$:
\begin{equation}
    \label{eq: masking}
    P_{\Omega}(\ten{A})_{i_1\dots i_d} =
    \begin{cases}
    0 \hspace{0.3in}\text{if }\hspace{0.1in} a_{ijk}\in TopK(\ten{A})\\
    a_{ijk} \hspace{0.1in}\text{  otherwise}
    \end{cases}
\end{equation}
Then we obtain our low-rank decomposition $\left(\ten{S},\ten{L}\right)$ by minimizing the reconstruction loss and considering only the bottom $K\%$ of entries in $\ten{A}$.
\begin{equation}
    \label{eq: tensor masking residual minimization}
    \begin{split}
    \ten{G}^{(1)},\dots,\ten{G}^{(d)} &= \argmin_{\ten{G}^{(1)},\dots,\ten{G}^{(d)}}\left\|P_{\Omega}\left(\llbracket \ten{G}^{(1)},\dots,\ten{G}^{(d)}\rrbracket-\ten{A}\right)\right\|_2\\
    \ten{L} &= \llbracket\ten{G}^{(1)},\dots,\ten{G}^{(d)}\rrbracket\\
    \ten{S} &= \ten{A}-\ten{L}.
    \end{split}
\end{equation}
This decomposition differs from the one obtained in \eqref{eq: tensor residual minimization} in that the tensor decomposition does not fit any of the high-magnitude entries which will remain unpruned. We use this initialization approach with the ``masking" reconstruction so that the initialization ignores high-magnitude entries that will remain un-pruned.
\section{Appendix}

\section{Introduction}\label{sec:intro}

Using machine learning to enable multiple applications on edge devices requires multiple task-specific persistent models \citep{yang2020co}. These models are used for tasks ranging from computer vision \citep{howard2019searching} to automatic speech recognition \citep{wang2021noisy}. The trend towards multiple applications and multiple models is constrained by the fact that off-chip memory reads incur high latency and power costs \citep{sze2017hardware}. Therefore, in this paper we target memory cost reduction. In this area low-rank tensor compression is a popular approach \citep{garipov2016ultimate,novikov2015tensorizing,lebedev2014speeding} that can achieve orders of magnitude compression, but can lead to significant accuracy loss.

Low-rank tensor compression has achieved impressive headline compression numbers \citep{garipov2016ultimate} and is suitable for on-device acceleration \citep{zhang2021fpga,deng2019tie} due to its reliance on standard dense linear algebra operations. However it is often applied to massively overparameterized architectures such as VGG or ResNet \citep{novikov2015tensorizing,hawkins2019bayesian}. Recent work benchmarking sparse pruning \citep{blalock2020state} raises the issue that compression techniques applied to overparameterized architectures may not reach the compression/accuracy Pareto frontier of SOTA compressed networks such as EfficientNet. Further, tensor compression enforces low-rank factorized structure on neural network weights. This factorized structure is already exploited by SOTA computer vision backbones via depth-wise separable convolutions. This motivates us to consider the following question: how can low-rank tensor compression contribute to SOTA computer vision architectures? In this paper we investigate whether low-rank tensor compression can be combined with sparse pruning to capture complementary coarse and fine structures respectively and outperform either sparse pruning or low-rank tensor factorization alone.

Specifically, we investigate two forms of low-rank plus sparse decomposition of neural network weights. First we consider additive structure in which the neural network weights can be decomposed as the sum of a low-rank component and a sparse component. Second we consider a low-rank or sparse structure in which the entries of neural network weights are taken from either a sparse pruned weight or a low-rank weight.

\subsection{Related Work}

Existing work has studied the combination of low-rank matrices and sparse representations to compress VGG-style models \citep{yu2017compressing} and ResNets \citep{gui2019model}. These methods apply additive low-rank plus sparse compression to the weights of neural networks and can achieve compression results that outperform sparse compression or low-rank compression alone. More recent work suggests that low-rank structure is a viable compression method for the EfficientNet model class \citep{gu2021towards} on CIFAR-10/100 computer vision tasks. An open question is whether low-rank matrix or tensor structure remains for efficient modern networks trained on ImageNet.

\subsection{Contributions}

Our main contribution is our study of low-rank structure remaining in recent SOTA efficient neural networks for computer vision tasks. We observe that trends in architecture design \citep{howard2017mobilenets,sandler2018mobilenetv2,howard2019searching} have removed low-rank structure from such networks by building in factorized weights. Therefore the direct application of low-rank matrix and tensor methods is challenging. To the best of our knowledge, our work is the first to consider the combination of low-rank tensor compression with sparse pruning. Further, our work is the first to study low-rank+sparse compression of weights for SOTA architectures that rely on efficient depthwise separable convolutions.

\begin{figure*}
    \centering
    \includegraphics[width=\textwidth]{figs/algo_flow.png}
    \caption{High-level proposed algorithm flow.}
    \label{fig: algo flow}
\end{figure*}

\section{Background}

\subsection{Tensor Formats}

We follow the notation and exposition of \citep{hawkins2020towards} to describe the low-rank tensor formats that we employ. This paper uses lower-case letters (e.g., $a$) to denote scalars, bold lowercase letters (e.g., $\mat{a}$) to represent vectors, bold uppercase letters (e.g., $\mat{A}$) to represent matrices, and bold calligraphic letters (e.g., $\ten{A}$) to denote tensors. A tensor is a generalization of a matrix, or a multi-way data array. An order-$d$ tensor is a $d$-way data array $\ten{A}\in \mathbb{R}^{I_1 \times I_2 \times \dots \times I_d}$, where $I_n$ is the size of mode $n$. The $(i_1, i_2, \cdots, i_d)$-th element of $\ten{A}$ is denoted as $a_{i_1i_2 \cdots i_d}$. An order-$3$ tensor is shown in Fig.~\ref{fig: visual formats} (a).

The number of entries in a tensor with mode size $n$ and $d$ modes is $n^d$. This exponential dependence on the tensor order is often referred to as the ``curse of dimensionality" and can lead to high costs in both computing and storage. Fortunately, many practical tensors have a low-rank structure, and this property can be exploited to reduce such costs. We will use four tensor decomposition formats (CP, Tucker, and TT/TTM) to reduce the parameters of neural networks.

First, we introduce a necessary decomposition to describe the CP format.
\begin{definition}
A $d$-way tensor $\ten{A} \in \mathbb{R}^{I_1\times \cdots \times I_d}$ is rank-1 if it can be written as a single outer product of $d$ vectors
\begin{equation}
\ten{A} =\mat{u}^{(1)} \circ \dots \circ \mat{u}^{(d)}, \; {\text{with}} \; \mat{u}^{(n)} \in\mathbb{R}^{I_n} \; \text{for}\; n=1,\cdots, d. \nonumber
\end{equation}
\end{definition}
Each element of $\ten{A}$ is $a_{i_1 i_2 \cdots i_d}=\prod \limits_{n=1}^d u_{i_n}^{(n)}$,
where $u_{i_n}^{(n)}$ is the $i_n$-th element of the vector $\mat{u}^{(n)}$.

\begin{definition}\label{Def: CP}
The CP factorization~\citep{carroll1970analysis,harshman1994parafac} expresses tensor $\ten{A}$ as the sum of multiple rank-1 tensors:
\begin{equation}
\ten{A} = \sum_{j=1}^R \mat{u}_j^{(1)} \circ \mat{u}_j^{(2)} \dots \circ \mat{u}_j^{(d)}.
\end{equation}
Here $\circ$ denotes an outer product operator. The minimal integer $R$ that ensures the equality is called the {\bf CP rank} of $\ten{A}$. To simplify notation we collect the rank-1 terms of the $n$-th mode into a factor matrix $ \fm{n} \in \mathbb{R}^{I_n \times R}$ with $\fm{n}(:,j) = \mat{u}_{j}^{(n)}$. A rank-$R$ CP factorization can be described with $d$ factor matrices $\{ \mat{U}^{(n)}\}_{n=1}^d$ using $R\sum_n I_n$ parameters.
\end{definition}

\input{figs/fomats}

Next we introduce a prerequisite definition for the low-rank Tucker format.
\begin{definition}
The  mode-$n$ product of a tensor $\ten{A}\in\mathbb{R}^{I_1\times\dots \times I_{n} \times \dots\times I_d}$ with a matrix $\mat{U}\in\mathbb{R}^{J\times I_n}$ is
\begin{equation}
    \label{eq: mode n product}
    \begin{split}
    \ten{B} &= \ten{A}\times_n \mat{U} \Longleftrightarrow
    b_{i_1\dots i_{n-1} j i_{n+1} \dots i_d}=\sum_{i_n=1}^{I_n} a_{i_1  \dots i_d}  u_{j i_n}  .
    \end{split}
\end{equation}
\end{definition}
The result is still a $d$-dimensional tensor $\ten{B}$, but the mode-$n$ size becomes $J$. In the special case $J=1$, the $n$-th mode diminishes and $\ten{B}$ becomes an order-$d-1$ tensor. The Tucker format is defined by a series of mode$-n$ products between a small core tensor and several factor matrices.

\begin{definition}\label{Def: Tucker}
The Tucker factorization~\citep{tucker1966some} expresses a $d$-way tensor $\ten{A}$ as a series of mode-$n$ products:
\begin{equation}
\label{eq:Tucker}
\ten{A} = \ten{G}\times_1 \fm{1}\times_2 \dots \times_d \fm{d}.
\end{equation}
Here $\ten{G}\in\mathbb{R}^{R_1\times \dots\times R_d}$ is a small core tensor, and $\fm{n}\in\mathbb{R}^{I_n\times {R}_n}$ is a factor matrix for the $n$-th mode. The {\bf Tucker rank} is the tuple $(R_1,\dots,R_d)$. A Tucker factorization with ranks $R_n=R$ requires $R^d+R\sum_n I_n$ parameters.
 \end{definition}
The product structure permits more expressive cross-dimension interactions than the CP model but may not result in large parameter reductions due to the exponential dependence on the tensor order $d$.

Finally we introduce two formats, Tensor-Train and Tensor-Train Matrix. Like the Tucker format, these formats permit more expressive interactions across dimensions. Unlike the Tucker format these formats remove the exponential dependence on the tensor order $d$.
\begin{definition}
\label{def: tensor train}
The tensor-train (TT) factorization~\citep{oseledets2011tensor} expresses a $d$-way tensor $\ten{A}$ as a collection of matrix products:
\begin{equation}
\label{eq:TT}
a_{i_1 i_2 \dots i_d} = \ft{1}(:,i_1,:)\ft{2}(:,i_2,:)\dots \ft{d}(:,i_d,:).
\end{equation}
Each  TT-core $\ft{n}\in \mathbb{R}^{R_{n-1}\times I_n \times R_{n}}$ is an order-$3$ tensor. The tuple $(R_0,R_1,\dots,R_d)$ is the {\bf TT-rank} and $R_0=R_d=1$.
\end{definition}
The TT format uses $\sum_n R_{n-1}I_nR_{n}$ parameters in total and leads to more expressive interactions than the CP format.

The TT-format is well-suited to tensors, but is equivalent to low-rank matrix format when the order is $d=2$. We can extend the TT-format to matrices by reshaping along each mode. Let $\mat{A}\in\mathbb{R}^{I\times J}$ be a matrix. We assume that $I$ and $J$ can be factored as follows:
\begin{equation}
\label{eq: TTM dimension factorization}
    I=\prod_{n=1}^d I_n,J=\prod_{n=1}^d J_n.
\end{equation}
We can reshape $\mat{A}$ into a tensor $\ten{A}$ with dimensions $I_1\times\dots\times I_d\times J_1\times\dots\times J_d$, such that the $(i,j)$-th element of $\mat{A}$ uniquely corresonds to the $(i_1, i_2, \cdots, i_d, j_1, j_2, \cdots, j_d)$-th element of $\ten{A}$. The TT decomposition can extended to compress the resulting order-$2d$ tensor as follows.

\begin{definition}
\label{def: tensor train matrix}
The tensor-train matrix (TTM) factorization expresses an order-$2d$ tensor $\ten{A}$ as $d$ matrix products:
\begin{equation}
\label{eq: ttm factorization}
a_{i_1\dots i_d j_1 \dots j_d} = \ft{1}(:,i_1,j_1,:)\ft{2}(:,i_2,j_2,:)\dots \ft{d}(:,i_d,j_d,:).
\end{equation}
Each TT-core $\ft{n}\in \mathbb{R}^{R_{n-1}\times I_n \times J_n \times R_{n}}$ is an order $4$ tensor. The tuple $(R_0,R_1,R_2,\dots,R_d)$ is the {\bf TT-rank} and as before $R_0=R_d=1$. This TTM factorization requires $\sum_n R_{n-1}I_n J_nR_{n}$ parameters to represent $\ten{A}$. \end{definition}

We provide a visual representation of the CP, Tucker, and TT formats in Fig.~\ref{fig: visual formats} (b) -- (d).

\subsection{Folding to high-order tensors.} In the previous section we described how to adapt the TT format to the matrix case by factorizing the dimensions of a matrix and viewing the matrix as a tensor. Our goal is to apply tensor decompositions to the weights of many architectures that include $1\times 1$ convolutions (MobileNet, EfficientNet) or fully-connected layers (Vision Transformer). These matrix-format weights are well-suited to the TTM decomposition, but not the CP and Tucker formats which collapse to low-rank matrix format. To apply the CP and Tucker decompositions to these matrix weights we can perform a reshape operation. A weight matrix $\mat{W}\in\mathbb{R}^{I\times J}$ can be folded into an order-$d$ tensor $\ten{A}\in\mathbb{R}^{I_1\times\dots\times I_d}$ where $IJ=\prod_n I_n$. We can also fold $\mat{W}$ to an order-$2d$ tensor $\ten{A} \in \mathbb{R}^ {I_1 \times \cdots \times I_d \times J_1 \times \cdots \times J_d }$ such that $w_{ij}=a_{i_1\cdots i_d j_1 \cdots j_d}$. While a convolution filter is already a tensor, we can reshape it to a higher-order tensor with reduced mode sizes. Reshaping imposes additional higher-order and hierarchical tensor structure.

\subsection{Sparse Pruning}
We prune a tensor weight $\ten{S}$ by applying a binary tensor mask $\ten{M}$ to $\ten{S}$ using the entry-wise product
\[
(\ten{M}\odot \ten{S})_{i_1\dots i_d}=m_{i_1\dots i_d}\times s_{i_1\dots i_d}.
\]
We focus on global magnitude pruning \cite{lecun1990optimal} throughout this work and use a cubic pruning schedule taken from \cite{zhu2017prune}. At each step we mask the bottom $100\times(s_t-s_{t-1})$ percent of the non-masked weights in $\ten{S}$ by setting their corresponding entries in $\ten{M}$ to zero where
\begin{equation}
    \label{eq: pruning}
        s_t = s_f\left(\frac{t}{n}\right)^3
\end{equation}
and $s_f$ is the desired final sparsity ratio, $t$ is the current step, and $n$ is the total number of steps. This process gradually increases the sparsity in $\ten{M}$ until the target sparsity $s_f$ is reached.

\subsection{Distillation}

The goal of knowledge distillation is to transfer the knowledge of an expensive teacher model to a cheaper student model \citep{hinton2015distilling}. In this setting ``expensive" refers to the memory cost of storing the neural network weights. The expensive teacher model $f_t$ is a neural network pretrained on ImageNet. The student model $f_s$ is the compressed teacher model obtained through sparse pruning and/or tensor compression. Given a training example with input $\mat{x}$ and label $\mat{y}$ the knowledge distillation loss is
\begin{equation}
    \label{eq: KD loss}
    \mathcal{L}\left(\mat{x},\mat{y}\right)=\alpha l(\mat{x},\mat{y})+(1-\alpha)T^2 KL\left(f_s(\mat{x})/T||f_t(\mat{x})/T\right)
\end{equation}
where $l$ is any standard loss function and $KL$ is the KL-divergence. In our setting $l$ is the cross-entropy loss. The KL term in the loss function penalizes the student output distribution when it differs from the teacher output distribution. Both outputs are temperature scaled by $T$. We set $\alpha=0.9$ and $T=3$ which is a standard choice \citep{hinton2015distilling,cho2019efficacy}. We note that another reasonable loss function is fine tuning on the training data only (set $\alpha=1$) which we do not explore in this work.
\section{Experiments}\label{sec: experiments}

In this section we report results obtained on ImageNet-1K. All results are obtained by training on a machine with 32 CPU cores, 8 Tesla V100 GPUs, and 128GB memory. We train each method for 50 epochs using a batch size of 256 and distillation settings $\alpha=0.9,T=3$. Each run takes approximately 12hrs. All reported ImageNet accuracy metrics are validation accuracy. In all experiments we set the tensor rank so that the low-rank factors required to represent $\ten{L}$ use $10\%$ as many elements as are in $\ten{A}$.

\subsection{Reconstruction and Initialization}

First we compare two variants of our method: additive reconstruction with residual initialization and masking reconstruction with masking initialization. For this experiment, and all other MobileNet experiments, we use SGD with momentum $0.9$ base learning rate $0.1$ and a decay of $0.7$ applied every $5$ epochs. The decay rate was selected using a small grid search over $\{0.5,0.7,0.9\}$ and selecting the best ImageNet-1K validation accuracy for the sparse pruned MobileNet-v3-Large baseline. In Figure \ref{fig: initialization} we compare the results of our residual initialization strategy with our masking initialization strategy. We observe no clear gains from either strategy. Based on these results we use the simpler residual initialization strategy with additive reconstruction for the remainder of our experiments.

\begin{figure}
     \centering
     \begin{subfigure}[t]{0.48\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/mbv3_small_init_acc_5.png}
         \caption{}
         \label{fig: mobilenetv3 small init}
     \end{subfigure}
     \hfill
     \begin{subfigure}[t]{0.48\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/mbv3_large_init_acc_5.png}
         \caption{}
         \label{fig: mobilenetv3 large init}
     \end{subfigure}
    \caption{Comparison of initialization strategies for low-rank+sparse tensor decomposition for the (a) MobileNet-v3-Small and (b) MobileNet-v3-Large architectures. There are no major distinctions between initialization strategies.\label{fig: initialization}}
\end{figure}

\subsection{MobileNetv3 Results}
\begin{figure}
     \centering
     \begin{subfigure}[t]{0.49\textwidth}
         \centering
         \includegraphics[height=2.0in]{figs/mbv3_small_acc_5.png}
         \caption{}
         \label{fig: mobilenetv3 small results}
     \end{subfigure}
     \hfill
     \begin{subfigure}[t]{0.49\textwidth}
         \centering
         \includegraphics[height=2.0in]{figs/mbv3_large_acc_5.png}
         \caption{}
         \label{fig: mobilenetv3 large results}
     \end{subfigure}
     \centering
     \begin{subfigure}[t]{0.49\textwidth}
         \centering
         \includegraphics[height=2.0in]{figs/efficientnet_b0_acc_5.png}
         \caption{}
         \label{fig: efficientnet b0 results}
     \end{subfigure}
     \hfill
     \begin{subfigure}[t]{0.49\textwidth}
         \centering
         \includegraphics[height=2.0in]{figs/deit_small_acc_5.png}
         \caption{}
         \label{fig: deit small results}
     \end{subfigure}
\caption{ImageNet top-5 accuracy comparison between our LR+S method with CP/TT/Tucker tensor format and the sparse pruning baseline. Results for (a) Mobilenet-v3-Small (b) MobileNet-v3-Large (c) EfficientNet-B0 (d) DEIT-Small.\label{fig: all results}}
\end{figure}

In Figures \ref{fig: mobilenetv3 small results} and \ref{fig: mobilenetv3 large results} we plot a comparison between our method with varying tensor formats (CP,Tucker,TT) and the sparse pruning baseline on MobileNet-v3-Small and MobileNet-v3-Large respectively. We observe that the combination of low-rank and sparse representations is competitive with the sparse pruning baseline. However we do not observe clear performance advantages.

\subsection{Other Models: EfficientNet and DEIT}

To test if our conclusions hold across architectural families we test on EfficientNet-B0 \citep{tan2019efficientnet}. We use the same optimizer settings as the MobileNet experiments, and report our results in Figure ~\ref{fig: efficientnet b0 results}. We also test our results on DEIT \citep{touvron2021training} which is a highly accurate Vision Transformer \citep{dosovitskiy2020image}. Specifically, we the the DEIT-Small model with resolution 224 and patch size 16. We train with the Adam optimizer with learning rate $1e-4$. In Figures \ref{fig: efficientnet b0 results} and \ref{fig: deit small results} we observe similar results as those from the MobileNet family of models. Low-rank plus sparse decomposition is competitive with sparse pruning across models, but does not significantly outperform sparse pruning.

\subsection{Effect on Model Weights}

In this section we investigate the effects of our compression methods on the model weights. We highlight a key challenge: many pre-trained models do not possess weights that are close to low-rank. In particular, models with a large number of FC layers or 1x1 convolutions are not ``approximately low-rank" even after reshaping from matrix to tensor format. In Figure \ref{fig: layerwise residuals} we plot the layer-wise relative errors of the low-rank decomposition, computed as
\begin{equation}
    \label{eq: relative error}
    Err = \frac{\|\ten{A}-\ten{L}\|_2}{\|\ten{A}\|_2}
\end{equation}
We plot the result of CP-format decomposition applied to the MobilNetv3-small and EfficientNet-B0 models. We observe that many layers have a high relative error, but also that several layers stand out in that the low-rank tensor component fits the original tensor $\ten{A}$ almost exactly.

\begin{figure}
     \centering
     \begin{subfigure}[b]{0.49\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/mobilenet_v3_small_global_residuals.png}
         \caption{}
         \label{fig: mobilenetv3 layerwise residuals}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.49\textwidth}
         \centering
         \includegraphics[width=0.9\textwidth]{figs/efficientnet_b0_global_residuals.png}
         \label{fig: efficientnet b0 layerwise residuals}
         \caption{}
     \end{subfigure}
\caption{Layer-wise relative error of the low-rank tensor-train factorization for each layer of (a) Mobilenet-v3-Small and (b) EfficientNet-B0. Other tensor types are similar. Layers with high relative error are the pointwise convolutions.\label{fig: layerwise residuals}}
\end{figure}
\begin{figure}[ht]
     \centering
     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/mobilenet_cp_spatial.png}
         \caption{MobileNetv3-Small spatial convolution before and after removing low-rank tensor factors.}
         \label{fig: mobilenetv3_spatial_visual}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/mobilenet_cp_spatial_hist.png}
         \caption{Effect of tensor factorization on MobileNetv3-Small spatial convolution weights.}
         \label{fig: mobilenetv3_spatial_hist}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/mobilenet_cp_pointwise.png}
         \caption{MobileNetv3-Small pointwise convolution before and after removing low-rank tensor factors.}
         \label{fig: mobilenetv3_pointwise_visual}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/mobilenet_cp_pointwise_hist.png}
         \caption{Effect of tensor factorization on MobileNetv3-Small pointwise convolution weights.}
         \label{fig: mobilenetv3_pointwise_hist}
     \end{subfigure}
        \caption{(a) Spatial convolutions possess regular structure that can be captured by low-rank tensor factorization. (b) Tensor factorization on spatial convolutions introduces overhead but leads to a weight distribution (in \textcolor{orange}{orange}) that is easier to prune. (c) Point-wise convolutions posses less regular structure, and tensor factorization induces fewer 0 entries. (d) Tensor factorization on pointwise convolutions introduces overhead and leads to a weight distribution (in \textcolor{orange}{orange}) that is not easier to prune.}
        \label{fig: pointwise spatial}
\end{figure}

We found that many, but not all, spatial convolutions were fit well by tensor decomposition and have low relative error. We provide an example in Figure \ref{fig: pointwise spatial} which gives representative examples of pointwise and spatial convolutions in MobileNetv3-Small. We detail the effect of our proposed compression method on the model weights. Our motivation is to ``preprocess" the weights by removing low-rank structure so that the remaining weight values are better-suited to sparse pruning. Therefore we investigate the effects of our decomposition approach on the weight distributions. We show how tensor compression can remove repeated coarse structure present in spatial convolutions, resulting in a sharply peaked weight distribution compared to the weights of the pretrained model. In contrast tensor decomposition captures less structure in the pointwise convolution and does not induce a weight distribution that is better-suited to sparse pruning.

\subsection{Low-Rank Only}

We also considered compressing models using only post-training tensor decomposition. We take a model and perform tensor decomposition on all convolutional kernels and FC layers except the last layer. This idea was first studied in \cite{lebedev2014speeding} on larger, more compressible architectures. We report our results in Figure \ref{fig: tensor only}. We followed the same experimental setup as before, and observed that it was difficult to achieve similar accuracy with low-rank decomposition alone. We hypothesize that this is the natural result of the observation from Figure \ref{fig: layerwise residuals}. Since the model weights are not ``aproximately low-rank" it is much more challenging to decompose a model and then fine-tune. It is possible that a longer training schedule (>>50 epochs), or training from scratch in low-rank tensor format, would resolve this issue.
\begin{figure}[ht]
     \centering
     \begin{subfigure}[b]{0.49\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/mobilenet_v3_small_tensor_only_acc_5.png}
         \caption{}
         \label{fig: mobilenet v3 small tensor only}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.49\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/mobilenet_v3_large_tensor_only_acc5.png}
         \caption{}
         \label{fig: mobilenet v3 large tensor only}
     \end{subfigure}
\caption{ImageNet top-5 accuracy comparison between sparse pruning and low-rank tensor decomposition on (a) MobileNet-v3-Small (b) MobileNet-v3-Large}
\label{fig: tensor only}
\end{figure}

redpajama_set_name: RedPajamaArXiv