text:
Q: Trying to tunnel through https via a proxy built from ground up browser never responds back? Due to a variety of reasons it became necessary to create our own proxy. Everything is working as it should through HTTP. As soon as we receive a CONNECT to tunnel through SSL is when everything goes wrong. What we do logically is take the CONNECT parse out the host and port so we know where we are sending future ssl requests and create a request to send back to the browser stating we have successfully made the ssl handshake like so:
HTTP/1.0 200 Connection established\r\nProxy-agent: test\r\n\r\n
What we expect to happen is that the browser once receiving this successful message will send us the next https request. However, instead we get sent another CONNECT request over and over.It is clear that is does not like the response we send back. The problem is that i'm not exactly sure why? Does the response back need to be sent back via an https socket? I just don't understand this process enough to move forward.
Here is my server class:
public class HttpServer extends Observable implements IWebServer, Runnable
{
int Port = -1;
int State = HttpState.IDLE;
ArrayList<WebTransactionEvent> History = new ArrayList<WebTransactionEvent>();
ArrayList<HttpService> myServices = new ArrayList<HttpService>();

SocketChannel myChannel = null;
boolean needResponse = false;
boolean shouldStop;
Logger logger = OpsToolsLogger.getLogger(HttpServer.class.getName());
Selector selector ;
static Hashtable<String, HttpServer> myInstances = new Hashtable<String, HttpServer>();
Hashtable<HttpTransaction, HttpService> myTaskTable = new Hashtable<HttpTransaction, HttpService>();
Vector<HttpTransaction> transactionQueue = new Vector<HttpTransaction>();

private HttpServer(){}

private HttpServer(int Port)
{
    logger.log(Level.WARNING, "HttpServer: startup - listening to port: " + Port);
    this.Port = Port;
    shouldStop = false;

    // Create the selector
    try {
            selector = Selector.open();
            ServerSocketChannel serverChannel = ServerSocketChannel.open();
            serverChannel.configureBlocking(false);
            serverChannel.socket().bind(new InetSocketAddress(Port));
            this.registerSocket(serverChannel);
    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
    new Thread(this).start();
}

public static HttpServer getInstance(String port)
{
    if( !myInstances.containsKey( port ) )
    {
        myInstances.put( port, new HttpServer(Integer.parseInt(port)));
    }

    return myInstances.get(port);
}

public int getState()
{
    return State;
}

public void stop()
{
    shouldStop = true;
}

public boolean needResponse()
{
    return needResponse;
}

public HttpTransaction getNextTransaction()
{
    if(transactionQueue.isEmpty())
    {
        return null;
    }
    //System.out.println("grabbing next trans");
    HttpTransaction temp = transactionQueue.firstElement();
    transactionQueue.remove(0);//pop trans from  queue
    return temp;
}

public void dropTransaction()
{
    myTaskTable.clear();
    needResponse = false;

}

public synchronized boolean respond(HttpTransaction transaction, IHttpResponse editedResponse, boolean closeConnection)
{
    logger.log(Level.FINE, "HttpServer: responding ");
    needResponse = false;
    if(myTaskTable.isEmpty())
    {
        return false;
    }

    //see if there isn't a service object registered with that transaction
    if(!myTaskTable.containsKey(transaction))
    {
        return false;
    }

    State = HttpState.SENDING_RESPONSE;
    ManipulatedHttpTransaction myTrans = (ManipulatedHttpTransaction) transaction;
    HttpResponse response = (HttpResponse) editedResponse;
    myTrans.setManipulatedResponse( response );
    HttpService serv = myTaskTable.get(transaction);
    if(!serv.respond(myTrans.getManipulatedResponse(), closeConnection))
    {
        History.add( new WebTransactionEvent( myTrans, WebTransactionEvent.TRANSACTION_ERROR ) );
        return false;
    }

    myTaskTable.remove(transaction);

    History.add( new WebTransactionEvent( myTrans, WebTransactionEvent.TRANSACTION_COMPLETED ) );

    needResponse = !myTaskTable.isEmpty();

    return true;
}

public void registerSocket(ServerSocketChannel theSocket)
{
    try {
        theSocket.register(selector, SelectionKey.OP_ACCEPT);
    } catch (ClosedChannelException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
}

@Override
public void run()
{
    try {

        while (!shouldStop ) {
            // Wait for an event
            selector.select();

            // Get list of selection keys with pending events
            Iterator it = selector.selectedKeys().iterator();

            // Process each key
            while (it.hasNext()) {
                // Get the selection key
                SelectionKey selKey = (SelectionKey)it.next();

                // Remove it from the list to indicate that it is being processed
                it.remove();

                // Check if it's a connection request
                if (selKey.isAcceptable()) {
                    // Get channel with connection request
                   ServerSocketChannel ssChannel = (ServerSocketChannel)selKey.channel();
                   SocketChannel theChannel =  ssChannel.accept();
                   if(theChannel != null)
                    {

                        logger.log(Level.FINEST, "HttpServer: Connection established");

                        try
                        {
                            theChannel.configureBlocking(false);
                        }
                        catch(Exception e)
                        {
                            logger.log(Level.WARNING, "myChannel = null ( configureBlocking() )");
                            //bytesRead = -1;
                        }

                        myServices.add( new HttpService(this, theChannel ) );
                        needResponse = true;

                    }

                    //needResponse = !myTaskTable.isEmpty();
                    //System.out.println("need response: "+ needResponse);

                }
            }
        }
    } catch (IOException e) {
    }
    //shutdown
    logger.log(Level.WARNING, "Server stopping - " + Port);
}

public ArrayList<WebTransactionEvent> getHistory()
{
    return new ArrayList<WebTransactionEvent>(History);
}

public boolean switchServerToSSL()
{
    //HttpService tempService = myTaskTable.get(PendingTransaction);
    //tempService.useSSL = true;
    return true;
}

/**
 * Adds the transaction from browser to the transaction queue and also ties it to a service by adding it to myTasks map
 * @param myTrans
 * @param httpService
 */
public void addTransaction(ManipulatedHttpTransaction myTrans,
        HttpService httpService) {
    // TODO Auto-generated method stub
    //ensure vector has room to add another transaction
    if(transactionQueue.capacity() <= transactionQueue.size())
        transactionQueue.ensureCapacity(transactionQueue.size() * 2);

    transactionQueue.add(myTrans);//add transaction to queue
    myTaskTable.put(myTrans, httpService);//tie the transaction toits service
   // System.out.println("server notifying proxy: " + myTrans.getFullURL());
    this.setChanged();
    this.notifyObservers(myTrans);
}

}
Here is portion in proxy that handles a CONNECT:
if(tempTransaction.getOriginatingRequest().getMethod().contentEquals("CONNECT"))
            {
                /*tell the browser that the connection exists
                 *
                 * Each time you connect to an SSL-protected website, Burp generates a server certificate for that host, signed by the CA certificate
                 *
                 * The server certificates presented to the client (i.e. a web browser) are dynamically generated/signed by the proxy and contain most of the same fields as the original webserver certificate. The subject DN, serial number, validity dates, and extensions are preserved. However, the issuer DN is now set to the name of the proxy's self-signed
                 * certificate and the public/private keys of the proxy are used in creating the forged certificate. These forged certificates are cached (in memory) by the proxy, for better performance
                 */
                HttpResponse tunnelResponse = new HttpResponse("HTTP/1.0 200 Connection established\r\nProxy-agent: Ops Assistant\r\n\r\n");
                tempTransaction.setResponse(tunnelResponse);

                if(!finishResponse2(tempTransaction,tempTransaction.getResponse(), false));
                {
                    //close the connection
                }

                myServer.switchServerToSSL();
            }

Here is section sends request back to browser:
 public boolean respond(IHttpResponse response, boolean closeConnection)
{
    isCloseConnectionRequested = closeConnection;

    try
    {
        if(useSSL)
        {
            ByteBuffer tmpBuffer = response.getData();
            tmpBuffer.position(0);
            myConnection.SecureWrite( tmpBuffer );
        }
        else
        {
            ByteBuffer tmpBuffer = response.getData();
            tmpBuffer.position(0);
            myConnection.Write(tmpBuffer);
        }
        if(closeConnection)
        {
            myChannel.close();
            myChannel = null;
        }
    }
    catch (Exception e)
    {
        isResponded = true;
        return false;
    }

    isResponded = true;
    return true;

}

Probably most important the socket class:
 public class SocketConnection implements IConnection
   {

public SocketChannel theSocketChannel;
public InetSocketAddress theRemoteAddress;
public int TimeoutThreshold;

private int TimeOutThreshold = 30;
private SSLEngine theSSLEngine;
private SSLContext theSSLContext;
private ByteBuffer inNetworkDataBuffer;
private ByteBuffer inAppDataBuffer;
private ByteBuffer outNetworkDataBuffer;
private ByteBuffer outAppDataBuffer;

//create outbound connection to host/port
public SocketConnection(String Host, int Port ) throws IOException
{
    theRemoteAddress = new InetSocketAddress( Host, Port);
    theSocketChannel = SocketChannel.open();
    theSocketChannel.configureBlocking(false);
    theSocketChannel.connect( theRemoteAddress );
    theSocketChannel.finishConnect();
}

//use existing socket connection
public SocketConnection(SocketChannel existingChannel) throws IOException
{
    theSocketChannel = existingChannel;
    theSocketChannel.configureBlocking(false);
    theRemoteAddress = new InetSocketAddress( existingChannel.socket().getInetAddress(), existingChannel.socket().getPort() );
}

public boolean setTimeOut(int newTimeOutThreshold)
{
    TimeOutThreshold = newTimeOutThreshold;
    return true;
}

public void waitForSocketToConnect() throws Exception
{
    int i = 0;
    while( !this.isConnected() )
    {
        this.finishConnect();
        if(i>=3000)
        {
            throw new Exception();
        }
        i++;

        try{Thread.sleep(10);}catch(Exception e){}
    }
}

public boolean Write( ByteBuffer DataToSend )
{
    try
    {
        //DataToSend.flip();
        int numBytesWritten = theSocketChannel.write(DataToSend);
        try
        {
            DataToSend.compact();
        }
        catch (ReadOnlyBufferException e)
        {
            DataToSend.rewind();
        }
    }
    catch (IOException e)
    {
        // Connection may have been closed
    }

    return true;
}

public ByteBuffer Read()
{
    ByteBuffer ResponseBytes = ByteBuffer.allocateDirect(0);

    try
    {
        ByteBuffer netBuffer = ByteBuffer.wrap(new byte[10000]);

        // Clear the buffer and read bytes from socket
        netBuffer.clear();

        int numBytesRead = theSocketChannel.read(netBuffer);
        if(numBytesRead == -1)
            return null; //-1 means we done return null as the flag
        netBuffer.flip();

        ByteBuffer tempBuffer = ByteBuffer.wrap(new byte[ ResponseBytes.limit() + netBuffer.limit() ]);
        ResponseBytes.position(0);
        netBuffer.position(0);
        tempBuffer.put(ResponseBytes);
        tempBuffer.put(netBuffer);
        netBuffer.flip();
        ResponseBytes = tempBuffer;

    }
    catch (IOException e)
    {
        // Connection may have been closed
        e = e;
        return ByteBuffer.wrap( e.getMessage().getBytes() );
    }

    return (ByteBuffer) ResponseBytes.flip();

}

public boolean SecureWrite( ByteBuffer DataToSend )
{
    boolean writeSuccess = true;

    try
    {
        //if we don't have a SSLEngine make one
        if(theSSLEngine==null)
        {
            setupSSL();
        }

        //Convert Data
        outAppDataBuffer.clear();
        outAppDataBuffer.put(DataToSend);
        outAppDataBuffer.flip();
        SSLEngineResult sslResult = theSSLEngine.wrap(outAppDataBuffer, outNetworkDataBuffer);
        outAppDataBuffer.compact();
        //outNetworkDataBuffer.flip();
        //int numBytesWritten = theSocketChannel.write(outNetworkDataBuffer);
        if(sslResult.getStatus() == SSLEngineResult.Status.OK)
        {
            if(sslResult.getHandshakeStatus() != SSLEngineResult.HandshakeStatus.NOT_HANDSHAKING)
            {
                // Write bytes
                outNetworkDataBuffer.flip();
                int numBytesWritten = theSocketChannel.write(outNetworkDataBuffer);
                outNetworkDataBuffer.compact();

                if(finishHandshake(sslResult))
                {
                    DataToSend.rewind();
                    return SecureWrite(DataToSend);
                }
                else
                {
                    return false;
                }
            }
            else
            {
                // Write bytes
                outNetworkDataBuffer.rewind();
                Write(outNetworkDataBuffer);
            }

        }
        else
        {

        }

    }
    catch(Exception e)
    {
        writeSuccess = false;
    }

    return writeSuccess;

}

public ByteBuffer SecureRead() throws ReadTimedOutException
{
    int timeElapsed = 0;
    ByteBuffer ResponseBytes = ByteBuffer.allocateDirect(0);

    try
    {
        //if we don't have a SSLEngine make one
        if(theSSLEngine==null)
        {
            setupSSL();
        }

        int consumedCount = 0;
        SSLEngineResult sslResult;
        do
        {
            //inNetworkDataBuffer.clear();
            inNetworkDataBuffer.put( Read() );
            inNetworkDataBuffer.flip();
            sslResult = theSSLEngine.unwrap( inNetworkDataBuffer, inAppDataBuffer );
            consumedCount += sslResult.bytesConsumed();
            inNetworkDataBuffer.compact();

            if( sslResult.getStatus() == SSLEngineResult.Status.OK )
            {

                if(sslResult.getHandshakeStatus() != SSLEngineResult.HandshakeStatus.NOT_HANDSHAKING)
                {

                    if(finishHandshake(sslResult))
                    {
                        return SecureRead();
                    }
                    else
                    {
                        return ByteBuffer.allocateDirect(0);
                    }
                }
                else
                {
                    timeElapsed = 0;
                    inAppDataBuffer.flip();
                    ByteBuffer tempBuffer = ByteBuffer.wrap(new byte[ ResponseBytes.limit() + inAppDataBuffer.limit() ]);
                    ResponseBytes.position(0);
                    inAppDataBuffer.position(0);
                    tempBuffer.put(ResponseBytes);
                    tempBuffer.put(inAppDataBuffer);
                    inAppDataBuffer.flip();
                    ResponseBytes = tempBuffer;
                    ResponseBytes.flip();
                }
            }
            else
            {
                //the status wasn't ok
                timeElapsed++;
            }
        }while(consumedCount < inNetworkDataBuffer.limit() && sslResult.getStatus() != SSLEngineResult.Status.OK);

    }
    catch (Exception e)
    {
        System.out.println(e.toString());
    }

    if(timeElapsed>=TimeOutThreshold)
    {
        throw new ReadTimedOutException();
    }

    return ResponseBytes;
}

public boolean Disconnect()
{
    try
    {
        theSocketChannel.close();
    }
    catch(Exception e)
    {
        return false;
    }

    return true;
}

public boolean isClosed()
{
    return !theSocketChannel.isOpen();
}

@Override
public String getHost()
{
    return theRemoteAddress.getHostName();
}

@Override
public int getPort()
{
    return theRemoteAddress.getPort();
}

public boolean isConnected()
{
    return theSocketChannel.isConnected();
}

@Override
public boolean hasSecure()
{
    return true;
}

public boolean finishConnect() throws Exception
{
    return theSocketChannel.finishConnect();
}

private void setupSSL() throws NoSuchAlgorithmException, KeyManagementException
{

    //create a new SSLEngine instance
    System.setProperty( "javax.net.debug", "ssl");
    TrustManager[] tm = new TrustManager[] { new NaiveTrustManager() };
    SSLContext theSSLContext = SSLContext.getInstance ("TLS");
    theSSLContext.init( new KeyManager[0], tm, new SecureRandom( ) );

    theSSLEngine = theSSLContext.createSSLEngine( theRemoteAddress.getHostName(), theRemoteAddress.getPort());
    theSSLEngine.setUseClientMode(true);

    inNetworkDataBuffer = ByteBuffer.wrap(new byte[theSSLEngine.getSession().getPacketBufferSize()]);
    inAppDataBuffer = ByteBuffer.wrap(new byte[theSSLEngine.getSession().getApplicationBufferSize()]);
    outNetworkDataBuffer = ByteBuffer.wrap(new byte[theSSLEngine.getSession().getPacketBufferSize()]);
    outAppDataBuffer = ByteBuffer.wrap(new byte[theSSLEngine.getSession().getApplicationBufferSize()]);

}

private boolean finishHandshake(SSLEngineResult sslResult)
{
    boolean bFinished = false;

    while(sslResult.getHandshakeStatus() != SSLEngineResult.HandshakeStatus.FINISHED)
    {
        if( sslResult.getStatus() == SSLEngineResult.Status.CLOSED )
        {
            bFinished = false;
            //break;
        }

        if(sslResult.getHandshakeStatus() == SSLEngineResult.HandshakeStatus.NEED_TASK)
        {
            Runnable task;
            while ((task=theSSLEngine.getDelegatedTask()) != null)
            {
                task.run();
            }

            try
            {
                //outNetworkDataBuffer.flip();
                sslResult = theSSLEngine.wrap(outAppDataBuffer, outNetworkDataBuffer);
                //outNetworkDataBuffer.compact();
            }
            catch (SSLException e)
            {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }

        }
        else if(sslResult.getHandshakeStatus() == SSLEngineResult.HandshakeStatus.NEED_WRAP)
        {
            try
            {
                outAppDataBuffer.flip();
                sslResult = theSSLEngine.wrap(outAppDataBuffer, outNetworkDataBuffer);
                outAppDataBuffer.compact();
            } catch (SSLException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
            if((sslResult.getStatus() == SSLEngineResult.Status.BUFFER_OVERFLOW) || (outNetworkDataBuffer.position() > 0))
            {
                try
                {
                    outNetworkDataBuffer.flip();
                    int numBytesWritten = theSocketChannel.write(outNetworkDataBuffer);
                    outNetworkDataBuffer.compact();
                }
                catch (Exception e)
                {
                    // TODO Auto-generated catch block
                    e.printStackTrace();
                }
            }
        }
        else if(sslResult.getHandshakeStatus() == SSLEngineResult.HandshakeStatus.NEED_UNWRAP)
        {
            try
            {
                int numBytes;
                //read data from the socket into inNetworkBuffer
                inNetworkDataBuffer.flip();
                sslResult = theSSLEngine.unwrap( inNetworkDataBuffer, inAppDataBuffer );
                inNetworkDataBuffer.compact();
                if(theSSLEngine.isInboundDone())
                {

                }
                else
                {
                    numBytes = theSocketChannel.read(inNetworkDataBuffer);
                    numBytes = numBytes;
                }

            }
            catch (Exception e)
            {
                e.printStackTrace();
                return false;
            }
        }
    }

    return true;
}

}
Anyone have any tips on how to best establish this handshake with the browser?

A: Have you read the Internet draft? The CONNECT is received in plaintext. You form the upstream connection and return the 'HTTP/1.0 200 Connection established' response. After that the proxy isn't processing requests and responses, it is just copying bytes in both directions, whatever they may happen to be. Specifically, the proxy isn't concerned with SSL in any way shape or form.


More people die in the United States each year from foodborne illness than from accidental fires. Foodborne illness can be prevented. Who is responsible for safety of food served to a consumer?
The owner of the food establishment.
All food workers in a food establishment.


text:
Sychesia dryas is a moth in the family Erebidae. It was described by Pieter Cramer in 1775. It is found on the West Indies, Guatemala, Costa Rica, French Guiana, Brazil, Peru, Ecuador, Bolivia, Suriname, Venezuela and Trinidad.

Subspecies
Sychesia dryas dryas
Sychesia dryas tupus Jordan, 1916

References

Moths described in 1775
Phaegopterina
Taxa named by Pieter Cramer

redpajama_set_name: RedPajamaWikipedia


text:
Podwysokie (ukr. Підвисоке) – wieś na Ukrainie w rejonie oratowskim, obwodu winnickiego.

Siedziba dawnej  w powiecie lipowieckim na Ukrainie.

Dwór
 murowany dwór wybudowany na przełomie XVIII w. i XIX w. przez Kłoszowskiego lub Klonowskiego. Podwysokie było rezydencją Stanisława Potockiego (zm. w 1870), wnuka Włodzimierza.

Przypisy

Linki zewnętrzne

 Podwysokie na stronie Rady Najwyższej Ukrainy

Wsie w rejonie oratowskim


text:
United States v. Spotted Bear
United States Court of Appeals, Eighth Circuit
United States of America Plaintiff- Appellee
Lonnie Dale Spotted Bear Defendant-Appellant
Submitted: November 15, 2018
Appeal from United States District Court for the District of North Dakota - Bismarck
Before COLLOTON, SHEPHERD, and STRAS, Circuit Judges.
STRAS, CIRCUIT JUDGE.
A jury found Lonnie Dale Spotted Bear guilty of four counts of sexual abuse involving three young female relatives. On appeal, he challenges the district court's[1]decision to allow the government to play video recordings of their forensic interviews for the jury. We affirm.
N.H.E., S.H., and M.S. accused Spotted Bear of molesting them at his home. As part of the investigation into their accusations, a specially trained social worker conducted a videotaped interview with each victim. After the investigation concluded, the government charged Spotted Bear with two counts of aggravated sexual abuse of a child under 12, one count of attempted aggravated sexual abuse of a child under 12, and one count of abusive sexual contact with a child under 12. See 18 U.S.C. §§ 2241(c), 2244(a)(5).
Spotted Bear's defense was that the three girls fabricated the allegations due to an unrelated family disagreement and then embellished them in response to pressure and coaxing by investigators. During Spotted Bear's trial, all three girls testified and described the abuse. To respond to the fabrication defense, the government played a portion of each girl's forensic interview for the jury. Their responses during the interviews were generally consistent with their answers at trial.
Spotted Bear never objected at trial to the portions of the recordings that the government played for the jury. Only now, on appeal, does he claim that they were inadmissible because they contained hearsay. See Fed. R. Evid. 802. In the absence of an objection, we review only for plain error, which requires Spotted Bear to show that the district court made a "clear or obvious" error that affected his substantial rights and that the error "seriously affect[ed] the fairness, integrity or public reputation of judicial proceedings." United States v. White Bull, 646 F.3d 1082, 1091 (8th Cir. 2011) (citation omitted); see also Fed. R. Crim. P. 52(b).
We analyze N.H.E.'s interview first, because the circumstances leading to its admission are different than for the other recordings. From the outset, Spotted Bear's theory was that N.H.E.'s trial testimony was an embellished version of what she had said during her forensic interview, which itself embellished upon her original allegations. In his opening statement, for example, Spotted Bear's counsel claimed that investigators encouraged N.H.E. "to say more stuff." And later, while cross-examining N.H.E., Spotted Bear's counsel asked whether she had "added some more things that [she] didn't tell [the forensic interviewer]." He followed up by inquiring whether, during the forensic interview, she had "added to [her] story." N.H.E. answered yes to both questions but did not explain what she added. This line of questioning, especially in light of Spotted Bear's theory of witness manipulation, could have left the jury with the mistaken impression that N.H.E. had significantly changed her story.
When a criminal defendant creates a false or misleading impression on an issue, we have held the government may "clarify, rebut, or complete [the] issue" with what would "otherwise [be] inadmissible evidence, including hearsay statements." United States v. Eagle, 515 F.3d 794, 801 (8th Cir. 2008). The government played the recording to allow the jury to judge for itself whether N.H.E. had in fact changed her story since her forensic interview. Cf. United States v. Smith, 591 F.3d 974, 982 (8th Cir. 2010) (holding that a defendant opened the door to the playing of a recording of a child-sex-abuse victim's forensic interview by asking the interviewer about alleged inconsistencies between the victim's trial testimony and her interview). Having opened the door to this evidence, Spotted Bear cannot now complain about its admission.[2] Accordingly, the district court did not err, much less plainly err, when it allowed the government to play the recording of N.H.E.'s forensic interview for the jury. See Eagle, 515 F.3d at 801; United States v. Womochil, 778 F.2d 1311, 1315 (8th Cir. 1985).
The remaining two recordings call for a different analysis because Spotted Bear's counsel did not open the door to their admission through his questioning of S.H. or M.S. Still, Spotted Bear failed to object to either recording, so he must satisfy the plain-error test to receive any relief. For two reasons, he has not met his burden of showing that the recordings affected his substantial rights. See United States v. Dominguez Benitez, 542 U.S. 74, 82 (2004) (stating that courts must "demand[] strenuous exertion" from defendants before granting relief on plain-error review).
First, the recordings were largely "cumulative of other government evidence." United States v. Worman, 622 F.3d 969, 977 (8th Cir. 2010). S.H.'s sister, for example, testified at length about S.H.'s abuse, and N.H.E. separately told the jury that S.H. had confided in her about the abuse, even before either girl had told her parents. The jury also learned about the specifics of M.S.'s abuse from two sources other than the recording: M.S.'s own trial testimony as well as the testimony of her mother. Her mother's testimony, in particular, echoed M.S.'s forensic interview, including the crucial fact that Spotted Bear molested her at his home on Thanksgiving. To be sure, some of this other evidence may also have been hearsay, but the fact that Spotted Bear does not challenge it means that we can consider it in evaluating whether he was prejudiced. See United States v. Peneaux, 432 F.3d 882, 895 (8th Cir. 2005).
         Second, the evidence of Spotted Bear's guilt was strong overall. See United States v. Gayekpar, 678 F.3d 629, 638 (8th Cir. 2012); see also United States v. Ramos-Caraballo, 375 F.3d 797, 804 (8th Cir. 2004) ("Especially [when] there is strong evidence of guilt, some improper repetition of testimony . . . in a generalized effort to bolster the witness[] matters little." (brackets, internal quotation marks and citation omitted)). Specifically, the jury heard that Spotted Bear engaged in a lengthy pattern of sexual abuse, that all three of his victims were female relatives under the age of 12, and that he molested them in a similar fashion. Cf. United States v. Never Misses A Shot, 781 F.3d 1017, 1027 (8th Cir. 2015) (stating that "evidence that the defendant committed similar sexual assaults or child molestations" can be used to prove guilt). In the end, the ...

redpajama_set_name: RedPajamaCommonCrawl


text:
Southampton High School is a public high school located in Courtland, Virginia, United States, in Southampton County, west of Franklin and east of Emporia. It is part of  Southampton County Public Schools. The school's current facility opened in 1993.

Athletic teams compete in the Virginia High School League's AA Southside District in Region I.

Notable alumni
Percy Ellsworth – NFL football safety
Jim Gillette, professional American football halfback
Cyrus Lawrence, football player
Riddick Parker, professional football player
Hunter Peck – Pitcher- Houston Astros
Greg Scott – NFL football defensive end

References

External links
Southampton High School

Schools in Southampton County, Virginia
Public high schools in Virginia
Educational institutions established in 1906
1906 establishments in Virginia


Hailing from Bondi Beach in Australia, Patrick's line of haircare products got us hooked as one of the best men's grooming brands out there. Now, they've introduced their line of skincare products with the same level of attention to detail to both the ingredients and packaging as found in their haircare line. In this exclusive North American release with Uncrate and Patrick's you'll find the FS1 Face Scrub and FW1 Face Wash. The scrub is comprised of volcanic sand and crushed diamonds, while the foaming wash contains Green Tea, White Tea, Lemon Myrtle, Vitamin C and other ingredients to ward off signs of premature aging. Each are produced in the USA, are free from parabens, sulfates and phthalates, are cruelty free, and 100% vegan.
While Patrick's FW1 Face Wash will cleanse your face, it does have a few more tricks up its sleeve. The foaming cleanser utilizes a range of ingredients like green tea, vitamin c, oat kernel extract, and sunflower seed oil to activate cellular regeneration, soothe skin, and reduce redness. Its antioxidant-rich formula protects the skin from environmental damages and also eliminates shine for a matte finish. The result is one of the best anti-aging men's face washes on the market, and equally as impressive is the Patrick's signature packaging it is housed in. FW1 is paraben, sulfate, and phthalate-free, so feel free to use it every day.
Fire and ice are generally foes but in this instance, they're living harmoniously in a dual-action exfoliant. Using volcanic sand and crushed diamonds, Patrick's FS1 Face Scrub unclogs pores and removes dead skin for a deep clean. In doing so, it improves skin tone while minimizing pores, reducing ingrown hairs, and can even fade small scars. With its added anti-aging peptides, multivitamins, and anti-irritants, the scrub also acts as a facial treatment that encourages the regeneration of cells, reduces redness, and retains moisture for smooth and hydrated skin.
Charcoal can pull contaminants out of your face, not just your whiskey, and this face wash removes excess oil like a...like a well-oiled machine, really. Activated charcoal extracts impurities from deep in the pores; olive oil lifts dirt and excess oils from the skin; jojoba creates a protective layer atop the skin that thwarts environmental toxins; and last, but not least, aloe adds extra moisture to keep the skin firm and healthy.
A proper night's rest is essential to good health. The Nue Co. Sleep Drops help ensure you clock in those necessary eight hours with a natural formula. Made from valerian root and passionflower, the non-habit-forming sleep aid slowly shuts down the body allowing you to drift off to sleep with ease, while also reducing anxiety with a blend of catnip and chamomile. Because of its wholesome ingredients, you'll be able to reap the rewards of your blissful slumber without any drowsy side effects.
Bixby prides itself for pinpointing the optimum drag of a comb for a certain hair type: That is, the ideal distance between each tooth (1.3mm) and length of each (19mm). This provides gentle separation for detangling or distributing product without any tugging or torturing. Available in black or amber, their wide tooth comb measures 133 x 5.5 x 35 mm, and arrives inside a tin travel case, and it's ideal for thicker hair types where more texture is needed for styling.
Barber meets bartender for a healthy, soothing shave. In its "Watermint Gin" shave cream, the fresh-faced pros at Blind Barber combine antiseptic and antibacterial juniper berries (a key ingredient in gin, as well) with the soothing and refreshing powers of watermint. It's gentle on all skin types--even dry, sensitive, or oily--and builds a quick, brushless lather right in your hands. Build a lather in palms with a nickel-sized amount (more as needed). Apply directly to warmed, freshly washed skin.
Inspire your mind and alert your body with MINERAL's CBD for FOCUS formula. FOCUS is a 1:1 cannabinoid to terpene blend with dense amounts of CBD and a custom terpene blend that features concentrated amounts alpha pinene to inspire alertness and improve mood. MINERAL creates synergistic relationships between Colorado grown cannabinoids and terpenes for targeted benefits. FOCUS is a terpene forward formula to reduce inflammation, improve mood and increase energy levels.
Your skin needs immediate attention after a shave, when it's most susceptible to burn and infection. Help it rebound pain free with this soothing aloe-packed aftershave lotion from Brickell. It also uses coconut oil to nourish and rehydrate the skin, as well as hyaluronic acid to help retain this moisture and restore elasticity. After shaving, splash the face with cold water to tighten pores. Then, apply a nickel-sized amount of product to the freshly shorn skin.
Just like their combs, Bixby's Beard Pick is the ideal tool for taming and styling your beard. Made from premium acetate, the same material found on your high-end sunglasses, the pick itself is just as stylish as you are in its tobacco tortoise finish. It arrives in a tin carrying case for throwing the pick in your bag and ensuring it's safety protected.
This soothing and smoothing soap invigorates the hands and senses with the scent of rose, cinnamon, and patchouli. It's a spinoff of Hudson Made's similarly named shower soap, and smells every bit as refreshing. Sulfate-free and ideal for sensitive hands, especially in the kitchen.
St. Patrick's day is this weekend, so you might want to go ahead and pick up some of the Hangover Tonic from Wooden Spoon Herbs. If you happen to have one too many Irish car bombs or too much green beer (or both), a couple of drops of this Hangover Tonic will help you bounce back Monday morning. It's packed with milk thistle seed and St. John's Wort that support healthy liver function while protecting it from damage. This helps those nasty toxins that are making you feel miserable move through your system faster — getting you back to feeling like yourself in no time. As for the ill-advised selfies with the leprechaun, it can't help you there.
Usage: Take 10 to 30 drops as needed.
Bixby prides itself for pinpointing the "optimum drag" of a comb: That is, the ideal distance between each tooth (1.3mm) and length of each (19mm). This provides gentle separation for detangling or distributing product without any tugging or torturing. Available in solid black or tobacco and carved from a single sheet of premium acetate, the fine-tooth comb from Bixby comes with a tin case for easy storage while traveling. The fine tooth comb is ideal for taming flyaways and facial hair, and best used for tighter and more polished styles.
Founded in Winston-Salem, North Carolina, Fulton & Roark produces some of the best solid cologne around. As their first line of products, their everyday solid colognes led to a full line of products that include shaving, soaps, shampoos, and innovative aftershave cloths that utilize quality ingredients and are made for guys on the go. The solid colognes and aftershave cloths in particular are ideal for air travel when space is a premium and liquids are cumbersome.
A more compact version of their full set, the Czech & Speake Air-Safe Manicure Set was built with globetrotters in mind. As the name implies, the manicure set is configured to comply with international carry-on restrictions, meaning it won't cause a fuss at security. Each of the Teflon-coated, TSA-friendly instruments are handmade in Germany with rounded edges. The Teflon coating repels grime, ensures proper hygiene, and provides longevity of the tools, while the rounded edges make air travel possible. Housed inside a grained leather case are nail-clippers, tweezers, safety scissors, and a crystal nail file. It's compact, handsome, and ready for a hop across the pond inside your carry aboard bag.


text:
National Sheep Association
NSA Community
Here you will find the latest updates on NSA activity, and responses to industry news or Government announcements.
For press enquiries, please see the latest statements below or contact [email protected].
7th January 2022 NSA Chief Executive reports from Oxford Farming Conference Ongoing disruption due to Covid19 meant the Oxford Farming Conference (OFC) and Real farming Conference both took place online for a second year running this week. NSA Chief Executive Phil Stocker linked into much of the OFC event and spoke in one of the fringe sessions. Read More
7th January 2022 NSA encouraged to see next stage of ELMS development underway NSA is welcoming the unveiling of the next two strands of the Environmental Land Management Scheme (ELMS) providing more guidance to farmers in England. Following quickly on the heels of details of the Sustainable Farming Scheme (SFI) released in December the two new schemes, Local Nature Recovery and Landscape Recovery, focus on creating and restoring natural habitats within the farmed environment. Read More
5th January 2022 NSA Breakfast Clubs return to inform and update in 2022 Following on from its success in 2021, the National Sheep Association (NSA)'s series of Breakfast Club webinars is returning for the new year with speakers looking forward to sharing useful and up to date information with sheep farmers and other interested parties throughout 2022. Read More
4th January 2022 'Time to focus on the sheep farmers and shepherds', says National Sheep Association, as it sets out commitments for 2022 As the new year begins the National Sheep Association (NSA) is looking ahead at its overarching priorities, setting out a series of commitments for 2022 to ensure security and success for the industry it works hard to protect. With 2022 set to offer some real opportunities for UK sheep farming but some equally impactful threats, NSA Chief Executive Phil Stocker explains the Association's focus for the year ahead and is keen to give emphasis to the role the farmer has to play. Read More
23rd December 2021 Two opportunities to join the National Sheep Association and provide a voice for the UK sheep industry NSA is looking for a Policy Manager to join the UK-wide team from March 2022 and help drive the high-profile work of the organisation at a time of immense change for the sheep sector. This role is to manage and coordinate NSA's policy work, ensuring effective internal and external communication and consistent representation of NSA at stakeholder and other relevant meetings. Read More
22nd December 2021 NSA astonished by detrimental impact of Australia deal on UK Farming National Sheep Association (NSA) is astonished by the continued dismissal by the Government on the detrimental impact the UK-Australia trade deal will have on UK Farming. Following the Department for International Trade's (DIT) own impact assessment which states a 'reallocation of resources away from agriculture, forestry, and fishing' of around £94 million, the Association is pressing for proper and urgent scrutiny of the deal by Parliament and the Trade and Agriculture Commission (TAC). Read More
17th December 2021 NSA angered by trade deal that threatens UK sheep sector Following the formal signing of a new trade deal between the UK and Australia yesterday (Thursday) evening that will give Australia free access to the UK sheep meat market NSA is expressing its grave disappointment. NSA Chief Executive Phil Stocker says: "NSA is disappointed that this deal has been pushed through and now signed without any resolution on how TRQs could be managed in a way to limit potential damage to the UK's own domestic trade. Read More
15th December 2021 NSA adds support to levy board decision to remove assurance scheme grant In response to the AHDB announcement this week that it is to discontinue its annual grant to Red Tractor Assurance, funded through levy payer funds, NSA Chief Executive Phil Stocker comments: "While NSA is a strong supporter of the principle of farm assurance in order to demonstrate the sector's credentials to the market place, and has also supported Red Tractor Assurance for many years, NSA is in agreement with this decision made by AHDB. Read More
13th December 2021 Add sustainably produced, British lamb to your Christmas shopping list says NSA As many consumers begin preparations for their festive dining the National Sheep Association (NSA) is recommending lamb as an additional meat to add to shopping baskets, highlighting its sustainable credentials as a positive reason to enjoy it this Christmas. Read More
8th December 2021 NSA passes on advice on how best to deal with a sheep worrying attack The National Sheep Association (NSA) is urging sheep farmers across the UK to update themselves with the latest guidance on how best to deal with a sheep worrying by dog attack on their flock. As the end of the year approaches, many sheep farmers will once again look back having experienced an increase in sheep worrying by dog attacks on their flocks during 2021. Read More
The Sheep Centre
WR13 6PH
VAT registered (No. 273 3469 44)
NSA is a Company Limited by Guarantee, registered in England (No. 37818).
It is a registered charity in England and Wales (No. 249255) and in Scotland (No. SCO42853).
NSA complies with the General Data Protection Regulation (GDPR).
© Copyright 2021 NSA, All Rights Reserved | Designed and Developed by H&H Reeds


text:
EQT V acquires Eurocom and CableTel
EQT V, a European private equity fund, has agreed to acquire Bulgaria's two leading cable TV operators, Eurocom and CableTel. EQT V intends to merge the two entities, invest into the new company together with CableTel's current shareholder, Ron Finley, and create a major player in the Bulgarian digital TV and broadband markets.
EQT V is acquiring 100 per cent of the shares in Eurocom from US-based private equity house Warburg Pincus. At the same time, EQT V acquires 70 per cent of the shares in CableTel from Gene Phillips. CableTel's other shareholder Ron Finley will remain minority shareholder by rolling over his 30 per cent shareholding in CableTel into the merged company and investing further equity.
The merged company will be the clear market leader in both Bulgaria and Macedonia with annual revenues of approximately E70 million and some 0.5 million households connected to the network.
The Bulgarian cable TV market is fragmented and digital TV and broadband penetration rates are lower than in the European Union as a whole. EQT V intends to accelerate the move into digital TV, broadband and telephony services by continuously investing into and offering a broad range of new and competitive services in combination with superior customer service.
EQT has a strong track record of fostering growth and value creation in the more than 70 companies that it has invested in over the last 14 years. EQT has previously developed Swedish cable TV operators StjärnTV and Com Hem into leading local providers of digital TV, broadband and telephony.
Macquarie, MBK seal C&M deal
Teracom takes control of Boxer
NDS buy out complete
PT Multimedia's acquires TVTEL
Shaw to sell TVB
Categories: Articles, Business, M&A


text:
\section{Introduction}
\label{sec:Intro}
Large-scale data collection is an integral component of the modern machine learning pipeline. The success of a learning algorithm critically depends on the quantity and quality of the input data. Although vast amounts of data are available, the information is often of a personal or sensitive nature. Models can leak substantial information about individual participants, even if we only release predictions or outputs ~\cite{fredrikson2014privacy,fredrikson2015model,shokri2017membership}. Privacy is, therefore, an important design factor for machine learning algorithms.

To protect against diverse and sophisticated attacks, $\epsilon$-differential privacy has emerged as a theoretically rigorous definition of privacy with robust guarantees~\cite{dwork2006differential}. Informally, an algorithm is differentially private if the inclusion (or exclusion) of any specific data record cannot substantially alter the output. In this paper, we consider the task of privately releasing a function $f_{\mathcal{D}}(q) \to \mathbb{R}$ that applies a pairwise operation $k(x,q)$ to each record $\{x_1, ... x_N\}$ in a dataset $\mathcal{D}$ and returns the sum. Although one could directly evaluate $f_{\mathcal{D}}(q)$ and release the result with the exponential mechanism, we can only evaluate $f_{\mathcal{D}}$ a finite number of times before the privacy budget runs out. The function release problem is to release a private summary $\mathcal{S}_{\mathcal{D}}$ of $f_{\mathcal{D}}$ that can answer an unlimited number of queries~\cite{hall2013differential}. We also seek error bounds for all queries\footnote{By ``all queries'', we mean all values of $q$. There is no polynomial algorithm for all general queries~\cite{ullman2013answering}.}, not just the global minimum as with empirical risk minimization~\cite{chaudhuri2011differentially} or a finite set of linear queries as in~\cite{hardt2012simple}.

Many machine learning problems can be solved in the function release framework. When $k(x,q)$ is a loss function, we can train a model by minimizing $f_{\mathcal{D}}$. When $k(x,q)$ is a kernel function, $f_{\mathcal{D}}$ is the kernel density estimate - a convenient way to approximate the likelihood function in a classification model. As a result, function release has received considerable theoretical attention. There are elegant, general and powerful techniques to release essentially any function~\cite{hall2013differential,alda2017bernstein,formalmirshani19a,wang2013efficient}. However, function release has not yet been widely adopted in practice because existing methods fail to scale beyond small, low-dimensional datasets. The practical utility of function release is plagued by issues such as quadratic runtime and exponential memory requirements. For instance, many algorithms release $f_{\mathcal{D}}$ via function approximation over an interpolation lattice, but the size of the lattice grows exponentially with dimensions.

\paragraph{Our Contribution:}

In this work, we propose a scalable approach to function release using the RACE sketch, a recent development in (non-private) data streaming algorithms. The RACE sketch can approximate pairwise kernel sums on streaming data for a specific class of kernels known as \textit{locality sensitive hash (LSH) kernels}. By restricting our attention to LSH kernel sums, we obtain fast streaming algorithms for private function release. RACE sketches consist of a small array of integers ($\sim$4 MB) and are well-suited to large-scale distributed settings. Private sketches are sought after by practitioners because they combine algorithmic efficiency with privacy guarantees~\cite{apple2017}. We show how to construct a private RACE sketch $\mathcal{S}_{\mathcal{D}}$ for the LSH kernel sum $f_{\mathcal{D}}$. We prove that our sketch is $\epsilon$-differentially private and derive pointwise error bounds for our approximation to $f_{\mathcal{D}}$. Our bounds are competitive with existing methods but come at a smaller computation cost. RACE easily scales to datasets with hundreds of dimensions and millions of entries.

Although the restriction to LSH kernels might seem limiting, we argue that most machine learning tasks can be performed using the RACE sketch. We show how to express classification, linear regression, kernel density estimation (KDE), anomaly detection and mode finding in terms of specific LSH kernel compositions. We conduct an exhaustive set of experiments with KDE, classification and linear regression. Our experiments show that RACE can release useful functions for many machine learning tasks with a competitive privacy-utility tradeoff.

\section{Background}
We consider a dataset $\mathcal{D}$ of $N$ points in $\mathbb{R}^d$. Although our analysis naturally extends to any metric space, we restrict our attention to $\mathbb{R}^d$ for the sake of presentation.

\subsection{Differential Privacy}
We use the well-established definition of differential privacy~\cite{dwork2006differential}.

\begin{definition}
\label{def:priv} \textbf{Differential Privacy}~\cite{dwork2006differential} A randomized function $\mathcal{A}$ is said to provide $(\epsilon,\beta)$-differential privacy if for all neighboring databases $\mathcal{D}$ and $\mathcal{D}'$ (which differ in at most one element) and all subsets $S$ in the codomain of $\mathcal{A}$,
$$\mathrm{Pr}[\mathcal{A}(\mathcal{D}) \in S] \leq e^{\epsilon}\mathrm{Pr}[A(\mathcal{D}')\in S] + \beta$$
\end{definition}
The parameter $\epsilon$ is the privacy budget. The privacy budget limits the amount of information that $\mathcal{A}(\mathcal{D})$ can leak about any individual element of $\mathcal{D}$. If $\beta > 0$, then $\mathcal{A}(\mathcal{D})$ might leak more information, but only with probability up to $\beta$. In this paper, we consider $\beta = 0$, which is simply called ``$\epsilon$-differential privacy.'' The Laplace mechanism~\cite{dwork2006differential} is a general method to satisfy $\epsilon$-differential privacy. By adding zero-mean Laplace noise to a real-valued function, we obtain differential privacy if the noise is scaled based on the sensitivity of the function (Definition~\ref{def:sens}).
\begin{definition}
\label{def:sens}
\textbf{Sensitivity}~\cite{dwork2006differential} For a function $\mathcal{A}:\mathcal{D}\rightarrow \mathbb{R}^d$, the L1-sensitivity of $\mathcal{A}$ is
$$\Delta = \sup||\mathcal{A}(\mathcal{D}) - \mathcal{A}(\mathcal{D}')||_1$$
where the supremum is taken over all neighboring datasets $\mathcal{D}$ and $\mathcal{D}'$
\end{definition}
\begin{theorem}\textbf{Laplace Mechanism}~\cite{dwork2006differential} Let $\mathcal{A} : \mathcal{D} \rightarrow \mathbb{R}^d$ be a non-private function with sensitivity $\Delta$ and let $\mathbf{z}\sim \mathrm{Lap}(\Delta / \epsilon)^d$ ($d$-dimensional i.i.d Laplace vector). Then the function $\mathcal{A}(\mathcal{D}) + \mathbf{z}$ provides $\epsilon$-differential privacy.
\end{theorem}

\subsection{Related Work}
There are several techniques to release the kernel sum $f_{\mathcal{D}}$ with differential privacy. A common approach is to decompose $f_{\mathcal{D}}$ into a set of weighted basis functions~\cite{wasserman2010statistical}. We truncate the basis expansion to $m$ terms and represent $f_{\mathcal{D}}$ as a set of weights in $\mathbb{R}^m$. The weights are made private via the Laplace mechanism and used to release a private version of $f_{\mathcal{D}}$. Each basis term in the representation increases the quality of the approximation but degrades the quality of the weights, since the privacy budget $\epsilon$ is shared among the $m$ weights. This is a bias-variance tradeoff: we trade variance in the form of Laplace noise with bias in the form of truncation error. The Fourier basis~\cite{hall2013differential}, Bernstein basis~\cite{alda2017bernstein}, trigonometric polynomial basis~\cite{wang2013efficient} and various kernel bases have all been used for private function release. Such methods are most effective when $f_{\mathcal{D}}$ is a smooth function.

An alternative set of techniques rely on functional data analysis ~\cite{formalmirshani19a} or synthetic databases ~\cite{hardt2012simple,balog2018differentially}. In~\cite{formalmirshani19a}, the authors use densities over function spaces to release a smoothed approximation to $f_{\mathcal{D}}$. The main idea of ~\cite{hardt2012simple} and ~\cite{balog2018differentially} is to release a set of weighted synthetic points that can be used to estimate $f_{\mathcal{D}}$. Table~\ref{table:relatedwork} compares these methods based on approximation error and computation.

\begin{table}[t]
\begin{center}
\begin{tabular}{ l | l | l | l }
\toprule
Method & Error Bound & Runtime & Comments \\
\midrule
\makecell[l]{Bernstein\\polynomials~\cite{alda2017bernstein}} & $d^{\frac{d}{d+H}}\left(\frac{1}{\epsilon}\log \frac{1}{\delta} \right)^{\frac{H}{d+H}}$ & $O(dNM^d)$& \makecell[l]{$M\geq 2$. Memory is also\\ exponential in $d$.} \\
\midrule
\makecell[l]{PFDA~\cite{formalmirshani19a}} &
$\frac{2}{\epsilon}\sqrt{\log \frac{2}{\beta} \log \frac{1}{\delta} \frac{C}{\phi}}$
& $O(dN^2)$ & \makecell[l]{$C$ and $\phi$ are task-dependent\\ $(\epsilon,\beta)$-differential privacy } \\
\midrule
\makecell[l]{MWEM~\cite{hardt2012simple}} & $N^{\frac{2}{3}}\left(\frac{\log N \log |Q|}{\epsilon}\right)^{1/3}$ & $O(dN|Q|)$ &  \makecell[l]{$Q$ is a set of query points. Holds \\ with probability $1 - 1 / \mathrm{poly}(|Q|)$ }\\
\midrule
\makecell[l]{Trigonometric\\polynomials~\cite{wang2013efficient}} & $\frac{1}{\epsilon}N^{\frac{2d}{2d+H}}$ & $O(dN^{1 + \frac{d}{2d+H}})$ & \makecell[l]{The result holds with probability\\ $1 - \delta$ for $\delta\geq10e^{-\frac{1}{5} N^{d / 2d + K}}$ }\\
\midrule
\makecell[l]{This work} & $\left(\frac{N}{\epsilon}\log \frac{1}{\delta}\right)^{\frac{1}{2}}$ & $O(dN)$ & \makecell[l]{
Applies only for LSH kernels. \\ Efficient streaming algorithm.}\\
\bottomrule
\end{tabular}
\caption{Summary of related methods to release the kernel sum $f_\mathcal{D} = \sum_\mathcal{D} k(\mathbf{x},\mathbf{q})$ for an $N$ point dataset $\mathcal{D}$ in $\mathbb{R}^d$. Unless otherwise stated, the error is attained with probability $1 - \delta$ and $\epsilon$-differential privacy. We hide constant factors and adjust results to estimate $f_\mathcal{D}$ rather than the KDE ($N^{-1}f_\mathcal{D}(\mathbf{q})$) when necessary. $H$ is a kernel smoothness parameter.}
\label{table:relatedwork}
\end{center}
\vspace{-0.5cm}
\end{table}

\subsection{Locality-Sensitive Hashing}
\label{sec:LSH}
\paragraph{LSH Functions:} An LSH family $\mathcal{F}$ is a family of functions $l(\mathbf{x}): \mathbb{R}^{d}\to \mathbb{Z}$ with the following property: Under $l(\mathbf{x})$, similar points have a high probability of having the same hash value. We say that a collision occurs whenever two points have the same hash code, i.e. $l(\mathbf{x}) = l(\mathbf{y})$. In this paper, we use a slightly different definition than the original~\cite{indyk1998approximate} because we require the collision probability at all points.

\begin{definition}
\label{def:lsh}
We say that a hash family $\mathcal{F}$ is locality-sensitive with collision probability $k(\cdot,\cdot)$ if for any two points $\mathbf{x}$ and $\mathbf{y}$ in $\mathbb{R}^d$, $l(\mathbf{x}) = l(\mathbf{y})$ with probability $k(\mathbf{x},\mathbf{y})$
under a uniform random selection of $l(\cdot)$ from $\mathcal{F}$.
\end{definition}

\paragraph{LSH Kernels:} When the collision probability $k(\mathbf{x},\mathbf{y})$ is a monotone decreasing function of the distance metric $\mathrm{dist}(\mathbf{x},\mathbf{y})$, one can show that $k$ is a positive semidefinite kernel function~\cite{coleman2020race}. We say that a kernel function $k(\mathbf{x},\mathbf{y})$ is an \textit{LSH kernel} if it forms the collision probability for an LSH family. For a kernel to be an LSH kernel, it must obey the conditions described in~\cite{chierichetti2012preserving}. A number of well-known LSH families induce useful kernels~\cite{gionis1999similarity}. For example, there are LSH kernels that closely resemble the cosine, Laplace and multivariate Student kernels~\cite{coleman2020race}.

\subsection{RACE Sketch}
LSH kernels are interesting because there are efficient algorithms to estimate the quantity
\begin{equation}
    f_\mathcal{D}(\mathbf{q}) = \sum_{\mathbf{x}\in\mathcal{D}} k(\mathbf{x},\mathbf{q})
\end{equation}
when $k(\mathbf{x},\mathbf{q})$ is an LSH kernel. In~\cite{coleman2020race}, the authors present a one-pass streaming algorithm to estimate $f_\mathcal{D}$. The algorithm produces a RACE (Repeated Array of Count Estimators) sketch $\mathcal{S}_\mathcal{D}\in \mathbb{Z}^{R\times W}$, a 2D array of integers that we index using LSH functions. This array is sufficient to report $f_\mathcal{D}(\mathbf{q})$ for any query $\mathbf{q}\in\mathbb{R}^d$. We begin by constructing $R$ functions $\{l_1(\mathbf{x}),...l_R(\mathbf{x})\}$ from an LSH family $\mathcal{F}$ with the desired collision probability. When an element $\mathbf{x}$ arrives from the stream, we hash $\mathbf{x}$ to get $R$ hash values, one for each row of $\mathcal{S}_\mathcal{D}$. We increment row $i$ at location $l_i(\mathbf{x})$ and repeat for all elements in the dataset. To approximate $f_\mathcal{D}(\mathbf{q})$, we return the mean of $\mathcal{S}_\mathcal{D}[r,l_r(\mathbf{q})]$ over the $R$ rows.

\paragraph{RACE for Density Estimation:} This streaming algorithm approximates the kernel density estimate (KDE) for all possible queries in a single pass. RACE is also a mergable summary with an efficient distributed implementation. In practice, RACE estimates the KDE with 1\% error using a 4 MB array, even for large high-dimensional datasets. To prove rigorous error bounds, observe that each row of $\mathcal{S}_\mathcal{D}$ is an unbiased estimator of $f_\mathcal{D}$. The main theoretical result of~\cite{coleman2020race} is stated below as Theorem~\ref{thm:race}.

\begin{theorem}
\label{thm:race}
\textbf{Unbiased RACE Estimator}\cite{coleman2020race} Suppose that $X$ is the query result for one of the rows of $\mathcal{S}_\mathcal{D}$. That is, $X= \mathcal{S}_\mathcal{D}[r,l_r(\mathbf{q})]$

\noindent\begin{minipage}{0.3\textwidth}
$$ \mathbb{E}[X] = f_\mathcal{D}(\mathbf{q})$$
\end{minipage}
\begin{minipage}{0.7\textwidth}
$$ \mathrm{var}(X)\leq \left(\tilde{f}_\mathcal{D}(\mathbf{q})\right)^2 = \left(\sum_{\mathbf{x}\in\mathcal{D}}\sqrt{k(\mathbf{x},\mathbf{q})}\right)^2$$
\end{minipage}

\end{theorem}

\paragraph{RACE for Empirical Risk Minimization:} Theorem~\ref{thm:race} holds for all collision probabilities, even those that are neither continuous nor positive semidefinite. Thus, we can use RACE to approximate the empirical risk for a variety of losses. Suppose we are given a dataset $\mathcal{D} = \{\mathbf{z}_1...\mathbf{z}_N\}$ of training examples and a loss function $L(\theta, \mathbf{z})$, where $\theta$ is a parameter that describes the model. Empirical Risk Minimization (ERM) consists of finding a parameter $\theta$ (and thus a model) that minimizes the mean loss over the training set.
RACE can approximate the empirical risk when $L(\theta, \mathbf{z})$ can be expressed in terms of LSH collision probabilities. Although we cannot analytically find the gradient of the RACE count values, derivative-free optimization~\cite{conn2009introduction} is highly effective for RACE sketches. With the right LSH family, RACE can perform many machine learning tasks in the one-pass streaming setting.

\section{Private Sketches with RACE}

We propose a private version of the RACE sketch. We obtain $\epsilon$-differential privacy by applying the Laplace mechanism to each count in the RACE sketch array. Algorithm~\ref{alg:sketch} introduces a differentially private method to release the RACE sketch, illustrated in Figure~\ref{fig:privateRACE}. It is straightforward to see that Algorithm~\ref{alg:sketch} only requires $O(NR)$ hash computations. Assuming fixed $R$, we have $O(dN)$ runtime.

\begin{minipage}{0.46\textwidth}
\begin{algorithm}[H]
\begin{algorithmic}
\STATE {\bf Input:} Dataset $\mathcal{D}$, privacy budget $\epsilon$, LSH family $\mathcal{F}$, dimensions $R\times W$
\STATE {\bf Output:} Private sketch $\mathcal{S}_{\mathcal{D}} \in \mathbb{Z}^{R\times W}$
\STATE {\bf Initialize:} $R$ independent LSH functions $\{l_1, ..., l_R\}$ from the LSH family $\mathcal{F}$
\STATE $\mathcal{S}_{\mathcal{D}} \leftarrow \mathbf{0}^{R\times W}$
\FOR{$\mathbf{x} \in \mathcal{D}$}
    \FOR {$r$ in $1$ to $R$}
        \STATE Increment $\mathcal{S}_{\mathcal{D}}[r,l_r(\mathbf{x})]$
    \ENDFOR
\ENDFOR
\STATE $\mathcal{S}_{\mathcal{D}} = \left \lfloor{\mathcal{S}_{\mathcal{D}} + Z}\right \rfloor$ where $Z \overset{\text{iid}}{\sim} \mathrm{Lap}(R\epsilon^{-1})$
 \end{algorithmic}
  \caption{Private RACE sketch}
 \label{alg:sketch}
\end{algorithm}
\end{minipage}
\begin{minipage}{0.46\textwidth}
\begin{algorithm}[H]
\begin{algorithmic}
\STATE {\bf Input:} Sketch $\mathcal{S}_{\mathcal{D}}$, query $\mathbf{q}$, the same $R$ LSH functions from Algorithm~\ref{alg:sketch}
\STATE {\bf Output:} Estimate of $N^{-1}f_{\mathcal{D}}(\mathbf{q})$
\STATE $\widehat{N} \leftarrow R^{-1} \sum_{i,j} \mathcal{S}_{\mathcal{D}}[i,j]$
\STATE $\hat{f}_{\mathcal{D}} \leftarrow 0$
\FOR {$r$ in $1$ to $R$}
    \STATE $\hat{f}_{\mathcal{D}} = \hat{f}_{\mathcal{D}} +
    \frac{1}{R} \mathcal{S}_{\mathcal{D}}[r,l_r(\mathbf{q})]$
\ENDFOR
\STATE \textbf{Return:} $\hat{f}_{\mathcal{D}}/ \widehat{N}$
\end{algorithmic}
\caption{RACE query}
 \label{alg:query}
\end{algorithm}
\vspace{0.55cm}
\end{minipage}

\subsection{Privacy}
For the purposes of Definition~\ref{def:priv}, we consider the function $\mathcal{A}: \mathcal{D} \rightarrow \mathbb{R}^{R\times W}$ to be Algorithm~\ref{alg:sketch}. The codomain of $\mathcal{A}$ is the set of all RACE sketches with $R$ rows and $W$ columns. Our main theorem is that the value returned by $\mathcal{A}(\mathcal{D})$ is $\epsilon$-differentially private. That is, our sketch is differentially private. We begin the proof by applying the parallel composition theorem~\cite{dwork2014algorithmic} to the counters in one row, which each see a disjoint partition of $\mathcal{D}$. Then, we apply the sequential composition theorem~\cite{dwork2014algorithmic} to the set of $R$ rows. Due to space limitations, we defer the full proof to the Appendix.

\begin{theorem}
For any $R > 0$, $W > 0$, and LSH family $\mathcal{F}$, the output of Algorithm~\ref{alg:sketch}, or the RACE sketch $\mathcal{S}_{\mathcal{D}}$, is $\epsilon$-differentially private.
\end{theorem}

\subsection{Utility}

Since one can construct many learning algorithms from a sufficiently good estimate of $f_{\mathcal{D}}$~\cite{alda2017bernstein}, we focus on utility guarantees for the RACE estimate of $f_{\mathcal{D}}(\mathbf{q})$. Since RACE is a collection of unbiased estimators for $f_{\mathcal{D}}(\mathbf{q})$, our proof strategy is to bound the distance between the RACE estimate and the mean. To bound the variance of the private RACE estimator, we add the independent Laplace noise variance $2R^2\epsilon^{-2}$ to the bound from Theorem~\ref{thm:race}. Theorem~\ref{thm:tradeoff} follows using the median-of-means procedure.

\begin{theorem}
\label{thm:tradeoff}
Let $\hat{f}_{\mathcal{D}}(\mathbf{q})$ be the median-of-means estimate using an $\epsilon$-differentially private RACE sketch with $R$ rows and $\tilde{f}_{\mathcal{D}}(\mathbf{q}) = \sum_{\mathcal{D}} \sqrt{k(\mathbf{x},\mathbf{q})}$. Then with probability $1 - \delta$,
$$|\hat{f}_{\mathcal{D}}(\mathbf{q}) - f_{\mathcal{D}}(\mathbf{q})| \leq \left(\frac{\tilde{f}^2_\mathcal{D}(\mathbf{q})}{R} + \frac{2}{\epsilon^2}R \right)^{1/2}\sqrt{32 \log 1/\delta}$$

\end{theorem}

Theorem~\ref{thm:tradeoff} suggests a tradeoff for which there is an optimal value of $R$. If we increase the number of rows $R$ in the sketch, we improve the estimator but must add more Laplace noise. To get our main utility guarantee, we choose an optimal $R$ that minimizes the error bound.

\begin{corollary}
\label{cor:util}
Put $R = \lceil\frac{1}{\sqrt{2}}\tilde{f}_\mathcal{D}(\mathbf{q})\epsilon\rceil$. Then the approximation error bound is
$$|\hat{f}_{\mathcal{D}}(\mathbf{q}) - f_{\mathcal{D}}(\mathbf{q})| \leq 16\left(\frac{\tilde{f}_\mathcal{D}(\mathbf{q})}{\epsilon}\log 1/\delta\right)^{1/2} \leq 16\sqrt{\frac{N}{\epsilon}\log 1/\delta}$$
\end{corollary}

If we divide both sides of Corollary~\ref{cor:util} by $f_{\mathcal{D}}$, we bound the relative (or percent) error rather than the absolute error. Corollary~\ref{cor:util} suggests that it is hardest to achieve a small relative error when $f_{\mathcal{D}}$ is small. This agrees with our intuition about how the KDE should behave under differential privacy guarantees. Fewer individuals make heavy contributions to $f_{\mathcal{D}}$ in low-density regions than in high-density ones, so the effect of the noise is worse.

\begin{figure*}[t]
\centering
\includegraphics[width=\textwidth,keepaspectratio]{images/private.png}
\vspace{-0.5cm}
\caption{Illustration of Algorithm~\ref{alg:sketch} for $\mathcal{S}_{\mathcal{D}}\in \mathbb{Z}^{3\times6}$. We hash each element in the stream with LSH functions $\{l_1,l_2,l_3\} \in \mathcal{F}$ having collision probability $k(\mathbf{x},\mathbf{y})$. In this example, $l_1(\mathbf{x}_i) = 3$, $l_2(\mathbf{x}_i) = 2$ and $l_3(\mathbf{x}_i) = 6$. We increment the highlighted cells. The addition of the Laplace noise is not shown in the figure, but is done by perturbing each count in $\mathcal{S}_{\mathcal{D}}$.}
\label{fig:privateRACE}
\vspace{-0.2cm}
\end{figure*}

\section{Applications}

Because RACE can release LSH kernel sums, our sketch is broadly useful for many algorithms in machine learning. In particular, we discuss private density estimation, classification, regression, mode finding, anomaly detection and diversity sampling using RACE.

\textbf{Kernel Density Estimation:} To use RACE for KDE, we select one or more LSH kernels and construct sketches with Algorithm~\ref{alg:sketch}. We require one RACE sketch for each kernel and bandwidth setting, and we return the result of Algorithm~\ref{alg:query}. Note that the query no longer has access to $N$, the number of elements in the private dataset. Therefore, we estimate $N$ directly from the private sketch.

\textbf{Mode Finding:} One can apply gradient-free optimization to KDE to recover the modes of the data distribution. This works surprisingly well, but in general the KDE is a non-convex function. One can also apply linear programming techniques that use information about the hash function partitions to identify a point in a partition with the highest count values.

\textbf{Naive Bayes Classification:} Using kernel density classification, a well-developed result from statistical hypothesis testing~\cite{john1995estimating}, we can construct classifiers with RACE under both the maximum-likelihood and maximum a posteriori (MAP) decision rules. Suppose we are given a training set $\mathcal{D}$ with $M$ classes $C_1,... C_M$ and a query $\mathbf{q}$. We can represent the empirical likelihood $\mathrm{Pr}[\mathbf{q}|C_i,\mathcal{D}]$ with a sketch of the KDE for class $i$. Algorithm~\ref{alg:query} returns an estimate of this probability, which may be used directly by a naive Bayes classifier or other type of probabilistic learner.

\textbf{Anomaly Detection / Sampling:} Anomaly detection can be cast as a KDE classification problem. If Algorithm~\ref{alg:query} reports a low density for $\mathbf{q}$, then the training set contains few elements similar to $\mathbf{q}$ and thus $\mathbf{q}$ is an outlier. This principle is behind the algorithms in~\cite{luo2018arrays} and~\cite{coleman2019diversified}.

\textbf{Linear Regression:} If we use an asymmetric LSH family~\cite{shrivastava2014asymmetric}, we can construct a valid surrogate loss for the linear regression loss $(\langle[\mathbf{x},y],[\theta,-1]\rangle)^2$. The key insight is that the signed random projection (SRP) LSH kernel is monotone in the inner product. If we apply SRP to both $[\mathbf{x},y]$ and $-[\mathbf{x},y]$, we obtain an LSH kernel with two components. One component is monotone increasing with the inner product, while the other is monotone decreasing. The resulting surrogate loss is a monotone function of the L2 loss and a convex function of $\theta$.

\section{Experiments}

\begin{table*}[t]
  \centering
  \begin{tabular}{ l|c|c|c|l|l  }
\toprule
Dataset & $N$ & $d$ & $\sigma$ & Description & Task\\
\midrule
NYC & 25k & 1 & 5k & NYC salaries (2018) & \multirow{4}[1]{*}{KDE} \\
SF & 29k & 1 & 5k & SF salaries (2018) &  \\
skin & 241k & 3 & 5.0 & RGB skin tones & \\
codrna & 57k & 8 & 0.5 & RNA genomic data & \\
\midrule
nomao & 34k & 26 & 0.6 & User location data & \multirow{3}[1]{*}{Classification}\\
occupancy & 17k & 5 & 0.5 & Building occupancy & \\
pulsar & 17k & 8 & 0.1 & Pulsar star data & \\
\midrule
airfoil & 1.4k & 9 & - & Airfoil parameters and sound level & \multirow{3}[1]{*}{Regression}\\
naval & 11k & 16 & - & Frigate turbine propulsion &  \\
gas & 3.6k & 128 & - & Gas sensor, different concentrations & \\
\bottomrule
\end{tabular}
\caption{Datasets used for KDE and classification experiments. Each dataset has $N$ entries with $d$ features. $\sigma$ is the kernel bandwidth.}
\label{tab:datasets}
\vspace{-0.2cm}
\end{table*}

\begin{table}[t]
\centering
\begin{tabular}{c|c|c|c|c}
\toprule
& PFDA & Bernstein & KME & RACE \\
\midrule
Preprocess & $> 3$ days & 2.3 days & 6 hr & 13 min \\
Query & - & 6.2 ms & 1.2 ms & 0.4 ms \\
\bottomrule
\end{tabular}
\vspace{0.1cm}
\caption{Computation time for KDE on the skin dataset. Note that we were unable to run PFDA on this dataset. Instead, we report the PFDA runtime for the (smaller) NYC dataset.}
\label{tab:computation}
\vspace{-0.5cm}
\end{table}

We perform an exhaustive comparison for KDE, classification and regression. For KDE, we estimate the density of the salaries for New York City (NYC) and San Francisco (SF) city employees in 2018, as well as the high-dimensional densities for the skin and codrna UCI datasets. We use the same LSH kernel as the authors of~\cite{coleman2020race}. We use UCI datasets for the regression and classification experiments. Most private KDE methods require the data to lie in the unit sphere or cube; we scale the inputs accordingly. Table~\ref{tab:datasets} presents the datasets used in our experiments. We make the following considerations for our baseline methods.

\textbf{KDE:} We implemented several function release methods from Table~\ref{table:relatedwork}. We also compare against the Fourier basis estimator~\cite{hall2013differential,wasserman2010statistical} and the kernel mean embedding (KME) method~\cite{balog2018differentially}. We use the Python code released by~\cite{balog2018differentially} and implemented the other methods in Python. To give a fair comparison, we profiled and optimized each of our baselines. We were unable to run Fourier and PFDA in more than one dimension because they both require function evaluations on a Bravais lattice, which scales exponentially. We show the density estimates and their errors in Figure~\ref{fig:KDEResults}.

\textbf{Classification:} We compare a maximum-likelihood RACE classifier against a regularized logistic regression classifier trained using objective perturbation~\cite{chaudhuri2011differentially}. We average over the Laplace noise and report the accuracy on a held-out test set in Figure~\ref{fig:ClassResults}.

\textbf{Regression:} We compare RACE regression against five algorithms: sufficient statistics perturbation (SSP), objective perturbation (ObjPert), posterior sampling (OPS), and adaptive versions of SSP and OPS (AdaSSP and AdaOPS~\cite{wang2018revisiting}). We use the MatLab code released by~\cite{wang2018revisiting} and report the mean squared error (MSE) on a held-out test set in Figure~\ref{fig:RegressionResults}.

\begin{figure*}[t]
\centering
  \includegraphics[width=\textwidth,keepaspectratio]{images/F1.pdf}
\vspace{-0.5cm}
\caption{Privacy-utility tradeoff for private function release methods. We report the L2 function error and the mean relative error for 2000 held-out queries.}
\label{fig:KDEResults}
\vspace{-0.2cm}
\end{figure*}

\textbf{Computation: } Table~\ref{tab:computation} displays the computation time needed to construct a useful function release. Note that Bernstein release can run faster if we use fewer interpolation points (see Table~\ref{table:relatedwork}), but we still required at least 12 hours of computation for competitive results. The Bernstein mechanism requires many binomial coefficient evaluations, which were expensive even when we used optimized C code. KME requires a large-scale kernel matrix computation, and PFDA required several days for an expensive eigenvalue computation. RACE construction time varies based on $R$, but is substantially faster than all baselines.

\begin{figure*}[t]
\centering
  \includegraphics[width=\textwidth,keepaspectratio]{images/F2.pdf}
\vspace{-0.5cm}
\caption{Binary classification experiments. We show the privacy-utility tradeoff for a private logistic regression classifier and the RACE max-likelihood classifier. Average over 10 repetitions. }
\label{fig:ClassResults}
\end{figure*}
\begin{figure*}[t]
\centering
  \includegraphics[width=\textwidth,keepaspectratio]{images/F3.pdf}
\vspace{-0.5cm}
\caption{Linear regression experiments. We show the privacy-utility tradeoff for RACE and several other linear regression methods. Average over 10 repetitions. }
\label{fig:RegressionResults}
\vspace{-0.5cm}
\end{figure*}

\section{Discussion}

\paragraph{Function Release at Scale:}
RACE is ideal for private function release in large-scale distributed applications. Although PFDA and the Bernstein mechanism have the strongest error bounds, they required days to produce the experimental results in Figure~\ref{fig:KDEResults}. This is a serious barrier in practice - it would require $> 2^{128} \approx 10^{38}$ computations to run the Bernstein mechanism on the UCI gas dataset. The RACE sketch has a small memory footprint, inexpensive streaming updates and can quickly run on high-dimensional datasets. We believe that RACE can make differentially private function release a viable tool for real-world applications.

Our sketch is also convenient to use and deploy in a production environment because it is relatively insensitive to hyperparameters. In general, we found that density estimation requires a larger sketch with more rows $R$ than classification or regression. Classification and regression problems benefit from a smaller hash range $W$ than function release. Higher dimensional problems also require a larger hash range. However, these choices are not critical and any $R \in [100,1\text{k}]$ with $W \in [100,1\text{k}]$ will provide good results. Hyperparameter tuning can be accomplished using very little of the overall privacy budget.

\paragraph{Privacy and Utility:}
Our sketch beats interpolation-based methods for private function release when the dataset has more than a few dimensions and when $f_{\mathcal{D}}$ is not smooth. Depending on smoothness, our $\sqrt{N\epsilon^{-1}}$ error rate improves upon the rates of~\cite{wang2013efficient} and~\cite{hardt2012simple} for LSH kernels. In our experiments (Figure~\ref{fig:KDEResults}), the Bernstein mechanism outperforms RACE on the SF dataset but fails to capture the nuances of the NYC salary distribution, which has sharp peaks. RACE preserves the details of $f_{\mathcal{D}}$ because the Laplace noise can only make local changes to the RACE structure. Suppose we generate a Laplace noise that is unusually large for one particular RACE counter. Queries that do not involve the problematic counter are unaffected. If we perturb one or two of the most important weights of a series estimator, the changes propagate to all queries.

Although RACE can only estimate LSH kernels, the space of LSH kernels is sufficiently large that we can release many useful functions. Our RACE surrogate loss outperformed objective perturbation~\cite{chaudhuri2011differentially} and compared favorably to other linear regression baselines. RACE also performed well in our classification experiments for $\epsilon > 10^{-1}$, providing a competitive utility tradeoff for practical privacy budgets. Our experiments show that RACE can privately release useful functions for many machine learning problems.

\section{Conclusion}
We have presented RACE, a differentially private sketch that can be used for a variety of machine learning tasks. RACE is competitive with the state of the art on tasks including density estimation, classification, and linear regression. RACE sketches can be constructed in the one-pass streaming setting and are highly computationally efficient. At the same time, our sketch offers good performance on many machine learning tasks and is reasonably efficient in terms of the privacy budget. Given the utility, simplicity and speed of the algorithm, we expect that RACE will enable differentially private machine learning in large-scale settings.

\section*{Broader Impact}

Privacy is a serious concern in machine learning and a barrier to the widespread adoption of useful technology. For example, machine learning models can substantially improve medical treatments, inform social policy, and help with financial decision-making. While such applications are overall beneficial to society, they often rely on sensitive personal data that users may not want to disclose. For instance, a 2019 survey found that a large majority ($\sim$ 80\%) of Americans are concerned with but do not understand how technology companies use the data they collect~\cite{pew2019}. Data management practices have recently come under increased scrutiny, especially after the enactment of the European GDPR Act in 2018. To improve public confidence in their services, Google~\cite{erlingsson2014rappor} and Apple~\cite{apple2017} both advertise differential privacy as a central part of their data collection process.

Differential privacy provides strong protection against malicious data use but is difficult to apply in practice. This is particularly true for functional data release, which is often prohibitively expensive in terms of computation cost. Our work substantially reduces the computational cost of this useful technique to achieve differential privacy. By reducing the computational requirement, we make privacy more accessible and attainable for data publishers, since large computing infrastructure is no longer needed to release private information. This directly reduces the economic cost of data privacy, an important consideration for governments and private companies. Our sketching method also allows private functional release to be applied to more real-world datasets, making it easier to release high-dimensional information with differential privacy.

\section*{Appendix}
This section contains proofs and detailed discussion of the theorems.
\setcounter{theorem}{2}
\setcounter{corollary}{0}

\subsection*{Privacy}
We consider a function $\mathcal{A}: \mathcal{D} \rightarrow \mathbb{R}^{R\times W}$ that takes in a dataset and outputs an $R\times W$ RACE sketch. $\mathcal{A}$ is a randomized function, where the randomness is over the choice of the LSH functions and the (independent) Laplace noise. In this context, the $\epsilon$-differentially private query is the RACE sketch $\mathcal{S}_{\mathcal{D}}$. Our main theorem is that the sketch is $\epsilon$-differentially private. Because differential privacy is robust to post-processing, we can then query $\mathcal{S}_{\mathcal{D}}$ as many times as desired once the sketch is released.

\paragraph{Proof Sketch:} The sketch $\mathcal{S}_{\mathcal{D}}$ consists of $R$ independent rows. Each row consists of $W$ count values. First, we prove privately release \textit{one} of the rows by adding Laplace noise with variance $\epsilon^{-1}$ (i.e. sensitivity $\Delta = 1$). This is described in Lemma~\ref{lem:row}. Then, we apply Lemma~\ref{lem:row} with $\epsilon / R$ to release \textit{all} the rows, proving the main theorem.
\begin{lemma}
\label{lem:row}
Consider one row of the RACE sketch, and add independent Laplace noise $\mathrm{Lap}(1/\epsilon)$ to each counter. The row can be released with $\epsilon$-differential privacy.
\end{lemma}
\begin{proof}
Consider just one of the $W$ counters. It is easy to see that the sensitivity of the counter is 1 because changing a single element from the dataset can only create a change of $\pm1$ in the counter. By the Laplace mechanism, this counter can be released with $\epsilon$-differential privacy by adding Laplace noise $\mathrm{Lap}(1 / \epsilon)$.

The row contains $W$ counters, and we can release each one with the Laplace mechanism. Therefore, we have $W$ mechanisms $\mathcal{M}_1, ... \mathcal{M}_W$ (one to release each counter) which are independently $\epsilon$-differentially private. The parallel composition theorem~\cite{dwork2014algorithmic} states that if each mechanism is computed on a disjoint subset of the dataset, then we can compute all of the mechanisms (i.e. release all of the counters) with $\epsilon$-differential privacy.

This is indeed the case for the RACE sketch. Under the LSH function, each element in the dataset maps to exactly one of the $W$ counters. Thus, the $W$ counters are computed on disjoint subsets of the dataset and we can release them with $\epsilon$-differential privacy.
\end{proof}
\begin{theorem}
For any $R > 0$, $W > 0$, and LSH family $\mathcal{F}$, the output of Algorithm 1, or the RACE sketch $\mathcal{S}_{\mathcal{D}}$, is $\epsilon$-differentially private.
\end{theorem}
\begin{proof}
The sketch is composed of $R$ independent rows. Consider the mechanism from Lemma~\ref{lem:row} to release a single row with $\epsilon$-differential privacy. The sequential composition theorem~\cite{dwork2014algorithmic} states that given $R$ mechanisms $\mathcal{M}_1, ... \mathcal{M}_R$ (one to release each row) which are independently $\epsilon$-differentially private, we can compute all the mechanisms (i.e. release all rows) with $R\epsilon$-differential privacy.

To construct the sketch, we apply Lemma~\ref{lem:row} to each row with $\epsilon/R$ differential privacy by adding independent Laplace noise $\mathrm{Lap}(R / \epsilon)$ to each counter. The sequential composition theorem guarantees that we can release all $R$ rows with $\epsilon$-differential privacy.
\end{proof}

\subsection*{Utility}

First, we introduce the median-of-means trick in Lemma~\ref{lem:med}. The median-of-means procedure is a standard analysis technique with many applications in the streaming literature. Lemma~\ref{lem:med} is a special case of Theorem 2.1 from~\cite{alon1999space}.
\begin{lemma}
\label{lem:med}
Let $X_1,...X_R$ be $R$ i.i.d. random variables with mean $\mathbb{E}[X] = \mu$ and variance $\leq \sigma^2$. To get the median of means estimate $\hat{\mu}$, break the $R$ random variables into $k$ groups with $m = R/k$ elements in each group.
$$ \hat{\mu} = \mathrm{median}\left(\sum_{i = 1}^m X_i, ..., \sum_{i = (k - 1)m + 1}^{km} X_i\right)$$

Put $k = 8 \log(1/ \delta)$ and $m = R / k$\footnote{For simplicity, we suppose $k$ and $m$ are integers that evenly divide $R$. See~\cite{alon1999space} for the complete analysis.}. Then with probability at least $1 - \delta$, the deviation of the estimate $\hat{\mu}$ from the mean $\mu$ is
$$ |\hat{\mu} - \mu|\leq \sqrt{32\sigma^2 \frac{\log(1/\delta)}{R}}$$
\end{lemma}

Each row $r$ of the sketch provides an estimator $X_r$ to be used in the median-of-means technique. By adding the Laplace noise variance, we can find $\sigma^2$ in Lemma~\ref{lem:med} and provide performance guarantees.

\begin{theorem}
Let $\hat{f}_{\mathcal{D}}(\mathbf{q})$ be the median-of-means estimate using an $\epsilon$-differentially private RACE sketch with $R$ rows and $\tilde{f}_{\mathcal{D}}(\mathbf{q}) = \sum_{\mathcal{D}} \sqrt{k(\mathbf{x},\mathbf{q})}$. Then with probability $1 - \delta$,
$$|\hat{f}_{\mathcal{D}}(\mathbf{q}) - f_{\mathcal{D}}(\mathbf{q})| \leq \left(\frac{\tilde{f}^2_\mathcal{D}(\mathbf{q})}{R} + \frac{2}{\epsilon^2}R \right)^{1/2}\sqrt{32 \log 1/\delta}$$

\end{theorem}

\begin{proof}

To use median-of-means, we must bound the variance of the RACE estimate $X_r = \mathcal{S}_{\mathcal{D}}[r,l_r(q)]$ after we add the Laplace noise. Since the Laplace noise is independent of the LSH functions, we simply add the Laplace noise variance $2R^{2}\epsilon^{-2}$ to the variance bound in Theorem 2.
$$ \mathrm{var}(X_r) \leq \sigma^2 = \left(\tilde{f}_{\mathcal{D}}\right)^2 + 2\frac{R^2}{\epsilon^2}$$
Using this variance bound with Lemma~\ref{lem:med} we have the following statement with probability $1 - \delta$.
$$|\hat{f}_{\mathcal{D}}(\mathbf{q}) - f_{\mathcal{D}}(\mathbf{q})| \leq \left(\frac{\tilde{f}^2_\mathcal{D}(\mathbf{q})}{R} + \frac{2}{\epsilon^2}R \right)^{1/2}\sqrt{32 \log 1/\delta}$$

\end{proof}

\begin{corollary}
Put $R = \lceil\frac{1}{\sqrt{2}}\tilde{f}_\mathcal{D}(\mathbf{q})\epsilon\rceil$. Then the approximation error bound is
$$|\hat{f}_{\mathcal{D}}(\mathbf{q}) - f_{\mathcal{D}}(\mathbf{q})| \leq 16\left(\frac{\tilde{f}_\mathcal{D}(\mathbf{q})}{\epsilon}\log 1/\delta\right)^{1/2} \leq 16\sqrt{\frac{N}{\epsilon}\log 1/\delta}$$
\end{corollary}
\begin{proof}
Take the derivative of $\sqrt{a / R + bR}$ with respect to $R$ to find that $R = \sqrt{a/b}$ minimizes the bound. Put $a = \tilde{f}^2_{\mathcal{D}}(\mathbf{q})$ and $b = 2/\epsilon^2$. The corollary is obtained by substituting $R = \frac{1}{\sqrt{2}}\tilde{f}_{\mathcal{D}}(\mathbf{q})\epsilon$ into Theorem~\ref{thm:tradeoff}. We may replace $\tilde{f}_{\mathcal{D}}(\mathbf{q})$ with $N$ in the inequality because
$$ \tilde{f}_{\mathcal{D}}(\mathbf{q}) = \sum_{\mathbf{x} \in \mathcal{D}} \sqrt{k(\mathbf{x},\mathbf{q}} \leq \sum_{\mathbf{x}\in\mathcal{D}}1 = N$$

\end{proof}

\bibliographystyle{plain}

redpajama_set_name: RedPajamaArXiv


The date for Blue-White weekend has been confirmed. Our guess was correct, it is April 22nd, 2006.
To that end, I also have confirmed reservations for all of the 2006 football games. If you'd like a spot, please contact me directly. It's about $50/per weekend. If you already have a spot, you've already received an email from me.
We Are . . Penn State!
I will be making reservations for the regular season (i.e. Northwestern and Illinois) this week and would like to know if you want a spot. Hotel reservations for the regular season need to be made and PAID FOR IN FULL in cash by the end of January (~$350/weekend/room). Several others are going to help front the cash to get the reservations set. Obviously, if you help out with cash, you get a spot.
If you wait to decide, hotels will be full, so please let me know now. Hope to see you all this season!


I am a Baptist because: Baptist churches are the only ones that come up to the following tests of the New Testament churches, namely they were founded by the Lord Jesus Himself; have had an unbroken perpetuity and a wilderness history; Christ the only Law giver, Head and Lord; doctrinal conformity to the New Testament model; missionary activities, and have been the sect every where spoken against for 1900 years. With "meekness and fear," and yet with cordial good will and sincere regard for all with whom we differ, we have given an answer to the questions asked as to the once delivered faith. If God should use it to bring one honest inquirer to the unity of faith and the fitly joining of such an one to a body of Christ or for the establishing in the faith of some who are al ready in such a body, we shall be greatly rejoiced; for as John said: "I rejoiced greatlythat I found of thy children walking in truth, as we have received a commandment from the Father" (II John 1:4).
What You Believe Makes a Difference!
To help inform others about historical Baptist Beliefs and Distinctives.
Why does it matter which Bible I use?
Who Started the Baptist Church?
We hope to help you find these answers and more right here.
Listen to some great Sermons!


text:
Q: What index will fix this query? How can I fix the sequential scan over 'items'? I already have an index for [league_id, buyout_amount]
"Limit  (cost=72399.08..72399.15 rows=25 width=553) (actual time=2037.893..2037.898 rows=25 loops=1)"
"  ->  Sort  (cost=72399.08..72817.16 rows=167231 width=553) (actual time=2037.891..2037.894 rows=25 loops=1)"
"        Sort Key: normalized_buyout, id"
"        Sort Method: top-N heapsort  Memory: 41kB"
"        ->  Seq Scan on items  (cost=0.00..67679.95 rows=167231 width=553) (actual time=0.764..1906.766 rows=183374 loops=1)"
"              Filter: ((buyout_amount <> 0::numeric) AND (league_id = 1))"
"Total runtime: 2037.949 ms"

This is the query
SELECT "items".* FROM "items"
WHERE (items.league_id = '1') AND (buyout_amount <> 0)
ORDER BY normalized_buyout ASC, items.id DESC
LIMIT 25

Current indexes (that are relevant to this query)
-- Index: index_items_on_buyout_amount

-- DROP INDEX index_items_on_buyout_amount;

CREATE INDEX index_items_on_buyout_amount
  ON items
  USING btree
  (buyout_amount);

-- Index: index_items_on_id

-- DROP INDEX index_items_on_id;

CREATE INDEX index_items_on_id
  ON items
  USING btree
  (id DESC);

-- Index: index_items_on_id_and_league_id

-- DROP INDEX index_items_on_id_and_league_id;

CREATE INDEX index_items_on_id_and_league_id
  ON items
  USING btree
  (id, league_id);

-- Index: index_items_on_id_and_normalized_buyout

-- DROP INDEX index_items_on_id_and_normalized_buyout;

CREATE INDEX index_items_on_id_and_normalized_buyout
  ON items
  USING btree
  (id DESC, normalized_buyout);

-- Index: index_items_on_league_id

-- DROP INDEX index_items_on_league_id;

CREATE INDEX index_items_on_league_id
  ON items
  USING btree
  (league_id);

-- Index: index_items_on_league_id_and_buyout_amount

-- DROP INDEX index_items_on_league_id_and_buyout_amount;

CREATE INDEX index_items_on_league_id_and_buyout_amount
  ON items
  USING btree
  (league_id, buyout_amount);

-- Index: index_items_on_league_id_and_id

-- DROP INDEX index_items_on_league_id_and_id;

CREATE INDEX index_items_on_league_id_and_id
  ON items
  USING btree
  (league_id, id DESC);

-- Index: index_items_on_normalized_buyout

-- DROP INDEX index_items_on_normalized_buyout;

CREATE INDEX index_items_on_normalized_buyout
  ON items
  USING btree
  (normalized_buyout);

A: I discovered my ORDER BY index wasn't being used. By adding a partial index WHERE buyout_amount <> 0 ORDER BY normalized_buyout, id DESC it started using the index.

A: Postgres can be particularly sensitive/stupid about casts: if you don't use constants that are the same type as the column, it (usually in my experience) casts the column to the constant, thus invalidating indexes.
Make sure your constants align with the column types.
Change
WHERE items.league_id = '1' -- constant is text
AND buyout_amount <> 0 -- constant is int

To:
WHERE items.league_id = 1 -- constant is int
AND buyout_amount <> 0.0 -- constant is decimal

redpajama_set_name: RedPajamaStackExchange


While people are talking about the winter of 2014, I have already prepared your summer break for August 2014, a 5-day photography workshop in the Tuscany of Auvergne, in France.
When you hear Tuscany you may immediately think of Italy. Before my discovery I did the same but now I know that in the heart of France there is a place called Tuscany of Auvergne, in Puy de Dôme.
So close your eyes and let your thoughts go ….think about…….sunflowers, garlic, picturesque villages, volcano chain, dramatic light and….French wine and food!
Join me during these 5-days and you can expect: diverse hillside landscape scenery in the heart of the Tuscany of Auvergne with picturesque villages. The spectacular and unique light conditions playing on the dramatic Massif Central mountain and volcano chains, will take your breath away. We will explore the Billom area through the maze of small hillside roads and hiking trails. Although the main focus will be landscape photography those of you who like macro, close-up or street photography will have plenty of opportunities to indulge during the walks in the field and the small villages. Throughout the day, you will get personal field instruction as well as photo feedback. The group is kept small so that every participant gets enough personal attention.Be assured of a great learning experience and of course a lot of fun sightseeing and hiking along short trails. You will get introduced to the specialties of the area, visiting local organic wine producers and photograph the vineyards in the soft evening light.


Idyllically positioned in peaceful large grounds, with surrounding oak woods and olive groves. Villa Aphrodite is a comfortable three bedroom country property with a lovely swimming pool, and is suitable for 6-8 people.
To the rear of the property, the covered sun terraces with wrought iron railings, have stone steps leading down into the gardens. Then there is a 40sqm swimming pool with children's section, equipped with sun beds and sun umbrellas. A beautiful stone curved seating area, is set to one side of the house, under the shade of a vast oak tree; a perfect spot to dine Al Fresco. Next there is a stone made barbeque and oven waiting for you to test your grilling/baking skills.
Delightfully relaxed atmosphere, great seclusion and privacy – no houses within view – makes it an ideal destination for sightseers, families with young children as well as romantic getaways.
Suitability: Ideal for those who seek a quiet and undisturbed holiday.
It is suitable for sightseeing, families of up to 8 members.
and there is also a sofa bed for 2 extra persons.
(one bathroom includes a washing machine and hair dryer). Kitchen: Kitchen includes: freezer, fridge, microwave, oven, coffee maker, mini-fridge, spices,cooking utensils, toaster, iron board, dish washer, 4 ring stove and a kitchenette.
Living Room: Comfortable seating for 6 people. Dining: Comfortable seating for 6 people.
Entertainment: LCD-TV, satellite or cable TV channels, DVD, CD, stereo system, radio, books and a ping-pong table. Exterior: Patio, terrace, balcony, veranda, barbeque, private parking, garden with 6 chairs and a children's playground.
There is a shower by the pool – towels provided.
From Ioannis Daskalogiannis Airport in Chania, which is letter B, to villa Afroditi, which is letter A, the suggested route is: 90/E75 and it is approximately 66.9 km, or 1 hour 17 mins by car.
From Souda Port in Chania, which is letter C, to villa Afroditi, which is letter A, the suggested route is: 90/E75 and it is approximately 50.2 km, or 53 mins by car.
From Heraclion Port, which is letter D, to villa Afroditi, which is letter A, the suggested route is: 90 and it is approximately 89.4 km, or 1 hour 34 mins by car.
and it is approximately 91.4 km, or 1 hour 35 mins by car.
Located near the traditional village of Prines. It is 7 kilometers away from the center of Rethymnon and only 5 kilometers from the nearest beach.
Prines is a traditional old village. Beautiful, with plentiful vegetation, mainly olive trees as well as oak trees. Ideal for those who love nature – peace and quiet of country life. It has the beautiful old church of St. Nicholas which was built in the 13th century and a spring of water that has been running for many centuries.
Prines has only a few small shops, as would be expected of a small country village. But all your basic needs will be met with a mini-market, some traditional cafes and an excellent tavern about 1km from Villa Afroditi.
The nearest beach is Koubes beach which is approximately 5 km away, while the beach of Petres is only 6 km away.
This villa was just brilliant. The location was idyllic-in the country but not to far from a village and town. Also,it was not too far from the highway, which made sight-seeing easy. The house was very good and the gardens and pool were excellent and well-maintained. The hosts were very friendly and were very generous in every respect. It was excellent value for money.It was, as the title suggests, Utopia in Crete and we would,without hesitation, return again.
A really authentic Greek villa, the moussaka, raki and apple cake supplied by the owners were amazing, if you want a chilled out holiday within reach of the amazing sights of this beautiful island then book the villa. You will need a car, there is no where within walking distance (apart from olives and goats) but this means its incredibly peaceful. There are a couple of resturants within 5 mins drive but with a purpose built bbq and an out door pizza oven (and very good local supermarkets, butchers and bakers) you can while away the evening on the terrace over looking the fantastic pool.
… use this form to ask for availability and rental rates for villa Afroditi.
I would like to know about the rental rates and availability for the dates I have selected above.


Where can I purchase the dresser?
Converting old dresser to foyer dresser ???
I like the first piece you showed - It could easily be fitted with 'legs', but think of them almost as as 'blocks' form-wise, in keeping with the squared elements on the front details of the piece, vs rounded bun or finial type feet. Your home has the elements of spanish, a little bit rustic in color and finish. I think with the right piece on the wall above, and a great rug, it would look substantial, a bit of rustic elegance, - beautiful. I have a similar piece, with a few more curves that at first I thought was much too spanish, but it has grown on me and become one of my favorite pieces in an Old World dining room setting. - build around it and have fun.