You will be shown an excerpt from a computer science scientific research paper, followed by a question about the paper. Please answer the question. Do not include any text in your response other than the answer.

Context: JFT is an internal Google dataset that has 100 million labeled imageswith 15,000 labels. When we did this work, Google’s baseline model forJFT was a deep convolutional neural network [7] that had been trained forabout six months using asynchronous stochastic gradient descent on alarge number of cores. This training used two types ofparallelism [2]. First, there were many replicas of the neural net runningon different sets of cores and processing different mini-batches fromthe training set. Each replica computes the average gradient on itscurrent mini-batch and sends this gradient to a sharded parameter server whichsends back new values for the parameters. These new values reflect allof the gradients received by the parameter server since the last timeit sent parameters to the replica. Second, each replica is spread overmultiple cores by putting different subsets of the neurons on eachcore. Ensemble training is yet a third type of parallelism that can bewrapped around the other two types, but only if a lot more cores areavailable. Waiting for several years to train an ensemble of models wasnot an option, so we needed a much faster way to improve the baselinemodel.

Question: What is the main difference between synchronous vs asynchronous SGD?
In training of Google’s baseline model of JFT, asynchronous stochastic gradient descent (SGD) involved running replicas of the neural net different sets of cores to compute gradients on given mini-batches, which are then sent to a shared parameter server which returns new values for the parameters. The general definitions of asynchronous and synchronous SGD, and, thus, their differences cannot be answered from this paper.