text:
## Found 1,176 Documents (Results 1–100)

100
MathJax

### Limiting spectral distribution of large dimensional Spearman’s rank correlation matrices. (English)Zbl 07542999

MSC:  62Hxx 62G30 62H20
Full Text:

### Random matrices with independent entries: beyond non-crossing partitions. (English)Zbl 07535859

MSC:  60B20 60B10
Full Text:

MSC:  62-XX
Full Text:

MSC:  62-XX
Full Text:

### The limiting distribution of the hook length of a randomly chosen cell in a random Young diagram. (English. Russian original)Zbl 07517612

Proc. Steklov Inst. Math. 316, No. 1, 268-279 (2022); translation from Tr. Mat. Inst. Steklova 316, 285-297 (2022).
MSC:  60-XX 05-XX
Full Text:

Full Text:

Full Text:

### Weakly protected nodes in random binary search trees. (English)Zbl 1483.05022

MSC:  05C05 05C80 60F05
Full Text:

Full Text:

### Common breaks in means for panel data under short-range dependence. (English)Zbl 07532134

MSC:  62F10 62F12 62-XX
Full Text:

Full Text:

### Some patterned matrices with independent entries. (English)Zbl 1479.60013

MSC:  60B20 60B10
Full Text:

MSC:  62G08
Full Text:

Full Text:

### On the asymptotic distribution of randomly weighted averages of random vectors. (English)Zbl 1473.62165

MSC:  62H05 62E20
Full Text:

Full Text:

### The limits of the sample spiked eigenvalues for a high-dimensional generalized Fisher matrix and its applications. (English)Zbl 1473.62159

MSC:  62H05 62H25
Full Text:

### Non-standard limits for a family of autoregressive stochastic sequences. (English)Zbl 1476.60035

MSC:  60E07 60G50 60J05
Full Text:

### Yaglom limit for stochastic fluid models. (English)Zbl 07408814

MSC:  68Q25 68R10 68U05
Full Text:

### A binary search scheme for determining all contaminated specimens. (English)Zbl 1478.92125

MSC:  92C99 62P10 60C99
Full Text:

Full Text:

Full Text:

Full Text:

### Test for homogeneity with unordered paired observations. (English)Zbl 1471.62263

MSC:  62F03 62F05 62E20
Full Text:

### De-biased sparse PCA: inference for eigenstructure of large covariance matrices. (English)Zbl 1473.62204

MSC:  62H25 62F25 62J07
Full Text:

### Fast tensorial JADE. (English)Zbl 1467.62109

MSC:  62H25 62E20
Full Text:

### Stochastic fluid model with jumps: the bounded model. (English)Zbl 1470.60215

MSC:  60J28 76M35
Full Text:

Full Text:

Full Text:

### Limiting spectral density of a dynamic nonconservative birth-death $$Q$$ matrix. (Chinese. English summary)Zbl 07494833

MSC:  60B20 60F05 62M15
Full Text:

MSC:  62Pxx
Full Text:

### A result on the limiting spectral distribution of random matrices with unequal variance entries. (English)Zbl 07460949

MSC:  60E10 60F15
Full Text:

### On nonparametric kernel-type estimate of the Bernoulli regression function. (English)Zbl 1473.62236

Jaiani, George (ed.) et al., Applications of mathematics and informatics in natural sciences and engineering, AMINSE 2019. Selected papers based on the presentations at the 4th conference, Tbilisi, Georgia, September 23–26, 2019. Cham: Springer. Springer Proc. Math. Stat. 334, 19-36 (2020).
MSC:  62J05 62G05 62E20
Full Text:

### Design of multiple-deferred state sampling plans for exponentiated half logistic distribution. (English)Zbl 07364944

MSC:  62N05 62D05 62P30
Full Text:

### Hypothesis testing for quantitative trait locus effects in both location and scale in genetic backcross studies. (English)Zbl 1467.62038

MSC:  62F03 62E20
Full Text:

### A fast estimation method for mean change point in massive data sets. (Chinese. English summary)Zbl 1474.62099

MSC:  62G05 62G10 62-08
Full Text:

Full Text:

### On the asymptotic distribution of the quadratic GMM estimator of a dynamic panel data model under a unit root. (English)Zbl 1459.62213

MSC:  62P20 62M07 62E20
Full Text:

### Asymptotic inference for $$\mathrm{AR}(1)$$ panel data. (English)Zbl 1474.62037

MSC:  62E20 62M10 62D20
Full Text:

Full Text:

### Limiting distribution of a sequence of functions defined on a Markov chain. (English)Zbl 1456.60061

MSC:  60F05 60J10
Full Text:

Full Text:

Full Text:

Full Text:

### Limiting distribution of periodic position measurements of a quantum harmonic oscillator. (English)Zbl 1448.81030

MSC:  81P15 81P40
Full Text:

Full Text:

Full Text:

### Modelling functional data with high-dimensional error structure. (English)Zbl 1444.62153

Aneiros, Germán (ed.) et al., Functional and high-dimensional statistics and related fields. Selected papers presented at the 5th international workshop on functional and operatorial statistics, IWFOS 2021, Brno, Czech Republic, June 23–25, 2021. Cham: Springer. Contrib. Stat., 99-106 (2020).
MSC:  62R10 62E20
Full Text:

### Existence of limiting distribution for affine processes. (English)Zbl 1451.60084

MSC:  60J25 60F05 60J80
Full Text:

### On the maximal multiplicity of block sizes in a random set partition. (English)Zbl 1442.05021

MSC:  05A18 05A17 60C05
Full Text:

### Steady-state imperfect repair models. (English)Zbl 1443.90156

MSC:  90B25 60K10 62N05
Full Text:

Full Text:

Full Text:

### Asymptotic sojourn time analysis of finite-source M/M/$$1$$ retrial queueing system with collisions and server subject to breakdowns and repairs. (English)Zbl 1436.90036

MSC:  90B22 60K25
Full Text:

### A combinatorial approach for discrete car parking on random labelled trees. (English)Zbl 1435.05180

MSC:  05C80 05C78 60K35
Full Text:

MSC:  62G32
Full Text:

### Central limit theorem for descents in conjugacy classes of $$S_n$$. (English)Zbl 1428.05019

MSC:  05A15 05E05
Full Text:

MSC:  62-XX
Full Text:

### Selection of modified chain Sampling plan (MChSP-1) with zero-inflated Poisson distribution based on quality region. (English)Zbl 07479612

MSC:  90-XX 62-XX
Full Text:

Full Text:

### Asymptotic analysis of finite-source M/M/1 retrial queueing system with collisions and server subject to breakdowns and repairs. (English)Zbl 1427.90097

MSC:  90B22 60K25
Full Text:

Full Text:

MSC:  60K35
Full Text:

### Transition and limiting distributions when covariates are available. (English)Zbl 1427.62014

MSC:  62E20 62H30 62M02
Full Text:

### Primes in prime number races. (English)Zbl 1473.11177

MSC:  11N05 11B83 11M26
Full Text:

### Distributions of a particle’s position and their asymptotics in the $$q$$-deformed totally asymmetric zero range process with site dependent jumping rates. (English)Zbl 1422.60159

MSC:  60K35 82C22
Full Text:

### Testing for parametric component of partially linear models with missing covariates. (English)Zbl 1420.62185

MSC:  62G08 62H15 62E20
Full Text:

### Some recent interactions of probability and number theory. (English)Zbl 1414.11098

MSC:  11K65 60F05 11N37
Full Text:

### Transformed statistics for tests of conditional independence in $$J \times K \times L$$ contingency tables. (English)Zbl 1417.62156

MSC:  62H17 62E20 62H10
Full Text:

Full Text:

### Maximum penalized likelihood estimation for the endpoint and exponent of a distribution. (English)Zbl 1412.62037

MSC:  62F10 62E10
Full Text:

### Central limit theorems for aggregate efficiency. (English)Zbl 1457.91211

MSC:  91B38 60F05
Full Text:

### Higher dimensional quasi-power theorem and Berry-Esseen inequality. (English)Zbl 1430.60028

MSC:  60F05 60E15 60C05
Full Text:

### Extreme values in ON/OFF models of teletraffic under permanent and periodic measurements. (English. Russian original)Zbl 1401.60098

Mosc. Univ. Math. Bull. 73, No. 4, 137-142 (2018); translation from Vestn. Mosk. Univ., Ser. I 73, No. 4, 9-15 (2018).
MSC:  60G70 60K30
Full Text:

Full Text:

Full Text:

Full Text:

### Quantum walks and search algorithms. 2nd edition. (English)Zbl 1457.81004

Quantum Science and Technology. Cham: Springer (ISBN 978-3-319-97812-3/hbk; 978-3-319-97813-0/ebook). xiv, 308 p. (2018).
Full Text:

### Structural change in nonstationary $$\mathrm{AR}(1)$$ models. (English)Zbl 1400.62192

MSC:  62M10 62F12 62E20
Full Text:

### A consistent estimator for spectral density matrix of a discrete time periodically correlated process. (English)Zbl 1410.62161

MSC:  62M10 62M15 62G20
Full Text:

### Large covariance and autocovariance matrices. (English)Zbl 1408.62007

Monographs on Statistics and Applied Probability 162. Boca Raton, FL: CRC Press (ISBN 978-1-138-30386-7/hbk; 978-1-351-39816-9/ebook). xxiii, 272 p. (2018).
Full Text:

### An iterative hard thresholding estimator for low rank matrix recovery with explicit limiting distribution. (English)Zbl 1394.62065

MSC:  62H12 62E20
Full Text:

### Powerful test based on conditional effects for genome-wide screening. (English)Zbl 1393.62080

MSC:  62P10 62H15
Full Text:

### On Schott’s and Mao’s test statistics for independence of normal random vectors. (English)Zbl 1463.62005

MSC:  62B05 62E20 62H15
Full Text:

Full Text:

### On limiting theorems for conditional causation probabilities of multiple-run-rules. (English)Zbl 1391.60172

MSC:  60J10 60C05 60F05
Full Text:

### Patterned random matrices. (English)Zbl 1400.15001

Boca Raton, FL: CRC Press (ISBN 978-1-138-59146-2/hbk; 978-0-429-94889-3/ebook). xxi, 267 p. (2018).
Full Text:

Full Text:

### Limiting distributions of spectral radii for product of matrices from the spherical ensemble. (English)Zbl 1385.15004

MSC:  15B52 15A18 60B20
Full Text:

Full Text:

### Limiting behaviour of Fréchet means in the space of phylogenetic trees. (English)Zbl 1394.62153

MSC:  62P10 60F05 62E20
Full Text:

MSC:  62P10
Full Text:

### Spectral radii of truncated circular unitary matrices. (English)Zbl 1376.15023

MSC:  15B10 15A18
Full Text:

### Some features of a finite-source M/GI/1 retrial queuing system with collisions of customers. (English)Zbl 1452.90123

Vishnevskiy, Vladimir M. (ed.) et al., Distributed computer and communication networks. 20th international conference, DCCN 2017, Moscow, Russia, September 25–29, 2017. Proceedings. Cham: Springer. Commun. Comput. Inf. Sci. 700, 186-200 (2017).
MSC:  90B22
Full Text:

### Simulation from quasi-stationary distributions on reducible state spaces. (English)Zbl 1425.60067

MSC:  60J27 62G09
Full Text:

Full Text:

### Analytical properties of an imperfect repair model and application in preventive maintenance scheduling. (English)Zbl 1394.90231

MSC:  90B25 62N05 90B35
Full Text:

### Matrix polynomial generalizations of the sample variance-covariance matrix when $$pn^{-1}\to y(0,\infty)$$. (English)Zbl 1390.15119

Indian J. Pure Appl. Math. 48, No. 4, 575-607 (2017); erratum ibid. 49, No. 4, 783-788 (2018).
MSC:  15B52 60B20
Full Text:

all top 5

all top 5

all top 5

all top 3

all top 3


text:
0%

Note - These are my notes on DeepLearning Specialization Part:
Regularization || Deeplearning (Course - 2 Week - 1) || Improving Deep Neural Networks(Week 1)

### Introduction:

And usually, you set this using your development set, or using [INAUDIBLE] cross validation. When you a variety of values and see what does the best, in terms of trading off between doing well in your training set versus also setting that two normal of your parameters to be small. Which helps prevent over fitting. So lambda is another hyper parameter that you might have to tune. And by the way, for the programming exercises, lambda is a reserved keyword in the Python programming language. So in the programming exercise, we’ll have lambd,
]without the a, so as not to clash with the reserved keyword in Python. So we use lambd to represent the lambda regularization parameter.

$$\text{minimize(Loss(Data|Model))}$$
we’ll now minimize loss+complexity, which is called structural risk minimization:

$$\text{minimize(Loss(Data|Model) + complexity(Model))}$$
Our training optimization algorithm is now a function of two terms: the loss term, which measures how well the model fits the data, and the regularization term, which measures model complexity.

Model complexity as a function of the weights of all the features in the model.
Model complexity as a function of the total number of features with nonzero weights. (A later module covers this approach.)
If model complexity is a function of weights, a feature weight with a high absolute value is more complex than a feature weight with a low absolute value.

We can quantify complexity using the L2 regularization formula, which defines the regularization term as the sum of the squares of all the feature weights:

$$L_2 \text{ regularization term} = ||\boldsymbol w||_2^2 = {w_1^2 + w_2^2 + … + w_n^2}$$

In this formula, weights close to zero have little effect on model complexity, while outlier weights can have a huge impact.

For example, a linear model with the following weights:

$${w_1 = 0.2, w_2 = 0.5, w_3 = 5, w_4 = 1, w_5 = 0.25, w_6 = 0.75}$$

Has an L2 regularization term of 26.915:

$$w_1^2 + w_2^2 + \boldsymbol{w_3^2} + w_4^2 + w_5^2 + w_6^2$$ $$= 0.2^2 + 0.5^2 + \boldsymbol{5^2} + 1^2 + 0.25^2 + 0.75^2$$ $$= 0.04 + 0.25 + \boldsymbol{25} + 1 + 0.0625 + 0.5625$$
$$= 26.915$$
But (w_3) (bolded above), with a squared value of 25, contributes nearly all the complexity. The sum of the squares of all five other weights adds just 1.915 to the L2 regularization term.

### Lambda Introduction

Model developers tune the overall impact of the regularization term by multiplying its value by a scalar known as lambda (also called the regularization rate). That is, model developers aim to do the following:

$$\text{minimize(Loss(Data|Model)} + \lambda \text{ complexity(Model))}$$
Performing L2 regularization has the following effect on a model

Encourages weight values toward 0 (but not exactly 0)
Encourages the mean of the weights toward 0, with a normal (bell-shaped or Gaussian) distribution.
Increasing the lambda value strengthens the regularization effect. For example, the histogram of weights for a high value of lambda
Some of your training examples of the losses of the individual predictions in the different examples, where you recall that w and b in the logistic regression, are the parameters. So w is an x-dimensional parameter vector, and b is a real number. And so to add regularization to the logistic regression, what you do is add to it this thing, lambda, which is called the regularization parameter. I’ll say more about that in a second. But lambda/2m times the norm of w squared. So here, the norm of w squared is just equal to sum from j equals 1 to nx of wj squared, or this can also be written w transpose w, it’s just a square Euclidean norm of the prime to vector w. And this is called L2 regularization.

When choosing a lambda value, the goal is to strike the right balance between simplicity and training-data fit:

If your lambda value is too high, your model will be simple, but you run the risk of underfitting your data. Your model won’t learn enough about the training data to make useful predictions.

If your lambda value is too low, your model will be more complex, and you run the risk of overfitting your data. Your model will learn too much about the particularities of the training data, and won’t be able to generalize to new data.

Note: Setting lambda to zero removes regularization completely. In this case, training focuses exclusively on minimizing loss, which poses the highest possible overfitting risk.
The ideal value of lambda produces a model that generalizes well to new, previously unseen data. Unfortunately, that ideal value of lambda is data-dependent, so you’ll need to do some tuning.

L2 Regularization and lambda (Regularization Parameter): relation
There’s a close connection between learning rate and lambda. Strong L2 regularization values tend to drive feature weights closer to 0. Lower learning rates (with early stopping) often produce the same effect because the steps away from 0 aren’t as large. Consequently, tweaking learning rate and lambda simultaneously may have confounding effects.

Early stopping means ending training before the model fully reaches convergence. In practice, we often end up with some amount of implicit early stopping when training in an online (continuous) fashion. That is, some new trends just haven’t had enough data yet to converge.

As noted, the effects from changes to regularization parameters can be confounded with the effects from changes in learning rate or number of iterations. One useful practice (when training across a fixed batch of data) is to give yourself a high enough number of iterations that early stopping doesn’t play into things.

### Implementing L2 Regularisation in a Neural Network

So this is how you implement L2 regularization for logistic regression. How about a neural network? In a neural network, you have a cost function(J) that's a function of all of your parameters, w[1], b[1] through w[L], b[L], where capital L is the number of layers in your neural network. And so the cost function is this, sum of the losses, summed over your m training examples.

#### L2 Norm ~ Weight Deacy

And it's for this reason that L2 regularization is sometimes also called weight decay. So if I take this definition of dw[l] and just plug it in here, then you see that the update is w[l] = w[l] times the learning rate alpha times the thing from backprop, +lambda of m times w[l]. Throw the minus sign there. And so this is equal to $$(w[l]- alpha * lambda / m) * (w[l]- alpha)$$ times the thing you got from backpop. And so this term shows that whatever the matrix w[l] is, you're going to make it a little bit smaller, right? This is actually as if you're taking the matrix w and you're multiplying it by $$1-alpha*lambda/m$$ . You're really taking the matrix w and subtracting $$alpha*lambda/m$$ with L2 Regularization penalty . Like you're multiplying matrix w by this number, which is going to be a little bit less than 1. So this is why L2 norm regularization is also called weight decay. Because it's just like the ordinally gradient descent, where you update w by subtracting alpha times the original gradient you got from backprop. But now you're also multiplying w by this thing, which is a little bit less than 1. So the alternative name for L2 regularization is weight decay. I'm not really going to use that name, but the intuition for it's called weight decay is that this first term here, is equal to this. So you're just multiplying the weight metrics by a number slightly less than 1. So that's how you implement L2 regularization in neural network. Now, one question that he has asked me is, why does regularization prevent over-fitting? Let's look at the next Section, and gain some intuition for how regularization prevents over-fitting.

### How do regularization Prevent Overfitting?

Why does regularization help with overfitting? Why does it help with reducing variance problems? Let's go through a couple examples to gain some intuition about how it works. So, recall that high bias, high variance. And I just write pictures from our earlier section that looks something like this.
Now, let's see a fitting large and deep neural network. I know I haven't drawn this one too large or too deep, unless you think some neural network and this currently overfitting. So you have some cost function like J of W, B equals sum of the losses.

$$\text{minimize(Loss(Data|Model)} + \lambda \text{ complexity(Model))}$$

So what we did for regularization was add this extra term that penalizes the weight matrices from being too large. So that was the Frobenius norm. So why is it that shrinking the L two norm or the Frobenius norm or the parameters might cause less overfitting? One piece of intuition is that if you crank regularisation lambda to be really, really big, they’ll be really incentivized to set the weight matrices W to be reasonably close to zero. So one piece of intuition is maybe it set the weight to be so close to zero for a lot of hidden units that’s basically zeroing out a lot of the impact of these hidden units. And if that’s the case, then this much simplified neural network becomes a much smaller neural network. In fact, it is almost like a logistic regression unit, but stacked most probably as deep. And so that will take you from this overfitting case much closer to the left to other high bias case. But hopefully there’ll be an intermediate value of lambda that results in a result closer to this just right case in the middle.
But the intuition is that by cranking up lambda to be really big they’ll set W close to zero, which in practice this isn’t actually what happens. We can think of it as zeroing out or at least reducing the impact of a lot of the hidden units so you end up with what might feel like a simpler network.
They get closer and closer as if you’re just using logistic regression.
The intuition of completely zeroing out of a bunch of hidden units isn’t quite right.
It turns out that what actually happens is they’ll still use all the hidden units, but each of them would just have a much smaller effect. But you do end up with a simpler network and as if you have a smaller network that is therefore less prone to overfitting.
So a lot of this intuition helps better when you implement regularization in the program exercise, you actually see some of these variance reduction results yourself.

### Another Intution with TanH as activation Function instead of Sigmoid

Here's another attempt at additional intuition for why regularization helps prevent overfitting. And for this, I'm going to assume that we're using the tanh activation function which looks like this.
This is a g of z equals tanh of z. So if that's the case, notice that so long as Z is quite small, so if Z takes on only a smallish range of parameters, maybe around here, then you're just using the linear regime of the tanh function. Is only if Z is allowed to wander up to larger values or smaller values like so, that the activation function starts to become less linear.

### Takeaway:

So the intuition you might take away from this is that if lambda, the regularization parameter, is large, then you have that your parameters will be relatively small, because they are penalized being large into a cos function. And so if the blades W are small then because Z is equal to W and then technically is plus b, but if W tends to be very small, then Z will also be relatively small. And in particular, if Z ends up taking relatively small values, just in this whole range, then G of Z will be roughly linear. So it's as if every layer will be roughly linear. As if it is just linear regression. And we saw in course one that if every layer is linear then your whole network is just a linear network. And so even a very deep network, with a deep network with a linear activation function is at the end they are only able to compute a linear function. So it's not able to fit those very very complicated decision. Very non-linear decision boundaries that allow it to really overfit right to data sets like we saw on the overfitting high variance case on the previous slide.

#### So just to summarize

If the regularization becomes very large, the parameters W very small, so Z will be relatively small, kind of ignoring the effects of b for now, so Z will be relatively small or, really,
I should say it takes on a small range of values. And so the activation function if is tanh, say, will be relatively linear. And so your whole neural network will be computing something not too far from a big linear function which is therefore pretty simple function rather than a very complex highly non-linear function. And so is also much less able to overfit. And again, when you enter in regularization for yourself in the program exercise, you'll be able to see some of these effects yourself.

#### Tip

Before wrapping up our def discussion on regularization, I just want to give you one implementational tip. Which is that, when implanting regularization, we took our definition of the cost function J and we actually modified it by adding this extra term that penalizes the weight being too large. And so if you implement gradient descent, one of the steps to debug gradient descent is to plot the cost function J as a function of the number of elevations of gradient descent and you want to see that the cost function J decreases monotonically after every elevation of gradient descent. And if you're implementing regularization then please remember that J now has this new definition. If you plot the old definition of J, just this first term, then you might not see a decrease monotonically. So to debug gradient descent make sure that you're plotting this new definition of J that includes this second term as well. Otherwise you might not see J decrease monotonically on every single elevation. So that's it for L two regularization which is actually a regularization technique that I use the most in training deep learning modules. In deep learning there is another sometimes used regularization technique called dropout regularization. Let's take a look at that in the next section.

### Dropout Regularization

In addition to L2 regularization, another very powerful regularization techniques is called "dropout." Let's see how that works. Let's say you train a neural network like the one on the left and there's over-fitting. Here's what you do with dropout. Let me make a copy of the neural network.
With dropout, what we're going to do is go through each of the layers of the network and set some probability of eliminating a node in neural network. Let's say that for each of these layers, we're going to- for each node, toss a coin and have a 0.5 chance of keeping each node and 0.5 chance of removing each node. So, after the coin tosses, maybe we'll decide to eliminate those nodes, then what you do is actually remove all the outgoing things from that no as well. So you end up with a much smaller, really much diminished network. And then you do back propagation training. There's one example on this much diminished network. And then on different examples, you would toss a set of coins again and keep a different set of nodes and then dropout or eliminate different than nodes. And so for each training example, you would train it using one of these neural based networks. So, maybe it seems like a slightly crazy technique. They just go around coding those are random, but this actually works. But you can imagine that because you're training a much smaller network on each example or maybe just give a sense for why you end up able to regularize the network, because these much smaller networks are being trained.

### Implementing Dropout

Let's look at how you implement dropout. There are a few ways of implementing dropout. I'm going to show you the most common one, which is technique called inverted dropout. For the sake of completeness, let's say we want to illustrate this with layer l=3. So, in the code I'm going to write- there will be a bunch of 3s here. I'm just illustrating how to represent dropout in a single layer. So, what we are going to do is set a vector d and d^3 is going to be the dropout vector for the layer 3.

### Inverted Dropout

That's what the 3 is to be np.random.rand(a). And this is going to be the same shape as a3. And when I see if this is less than some number, which I'm going to call keep.prob. And so, keep.prob is a number. It was 0.5 on the previous time, and maybe now I'll use 0.8 in this example,
and there will be the probability that a given hidden unit will be kept. So keep.prob = 0.8, then this means that there's a 0.2 chance of eliminating any hidden unit. So, what it does is it generates a random matrix. And this works as well if you have factorized. So d3 will be a matrix. Therefore, each example have a each hidden unit there's a 0.8 chance that the corresponding d3 will be one, and a 20% chance there will be zero. So, this random numbers being less than 0.8 it has a 0.8 chance of being one or be true, and 20% or 0.2 chance of being false, of being zero. And then what you are going to do is take your activations from the third layer, let me just call it a3 in this low example. So, a3 has the activations you computate. And you can set a3 to be equal to the old a3, times- There is element wise multiplication. Or you can also write this as a3* = d3. But what this does is for every element of d3 that's equal to zero. And there was a 20% ch ance of each of the elements being zero, just multiply operation ends up zeroing out, the corresponding element of d3. If you do this in python, technically d3 will be a boolean array where value is true and false, rather than one and zero. But the multiply operation works and will interpret the true and false values as one and zero. If you try this yourself in python, you'll see. Then finally, we're going to take a3 and scale it up by dividing by 0.8 or really dividing by our keep.prob parameter. So, let me explain what this final step is doing. Let's say for the sake of argument that you have 50 units or 50 neurons in the third hidden layer. So maybe a3 is 50 by one dimensional or if you- factorization maybe it's 50 by m dimensional. So, if you have a 80% chance of keeping them and 20% chance of eliminating them. This means that on average, you end up with 10 units shut off or 10 units zeroed out. And so now, if you look at the value of z^4, z^4 is going to be equal to $$w^4 * a^3 + b^4$$. And so, on expectation, this will be reduced by 20%. By which I mean that 20% of the elements of a3 will be zeroed out. So, in order to not reduce the expected value of z^4, what you do is you need to take this, and divide it by 0.8 because this will correct or just a bump that back up by roughly 20% that you need. So it's not changed the expected value of a3. And, so this line here is what's called the inverted dropout technique. And its effect is that, no matter what you set to keep.prob to, whether it's 0.8 or 0.9 or even one, if it's set to one then there's no dropout, because it's keeping everything or 0.5 or whatever, this inverted dropout technique by dividing by the keep.prob, it ensures that the expected value of a3 remains the same. And it turns out that at test time, when you trying to evaluate a neural network, which we'll talk about on the next slide, this inverted dropout technique, there is there is line to are due to the green box at dropping out. This makes test time easier because you have less of a scaling problem. By far the most common implementation of dropouts today as far as I know is inverted dropouts. I recommend you just implement this. But there were some early iterations of dropout that missed this divide by keep.prob line, and so at test time the average becomes more and more complicated. But again, people tend not to use those other versions. So, what you do is you use the d vector, and you'll notice that for different training examples, you zero out different hidden units. And in fact, if you make multiple passes through the same training set, then on different pauses through the training set, you should randomly zero out different hidden units. So, it's not that for one example, you should keep zeroing out the same hidden units is that, on iteration one of grade and descent, you might zero out some hidden units. And on the second iteration of great descent where you go through the training set the second time, maybe you'll zero out a different pattern of hidden units. And the vector d or d3, for the third layer, is used to decide what to zero out, both in for prob as well as in that prob. We are just showing for prob here. Now, having trained the algorithm at test time, here's what you would do. At test time, you're given some x or which you want to make a prediction. And using our standard notation, I'm going to use a^0, the activations of the zeroes layer to denote just test example x. So what we're going to do is not to use dropout at test time in particular which is in a sense. $$Z^1= w^1.a^0 + b^1. a^1 = g^1(z^1 Z). Z^2 = w^2.a^1 + b^2. a^2 =...$$ And so on. Until you get to the last layer and that you make a prediction y^. But notice that the test time you're not using dropout explicitly and you're not tossing coins at random, you're not flipping coins to decide which hidden units to eliminate. And that's because when you are making predictions at the test time, you don't really want your output to be random. If you are implementing dropout at test time, that just add noise to your predictions.

#### Introspect Results:

In theory, one thing you could do is run a prediction process many times with different hidden units randomly dropped out and have it across them. But that's computationally inefficient and will give you roughly the same result; very, very similar results to this different procedure as well. And just to mention, the inverted dropout thing, you remember the step on the previous line when we divided by the cheap.prob. The effect of that was to ensure that even when you don't see men dropout at test time to the scaling, the expected value of these activations don't change. So, you don't need to add in an extra funny scaling parameter at test time. That's different than when you have that training time. So that's dropouts. And when you implement this in week's premier exercise, you gain more firsthand experience with it as well. But why does it really work? What I want to do the next section is give you some better intuition about what dropout really is doing. Let's go on to the next section.

## Understanding Dropout

Drop out does this seemingly crazy thing of randomly knocking out units on your network. Why does it work so well with a regularizer? Let's gain some better intuition. In the previous section, I gave this intuition that drop-out randomly knocks out units in your network. So it's as if on every iteration you're working with a smaller neural network, and so using a smaller neural network seems like it should have a regularizing effect.

#### Why Does Dropout Work ?

Here's a second intuition which is, let's look at it from the perspective of a single unit. Let's say this one. Now, for this unit to do his job as for inputs and it needs to generate some meaningful output. Now with drop out, the inputs can get randomly eliminated. Sometimes those two units will get eliminated, sometimes a different unit will get eliminated.
So, what this means is that this unit, which I'm circling in purple, it can't rely on any one feature because any one feature could go away at random or any one of its own inputs could go away at random. Some particular would be reluctant to put all of its bets on, say, just this input, right? The weights, we're reluctant to put too much weight on any one input because it can go away. So this unit will be more motivated to spread out this way and give you a little bit of weight to each of the four inputs to this unit. And by spreading all the weights, this will tend to have an effect of shrinking the squared norm of the weights. And so, similar to what we saw with L2 regularization, the effect of implementing drop out is that it shrinks the weights and does some of those outer regularization that helps prevent over-fitting. But it turns out that drop out can formally be shown to be an adaptive form without a regularization. But L2 penalty on different weights are different, depending on the size of the activations being multiplied that way. But to summarize, it is possible to show that drop out has a similar effect to L2 regularization. Only to L2 regularization applied to different ways can be a little bit different and even more adaptive to the scale of different inputs. One more detail for when you're implementing drop out. Here's a network where you have three input features. This is seven hidden units here, seven, three, two, one. So, one of the parameters we had to choose was the cheap prop which has a chance of keeping a unit in each layer. So, it is also feasible to vary key prop by layer. So for the first layer, your matrix W1 will be three by seven. Your second weight matrix will be seven by seven. W3 will be seven by three and so on. And so W2 is actually the biggest weight matrix, because they're actually the largest set of parameters would be in W2 which is seven by seven. So to prevent, to reduce over-fitting of that matrix, maybe for this layer, I guess this is layer two, you might have a key prop that's relatively low, say zero point five, whereas for different layers where you might worry less about over-fitting, you could have a higher key prop, maybe just zero point seven. And if a layers we don't worry about over-fitting at all, you can have a key prop of one point zero.

#### For clarity

These numbers I'm drawing on the purple boxes, These could be different key props for different layers. Notice that the key prop of one point zero means that you're keeping every unit and so, you're really not using drop out for that layer. But for layers where you're more worried about over-fitting, really the layers with a lot of parameters, you can set the key prop to be smaller to apply a more powerful form of drop out. It's kind of like cranking up the regularization parameter lambda of L2 regularization where you try to regularize some layers more than others. And technically, you can also apply drop out to the input layer, where you can have some chance of just maxing out one or more of the input features. Although in practice, usually don't do that that often. And so, a key prop of one point zero was quite common for the input there. You can also use a very high value, maybe zero point nine, but it's much less likely that you want to eliminate half of the input features. So usually key prop, if you apply the law, will be a number close to one if you even apply drop out at all to the input there. So just to summarize, if you're more worried about some layers overfitting than others, you can set a lower key prop for some layers than others.

### Downside of Dropout

The downside is, this gives you even more hyper parameters to search for using cross-validation. One other alternative might be to have some layers where you apply drop out and some layers where you don't apply drop out and then just have one hyper parameter, which is a key prop for the layers for which you do apply drop outs. And before we wrap up, just a couple implementational tips. Many of the first successful implementations of drop outs were to computer vision. So in computer vision, the input size is so big, inputting all these pixels that you almost never have enough data. And so drop out is very frequently used by computer vision. And there's some computer vision researchers that pretty much always use it, almost as a default. But really the thing to remember is that drop out is a regularization technique, it helps prevent over-fitting. And so, unless my algorithm is over-fitting, I wouldn't actually bother to use drop out. So it's used somewhat less often than other application areas. There's just with computer vision, you usually just don't have enough data, so you're almost always overfitting, which is why there tends to be some computer vision researchers who swear by drop out. But their intuition doesn't always generalize I think to other disciplines. One big downside of drop out is that the cost function J is no longer well-defined. On every iteration, you are randomly killing off a bunch of nodes. And so, if you are double checking the performance of grade and dissent, it's actually harder to double check that you have a well defined cost function J that is going downhill on every iteration. Because the cost function J that you're optimizing is actually less. Less well defined, or is certainly hard to calculate.

So you lose this debugging tool to will a plot, a graph like this. So what I usually do is turn off drop out, you will set key prop equals one, and I run my code and make sure that it is monotonically decreasing J, and then turn on drop out and hope that I didn’t introduce bugs into my code during drop out. Because you need other ways, I guess, but not plotting these figures to make sure that your code is working to greatness and it’s working even with drop outs. So with that, there’s still a few more regularization techniques that are worth your knowing. Let’s talk about a few more such techniques in the next section.

## Other regularization methods

In addition to L2 regularization and drop out regularization there are few other techniques to reducing over fitting in your neural network.

### Data Augmentation

Let's take a look. Let's say you fitting a CAD crossfire. If you are over fitting getting more training data can help, but getting more training data can be expensive and sometimes you just can't get more data. But what you can do is augment your training set by taking image like this.

### Early Stopping

There's one other technique that is often used called early stopping. So what you're going to do is as you run gradient descent you're going to plot your, either the training error, you'll use 01 classification error on the training set. Or just plot the cost function J optimizing, and that should decrease monotonically, like so, all right? Because as you trade, hopefully, you're trading around your cost function J should decrease.
So with early stopping, what you do is you plot this, and you also plot your dev set error. And again, this could be a classification error in a development sense, or something like the cost function, like the logistic loss or the log loss of the dev set. Now what you find is that your dev set error will usually go down for a while, and then it will increase from there. So what early stopping does is, you will say well, it looks like your neural network was doing best around that iteration, so we just want to stop trading on your neural network halfway and take whatever value achieved this dev set error. So why does this work? Well when you've haven't run many iterations for your neural network yet your parameters w will be close to zero. Because with random initialization you probably initialize w to small random values so before you train for a long time, w is still quite small. And as you iterate, as you train, w will get bigger and bigger and bigger until here maybe you have a much larger value of the parameters w for your neural network. So what early stopping does is by stopping halfway you have only a mid-size rate w.

And so similar to L2 regularization by picking a neural network with smaller norm for your parameters w, hopefully your neural network is over fitting less. And the term early stopping refers to the fact that you’re just stopping the training of your neural network earlier.

format: plain


text:
# #MonthOfJulia Day 18: Plotting

There’s a variety of options for plotting in Julia. We’ll focus on those provided by Gadfly, Bokeh and Plotly and.

Gadfly is the flavour of the month for plotting in Julia. It’s based on the Grammar of Graphics, so users of ggplot2 should find it familiar.

To start using Gadfly we’ll first need to load the package. To enable generation of PNG, PS, and PDF output we’ll also want the Cairo package.

julia> using Gadfly
julia> using Cairo

You can easily generate plots from data vectors or functions.

julia> plot(x = 1:100, y = cumsum(rand(100) - 0.5), Geom.point, Geom.smooth)
julia> plot(x -> x^3 - 9x, -5, 5)

Gadfly plots are by default rendered onto a new tab in your browser. These plots are mildly interactive: you can zoom and pan across the plot area. You can also save plots directly to files of various formats.

julia> dampedsin = plot([x -> sin(x) / x], 0, 50)
julia> draw(PNG("damped-sin.png", 800px, 400px), dampedsin)

Let’s load up some data from the nlschools dataset in R’s MASS package and look at the relationship between language score test and IQ for pupils broken down according to whether or not they are in a mixed-grade class.

julia> using RDatasets
julia> plot(dataset("MASS", "nlschools"), x="IQ", y="Lang", color="COMB",

Those two examples just scratched the surface. Gadfly can produce histograms, boxplots, ribbon plots, contours and violin plots. There’s detailed documentation with numerous examples on the homepage.

Watch the video below (Daniel Jones at JuliaCon 2014) then read on about Bokeh and Plotly.

## Bokeh

Bokeh is a visualisation library for Python. Bokeh, like D3, renders plots as Javascript, which is viewable in a web browser. In addition to the examples on the library homepage, more can be found on the homepage for Julia’s Bokeh package.

The first thing you’ll need to do is install the Bokeh library. If you already have a working Python installation then this is easily done from the command line:

\$ pip install bokeh

Next load up the package and generate a simple plot.

julia> using Bokeh
julia> autoopen(true);
julia> x = linspace(0, pi);
julia> y = cos(2 * x);
julia> plot(x, y, title = "Cosine")
Plot("Cosine" with 1 datacolumns)

The plot will be written to a file bokeh_plot.html in the working directory, which will in turn be opened by the browser. Use plotfile() to change the name of the file. The plot is interactive, with functionality to pan and zoom as well as resize the plot window.

## Plotly

The Plotly package provides a complete interface to plot.ly, an online plotting service with interfaces for Python, R, MATLAB and now Julia. To get an idea of what’s possible with plot.ly, check out their feed. The first step towards making your own awesomeness with be loading the package.

using Plotly

Next you should set up your plot.ly credentials using Plotly.set_credentials_file(). You only need to do this once because the values will be cached.

Data series are stored in Julia dictionaries.

julia> p1 = ["x" => 1:10, "y" => rand(0:20, 10), "type" => "scatter", "mode" => "markers"];
julia> p2 = ["x" => 1:10, "y" => rand(0:20, 10), "type" => "scatter", "mode" => "lines"];
julia> p3 = ["x" => 1:10, "y" => rand(0:20, 10), "type" => "scatter", "mode" => "lines+markers"];

julia> Plotly.plot([p1, p2, p3], ["filename" => "basic-line", "fileopt" => "overwrite"])
Dict{String,Any} with 5 entries:
"error"    => ""
"message"  => ""
"warning"  => ""
"filename" => "basic-line"
"url"      => "https://plot.ly/~collierab/17"

You can either open the URL provided in the result dictionary or do it programatically:

julia> Plotly.openurl(ans["url"])

By making small jumps through similar hoops it’s possible to create some rather intricate visualisations like the 3D scatter plot below. For details of how that was done, check out my code on github.

Obviously plotting and visualisation in Julia are hot topics. Other plotting packages worth checking out are PyPlot, Winston and Gaston. Come back tomorrow when we’ll take a look at using physical units in Julia.

The post #MonthOfJulia Day 18: Plotting appeared first on Exegetic Analytics.


text:
Max Karl Ernst Ludwig Planck, FRS (/plɑːŋk/; 23 April 1858 – 4 October 1947) was a German theoretical physicist whose work on quantum theory won him the Nobel Prize in Physics in 1918.

Planck made many contributions to theoretical physics, but his fame as a physicist rests primarily on his role as an originator of quantum theory, which revolutionized human understanding of atomic and subatomic processes. However, his name is also known on a broader academic basis, through the renaming in 1948 of the German scientific institution, the Kaiser Wilhelm Society (of which he was twice president), as the Max Planck Society (MPS). The MPS now includes 83 institutions representing a wide range of scientific directions.

Early life and career

Planck came from a traditional, intellectual family. His paternal great-grandfather and grandfather were both theology professors in Göttingen; his father was a law professor in Kiel and Munich.

Planck was born in Kiel, Holstein, to Johann Julius Wilhelm Planck and his second wife, Emma Patzig. He was baptised with the name of Karl Ernst Ludwig Marx Planck; of his given names, Marx (a now obsolete variant of Markus or maybe simply an error for Max, which is actually short for Maximilian) was indicated as the primary name. However, by the age of ten he signed with the name Max and used this for the rest of his life.

He was the 6th child in the family, though two of his siblings were from his father's first marriage. Among his earliest memories was the marching of Prussian and Austrian troops into Kiel during the Second Schleswig War in 1864. In 1867 the family moved to Munich, and Planck enrolled in the Maximilians gymnasium school, where he came under the tutelage of Hermann Müller, a mathematician who took an interest in the youth, and taught him astronomy and mechanics as well as mathematics. It was from Müller that Planck first learned the principle of conservation of energy. Planck graduated early, at age 17. This is how Planck first came in contact with the field of physics.

Planck was gifted when it came to music. He took singing lessons and played piano, organ and cello, and composed songs and operas. However, instead of music he chose to study physics.

The Munich physics professor Philipp von Jolly advised Planck against going into physics, saying, "in this field, almost everything is already discovered, and all that remains is to fill a few holes." Planck replied that he did not wish to discover new things, but only to understand the known fundamentals of the field, and so began his studies in 1874 at the University of Munich. Under Jolly's supervision, Planck performed the only experiments of his scientific career, studying the diffusion of hydrogen through heated platinum, but transferred to theoretical physics.

In 1877 he went to the Friedrich Wilhelms University in Berlin for a year of study with physicists Hermann von Helmholtz and Gustav Kirchhoff and mathematician Karl Weierstrass. He wrote that Helmholtz was never quite prepared, spoke slowly, miscalculated endlessly, and bored his listeners, while Kirchhoff spoke in carefully prepared lectures which were dry and monotonous. He soon became close friends with Helmholtz. While there he undertook a program of mostly self-study of Clausius's writings, which led him to choose thermodynamics as his field.

In October 1878 Planck passed his qualifying exams and in February 1879 defended his dissertation, Über den zweiten Hauptsatz der mechanischen Wärmetheorie (On the second law of thermodynamics). He briefly taught mathematics and physics at his former school in Munich.

In June 1880, he presented his habilitation thesis, Gleichgewichtszustände isotroper Körper in verschiedenen Temperaturen (Equilibrium states of isotropic bodies at different temperatures).

With the completion of his habilitation thesis, Planck became an unpaid private lecturer (Privatdozent) in Munich, waiting until he was offered an academic position. Although he was initially ignored by the academic community, he furthered his work on the field of heat theory and discovered one after another the same thermodynamical formalism as Gibbs without realizing it. Clausius's ideas on entropy occupied a central role in his work.

In April 1885 the University of Kiel appointed Planck as associate professor of theoretical physics. Further work on entropy and its treatment, especially as applied in physical chemistry, followed. He published his Treatise on Thermodynamics in 1897. He proposed a thermodynamic basis for Svante Arrhenius's theory of electrolytic dissociation.

Within four years he was named the successor to Kirchhoff's position at the Friedrich-Wilhelms-Universität in Berlin – presumably thanks to Helmholtz's intercession – and by 1892 became a full professor. In 1907 Planck was offered Boltzmann's position in Vienna, but turned it down to stay in Berlin. During 1909, as a University of Berlin professor, he was invited to become the Ernest Kempton Adams Lecturer in Theoretical Physics at Columbia University in New York City. A series of his lectures were translated and co-published by Columbia University professor A. P. Wills. He retired from Berlin on 10 January 1926, and was succeeded by Erwin Schrödinger.

Family

In March 1887 Planck married Marie Merck (1861–1909), sister of a school fellow, and moved with her into a sublet apartment in Kiel. They had four children: Karl (1888–1916), the twins Emma (1889–1919) and Grete (1889–1917), and Erwin (1893–1945).

After the apartment in Berlin, the Planck family lived in a villa in Berlin-Grunewald, Wangenheimstrasse 21. Several other professors of Berlin University lived nearby, among them theologian Adolf von Harnack, who became a close friend of Planck. Soon the Planck home became a social and cultural centre. Numerous well-known scientists, such as Albert Einstein, Otto Hahn and Lise Meitner were frequent visitors. The tradition of jointly performing music had already been established in the home of Helmholtz.

After several happy years, in July 1909 Marie Planck died, possibly from tuberculosis. In March 1911 Planck married his second wife, Marga von Hoesslin (1882–1948); in December his fifth child Hermann was born.

During the First World War Planck's second son Erwin was taken prisoner by the French in 1914, while his oldest son Karl was killed in action at Verdun. Grete died in 1917 while giving birth to her first child. Her sister died the same way two years later, after having married Grete's widower. Both granddaughters survived and were named after their mothers. Planck endured these losses stoically.

In January 1945, Erwin, to whom he had been particularly close, was sentenced to death by the Nazi Volksgerichtshof because of his participation in the failed attempt to assassinate Hitler in July 1944. Erwin was executed on 23 January 1945.

Professor at Berlin University

As a professor at the Friedrich-Wilhelms-Universität in Berlin, Planck joined the local Physical Society. He later wrote about this time: "In those days I was essentially the only theoretical physicist there, whence things were not so easy for me, because I started mentioning entropy, but this was not quite fashionable, since it was regarded as a mathematical spook". Thanks to his initiative, the various local Physical Societies of Germany merged in 1898 to form the German Physical Society (Deutsche Physikalische Gesellschaft, DPG); from 1905 to 1909 Planck was the president.

Planck started a six-semester course of lectures on theoretical physics, "dry, somewhat impersonal" according to Lise Meitner, "using no notes, never making mistakes, never faltering; the best lecturer I ever heard" according to an English participant, James R. Partington, who continues: "There were always many standing around the room. As the lecture-room was well heated and rather close, some of the listeners would from time to time drop to the floor, but this did not disturb the lecture". Planck did not establish an actual "school"; the number of his graduate students was only about 20, among them:

1897 Max Abraham (1875–1922)
1904 Moritz Schlick (1882–1936)
1906 Walther Meissner (1882–1974)
1906 Max von Laue (1879–1960)
1907 Fritz Reiche (1883–1960)
1912 Walter Schottky (1886–1976)
1914 Walther Bothe (1891–1957)

In 1894 Planck turned his attention to the problem of black-body radiation. He had been commissioned by electric companies to create maximum light from lightbulbs with minimum energy. The problem had been stated by Kirchhoff in 1859: "how does the intensity of the electromagnetic radiation emitted by a black body (a perfect absorber, also known as a cavity radiator) depend on the frequency of the radiation (i.e., the color of the light) and the temperature of the body?". The question had been explored experimentally, but no theoretical treatment agreed with experimental values. Wilhelm Wien proposed Wien's law, which correctly predicted the behaviour at high frequencies, but failed at low frequencies. The Rayleigh–Jeans law, another approach to the problem, created what was later known as the "ultraviolet catastrophe", but contrary to many textbooks this was not a motivation for Planck.

Planck's first proposed solution to the problem in 1899 followed from what Planck called the "principle of elementary disorder", which allowed him to derive Wien's law from a number of assumptions about the entropy of an ideal oscillator, creating what was referred-to as the Wien–Planck law. Soon it was found that experimental evidence did not confirm the new law at all, to Planck's frustration. Planck revised his approach, deriving the first version of the famous Planck black-body radiation law, which described the experimentally observed black-body spectrum well. It was first proposed in a meeting of the DPG on 19 October 1900 and published in 1901. This first derivation did not include energy quantisation, and did not use statistical mechanics, to which he held an aversion. In November 1900, Planck revised this first approach, relying on Boltzmann's statistical interpretation of the second law of thermodynamics as a way of gaining a more fundamental understanding of the principles behind his radiation law. As Planck was deeply suspicious of the philosophical and physical implications of such an interpretation of Boltzmann's approach, his recourse to them was, as he later put it, "an act of despair ... I was ready to sacrifice any of my previous convictions about physics."

The central assumption behind his new derivation, presented to the DPG on 14 December 1900, was the supposition, now known as the Planck postulate, that electromagnetic energy could be emitted only in quantized form, in other words, the energy could only be a multiple of an elementary unit:

${displaystyle E=hnu }$

where h is Planck's constant, also known as Planck's action quantum (introduced already in 1899), and ν (the Greek letter nu, not the Roman letter v) is the frequency of the radiation. Note that the elementary units of energy discussed here are represented by and not simply by ν. Physicists now call these quanta photons, and a photon of frequency ν will have its own specific and unique energy. The total energy at that frequency is then equal to multiplied by the number of photons at that frequency.

At first Planck considered that quantisation was only "a purely formal assumption ... actually I did not think much about it..."; nowadays this assumption, incompatible with classical physics, is regarded as the birth of quantum physics and the greatest intellectual accomplishment of Planck's career (Ludwig Boltzmann had been discussing in a theoretical paper in 1877 the possibility that the energy states of a physical system could be discrete). The discovery of Planck's constant enabled him to define a new universal set of physical units (such as the Planck length and the Planck mass), all based on fundamental physical constants upon which much of quantum theory is based. In recognition of Planck's fundamental contribution to a new branch of physics, he was awarded the Nobel Prize in Physics in 1918.

Subsequently, Planck tried to grasp the meaning of energy quanta, but to no avail. "My unavailing attempts to somehow reintegrate the action quantum into classical theory extended over several years and caused me much trouble." Even several years later, other physicists like Rayleigh, Jeans, and Lorentz set Planck's constant to zero in order to align with classical physics, but Planck knew well that this constant had a precise nonzero value. "I am unable to understand Jeans' stubbornness – he is an example of a theoretician as should never be existing, the same as Hegel was for philosophy. So much the worse for the facts if they don't fit."

Max Born wrote about Planck: "He was, by nature, a conservative mind; he had nothing of the revolutionary and was thoroughly skeptical about speculations. Yet his belief in the compelling force of logical reasoning from facts was so strong that he did not flinch from announcing the most revolutionary idea which ever has shaken physics."

Einstein and the theory of relativity

In 1905, the three epochal papers of the hitherto completely unknown Albert Einstein were published in the journal Annalen der Physik. Planck was among the few who immediately recognized the significance of the special theory of relativity. Thanks to his influence, this theory was soon widely accepted in Germany. Planck also contributed considerably to extend the special theory of relativity. For example, he recast the theory in terms of classical action.

Einstein's hypothesis of light quanta (photons), based on Heinrich Hertz's 1887 discovery (and further investigation by Philipp Lenard) of the photoelectric effect, was initially rejected by Planck. He was unwilling to discard completely Maxwell's theory of electrodynamics. "The theory of light would be thrown back not by decades, but by centuries, into the age when Christiaan Huygens dared to fight against the mighty emission theory of Isaac Newton ..."

In 1910, Einstein pointed out the anomalous behavior of specific heat at low temperatures as another example of a phenomenon which defies explanation by classical physics. Planck and Nernst, seeking to clarify the increasing number of contradictions, organized the First Solvay Conference (Brussels 1911). At this meeting Einstein was able to convince Planck.

Meanwhile, Planck had been appointed dean of Berlin University, whereby it was possible for him to call Einstein to Berlin and establish a new professorship for him (1914). Soon the two scientists became close friends and met frequently to play music together.

World War I

At the onset of the First World War Planck endorsed the general excitement of the public, writing that, "Besides much that is horrible, there is also much that is unexpectedly great and beautiful: the smooth solution of the most difficult domestic political problems by the unification of all parties (and) ... the extolling of everything good and noble."

Nonetheless, Planck refrained from the extremes of nationalism. In 1915, at a time when Italy was about to join the Allied Powers, he voted successfully for a scientific paper from Italy, which received a prize from the Prussian Academy of Sciences, where Planck was one of four permanent presidents.

Planck also signed the infamous "Manifesto of the 93 intellectuals", a pamphlet of polemic war propaganda (while Einstein retained a strictly pacifistic attitude which almost led to his imprisonment, being spared by his Swiss citizenship). But in 1915 Planck, after several meetings with Dutch physicist Lorentz, revoked parts of the Manifesto. Then in 1916 he signed a declaration against German annexationism.

Post War and Weimar Republic

In the turbulent post-war years, Planck, now the highest authority of German physics, issued the slogan "persevere and continue working" to his colleagues.

In October 1920 he and Fritz Haber established the Notgemeinschaft der Deutschen Wissenschaft (Emergency Organization of German Science), aimed at providing financial support for scientific research. A considerable portion of the money the organization would distribute was raised abroad.

Planck also held leading positions at Berlin University, the Prussian Academy of Sciences, the German Physical Society and the Kaiser Wilhelm Society (which in 1948 became the Max Planck Society). During this time economic conditions in Germany were such that he was hardly able to conduct research. In 1926 Planck became a foreign member of the Royal Netherlands Academy of Arts and Sciences.

During the interwar period, Planck became a member of the Deutsche Volks-Partei (German People's Party), the party of Nobel Peace Prize laureate Gustav Stresemann, which aspired to liberal aims for domestic policy and rather revisionistic aims for politics around the world.

Planck disagreed with the introduction of universal suffrage and later expressed the view that the Nazi dictatorship resulted from "the ascent of the rule of the crowds".

Quantum mechanics

At the end of the 1920s Bohr, Heisenberg and Pauli had worked out the Copenhagen interpretation of quantum mechanics, but it was rejected by Planck, and by Schrödinger, Laue, and Einstein as well. Planck expected that wave mechanics would soon render quantum theory—his own child—unnecessary. This was not to be the case, however. Further work only cemented quantum theory, even against his and Einstein's philosophical revulsions. Planck experienced the truth of his own earlier observation from his struggle with the older views in his younger years: "A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it."

Nazi dictatorship and World War II

When the Nazis seized power in 1933, Planck was 74. He witnessed many Jewish friends and colleagues expelled from their positions and humiliated, and hundreds of scientists emigrated from Nazi Germany. Again he tried to "persevere and continue working" and asked scientists who were considering emigration to remain in Germany. He hoped the crisis would abate soon and the political situation would improve.

Otto Hahn asked Planck to gather well-known German professors in order to issue a public proclamation against the treatment of Jewish professors, but Planck replied, "If you are able to gather today 30 such gentlemen, then tomorrow 150 others will come and speak against it, because they are eager to take over the positions of the others." Under Planck's leadership, the Kaiser Wilhelm Society (KWG) avoided open conflict with the Nazi regime, except concerning Fritz Haber. Planck tried to discuss the issue with Adolf Hitler but was unsuccessful. In the following year, 1934, Haber died in exile.

One year later, Planck, having been the president of the KWG since 1930, organized in a somewhat provocative style an official commemorative meeting for Haber. He also succeeded in secretly enabling a number of Jewish scientists to continue working in institutes of the KWG for several years. In 1936, his term as president of the KWG ended, and the Nazi government pressured him to refrain from seeking another term.

As the political climate in Germany gradually became more hostile, Johannes Stark, prominent exponent of Deutsche Physik ("German Physics", also called "Aryan Physics") attacked Planck, Sommerfeld and Heisenberg for continuing to teach the theories of Einstein, calling them "white Jews". The "Hauptamt Wissenschaft" (Nazi government office for science) started an investigation of Planck's ancestry, claiming that he was "1/16 Jewish"; however, Planck himself denied this.

In 1938, Planck celebrated his 80th birthday. The DPG held a celebration, during which the Max-Planck medal (founded as the highest medal by the DPG in 1928) was awarded to French physicist Louis de Broglie. At the end of 1938, the Prussian Academy lost its remaining independence and was taken over by Nazis (Gleichschaltung). Planck protested by resigning his presidency. He continued to travel frequently, giving numerous public talks, such as his talk on Religion and Science, and five years later he was sufficiently fit to climb 3,000-meter peaks in the Alps.

During the Second World War the increasing number of Allied bombing missions against Berlin forced Planck and his wife to temporarily leave the city and live in the countryside. In 1942 he wrote: "In me an ardent desire has grown to persevere this crisis and live long enough to be able to witness the turning point, the beginning of a new rise." In February 1944 his home in Berlin was completely destroyed by an air raid, annihilating all his scientific records and correspondence. His rural retreat was threatened by the rapid advance of the Allied armies from both sides. After the end of the war Planck, his second wife, and his son by her were brought to a relative in Göttingen, where Planck died on 4 October 1947. His grave is situated in the old Stadtfriedhof (City Cemetery) in Göttingen.

In 1945, Planck's son Erwin was arrested due to the attempted assassination of Hitler in the July 20 plot. Erwin consequently died at the hands of the Gestapo; his death destroyed much of Max Planck's will to live.

Religious views

Planck was a member of the Lutheran Church in Germany. However, Planck was very tolerant towards alternative views and religions. In a lecture in 1937 entitled "Religion und Naturwissenschaft" he suggested the importance of these symbols and rituals related directly with a believer's ability to worship God, but that one must be mindful that the symbols provide an imperfect illustration of divinity. He criticized atheism for being focused on the derision of such symbols, while at the same time warned of the over-estimation of the importance of such symbols by believers.

Max Planck said in 1944, "As a man who has devoted his whole life to the most clear headed science, to the study of matter, I can tell you as a result of my research about atoms this much: There is no matter as such. All matter originates and exists only by virtue of a force which brings the particle of an atom to vibration and holds this most minute solar system of the atom together. We must assume behind this force the existence of a conscious and intelligent mind. This mind is the matrix of all matter."

Planck regarded the scientist as a man of imagination and faith. He said: "Both Religion and science require a belief in God. For believers, God is in the beginning, and for physicists He is at the end of all considerations… To the former He is the foundation, to the latter, the crown of the edifice of every generalized world view".

On the other hand, Planck wrote, "...'to believe' means 'to recognize as a truth,' and the knowledge of nature, continually advancing on incontestably safe tracks, has made it utterly impossible for a person possessing some training in natural science to recognize as founded on truth the many reports of extraordinary occurrences contradicting the laws of nature, of miracles which are still commonly regarded as essential supports and confirmations of religious doctrines, and which formerly used to be accepted as facts pure and simple, without doubt or criticism. The belief in miracles must retreat step by step before relentlessly and reliably progressing science and we cannot doubt that sooner or later it must vanish completely."

Later in life, Planck's views on God were that of a deist. For example, six months before his death a rumour started that Planck had converted to Catholicism, but when questioned what had brought him to make this step, he declared that, although he had always been deeply religious, he did not believe "in a personal God, let alone a Christian God."


text:
# Neural network cost function - why squared error?

Question: Why is the squared error most often used for training neural networks?

Context:

• The key factor that informs these adjustments is how "wrong" the untrained network is.
• The actual output and the desired output (training examples) will be different in an untrained network.
• This difference (target - actual) is the error. There is an error for each output layer node.

Learning:

• The link weights are adjusted so that the overall error of the network is minimised for each training example.
• Most guides will use a cost function $Cost=\sum_{output\ nodes}{(target-actual)^2}$ to minimise. They often don't give a reason for this function, but if they do they say:

1. the (target-actual)2 is always positive so errors of different signs don't cancel out, misrepresenting how "wrong" the network is,
2. the cost function is differentiable so we can work out the sensitivity of the error to individual weights, the partial differential $\frac{\delta{Cost}}{\delta{w}}$.
3. parallels to linear regression, where errors are assumed to be Gaussian, that is, distributed normally around the true value

Challenge:

• We don't need the cost function to always be positive because we don't actually sum over the errors from each node. Instead we consider each output node in turn when we propagate the errors back. The errors at the output nodes are independent of each other.
• You can find $\frac{\delta{Error}}{\delta{w}}$ from the simpler $Error=(target-actual)$. There is no problem deriving the partial differentials from this $Error$ expression. Instead of trying to minimise it, we instead move towards zero from either direction.
• There is no reason to believe the errors in the output of a neural network are distributed normally.

Derivation of Simpler Weight Learning

• The error at the $jth$ output node is (target - actual), or $e_j = (t_j - o_j)$.
• The actual output, $o_j = sigmoid(\sum_i{w_i.x_i})$ where $w_i$ is weight of the $ith$ link to that $jth$ node, and $x_i$ is the output from the preceding $ith$ node. Sigmoid is the popular squashing function $\frac{1}{(1+e^{-x})}$.
• So $\frac{\delta{e_j}}{\delta{w_k}} = \frac{\delta{}}{\delta{w_k}}(t_j - o_j)= \frac{\delta{}}{\delta{w_k}}(-sigmoid(\sum_i{w_i.x_i}))$ because we can ignore the constant $t_j$. We can also see that only one $w_{i=k}$ matters as other $w_{i\neq k}$ don't contribute to the differential.
• That leaves us with $\frac{\delta{e_j}}{\delta{w_k}} = - \frac{\delta{}}{\delta{w_k}}(sigmoid(w_k. x_k)) = - (w_k.x_k)(1-w_k.x_k).x_k$ because we know how to differentiate the sigmoid.

So we have an expression for $\frac{\delta{e_j}}{\delta{w_k}}$ which can be used to guide the iterative refinement of the weight $w_k$ so that $e_j$ moves towards zero.

Let's unfold that a bit more. If the error at a node $e_j = (t_j - o_j)$ is positive, and the gradient with respect to weight $w_k$ is positive, we reduce the weight. If the gradient is negative, we increase the weight. In this way we inch towards zero error.

The opposite applies if the error is negative. If the gradient is positive we increase the weight a bit, and if it is negative, we decrease the weight a bit.

So what's wrong with this analysis? And why do so many textbooks, papers and guides not explain their choice of the squared error as a cost function?

I've looked through dozens of papers, dozens of website guides, and about 10 properr textbooks.

• you can use any cost function, but it is equivalent to change the activation function of the output neurons Jan 27 '16 at 23:31
• You are thinking in one dimension. Network often has multiple outputs. So tell me how do you measure the deviation of two vectors $v$ and $y$ in $R^3$? Your answer leads you to the square loss Oct 11 at 1:26

Sum of squares is quite standard cost function in linear regression modelling. For multivariate classification, more commonly cross entropy is used, in addition to the softmax formula applied to the multivariate output.

When it comes to your question, I will base my answer on the assumption that what you are proposing here is the usage of the new (simpler?) error function: $Cost = (target - output)$ (written in the vector form). (Sorry, I do not have reputation high enough to ask you for a clarification in the comment.)

First, let us recall that vanilla gradient update per weight in a neural network is as follows: $\ w_i = \ w_i - \eta \cdot \frac{\delta{Cost}}{\delta{w_i}} \tag{1}$ where $\eta$ is the learning rate.

Now let's look at the example of logistic neuron having two inputs $x_1$ and $x_2$, two weights $w_1$ and $w_2$ (associated with each input) and one bias $b$. Logistic neuron applies sigmoid/logistic function $\sigma(z)$ to the value $z = x_1*w_1 + x_2*w_2 + b$, so the produced output of the neuron is as follows:

$output = \sigma(z) \equiv \frac{1}{1+e^{-z}}\tag{2}$

Suppose we are using sum of squares cost function, with one training sample in online learning:

$Cost=1/2*{(target-output)^2}\tag{3}$

We would have the following derivative for weight $w_1$ (based on (3) and (2)):

$\frac{\delta{Cost}}{\delta{w_1}} = \frac{\delta{Cost}}{\delta\sigma(z)}*\frac{\delta\sigma(z)}{\delta\ z}*\frac{\delta\ z}{\delta \ w_1} = -(target - output)*\sigma(z)(1-\sigma(z))*x_1 \tag{4}$

So when we put (4) into (1), we would have the following weight update:

$\ w_1 = \ w_1 + \eta*(target - output)*\sigma(z)(1-\sigma(z))*x_1 \tag{5}$

So each weight is updated proportionally to the difference between the predicted (output) and true (target value) and proportionally to the value of the input.

Instead, if we would use the error function $Cost = (target - output)$ and modify formula (1) as follows:

$\ w_i = \ w_i (-/+) \eta \cdot \frac{\delta{Cost}}{\delta{w_i}} \tag{6}$ so that is uses + if (target - output) is negative and - otherwise, then formula (5) would look as follows:

$\ w_1 = \ w_1 (-/+) \eta*\sigma(z)(1-\sigma(z))*x_1 \tag{7}$

As you can notice, derivatives would not be proportional to the difference between the predicted (output) and true (target value) output. In my opinion, there is nothing wrong there, but with first approach, formula (1) is consistent and it works with many cost functions, and what is more important, the learning process is faster as the updated weights will not oscillate around optimal solution - as the learning converges derivatives are smaller and smaller (I did not try to simulate this.)

• And what if one is using ReLU, or any ReLU-like activations? And in the case of the last layer, maybe, even no activation at all? Jun 6 '18 at 22:33

Understanding this answer demands some basic familarity with probability and gaussian distributions. If you are commfortable with these topics, then continue, else first look up for them in the internet and then come back to this answer.

The mean square error is a consequence of performing maximum log-likelihood estimation over the conditional probability distribution of the output. Maximizing the log likelohood is same as minimizing binary cross entropy between the true data generating distribution and the distribution learned by the model. Since this question is not about explaining cross entropy, I'll give a reference and skip that. https://rdipietro.github.io/friendly-intro-to-cross-entropy-loss/

What is maximum likelihood estimation ?

Maximum likelihood estimation (MLE) is a method of estimating the parameters of a statistical model, given observations.

Linear regression as maximum likelihood

The maximum likelihood estimator can readily be generalized to the case where our goal is to estimate a conditional probability P(y | x; θ) in order to predict y given x. This is actually the most common situation because it forms the basis for most supervised learning. If X represents all our inputs and Y all our observed targets, then the conditional maximum likelihood estimator is The above screenshots were taken from the Deeplearning book by Ian Goodfellow et al. To read more about estimators read this chapter: http://www.deeplearningbook.org/contents/ml.html

Now we have the base to consider maxmimum likelihood estimation in the context of neural networks.

Mean square error in neural networks

The above screenshots were taken from book: Pattern recognition and machine learning by Bishop.

Hope this helps in understanding use of mean squared error in machine learning. If you have any queries or issues regarding this answer, please write them down in comments.

• This answer shows the equivalence between maximizing the likelihood function and minimizing the sum-of-squares error function. Jul 26 '18 at 7:58

Sum of squares is a usual way to turn multiobjective optimisation (we want all the nodes to have low error) to single objective (we optimise over a single number) , which is much more convenient for usual optimisation like gradient descent.

Since the error is not expected to be zero everywhere, the squares expresses that we would like the errors to have similar values (rather than having some nodes with very small error and others with very large). Scale is an issue in this reasoning, but since the output nodes values are bounded by sigmoidal functions this is not an issue.

I think your reasoning ultimately expresses the sum of absolute values of the errors. Indeed, when you can see that optimising a sum of absolute values is quite similar to optimising each one separately.

Sorry if I am being simplistic, I wrote this on my mobile, just to provide some intuition, not a complete answer

• Thanks for taking the time to reply, I appreciate it. The bit I don't understand is this .... if consider updating the link weight W which connects to an output node N ... our calculus only takes into account the absolute error produced by N and no other node. .... so I am struggling to understand the "global" effect of summing and squaring over all output node errors ... when in fact the mechanics of weight update only take the "local" error relevant to the linked node..... I do appreciate you taking the time .. I will keep trying. Jan 22 '16 at 15:19
• thinking about this further ... if I'm updating a weight W and using the error out of output node N .... the fact that a different node M has a huge error doesn't affect the update to weight W .. which is why this is "local" not "global" as some have suggested the sum squared cost function tries to be. Jan 22 '16 at 15:45
• A friend suggested the answer to this lies in the fact that the cost function should be defined over the entire training data set not per training example. In this case the errors being summed over many training examples must be such that they don't cancel out .. And squaring the (target-output) is one valid way of doing this. Some approaches update weights after each training example (online learning? stochastic gradient descent?) and this works only because the cost function per example is sufficiently similar to the true global cost function where taking small gradient descent steps. Jan 22 '16 at 20:37

I derived weight update expressions using the naive error cost function.

The key here is that we're not minimising it, we're trying to get it to zero - from both directions.

Seems to work well! See blog post, with example code and results.

http://makeyourownneuralnetwork.blogspot.co.uk/2016/01/why-squared-error-cost-function.html


text:
# Volcano Watch — Global Positioning System accuracy improved for volcanologists and all civilians

Release Date:

On May 1, 2000, President Clinton announced the United States' decision to stop degrading Global Positioning System (GPS) accuracy. The act has made it possible for civilians to obtain positions as precise as 15 meters (50 ft) using handheld portable GPS receivers. It benefits us in the volcano world, too.

On May 1, 2000, President Clinton announced the United States' decision to stop degrading Global Positioning System (GPS) accuracy. The act has made it possible for civilians to obtain positions as precise as 15 meters (50 ft) using handheld portable GPS receivers. It benefits us in the volcano world, too.

GPS is a satellite-based system that provides accurate location and timing data to users worldwide. Its origin was for military defense purposes. The Department of Defense had a long-standing policy of intentional degradation of the satellite signals, a process called Selective Availability.

GPS works by a system of precise clock-based radio signals. Selective availability works by allowing the clocks in the satellites to drift. At a stationary ground-based receiver, the tiny variations in the time signals make it appear as though the distance to the satellite is changing. Special codes were required to compensate for the clock drift. Anyone using the system without codes had to operate dual receivers in order to correct for the clock variations.

We use GPS in several ways at the Hawaiian Volcano Observatory. Foremost is for our surface-deformation studies. Permanently sited and temporary GPS receivers transmit data about their position to HVO, and small movements in ground surface, as little as 1-2 mm (one-tenth of an inch), are detected. These changes may indicate magmatic changes in the shallow crust that lead to changes in eruption status at Kīlauea and Mauna Loa volcanoes.

We have always surmounted the problem of degraded GPS signals by operating numerous sites and sampling the data simultaneously. Drift in the satellite clock is canceled out by this method. Consequently, we'll see no substantial improvements to our detailed deformation monitoring from these recent changes.

Probably the greatest immediate enhancement will be in our geologic mapping and sample-site information. We map new lava flows using portable GPS receivers. Since 1996 we've had special Department of Defense-authorized GPS receivers that possess the precise codes needed to maintain position despite degradation of satellite signals. But these receivers could only be used by our salaried employees.

Volunteers on our staff could use commercially available receivers, but the precision of these instruments was no better than from 50 to 100 m (160-320 ft). Now these same receivers maintain precision of about 15 m (50 ft), and relative positioning from point to point is commonly much better. Thus, we're able to use our volunteer staff to much greater advantage in helping us monitor the distribution of new lava flows erupted at Kīlauea volcano.

We and Earth scientists worldwide have been using portable GPS receivers to record the position of drill holes, outcrops, and samples collected for radiocarbon dating or chemical analysis. When satellite signals are intentionally degraded, these location data are rarely much better than the reckoning we customarily do using map and compass or air photos. With degradation stopped, handheld GPS positions used in open country are now as precise as the topographic maps of the region. Under forest canopy, especially the dense Hawaiian rain forest, all portable GPS receivers have poor reception and, consequently, poor positional data.

Many hikers and hunters in Hawaii have handheld GPS receivers. A typical unit costs about \$150 and may be ordered from any outdoor-equipment store. These users will realize a dramatic improvement in GPS accuracy now that signal degradation has been stopped. They won't be able to detect ground surface changes of a few millimeters, but they could map a new lava flow with as much precision as we can.

### Volcano Activity Update

Eruptive activity of Kīlauea Volcano continued unabated during the past week. Lava is erupting from Puu Oo and flowing through a network of tubes toward the coast near the eastern boundary of Hawaii Volcanoes National Park. The tube system is now well-established, and surface flows from breakouts of the tube system are seen uncommonly on the coastal flats these days. Lava is entering the ocean mainly at Wahaula and at a site 1.1 kilometers (1200 yards) to the west. The Wahaula site is explosive intermittently. The ocean-entry areas are extremely hazardous, with explosions accompanying sudden collapses of the new land. The active lava flows are hot and have places with very thin crust. The steam clouds are highly acidic and laced with glass particles.

No felt earthquakes were reported this week.


text:
American Institute of Mathematical Sciences

November  2016, 21(9): 3053-3073. doi: 10.3934/dcdsb.2016087

Well-posedness of stochastic primitive equations with multiplicative noise in three dimensions

 1 Jiangsu Provincial Key Laboratory for Numerical Simulation of Large Scale Complex Systems, School of Mathematical Science, Nanjing Normal University, Nanjing 210023 2 Department of Applied Mathematics, Nanjing University of Finance and Economics, Nanjing 210023, China

Received  November 2015 Revised  February 2016 Published  October 2016

Three dimensional primitive equations with a small multiplicative noise are studied in this paper. The existence and uniqueness of solutions with small initial value in a fixed probability space are obtained. The proof is based on Galerkin approximation, Itô's formula and weak convergence methods.
Citation: Hongjun Gao, Chengfeng Sun. Well-posedness of stochastic primitive equations with multiplicative noise in three dimensions. Discrete & Continuous Dynamical Systems - B, 2016, 21 (9) : 3053-3073. doi: 10.3934/dcdsb.2016087
References:

show all references

References:
 [1] Christian Beck, Lukas Gonon, Martin Hutzenthaler, Arnulf Jentzen. On existence and uniqueness properties for solutions of stochastic fixed point equations. Discrete & Continuous Dynamical Systems - B, 2020  doi: 10.3934/dcdsb.2020320 [2] Stefano Bianchini, Paolo Bonicatto. Forward untangling and applications to the uniqueness problem for the continuity equation. Discrete & Continuous Dynamical Systems - A, 2020  doi: 10.3934/dcds.2020384 [3] Yichen Zhang, Meiqiang Feng. A coupled $p$-Laplacian elliptic system: Existence, uniqueness and asymptotic behavior. Electronic Research Archive, 2020, 28 (4) : 1419-1438. doi: 10.3934/era.2020075 [4] Leilei Wei, Yinnian He. A fully discrete local discontinuous Galerkin method with the generalized numerical flux to solve the tempered fractional reaction-diffusion equation. Discrete & Continuous Dynamical Systems - B, 2020  doi: 10.3934/dcdsb.2020319 [5] Yongxiu Shi, Haitao Wan. Refined asymptotic behavior and uniqueness of large solutions to a quasilinear elliptic equation in a borderline case. Electronic Research Archive, , () : -. doi: 10.3934/era.2020119 [6] Ahmad Z. Fino, Wenhui Chen. A global existence result for two-dimensional semilinear strongly damped wave equation with mixed nonlinearity in an exterior domain. Communications on Pure & Applied Analysis, 2020, 19 (12) : 5387-5411. doi: 10.3934/cpaa.2020243 [7] Xuhui Peng, Rangrang Zhang. Approximations of stochastic 3D tamed Navier-Stokes equations. Communications on Pure & Applied Analysis, 2020, 19 (12) : 5337-5365. doi: 10.3934/cpaa.2020241 [8] Marion Darbas, Jérémy Heleine, Stephanie Lohrengel. Numerical resolution by the quasi-reversibility method of a data completion problem for Maxwell's equations. Inverse Problems & Imaging, 2020, 14 (6) : 1107-1133. doi: 10.3934/ipi.2020056 [9] Zexuan Liu, Zhiyuan Sun, Jerry Zhijian Yang. A numerical study of superconvergence of the discontinuous Galerkin method by patch reconstruction. Electronic Research Archive, 2020, 28 (4) : 1487-1501. doi: 10.3934/era.2020078 [10] Yue Feng, Yujie Liu, Ruishu Wang, Shangyou Zhang. A conforming discontinuous Galerkin finite element method on rectangular partitions. Electronic Research Archive, , () : -. doi: 10.3934/era.2020120 [11] Jerry L. Bona, Angel Durán, Dimitrios Mitsotakis. Solitary-wave solutions of Benjamin-Ono and other systems for internal waves. I. approximations. Discrete & Continuous Dynamical Systems - A, 2021, 41 (1) : 87-111. doi: 10.3934/dcds.2020215 [12] Mark F. Demers. Uniqueness and exponential mixing for the measure of maximal entropy for piecewise hyperbolic maps. Discrete & Continuous Dynamical Systems - A, 2021, 41 (1) : 217-256. doi: 10.3934/dcds.2020217 [13] Shenglan Xie, Maoan Han, Peng Zhu. A posteriori error estimate of weak Galerkin fem for second order elliptic problem with mixed boundary condition. Discrete & Continuous Dynamical Systems - B, 2020  doi: 10.3934/dcdsb.2020340 [14] Zedong Yang, Guotao Wang, Ravi P. Agarwal, Haiyong Xu. Existence and nonexistence of entire positive radial solutions for a class of Schrödinger elliptic systems involving a nonlinear operator. Discrete & Continuous Dynamical Systems - S, 2020  doi: 10.3934/dcdss.2020436 [15] Zhilei Liang, Jiangyu Shuai. Existence of strong solution for the Cauchy problem of fully compressible Navier-Stokes equations in two dimensions. Discrete & Continuous Dynamical Systems - B, 2020  doi: 10.3934/dcdsb.2020348 [16] Thabet Abdeljawad, Mohammad Esmael Samei. Applying quantum calculus for the existence of solution of $q$-integro-differential equations with three criteria. Discrete & Continuous Dynamical Systems - S, 2020  doi: 10.3934/dcdss.2020440 [17] Gongbao Li, Tao Yang. Improved Sobolev inequalities involving weighted Morrey norms and the existence of nontrivial solutions to doubly critical elliptic systems involving fractional Laplacian and Hardy terms. Discrete & Continuous Dynamical Systems - S, 2020  doi: 10.3934/dcdss.2020469 [18] Peter Poláčik, Pavol Quittner. Entire and ancient solutions of a supercritical semilinear heat equation. Discrete & Continuous Dynamical Systems - A, 2021, 41 (1) : 413-438. doi: 10.3934/dcds.2020136 [19] Jianhua Huang, Yanbin Tang, Ming Wang. Singular support of the global attractor for a damped BBM equation. Discrete & Continuous Dynamical Systems - B, 2020  doi: 10.3934/dcdsb.2020345 [20] Siyang Cai, Yongmei Cai, Xuerong Mao. A stochastic differential equation SIS epidemic model with regime switching. Discrete & Continuous Dynamical Systems - B, 2020  doi: 10.3934/dcdsb.2020317

2019 Impact Factor: 1.27


text:
# Exponential sums and congruences

1. Oct 27, 2008

### mhill

let be the exponential sum

$$S= \sum_{n=1}^{N}e( \frac{f(x)}{p})$$

$$e(x)= exp( 2i \pi x)$$

my conjecture is that since the complex exponential takes its maximum value '1' when x is equal to an integer then

$$Re(S)= \Pi (f,N)$$ with $$\Pi (f,N)$$ is the number of solutions on the interval (1,N) of the congruence

$$f(x) =0 mod(p)$$ and f(x) is a Polynomial.

2. Nov 4, 2008

### yasiru89

Forgive me if this is a stupid question- but what's [itex]p[/tex]? Or did you mean [itex]n[/tex] instead or [itex]p[/tex] as the number of prime factors of [itex]n[/tex] or something?

3. Nov 5, 2008

any prime

4. Nov 5, 2008

### yasiru89

I still don't get over what exactly that summation for [itex]S[/tex] is done. A clarification please?

5. Nov 6, 2008

### mhill

o sorry.. i should have written

$$S= \sum_{n=1}^{N}e( \frac{f(n)}{p})$$

the sum is taken over 'n' but if the prime 'p' divides f(n) then the complex exponential is equal to '1'

6. Nov 6, 2008

### yasiru89

Okay then [itex]p[/tex] is a prime of one's choosing.

We have,

$$S = \sum_{n = 1}^{N} \exp{\left(\frac{2\pi i}{p}f(n)\right)}$$

Then,
$$\Re(S) = \sum_{n = 1}^{N} \cos{\left(\frac{2\pi}{p}f(n)\right)}$$

If [itex]f(n)[/tex] is a multiple of [itex]p[/tex], then the the real part of [itex]S[/tex] will 'count' each solution of that congruence, but what about certain [itex]f(n)[/tex] values that don't and hence give rise to non-zero real and imaginary components? They won't be 1 in a single go, but they can possibly accumulate to values greater than 1 I think. So some bounds for such a theorem also become necessary if I haven't missed anything.

Last edited: Nov 6, 2008