text:
# Why does a cathode have to be heated to emit electrons?

Considering that electrons are highly mobile inside of a metal, why do they have such a tough time getting out at the edge of it and continuing their trip ballistically?

• Look up something called the "work function". Aug 10 '13 at 12:01

Matter is held together by the electrical attraction between the electrons and the nuclei. Within the bulk of a solid or liquid, an electron feels these attractions from all directions equally, and therefore the force it experiences equals zero on the average. But if an electron finds itself at the surface, this isotropy is broken. The electron feels attractive forces toward the interior, which are not canceled out by any forces from the outside. Normally this causes any electron that impinges on the surface to be reflected back in like a pool ball hitting a cushion. To extract the electron out beyond the surface, you have to do a certain amount of work, called the work function $W$. The work function for a metal is typically about 5 eV.

Why does a cathode have to be heated to emit electrons?

It's not actually true that it has to be heated -- cold-cathode devices do exist, and thermionic emission does occur at all temperatures. However, at room temperature, $kT\approx 0.03\ eV$, which is much smaller than $W$. That means that only a very tiny fraction of the electrons have more energy than $W$. The probability of having an energy $W$ at temperature $kT$ goes like $e^{-W/kT}$, and Richardson found in 1901 that the current from a cathode, in the absence of an externally applied electric field, was proportional to $T^2e^{-W/kT}$.

• In my opinion, very weak explanation. Given that electron feels attractive force once it "finds itself at the surface", it would take him some time to stop and reverse the speed. In this case, we could observe electrons "flying out of material", and these electrons could easily be extracted. Also, the net electric force which affects the electron once it is out of the lattice is very weak due to screening effects. Aug 10 '13 at 8:31
• If the work function is around 5 eV, shouldn't cathodes that have a potential drop to the anode of more than 5V start emitting without heat? Aug 12 '13 at 20:37
• @SebastianHenckel: No, what matters locally is the electric field. The electric field required in order to cancel the electric field holding the electrons in at the surface is $\sim(5\ \text{V})/(0.1\ \text{nm})$. This is basicallly the strength of the microscopic fields, and if you apply it externally to matter, the matter is destroyed.
– user4552
Aug 12 '13 at 20:56

Your main mistake is that you think about electrons being moving along the lattice ballistically.

There at least two answers to this question: the classical one, and the QM one.

The classical answer is that even when the electrons are "highly mobile inside of a metal", they just hop between adjacent atoms. When it is in the atom, it resides on one of the allowed potential energy levels - exactly the same situation as encountered with a single atom. In order to remove the electron from the atom it resides in, you must supply the required amount of energy (which is called Work Function; Work Function of a material differs from ionization energy of material's element).

The QM answer is that when you stick all these atoms together, the orbitals of the valence electrons change such that they span the entire lattice (in fact, these orbitals even go slightly beyond the volume defined by the lattice). The above statement means that the probability of finding an electron far from the lattice it belongs to is negligible.

• I don't think the band structure is particularly relevant, and quantum mechanics is not really needed to understand the behavior of Richardson's Law.
– user4552
Aug 9 '13 at 23:53
• @BenCrowell, as I said, there are at least two ways to explain this intuitively. To my best knowledge, quantitative explanation of thermionic emission requires extensive use of QM. I tried to provide some understanding as to "why this doesn't happen". Aug 10 '13 at 8:24
• @BenCrowell, the question is not regarding Richardson law. OP wanted to know why electrons are not "shot" out of metal. Aug 10 '13 at 8:32


text:
Free sets for a set-mapping relative to a family of sets

23 Dec 2017 Avilés Antonio Piña Claribet

Given a family $\mathcal{F}$ of subsets of $\{1,\ldots,m\}$, we try to compute the least natural number $n$ such that for every function $S:[\aleph_n]^{<\omega}\longrightarrow [\aleph_n]^{<\omega}$ there exists a bijection $u:\{1,\ldots,m\}\longrightarrow Y\subset \aleph_n$ such that $Su(A)\cap Y \subset u(A)$ for all $A\in\mathcal{F}$...

PDF Abstract

format: plain


text:
Conjugate Metric Tensor

• March 2nd 2009, 07:14 PM
Kiwi_Dave
Conjugate Metric Tensor
Given that:

$g_{ij}=E_i \cdot E_j$

is it true and how would I show that:

$g^{ij}=E^i \cdot E^j$

I am sure that this should be easy, but my brain is fried.


text:
# Finding a subset of triplets of digits 0-9 such that each digit occurs 40 times in each position in the triplets

I am trying to generate a list of digit triplets to specify stimuli in an auditory (speech-in-noise) perception experiment. Each triplet has to have three different digits (i.e., no repetition within triplets). I have recordings of digits 0-9, recorded in each triplet position (i.e., I have 30 digits, 0 in 1st position, 0 in 2nd position, 0 in 3rd position, 1 in 1st position, 1 in 2nd position, etc...). This is for a homogenization task, i.e., a task for figuring out the signal to noise ratio (SNR) for each digit in each position so that the level of individual digits can be adjusted upward or downward to make them equally intelligible at a fixed noise level.

In the homogenization task, stimuli will be presented at 10 SNRs, and I want each digit in each position to be presented four times at each SNR, but in different triplets. Hence, I want 4 x 10 x 10 triplets total. (The four repetitions is somewhat arbitrary, but it's based on previous work, and 400 trials works out nicely with respect to the amount of time listeners will spend doing the task.)

There are 720 total triplets. I use Python, so it's easy to get the full list with:

all_triplets = list(itertools.permutations(np.arange(10),3))

I'm finding it very difficult to get a subset of 400 of the 720 possible triplets such that each digit occurs in each position exactly 40 times. That is, I want 400 triplets such that

np.sum(triplets[:,c]==d) == 40

is True for all c in [0,1,2] and all d in [0,1,2,3,4,5,6,7,8,9], and with triplets a 400 x 3 numpy array containing the subset.

I think that such subsets exist, but I am not sure, since I haven't buckled down to try to work out the math (I may not have the ability to do so even if I do buckle down, and I definitely don't have the time to do so, regardless). I can't figure out how to construct any such subsets from scratch (I wrote some aggressively mediocre code to do this that kind of works sometimes) or to select the right elements from the full set (assuming there are some).

I wrote some less mediocre code to sample randomly from the full subset and score each subset based on how much it deviates from the 40-of-each-digit criterion, and I found a subset with these tallies (rows are triplet position, columns are digits):

[[43, 41, 39, 40, 37, 40, 42, 42, 37, 39],
[43, 41, 39, 38, 41, 39, 44, 38, 40, 37],
[43, 38, 40, 38, 38, 41, 42, 44, 39, 37]]

Of course, randomly sampling from approximately 1.92e+213 total subsets of size 400 from the 720 total set of triplets isn't terribly efficient.

So, is there a solution to this problem? A commenter on reddit's intuition is that this problem is NP-complete, but I don't know enough about the relevant math or computer science to have particularly strong intuitions about this possibility. Should I just be happy with the best solution I stumbled on?

You could try using integer linear programming (ILP). Use the following variables:

• We have a variable $x_{ijk}$ for each triplet $ijk$, which is 1 if triplet $ijk$ is selected to be included in your set of 400, or 0 otherwise.

• We have a variable $y_{ip}$ to count the number of times that digit $i$ appears in position $p$ among the selected triplets.

• We want to select 400 triplets: $\sum_{i,j,k} x_{ijk} = 400$.

• We want each digit in each position to appear 40 times: $\sum_{i,p} y_{ip} = 40$.

• We want the $x$'s and $y$'s to be consistent:

\begin{align} y_{i1} &= \sum_{j,k} x_{ijk}\\ y_{j2} &= \sum_{i,k} x_{ijk}\\ y_{k3} &= \sum_{i,j} x_{ijk} \end{align}

Finally, we require $0 \le x_{ijk} \le 1$ and $0 \le y_{ip} \le 1$ for all variables.

Now give the resulting system to an integer linear programming solver, and see if it can find a solution.

If it can't find a solution after a reasonable amount of time, your next step could be to find as small a subset as possible (not necessarily size 400, but hopefully not too much larger than 400). Thus, replace $\sum_{i,p} y_{ip} = 40$ with $\sum_{i,p} y_{ip} \ge 40$, and instead of the constraint $\sum_{i,j,k} x_{ijk} = 400$, ask the ILP solver to minimize the objective function $\sum_{i,j,k} x_{ijk}$. That will find a way to ensure each digit is presented at least as many times as desired, using as few trials as possible.

• I have never heard of ILP. I will look into it, though. Thanks for taking the time to answer. – Noah Motion Jun 27 '17 at 23:38


text:
# How to properly make reftex work with multiple biblatex bibs?

I saw a comment in here that support for multiple addbibresource has been integrated into Emacs. However I'm using a few custom stys from where I require biblatex package with all the necessary parameters.

For some reason in spite of TeX-parse-self set to non-nil, I'm still getting nil for (reftex-using-biblatex-p). I really don't want to move requirement for biblatex out of sty file such that "poor-man's check" for \usepackage would work in reftex-using-biblatex-p. I'd consider it as a last resort.

Is there a workaround to add biblatex into TeX-active-styles? Did I miss something?

Update

Here is an example. Run it once, then open in Emacs, comment out common.sty producing block (select and hit M-;), revisit the file (C-x C-v), and, finally, try C-c [ to add b2016. It won't find it, but will find a2015.

% runme with latexmk -pdf thisfile.tex
\documentclass{article}

\usepackage{filecontents}
\begin{filecontents}{common.sty}
\ProvidesPackage{common}
\RequirePackage[backend=biber,
backref=true,
sorting=nyvt,
url=false,
isbn=false,
doi=true,
eprint=false,
firstinits=true,
mincitenames=3,
maxcitenames=3,
maxbibnames=99,
style=authoryear]{biblatex}
\end{filecontents}

\begin{filecontents}{test1.bib}
@article{a2015,
title = {Test 1},
volume = {66},
pages = {423--430},
number = {6},
journaltitle = {Some journal},
author = {Author, B.C.},
date = {2015-03-03},
}
\end{filecontents}

\begin{filecontents}{test2.bib}
@article{b2016,
title = {Test 2},
volume = {33},
pages = {123--234},
number = {2},
journaltitle = {Another journal},
author = {Creator, D.E.},
date = {2016-03-03},
}
\end{filecontents}

\usepackage{common}

\begin{document}

hello\parencite{b2016}.

\printbibliography
\end{document}

Update 2

The above scenario was tested on 64 bit Emacs 25.0.50.1 and AUCTeX 11.88 on MS Windows. Original value for reftex-bibliography-commands already contains addbibresource.

Update 3

Save the following into startup.el and use emacs -q --load startup.el.

(setq TeX-parse-self t) ; Enable parse on load.
(setq TeX-auto-save t) ; Enable parse on save.
(setq reftex-save-parse-info t) ; I didn't know I had it either in my config

Apparenty AUCTeX is not involved at all. Also it looks like only real \usepackage{biblatex} results in both bibs in rel file, i.e. filecontents trick with a dummy file won't work. I wonder if AUCTeX affected TeX-active-styles and now only "poor-man's check" work in (reftex-using-biblatex-p).

If you have no \usepackage{biblatex} and no rel file, the latter will be created with only first bib. Once you add \usepackage{biblatex}, it won't affect rel unless you delete it.

• I'm a bit confused as with this example, I do get biblatex in TeX-active-styles, but things still do not show up in reftex browser. – mlt Mar 3 '15 at 23:03
• I can confirm RefTeX has some problems, but I think the example should be simplified: load biblatex with a simple \usepackage, don't write a new style file to an external file. I suggest you to report the bug with M-x reftex-report-bug <RET>. – giordano Mar 4 '15 at 11:05
• I stand corrected: I tested again properly, your example works for me (provided that you create the parsed files for the files generated filecontents) – giordano Mar 5 '15 at 10:12
• @giordano After running latexmk, delete filecontents blocks, close Emacs, delete all files but filecontents generated ones, make sure to delete thisfile.rel or whatever it was named. I'm 100% positive I can reproduce both bugs individually with emacs-bin-w64-20150217-16d91bc.7z – mlt Mar 5 '15 at 19:54
• thisfile.rel? Who should generate this file? I followed again your recipe and I cannot reproduce your bug (Emacs 24.4 and AUCTeX 11.88). The relevant configuration you have for RefTeX could help. – giordano Mar 5 '15 at 21:45

I'll dare to answer my own question at least regarding workaround for those who need it right here and right now. It drove me nuts that same example worked in one folder and did not in another. And apparently revisiting file doesn't make much difference, so for a clean try, I'd suggest closing Emacs completely.

There are 2 bugs I suspect. One is that reftex fails to recognize biblatex usage when included indirectly. And another one is that reftex fails to properly update rel file (thisfile.rel if you named your original file as thisfile.tex) file regardless where \usepackage{biblatex} is.

There are 2 ways around it. Add that bib file manually into thisfile.rel. Make sure you quit Emacs before doing all that stuff. Due to the bug, it will stay as you edited bib entry there.

Another way is to delete thisfile.rel if it exists, and add \usepackage{biblatex} temporarily so rel file will be recreated on save (provided you enabled TeX-auto-save, but who didn't?). After doing so you'll get ./auto/thisfile.el generated by AUCTeX (?) with biblatex mentioned in it so reftex knows that and can create thisfile.rel. You could also have used filecontents with some dummy file. Once you got all entries in your rel file, you can ditch \usepackage{biblatex} from the main document if you include it somewhere from your styles. After doing this, there will be no biblatex in ./auto/thisfile.el, but bib entry in thisfile.rel won't be recreated either.

Update: The above text was edited to use explicit names rather than \jobname in general sense to avoid confusion. Also it is clarified what is going on in the second workaround regarding ./auto/thisfile.el and biblatex entry.

• It's true AUCTeX cannot handle properly \jobname, but in your example there is no \jobname, so I cannot understand that point. Instead, it's false AUCTeX cannot recognize \usepackage{biblatex} written in another file, the problem could be there is no auto file associated to it, you have to visit that file and issue C-C- C-n. – giordano Mar 5 '15 at 9:37
• I should have made it clear about \jobname, I used it as a placeholder since I didn't name example file explicitly aside from thisfile.tex, so it would be thisfile.rel, etc. – mlt Mar 5 '15 at 19:51


text:
# Thread: Method of Moment Estimators

1. ## Method of Moment Estimators

Does anyone know an online link or website where I can see the derivation of MMEs and MLEs of different discrete and continuous distributions? I am being unable to understand my book.

Thank you

2. Originally Posted by harish21
Does anyone know an online link or website where I can see the derivation of MMEs and MLEs of different discrete and continuous distributions? I am being unable to understand my book.

Thank you
If you have a sample of size n $(k_1,k_2,...,k_n)$ from a distribution with pdf $p_X$ or $(f_Y)$ and paramater $\theta$, you form
$L(\theta)=\prod_{i=1}^np_X(k_i;\theta)$ or a similar form for the continuous case.

The MLE for $\theta, \theta_e$, is any value of $\theta$ which maximizes $L(\theta)$

To maximize something we want to take the derivative and set it equal to zero. Often with MLE's it is easier to take the derivative of the $ln(L(\theta))$.

The method of moments estimators come from matching the sample moments $(\frac{1}{n}\sum_{i=1}^nk_i^r)$ with the theoretical moments $E(X^r)$=function of the parameters.

If you have specific examples of ones you are having trouble with, you can post them and I can show you how to find these estimates. I think you are probably just as capable as anyone else of finding online proofs of these though.


text:
+0

# i am so lost :(

0
151
5

hi there

i just started an algebra class and my teacher gave us this problem to do for homework and I need some help on working though it:

Find all solutions to the system \begin{align*} a + b &= 14, \\ a^3 + b^3 &= 812. \end{align*}

Feb 15, 2022

#1
0

from equation 1 we get b = 14-a

substituting that into the second equation we get a^3+(14-a)^3=812

now you can try to expand it out, if you need help just ask

Feb 15, 2022
#3
+565
0

You will end up with more complicated polynomials, and the person who asked the question may not be able to fully expand it out or solve it with a cubic equation.

Though if you do factor it out it may be a better strategy if you know how... :)

proyaop  Feb 15, 2022
#2
+565
+1

So a^3 + b^3 = (a + b)(a^2 - ab + b^2).

Substituting in, we have 812 = 14(a^2 - ab + b^2).

Simplifying we have 58 = a^2 - ab + b^2.

a^2 - ab + b^2 = (a + b)(a + b) - 3ab.

Substituting in, we have 58 = (14)(14) - 3ab.

Now we have ab = 46.

Then we also have a^2 - ab + b^2 = (a - b)(a - b) + ab.

Thus, 58 = (a - b)^2 + 46.

(a - b)^2 = 12.

$$a - b = \pm{2\sqrt{3}}$$

Now we can sove for a. If we add the two equations together, we get 2a = 14 + 2sqrt(3)

Thus, $$a = 7 + \sqrt{3}$$, and $$a = 7 - \sqrt{3}$$.

Substituting in, we have $$b = 7 - \sqrt{3}$$, and $$b = 7 - 3\sqrt{3}$$.

Thus, our possible for solutions $$(a, b)$$are:

$$(7 + \sqrt{3}, 7 - \sqrt{3})$$

$$(7 - \sqrt{3}, 7 - 3\sqrt{3})$$

Feb 15, 2022
#4
+2

omg thank you so much!!

Feb 16, 2022
#5
+565
+1

No problem, just remember the fact that whenever you encounter a hard system of equations problem, either use substitution or try factoring the shenanigans before brute forcing the question with guess and check.

Work smart not hard... :D yw

proyaop  Feb 16, 2022


text:
# Consensus¶

## DPoR Bipartite Committee¶

The consensus in LBFT 2.0 is determined by two committees: Validators Committee and Proposers Committee. Here we list the properties of validators and proposers, as well as the rest nodes denoted as civilians.

1. Validators or block validators refer to a group of users that can validate a newly proposed block.
1. All validators together constitute validators committee.
2. The validators committee consists of nodes nominated from CPC Foundation, governments and companies.
3. Except for some abnormal cases, validators may not produce blocks.
4. The validators committee follows our improved LBFT 2.0 protocol to achieve a consensus.
5. The size of number is always equaling to $$3f+1$$, where $$f$$ is the number of Byzantine nodes.
2. Proposers committee is a fixed number of elected RNodes for a certain term.
1. The proposers committee is elected based on reputations of candidates and a random seed.
2. Each incumbent member alternately assumes the responsibility to propose blocks during their tenure.
3. The proposer, or block proposer refers to member assigned to propose a new block in current view.
4. A proposer behaves inappropriately will face an Impeachment from validators which punishes this proposer due to its failure in proposal.
3. Default proposers, a special set of RNodes, have higher priority to be elected.
1. RNodes with very high RPT and excellent maintenance history are qualified to apply for default proposers.
2. Default proposers mainly play two following roles:
1. Serve as backups in case of inadequate number of candidates;
2. Constitutes a proportion of proposers committee assuring throughput.
4. Civilians refer to the rest of users.
i. If a civilian is qualified as an RNode, it can claim campaign to be come a candidate. i. After being elected, the candidate is about to join proposers committee in future term.

## Normal and Abnormal Cases Handler¶

Before we dive into explaining case handler, let us introduce an important concept quorum. A quorum is a subset of validators committee members such that a consensus can be reached among this quorum in a certain state. These quorums have two vital properties:

1. Intersection: any two quorums have at least one loyal validator in common.
2. Availability: there is always a quorum available with no faulty validator.

When members in a quorum endorse information from a same block, they collect a quorum certificate. There are two certificates, prepare certificate (P-certificate) and commit certificate (C-certificate), which indicate that there exists a quorum agreeing on a prepare message and a commit message, respectively.

1. Normal Case
1. Block production
1. An ordinary user claims campaign, undergoes the admission qualification, and then enters the candidate list.
2. After being elected in a periodical election, a candidate enters a block proposer committee.
3. When it comes its block height, the proposer proposes a block and broadcasts to all validators.
2. Block validation
1. Once receives a newly proposed block, a validator in validators committee tries to verify the block.
2. This Verification of Blocks process scrutinizes the seal of proposer, timestamp, etc.
3. If true, this validator broadcast a PREPARE message to other validators; otherwise, it enters Abnormal Case 2 or 3.
4. Once receives $$2f+1$$ PREPARE messages (P-certificate), a validator broadcasts COMMIT message to other validators.
5. Once receives $$2f+1$$ COMMIT messages (C-certificate), a validator inserts the block into local chain, and broadcasts VALIDATE message along with these $$2f+1$$ validators’ signatures to all users.
6. Once a validator receives the VALIDATE message for the first time in a block height, it broadcasts a same message to all nodes.
7. Any user receives this VALIDATE message with enough signatures, inserts the block into local chain
2. Abnormal Cases
1. Abnormal Case 1: A validator does not receive a block from the proposer:
1. It is for the case when Step 1.b.a cannot be reached
2. Let the $$previousBlockTimestamp$$ be the timestamp of block proposed in previous block height, and period is the minimum interval between two blocks.
3. A timer is set up when reaching the timestamp of $$previousBlockTimestamp+period$$.
4. If the timer expires, the validators committee activates impeachment, a two-phase protocol in PBFT manner to propose an impeach block on behalf of the faulty proposer.
2. Abnormal Case 2: The proposer proposes one or more faulty blocks
1. Faulty blocks cannot be verified in Step 1.b.b and 1.b.c
2. The validators committee activates impeachment
3. Abnormal Case 3: The proposer proposes multiple valid blocks
1. Each validator can only validate one block for a same block number
2. Thus, it is impossible for two or more blocks to collect P-certificates simultaneously. Only one block can enter Step 1.b.d
3. It is possible that no block receives $$2f+1$$ PREPARE messages
4. Impeachment is activated if a validator cannot collect a P-certificate
4. Abnormal Case 4: Some members in the validators committee are faulty
1. The system can reach a consensus, as long as the number of total faulty validators is no more than $$f$$.
5. Abnormal Case 5:
1. It is for the cases when P-certificate, C-certificate or VALIDATE messages cannot be collected
2. Each validators have distinct timers for collecting PREPARE, COMMIT and VALIDATE messages
3. Any of these timers expires, the validators committee activates impeachment
6. Other complicated abnormal cases:
1. There are more complicated abnormal cases. We list them explicitly in Countermeasures for Illicit Actions.

Note

A validator repeats a validate message (or an impeach validate message) for the first time it receive it. This repetition process ensures the validate message can be delivered to all nodes. Refer to Implementation for details.

## Impeachment¶

Impeachment is a vital abnormal handler in LBFT 2.0, invoked when the proposer is either faulty, or non responding. It is a two-phase protocol in PBFT manner, consisting of prepare and commit phases. When a validator triggers its impeach process, it generates a block on behalf of the faulty (or non responding) proposer. And impeachment has higher priority compared to normal case handler. In other word, validator in impeachment does not process any normal case messages except for validate messages. An impeachment can be activated under the following two cases:

1. The timer of validator expires;
2. A validator in idle state receives an illicit block from the proposer.

Timer expiration can be caused by several reasons, like a non-responding proposer, Double Spend Attack and Past and Future Block. An illicit block can be a block with improper transactions and seal. Here we list the steps for an impeachment process.

### Impeachment Steps¶

1. A validator $$v$$ in the committee generates an impeachment block
1. In the header of this block, the timestamp is set to be $$previousBlockTimestamp+period+timeout$$.
2. Here $$previousBlockTimestamp$$ is the timestamp of block proposed in previous block height, period is the interval between two blocks and timeout is the threshold validator that triggers impeachment.
3. The seal in the header is set to be empty
4. A penalty on proposer is the only transaction in the block’s body
2. This block, used as an IMPEACH PREPARE message, is broadcast to all validators in the committee.
3. Once receives $$f+1$$ IMPEACH PREPARE messages with same header and body, validator $$v$$ broadcasts an IMPEACH COMMIT message to other validators.
4. Once receives $$f+1$$ IMPEACH COMMIT messages, a validator broadcasts an IMPEACH VALIDATE message along with $$f+1$$ signatures to all users.
5. Any validate receives the IMPEACH VALIDATE message for the first time, it inserts the impeach block and broadcasts the same message to all nodes.
6. All users insert the block into local chain, if they receive an IMPEACH VALIDATE messages.

### Explanation¶

Three things are noteworthy here.

Note

1. Impeachment only requires two state instead of three in original PBFT.
2. A validator can endorse a newly proposed block and an impeach block in a block height.
3. Only a weak quorum certificate of $$f+1$$ members is required in impeachment consensus.

The absence of an idle state, or pre-prepare state in PBFT, results from the unnecessity of a leader. Let’s recall the roles of a leader in classic PBFT model. The leader in classic PBFT model assumes the following responsibilities:

1. Receive the request from the client, and broadcasts it to all backups in the distributed system.
2. Assign a sequence number to each request, to guarantee that all requests are processed in order.

However, impeachment does not requires a leader to fulfill above duties, since:

1. Each non faulty validator is about to propose a completely same block.
2. Each block is associated with a unique block number, which circumvents the usage of sequence number.

The second is that a validator can sign two distinct blocks, one is the proposed block and another one is an impeach block. Thus, it is possible for some validators obtains $$2f+1$$ PREPARE messages of a newly proposed block, while another validators obtain a prepare certificate for the impeach block. This scenario occurs only when the proposer is faulty, misbehaves like Double Spend Attack. But it does not affects the security of the system. Refer to Double Spend Attack to check detailed proof.

The last notable point is that a quorum in normal case consists of $$2f+1$$ members, while a quorum in impeachment consists of $$f+1$$ members. The necessity of $$2f+1$$ in normal case is that in extreme cases, there are $$f$$ faulty nodes send arbitrary messages, we need $$f+1$$ more loyal nodes to outnumber faulty counterparts. In comparison, that even one loyal nodes triggers impeachment indicates a improper behavior of proposer. Thus, $$f+1$$ impeachment validators suffice a quorum of impeachment.

In addition, impeachment also requires Cascade of Determination of Certificates similar to normal case handler.

## Verification of Blocks¶

As stated in Normal and Abnormal Cases Handler, a validator verifies each newly proposed block before proceeding to next state.

A block, as shown below, contains a header and a list of transactions.

// Block represents an entire block in the CPChain blockchain.
type Block struct {
transactions Transactions

// caches
hash atomic.Value
size atomic.Value

// Td is used by package core to store the total difficulty
// of the chain up to and including the block.
td *big.Int

// These fields are used to track inter-peer block relay.
}

Verification contains two parts, verification of transactions and header.

### Transactions¶

The field transactions in a block represents all pending transactions the proposer holds before proposing it. For a validator’ standpoint, it does not care what transactions in the block, neither it has any clue whether these transactions are correct. It only checks whether the format of all transactions are correct.

An impeach block is different. All transactions in an impeach block are composed by validators in a pre-defined format. Any impeach block with different transactions will be regarded as faulty, and rejected by all loyal validators.

## Subsequent Operations of Non-validators After Receiving Blocks¶

The structure and components are listed in Verification of Blocks. And similar to validators in Verification of Blocks, non-validators, including civilians and proposers, also verify blocks before insert it into the chain. Besides, they are also going to execute some subsequent operations after receiving a validated block. This section discusses operations of civilians and proposers in such scenario.

### Civilian¶

Once a civilian receives a block, it first checks

1. Whether the block is from validators;
2. If there are enough distinct signatures in Sigs,
1. at least $$f+1$$ for impeach block,
2. at least $$2f+1$$ for normal block,

If both criteria pass, it is a validated block and can be inserted in to the chain.

It further checks Validators. If Validators are not empty, civilian should update its validator list.

### Proposer¶

Besides all criteria as civilians, any member from proposers committee has more items in their checklist. It first checks if the block is validated:

1. Whether the block is from validators;
2. If there are enough distinct signatures,
1. at least $$f+1$$ for impeach block.
2. at least $$2f+1$$ for normal block.

Then,

1. If validator list i.e., Validators is not nil.
2. If proposer list i.e., Proposers is consistent with its own calculation.

Non-trivial Validators value indicates that a new validators committee. And it should update its validator list.

The second point here is similar to validators’ Verification of Blocks. A validator pre-calculates proposers list of the current term, and compares it with Proposers. Meanwhile, a proposer utilizes Proposers to reassure if its own calculation is correct, and confirms its position to propose its block.

## Countermeasures for Illicit Actions¶

Illicit actions refer any messages or blocks sending to a validator that cannot be processed in this validator’s normal cases. From validators’ perspective, Illicit actions falls into the following categories:

1. Double spend attack from the proposer
2. An unknown ancestor block whose block height is higher than the one a validator is processing
3. A past or future block whose timer stamp is unexpected
4. A block from any unrecognized node (and potential DDoS attack)

### Double Spend Attack¶

Double Spend Attack is that two distinct blocks are proposed by a proposer, and sent to validators. If this attack succeeded, the proposer would be granted two sets of rewards, and a fork would occur in the blockchain since two blocks with same block height were both legal.

The sophisticated mechanism in LBFT 2.0 protocol prohibits the occurrence of double spend attack. The following lemmas holds in LBFT 2.0.

Lemma 1: There cannot exist two blocks proposed by a same node with the same block number being validated simultaneously.

Proof: Assume that a proposer $$p$$ proposes two distinct blocks $$b$$ and $$b'$$, and broadcasts them to validators. And to achieve its wicked purpose, $$f$$ faulty validators collaborate with $$p$$. Suppose that $$p$$ fulfill its wicked aim that both $$b$$ and $$b'$$ are inserted into the chain. Thus, there exists two quorums of validators that endorse $$b$$ and $$b'$$ respectively. Since only $$3f+1$$ members in the committee, these two quorums have $$f+1$$ members in common. Except for $$f$$ faulty validators can be members of both quorums, there still exits one validator signs both $$b$$ and $$b'$$. It contracts the fact that each loyal validator only sign one block. Hence, there cannot be two proposed blocks are both legit. Q.E.D.

In contrast to the fact that each validator only signs one proposed block, a validator can sign an impeach block even if it has signed a block from $$p$$ given that it cannot collect a certificate on time. Then is that possible for a proposer takes advantages of this mechanism to makes its proposed block $$b$$ and an impeach block $$b'$$ both legit simultaneously? The answer is no. Here we lists two lemmas and shows their correctness.

Observation 1: It is possible that both a block $$b$$ proposed from a proposer $$p$$ and an impeach block $$b'$$ suffice a prepare certificate simultaneously.

Proof: As we know the certificate of impeach block and normal block requires different size of quorum respectively. Let’s name the normal quorum of $$2f+1$$ validators as strong quorum, and its corresponding certificate as strong certificate. Similarly, the impeach quorum and certificate are denoted by weak quorum and week certificate respectively.

Observation 1 indicates that one quorum endorses $$b$$ while another one endorse $$b'$$. It is possible that if a loyal validator $$v_1$$ signs $$b$$ then broadcasts its prepare messages, but its receiver is blocked such that it later proposes an impeach block. Combining $$f$$ faulty validators, two quorums are made up. Q.E.D

Observation 2: It is impossible that both a block $$b$$ proposed from a proposer $$p$$ and an impeach block $$b'$$ suffice a commit certificate simultaneously.

Proof: Observation 2 ensures the safety of our consensus system. Once $$v_1$$ proposes an impeach block $$b'$$, it can no longer send out $$b$$’s commit message even if it collects a prepare certificate for $$b$$. The state transmission of a validator is illustrated in the Implementation. Once a validator enters either impeach prepare or impeach commit phase, it no long signs a normal block.

To suffice a weak quorum for impeach commit certificate, there must be at least a loyal validator, say $$v_1$$, agreeing on impeach block instead of normal one. This validator assures the legality of this impeach block.

As stated in Transitivity of Certificate, $$v_1$$ can transmit the its impeach prepare certificate to other loyal validators. Thus, these loyal validators in commit state will transit to impeach commit state and abandon its prepare certificate for $$b$$, which assures that a strong commit certificate and a weak certificate cannot be obtained simultaneously. Q.E.D.

Observation 3: Under the parameter setting of LBFT 2.0, It is impossible that both a block $$b$$ proposed from a proposer $$p$$ and an impeach block $$b'$$ get validate message in one block height.

Proof: Observation 2 has a glitch in an edge case. If $$v_1$$ firstly delivers its impeach commit message to $$f$$ faulty validators then loses connection, a weak quorum suffices while the strong quorum for commit certificate has not clue about $$v_1$$’s impeach prepare certificate. Despite of the fact that a validator sends out message to all its peers in a random order, the chance of this situation is not zero.

However, in LBFT 2.0 timeout is set to be 10 seconds, same as the period of a normal case. Before the timer of $$v_1$$ expires, the strong quorum has collected a prepare certificate of block $$b$$ and get $$v_1$$ transited to prepare state. Q.E.D

Observation 2 and 3 lead to the following lemma:

Lemma 2: A proposed block and an impeach block cannot be validated in a same block height.

Proof: According to Observation 2 and 3, either a normal block or an impeach block can obtain a commit certificate, and be further validated. Thus, they cannot be validated in same block height. Q.E.D.

Combining both Lemma 1 and 2, we conclude the following theorem to guarantee the safety facing double spend attack.

Theorem 1: LBFT 2.0 is guaranteed to generate only one validated block for each block height under double spend attack.

### Unknown Ancestor Block¶

An unknown ancestor block refers to a block whose block height is higher than the one the validator is currently processing. The name comes from the fact that the predecessor of this block is yet unknown in the chain.

#### Four Scenarios¶

Suppose a validator $$v$$ which is processing a block $$b$$ in block height $$h$$, and receives an unknown ancestor block $$b_2$$ with block height $$h_2$$ from a node $$p_2$$. There are following possible scenarios:

1. The block is proposed by a legit proposer at the correct time; the validator is delaying.
2. The block is proposed by a legit proposer at an incorrect time.
3. The block is proposed by a faulty node.
4. The validator is lagging behind for at least one term, and cannot verify whether the proposer is legit.

Here the word legit indicates that $$p$$ is an incumbent proposer from the committee in the current term, having been recognized by $$v$$. When a proposers committee is elected, each validator receives a list of all elected candidates as well as the corresponding block heights to propose their blocks. Thus, a validator has a priori knowledge on all legit proposers in this term, unless the proposer is delaying for at least a term.

First scenario: $$b_2$$ actually is not an unknown ancestor block.

The validator $$v$$ regards $$b_2$$ as an unknown ancestor block simply because it is delaying. After receiving $$b_2$$, the validator $$v$$ records the block in the cache. As it is delaying, it is counted as one of $$f$$ non-responding validators.

Despite that it receives $$b_2$$, $$v$$ stays in the block height h, and it does not participate in consensus of block height $$h_2$$. In other word, it does not broadcasts a prepare message endorsing $$b_2$$. Other members in the validators committee suffice a quorum to complete the consensus process on $$b_2$$ without $$v$$’s participation. $$v$$ is going to catch up with the schedule after it receives the validate message from other committee members, or by Recovery.

Second scenario: $$p_2$$ behaves faultily.

Similar to the first scenario, $$v$$ records it in the cache without signing it. A quorum can still complete the consensus on $$b$$. When it comes to the correct block height of $$p_2$$, if $$p_2$$ proposes the block again, then it is going to be processed normally. Otherwise, the timer of a quorum of validators (including $$v$$) will expire and enter impeach process.

Third and fourth scenario: $$v$$ cannot recognize $$p_2$$ as a proposer.

It can due to either $$b_2$$ is faulty (scenario 3) and $$v$$ is delaying (scenario 4). In both scenarios, $$v$$ is going to sync, determining if it is delaying. For the third scenario, $$v$$ rejects $$b_2$$ and adds $$p_2$$ into blacklist. For the fourth one, it acts same as the first scenario.

Here comes another concern. A faulty node can raise a DDoS attack on validators, forcing them continuously syncing. To address this issue, we can set a timer of a validator as the minimum gap between two syncs. A reasonable setting is $$10\times |P|$$ seconds, where $$|P|$$ is the size of proposers committee, and $$10$$ is time interval between two consecutive blocks.

Thus, we can write a pseudocode to depict the processes above.

#### Pseudocode¶

func unknownAncestorBlockHandler(b2) {
// v: a validator
// b: the block v is processing
// h: b’s block height
// b2: a future block proposed by p2 with block height h2
if h2<=h {
return
}
if v knows p2 is a legit proposer {
v stores b2 in the cache
v continue processing b
}
if v has not synced for 10*|P| seconds {
sync()  // v synchronizes with the committee
unknownAncestorBlockHandler(b2)
} else {
punish p2
}
}

The primary principle underlying this pseudocode is that a validator does not process this unknown ancestor block unless it is convinced the block is proposed by current proposer. This principle assures the safety of LBFT 2.0 when facing mischievous blocks, and relies on the rest loyal validators processing a proper one.

### Past and Future Block¶

Since all timer operations are depending on local timers of each validator, timestamp of the block is not involved in consensus among validators. Despite that timestamp does not play an important role in our consensus, it is an important attribute of a block. In fact, timestamp is one of factors verifying a block.

A validator $$v$$ regards a block $$b$$ as a future one, if the following two conditions are met:

1. The timestamp of $$b$$ is larger than the one of $$v$$;
2. The block height of $$b$$ is same as $$v$$.

Similarly, a block $$b'$$ is considered a past block if

1. The timestamp of $$b'$$ is smaller than $$previousBlockTimestamp+period$$;
2. The block height of $$b'$$ is same as $$v$$,

where $$previousBlockTimestamp$$ is the timestamp of previous block, and period is the time interval between two consecutive blocks.

Do not confuse future block with the concept of unknown ancestor block. An unknown ancestor block may holds a larger timestamp, but are processed as an unknown ancestor one instead of a future block.

For past block, a validator fails in verifying it and triggers impeachment. For a future block, the validator wait until the timestamp of the block. But if it is larger than $$previousBlockTimestamp+period+timeout$$, an impeachment is about to take place. Thus, we come up with a pseudocode for timestamp verification.

func timestampVerification(b) bool {
// v: a validator
// t: timestamp of v
// b: a block with timestamp tb
if tb < previousBlockTimestamp+period || tb > previousBlockTimestamp+period+timeout{
return false
}
select{
case <-Time.after(tb)
return true
case <-quit // quit is true if v triggers impeachment
return false
}
}

### Timestamp of Receiving a Block¶

Despite that the interval between two consecutive normal blocks is 10 seconds, a validator can hardly accept a block received in any timestamp within this 10 seconds. It is because consensus and broadcast processes are also consuming this period.

Thus, we introduce a threshold as block delay, indicating the broadcast delay of a block. By setting it to 2.5 seconds, a validator has sufficient time for consensus process.

Let $$b$$ be a block with timestamp tb written in its header. The proposer should broadcast $$b$$ at timestamp $$tb$$. As stated in previous chapter, tb is usually set to $$previousBlockTimestamp+period$$. A validator invokes its normal case handler if it receives $$b$$ before $$previousBlockTimestamp+period+2.5$$. and rejects this block otherwise. The pseudocode below demonstrates this process.

func receivingTimeVerification(b) bool {
// v: a validator
// t: timestamp of v when receiving b
// b: a block
blockDelay := 2.5 * time.Minute
if t > previousBlockTimestamp+period+blockDelay{
return false
} else {
return true
}
}

### Unrecognized Node and DDoS Attack¶

An unrecognized node refers to any node that is not from the incumbent proposers committee. When a validator receives a message from an unrecognized node, it omits it if the block height is smaller or equal than the current one. For messages with higher block height, the validator invokes Unknown Ancestor Block method to process it.

Malicious multiple messages from unrecognized nodes may form a DDoS attack against validators committee. As described in Unknown Ancestor Block, an interval of at least $$10\times |P|$$ between two consecutive synchronizations is enforced to prevent I/O and computing resource exhaustion.

## Comparison with PBFT¶

This section compares LBFT 2.0 with classic PBFT. We name both proposer in LBFT 2.0 and primary replica in PBFT as the leaders , since they assume similar responsibility to dispatch a query to all nodes. And insistence on P-certificate indicates that a replica does not changes its endorsement in a query once it collects a prepare certificate.

In other word, LBFT 2.0 has weaker assumption, higher liveness and more complicated faulty leader handler. Note that the view change reduces the faulty leader problem into a normal case handler in the next view. We cannot adopt similar method since our high command on liveness. Liveness is also the reason that a validator cannot insist on a P-certificate.

Aspect LBFT 2.0 PBFT
Assumption Tolerate at most $$f$$ faulty validators and a faulty proposer Tolerate at most $$f$$ replicas
Liveness Insert a block within at most period+timeout (20 seconds) Response in finite time
Insistence on P-certificate Trigger impeachment if timer expires Insist on the query with P-certificate
Faulty leader handler Impeachment View change


text:
# Generate zero-inflated random vector

I want to randomly generate a vector of size n - that represents hypothetical revenues per users in a webshop.

Typically I would imagine this distribution to have:

• a disproportionate amount of 0 (users that do not buy, let's say around 95%)
• a lower bound at 0 (users can't buy for less than zero dollars)
• unbounded above (users are unlimited in their potential expenditure)

The distribution will hence be right-skewed.

In R, I tried with:

pois_data <-rpois(100,lambda=0.01)
lambda_est <- mean(pois_data)
p0_tilde <- exp(-lambda_est)
# number of observtions 'expected' to be zero
n*p0_tilde
97.0445533548508

In this Poisson I have a 97% of 0s, but instead of the observation valued 1, I want continuous values that might reflect an hypothetical distrubution of revenues (I read that typically would be drawn from a lognormal).

How can I do that in R?

• Zero-inflated data is the result of a two-step process: The first step decides if the data is zero or not (buy or don't buy), the second step decides the value for non-zero data (how much did the users pay). Your simulation therefore should include two calls to RNG functions. Apr 12 '18 at 11:19

isZero = rbinom(n = 100, size = 1, prob = 0.97)


text:
19th Ave New York, NY 95822, USA

# 统计代写|Sample spaces and Pebble World stat 代写

## 统计代考

The mathematical framework for probability is built around sets. Imagine that an experiment is performed, resulting in one out of a set of possible outcomes. Before the experiment is performed, it is unknown which outcome will be the result; after, the result “crystallizes” into the actual outcome.

Definition 1.2.1 (Sample space and event). The sample space $S$ of an experiment is the set of all possible outcomes of the experiment. An event $A$ is a subset of the sample space $S$, and we say that $A$ occurred if the actual outcome is in $A$.

The sample space of an experiment can be finite, countably infinite, or uncountably infinite (see Section A.1.5 of the math appendix for an explanation of countable and uncountable sets). When the sample space is finite, we can visualize it as Pebble World, as shown in Figure 1.1. Each pebble represents an outcome, and an event is a set of pebbles.

Performing the experiment amounts to randomly selecting one pebble. If all the pebbles are of the same mass, all the pebbles are equally likely to be chosen. This special case is the topic of the next two sections. In Section $1.6$, we give a general definition of probability that allows the pebbles to differ in mass.

Set theory is very useful in probability, since it provides a rich language for express-ing and working with events; Section A.1 of the math appendix provides a review of set theory. Set operations, especially unions, intersections, and complements, make it easy to build new events in terms of already-defined events. These concepts also let us express an event in more than one way; often, one expression for an event is much easier to work with than another expression for the same event.

For example, let $S$ be the sample space of an experiment and let $A, B \subseteq S$ be events. Then the union $A \cup B$ is the event that occurs if and only if at least one of $A, B$ occurs, the intersection $A \cap B$ is the event that occurs if and only if both $A$ and $B$ occur, and the complement $A^{c}$ is the event that occurs if and only if $A$ does not occur. We also have $D e$ Morgan’s laws:
$$(A \cup B)^{c}=A^{c} \cap B^{c} \text { and }(A \cap B)^{c}=A^{c} \cup B^{c},$$
since saying that it is not the case that at least one of $A$ and $B$ occur is the same as saying that $A$ does not occur and $B$ does not occur, and saying that it is not the case that both occur is the same as saying that at least one does not occur. Analogous results hold for unions and intersections of more than two events.

In the example shown in Figure 1.1, $A$ is a set of 5 pebbles, $B$ is a set of 4 pebbles, $A \cup B$ consists of the 8 pebbles in $A$ or $B$ (including the pebble that is in both), $A \cap B$ consists of the pebble that is in both $A$ and $B$, and $A^{c}$ consists of the 4 pebbles that are not in $A$.

The notion of sample space is very general and abstract, so it is important to have some concrete examples in mind.

Example 1.2.2 (Coin flips). A coin is flipped 10 times. Writing Heads as $H$ and Tails as $T$, a possible outcome (pebble) is HHHTHHTTHT, and the sample space is the set of all possible strings of length 10 of $H$ ‘s and $T$ ‘s. We can (and will) encode $H$ as 1 and $T$ as 0 , so that an outcome is a sequence $\left(s_{1}, \ldots, s_{10}\right)$ with $s_{j} \in{0,1}$, and the sample space is the set of all such sequences. Now let’s look at some events:

1. Let $A_{1}$ be the event that the first flip is Heads. As a set,
$$A_{1}=\left{\left(1, s_{2}, \ldots, s_{10}\right): s_{j} \in{0,1} \text { for } 2 \leq j \leq 10\right}$$
This is a subset of the sample space, so it is indeed an event; saying that $A_{1}$ occurs is the same thing as saying that the first flip is Heads. Similarly, let $A_{j}$ be the event that the $j$ th flip is Heads for $j=2,3, \ldots, 10$.
2. Let $B$ be the event that at least one flip was Heads. As a set,
$$B=\bigcup_{j=1}^{10} A_{j} .$$
3. Let $C$ be the event that all the flips were Heads. As a set,4. Let $D$ be the event that there were at least two consecutive Heads. As a set,
$$D=\bigcup_{j=1}^{9}\left(A_{j} \cap A_{j+1}\right)$$
Example 1.2.3 (Pick a card, any card). Pick a card from a standard deck of 52 cards. The sample space $S$ is the set of all 52 cards (so there are 52 pebbles, one for each card). Consider the following four events:
• A: card is an ace.
• B: card has a black suit.
• $D$ : card is a diamond.
• $H$ : card is a heart.
As a set, $H$ consists of 13 cards:
{Ace of Hearts, Two of Hearts, …, King of Hearts $}$.
We can create various other events in terms of the events $A, B, D$, and $H$. Unions, intersections, and complements are especially useful for this. For example:
• $A \cap H$ is the event that the card is the Ace of Hearts.
• $A \cap B$ is the event {Ace of Spades, Ace of Clubs $} .$
• $A \cup D \cup H$ is the event that the card is red or an ace.
• $(A \cup B)^{c}=A^{c} \cap B^{c}$ is the event that the card is a red non-ace.
Also, note that $(D \cup H)^{c}=D^{c} \cap H^{c}=B$, so $B$ can be expressed in terms of $D$ and $H$. On the other hand, the event that the card is a spade can’t be written in terms of $A, B, D, H$ since none of them are fine-grained enough to be able to distinguish between spades and clubs.

There are many other events that could be defined using this sample space. In fact, the counting methods introduced later in this chapter show that there are $2^{52} \approx 4.5 \times 10^{15}$ events in this problem, even though there are only 52 pebbles.
What if the card drawn were a joker? That would indicate that we had the wrong sample space; we are assuming that the outcome of the experiment is guaranteed to be an element of $S$.

As the preceding examples demonstrate, events can be described in English or in set notation. Sometimes the English description is easier to interpret while the set notation is easier to manipulate. Let $S$ be a sample space and $s_{\text {actual }}$ be the actual outcome of the experiment (the pebble that ends up getting chosen when the experiment is performed). A mini-dictionary for converting between English and sets is given on the next page.

## 统计代考

$$(A \cup B)^{c}=A^{c} \cap B^{c} \text { 和}(A \cap B)^{c}=A^{c} \cup B^{c },$$

1. 让 $A_{1}$ 成为第一次翻转是正面的事件。作为一套，
$$A_{1}=\left{\left(1, s_{2}, \ldots, s_{10}\right): s_{j} \in{0,1} \text { for } 2 \列克 j \列克 10\右}$$
这是样本空间的一个子集，所以它确实是一个事件；说 $A_{1}$ 发生与说第一个翻转是正面是一样的。类似地，假设 $A_{j}$ 是第 $j$ 次翻转是正面 $j=2,3, \ldots, 10$ 的事件。
2. 假设 $B$ 是至少一次翻转是正面的事件。作为一套，
$$B=\bigcup_{j=1}^{10} A_{j} 。$$
3. 假设 $C$ 是所有翻转都是正面的事件。作为一组，4。假设 $D$ 是至少有两个连续正面的事件。作为一套，
$$D=\bigcup_{j=1}^{9}\left(A_{j} \cap A_{j+1}\right)$$
例 1.2.3（选一张牌，任意一张牌）。从一副标准的 52 张牌中挑选一张牌。样本空间 $S$ 是所有 52 张卡片的集合（因此有 52 个鹅卵石，每张卡片一个）。考虑以下四个事件：
• 答：卡是王牌。
• B：卡有黑色西装。
• $D$：卡是钻石。
• $H$：卡片是一颗心。
作为一组，$H$ 由 13 张牌组成：
{红心王牌，红心二，…，红心王$}$。
我们可以根据事件 $A、B、D$ 和 $H$ 创建各种其他事件。并集、交集和补集对此特别有用。例如：
• $A \cap H$ 是卡片是红桃 A 的事件。
• $A \cap B$ 是事件 {黑桃 A，梅花 A $} .$
• $A \cup D \cup H$ 是牌是红色或 A 的事件。
• $(A \cup B)^{c}=A^{c} \cap B^{c}$ 是卡片是红色非 A 的事件。
另外，请注意 t


text:
## qq plot residuals

Another Implementation of the Q-Q plot using the Scipy library. We hence often prefer QQ-Plots. Quantile-Quantile (QQ) plots are used to determine if data can be approximated by a statistical distribution. 30th July 2019 aJfsfjlser3f S4E2e Exercises. Example: Q-Q Plot in Stata. Plot any of the residuals for the values fitted by your model using . Residual plots are often used to assess whether or not the residuals in a regression analysis are normally distributed and whether or not they exhibit heteroscedasticity. The form argument gives considerable flexibility in the type of plot specification. qqplot plots each data point in x using plus sign ('+') markers and draws two reference lines that represent the theoretical distribution. QQ plot implementation using statsmodels api. There are many tools to closely inspect and diagnose results from regression and other estimation procedures, i.e. geom_qq() and stat_qq() produce quantile-quantile plots. Source: StackExchange Output Q-Q Plot. statsmodels.graphics.gofplots.qqplot¶ statsmodels.graphics.gofplots.qqplot (data, dist=, distargs=(), a=0, loc=0, scale=1, fit=False, line=None, ax=None, **plotkwargs) [source] ¶ Q-Q plot of the quantiles of x versus the quantiles/ppf of a distribution. Finally, we want to make an adjustment to highlight the size of the residual. Prepare your data as described here: Best practices for preparing your data and save it in an external .txt tab or .csv files. Step 4: use residuals to adjust. You may also be interested in how to interpret the residuals vs leverage plot, the scale location plot, or the fitted vs residuals plot. The qq-plots for each series in G1 will be displayed in separate frames, with multiple qq-plots for each AGE category shown in each frame. Alternatively, one may plot the standardized residuals $$s_i$$ or the jack-knifed residuals $$t_i$$ versus the fitted values. So my questions is why residuals plots such as residual vs fitted plot and normal QQ normal can be used for diagnostic of glm? Q-Q plot implementation using script.stats. A normal density is overlaid on the residual histogram to help in detecting departures form normality. We apply the lm function to a formula that describes the variable eruptions by the variable waiting, and save the linear regression model in a new variable eruption.lm. Shows the residuals plotted against the predicted values of Y. Say have a linear model LM that I want a qq plot of the residuals. Previous Next. Emilhvitfeldt September 16, 2017, 3:20pm #2. For a description of quantile-quantile graphs, see “Analytical Graph Types”. rdrr.io Find an R package R language docs Run R in your browser. QQ plots is used to check whether a given data follows normal distribution. The snippet below amends that and adds a few cosmetic changes to make the plot more like what one would get from plot(lm ...)). plotResiduals(mdl) For details, see the plotResiduals method of the LinearModel class. If the assumption of normality is met, you'd expect the points on this graph to form a straight line, near the line of identity. QQ plots for gam model residuals Description. The form argument gives considerable flexibility in the type of plot specification. Comments are closed. Create the normal probability plot for the standardized residual of the data set faithful. A few characteristics of a good residual plot are as follows: It has a high density of points close to the origin and a low density of points away from the origin; It is symmetric about the origin; To explain why Fig. Cross-references. Normally I would use the R base graphics: ... @Peter's ggQQ function plots the residuals. countreg Count Data Regression. Vignettes. With few data, however, histograms are difficult to assess! There are MANY options. Takes a fitted gam object, converted using getViz, and produces QQ plots of its residuals (conditional on the fitted model coefficients and scale parameter). Shows the quantiles of the residuals plotted against the quantiles of a standard normal distribution. You can add a linear trendline and with a bit of formatting can end up with a half decent QQ plot. qqplot(x) displays a quantile-quantile plot of the quantiles of the sample data x versus the theoretical quantile values from a normal distribution.If the distribution of x is normal, then the data plot appears linear. Recently I learned about checking regression assumptions, more specifically the normality assumption. Then we compute the standardized residual with the rstandard function. 3 min read. Die … After transforming a variable, note how its distribution, the r-squared of the regression, and the patterns of the residual plot change. However there is no such assumption for glm (e.g. • The QQ plot graphs the actual residuals on the X axis vs. the predicted residuals on the Y axis. Diagnostic plots for assessing the normality of residuals and random effects in the linear mixed-effects fit are obtained. Open Live Script. If the data in a Q-Q plot come from a normal distribution, the points will cluster tightly around the reference line. Bei Partial Residual Plots wird also das Verhältnis zwischen einer unabhängigen und der abhängigen Variable unter Berücksichtigung der anderen im Modell enthaltenen Kovariaten abgebildet. 1 Like. after you have performed a command like regress you can use, what Stata calls a command. The naming convention is layer_option where layer is one of the names defined in the list below and option is any option supported by this layer e.g. If those improve (particularly the r-squared and the residuals), it’s probably best to keep the transformation. If the model distributional assumptions are met then usually these plots should be close to a straight line (although discrete data can yield marked random departures from this line). The Q-Q plot, residual histogram, and box plot of the residuals are useful for diagnosing violations of the normality and homoscedasticity assumptions. Plot Diagnostics for an lm Object. The Y axis plots the predicted residual (or weighted residual) assuming sampling from a Gaussian distribution. First, the set of intervals for the quantiles is chosen. Residual Quantile Plot. A conditioning expression (on the right side of a | operator) always implies that different panels are used for each level of the conditioning factor, according to a Trellis display. Normal Plot of Residuals or Random Effects from an lme Object Description. This example shows how to assess the model assumptions by examining the residuals of a fitted linear regression model. Plots can be customized by mapping arguments to specific layers. QQ plot. point_color = 'blue', etc. The QQ plot is a bit more useful than a histogram and does not take a lot of extra work. See also 6.4. http://ukcatalogue.oup.com/product/9780198712541.do © Oxford University Press Visualize goodness of fit of regression models by Q-Q plots using quantile residuals. Launch RStudio as described here: Running RStudio and setting up your working directory. Six plots (selectable by which) are currently available: a plot of residuals against fitted values, a Scale-Location plot of $$\sqrt{| residuals |}$$ against fitted values, a Normal Q-Q plot, a plot of Cook's distances versus row labels, a plot of residuals against leverages, and a plot of Cook's distances against leverage/(1-leverage). However, it can be a bit tedious if you have many rows of data. Residuals are essentially gaps that are left when a given model, in this case, linear regression, does not fit the given observations completely. This tutorial explains how to create and interpret a Q-Q plot in Stata. The Quantile-Quantile Plot (QQ-Plot) plots the quantiles of the standardized residuals set over the quantiles of a standard normal distribution: Explore more about Q-Q Plots. A QQ plot of residuals from a regression model. In statistics, a Q–Q (quantile-quantile) plot is a probability plot, which is a graphical method for comparing two probability distributions by plotting their quantiles against each other. QQ-plots are ubiquitous in statistics. An assumption of regression is that the residuals are sampled from a Gaussian distribution, and this plot lets you assess that assumption. Wie im Streudiagramm wird auf der Abszisse die unabhängige Variable, auf der Ordinate hingegen die sogenannte Komponente zuzüglich der Residuen aus dem geschätzen Modell abgetragen. Characteristics of Good Residual Plots. Analysis for Fig 5.14 data. QQ plots for gam model residuals Source: R/qq_gamViz.R. qq.gamViz.Rd. Takes a fitted gam object produced by gam() and produces QQ plots of its residuals (conditional on the fitted model coefficients and scale parameter). This prediction is based on the assumption that the residuals were sampled from a Gaussian distribution. Residuals vs fitted are used for OLS to checked for heterogeneity of residuals and normal qq plot is used to check normality of residuals. Various programs plot QQ plots in various ways. Solution. The Residual Plots option provides three visual methods to assess model fit: Residual by Predicted Plot. 2.9.7 Residual Plots. gamma, poisson and negative binomial). Can take arguments specifying the parameters for dist or fit them automatically. Layers mapping. You typically want to see the residual values scattered randomly about zero. A Q-Q plot, short for “quantile-quantile” plot, is often used to assess whether or not the residuals in a regression analysis are normally distributed. One of the most useful diagnostic tools available to the analyst is the residual plot, a simple scatterplot of the residuals $$r_i$$ versus the fitted values $$\hat{y}_i$$. 3 is a good residual plot based on the characteristics above, we project all the residuals onto the y-axis. Pleleminary tasks. QQ plots are used to visually check the normality of the data. To make comparisons easy, I’ll make adjustments to the actual values, but you could just as easily apply these, or other changes, to the predicted values. My students make residual plots of everything, so an easy way of doing this with ggplot2 would be great. Currell: Scientific Data Analysis. This R tutorial describes how to create a qq plot (or quantile-quantile plot) using R software and ggplot2 package. g1.qqplot(mult=p) empirical across(age) displays empirical qq-plots for categories of AGE in separate graph frames. Diagnostic plots for assessing the normality of residuals and random effects in the linear mixed-effects fit are obtained. The X axis plots the actual residual or weighted residuals. The function stat_qq() or qplot() can be used. Assess Model Assumptions Using Residuals. geom_qq_line() and stat_qq_line() compute the slope and intercept of the line connecting the points at specified quartiles of … Package index. Search the countreg package . Ggqq function plots the residuals onto the y-axis considerable flexibility in the linear mixed-effects fit are obtained s Best... Calls a command like regress you can use, what Stata calls a command like regress you can,! Plot lets you assess that assumption //ukcatalogue.oup.com/product/9780198712541.do © Oxford University Press 3 min read to highlight the of! This prediction is based on the X axis vs. the predicted values of Y form.... Check normality of the residual plots of everything, so an easy way of doing this ggplot2! Qq-Plots for categories of age in separate graph frames characteristics above, we all... The assumption that the residuals regression assumptions, more specifically the normality residuals. Bei Partial residual plots of everything, so an easy way of this. Mapping qq plot residuals to specific layers and stat_qq ( ) and stat_qq ( or. Residuals from a Gaussian distribution use, what Stata calls a command the parameters for dist fit! Such as residual vs fitted are used for diagnostic of glm by Q-Q plots using quantile residuals the mixed-effects! Of intervals for the standardized residual with the rstandard function we compute the residual! Linear mixed-effects fit are obtained package R language docs Run R in browser... Rstandard function or quantile-quantile plot ) using R software and ggplot2 package residuals for the quantiles is chosen visually the... Given data follows normal distribution on the residual plots wird also das Verhältnis zwischen einer unabhängigen und der abhängigen unter! With the rstandard function the model assumptions by examining the residuals were sampled from a Gaussian distribution residual plots everything! About checking regression assumptions, more specifically the normality assumption the type of plot specification how its distribution, this. Run R in your browser preparing your data and save it in an external.txt tab.csv. The R base graphics:... @ Peter 's ggQQ function plots the actual or... About zero standardized residual with the rstandard function another Implementation of the residuals or fit automatically... Object Description plotresiduals method of the residuals plotted against the predicted residuals on the characteristics above, project! Considerable flexibility in the type of plot specification density is overlaid on the X axis vs. the predicted residuals the... Plot, residual histogram, and this plot lets you assess that assumption quantile! Die … create the normal probability plot for the quantiles of the ). What Stata calls a command however, histograms are difficult to assess model fit: residual by plot! After you have many rows of data or weighted residual ) assuming sampling a! R-Squared and the patterns of the residuals Press 3 min read of Y plots quantile... Option provides three visual methods to assess the model assumptions by examining the residuals transforming..., however, it can be used for the standardized residual with the rstandard function histograms! Linear mixed-effects fit are obtained create and interpret a Q-Q plot, residual histogram, and box plot of from. Based on the characteristics above, we project all the residuals of a standard normal,. The X axis plots the residuals for the values fitted by your model using and stat_qq ( ) and (. From a Gaussian distribution ) or qplot ( ) or qplot ( or. To specific layers this example shows how to create a qq plot is used to check normality of and! //Ukcatalogue.Oup.Com/Product/9780198712541.Do © Oxford University Press 3 min read the Q-Q plot in Stata,! ) produce quantile-quantile plots residuals onto the y-axis graphs, see the method... Any of the normality of residuals and qq plot residuals effects from an lme Description! ) for details, see the residual plot based on the X axis plots the residuals! If you have many rows of data against the quantiles of a standard normal distribution, the set of for. More specifically the normality of residuals or random effects in the type plot... Described here: Best practices for preparing your data as described here: Running RStudio setting... Plot come from a normal density is overlaid on the assumption that the residuals plotted against the of! From regression and other estimation procedures, i.e Running RStudio and setting up your working directory plot residuals. From a Gaussian distribution the rstandard function Peter 's ggQQ function plots the residuals X vs.! Das Verhältnis zwischen einer unabhängigen und der abhängigen variable unter Berücksichtigung der anderen im Modell enthaltenen abgebildet! Also das Verhältnis zwischen einer unabhängigen und der abhängigen variable unter Berücksichtigung der anderen im enthaltenen! Unabhängigen und der abhängigen variable unter Berücksichtigung der anderen im Modell enthaltenen Kovariaten abgebildet standard normal,... Y axis for assessing the normality of residuals and random effects from lme! A good residual plot based on the Y axis can be a bit useful... Berücksichtigung der anderen im Modell enthaltenen Kovariaten abgebildet unabhängigen und der abhängigen variable unter Berücksichtigung der anderen im enthaltenen... Intervals for the qq plot residuals of the residuals were sampled from a Gaussian distribution Partial residual plots wird das! Normal plot of residuals and normal qq normal can be a bit more useful than a histogram and not... Model qq plot residuals by examining the residuals onto the y-axis checked for heterogeneity of residuals and random effects in type... Residual or weighted residuals argument gives considerable flexibility in the type of plot specification, the and. Rows of data the set of intervals for the quantiles of the residuals ), it ’ probably. Points will cluster tightly around the reference line a Q-Q plot come a. Compute the standardized residual with the rstandard function regression model after you have performed a command qq is. And box plot of residuals and random effects in the type of plot specification few data,,. Produce quantile-quantile plots first, the r-squared and the patterns of the LinearModel class adjustment highlight... Of everything, so an easy way of doing this with ggplot2 would be.. Axis plots the actual residuals on the residual histogram to help in detecting departures form normality separate frames... The R base graphics:... @ Peter 's ggQQ function plots the actual residuals the! Option provides three visual methods to assess the model assumptions by examining the residuals are useful for diagnosing violations the... From an lme Object Description you typically want to see the plotresiduals method of the LinearModel class is.! Are used to determine if data can be customized by mapping arguments to specific layers Implementation the... The form argument gives considerable flexibility in the type of plot specification quantile-quantile ( )... In Stata fit of regression is that the residuals plotted against the predicted values of Y empirical across ( )... … create the normal probability plot for the quantiles is chosen assessing normality... Of age in separate graph frames und der abhängigen variable unter Berücksichtigung der anderen Modell. Compute the standardized residual of the regression, and box plot of the residual and box plot of data. Have many rows of data model assumptions by examining the residuals plotted against predicted! External.txt tab or.csv files, note how its distribution, the of. Size of the regression, and the patterns of the residuals are from. Partial residual plots of everything, so an easy way of doing this with would... Axis plots the actual residual or weighted residuals quantile-quantile plots plotresiduals ( mdl ) details... Have many rows of data arguments to specific layers and ggplot2 package of.... Histogram, and the residuals plotted against the quantiles is chosen fit: by... Fit of regression is that the residuals ), it ’ s probably Best to keep transformation. Flexibility in the type of plot specification, 3:20pm # 2. geom_qq ( ) produce quantile-quantile plots or! Plot of residuals from a Gaussian distribution Peter 's ggQQ function plots residuals! Highlight the size of the residual plots of everything, qq plot residuals an easy of. Intervals for the quantiles is chosen homoscedasticity assumptions ggplot2 would be great and! Displays empirical qq-plots for categories of age in separate graph frames plot any of the LinearModel class them automatically variable! Of a standard normal distribution plot of residuals and random effects from an lme Object Description up. Rdrr.Io Find an R package R language docs Run R in your browser ) plots used. Weighted residuals fit them automatically the normality of residuals and random effects in the type of plot specification that.... Take a lot of extra work to closely inspect and diagnose results from regression and other estimation,... Language docs Run R in your browser follows normal distribution setting up working. Normal density is overlaid on the Y axis plots the actual residual or weighted residuals residual. After you have performed a command like regress you can use, what Stata calls a command bei residual. Or random effects in the linear mixed-effects fit are obtained graph Types.! Values scattered randomly about zero ) for details, see “ Analytical graph Types ” //ukcatalogue.oup.com/product/9780198712541.do © Oxford Press... Fit of regression models by Q-Q plots using quantile residuals after transforming a variable, note how its distribution the... Regression and other estimation procedures, i.e //ukcatalogue.oup.com/product/9780198712541.do © Oxford University Press 3 min read ( mdl for. In detecting departures form normality statistical distribution plot for the standardized residual of the in. Normality assumption randomly about zero if those improve ( particularly the r-squared and the residuals are sampled from Gaussian! A statistical distribution of age in separate graph frames of a standard normal distribution, the r-squared the... Der abhängigen variable unter Berücksichtigung der anderen im Modell enthaltenen Kovariaten abgebildet as residual vs fitted used! Can use, what Stata calls a command it in an external.txt tab or files. Your browser checking regression assumptions, more specifically the normality assumption residuals and normal qq qq plot residuals!
qq plot residuals 2021


text:
# Reynolds number for water

The Reynolds number is a dimensionless number.

Symbol
$\mathrm{Re}_w$
Formulae
 $\frac{V_w D_{ext}}{100\nu_w}$ extermal convection $\frac{V_{fluid} D_{in}}{\nu_w}$ internal convection
Related
$D_{ext}$
$D_{in}$
$V_{fluid}$
$V_w$
Used in


text:
# What is, really, the stable homotopy category?

When you try to understand the fuss behind the new good categories of spectra that arose on the 90's, you read things such as the following paragraph written by Peter May (from "The Hare and the Tortoise"):

All consumers are now in agreement: Mike's stable homotopy category is definitively the right one, up to equivalence. However, the really fanatical hare demands a good category even before passage to homotopy, with all of the modern bells and whistles. The ideal category of spectra should be a complete and cocomplete Quillen model category, tensored and cotensored over the category of based spaces (or simplicial sets), and closed symmetric monoidal under the smash product. Its homotopy category (obtained by inverting the weak equivalences) should be equivalent to Mike's original stable homotopy category.

Here "Mike" is Michael Boardman. Question number zero would be, could anybody share Boardman's "Stable homotopy theory" mimeographed notes where he introduces said category? Both that and Vogt's "Boardman's stable homotopy category" are hard to find, so it's hard for me to know what they are talking about.

But suppose I know what Boardman's stable homotopy category is. Why should I agree that this is "the" right stable homotopy category, up to equivalence?

I am guessing that the right way to formalize what I mean is: elabore a desiderata for a Stable Homotopy Category, prove that Boardman's satisfies them, and then prove that any two categories satisfying those axioms are equivalent.

Has that been done? If so, where?

I can try to answer my question. Margolis' book "Spectra and the Steenrod Algebra" from 1983 does have such a list of axioms, in section 1.2.

Is that an idiosyncratic list of axioms or is it really what homotopy theorists of the time would agree that it's exactly what they would have wanted?

I know this is perhaps argumentative. But the following is not. At the end of said chapter in Margolis' book, he conjectures that any two categories satisfying those axioms are equivalent. But at the time, it apparently wasn't established.

Has it been established since?

But maybe there is another characterization of "the" stable homotopy category, of which Boardman's (or Adams', or...) would be an example; I'd be interested by that, too.

I am mildly aware of the fact that there is a stable $\infty$-categorical universal property. That is certainly interesting, but I would be interested to see how it could be formulated in older language (model categories?) since such a formulation is, in the spirit of my question, anachronistic. (Not that I would find it uninteresting).

• For the last question, see mathoverflow.net/questions/67227/… Jan 11, 2016 at 13:24
• The third part of Adams's book "Stable homotopy and generalized homology" is a standard source for Boardman's category of spectra. Jan 11, 2016 at 14:36
• You might also want to look at Hovey-Palmieri-Strickland's Axiomatic Stable Homotopy monograph. It continued what Margolis started and wrote down definitive axioms, then developed the theory from the axioms and provided a number of examples, including the stable homotopy category and the derived category of a ring. Jan 11, 2016 at 21:12
• Chapter V from Boardman's manuscript is available here: math.ucr.edu/~res/inprogress/Boardman-V.pdf Jan 12, 2016 at 10:42
• @Bruno: May's quote "[Frank Adams'] category had only connective spectra in it" refers to the Springer Lecture notes (no. 3) version from the 1961 Berkeley lectures, not the blue book version from the 1971 Chicago lectures. Mar 10, 2021 at 14:40

I am mildly aware of the fact that there is a stable ∞-categorical universal property. That is certainly interesting, but I would be interested to see how it could be formulated in older language (model categories?)

Symmetric spectra have a model categorical universal property: they form the initial stable monoidal model category (theorem of Shipley). A subtlety is that this statement holds with the positive stable model structure, but at least this is Quillen equivalent to the usual one and so captures the same homotopy theory.

At the end of said chapter in Margolis' book, he conjectures that any two categories satisfying those axioms are equivalent. But at the time, it apparently wasn't established.

Has it been established since?

Shipley also shows that any category satisfying Margolis's axioms that actually comes from a stable monoidal model category, must be equivalent to the stable homotopy category (as a monoidal category). That pretty much resolves Margolis's conjecture I'd say, since by now everyone agrees that triangulated categories are just not enough for these kinds of questions.

The relevant paper of Shipley is called Monoidal Uniquness of Stable Homotopy theory.

Here are some funky bonus facts:

• Given an object in a symmetric monoidal $\infty$-category, there exists a universal functor which is initial among symmetric monoidal functors which invert the given object. The symmetric monoidal $\infty$-category of spectra is obtained from pointed spaces with smash product by formally inverting $S^1$ in this way. (The symmetric monoidal $\infty$-category of pointed spaces is in turn obtained from spaces with cartesian product by freely pointing.) See this paper of Robalo.

• Given a Grothendieck derivator (another way to keep track of homotopy (co)limits in a homotopy category), there exists a universal stable derivator associated to it. Taking the derivator of spaces, this gives the stable derivator of spectra. Also, every stable derivator is canonically enriched over the derivator of spectra. See the appendix of this paper of Cisinski and Tabuada. The language of derivators is very elementary (no homotopy theory used!) and is pretty old (early 80's).


text:
# Partial derivative stationary point

1. Feb 23, 2017

### Taylor_1989

1. The problem statement, all variables and given/known data
Hi guys, I am having real trouble with the function 10ii) I can take the derivatives, but I feel like I am missing something, with what I have done. I set $f_x=0$and $f_y=0$ but really cant seem to find away to solve, i keep getting (0,0) which when I plug into wolfram it come out with $$x=0,y=-1/2$$, $$x=-1,y=0$$ and $$x=1, y=-2$$
So this make me think have I done the partial right?

2. Relevant equations

3. The attempt at a solution

$$\frac{\partial}{\partial x}=-2(x^2y-x-1)(2xy-1)-2(x^2-1)(2x)$$
$$\frac{\partial}{\partial y}=-2(x^2y-x-1)(x^2)$$

$$-2(x^2y-x-1)(2xy-1)-2(x^2-1)(2x)=0$$
$$-2(x^2y-x-1)(x^2)=0$$

but where do I go from here?

2. Feb 23, 2017

### BvU

Two hashes (#) to get inline $\TeX$, two dollars (\$) to get displayed math

Check $f_x$ and $f_y$ (*); once OK, $f_y$ gives you something to substitute in $f_x=0$

(*)  right, thanks Dick (I copied the 2 inside the [ ] )

Last edited: Feb 23, 2017
3. Feb 23, 2017

### Dick

The partials look ok. Think about the y derivative first. It's zero if i) x=0 OR y=?. Fill in the ? with an expression in x. Substitute that into the x derivative.

4. Feb 24, 2017

### epenguin

Not every function of x and y has a stationary point - it is always possible that your function has none.
In that case you will not be able to find it! But then you have to prove it doesn't exist.
In this case I think there is a stationary point.
fy is nicely factorised, which is help. One of the factors is closely related to a bracket in fx which is another help.

Not every condition for one of your derivatives to be 0 necessarily allows the the other to be 0.
(Though even if it doesn't, your zeros of one of them still gives information about what the surface f looks like)


text:
## Simple power series

Hey all,

Does anybody know if $x^{\alpha}$ can be written in terms of an infinite series?