text:
## WeBWorK Main Forum

### Argument isn't numeric in localtime

by Darwyn Cook -
Number of replies: 10
I am getting a couple of errors after putting a gateway exam in a course. There are two errors:
1) Argument "" isn't numeric in numeric ge (>=) at /opt/webwork/webwork2/lib/WeBWorK/ContentGenerator/ProblemSets.pm line 399.2) Argument "" isn't numeric in localtime at /opt/webwork/webwork2/lib/WeBWorK/ContentGenerator/ProblemSets.pm line 478.The errors are only present in the two courses that have these gateways, so I have to believe they are causing the problem. FWIW the gateways use questions that are drawn from a group of questions.Webwork and the gateways are working fine, so no real issue there.We are running WW on Centos 4.8 with a cvs update two weeks ago using
rel-2-4-patches webwork2 (is that 2.4.7 or do you need -dP?).

P.S. Looking at a similar post from Rob Owen in 2007 I checked the dates for the sets, they all look to be correctly formatted.

### Re: Argument isn't numeric in localtime

by Gavin LaRose -
Hi Darwyn,

The error message you're getting appears to arise from the formatting of the dates when a gateway test was created and is due that occurs in the ProblemSets list. This is an odd error (I think), because if the test was created it should have a creation and due date. How was (were) the gateway assignments (and the assignments, including gateway tests) that are on the offending ProblemSets pages created?

Thanks,
Gavin

### Re: Argument isn't numeric in localtime

by Darwyn Cook -
Well here is a longwinded answer to an easy question. The exams were originally created in a separate Gateway course over a year ago. As I remember the course gateway was created as a gateway and then the problem sets were added (although that could have happened in the opposite order).

I hadn't noticed this before, but in the Gateway course there is only error, the line 399 error. The gateways are used to create printed exams, so I never bothered with the errors. This year I decided to let the students practice online, so I imported the Gateways into my differential equations course.

The set def file for the integrals gateway I imported:

openDate = 10/25/2008 at 12:04am EDT
dueDate = 12/10/2009 at 11:04pm EST
answerDate = 12/17/2009 at 11:04pm EST
assignmentType = gateway
attemptsPerVersion =
timeInterval =
versionsPerInterval =
versionTimeLimit =
problemRandOrder = 0
problemsPerPage =
hideScore = N
hideWork = N
capTimeLimit = 0
problemList =
alfredLibrary/gateway/calcII/elementary/prob1.pg, 1,
alfredLibrary/gateway/calcII/elementary/prob2.pg, 1,
alfredLibrary/gateway/calcII/elementary/prob3.pg, 1,
alfredLibrary/gateway/calcII/elementary/prob4.pg, 1,
alfredLibrary/gateway/calcII/elementary/prob5.pg, 1,
alfredLibrary/gateway/calcII/elementary/prob6.pg, 1,
alfredLibrary/gateway/calcII/elementary/prob7.pg, 1,
alfredLibrary/gateway/calcII/elementary/prob8.pg, 1,
alfredLibrary/gateway/calcII/elementary/prob9.pg, 1,
group:substitution_integrals, 1,

There are two gateways, one for derivatives and fne or integrals, here is the dates in the homework sets editor:

 Welcome_to_WeBWorK 1 58/58 Yes 01/19/2010 at 08:20am EST 01/20/2010 at 11:59pm EST 01/21/2010 at 05:10pm EST Introduction_to_Differential_Equations 7 58/58 Yes 01/20/2010 at 08:20am EST 01/25/2010 at 11:59pm EST 05/31/2010 at 10:33pm EDT Initial_Value_Problems 8 58/58 Yes 01/22/2010 at 07:20am EST 01/27/2010 at 10:59pm EST 01/29/2010 at 10:59pm EST Introduction_to_Modeling 10 58/58 Yes 01/27/2010 at 09:33pm EST 02/01/2010 at 10:59pm EST 02/03/2010 at 10:59pm EST Direction_Fields 11 58/58 Yes 01/29/2010 at 07:20am EST 02/03/2010 at 10:59pm EST 05/31/2010 at 10:33pm EDT Integrals_Gateway 10 58/58 Yes 01/20/2010 at 08:24am EST 05/10/2010 at 11:59pm EDT 05/11/2010 at 11:59pm EDT Derivatives_Gateway 16 58/58 Yes 01/20/2010 at 08:24am EST 05/10/2010 at 11:59pm EDT 05/11/2010 at 11:59pm EDT Separable_Equations 13 58/58 Yes 05/17/2010 at 10:33pm EDT 05/24/2010 at 10:33pm EDT 05/31/2010 at 10:33pm EDT Linear_Equations 9 58/58 Yes 05/17/2010 at 10:33pm EDT 05/24/2010 at 10:33pm EDT 05/31/2010 at 10:33pm EDT Linear_Models 9 58/58 Yes 05/17/2010 at 10:33pm EDT 05/24/2010 at 10:33pm EDT 05/31/2010 at 10:33pm EDT Non_Linear_Models 7 58/58 Yes 05/17/2010 at 10:33pm EDT 05/24/2010 at 10:33pm EDT 05/31/2010 at 10:33pm EDT Eulers_Method 10 58/58 Yes 05/17/2010 at 10:33pm EDT 05/24/2010 at 10:33pm EDT 05/31/2010 at 10:33pm EDT quotient_rule_derivatives 8 0/58 No 10/10/2008 at 11:14pm EDT 06/17/2010 at 10:59pm EDT 06/18/2010 at 10:59pm EDT chain_rule_derivatives 10 0/58 No 10/10/2008 at 11:28pm EDT 06/17/2010 at 10:59pm EDT 06/18/2010 at 10:59pm EDT miscellaneous_derivatives 4 0/58 No 10/10/2008 at 11:50pm EDT 06/17/2010 at 10:59pm EDT 06/18/2010 at 10:59pm EDT product_rule_derivatives 8 0/58 No 01/25/2010 at 09:59pm EST 06/17/2010 at 10:59pm EDT 06/18/2010 at 10:59pm EDT substitution_integrals 4 0/58 No 10/24/2008 at 09:13pm EDT 10/31/2010 at 09:13pm EDT 11/03/2010 at 09:13pm EDT

### Re: Argument isn't numeric in localtime

by Gavin LaRose -
Hi Darwyn,

Thanks. What do the data associated with the actual set that a student has taken look like? e.g., if you select the student and "view/edit all sets for one user"? I'm curious what shows up in the 'due date' and 'answer date' field for the versions of the sets that the student(s) take (and which are on the ProblemSets page that is generating this error message). Do you have access to the mysql server? If so, I'd also be interested in seeing what the value for the version_creation_time column in the course_set_user is for that set.

The error message(s) say that these field are blank, which I think shouldn't be the case if the version was created by taking the test. If the test version were created in some other way it's possible that they could be blank---but I'm not thinking of how that could be the case.

Gavin

### Re: Argument isn't numeric in localtime

by Darwyn Cook -
Well you convinced me that the problem was specific to these sets, so I have found one solution: I just made new sets with the same problems and they don't have these errors.

If you want I can still look in the mysql server.

### Re: Argument isn't numeric in localtime

by Darwyn Cook -
I was a bit premature with my optimism. Checking again, remaking the problem set did not help. I tried making a different gateway in another course with just one problem in it, same issue, so it looks like a site issue.
Here is one of the pieces of info you requested, I will have to get the mysql info tomorrow.

Assigned Edit set for admin Dates
Derivatives_Gateway
 Open: 01/20/2010 at 08:24am EST Due : 05/10/2010 at 11:59pm EDT Ans : 05/11/2010 at 11:59pm EDT

test 1
 Open: 01/20/2010 at 08:24am EST Due : 05/10/2010 at 11:59pm EDT Ans : 05/11/2010 at 11:59pm EDT

test 2
 Open: 01/20/2010 at 08:24am EST Due : 05/10/2010 at 11:59pm EDT Ans : 05/11/2010 at 11:59pm EDT

test 3
 Open: 01/20/2010 at 08:24am EST Due : 05/10/2010 at 11:59pm EDT Ans : 05/11/2010 at 11:59pm EDT

test 4
 Open: 01/20/2010 at 08:24am EST Due : 05/10/2010 at 11:59pm EDT Ans : 05/11/2010 at 11:59pm EDT

Integrals_Gateway
 Open: 01/20/2010 at 08:24am EST Due : 05/10/2010 at 11:59pm EDT Ans : 05/11/2010 at 11:59pm EDT

test 1
 Open: 01/20/2010 at 08:24am EST Due : 05/10/2010 at 11:59pm EDT Ans : 05/11/2010 at 11:59pm EDT

test 2
 Open: 01/20/2010 at 08:24am EST Due : 05/10/2010 at 11:59pm EDT Ans : 05/11/2010 at 11:59pm EDT

test 3
 Open: 01/20/2010 at 08:24am EST Due : 05/10/2010 at 11:59pm EDT Ans : 05/11/2010 at 11:59pm EDT

test 4
 Open: 01/20/2010 at 08:24am EST Due : 05/10/2010 at 11:59pm EDT Ans : 05/11/2010 at 11:59pm EDT

test 5
 Open: 01/20/2010 at 08:24am EST Due : 05/10/2010 at 11:59pm EDT Ans : 05/11/2010 at 11:59pm EDT

### Re: Argument isn't numeric in localtime

by Gavin LaRose -
Hi Darwyn,

Thanks, that's helpful. The first thing is that it seems to clear up that the due and answer dates are correctly set (as we'd hope).

I think this means that the line 399 error is probably looking at the attempts_per_version value for the set. Is there any chance that is empty? (On the set editing page, that's the "Number of Graded Submissions per Test" field.)

The line 478 error has to do with the stored time when the set version was created. I can't see why that would be empty; when the set version is created it should be read from the system time. If you can see if the version_creation_time in the database is blank for one of these sets that would probably be useful.

Gavin

### Re: Argument isn't numeric in localtime

by Darwyn Cook -
Gavin,
You nailed the 399 error, the number of graded submissions per test was blank, changing that to 100 cleared those errors.

You may have noticed that I have studiously avoided your requests for information from the database, I hope to have that tomorrow.

FWIW, the line 478 errors are in the course I imported the Gateways into, they do not appear in the course they were exported from.

### Re: Argument isn't numeric in localtime

by Gavin LaRose -
Hi Darwyn,

One more thought/question: the line 478 error is in the course you imported gateways to. Did you import the gateway set (that is, the template set that is assigned to all users), or the set version (that is, the test that shows up as having been taken by the user)? If the latter, the mystery is probably cleared up---if you export a set version, it won't export a large amount of the version data (it never occurred to me that would be desirable), and the version creation time is one of those fields that would therefore be blank when the set was built in the new course, which would result in the error you're seeing.

And if that's the case, it suggests that I might want to make the set exporting function a little more robust with respect to gateway data.

Thanks,
Gavin

### Re: Argument isn't numeric in localtime

by Darwyn Cook -
In the homework sets editor I used the export radio dialog button to export the integrals and derivatives gateway along with the sets that I used to make up the groups for some of the questions.
I used the file manager to download those set .def files to my computer, and then used file manager in the other course to upload them. Then I used the import set feature in homework sets editor to import them.

I don't think it is possible to do that with a single test that someone has taken.


text:
# Why is the ML ratio test of two normal means with different variances impossible?

Say I have IID samples from possibly different means and exclusive variances, $$X_{1i} \sim N(\mu_1,\sigma_1^2), X_{2j} \sim N(\mu_2,\sigma_2^2)$$, where $$(i = 1, \cdots, m, j = 1, \cdots, n)$$, and $$\sigma_1 \neq \sigma_2$$.

Test : $$H_0 : \mu_1 = \mu_2$$ vs. $$H_1 : \mu_1 \neq \mu_2$$.

According to my professor, there is no maximum likelihood ratio test with significance level $$\alpha$$ for this model.

We exactly know that there can be test (t-test) for different mean-same variance test, and F-test for the comparison of variances from possibly different mean, and different variance model.

But I cannot find the formal proof for this. He mentioned that this fact can be driven from techniques of Harmonic Analysis, but is there anyone to exactly show what it says?

• Are $\sigma_1,\sigma_2$ known? – StubbornAtom Oct 23 '18 at 5:45

## 1 Answer

Hi: For a test to be optimal, a necessary condition ( there are others ) is that the test statistic used for the test has to be a pivotal quantity. Being a pivotal quantity means that the statistic has to have a distribution that's independent of all the parameters of the distribution except for the one being tested.

So, in the simple case where say, there's one mean being tested and normality is assumed, it's possible to divide the parameter being tested, $$\beta$$, by the variance ( or the estimated variance ) and come up with a test-statistic that is a pivotal quantity. ( because it's normal zero one under the null ).

In the case of 2 means and 2 different variances, it's not possible to divide the numerator by something and obtain a pivotal quantity. I've never seen a proof of the impossibility of this in a statistics text so your professor is probably correct that the proof uses harmonic analysis. But I hope that I atleast somewhat explained what is trying to be proven so that you know what to look for when searching for a proof. Googling for Sattherwaite's Approximation may help during your search also. Good luck.

P.S: Another necessary condition ( and maybe sufficient, I think ) for optimality of the test is that the test statistic should be able to be derived using the neyman pearson lemma. But, as far what your professor is talking about in regard to harmonic analysis, my guess is that he's referring to the pivotal quantity part of the necessary condition.

format: plain


text:
## Journal of the Mathematical Society of Japan

### On Alexander polynomials of certain $(2,5)$ torus curves

#### Abstract

In this paper, we compute Alexander polynomials of a torus curve $C$ of type $(2,5)$, $C:\,f(x,y)=f_{2}(x,y)^{5}+f_{5}(x,y)^{2}=0$, under the assumption that the origin $O$ is the unique inner singularity and $f_{2}=0$ is an irreducible conic. We show that the Alexander polynomial remains the same with that of a generic torus curve as long as $C$ is irreducible.

#### Article information

Source
J. Math. Soc. Japan Volume 62, Number 1 (2010), 213-238.

Dates
First available in Project Euclid: 5 February 2010

https://projecteuclid.org/euclid.jmsj/1265380429

Digital Object Identifier
doi:10.2969/jmsj/06210213

Mathematical Reviews number (MathSciNet)
MR2648221

Zentralblatt MATH identifier
1184.14045

Keywords
torus curve Alexander polynomial

#### Citation

KAWASHIMA, Masayuki; OKA, Mutsuo. On Alexander polynomials of certain $(2,5)$ torus curves. J. Math. Soc. Japan 62 (2010), no. 1, 213--238. doi:10.2969/jmsj/06210213. https://projecteuclid.org/euclid.jmsj/1265380429

#### References

• E. Artal Bartolo, Sur les couples des Zariski, J. Algebraic Geom., 3 (1994), 223–247.
• B. Audoubert, C. Nguyen and M. Oka, On alexander polynomials of torus curves, J. Math. Soc. Japan, 57 (2005), 935–957.
• B. Audoubert, T. C. Nguyen and M. Oka, Erratum to: “On Alexander polynomials of torus curves” [J. Math. Soc. Japan, 57 (2005), 935–957; mr2183580], J. Math. Soc. Japan, 59 (2007), 911–912.
• A. I. Degtyarev, Alexander polynomial of a curve of degree six, J. Knot Theory Ramifications, 3 (1994), 439–454.
• H. Esnault, Fibre de Milnor d'un cône sur une courbe plane singulière, Invent. Math., 68 (1982), 477–496.
• M. Kawashima, Classification of local singularities on torus curves of type (2,5), Tokyo Journal of Mathematics, to appear.
• A. Libgober, Alexander polynomial of plane algebraic curves and cyclic multiple planes, Duke Math. J., 49 (1982), 833–851.
• A. Libgober, Alexander invariants of plane algebraic curves, In Singularities, Part 2 (Arcata, Calif., 1981), Proc. Sympos. Pure Math., 40, Amer. Math. Soc., Providence, RI, 1983, pp. 135–143.
• F. Loeser and M. Vaquié, Le polynôme d'Alexander d'une courbe plane projective, Topology, 29 (1990), 163–173.
• J. Milnor, Singular Points of Complex Hypersurface, 61, Ann. of Math. Stud., Princeton Univ. Press, 1968.
• M. Namba, Geometry of projective algebraic curves, Decker, New York, 1984.
• M. Oka, Non-degenerate complete intersection singularity, Hermann, Paris, 1997.
• M. Oka, Alexander polynomial of sextics, J. Knot Theory Ramifications, 12 (2003), 619–636.
• M. Oka, A survey on Alexander polynomials of plane curves, Singularités Franco-Japonaise, Séminaire et congrès, 10, Soc. Math. France, Paris, 2005, pp. 209–232.
• R. Randell, Milnor fibers and Alexander polynomials of plane curves, In Singularities, Part 2 (Arcata, Calif., 1981), Proc. Sympos. Pure Math., 40, Amer. Math. Soc., Providence, RI, 1983, pp. 415–419.
• C. T. C. Wall, Singular points of plane curves, 63, London Mathematical Society Student Texts, Cambridge University Press, Cambridge, 2004.
• O. Zariski, On the irregularity of cyclic multiple planes, Ann. of Math. (2), 32 (1931), 485–511.


text:
which investment will produce more money: $500 invested at 8% compound annually for three years or$600 invested at 7% compound annually for three years? how much more money?
2. $A_1=500(1+.08)^3= \dots$
$A_2=600(1+.07)^3= \dots$


text:
Scroll To Top

## Sunday, March 24, 2013

MobyT | 1:59 PM

### Anthony Watts proclaims his Greend Credentials

Anthony Watts has put up a post about his solar installations.  Just in case any of his readers thought he was becoming altruistic, he assures them that his reasons for adopting clean energy are the same as his reasons for opposing clean energy:

$$MONEY$$.

The price of electricity in California rivals the prices charged in Australia and, where there is a reasonable feed-in tariff, installing grid-connected solar PV is a no-brainer.

Anthony reassures his readers that he still holds the same values as the rest of the 8% Dismissives (the Almighty Dollar is the sole reason for doing anything) and makes it clear that it's greed not green that drove his decision.

(Anyone interested in comparing PV installations at different locations Click Here.  The data is mainly from PV householders in Australia but looks as if people from other countries are now contributing data as well.)

### Greend with Envy

"My PV is bigger than your PV, so there!"

Anthony reveals his insecurity and green envy as he wonders if climate hawks Joe Romm, Bill McKibben, Michael Mann and James Hansen have a solar PV installation.  Since Anthony can't hold a candle to them when it comes to science and advocacy, he knows the best he can long for is to hold a solar panel :D

### Behaviour more important that beliefs

In the end, it is not so much the beliefs, attitudes, values or world view of individuals that will make the difference, it's behaviour.  It matters less what baser motives led Anthony to install solar PV, the effect is the same - clean energy replaced dirty energy.

Most decisions we make are driven by a mix of motives.  Except for very few very wealthy individuals, cost is a factor as well as values.  I'll bet somewhere in there even Anthony Watts had an altruistic thought creep into his brain, that he may be doing some good for the world, not just his hip-pocket.

Ultimately household installations aren't enough.  We need to replace all fossil-fuel electricity generators with clean renewable energy generation, fed by solar, wind, wave, geo-thermal or other source.

With people like Watts saying to his luddite followers that 'going green' doesn't mean 'going soft', it might help lower their resistance such that clean energy becomes if not desirable at least acceptable in their little minds.  (On his blog, Watts still pushes for oil and coal and promotes 'we're heading for an ice age' articles, so there's a long way to go.)

### Climate Hawks - keep working on Businesses and Governments

As long as there are enough Joe Romms and James Hansens and Michael Manns and Bill McKibbens and David Karolys and Tim Flannerys and Will Steffens in the world, and enough big and small companies all around the world doing the R&D and mass production, the price of clean energy will continue to fall and the shift to clean energy will continue.

Although clean energy (eg wind and solar) is growing exponentially, we're not yet making the shift quickly enough.

1. Sou you have your bones to pick with HotCopper {I visited and their intro page was enough to chase me away. Life's too short.}

And I got my bone to pick with fake skeptics in particular the theater over at SkepticForum.

After I shared a little something I posted after sharing a quotes and links from this post and "Has Anthony Watts Become a Leftist-Warmist?" and listened to that interview with Anthony Watts, I couldn't resist posting this:
~ ~ ~ ~ ~ ~ ~

"Oy vey,
What a revealing interview - Anthony Watts clearly exposes his tunnel vision and loopy thinking.

And the interviewer,
jeez, talk about a text book example of one sided {dare I say goal oriented} questioning.

Quote [ Watts himself has said the reason he backed off from viewing global warming as a threat was because he feared that policies to address the problem might involve a rise in taxation. (See about 58 seconds into this video interview = http://www.youtube.com/watch?v=UmIJCGQzCiU ]
~ ~ ~

For all the talk about the profit motive influencing scientists,
not one question about the masses of money seeded by a very few {incredibly wealthy and disgusting hubristic and frighteningly shallow self-obsessive thinking} who through their "ThinkTank" have strategically attacked science as though it was a football game and winning was all that mattered - rather than a struggle to better understand.

We see that right here at SkepticForum - where utter nonsense can be repeated endlessly and all rhetorical tricks are welcome - because the arguing is considered more important than the learning."

http://www.skepticforum.com/viewtopic.php?f=40&t=19892

1. Good one - thanks Peter :D That's an interesting discussion board.

2. sorry about the typos. :-(

Instead of commenting as "Anonymous", please comment using "Name/URL" and your name, initials or pseudonym or whatever. You can leave the "URL" box blank. This isn't mandatory. You can also sign in using your Google ID, Wordpress ID etc as indicated. NOTE: Some Wordpress users are having trouble signing in. If that's you, try signing in using Name/URL or OpenID. Details here.


text:
# [Case-study preview] Speeding up Stan by reducing redundant computation

Hey folks, I mentioned during the general meeting last week that I was working on a case study walking through an idea I had for speeding things up in Stan for models with contrast matrices with high redundancy. I’m still working on the prose part, but I thought I’d post a link to the code in case anyone wants to take a peek. I’ve included comments in both the R and Stan files to try to describe what’s going on.

The data scenario I use as an example is actually the same as from the “Multivariate Priors for Hierarchical Models” section of the Stan User’s Guide, with the tweak that I use terminology from my neck of the woods (Psychology). So the data come from multiple human “subjects”, each of which is randomly assigned to one of two groups, then all subjects are observed multiple times in each of two conditions.

When there are 100 participants and 100 observations in each condition per participant, the “_slower” model (pretty much just the final optimized model from the manual) takes about 25 minutes to achieve 4e3 samples while the “_faster” model takes about 4 minutes.

The geometry of the parameter space should be identical between these models, all I’m doing is identifying computations that are redundant, doing them once and indexing into them rather than computing them fresh every time. So there shouldn’t ever be any cost to using this trick and I think it should be possible to automate it in packages like rstanarm and brms.

19 Likes

Just had a thought while procrastinating on the prose: design matrices tend to have high regularity within columns (ex. intercept column, effect columns where entries are simply *-1 of each other) so I wonder if that regularity could be used to further reduce computation, or whether dot_product() is already super optimized that breaking it apart and doing the sums explicitly would just slow things down.

@mike-lawrence The trick here is that there are multiple observations with sort of the same right hand side, right?

This sounds similar to the thing where you replace a bunch of bernoullis that all have the same right hand side with a binomial.

I did some thinking about a similar thing at some point. I decided for intercept only models (or at least for the intercept part of a hierarchical regression) we could get somewhere, but I never wrestled it into a form where I could write a clean function we might put in Math (Group orderings in regression).

@andrewgelman is often forwarding e-mails from people who have large hierarchical regressions and they’re happy with the regression but it’s just too slow cause they have an absurd number of data points.

Since you already started a case study, do you think you could write and show examples of these other things to? Like expand the contents to “Last Mile Optimizations for Hierarchical Regressions” maybe? The idea being, after everything else is done, these are things you can do to make your model faster. They don’t work for everyone, so it’s hard to make them general, but there is a checklist of things people can look through.

In that theme, I think it would make sense to add a section on using the GLMs (https://mc-stan.org/docs/2_22/functions-reference/bernoulli-logit-glm.html) and reduce_sum. So that’s 5 things, which is a pretty sizeable case study.

And this is purposefully separate from the model approximation things you might try to do (or switching to lme4 that ends up happening a lot).

2 Likes

I hadn’t thought of it precisely like that but yes that’s correct and a good way to describe it succinctly/generalizably.

I’d already been planning to add a second model with largely the same structure but with a binomial outcome to add the sufficient statistics trick as an additional speed-up in that situation. Are there any other optimization you were thinking of?

1 Like

Ooooh right, that’s the name for this stuff. I’d forgotten.

So multivariate normal + bernoulli/binomial

And the other thing I was talking about is there’s redundant calculation on intercepts a lot.

So for instance, if you have 4 age groups, 5 income groups, and 50 states and you index age with i, income with j, and states with k, then you might have a bunch of intercepts:

\mu_i + \mu_j + \mu_k

And then there are 20 * 50 = 1000 possible right hand sides there, but like your survey could definitely have been more than 1000 people! So you could get savings in kinda the same way.

But that doesn’t always work, like in a lot of hierarchical models you would have more groups than people. If we have N < 1000 rows of a dataframe and our model includes these terms:

\mu_i + \mu_j + \mu_k

That will take 3 * N loads and 2 * N adds. But! What if we break this in to two pieces:

\mu_{ij} = \mu_i + \mu_j

\mu_{ij} + \mu_k

Now the first thing takes 2 * 20 loads + 20 adds, and the second thing takes 2 * N loads and N adds, which is less work.

You could apply this recursively to reduce the amount of copying you gotta do. These hierarchical models are memory bound so this is significant. You’re moving around 2/3 of the memory so you’d expect a comparable speedup!

And then also I think always feeding whatever you do into the appropriate glm and maybe using reduce_sum makes sense. I believe the glms work via some sufficient statistics stuff too.

Yes, I’m totally into this hypothetical document explaining how to fit big hierarchical regressions without having to run the computer overnight.

1 Like

@andrewgelman I’ve clearly failed to make time to write this up. Maybe a good project (parsing what I’ve done, adding prose) for one of your many aspiring students? I think it’s upper-year undergraduate or maybe early-year graduate level. And possibly extra credit for adding it to rstanarm/brms as something that’s done for the user automatically.

3 Likes

All volunteer work is successful as long as you enjoyed doing it! Finishing projects are for work, yuck.

3 Likes

AKA i.i.d.? (Two TLAs in a single A-only sentence.) I discuss this in the efficiency chapter of the user’s guide in the section Exploiting sufficient statistics, which is another way to think about it. The sum of bernoulli outcomes is sufficient if they are i.i.d.

If it can be done in transformed data, it only happens once and there’s almost no relative overhead compared to model fitting.


text:
# Resetting std::stringstream to state before read

## Recommended Posts

Hello,

I need to peek into my stringstream to find out the position of the next occurence of certain characters, like "#". I use std::getline for that, and after that, the stream needs to be reset to its original state:

size_t getCharPosition(std::stringstream& stSourceStream, char c)
{
const auto position = stSourceStream.tellg();

std::string stTemp;
std::getline(stSourceStream, stTemp, c);

const auto newPosition = stSourceStream.tellg();
stSourceStream.seekg(position, std::ios::beg);

return (size_t)newPosition;
}

Now this normally works well, unless c doesn't occur in the stream, in which case after that operation, the stream isn't valid anymore, and I can't read from it again, probably because I read over the end of the stream and some internal flag is set.

Now the question is, what do I have to do additionally to restore the stream correctly? I'd also take a entire different/better method for finding out the next position of a character in the stream. Thanks!

##### Share on other sites
std::basic_ios::clear() should do the trick.

##### Share on other sites

The one time in the standard libary where clear doesn't clear the whole damn thing... thanks :D

## Create an account

Register a new account

• ### Forum Statistics

• Total Topics
628298
• Total Posts
2981890

• 9
• 9
• 11
• 10
• 10


text:
# std::unordered_map::max_size

 size_type max_size() const noexcept; (since C++11)

Returns the maximum number of elements the container is able to hold due to system or library implementation limitations, i.e. std::distance(begin(), end()) for the largest container.

## Contents

(none)

### Return value

Maximum number of elements.

Constant.

### Notes

This value typically reflects the theoretical limit on the size of the container. At runtime, the size of the container may be limited to a value smaller than max_size() by the amount of RAM available.

### Example

#include <iostream>
#include <unordered_map>

int main()
{
std::unordered_map<char,char> s;
std::cout << "Maximum size of a 'unordered_map' is " << s.max_size() << "\n";
}

Possible output:

Maximum size of a 'unordered_map' is 18446744073709551615


text:
the best way to ensure the accuracy and safety of your accounts is to:

# the best way to ensure the accuracy and safety of your accounts is to:

S
9.9k points
interest is: a. a charge for lending money to a bank b. the amount owed for borrowing money c. the amount added into your savings when opening a bank account d. a charge for the convenience of accessing money stored in your bank accountB
To earn as much interest as possible, you should open a savings account that earns _______ interest and has the ______ interest rate. a) compound; lowest b) compound; highest c) simple; lowest d) simple; highestA
John just opened a savings account and wants to maximize the amount of interest he earns. Which of the following actions would enable him to earn MORE interest a) Selecting an account with a high interest rate b) Leaving his money in the account for long period of time c) Transferring money into his checking account each month d) Both A BD
You are opening a savings account that earns compound interest. Which compounding frequency will earn you the MOST money? a) Compounding 1 time a year b) Compounding 4 times a year c) Compounding monthly d) Compounding dailyD
Which of the following is a type of savings vehicle? a) Certificate of Deposit b) Money Market Account c) Checking Account d) Both A BA
Which of the following is generally true about savings vehicles? a) Savings vehicles are only useful for long-term investments b) People should evaluate different forms of savings vehicles based on their needs c) Savings vehicles are never insured d) All of the aboveB
Which of the following savings vehicles usually requires a high minimum balance? a) Simple savings account b) Certificate of Deposit (CD) c) Checking account d) All of the aboveB
Which of the following accounts will give you the LEAST access to your money a) Simple savings account b) Certificate of Deposit (CD) c) Checking account d) cashB
The purpose of a budget is to: a) help you plan how you will spend the money you earn or receive b) stop you from spending too much money c) increase the balance of your savings account d) tell you how much you owe the government in taxesA
When it comes to saving money, what is a good rule of thumb. a) Keep most of your savings in your checking account b) Put aside money for savings each month c) Choose the savings account with the lowest interest rate d) Putting anything you can't afford on your credit cardB
The three parts of the Federal Reserve System are the Reserve Banks, the Federal Open Market Committee (FOMC), and the a) Board of Governors b) United States Mint c) Department of the Treasury d) Secretary of StateA
Which of the following is NOT one of the responsibilities of the Federal Reserve a) Supervising the Reserve banks b) Setting the nation's monetary policy c) Establishing the federal budget d) Maintaining a stable banking systemD
Which of the following is a unique feature of credit unions? a)Credit unions are typically owned and run by their members b)Credit unions limit membership to certain people or groups c) Credit unions offer a wide variety of banking services d) Both A & BD
Which type of account is typically the MOST liquid? a) Checking account b) Savings account c) Certificate of Deposit d) Exchanged Traded FundA
Savings accounts usually offer _________ interest rates than checking accounts. It is _________ to access your money in a savings account than in a checking account a) lower; harder b) lower; easier c) higher; harder d) higher; easier...
What part of a check is the LEAST important? a) check number b) signature line c) memo line d) routing number...
Which of the following represent typical account fees? a) ATM fee b) Service fee c) Minimum balance fee d) Routing number...
What's the best strategy for avoiding ATM fees? a) Onlu use ATMs close to your house b) Only use ATMs to withdraw cash c) Only use ATMs in your bank's network d) Only use ATMs you have used before...
What's the purpose of balancing or monitoring your checking account? a) To find out which checks you have bounced b) To help you calculate how much money you have in your account c) To keep an estimation of how many checks you write d) To stop identity theft from occurring...
The best way to ensure the accuracy and safety of your accounts is to: a) Monitor your online accounts regularly b) Shred your paper account statements weekly c) Call your customer service center daily d) Balance your checkbook annually...
Which of the following payment types requires you to pay upfront? a) Money order b) Cashier's check c) Pre-paid card d) All of the aboveD
Which of the following statements comparing debit cards to credit cards is TRUE a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
a) b) c) d)...
the best
studybud

Surround your text in *italics* or **bold**, to write a math equation use, for example, $x^2+2x+1=0$ or $$\beta^2-1=0$$

Use LaTeX to type formulas and markdown to format text. See example.


text:
# pyspark.sql.GroupedData¶

class pyspark.sql.GroupedData(jgd, df)[source]

A set of methods for aggregations on a DataFrame, created by DataFrame.groupBy().

New in version 1.3.

Methods

 agg(*exprs) Compute aggregates and returns the result as a DataFrame. apply(udf) It is an alias of pyspark.sql.GroupedData.applyInPandas(); however, it takes a pyspark.sql.functions.pandas_udf() whereas pyspark.sql.GroupedData.applyInPandas() takes a Python native function. applyInPandas(func, schema) Maps each group of the current DataFrame using a pandas udf and returns the result as a DataFrame. avg(*cols) Computes average values for each numeric columns for each group. cogroup(other) Cogroups this group with another group so that we can run cogrouped operations. Counts the number of records for each group. max(*cols) Computes the max value for each numeric columns for each group. mean(*cols) Computes average values for each numeric columns for each group. min(*cols) Computes the min value for each numeric column for each group. pivot(pivot_col[, values]) Pivots a column of the current DataFrame and perform the specified aggregation. sum(*cols) Computes the sum for each numeric columns for each group.


text:
Navigate back to the activity.

### Main Ideas

• Visualizing the geometric relationship between the gradient and a scalar field
• Clarification of how gradients look in two and three dimensions.
• Increased familiarity with Maple and/or Mathematica

Estimated Time: 10 - 15 minutes

See files on activity page.

### Prerequisite Knowledge

A rudimentary understanding how a gradient acts on a scalar field.

### Activity: Introduction

This activity is a great follow up to acting out the gradient.

The students should first look at all the graphs inside the maple/mathematica files. Then have the students plot some functions of their own.

### Extensions

This activity is part of a sequence of activities which address the Geometry of Vector Fields. The following activities are included in this sequence.

• Preceding activity:
• Curvilinear Basis Vectors: This kinesthetic activity students are asked to point in $\hat{r}$, $\hat{\theta}$, $\hat{\phi}$, and $\hat{z}$ directions in reference to an origin within the classroom which begins class discussion regarding about the directions of curvilinear basis vectors at various points in space.
• Follow-up activities:
• Drawing Electric Field Vectors: In direct analogy to Drawing Equipotential Surfaces, this small group activity has students sketch the electric field due to a quadrupole in the plane of the charges.
• Visualizing Electric Flux: This computer visualization activity uses Mathematica to explore the effects of placing a point charge inside, outside, and on a cubical Gaussian surface which allows students to visualize the electric flux of a point charge through a Gaussian surface in different locations with respect to the point charge.
• Visualizing Divergence: This computer visualization activity has students predict the sign of divergence at various points in many vector fields generated by a Mathematica notebook.
• Visualizing Curl: Similar to Visualizing Divergence, this activity uses a Mathematica notebook of various vector fields to assist students in the geometric interpretation of the curl of a vector field by predicting the sign of the curl at various points in vector fields.

This activity is part of a sequence of activities which address the Geometry of the Gradient. The following activities are included within this sequence.

• Preceding activities:
• Acting Out the Gradient: This kinesthetic activity introduces students to the geometric concept of the gradient through an imaginary elliptic hill in the classroom where students use their arms to represent the gradient at their local point within the classroom.
• Navigating a Hill: In this small group activity, students determine various aspects of local points on an elliptic hill which is a function of two variables. The gradient is emphasized as a local quantity which points in the direction of greatest change at a point in the scalar field. This activity emphasizes the gradient as a local quantity and requires students to perform calculations on a given scalar field.

##### Views

New Users

Curriculum

Pedagogy

Institutional Change

Publications


text:
Interaction at a distance feels like magic.

Magic might be useful if you’re an elf, wizard or unicorn, but such things only exist in stories.

However, there’s something much better than magic: physics!

Wireless interaction is all about physics: radio waves (a type of electromagnetic radiation, similar to visible light) have some sort of property (such as their amplitude, phase or pulse width) modulated by a transmitter in such a way that information can be encoded and, thus, broadcast. When radio waves encounter an electrical conductor (i.e. an aerial), they cause an alternating current from which the information in the waves can be extracted and transformed back into its original form.

## Layers upon Layers¶

If you remember, networks are built in layers.

The most fundamental requirement for a network is some sort of connection that allows a signal to get from one device to the other. In our networking tutorial we used wires connected to the I/O pins. Thanks to the radio module we can do away with wires and use the physics summarised above as the invisible connection between devices.

The next layer up in the network stack is also different from the example in the networking tutorial. With the wired example we used digital on and off to send and read a signal from the pins. With the built-in radio on the micro:bit the smallest useful part of the signal is a byte.

## Bytes¶

A byte is a unit of information that (usually) consists of eight bits. A bit is the smallest possible unit of information since it can only be in two states: on or off.

Bytes work like a sort of abacus: each position in the byte is like a column in an abacus - they represent an associated number. In an abacus these are usually thousands, hundreds, tens and units (in UK parlance). In a byte they are 128, 64, 32, 16, 8, 4, 2 and 1. As bits (on/off signals) are sent over the air, they are re-combined into bytes by the recipient.

Have you spotted the pattern? (Hint: base 2.)

By adding the numbers associated with the positions in a byte that are set to “on” we can represent numbers between 0 and 255. The image below shows how this works with five bits and counting from zero to 32:

If we can agree what each one of the 255 numbers (encoded by a byte) represents ~ such as a character ~ then we can start to send text one character per byte at a time.

Funnily enough, people have already thought of this ~ using bytes to encode and decode information is commonplace. This approximately corresponds to the Morse-code “protocol” layer in the wired networking example.

A really great series of child (and teacher) friendly explanations of “all things bytes” can be found at the CS unplugged website.

The problem with radio is that you can’t transmit directly to one person. Anyone with an appropriate aerial can receive the messages you transmit. As a result it’s important to be able to differentiate who should be receiving broadcasts.

The way the radio built into the micro:bit solves this problem is quite simple:

• It’s possible to tune the radio to different channels (numbered 0-100). This works in exactly the same way as kids’ walkie-talkie radios: everyone tunes into the same channel and everyone hears what everyone else broadcasts via that channel. As with walkie-talkies, if you use adjacent channels there is a slight possibility of interference.

Of course, the micro:bit is still receiving broadcast messages for other address/group combinations. The important thing is you don’t need to worry about filtering those out. Nevertheless, if someone were clever enough, they could just read all the wireless network traffic no matter what the target address/group was supposed to be. In this case, it’s essential to use encrypted means of communication so only the desired recipient can actually read the message that was broadcast. Cryptography is a fascinating subject but, unfortunately, beyond the scope of this tutorial.

## Fireflies¶

This is a firefly:

It’s a sort of bug that uses bioluminescence to signal (without wires) to its friends. Here’s what they look like when they signal to each other:

The BBC have rather a beautiful video of fireflies available online.

We’re going to use the radio module to create something akin to a swarm of fireflies signalling to each other.

First import radio to make the functions available to your Python program. Then call the radio.on() function to turn the radio on. Since the radio draws power and takes up memory we’ve made it so you decide when it is enabled (there is, of course a radio.off() function).

At this point the radio module is configured to sensible defaults that make it compatible with other platforms that may target the BBC micro:bit. It is possible to control many of the features discussed above (such as channel and addressing) as well as the amount of power used to broadcast messages and the amount of RAM the incoming message queue will take up. The API documentation contains all the information you need to configure the radio to your needs.

Assuming we’re happy with the defaults, the simplest way to send a message is like this:

radio.send("a message")

The example uses the send function to simply broadcast the string “a message”. To receive a message is even easier:

new_message = radio.receive()

As messages are received they are put on a message queue. The receive function returns the oldest message from the queue as a string, making space for a new incoming message. If the message queue fills up, then new incoming messages are ignored.

That’s really all there is to it! (Although the radio module is also powerful enough that you can send any arbitrary type of data, not just strings. See the API documentation for how this works.)

Armed with this knowledge, it’s simple to make micro:bit fireflies like this:

# A micro:bit Firefly.
# By Nicholas H.Tollervey. Released to the public domain.
import random
from microbit import display, Image, button_a, sleep

# Create the "flash" animation frames. Can you work out how it's done?
flash = [Image().invert()*(i/9) for i in range(9, -1, -1)]

# The radio won't work unless it's switched on.

# Event loop.
while True:
# Button A sends a "flash" message.
if button_a.was_pressed():
if incoming == 'flash':
# If there's an incoming "flash" message display
# the firefly flash animation after a random short
# pause.
sleep(random.randint(50, 350))
display.show(flash, delay=100, wait=False)
# Randomly re-broadcast the flash message after a
# slight delay.
if random.randint(0, 9) == 0:
sleep(500)

The import stuff happens in the event loop. First, it checks if button A was pressed and, if it was, uses the radio to send the message “flash”. Then it reads any messages from the message queue with radio.receive(). If there is a message it sleeps a short, random period of time (to make the display more interesting) and uses display.show() to animate a firefly flash. Finally, to make things a bit exciting, it chooses a random number so that it has a 1 in 10 chance of re-broadcasting the “flash” message to anyone else (this is how it’s possible to sustain the firefly display among several devices). If it decides to re-broadcast then it waits for half a second (so the display from the initial flash message has chance to die down) before sending the “flash” signal again. Because this code is enclosed within a while True block, it loops back to the beginning of the event loop and repeats this process forever.


text:
# Work done by adiabatic expansion of gas

The formula for the work done by adiabatic expansion of gas is

$$W = C_V\,\mathrm dT.$$

Why the volumetric heat capacity $$C_V$$ is constant and not the heat capacity at constant pressure $$C_p?$$ If the system is at constant volume, it is a isovolumetric process and hence no work can be done in that case.

• Probably better to write $dW = C_V dT$, since it is a differential amount of work. Feb 21, 2021 at 10:37

Pretty simple, $$\Delta U = nC_V\Delta T$$ no matter what the process is. This is called a state function, i.e. it is independent of the path taken and only depends on the final and initial conditions here initial and final temperature.

However, work done is path dependent, so it depends on how you change pressure and volume. Let's say you take a gas from $$p_1,$$ $$V_1,$$ $$T_1$$ to $$p_2,$$ $$V_2,$$ $$T_2.$$ You can say $$\Delta U = nC_V(T_2 - T_1),$$ however we need to know the process (isothermal, adoabetic, polytropic) for finding work done.

• One caveat... This only applies to an ideal gas. In a real gas, the internal energy is also a function of volume. Oct 15, 2017 at 12:08

Part 1. Let's consider first law of thermodynamics.

$\delta Q = dU + \delta W$ (classical sign convention)

or

$\delta Q = dU - \delta W$ (IUPAC sign convention)

In the case of adiabatic process, $\delta Q = 0$.

$0 = dU - \delta W$

$dU = \delta W$

$dU = C_VdT$ (by definition)

$C_VdT = \delta W$

After integration, assuming CV is constant.

$C_V\Delta T = W$

Part 2. Why not Cp?

By definition, $\delta Q_p = dH = C_pdT$. Obviously, we don't have an isobaric process here.

P. S. Yes, those sign convention differences also make me feel bad.

The total differential of the internal energy (at constant composition and allowing only pV work) is usually written as

$$dU = -pdV + TdS$$

However the differential $$dU$$ can be written in terms of other variables, for instance p and T, in which case:

\begin{align} dU &= \left(\frac{\partial U}{\partial V} \right)_T dV + \left(\frac{\partial U}{\partial T} \right)_V dT \\ &= \left[-p + T \left( \frac{\partial S}{\partial V} \right)_T \right] dV + C_V dT \\ &= \left[-p + T \left( \frac{\partial P}{\partial T} \right)_V \right] dV + C_V dT \end{align}

The last equality follows from one of Maxwell's relations. This expression btw is known as the thermodynamic equation of state. For an ideal gas it follows that

$$dU = C_V dT$$

Note that the change in energy is independent of changes in pressure and volume, it depends only on changes in T.

If the heat capacity is constant over the temperature change of interest then you may write

$$\Delta U = C_V \Delta T$$


text:
In today’s tutorial we are going to build off of our last blog post by using the function we created to aggregate the ENTIRE history of team performance data. Once the data is collected we will analyze it primarily using dplyr and wrap up with some ggvis generated visualizations.

As an ode to one of the a music group near to my heart I have decided to assign naming rights to this process that will help you tackle data questions big and small.

### Let’s Get It Started In Here

First thing we need after to do after firing up is load/install the packages we’re going to use.

options(stringsAsFactors = F)
###Load Packages and Bring In Function
c('pipeR','dplyr','rvest','RCurl','ggvis') -> packages
#install.packages('ggvis') if you don't have this package you best install it
lapply(packages,library,character.only = T)
##
## Attaching package: 'dplyr'
##
## The following object is masked from 'package:stats':
##
##     filter
##
## The following objects are masked from 'package:base':
##
##     intersect, setdiff, setequal, union
##
##
## Attaching package: 'rvest'
##
## The following object is masked from 'package:utils':
##
##     history
##
## Loading required package: bitops

Next, we have to feed the function we built into R.

This process is somewhat complicated since Github uses https. I highly reccomend that if you have any questions about the functions being used here, or anywhere in this tutorial, that you use R’s fantastic help function by enter ?functionName

'https://raw.githubusercontent.com/abresler/blog_code/master/nba/functions/getBREFTeamStats.R' -> dope_function
dope_function %>>% (getURL(url = .,ssl.verifypeer=FALSE)) %>>%
(parse(text = .)) -> get
eval(expr = get)

## Getting Loopy to Suck In All the Data

You should see the function getBREFTeamStatTable in your workspace but before we can use it must decide which data we want to investigate. How about the entire history of NBA team statistics

After some perusing it looks like the first official season was 1949-1950. Perfect we know what to feed our function to make her happy, every season end from 1950 to 2015.

How do we go about that you ask? Though R has a multiple options up to this task, today we are going to use the magic of a for loop. Since every season besides the 2014-2015 is in the record books we are going to separate historic seasons from this current 2014-2015 season and then bind the 2 data frames together that way we are only scraping the historic data once.

### Historic Seasons
1950:2014 -> season_ends #these are the years
data.frame() %>>% tbl_df -> all_historic_years #create an empty data frame which will contain all the years

#our FOR loop for every season end from 1950 to 2014
for(s in season_ends){
getBREFTeamStatTable(season_end = s, date = T,table_name = 'team')-> table
s -> table$season_end #add a numeric year to make sorting easy all_historic_years %>>% rbind_list(table) -> all_historic_years #bind year with the master table table %>>% rm #remove the year table to free up memory } #Save Historic Data Somewhere #all_historic_years %>>% #write.csv('2014/november/loop_analyze_visualize/historic_team_data_1950_2014.csv',row.names = F) #Pull in 2015 getBREFTeamStatTable(table_name = 'team',season_end = 2015, date = T) -> data_2015 2015 -> data_2015$season_end

#Bind Both Together -- sort descending
all_historic_years %>>%
rbind_list(data_2015) %>>%
arrange(desc(season_end)) -> all_nba_team_data

That was pretty easy

### But Did It Work?

season table_name bref_team_id team g mp fg fga fg. X3p X3pa X3p. X2p X2pa
2014-2015 team_stats DAL Dallas Mavericks 9.00 2160.00 364.00 755.00 0.48 80.00 227.00 0.35 284.00 528.00
2014-2015 team_stats POR Portland Trail Blazers 9.00 2160.00 353.00 775.00 0.46 97.00 251.00 0.39 256.00 524.00
2014-2015 team_stats TOR Toronto Raptors 9.00 2160.00 329.00 742.00 0.44 64.00 195.00 0.33 265.00 547.00
2014-2015 team_stats PHO Phoenix Suns 8.00 1970.00 304.00 674.00 0.45 69.00 205.00 0.34 235.00 469.00
2014-2015 team_stats GSW Golden State Warriors 8.00 1920.00 309.00 629.00 0.49 76.00 200.00 0.38 233.00 429.00
2014-2015 team_stats BOS Boston Celtics 7.00 1680.00 289.00 615.00 0.47 55.00 183.00 0.30 234.00 432.00
Only one person can accurately describe the sense of achievement and accomplishment we’ve earned for successfully overcoming this data hurdle.

## Time to Analyze

Now that we are in possession of the 1363 row data frame containing every team’s statistical performance since the 1949-50 season it’s time to explore it in order to better understand the data and find interesting ideas through visualization.

One nice way to quickly analyze a bunch of a data frame's numeric data is with the summary function. Let’s use it on our NBA data frame.

all_nba_team_data %>>% summary #summary of the variables
##     season           table_name        bref_team_id
##  Length:1363        Length:1363        Length:1363
##  Class :character   Class :character   Class :character
##  Mode  :character   Mode  :character   Mode  :character
##
##
##
##
##      team                 g              mp              fg
##  Length:1363        Min.   : 6.0   Min.   : 1465   Min.   : 214
##  Class :character   1st Qu.:82.0   1st Qu.:19755   1st Qu.:2932
##  Mode  :character   Median :82.0   Median :19805   Median :3179
##                     Mean   :78.3   Mean   :19087   Mean   :3087
##                     3rd Qu.:82.0   3rd Qu.:19855   3rd Qu.:3491
##                     Max.   :82.0   Max.   :20080   Max.   :3980
##                     NA's   :1      NA's   :141     NA's   :1
##       fga            fg.              X3p             X3pa
##  Min.   : 496   Min.   :0.3100   Min.   : 10.0   Min.   :  75.0
##  1st Qu.:6526   1st Qu.:0.4410   1st Qu.:133.8   1st Qu.: 423.5
##  Median :6899   Median :0.4570   Median :333.0   Median : 963.5
##  Mean   :6784   Mean   :0.4538   Mean   :329.3   Mean   : 943.4
##  3rd Qu.:7380   3rd Qu.:0.4748   3rd Qu.:493.2   3rd Qu.:1388.2
##  Max.   :9295   Max.   :0.5450   Max.   :891.0   Max.   :2371.0
##  NA's   :1      NA's   :1        NA's   :379     NA's   :379
##       X3p.             X2p            X2pa           X2p.
##  Min.   :0.1040   Min.   : 168   Min.   : 358   Min.   :0.3100
##  1st Qu.:0.3140   1st Qu.:2466   1st Qu.:5249   1st Qu.:0.4550
##  Median :0.3430   Median :2785   Median :5996   Median :0.4750
##  Mean   :0.3293   Mean   :2849   Mean   :6102   Mean   :0.4682
##  3rd Qu.:0.3620   3rd Qu.:3449   3rd Qu.:7197   3rd Qu.:0.4910
##  Max.   :0.4280   Max.   :3972   Max.   :9295   Max.   :0.5580
##  NA's   :379      NA's   :1      NA's   :1      NA's   :1
##        ft            fta            ft.              orb
##  Min.   :  99   Min.   : 124   Min.   :0.6250   Min.   :  57
##  1st Qu.:1486   1st Qu.:1984   1st Qu.:0.7310   1st Qu.: 919
##  Median :1646   Median :2196   Median :0.7510   Median :1044
##  Mean   :1635   Mean   :2183   Mean   :0.7498   Mean   :1022
##  3rd Qu.:1836   3rd Qu.:2443   3rd Qu.:0.7700   3rd Qu.:1172
##  Max.   :2434   Max.   :3411   Max.   :0.8320   Max.   :1520
##  NA's   :1      NA's   :1      NA's   :1        NA's   :260
##       drb            trb            ast            stl
##  Min.   : 174   Min.   : 243   Min.   : 115   Min.   :  34.0
##  1st Qu.:2335   1st Qu.:3374   1st Qu.:1679   1st Qu.: 585.5
##  Median :2440   Median :3533   Median :1854   Median : 655.0
##  Mean   :2363   Mean   :3610   Mean   :1812   Mean   : 647.5
##  3rd Qu.:2545   3rd Qu.:3763   3rd Qu.:2041   3rd Qu.: 731.5
##  Max.   :3074   Max.   :6131   Max.   :2575   Max.   :1059.0
##  NA's   :260    NA's   :18     NA's   :1      NA's   :260
##       blk             tov             pf            pts
##  Min.   : 21.0   Min.   :  76   Min.   : 101   Min.   :  614
##  1st Qu.:347.0   1st Qu.:1175   1st Qu.:1731   1st Qu.: 7820
##  Median :399.0   Median :1281   Median :1875   Median : 8340
##  Mean   :399.5   Mean   :1279   Mean   :1833   Mean   : 8047
##  3rd Qu.:460.0   3rd Qu.:1433   3rd Qu.:2028   3rd Qu.: 8881
##  Max.   :716.0   Max.   :2011   Max.   :2470   Max.   :10371
##  NA's   :260     NA's   :260    NA's   :1      NA's   :1
##      pts.g       playoff_team     scrape_time
##  Min.   : 70.0   Mode :logical   Min.   :2014-11-14 16:36:47
##  1st Qu.: 96.6   FALSE:584       1st Qu.:2014-11-14 16:37:02
##  Median :102.4   TRUE :779       Median :2014-11-14 16:37:12
##  Mean   :102.5   NA's :0         Mean   :2014-11-14 16:37:11
##  3rd Qu.:108.6                   3rd Qu.:2014-11-14 16:37:22
##  Max.   :126.5                   Max.   :2014-11-14 16:37:31
##  NA's   :1
##    season_end
##  Min.   :1950
##  1st Qu.:1978
##  Median :1992
##  Mean   :1990
##  3rd Qu.:2004
##  Max.   :2015
##

We now can see all sorts interesting things, for example the highest ever team field goal percentage was 54.5% [the 1984-85 Lakers]. I wonder if we can find quickly find all the unique NBA teams that ever played? Super easy to do in R

all_nba_team_data %>>%
select(team) %>>% #select the team
arrange(team) %>>%  #sort it so
unique %>>% # gives us the unique results
unlist %>>% #comes in a list form so need to unlist it
as.character ## every NBA Team
##  [1] "Anderson Packers"
##  [2] "Atlanta Hawks"
##  [3] "Baltimore Bullets"
##  [4] "Boston Celtics"
##  [5] "Brooklyn Nets"
##  [6] "Buffalo Braves"
##  [7] "Capital Bullets"
##  [8] "Charlotte Bobcats"
##  [9] "Charlotte Hornets"
## [10] "Chicago Bulls"
## [11] "Chicago Packers"
## [12] "Chicago Stags"
## [13] "Chicago Zephyrs"
## [14] "Cincinnati Royals"
## [15] "Cleveland Cavaliers"
## [16] "Dallas Mavericks"
## [17] "Denver Nuggets"
## [18] "Detroit Pistons"
## [19] "Fort Wayne Pistons"
## [20] "Golden State Warriors"
## [21] "Houston Rockets"
## [22] "Indiana Pacers"
## [23] "Indianapolis Olympians"
## [24] "Kansas City Kings"
## [25] "Kansas City-Omaha Kings"
## [26] "Los Angeles Clippers"
## [27] "Los Angeles Lakers"
## [28] "Memphis Grizzlies"
## [29] "Miami Heat"
## [30] "Milwaukee Bucks"
## [31] "Milwaukee Hawks"
## [32] "Minneapolis Lakers"
## [33] "Minnesota Timberwolves"
## [34] "New Jersey Nets"
## [35] "New Orleans Hornets"
## [36] "New Orleans Jazz"
## [37] "New Orleans Pelicans"
## [38] "New Orleans/Oklahoma City Hornets"
## [39] "New York Knicks"
## [40] "New York Nets"
## [41] "Oklahoma City Thunder"
## [42] "Orlando Magic"
## [45] "Phoenix Suns"
## [46] "Portland Trail Blazers"
## [47] "Rochester Royals"
## [48] "Sacramento Kings"
## [49] "San Antonio Spurs"
## [50] "San Diego Clippers"
## [51] "San Diego Rockets"
## [52] "San Francisco Warriors"
## [53] "Seattle SuperSonics"
## [54] "Sheboygan Red Skins"
## [55] "St. Louis Bombers"
## [56] "St. Louis Hawks"
## [57] "Syracuse Nationals"
## [58] "Toronto Raptors"
## [59] "Tri-Cities Blackhawks"
## [60] "Utah Jazz"
## [61] "Vancouver Grizzlies"
## [62] "Washington Bullets"
## [63] "Washington Capitols"
## [64] "Washington Wizards"
## [65] "Waterloo Hawks"

### Look at Us, Master's of This Data

Wow I never knew there was a team called the Chicago Stags.

I wonder if this arrogant troll knew that or could tell us how many official NBA teams there have been since 1949-50??

### Time Create Some New Variables

As fun as it is to explore the variables we already have in the data frame we need to move on to creating some of our own. One thing this data frame doesn't tell us is how many points each team scored during the course of the season. I also see a metric that captures the number of points per field goal attempt. Seems like a daunting task, it would be in Excel but we hate Excel and don't need it when we have nuclear weapons like R. Yet again there are countless ways to achieve both of these things in R but my favorite way is to use dplyr's mutate function.

all_nba_team_data %>>%
filter(!is.na(g)) %>>%
mutate(points_total = g * pts.g,
points_per_fga = points_total / fga) -> all_nba_team_data
all_nba_team_data %>>%
select(team,season,points_total, points_per_fga) #make sure it worked
## Source: local data frame [1,362 x 4]
##
##                      team    season points_total points_per_fga
## 1        Dallas Mavericks 2014-2015        963.9       1.276689
## 2  Portland Trail Blazers 2014-2015        948.6       1.224000
## 3         Toronto Raptors 2014-2015        948.6       1.278437
## 4            Phoenix Suns 2014-2015        838.4       1.243917
## 5   Golden State Warriors 2014-2015        838.4       1.332909
## 6          Boston Celtics 2014-2015        732.2       1.190569
## 7           Brooklyn Nets 2014-2015        831.2       1.272894
## 8        Sacramento Kings 2014-2015        934.2       1.326989
## 9           Chicago Bulls 2014-2015        932.4       1.307714
## 10        Houston Rockets 2014-2015        826.4       1.341558
## ..                    ...       ...          ...            ...

Cool like Joe Johnson at end of a game with the Brooklyn Nets looking for the win, but we went to all that work to add those new columns we should at least explore them a little. How about trying to find which team had the highest ever points per field goal attempt?

all_nba_team_data %>>%
filter(max(points_per_fga) == points_per_fga) %>>%
select(team, season_end, points_per_fga, playoff_team)
## Source: local data frame [1 x 4]
##
##        team season_end points_per_fga playoff_team
## 1 Utah Jazz       1995       1.376369         TRUE

Well, well, well those 1994-95 Utah Jazz. That was a pretty good team with superstars like Felton Spencer, John Crotty, Blue Edwards, Adam Keefe and this bad ass at age 35.

I don’t know, maybe that team had a few other decent players, just can’t think of them right now! This whole analysis thing has been fun and we just scratched the surface of what R can do but we came to here visualize, let’s do it.

### Visualize What You Can’t C

No better way to think about it than those lyrics from the late Tupac Shakur. We can see the data of course, all 44,946 pieces of it, but its hard to really see what’s going on in a tabular format. That, ladies and gentleman, is what data visualization is for.

### What Do We Visualize?

Before we get started visualizing we need to decide what exactly we want to explore. I keep hearing about this Daryl Morey character and how he is all into all these crazy things including data science. I also hear rumors that he is encouraging a game-plan centered around shooting tons of three pointers and even tried to formulate his roster around doing that.

Can data analysis tell us anything about this and the history of the three point shot? Of course it can.

### Step 1: Filter Down to the Three Point Era

First we need to create a data frame that includes only the seasons since the advent of the Three Point Shot Era. We could use Google to find the answer but we are learning we are data scientists so let’s use R to answer the question of what years cover this era. How can R do this? When we looked at the summary of our data frame we noticed that X3p., x3p and X3pa columns all contained NA values. Hmmm, maybe thats the answer, if we filter out NAs in one of those columns intuition says that should work.

all_nba_team_data %>>%
filter(!is.na(X3pa)) -> era3pt_shot #create the 3pt era data frame by filter  out NA remember that that symbol ! means NOT and is.na is a function to find NA values

### It Worked, JEAH

Looks like R got us the right answer and if you don’t trust R, this should do the trick.

We have one item before to take care of before getting to the best part and that’s adding a discrete column for the decade which we intend utilize to add some spice to our visualizations..

era3pt_shot$season_end %>>% #select season end cut(breaks = c(0,1989,1999,2009,2015), #use cut to tell it end of each decade labels = c('1980s','1990s','2000s','2010s') #label the new column ) -> era3pt_shot$decade

### Step 3B: Visualization Time

Alright we have what we need, its #DataViz time.

Let’s look at a colored Scatter Plot of Total Three Point Shot Attempts against Total Field Goal Attempts. Note since these variables are not calculated in real terms across the seasons [remember our 2015 season data is on going] we must filter the 2014-15 data in order to keep things consistent and not ruin the visualization.

### Time to Admire Our First Visualization

era3pt_shot %>>%
filter(!season_end == 2015) %>>% #filter out 2015
ggvis(x =~fga, y =~X3pa, fill=~factor(decade), shape=~factor(decade), fillOpacity := 0.65) %>>% #fga vs 3pt attempts colored by decade
layer_points() %>>% #ggvis nomenclature for scatter
add_axis("x", title = "Field Goal Attempts",
title_offset = 50) %>>% #fix x title
add_axis("y", title = "3PT Shot Attempts",
title_offset = 55) %>>% #fix y title
title = "Decade") #clean up the legend

Look at that beautiful ggvis scatter plot. What do we see here? Well first it looks like back in the 1980s and 1990s team’s on average used to take significantly more field goal attempts, we even see one insane out-lier, the absolutely DREADFUL 1990-91 Denver Nuggets who attempted an insane 8,668 field goals [bonus challenge to those up for it, try to use the 2 lines of dplyr I used to figure this out].

In addition to that it, looks like as we move through the decades, despite less overall field goal attempts, teams appear to be taking more 3 point shots. There are all sorts of factors that may have influenced this pattern outside of the magical powers of Daryl Morey, I don’t want to give it away but if you have time, try to investigate how the NBA has changed the Three Point Shot rules since its adoption in the 1979-80 season. Though we won’t get into it in this post, it looks like the data during the 1990s and the 2010s would be ripe for Clustering Analysis [hint there maybe actual clusters in what we see or these very apparent clusters data maybe the result of different external force].

This visualization is quite interesting and the more we look and think about it the more potential investigative topics may come across but this visualization isn’t fair, it's missing the 2015 season data and it isn't easy to see if there have been trends in how the 3 point shot progressed as the years have passed.

### Step 4: Figure Out A Way to Create a Real Variable that Compares All Season’s Equally

I know a way, lets add a new column that is an apples to apples comparison across all seasons since 1979-80 but what? Hmm… what are potential real variables that could be used to try to generate something like this? Two good options come to mind, minutes and games played. We could use either, but for the purposes of this next analysis let’s use minutes played. Since we are investigating three point shot attempts lets select that and divide by total minutes played.

era3pt_shot %>>%
mutate(X3pa_per_min = X3pa/mp) -> era3pt_shot #add the new variable

### Step 5: Time to Visualize It, the Iggy Azelea Way

Ok we have the variables we need let’s get to it. For this visualization we want to look at 3 Point Shot Attempts Per Minute by Season End.

We could do this visually in any number of ways but since I want to get fancy and showcase the power of easy regression analysis in ggvis thanks to the hard work of the man, the myth, the legend Hadley Wickham we are going to stick with a scatter plot and layer on a regression line.

era3pt_shot %>>%
ggvis(x =~season_end, y =~X3pa_per_min, fill = ~factor(decade), fillOpacity := 0.45) %>>%
add_axis("x", title = "", values = seq(from = 1980,to =2015,1), format="####", title_offset = 50,
properties = axis_props(
ticks = list(stroke = "black"),
majorTicks = list(strokeWidth = 2),
labels = list(
angle = 50,
fontSize = 11,
align = "left",
baseline = "middle",
dx = 3
))) %>>%
add_axis("y", title = "3PT Point Attempts Per Minute",
title_offset = 55, ticks = 20) %>>%
layer_model_predictions(model = 'lm', stroke =~factor(decade), se  = T) %>>%
properties = legend_props(
title = list(fontSize = 12),
labels = list(fontSize = 10, dx = 10),
symbol = list(stroke = "black", strokeWidth = 1,
size = 100)
))

#### Damn That 1362 Point Plot is FINE

Look at this amazing regression scatter plot highlighting every single 3 point shot attempt per minute since the introduction of the shot. It appears there was certainly a an upward trend from the 1980’s into the late 1990s and another starting in the mid 2000s through this season. There are a host of potential explanatory factors and other follow on analysis we could perform based on this viz, some which may be topics in forthcoming posts, but before we call it a day we want to look at, and visualize, one last item.

When looking at this graph it may be should be come apparent that, with only a few exceptions, each season’s leader in three point shot attempts appears to be higher than the prior season’s leader, signifying a potential trend. In fancy statistical words, this possible trend appears to display linearity. Appearances can deceive though and we need to thoroughly investigate this hypothesis in keeping with the data ninja code of mathematically proving the possibility of a relationship.

There are a number of ways to do this while adhering to this code, all of which R or any good programming language makes easy for you. In this case we are going to use the results of a linear model, the easiest of which to do in R uses the lm function.

### Step 6: Let’s Do It, Linear Regression That Is.

era3pt_shot %>>%
group_by(season_end) %>>% #group by season
filter(X3pa_per_min == max(X3pa_per_min)) %>>%  #take the max of each year
(lm(X3pa_per_min ~ season_end, data = .)) %>>% #apply lm against season
summary #let's look at the summary data to see the fit
##
## Call:
## lm(formula = X3pa_per_min ~ season_end, data = .)
##
## Residuals:
##       Min        1Q    Median        3Q       Max
## -0.019115 -0.006928 -0.000761  0.002920  0.032767
##
## Coefficients:
##               Estimate Std. Error t value Pr(>|t|)
## (Intercept) -6.5236007  0.3623104  -18.01   <2e-16 ***
## season_end   0.0033032  0.0001814   18.21   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.01131 on 34 degrees of freedom
## Multiple R-squared:  0.907,  Adjusted R-squared:  0.9043
## F-statistic: 331.7 on 1 and 34 DF,  p-value: < 2.2e-16

#### Good Job Intuition, We Were Right That’s a 90% Fit to the Data

Our instincts served us well. There is unquestionably a linear relationship over time that indicates that the team that leads the league in 3 point shot attempts per minute has been going up year by year. What this means for the future we can’t say definitively and I by no means would extrapolate that this trend will continue, but historically it has. Now for our final visualization let’s look at who the teams were that actually lead the league in three point shot attempts per minute were.

### Step 7: Create A Better “Real” Variable to Plot With

Before we do this, although I like our real statistic we created it is kind of hard to process what it is actually telling us, put it this way could we use this to explain this trend these basketball loving ladies?

Maybe, but I think we could do better. We know that there are 5 players on the court per team at a time, that there are a minimum of 4 quarters each of 12 minute length. That means, excluding overtime, there are 240 available minutes per game. So we can take our new variable, and multiply it by 240 to get a new version of it that contains the same information but per 240 minutes, essentially it is a roundabout that should nearly mimic three shot attempts per game if we were to calculate it. Let’s add our new variable and then create a new data frame containing only the top performing teams by season.

era3pt_shot %>>%
mutate(X3pa_per_240_min = X3pa_per_min * 240) %>>%
group_by(season_end) %>>%
filter(X3pa_per_min == max(X3pa_per_min)) %>>%
select(season_end, team, X3pa_per_240_min) -> top_teams

### Step 8: Lets Visualize These Teams

Now that we have data and new variable this let’s quickly visualize it, but since I am stickler for colors matching the entities they are associated with I want to find the right colors for each team. Fortunately I’ve already done this and we can just read in the file with this data and add in the 2 teams missing teams [the San Diego Clippers and the late Seattle Supersonics, RIP].

#Bring in Correct Colors
'http://asbcllc.com/data/NBA/team_colors.csv' %>>% read.csv %>>% data.frame %>>%
tbl_df -> active_colors
data.frame(team = top_teamsteam %>>% unique) %>>% tbl_df %>>% arrange((team)) -> teams teams %>>% merge(active_colors,all.x = T) -> teams '#EE2944' -> teams[14,2] #add the San Diego Clippers '#266A2E'-> teams[15,2] #add the Sonics We are now ready to create our final visualization. top_teams %>>% ggvis(x =~season_end, #season y =~X3pa_per_240_min,#by our 240 stat showi text := ~team, #plot the team name fill = ~team) %>>% #color the team add_axis("x", title = "", #no title values = seq(from = 1980,to =2015,1), #plot the years format="####", title_offset = 50, #format the axis properties = axis_props( ticks = list(stroke = "black"), majorTicks = list(strokeWidth = 2), labels = list( angle = 50, fontSize = 11, align = "left", baseline = "middle", dx = 3 ))) %>>% add_axis("y", title = "3PT Attempts Per 240 Available Minutes", format="####", #uses d3 axis format title_offset = 55, ticks = 20) %>>% #fix title and ticks layer_text() %>>% #adds the text scale_nominal(property = 'fill', #use our pretty colors domain = as.character(teamsteam),
range = as.character(teams\$primary_color)) %>>%
title = "Team", #need a pretty legend
properties = legend_props(
title = list(fontSize = 12),
labels = list(fontSize = 10, dx = 10),
symbol = list(stroke = "black", strokeWidth = 1,
size = 100)
))

Wow look at this amazing chart. This clearly validates the linearity we explored earlier and shows that in recent times this Daryl Morey guy has clearly pushed for and structured teams around taking lots of three point shots. What is even more interesting is that there appears to be another candy loving, last minute ETO signing, father of 8 that is nearly universally despised by Nets and Lakers fans whose presence on a roster, whether by design or chance, appears as a constant in all but 1 of the team’s that have lead the NBA in 3 point shot attempts per 240 minutes since the 2009-10 season. Any idea who he may be? Here’s a hint

## Wrapping It All Up, You Can Now Forever Be Down with L.A.V.

Well it looks like what we heard about Daryl Morey is true. He is optimizing his roster to do crazy things like taking insane amounts of threes. Will the strategy work and can the Rockets keep up this barrage of 3 point shots, only future us in late June know for sure, but if I were a betting man, I’d think both could happen but execution of this strategy lead to a championship? Again no one knows for sure, but looking at this data there were 2 teams that led the league in our 3 point shot attempts per 240 minutes who did win NBA titles. Who you ask? Look into it, but they played in a state home to Cowboys and were coached by a guy who nearly was killed during a game by the baddest of the bad power forwards to play the game, Kermit Washington.

We did alot today, we learned how to loop through a function, dug deeper into analyzing data and introduced the all powerful art of data visualization. We got down with L.A.V., something we should now be able anytime, anywhere as long as we have some data and

That wraps up today’s post. Here is the R source code and as always don’t hesitate to reach out to me with any questions or feedback on Twitter. I’ll be back soon with another post.


text:
You are currently browsing the tag archive for the ‘puzzle’ tag.

Among several things, these days, I have been doing some (serious) reading of literature on psychology, cognitive development, learning, linguistics, philosophy and a few other subjects. Well, the ones I just named happen to be parts of interdisciplinary areas, which are precisely the ones I am interested in. Of course, on many levels those parts also have a lot to do with mathematics, especially mathematical education. Ok, that was just a little background I wanted to provide for the content of today’s post.

I need to do a small (online) experiment in order to test a hypothesis, which will be the subject of my next post. Let me not reveal too much for now. The experiment is in the form of two puzzles that I ask readers (you!) to solve. They are both “multiple choice” puzzles with exactly two correct answers to each. Please bear in mind that this is NOT an IQ test. It is also not meant to test how good you are at solving puzzles individually. I am really interested in “aggregate” results. That is, for testing my hypothesis, I am only interested in what the majority thinks are the right answers. What is more, you won’t be graded, and no one (not even me) will ever know if you got the right answers. Please submit answers to both the puzzles.

Lastly, please don’t cheat or try to look for answers offline or online. As I said, this is NOT a test!

Let us now look at the puzzles.

Puzzle 1:

There are four cards, labelled either X or Y on one side and either 3 or 7 on the other. They are laid out in a row with their top (visible) sides shown like this: X Y 3 7. A rule states: “If X is on one side then there must be a 3 on the other.” Which two cards do you need to turn over to find out if this rule is true?

1) X

2) Y

3) 3

4) 7

Puzzle 2:

As you walk into a bar, you see a large sign that reads, “To drink alcohol here you must be over 18.” There are four people in the bar. You know the ages of two of them, and can see what the other two are drinking. The situation is: Alisa is drinking beer; Dymphna is drinking Coke; Maureen is 30 years old; Lauren is 16 years old. Which two people would you need to talk to in order to check that the “over 18 rule” for drinking alcohol is being followed?

1) Alisa

2) Dymphna

3) Maureen

4) Lauren

Note: I will keep this “poll” open for a week to collect as much data as possible. Thanks!

Update: It seems many readers weren’t aware of the short duration of the “poll” and that they would have very much liked to participate. So, I am extending the poll till Jan 7, 2010 for them.  (Doing so would also help me in collecting more data.)

Since the cat’s out of the bag and we’ve had some public discussion of our first Problem of the Week, I thought I’d officially kick things off with a new problem, this time under the ground rules discussed in our last post. This one comes from our regular reader John Smith, who wrote me this problem in email. It comes in two parts; here’s the first:

Consider a rectangle with sides of integer lengths $a$ and $b$, subdivided into $ab$ unit squares, and a diagonal line from corner to corner. Show that the number of unit squares that the line crosses is $a+b-\gcd(a,b)$. (Count only those cases where the line crosses the interior of a square.)

A little while later, John wrote me a follow-up question, which will be our part two:

What would be the $n$-dimensional analogue of the previous problem? How would you prove it?

Please submit your solutions through email to topological[dot]musings[At]gmail[dot]com, rendering the stuff in brackets in the obvious way; please do not reply in comments! Solutions should arrive by May 20 (11:59pm UTC time if you want to be exact). We will post one of the correct solutions (or otherwise our own), but all correct solutions will be acknowledged, and successful solvers will have their names entered into our Problem-Solving Hall of Fame. So get your answers in, and happy solving!

[Added May 15: if you just want to solve part one, that's fine -- credit will be awarded for correct solutions. In general for multi-part problems, we ask that you submit all parts you intend to solve in one mailing -- thanks.]

• 255,366 hits


text:
# The components of the space of projectively flat bundles over a Riemann surface

Recently, I'm reading the paper "Analytic structures on the space of flat vector bundles over a compact Riemann surface" by Gunning. In the introduction of this paper, Gunning says that the set $H^1(M,PL(n, \mathbb{C}))$ of projectively flat bundles over a compact Riemann surface $M$ has $n$ components, which can be put in one-to-one correspondence with one another. Then he claims that this property follows directly from the short exact sequence(the details are omitted) $$0 \rightarrow \mathbb{Z}_n \rightarrow SL(n, \mathbb{C}) \rightarrow PL(n, \mathbb{C}) \rightarrow 0,$$ where $\mathbb{Z}_n = \mathbb{Z}/(n\mathbb{Z})$.

I think we could obtain the following exact sequence of cohomology groups from the above one $$0 \rightarrow H^0(M,\mathbb{Z}_n) \rightarrow H^0(M, SL(n, \mathbb{C})) \rightarrow H^0(M,PL(n,\mathbb{C})) \rightarrow H^1(M,\mathbb{Z}_n) \rightarrow H^1(M, SL(n, \mathbb{C})) \rightarrow H^1(M,PL(n,\mathbb{C})) \rightarrow H^2(M,\mathbb{Z}_n)$$ since $\mathbb{Z}_n$ is in the center of $SL(n, \mathbb{C})$. From the connectness of $M$, we have (right?) $$H^0(M,\mathbb{Z}_n) = \mathbb{Z}_n, H^0(M, SL(n, \mathbb{C}))=SL(n, \mathbb{C}), H^0(M,PL(n,\mathbb{C}))=PL(n,\mathbb{C}), H^2(M,\mathbb{Z}_n)=\mathbb{Z}_n.$$

Therefore, we get an exact sequence $$0 \rightarrow H^1(M,\mathbb{Z}_n) \rightarrow H^1(M, SL(n, \mathbb{C})) \rightarrow H^1(M,PL(n,\mathbb{C})) \xrightarrow{\delta} \mathbb{Z}_n$$

I guess these $n$ components arise from the preimages of $\delta$. But how to prove

1. The map $\delta$ is surjective.
2. For any $a_1, a_2 \in \mathbb{Z}_n$, there exists a natural one-to-one correspondence between $\delta^{-1}(a_1)$ and $\delta^{-1}(a_2)$.

Furthermore, we know that for any projective bundle $\phi$ over a compact Riemann surface $M$, there always exists a holomorphic vector bundle $E$ such that $\phi = \mathbb{P}(E)$. Hence, if $\phi \in H^1(M,PL(n,\mathbb{C}))$, we can find a holomorphic vector bundle $E$ with $\phi = \mathbb{P}(E)$ over $M$. But we don't know whether $E$ is flat or not (or $\deg(E)$ is zero or not). I think

1. $\delta(\mathbb{P}(E)) = [\deg(E)] \in \mathbb{Z}_n$.

Who can help me prove them or give a counterexample? Thank you very much!

• Inside $\textbf{PGL}(n,\mathbb{C})$, consider the torus $T'$ that consists of equivalence classes of diagonal $n\times n$ matrices where the first $n-1$ diagonal entries are equal, say $s$, and the last diagonal entry can be different, say $t$. Via the map that sends such a matrix to $st^{-1}$, this torus is equivalent to $\mathbb{C}^\times$. The inverse image $T$ of $T'$ in $\textbf{SL}(n,\mathbb{C})$ is a copy of $\mathbb{C}^\times$ via the entry $s$, where now $t^{-1}$ equals $s^{n-1}$. Thus the map between tori is $s\mapsto s^n$ . . . – Jason Starr Jun 13 '17 at 11:37
• . . . Consider the long exact sequence of cohomology coming from the short exact sequence of Abelian groups, $$1 \to \mu_n \to T \to T' \to 1,$$ $$H^1(M,\mathbb{C}^\times)\xrightarrow{(-)^n} H^1(M,\mathbb{C}^\times) \xrightarrow{\delta} H^2(M,\mu_n).$$ Via first Chern classes, this is $$H^2(M,\mathbb{Z}(1)) \xrightarrow{n\cdot (-)} H^2(M,\mathbb{Z}(1)) \to H^2(M,\mathbb{Z}/n\mathbb{Z}(1)).$$ This is clearly surjective, since $H^3$ is zero. Starting from a $T'$-torsor whose first Chern class is not $n$-divisible, the associated $\textbf{PGL}(n,\mathbb{C})$-torsor generates $H^2(M,\mu_n)$. – Jason Starr Jun 13 '17 at 11:44
• @JasonStarr Thanks! Do you mean the weight zero Hodge structure by $H^2(M,\mathbb{Z}(1))$? I still have something confused in your argument. By your construction, we know that the exact sequences $$0 \rightarrow \mu_n \rightarrow T \rightarrow T' \rightarrow 0$$ and $$0 \rightarrow\mu_n \rightarrow \mathbb{C}^{\times} \xrightarrow{(-)^n} \mathbb{C}^{\times} \rightarrow 0$$ are equivalent. – swalker Jun 14 '17 at 8:40
• Note that in the sequence $H^1(M, \mathbb{C}^{\times}) \xrightarrow{(-)^n}H^1(M, \mathbb{C}^{\times}) \xrightarrow{\delta} H^2(M, \mu_n)$, the map $\delta = 0$, i.e. for any $L' \in H^1(M, \mathbb{C}^{\times}), \delta(L')=0$. As a consequence, the image of any $T'$-torsor is just one point in $\mu_n$. So how to construct the associated $\mathbf{PGL}(n,\mathbb{C})$-torsor and prove this torsor generates $H^2(M, \mu_n)$(It should use the last sequence to prove it, but how to use it?) And I want to know what gives the one-to-one correspondence between the components(Question 2). Thank you. – swalker Jun 14 '17 at 8:41
• The map $\delta$ is not the zero map. The self-map on $H^1(M,\mathbb{C}^\times)$ of raising to the $n^{\text{th}}$ power corresponds to multiplication by $n$ of the first Chern class. The multiplication by $n$ map is not surjective on $H^1(M,\mathbb{Z}(1))$. The raising to the $n^{\text{th}}$ power map is not surjective on $H^1(M,\mathbb{C}^\times)$. – Jason Starr Jun 14 '17 at 10:19

Edit. The OP has clarified that all of the sheaves are sheaves of locally constant functions, rather than sheaves of continuous functions or $C^\infty$ functions. In that case, it is better to consider principal bundles that are induced from an Abelian subgroup of the normalizer of the maximal torus that is not contained in the maximal torus. For instance, let $\Gamma'$ be the subgroup of $\textbf{PGL}(n,\mathbb{C})$ generated by the equivalence classes of the following two matrices, $$A=\left[\begin{array}{cccccc} 0 & 1 & 0 & \dots & 0 & 0\\ 0 & 0 & 1 & \dots & 0 & 0\\ 0 & 0 & 0 & \dots & 0 &0\\ \vdots & \vdots & \vdots & \ddots & \vdots & \vdots\\ 0 & 0 & 0 &\dots & 0 & 1 \\ 1 & 0 & 0 & \dots & 0 & 0 \end{array} \right], \ \ B=\left[\begin{array}{cccccc} 1 & 0 & 0 & \dots & 0 & 0\\ 0 & z & 0 & \dots & 0 & 0 \\ 0 & 0 & z^2 & \dots & 0 & 0 \\ \vdots & \vdots & \vdots & \ddots & \vdots & \vdots \\ 0 & 0 & 0 &\dots & z^{n-2} & 0 \\ 1 & 0 & 0 & \dots & 0 & z^{n-1} \end{array} \right],$$ where $z$ is a primitive $n^{\text{th}}$ root of unity. The group $\Gamma'$ is isomorphic to $(\mathbb{Z}/n\mathbb{Z})\times \mu_n$. The inverse image $\Gamma$ of $\Gamma'$ in $\textbf{SL}(n,\mathbb{C})$ is a non-Abelian group. It is a central extension of $\Gamma'$ by $\mu_n\cdot \text{Id}_{n\times n}$. The commutation relation is $ABA^{-1}B^{-1} = z\cdot \text{Id}_{n\times n}$. The central extension of finite groups, $$1 \to \mu_n\cdot \text{Id}_{n\times n} \to \Gamma \to (\mathbb{Z}/n\mathbb{Z})\times \mu_n \to 1,$$ gives a long exact sequence of non-Abelian cohomology with a connecting map, $$\delta:H^1(M;(\mathbb{Z}/n\mathbb{Z})\times \mu_n) \to H^2(M;\mu_n).$$ This connecting map is what is usually called the Weil pairing. There are many ways to construct the Weil pairing; the construction above is just one (probably not the best, since it is a little unclear how this behaves when you replace the integer $n$ by a multiple of $n$).

Because $\Gamma$ is a nilpotent group of nilpotency class $2$, every group homomorphism, $$\rho:\pi_1(M;\ast) \to \Gamma,$$ factors through the $2$-nilpotent quotient $\Pi(M)$ of $\pi_1(M;\ast)$. This fits into a central extension, $$0 \to \left(\bigwedge^2_{\mathbb{Z}} H_1(M;\mathbb{Z})\right)/H_2(M;\mathbb{Z})\to \Pi \to H_1(M;\mathbb{Z})\to 0,$$ where a generator $\gamma$ of $H_2(M;\mathbb{Z})$ maps to the usual relation in $\bigwedge^2H_1(M;\mathbb{Z})$, $$\gamma \mapsto \alpha_1\wedge \beta_1 + \dots + \alpha_g\wedge \beta_g$$ for the standard presentation, $$\pi_1(M,\ast) = \langle \alpha_1,\dots,\alpha_g,\beta_1,\dots,\beta_g: [\alpha_1,\beta_1]\cdots [\alpha_g,\beta_g] \rangle.$$

So now, for every integer $m$, consider the homomorphism $$\sigma_m: \Pi \to \Gamma',$$ $$\alpha_1 \mapsto [A], \ \beta_1 \mapsto [B^m], \ \alpha_{i>1} \mapsto [\text{Id}_{n\times n}], \ \beta_{j>1}\mapsto [\text{Id}_{n\times n}].$$ This maps $\gamma$ to $z^m\cdot \text{Id}_{n\times n}$. Thus, $\sigma_m$ lifts to a representation $\rho_m:\Pi \to \Gamma$ if and only if $m$ is divisible by $n$. When $m$ is not divisible by $m$, the commutator of $A$ and $B^m$ maps to $z^m\text{Id}_{n\times n}$. Thus the corresponding $\Gamma'$-principal bundle over $M$ corresponding to $\sigma_m$ has image under $\delta$ equal to $m$ times the generator of $H^2(M;\mu_n)$.

Since the long exact sequence in non-Abelian cohomology associated to a central extension is functorial, it follows that for the associated $\textbf{PGL}(n,\mathbb{C})$-principal bundle over $M$, the image under $\delta$ also equals $m$ times the generator. Thus, $\delta$ is surjective.

Original post. I will add some details. Let $M$ be a compact, closed, oriented, real manifold of real dimension $2$. Denote by $\textbf{SL}(n,\mathbb{C})$, resp. $\textbf{PGL}(n,\mathbb{C})$, the real Lie group of $n\times n$ complex matrices with determinant $1$, resp. equivalence classes of $n\times n$ complex matrices up to scaling.

Denote by $T'$ the Lie subgroup of $\textbf{PGL}(n,\mathbb{C})$ consisting of equivalence classes of diagonal $n\times n$ matrices such that the first $(n-1)$ diagonal entries are all equal, say $s$, and the last diagonal entry, $t$, need not equal $s$. Denote by $T$ the inverse image of this Lie subgroup in $\textbf{SL}(n,\mathbb{C})$. This is the set of the same type of matrices, but now under the restriction that $t^{-1}$ equals $s^{n-1}$. There is a short exact sequence of Lie groups, $$1 \to \mu_n \to T \to T' \to 1,$$ where $\mu_n$ denotes the finite subgroup of scalar multiples of the identity matrix by $n^{\text{th}}$ roots of $1$. The morphism $T\to T'$ is a submersion, and even a finite unbranched cover.

For every Lie group $G$, denote by $\underline{G}$ the corresponding sheaf on $M$ of $C^\infty$ functions from open subsets of $M$ to $G$ (it would be fine to work with continuous functions). Then the short exact sequence of Lie groups above gives a short exact sequence of sheaves of Abelian groups. That short exact sequence of sheaves of Abelian groups gives a long exact sequence of cohomology. The relevant portion of that sequence is as follows, $$H^1(M,\underline{T}) \to H^1(M,\underline{T'}) \xrightarrow{\delta} H^2(M,\mu_n).$$ Both $T$ and $T'$ are isomorphic to the Lie group $\mathbb{C}^\times$ via the isomorphisms, $$T\xrightarrow{\cong} \mathbb{C}^\times, \ \ \text{Diag}(s,\dots,s,s^{-(n-1)}) \mapsto s,$$ $$T'\xrightarrow{\cong} \mathbb{C}^\times, \ \ [\text{Diag}(s,\dots,s,t)] \mapsto st^{-1}.$$ Via these isomorphisms, both $H^1(M,\underline{T})$ and $H^1(M,\underline{T'})$ are isomorphic to $H^1(M,\underline{C}^\times)$. The diagram becomes, $$H^1(M,\underline{\mathbb{C}}^\times) \xrightarrow{(-)^n} H^1(M,\underline{\mathbb{C}}^\times) \xrightarrow{\delta} H^2(M,\mu_n).$$ Using the exponential sequence of Lie groups, $$0 \to \mathbb{Z}(1) \to \mathbb{C} \xrightarrow{\text{Exp}} \mathbb{C}^\times \to 0,$$ there is a long exact sequence of cohomology. The connecting maps of that long exact sequence give a commutative diagram, $$\begin{array}{ccccc} H^1(M,\underline{\mathbb{C}}^\times) &\xrightarrow{(-)^n} & H^1(M,\underline{\mathbb{C}}^\times) & \xrightarrow{\delta} & H^2(M,\mu_n) \\ \downarrow & & \downarrow & & \downarrow \\ H^2(M,\mathbb{Z}) &\xrightarrow{n(-)} & H^2(M,\mathbb{Z}(1)) & \xrightarrow{} & H^2(M,\mathbb{Z}/n\mathbb{Z}(1)) \end{array}.$$ The last vertical map is an isomorphism; the others are the first Chern class maps. The first Chern class maps are surjective. Finally, because $H^3(M,\mathbb{Z}(1))$ is zero, it follows that for every element of $H^2(M,\mu_n)$, there exists a principal bundle for $T'$ whose associated class in $H^1(M,\underline{T'})$ maps to the specified element of $H^2(M,\mu_n)$. Since $T'$ is a Lie subgroup of $\textbf{PGL}(n,\mathbb{C})$, there is an associated principal bundle for $\textbf{PGL}(n,\mathbb{C})$. By the functoriality of non-Abelian cohomology, this principal bundle for $\textbf{PGL}(n,\mathbb{C})$ has the same image in $H^2(M,\mu_n)$.

Finally, the long exact sequence of non-Abelian cohomology is part of the basic setup of non-Abelian cohomology. In general it is not true that there is any canonical isomorphism of the fibers of the connecting map with the pointed set preceding in the sequence. What is true is that there is a "twisting" construction. There is an excellent discussion of this in Serre's "Galois cohomology". You can also find some notes on my webpage: http://www.math.stonybrook.edu/~jstarr/papers/Escola_07_08d_nocomments.pdf

• I apologize for my ambiguous notation, I probably should give the precise definitions for ${\Bbb C}^{\times}, SL(n,{\Bbb C})\cdots$ at first. Here ${\Bbb C}^{\times}$ denotes the constant sheaf, i.e. a sheaf of constant maps from open subsets to ${\Bbb C}^{\times}$. And $H^1(M, {\Bbb C}^{\times})$ is the set of all flat complex line bundles. The definitions of $SL(n,{\Bbb C})$ and $H^1(M, SL(n,{\Bbb C}))$ are similar. My first question is for any $\beta \in H^1(M, {\Bbb Z}_n)$, can we find a holomorphic vector bundle $E$, which is projectively flat, such that $\delta({\Bbb P}(E)) = \beta$. – swalker Jun 15 '17 at 13:19


text:
# Creating a Simple CS Signal: Rectangular-Pulse BPSK

We’ll use this simple textbook signal throughout the CSP Blog to illustrate and tie together all the different aspects of CSP.

To test the correctness of various CSP estimators, we need a sampled signal with known cyclostationary parameters. Additionally, the signal should be easy to create and understand. A good candidate for this kind of signal is the binary phase-shift keyed (BPSK) signal with rectangular pulse function.

PSK signals with rectangular pulse functions have infinite bandwidth because the signal bandwidth is determined by the Fourier transform of the pulse, which is a sinc() function for the rectangular pulse. So the rectangular pulse is not terribly practical–infinite bandwidth is bad for other users of the spectrum. However, it is easy to generate, and its statistical properties are known.

So let’s jump in. The baseband BPSK signal is simply a sequence of binary ($\pm$ 1) symbols convolved with the rectangular pulse. The MATLAB script make_rect_bpsk.m does this and produces the following plot:

The signal alternates between amplitudes of +1 and -1 randomly. After frequency shifting and adding white Gaussian noise, we obtain the power spectrum estimate:

The power spectrum plot shows why the rectangular-pulse BPSK signal is not popular in practice. The range of frequencies for which the signal possesses non-zero average power is infinite, so it will interfere with signals “nearby” in frequency. However, it is a good signal for us to use as a test input in all of our CSP algorithms and estimators.

The MATLAB script that creates the BPSK signal and the plots above is here. It is an m-file but I’ve stored it in a .doc file due to WordPress limitations I can’t yet get around.

I'm a signal processing researcher specializing in cyclostationary signal processing (CSP) for communication signals. I hope to use this blog to help others with their cyclo-projects and to learn more about how CSP is being used and extended worldwide.

## 2 thoughts on “Creating a Simple CS Signal: Rectangular-Pulse BPSK”

1. Amitoz says:

Hello,
Thanks for creating this blog.
I am supposed to work on applying deep learning methods for denoising or separating cyclostationary signals. My background is mainly in machine learning, image processing and computer vision. Since I am not from Digital Signal Processing or Digital Communication background, It would be great if you can recommend a or few resources to grasp the basics of these first.

1. Thank you for your interest in the CSP Blog!

I suppose since your background contains image processing, you do actually have a lot of the basics of DSP for communication-signal processing, such as transform theory, filtering/convolution, and perhaps even spectrum estimation. So I’m thinking what you are currently lacking is more in the areas of mathematical modeling of communication signals and the theory and practice of cyclostationary signal processing.

On CSP, check out my papers in My Papers and, perhaps more importantly, the papers on CSP in The Literature. You might also read/skim all the posts here at the CSP Blog, in their order of appearance (because I start out slowly and build up to more advanced topics).

For communication-signal modeling, a good textbook is Proakis (The Literature [R44]). I’m sure there are online things too. Perhaps Khan Academy?

Finally, for full disclosure, see evidence of my current attitude toward the Machine Learners and their foray into CSP: