text:
# Berry-Esseen type bounds with lower moment assumptions

Let $D(\epsilon,C)$ be the collection of all random variables $X$ on $\mathbb{R}$ such that $E[X]=0$, $E[X^2]=1$, and $E[|X|^{2+\epsilon}]\leq C$. Define a function $L_{\epsilon,C}(n)$ by $$L_{\epsilon,C}(n) = \sup_{X\in D(\epsilon,C)} \sup_{x\in\mathbb{R}} \left\vert P\left(\frac{X_1+X_2+\ldots+X_n}{\sqrt{n}}<x\right)-P\left(Z < x\right)\right\vert$$ where $X_i$ are independent copies of $X$, and $Z$ is a standard normal.

Having made these definitions, the Berry-Esseen theorem tells us that $$L_{1,C}(n) = O(C n^{-1/2})$$ It is also not hard to use the Lindeberg-Feller theorem to show that for any fixed $\epsilon>0$ and $C\in \mathbb{R}$ $$\lim_{n\to\infty} L_{\epsilon,C}(n) = 0$$ Which leads to my question -- can we get any more quantitative result than that this goes to zero, without assuming $\epsilon=1$? Or, in the other direction, is it known that we can't?

$\newcommand{\ep}{\epsilon}$
To answer your question, one can use e.g. a result by Bikelis, which states the following: Let $X_1,\dots,X_n$ be independent zero-mean random variables such that $E|X_i|^{2+\ep}<\infty$ for some $\epsilon\in[0,1]$ and all $i$. Then for all real $x$ $$\Big|P\Big(\sum_1^n X_i<x\sqrt B_n\Big)-P(Z<x)\Big| \le \frac A{B_n^{1+\ep/2}(1+|x|)^{2+\ep}}\,\sum_1^n E|X_i|^{2+\ep},$$ where $A$ is a finite universal constant, $B_n:=\sum_1^n EX_i^2$, and $Z$ is a standard normal random variable.
In you specific case, it follows that $L_{\epsilon,C}\le AC/n^{\epsilon/2}$.

format: plain


text:
View Single Post
 2021-02-27, 07:39 #1215 sweety439   Nov 2016 2,819 Posts Riesel base 185 Code: 1,0 2,2 3,1 4,1 5,4 6,1 7,1 8,8 9,(partial algebra factors) 10,6783 11,4 12,8 13,1 14,4 15,2 16,3 With CK=17 All k where k = m^2 and m = = 3 or 5 mod 8: for even n let k = m^2 and let n = 2*q; factors to: (m*185^q - 1) * (m*185^q + 1) odd n: factor of 2 This includes k = 9 k=10 prime given by CRUS k = 1 remain at n=66337, see post https://mersenneforum.org/showpost.p...&postcount=225


text:
# A 2 m long wire of resistance 4 Omega and diameter 0.64 mm is coated with plastic insulation of thickness 0.66 mm. A current of 5A flows through the

0 votes
34 views
in Physics
A 2 m long wire of resistance 4 Omega and diameter 0.64 mm is coated with plastic insulation of thickness 0.66 mm. A current of 5A flows through the wire. Find the temperature difference across the insulation in the steady state. Thermal conductivity of plastic is 0.16 xx (10^-2) cal//s cm.^@C.

## 1 Answer

0 votes
by (24.2k points)
Correct Answer - 4.7^(@)C

0 votes
1 answer
0 votes
1 answer
0 votes
1 answer
0 votes
1 answer
0 votes
1 answer


text:
## July 5, 2010

### Day on Higher Category Theory in Utrecht

#### Posted by Urs Schreiber

A small workshop on higher category theory takes place

Tuesday, July 13, 2010, beginning 10am, in Utrecht.

For details see the webpage Day on Higher Category Theory - July 2010.

The talks are

If you want to participate or have any questions, please drop me a note, here or by email.

Posted at July 5, 2010 4:57 PM UTC

TrackBack URL for this Entry:   http://golem.ph.utexas.edu/cgi-bin/MT-3.0/dxy-tb.fcgi/2238

### Re: Day on Higher Category Theory in Utrecht

We added one more talk at short notice to the program:

Posted by: Urs Schreiber on July 9, 2010 4:27 PM | Permalink | Reply to this

### Re: Day on Higher Category Theory in Utrecht

There are now pdf slides available for some (most) of the talks. The links can be found on the workshop website.

Posted by: Urs Schreiber on July 19, 2010 10:37 AM | Permalink | Reply to this

Post a New Comment


text:
# Converting N-S example to use a mixed space

As an exercise in troubleshooting a more complex variational problem which I’m attempting to solve with fenics, I am trying to convert this incompressible N-S example to use a mixed function space to solve for velocity and pressure simultaneously, rather than using Chorin’s method.

I replaced the function space definitions with

# Define function spaces (P2-P1)
P2 = VectorElement("P", mesh.ufl_cell(), 2)
P1 = FiniteElement("P", mesh.ufl_cell(), 1)
TH = MixedElement([P2, P1])
W = FunctionSpace(mesh, TH)

# Define trial and test functions
w = TrialFunction(W)
u, p = split(w)

w = TestFunction(W)
v, q = split(w)

as well as the function definitions:

w0 = Function(W)
u0, p0 = split(w0)
w1 = Function(W)
u1, p1 = split(w1)

The weak formulation I’m attempting to implement is stated as
$$\begin{cases} (D u_h^n, \vec{v}_h) + (\mathcal{I}(\vec{u}_h^n) \cdot \nabla \vec{u}_h^n, \vec{v}_h) + \frac{1}{Re}(\nabla \vec{u}_h^n, \nabla \vec{v}_h) - (p_{h}, \nabla \cdot \vec{v}_{h}) = (\vec{f}^n, \vec{v}_h) & \forall \vec{v}_h \in \vec{X}_{h} \\ %\label{eq:serial3}\\ (\nabla \cdot \vec{u}_h^n, r_h) = 0 & \forall r_h \in Q_{h} \end{cases}$$
where D is the discrete time derivative and \mathcal{I} is an interpolation operator; for the moment I’m using the previous timestep.

I couldn’t figure out how to solve these equations simultaneously, so I combined them in my implementation:

F1 = (1/k)*inner(u - u0, v)*dx + inner(grad(u0)*u, v)*dx + \
a1 = lhs(F1)
L1 = rhs(F1)

I also changed the domain of the boundary conditions to use W.sub(0), etc. as well as the solve and assign in the time-stepping loop to use the combined vector, e.g.:

solve(A1, w1.vector(), b1, "gmres", "default")
...
w0.assign(w1)

When I try to execute the solver, my forms compile ok, but I get this error on the very first iteration:

*** -------------------------------------------------------------------------
*** DOLFIN encountered an error. If you are not able to resolve this issue
*** using the information listed below, you can ask for help at
***
***
*** Remember to include the error message listed below and, if possible,
*** include a *minimal* running example to reproduce the error.
***
*** -------------------------------------------------------------------------
*** Error:   Unable to solve linear system using PETSc Krylov solver.
*** Reason:  Solution failed to converge in 0 iterations (PETSc reason DIVERGED_PC_FAILED, residual norm ||r|| = 0.000000e+00).
*** Where:   This error was encountered inside PETScKrylovSolver.cpp.
*** Process: 0
***
*** DOLFIN version: 2019.1.0
*** Git changeset:
*** -------------------------------------------------------------------------

Note that this is using the GMRES solver like the original example. If I remove “gmres” and use the default solver, I see no apparent update in the solution.

Can anyone point out anything I’m doing that’s obviously wrong? Other examples of solving coupled problems, involving N-S in particular, might also be helpful.

Yes, it’s not possible to use GMRES to solve a saddle point problem without a suitable (block) preconditioner. Even then it’s a complicated problem. Better use direct solvers until you’re sure that your code runs fine.

You’d have to provide a minimal working example for us to see what’s wrong with your code using direct solvers.

For inspiration, check out:

https://fenicsproject.org/docs/dolfin/latest/python/demos/stokes-iterative/demo_stokes-iterative.py.html

Introduction to iterative solvers for saddle point problems: https://doi.org/10.1017/S0962492904000212

Thank you for the references, the paper in particular looks like it will be quite helpful.

My full code follows, though as I noted above I have attempted to keep it as similar to the original example as possible:

from dolfin import *

# Print log messages only from the root process in parallel
parameters["std_out_all_processes"] = False;

mesh = Mesh("lshape.xml")

# Define function spaces (P2-P1)
P2 = VectorElement("P", mesh.ufl_cell(), 2)
P1 = FiniteElement("P", mesh.ufl_cell(), 1)
TH = MixedElement([P2, P1])
W = FunctionSpace(mesh, TH)

# Define trial and test functions
w = TrialFunction(W)
u, p = split(w)

w = TestFunction(W)
v, q = split(w)

# Set parameter values
dt = 0.01
T = 3
nu = 0.01

# Define time-dependent pressure boundary condition
p_in = Expression("sin(3.0*t)", t=0.0, degree=2)

# Define boundary conditions
noslip  = DirichletBC(W.sub(0), (0, 0),
"on_boundary && \
(x[0] < DOLFIN_EPS | x[1] < DOLFIN_EPS | \
(x[0] > 0.5 - DOLFIN_EPS && x[1] > 0.5 - DOLFIN_EPS))")
inflow  = DirichletBC(W.sub(1), p_in, "x[1] > 1.0 - DOLFIN_EPS")
outflow = DirichletBC(W.sub(1), 0, "x[0] > 1.0 - DOLFIN_EPS")
bcu = [noslip]
bcp = [inflow, outflow]

# Create functions
w0 = Function(W)
u0, p0 = split(w0)  # the previous timestep solution
w1 = Function(W)
u1, p1 = split(w1)  # current pressure solution
#w1 = Function(W)
#u1, p1 = split(w1)  # current pressure solution

# Define coefficients
k = Constant(dt)
f = Constant((0, 0))

# velocity step
F1 = (1/k)*inner(u - u0, v)*dx + inner(grad(u0)*u, v)*dx + \
a1 = lhs(F1)
L1 = rhs(F1)

# Assemble matrices
A1 = assemble(a1)

# Use amg preconditioner if available
prec = "amg" if has_krylov_solver_preconditioner("amg") else "default"

# Create files for storing solution
ufile = File("results/velocity.pvd")
pfile = File("results/pressure.pvd")

# Time-stepping
t = dt
while t < T + DOLFIN_EPS:

# Update pressure boundary condition
p_in.t = t

# Compute tentative velocity step
begin("Computing velocity and pressure")
b1 = assemble(L1)
[bc.apply(A1, b1) for bc in bcu]
solve(A1, w1.vector(), b1) #, "gmres", "default")
end()

#u1, p1 = split(w1)

# Plot solution
plot(p1, title="Pressure", rescale=True)
plot(u1, title="Velocity", rescale=True)

# Save to file
#ufile << u1
#pfile << p1

# Move to next time step
w0.assign(w1)
t += dt
print("t =", t)

# Hold plot
import matplotlib.pyplot as plt
plt.show()

You have to reassemble A1 at every time step, since a1 depends on u0.

I added an assemble for A1, and it seems to have made no difference:

    begin("Computing velocity and pressure")
b1 = assemble(L1)
A1 = assemble(a1)
[bc.apply(A1, b1) for bc in bcu]
solve(A1, w1.vector(), b1) #, "gmres", "default")
end()

Is there anything else obvious I’m missing?

You are not applying the pressure conditions (bcp) to your problem which is supposed to drive your flow.

I’m going to be kicking myself about that oversight – I overlooked those b.c. when attempting to adapt the example.

So after correcting the pressure boundary conditions, I’m seeing the solution change in time, which is progress. On the other hand, the solution seems to be pretty poor. I attempted to add a zero-mean pressure conditions by inserting

    u1, p1 = split(w1)

# enforce zero-mean pressure condition
p1 = p1 - assemble(p1 * dx)

into the time-stepping, but it had no noticeable effect. I will try playing with the interpolation operator, but are there any other suggestions on how to improve this direct scheme?

You are using a poor linearization, inner(grad(u0)*u. Have you considered solving the non-linear problem with a NewtonSolver?

I was able to obtain a solution using a NewtonSolver as you suggested, with some significant instability in the pressure near the inlet:

I had great difficulty getting the newton iterations to converge with the problem as originally stated – specifically the inlet pressure condition that oscillates between positive and negative. I modified the problem to oscillate between a positive pressure and 0 instead to get the result pictured.

Are there other ways to improve the quality of the solution using a coupled solve as I’m trying to do here? I have tried enforcing a zero mean-pressure conditions like so:

pdofs = W.sub(1).dofmap()
w.vector()[pdofs.dofs()] -= assemble(p1 * dx)

But I’m not at all sure that’s correct. I intend to try a better linearization as well and see how that fares.

I do not think that this is the correct condition to apply, as this is only used when the pressure can only be determined up to a constant. Since you use pressure Dirichlet conditions, this is not the case. However, due to the use of pressure driven flow, you need to do integration by parts carefully, as the common boundary condition related to pressure is du/dn=pn.
I will attach what I have found to be a very good source on Navier-Stokes.


text:
# Projective texturing in HLSL (water reflection)

This topic is 4601 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

Hey, I want to project a texture to create a water reflection and have managed to do this with the FFP. But now I want to manipulate the pixels (with a pixelshader) on the water surface to make it look like its moving (so it looks like real water). Is it so, that if I want to use a pixelshader I also need to project my texture in a vertexshader? (I must forget about my working FFP projective texture?.. or can I do it any other way?). I don't really understand how to project a texture using a vertexshader in HLSL. What values does my vertexshader need? What does it look like? If anyone could show me a sample of projective texturing in HLSL it would be great. I have searched the forums but haven't find any good ones that ONLY shows projective texturing in HLSL... Thanks in advance!

##### Share on other sites
here you can find all the information you will need:

Article on water

Regards,
Kenzo

##### Share on other sites
this can be acheived easily enough:

first mul your Input vertex postion with the worldspace projection matrix
and set the value to a float4 called pPos

then, having defined an output float3 called 'project', output these values:
Out.project.x = 0.5 * (pPos.w + pPos.x);
Out.project.y = 0.5 * (pPos.w - pPos.y);
Out.projectr.z = pPos.w;

float2 projcoord=(In.project.xy / In.project.z);

then just sample your texture like:
float4 reflect=Tex2d(reflectsampler,projcoord);

or you can offet these coords with a normal map or whatever to make waves..just sample a normal map:

EDIT: use regular texture coords for the bump map
float4 bump=Tex2d(bumpsampler,In.Tex.xy);
float2 wavecoords = projcoord + (bump.xy * 0.2f) ; //bias it with a scale
float4 reflect=Tex2d(reflectsampler,wavecoords);

[Edited by - Matt Aufderheide on May 8, 2006 6:28:36 AM]

##### Share on other sites
Thanks for the reply. I couldn't get it to work... This is what my shader looks like now:
float4x4 worldViewProj : WorldViewProjection;float4x4 world   : World;float4x4 worldInverseTranspose : WorldInverseTranspose;float4x4 viewInverse : ViewInverse;texture diffuseTexture : Diffuse<	string ResourceName = "default_color.dds";>;struct vertexInput {    float3 position				: POSITION;    float3 normal				: NORMAL;    float4 diffuseColor			: COLOR0;    float4 texCoordDiffuse		: TEXCOORD0;};struct vertexOutput {	float4 pos : POSITION;    float4 project  : TEXCOORD0;};vertexOutput VS_TransformAndTexture(vertexInput IN) {	vertexOutput OUT;		OUT.pos = mul(float4(IN.position.xyz , 1.0), worldViewProj);		float4 pPos = mul(IN.position, worldViewProj);         OUT.project.x = 0.5 * (pPos.w + pPos.x);	OUT.project.y = 0.5 * (pPos.w - pPos.y);	OUT.project.z = pPos.w;	OUT.project.w = 0; // ??    return OUT;}sampler TextureSampler = sampler_state {    texture = <diffuseTexture>;};float4 PS_Textured( vertexOutput IN): COLOR{	float2 projcoord = (IN.project.xy / IN.project.z);	float4 reflect = tex2D(TextureSampler, projcoord);	return reflect;}technique textured{    pass p0     {				VertexShader = compile vs_2_0 VS_TransformAndTexture();		PixelShader  = compile ps_2_0 PS_Textured();    }}

In my code I'm setting the values like this:
effect.SetTechnique("textured");effect.SetValue("diffuseTexture", billboard.Texture[0]);effect.SetValue("worldViewProj", Screen.Device.Transform.World * Screen.Device.Transform.View * Screen.Device.Transform.Projection);

I'm not that good at shaders yet, maybe you can explain whats wrong with it?

##### Share on other sites
Quote:
 Original post by kobingo vertexOutput OUT; OUT.pos = mul(float4(IN.position.xyz , 1.0), worldViewProj); float4 pPos = mul(IN.position, worldViewProj); OUT.project.x = 0.5 * (pPos.w + pPos.x); OUT.project.y = 0.5 * (pPos.w - pPos.y); OUT.project.z = pPos.w; OUT.project.w = 0; // ??

I don´t know if this is the source of your problem, but OUT.pos is already what Matt referred to as pPos. Furthermore I´m not sure whether
float4 pPos = mul(IN.position, worldViewProj)

would use 0 or 1 as w-coordinate, but it will definitely assume a w-coordinate, as you need a 4-dimensional vector to perform that multiplication and IN.position is only a 3-dimensional vector.

So I would just use the calculated OUT.pos to calculate OUT.project, because I think the w-coordinate in that multiplication should be 1. I´d change that code snippet to:
	vertexOutput OUT;		OUT.pos = mul(float4(IN.position.xyz , 1.0), worldViewProj);		//      float4 pPos = mul(IN.position, worldViewProj);             OUT.project.x = 0.5 * (OUT.pos.w + OUT.pos.x);	OUT.project.y = 0.5 * (OUT.pos.w - OUT.pos.y);	OUT.project.z = OUT.pos.w;

OUT.project should only be float3, as there is no need for the fourth component.

Hope that helps.
Good luck!

##### Share on other sites
Matches is correct..and normally I would just define the input position as a float4 right at the start to avoid any goofiness..

so you just do this:
{//inputs
float4 inPos: POSITION,
....
}

float4 pPos=mul(inPos,matWorldViewProj);
Out.pos=pPos;

Out.project.x = 0.5 * (pPos.w + pPos.x);
Out.project.y = 0.5 * (pPos.w - pPos.y);
Out.project.z = pPos.w;

or of ourse you can operate using the Out.pos if you want, but i dont like operating on my outs for some reason..

(btw see my edit for the bump map sampling..)

##### Share on other sites
Thanks alot, that worked! :-) Now I must experiment with the pixelshader to get it to look like waves...

##### Share on other sites
float4 bump=tex2D(NormalSampler,IN.project.xy);float2 wavecoords = projcoord + (bump.xy * 0.2f) ; //bias it with a scalereflect=tex2D(TextureSampler,wavecoords);

I got some weird effects, parts of the texture look correct while others didn't. It seemed like it wasn't really projected correctly anymore... Any ideas?

Also, what do you mean by "bias it with a scale"?

##### Share on other sites
When I change the line:
float4 bump=tex2D(NormalSampler,IN.project.xy);
to this:
float4 bump=tex2D(NormalSampler,projcoord);

I looks a little bit better, but still a bit weird... Can anyone spot the error?

1. 1
2. 2
3. 3
Rutin
15
4. 4
5. 5

• 10
• 9
• 9
• 11
• 11
• ### Forum Statistics

• Total Topics
633687
• Total Posts
3013327
×


text:
# Breaking Horizontal Rule Running Across Adjacent Columns in a Table

Is there any way to force a break in the horizontal rule running across two adjacent cells of a table. To be more precise, here is what I'm looking for:

Notice how I wish the lines to fall short of the left and right borders of the table as well. I would really appreciate it if someone can provide a simple and compact solution.

This is the usual scenario for comparing against:

Here is the simple code for quick access:

\begin{tabular}{ccccc}
\hline
\multirow{2}{*}{C1} & \multicolumn{2}{c}{C2} & \multicolumn{2}{c}{C3} \\ \cline{2-3} \cline{4-5}
& C21 & C22              & C31 & C32            \\ \hline
Val1                & Val2 & Val3            & Val4 & Val5 \\

\end{tabular}

-
Please add a minimal working example. –  Svend Tveskæg Apr 6 '13 at 6:12
While code snippets are useful in explanations, it is always best to compose a fully compilable MWE that illustrates the problem including the \documentclass and the appropriate packages so that those trying to help don't have to recreate it. –  Peter Grill Apr 6 '13 at 6:48
@PeterGrill - Will keep that in mind from now on. Thank you very much –  Chatterjee Apr 6 '13 at 6:56

You could use the booktabs package:

## Code:

\documentclass{article}
\usepackage{multirow}
\usepackage{booktabs}

\begin{document}
\begin{tabular}{ccccc}
\hline
\multirow{2}{*}{C1} & \multicolumn{2}{c}{C2} & \multicolumn{2}{c}{C3} \\
\cmidrule(lr){2-3} \cmidrule(lr){4-5}
& C21 & C22              & C31 & C32            \\
\cmidrule(lr){1-1} \cmidrule(lr){2-3} \cmidrule(lr){4-5}
Val1                & Val2 & Val3            & Val4 & Val5 \\

\end{tabular}
\end{document}

-


text:
Acorns from 19 red oak (Quercus rubra L.) trees were collected in fall 1988. Whips were produced in 1989 and field planted in 1990. Mother tree identity was maintained throughout the study. Growth during the whip production phase was rapid; the average height growth for the 19 families was 122 cm (approximately 4 ft). The best family averaged 167 cm (5.5 ft), the worst 73 cm (2.5 ft) with average individual whip values of $11.20 and$2.80, respectively. When transplanted to nursery fields, the whips had high survival and rapid growth rates. Average individual tree value at the end of three growing seasons ranged from $38.60 to$11.20 for families 6 and 12, respectively. All the traits measured in this study: height, caliper, % acceptable and individual tree value, have relatively high heritability estimates, indicating that traits are under genetic control and thus subject to manipulation by nursery mangers and breeders. The family that produced the highest value whips, did not produce the highest value finish stock. Average finish stock value for a family was determined more by the percentage of the acceptable trees (a measure of quality), than by plant height or caliper (measures of growth).

This content is only available as a PDF.

## Author notes

Published as contribution 172-93 of the Ohio Agriculture Research and Development Center, The Ohio State University, Columbus, OH.

2Associate Professor.

3Associate Professor, Department of Forestry, North Carolina State University, Raleigh, NC 27695-8002.


text:
# Exothermic thermodynamics and data table

Bio-diesel and thermodynamics last updated save as pdf exothermic reactions are the reactions that release energy multiply the average bond energy (table 1 . Thermodynamics the study of the transformations of energy from one form into another exothermic: ∆h 0 table 81 entropy the . Solute to predict if an enthalpy change of solution will be exothermic or endothermic introduction thermodynamics you must determine what type of data. Thermodynamics-enthalpy of reaction and hess'l law page 5 c w-8 data tables part i data table determination of the heat capacity of the calorimeter initial temperature (c). Their thermodynamic surroundings the laboratory measurement of the heat energy changes that record temperature values in your data table every 15 seconds wait .

Thermodynamics is the study of energy changes accompanying physical and chemical changes the term itself clearly suggests what is happening -- thermo, from temperature, meaning energy, and dynamics, which means the change over time. Heat of solution data for aqueous solutions solute products heat of solution exothermic ch 2o 2(l) (−57 kj/mol of heat of solution). Use the thermodynamic data table on the following page to help you calculate and fill out table 1 write the salt/ppt equation for each precipitation reaction indicated agcl(s) formation is. Thermodynamic favorability entropy and free energy using the following standard enthalpy of reaction data, 2c(s) + 3h 2 (g) → c 2 h 6 tables of standard .

Define: enthalpy, entropy, heat of fusion, heat of vaporization, hess's law, exothermic, endothermic know the first, second, and third laws of thermodynamics calculate the heat required to heat water in all 3 phases, and between phases. 0504 endothermic and exothermic reactions lab report directions: complete the laboratory activity carefully, filling in the data table and answering the questions below copy and paste this page into a document and save your fi. (d) use the data in the table below to calculate the value of the standard molar entropy, s° for o2(g) at 25°c standard molar entropy, s ° (j k -1 mol -1 ).

In thermodynamics, these two types of reactions are classified as exothermic or endothermic, respectively an easy way to remember the difference between these two reaction types is by their prefixes: endo- means to draw in, and exo- means to give off. Exothermic reactions are essentially reactions that release energy as heat (in part, at least) to the surroundings, and may cause an explosion or combustion, depending on the amount of heat release. For exothermic reactions, (h is negative, whereas (h is positive for endothermic reactions a negative (h indicates the products are at a lower energy state than the reactants a positive (h indicates the products are at a higher energy state than the reactants. Table of contents 2017 | volume 8, issue 2 and querying of data and content of the open access journal through the online mode thermodynamics & catalysis . The release of heat means that this is an exothermic reaction the sign of the enthalpy of the reaction reveals the direction of heat flow data in table 3 to .

Periodic table of the elements thermodynamics by looking at whether q is exothermic or endothermic we can determine a relationship between $$\delta h$$ and . Thermodynamics there always seems to be at least one free response question that involves in the table below are data that show the percent of co in the . Thermodynamics - enthalpy of reaction and hess's law the heat transfers for exothermic reactions using a calorimeter, most of the heat released is absorbed by the . Journal of thermodynamics and catalysis discusses the latest research innovations and important developments in this field contain data relating to .

## Exothermic thermodynamics and data table

In high school i learned that an exothermic reactions releases energy, while an endothermic reaction needs energy to occur now i learned that there is a separate, somewhat similar classification s. As the water heats, record the temperature at which you observe bubbles, steam, and true boiling in data table c 11 once boiling begins, read the temperature every 30 seconds for three minutes. Exothermic typical units for δh° are j/mol or exp 15 thermodynamics of the solution process data and results fill in the following data table: solution .

An example of an exothermic reaction is the mixture of sodium and chlorine to yield table salt endothermic and exothermic reactions are chemical reactions that . Thermodynamics thermodynamics: the left: for an exothermic reaction the enthalpy of the system these values can be obtained from data tables nh3(g) + hcl(g) . Kinetics and thermodynamics of reactions key words: enthalpy, entropy, gibbs free energy, heat of reaction, energy of from the table of thermodynamic data, the . Enthalpy, entropy, and spontaneity search this site home thermodynamic data total kinetic energy (1 st law of thermodynamics) negative: exothermic .

There are a few ways that exothermic reactions can used as hot packs the reaction we will study today mixes anhydrous magnesium sulfate (mgso mass in data table . Standard thermodynamic properties of chemical substances this table gives the standard state chemical thermodynamic properties of about 2400 individual substances in the crystalline, l iquid, and gaseous.

Exothermic thermodynamics and data table
Rated 3/5 based on 50 review

2018.


text:
Register FAQ Search Today's Posts Mark Forums Read

 2016-12-26, 06:01 #727 Mark Rose     "/X\(‘-‘)/X\" Jan 2013 1011011100012 Posts I wanted to see how changing the CPU frequency would affect the power consumption of my i5-6600 DDR4-2133 compute cluster. I created a spreadsheet with timings of 4 threads and 4 workers. We've noticed before Skylake gets more throughput using 4 threads instead of 4 workers, but it turns out that changes with CPU frequency. When I underclock the CPU to 3.3 GHz, using 4 workers begins to be more performant. Spreadsheet. Underclocking is proving to be an efficiency win. I haven't yet played with undervolting. I also intend to gather more data while running 3 threads/workers. So far 4 threads/workers at 3.3 GHz is faster than 3 threads at 3.7 GHz. I haven't yet disabled a core in the BIOS to check power consumption differences. I strongly suspect 4 cores will win.
2016-12-26, 09:07   #728
S485122

Sep 2006
Brussels, Belgium

3×7×79 Posts

Quote:
 Originally Posted by Mark Rose ... Underclocking is proving to be an efficiency win. I haven't yet played with undervolting. ...
This is why the low voltage versions of those CPU are interesting : the i5-6600T for instance.

Jacob

2016-12-26, 13:43   #729
axn

Jun 2003

132·29 Posts

Quote:
 Originally Posted by Mark Rose We've noticed before Skylake gets more throughput using 4 threads instead of 4 workers, but it turns out that changes with CPU frequency. When I underclock the CPU to 3.3 GHz, using 4 workers begins to be more performant.
Based on the data, I think this statement is too strong. Even at 3.3 GHz, 1 worker is more performant that 4 workers, except for 3 FFTs where the difference is so small as to be negligible. Of course, the trend is there, so maybe 3.2 & below might show a clear superiority for the 4 workers.

 2016-12-26, 14:03 #730 lavalamp     Oct 2007 Manchester, UK 17·79 Posts Of course, the real metric should be performance / total cost of ownership. Add up the cost of the computer, as well as the cost of all previously used, and expected future use of electricity (based on chosen clock speed). Optionally also include the cost of your time working on setting up and configuring the machine. :)
2016-12-26, 14:28   #731
retina
Undefined

"The unspeakable one"
Jun 2006
My evil lair

2·3·1,019 Posts

Quote:
 Originally Posted by lavalamp Add up the cost of the computer, as well as the cost of all previously used, and expected future use of electricity (based on chosen clock speed). Optionally also include the cost of your time working on setting up and configuring the machine. :)
And if you lose one week of work trying to figure out the sweet spot and gain 3% then you have to work 24/7 for 32 weeks just to break even.

2016-12-26, 19:30   #732
Mark Rose

"/X\(‘-‘)/X\"
Jan 2013

292910 Posts

Quote:
 Originally Posted by axn Based on the data, I think this statement is too strong. Even at 3.3 GHz, 1 worker is more performant that 4 workers, except for 3 FFTs where the difference is so small as to be negligible. Of course, the trend is there, so maybe 3.2 & below might show a clear superiority for the 4 workers.
If you plot the slopes of iter/sec/ghz, the threads slope is steeper. I'll take more measurements and update the thread when done.

Most impressive to me so far is that I can run at 3.3 GHz for 95% performance for 86% of running costs, before undervolting.

Quote:
 Originally Posted by lavalamp Of course, the real metric should be performance / total cost of ownership. Add up the cost of the computer, as well as the cost of all previously used, and expected future use of electricity (based on chosen clock speed). Optionally also include the cost of your time working on setting up and configuring the machine. :)
It would have been much more cost efficient to go with the i5-6400, but I am hoping the i5-6600 will have better resale value in the future. Likewise I went with 32 GB of RAM. The additional $100/node also gives me more flexibility if I want to use the cluster for anything else. The four nodes share a single power supply. Quote:  Originally Posted by retina And if you lose one week of work trying to figure out the sweet spot and gain 3% then you have to work 24/7 for 32 weeks just to break even. It would be hard to lose a week of work. I'm only experimenting on a single node, and when I am measuring the power draw I'm doing DC. Testing stability with undervolting will certainly take more time! 2016-12-26, 23:28 #733 lavalamp Oct 2007 Manchester, UK 17·79 Posts Quote:  Originally Posted by Mark Rose It would have been much more cost efficient to go with the i5-6400, but I am hoping the i5-6600 will have better resale value in the future. Unless you plan on selling in the near-future, I would have thought the resale value on these would be essentially nil. Intel will release the 7xxx series early next year, and then (at least they claim) the 8xxx series later in the year, meaning you'll already be 2 generations behind. How long do you plan on running these machines, and what do you project an extra ~10W / machine will cost over such a time? Edit: A rough calculation for 4 machines. 10W / machine is ~1kWh / day, and based on these figures electricity costs 11.16 c/kWh in Toronto now, but I'll round it up to 12 c/kWh. Over 4 years that comes to$175.

Last fiddled with by lavalamp on 2016-12-26 at 23:35

2016-12-27, 02:58   #734
Mark Rose

"/X\(‘-‘)/X\"
Jan 2013

29×101 Posts

Unfortunately I wasn't able to underclock my CPU. I didn't know the fixed multiplier also prevents underclocking. I have yet to try undervolting.

Quote:
 Originally Posted by lavalamp Unless you plan on selling in the near-future, I would have thought the resale value on these would be essentially nil. Intel will release the 7xxx series early next year, and then (at least they claim) the 8xxx series later in the year, meaning you'll already be 2 generations behind.
There is a vibrant used component market here. I intend to sell the whole lot to a used system builder. Of greater concern with selling is what AMD Ryzen will do.

Quote:
 How long do you plan on running these machines, and what do you project an extra ~10W / machine will cost over such a time? Edit: A rough calculation for 4 machines. 10W / machine is ~1kWh / day, and based on these figures electricity costs 11.16 c/kWh in Toronto now, but I'll round it up to 12 c/kWh. Over 4 years that comes to $175. I wish electricity were so cheap. The actual electricity cost, once all the taxes, fees, etc., are factored in, is 17.7¢/kWh for the first 600 kWh and 19.5¢/kWh after. In the cooler months when not using AC, I've been using about 550 kWh with the cluster running. This past summer my highest monthly usage was about 1250 kWh. For simplicity I'll say the cluster incrementally costs me 18.5¢/kWh. The cluster, not including the Ethernet switch, consumes 370 watts when running at stock clocks. That's 266 kWh/month, or$591/year. So if I can save 50 watts across the cluster, that's 36 kWh/month, or \$80/year. I figure I'll run the cluster at most 2.5 years more.

2017-02-14, 17:15   #736

"Kieren"
Jul 2011
In My Own Galaxy!

2·3·1,693 Posts

Quote:
 TL;DR: get dual rank memory.
I wish it were easier to determine which parts are dual.

 2017-03-08, 22:29 #737 VictordeHolland     "Victor de Hollander" Aug 2011 the Netherlands 117610 Posts Just for fun I ran the Prime95 benchmark on my old 11'' netbook with AMD E-350 @1.6GHz Code: AMD E-350 Processor CPU speed: 1596.06 MHz, 2 cores CPU features: 3DNow! Prefetch, SSE, SSE2 L1 cache size: 32 KB L2 cache size: 512 KB L1 cache line size: 64 bytes L2 cache line size: 64 bytes L1 TLBS: 40 L2 TLBS: 512 Prime95 32-bit version 28.10, RdtscTiming=1 Best time for 1024K FFT length: 99.916 ms., avg: 103.876 ms. Best time for 1280K FFT length: 142.989 ms., avg: 149.942 ms. Best time for 1536K FFT length: 167.401 ms., avg: 174.861 ms. Best time for 1792K FFT length: 196.051 ms., avg: 213.505 ms. Best time for 2048K FFT length: 222.136 ms., avg: 231.517 ms. Best time for 2560K FFT length: 275.761 ms., avg: 289.138 ms. Best time for 3072K FFT length: 347.029 ms., avg: 357.043 ms. Best time for 3584K FFT length: 403.094 ms., avg: 417.261 ms. Best time for 4096K FFT length: 458.478 ms., avg: 477.660 ms. Best time for 5120K FFT length: 645.214 ms., avg: 656.045 ms. Best time for 6144K FFT length: 778.602 ms., avg: 801.066 ms. Best time for 7168K FFT length: 936.351 ms., avg: 955.565 ms. Best time for 8192K FFT length: 1056.663 ms., avg: 1081.029 ms. Timing FFTs using 2 threads. Best time for 1024K FFT length: 50.489 ms., avg: 51.903 ms. Best time for 1280K FFT length: 71.734 ms., avg: 74.501 ms. Best time for 1536K FFT length: 83.726 ms., avg: 86.498 ms. Best time for 1792K FFT length: 98.136 ms., avg: 101.268 ms. Best time for 2048K FFT length: 111.295 ms., avg: 113.926 ms. Best time for 2560K FFT length: 137.782 ms., avg: 144.518 ms. Best time for 3072K FFT length: 172.050 ms., avg: 178.164 ms. Best time for 3584K FFT length: 203.060 ms., avg: 208.392 ms. Best time for 4096K FFT length: 230.258 ms., avg: 236.426 ms. Best time for 5120K FFT length: 319.971 ms., avg: 327.913 ms. Best time for 6144K FFT length: 397.567 ms., avg: 418.475 ms. Best time for 7168K FFT length: 477.554 ms., avg: 487.654 ms. Best time for 8192K FFT length: 524.522 ms., avg: 534.575 ms. Timings for 1024K FFT length (2 cpus, 2 workers): 104.66, 98.42 ms. Throughput: 19.72 iter/sec. Timings for 1280K FFT length (2 cpus, 2 workers): 149.56, 140.02 ms. Throughput: 13.83 iter/sec. Timings for 1536K FFT length (2 cpus, 2 workers): 176.08, 163.24 ms. Throughput: 11.81 iter/sec. Timings for 1792K FFT length (2 cpus, 2 workers): 198.76, 186.35 ms. Throughput: 10.40 iter/sec. Timings for 2048K FFT length (2 cpus, 2 workers): 223.87, 209.46 ms. Throughput: 9.24 iter/sec. Timings for 2560K FFT length (2 cpus, 2 workers): 288.10, 267.54 ms. Throughput: 7.21 iter/sec. Timings for 3072K FFT length (2 cpus, 2 workers): 356.97, 335.54 ms. Throughput: 5.78 iter/sec. Timings for 3584K FFT length (2 cpus, 2 workers): 429.52, 401.99 ms. Throughput: 4.82 iter/sec. Timings for 4096K FFT length (2 cpus, 2 workers): 497.72, 464.37 ms. Throughput: 4.16 iter/sec. Timings for 5120K FFT length (2 cpus, 2 workers): 695.86, 655.48 ms. Throughput: 2.96 iter/sec. Timings for 6144K FFT length (2 cpus, 2 workers): 792.26, 738.64 ms. Throughput: 2.62 iter/sec. Timings for 7168K FFT length (2 cpus, 2 workers): 986.58, 920.18 ms. Throughput: 2.10 iter/sec. Timings for 8192K FFT length (2 cpus, 2 workers): 1096.14, 1023.91 ms. Throughput: 1.89 iter/sec.

 Similar Threads Thread Thread Starter Forum Replies Last Post Xyzzy Lounge 39 2021-03-12 14:19 Oddball Riesel Prime Search 5 2010-08-02 00:11 rogue Soap Box 19 2009-10-28 19:17 Xyzzy Lounge 10 2006-09-28 00:36 Xyzzy Factoring 65 2005-09-05 08:16

All times are UTC. The time now is 07:22.

Mon Apr 12 07:22:03 UTC 2021 up 4 days, 2:02, 1 user, load averages: 2.46, 2.53, 2.32


text:
# Is there a way to shorten a set of conditionals like this?

I have a long set of IF statements that basically set a null object Value to 0, I feel that because i am doing the same action each time their has to be a simple way to make this allot shorter. It just looks like something I wouldn't normally see in code.

My code looks like this:

if ($data->sl1 === NULL){$data->sl1 = "0";}
if ($data->sl2 === NULL){$data->sl2 = "0";}
if ($data->sl3 === NULL){$data->sl3 = "0";}
if ($data->sl4 === NULL){$data->sl4 = "0";}
if ($data->sl5 === NULL){$data->sl5 = "0";}
if ($data->sl6 === NULL){$data->sl6 = "0";}
if ($data->sl7 === NULL){$data->sl7 = "0";}
if ($data->sl8 === NULL){$data->sl8 = "0";}
if ($data->sl9 === NULL){$data->sl9 = "0";}
if ($data->sl10 === NULL){$data->sl10 = "0";}
if ($data->sl11 === NULL){$data->sl11 = "0";}
if ($data->sl12 === NULL){$data->sl12 = "0";}

if ($data->sn1 === NULL){$data->sn1 = "0";}
if ($data->sn2 === NULL){$data->sn2 = "0";}
if ($data->sn3 === NULL){$data->sn3 = "0";}
if ($data->sn4 === NULL){$data->sn4 = "0";}
if ($data->sn5 === NULL){$data->sn5 = "0";}
if ($data->sn6 === NULL){$data->sn6 = "0";}
if ($data->sn7 === NULL){$data->sn7 = "0";}
if ($data->sn8 === NULL){$data->sn8 = "0";}
if ($data->sn9 === NULL){$data->sn9 = "0";}
if ($data->sn10 === NULL){$data->sn10 = "0";}
if ($data->sn11 === NULL){$data->sn11 = "0";}
if ($data->sn12 === NULL){$data->sn12 = "0";}

if ($data->sr1 === NULL){$data->sr1 = "0";}
if ($data->sr2 === NULL){$data->sr2 = "0";}
if ($data->sr3 === NULL){$data->sr3 = "0";}
if ($data->sr4 === NULL){$data->sr4 = "0";}
if ($data->sr5 === NULL){$data->sr5 = "0";}
if ($data->sr6 === NULL){$data->sr6 = "0";}
if ($data->sr7 === NULL){$data->sr7 = "0";}
if ($data->sr8 === NULL){$data->sr8 = "0";}
if ($data->sr9 === NULL){$data->sr9 = "0";}
if ($data->sr10 === NULL){$data->sr10 = "0";}
if ($data->sr11 === NULL){$data->sr11 = "0";}
if ($data->sr12 === NULL){$data->sr12 = "0";}

One way I know could work would simply be to iterate through the entire object with a simple foreach like this:

foreach ($data as$item) {
if ($item === NULL){$item = "0";}
}

But let's say I don't want to change every single null object to 0. Do I have any other option?

I am using PHP in my example, but I am curious to hear answers for other languages as well.

• So there are some conditions? A subset of lists matching certain conditions? It's hard to make one up with no background. Feb 12 '14 at 23:03
• The variable names could be a little more descriptive, but take a look at get_object_vars. Feb 12 '14 at 23:05
• I wish they could be, but these are names from a parsed CSV provided by our client, my current solution is fine for them, but i am curious for future projects if i can do this in a quicker way. Feb 12 '14 at 23:06
• I might be wrong but it seems get_object_vars still iterates through the entire object doesn't it? Feb 12 '14 at 23:08
• It does - it gives you all of the public members in an array. You can take the result and loop over it. C# gives similar functionality through reflection, but has the advantage of being strongly typed (so you can check what type of parameter it is). Feb 12 '14 at 23:11

In short, the CSV parser is using the wrong data types. Whenever you have contiguous variable names, it screams "use an array!" If you at all, possibly can, use an array. That allows you to operate on it much, much more naturally, it's more flexible, and it's way less prone to human error.

If you used an array, this would be much, much simpler:

foreach ($this->sl as$k => $sl) {$this->sl = ($sl === null) ? "0" :$sl;
}

Since you seemingly can't use an array, your best bet is probably to either process it into an array, or use some nasty variable-variable hackiness to pretend it's an array.

$sl = array(); for ($i = 1; isset($this->{'sl' .$i}; ++$i) {$sl[] = $this->{'sl' .$i};
}

Or if (for some reason) you don't want to use an array:

for ($i = 1; isset($this->{'sl' . $i}; ++$i) {
$this->{'sl' .$i} = ($this->{'sl' .$i} === null) ? "0" : $this->{'sl' .$i};
}

• The reason i can't use an array is because each item (ss1 ss2 ss3) is a unique column in the database, so each one is returned separately. I agree the names are very stupid, but the mutual fund we are working with prefers to use a shorthand to represent their data like that. How ever your last solution would indeed work, i hadn't thought to treat the object name as a string like that. Feb 12 '14 at 23:16
• @AndrewFont You could always unpack the array back into columns. That is an unfortunate situation though. It sounds like it's likely badly normalized database. I'm sure that's completely unchangeable though :/. Feb 12 '14 at 23:18
• Also i made that terrible parser =(, but i was forced to feed the data to the specific columns in the database, so i was not able to change the naming scheme for them, so i chopped it up and saved each piece of information where it needed to go Feb 12 '14 at 23:18
• Yea this project was kind of a data nightmare because everything had strange shorthand names and the database schema for each and every CSV i had to parse was made by a different developer then the last. Feb 12 '14 at 23:20


text:
# The N-clock model: variational analysis for fast and slow divergence rates of N

created by cicalese on 02 Apr 2021

[BibTeX]

Preprint

Inserted: 2 apr 2021
Last Updated: 2 apr 2021

Year: 2020

ArXiv: 2012.09548 PDF

Abstract:

We study a nearest neighbors ferromagnetic spin system on the square lattice in which the spin field is constrained to take values in a discretization of the unit circle consisting of $N$ equi-spaced vectors, also known as $N$-clock model. We find a fast rate of divergence of $N$ with respect to the lattice spacing for which the $N$-clock model has the same discrete-to-continuum variational limit of the $XY$ model, in particular concentrating energy on topological defects of dimension 0. We prove the existence of a slow rate of divergence of $N$ at which the coarse-grain limit does not detect topological defects, but it is instead a $BV$-total variation. Finally, the two different types of limit behaviors are coupled in a critical regime for $N$, whose analysis requires the aid of Cartesian currents.

Credits | Cookie policy | HTML 5 | CSS 2.1


text:
# V. A. Tesko

orcid.org/0000-0002-6071-1985
Search this author in Google Scholar

Articles: 5

### The investigation of Bogoliubov functionals by operator methods of moment problem

Methods Funct. Anal. Topology 22 (2016), no. 1, 1-47

The article is devoted to a study of Bogoliubov functionals by using methods of the operator spectral theory being applied to the classical power moment problem. Some results, similar to corresponding ones for the moment problem, where obtained for such functionals. In particular, the following question was studied: under what conditions a sequence of nonlinear functionals is a sequence of Bogoliubov functionals.

### The integration of operator-valued functions with respect to vector-valued measures

Volodymyr Tesko

Methods Funct. Anal. Topology 18 (2012), no. 3, 288-304

We investigate the $H$-stochastic integral introduced in [24]. It is known that this integral generalizes the classical Ito stochastic integral and the Ito integral on a Fock space. In the present paper we construct and study an extension of the $H$-stochastic integral which will generalize the Hitsuda-Skorokhod integral.

### One generalization of the classical moment problem

Volodymyr Tesko

Methods Funct. Anal. Topology 17 (2011), no. 4, 356-380

Let $\ast_P$ be a product on $l_{fin}$ (a space of all finite sequences) associated with a fixed family $(P_n)_{n=0}^{\infty}$ of real polynomials on $\mathbb{R}$. In this article, using methods from the theory of generalized eigenvector expansion, we investigate moment-type properties of $\ast_P$-positive functionals on $l_{fin}.$ If $(P_n)_{n=0}^{\infty}$ is a family of the Newton polynomials $P_n(x)=\prod_{i=0}^{n-1}(x-i)$ then the corresponding product $\star=\ast_P$ is an analog of the so-called Kondratiev--Kuna convolution on a "Fock space". We get an explicit expression for the product $\star$ and establish a connection between $\star$-positive functionals on $l_{fin}$ and a one-dimensional analog of the Bogoliubov generating functionals (the classical Bogoliubov functionals are defined correlation functions for statistical mechanics systems).

### A stochastic integral of operator-valued functions

Volodymyr Tesko

Methods Funct. Anal. Topology 14 (2008), no. 2, 132-141

In this note we define and study a Hilbert space-valued stochastic integral of operator-valued functions with respect to Hilbert space-valued measures. We show that this integral generalizes the classical Ito stochastic integral of adapted processes with respect to normal martingales and the Ito integral in a Fock space.

### A construction of generalized translation operators

V. A. Tesko

Methods Funct. Anal. Topology 10 (2004), no. 4, 86-92


text:
# renewcommand \section* or adding number count

#### ppeetteerr

I have two master files (a,b) that use the same content file (c). when I compile with file a I wish the section and subsection titles to have the characteristics of \section (i.e. section numbering) yet when I compile with b that it would have the characteristics of \section* I thought I could use the renew command on section* but that doesn't seem to work.

Peter

Oops solved it using the command
\setcounter{secnumdepth}{-1}

Last edited:


text:
## COCI '14 Contest 1 #1 Prosjek

View as PDF

Points: 5 (partial)
Time limit: 1.0s
Memory limit: 32M

Problem type
Allowed languages
Ada, Assembly, Awk, Brain****, C, C#, C++, COBOL, CommonLisp, D, Dart, F#, Forth, Fortran, Go, Groovy, Haskell, Intercal, Java, JS, Kotlin, Lisp, Lua, Nim, ObjC, OCaml, Octave, Pascal, Perl, PHP, Pike, Prolog, Python, Racket, Ruby, Rust, Scala, Scheme, Sed, Swift, TCL, Text, Turing, VB, Zig

Mirko is practicing arithmetic operations in an interesting way during math class. First, he writes a sequence of integers . Then, underneath the first sequence, he writes another sequence of integers which he gets by replacing every number from the sequence with the average value of all the numbers before the current one, including it.

For example, if the first sequence of integers is equal to

then the second sequence of integers is going to be

in other words

You are given the second sequence of integers . Determine the first sequence of integers to check Mirko's calculations.

#### Input

The first line of input contains the integer , the length of sequence .
The second line of input contains the sequence of spaced-separated integers .

#### Output

The first and only line of output must contain a sequence of space-separated integers .
Please note: The input data will be such that the elements from the sequence are integers .

#### Sample Input 1

1
2

#### Sample Output 1

2

#### Sample Input 2

4
3 2 3 5

#### Sample Output 2

3 1 5 11

#### Sample Input 3

5
1 2 2 3 4

#### Sample Output 3

1 3 2 6 8


text:
Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Exposed soil and mineral map of the Australian continent revealing the land at its barest

Abstract

Multi-spectral remote sensing has already played an important role in mapping surface mineralogy. However, vegetation – even when relatively sparse – either covers the underlying substrate or modifies its spectral response, making it difficult to resolve diagnostic mineral spectral features. Here we take advantage of the petabyte-scale Landsat datasets covering the same areas for periods exceeding 30 years combined with a novel high-dimensional statistical technique to extract a noise-reduced, cloud-free, and robust estimate of the spectral response of the barest state (i.e. least vegetated) across the whole continent of Australia at 25 m2 resolution. Importantly, our method preserves the spectral relationships between different wavelengths of the spectra. This means that our freely available continental-scale product can be combined with machine learning for enhanced geological mapping, mineral exploration, digital soil mapping, and establishing environmental baselines for understanding and responding to food security, climate change, environmental degradation, water scarcity, and threatened biodiversity.

Introduction

Most natural terrestrial land surfaces consist of mixtures of vegetation, soil and bedrock. These materials reflect and absorb radiation across different wavelengths resulting in an observation of a spectral response. Our ability to accurately distinguish the surface materials from these spectral signatures depends on the spatial and spectral resolution of the instrument, the state of the atmosphere and homogeneity of the surface relative to the resolution of the sensor. However, as green and dry vegetation both strongly interact with electromagnetic radiation across the same wavelength region that is used for mineral mapping1,2,3, the ability to separate – or at least recognise – the influence of the vegetation is critical to the success of mapping the characteristics of soil and rock. Indeed, just 10–30% vegetation coverage can significantly impede the recognition of mineral spectral features1,2,4,5.

Where geo-botanical relationships are strong (i.e. plant species or communities are strongly linked to the type of bedrock or soil), indirect measures can allow inferences to be made on the nature of the substrate6,7. However, in the absence of geo-botanical inference, effective mapping of soils using remote sensing is restricted to areas where the vegetation cover is either sparse or absent due to drought or cultivation8. As such, most remotely sensed mineral mapping studies have covered areas of relatively sparse vegetation cover9,10,11,12. In Australia, a significant obstacle to geological remote sensing are fire scars in the arid zones as they create a complex pattern of vegetation densities that makes it difficult to map bedrock and soil13.

Recently, some studies have attempted to exploit the full time series archive of Landsat observations, collected over the last 30 years, to produce per-pixel mosaics of the barest earth (i.e. exposed soil)14,15. However, their approaches were only attempted in small geographic areas and their methodologies are based on user-defined thresholds and data-mining techniques that are unlikely to be sufficiently complex and scalable to correctly remove all the non-bare responses in the observations. This is especially true for a large continent such as Australia, which exhibits a very diverse range of climates, terrestrial land surfaces and biophysical changes through time. Scaling methods to a continental archive of data presents a major challenge in the analysis of big datasets called heterogeneity whereby outliers and various states are no longer sparse but become proper sub-populations in the data that are difficult to disentangle16.

In this paper, we tackle these technical issues and provide the first continental-scale mosaic of Australia at its barest state using the full temporal archive of Landsat observations. We achieve this by proposing a statistical estimator of the barest spectra that is both robust to contamination (such as cloud cover, shadows, detector saturation and pixel corruption) and, most importantly, correctly maintains the relationship between all the spectral wavelengths enabling the application of machine learning and spatial statistics to further separate vegetation and mineral spectra17,18. The result of our approach is the generation of two continental-scale composite Barest Earth mosaics of Australia, one of shorter temporal depth using only Landsat 8 observations and the other using the full 30 + year archive combining Landsat 5, 7 and 8.

Results

Data and approach

The approach starts by considering ~16 billion 6-dimensional time series across the continent of Australia where each time series represents the variation of the BLUE, GREEN, RED, NIR, SWIR1 and SWIR2 wavelength bands in a 250-m2 pixel (see Table 1). These time series were generated from imagery of the Australian continent collected by the Landsat series of satellites over a period of more than 30 years. These images were geographically aligned and stacked, so that the pixels line up through time, to obtain a spatial-spectral-temporal (tensor) data set. We processed all data using the Open Data Cube (previously known as Australian Geoscience Data Cube19,20). Differences between Landsat sensors were minimised by ensuring that all observations were atmospherically, BRDF and topographically corrected to measurements of surface reflectance using the NBAR/T approach21. This surface reflectance correction, that uses several ancillary datasets to get a highly accurate atmospheric reading, is only available over the Australian continent.

For each time series, our aim is to obtain an estimate of the barest state observed relating to either soil or exposed rock. To achieve this, we holistically consider each pixel observation through time as 6-dimensional vector. The first ingredient of our method is to apply a high-dimensional statistic called a weighted geometric median (WGM) that combines ideas from the classic (one-dimensional) weighted median22 with the high-dimensional geometric median (GM) proposed by Weber23; see methods. Recently, the GM has attracted significant interest in the mathematical community due to its remarkable robustness and confidence boosting properties24,25,26,27. These properties and mathematical results easily extend to the weighted case (that we propose) as well. Interestingly, there are many ways to extend the concept of a median to higher dimensions28 but the GM is one of the more mathematically tractable of those proposed extensions. Our motivation to include weights into the definition of the GM, giving the WGM, is to focus the WGM on observations through time that exhibit the least amount of vegetation and the high level of robustness of the WGM removes the influence of the other sub-populations in the data. A key point is that by weighting (as opposed to filtering) we do not throw away any information. This allows the resulting pixel composite mosaic to be completely cloud-free and for the spectral signature of vegetation to be minimised. This approach also behaves well where the vegetation never achieves bareness, such as in areas with dense tree cover, as we get a result for every pixel. In other words, if a pixel consistently remains green through time then the resulting mosaic will have a green response at that location. This is a useful characteristic if a spatial model is to be applied to the product as our mosaic does not containing any missing data. The second ingredient to our approach is the design of an optimal weighting scheme that ranks, as opposed to filtering, the bareness of pixels and separate out the bare states from others, if they are present; see methods. The weighting scheme was derived by fitting an ensemble of regression models based on spectral features over various training areas exhibiting a high variability of biophysical states. A single model was derived from this ensemble using a loss function that minimises vegetation response and variability over these areas. The WGM then robustly summarises this bare state for each time series as a 6-dimensional vector (pixel) and combining all these pixels gives the continental-scale mosaic (Fig. 1).

Validation

We validated our Barest Earth mosaic three ways. First, we considered a small test area in Canberra where grass is commercially grown using pivot irrigation and then partially harvested throughout the year. This gives observations through time ranging from partially exposed bare soil to partial grassy cover but where a completely bare crop circle is never observed for any given time period (Fig. 2). Focusing on specific pixels within this area, we show in Fig. 3 the variation in spectral responses and our WGM estimate of the barest state. This gave confidence that our approach was able to separate the complex spatial-temporal-spectral dynamics into its constituents at each pixel to obtain a consistently bare crop circle. Most surprisingly, the resulting (synthetic) Barest Earth output in our test area is visually indistinguishable from a true observation and displays remarkable spatial smoothness even though the approach does not use any spatial information. This is consistent with recent findings29.

The second part of our validation consisted of generating two continental-scale composite Barest Earth mosaics of Australia. The first, Landsat-8 Barest Earth mosaic, only uses Landsat-8 OLI (Operational Land Imager) observations (from 2013 to the present day), and the second, Landsat 30 + Barest Earth, uses a deeper time series from 16 March 1983 to the present day, with observations that combine Landsat-5 Thematic Mapper (TM), Landsat-7 Enhanced Thematic Mapper + (ETM+), and Landsat-8 OLI. Our hypothesis was that the mosaic generated from the longer time series would have less vegetation response and less influence from burn scars as the algorithm had more opportunity to potentially observe a barer state over this longer time period. This hypothesis was validated by comparing the two mosaics in various areas across the continent. It is quite clear that the increased temporal depth allows us to overcome the difficulty in mapping fire prone landscapes that preserve vegetation patterns which correlate with fire scars of different ages and intensities13. We see that this effect is significantly reduced in the Landsat 30 + mosaic improving our ability to map soil and rock characteristics (Fig. 4).

The third stage of our validation exercise involved a comparison with existing datasets at Geoscience Australia to assess the effectiveness of the approach in minimising the influence of vegetation and to ensure the spectra are consistent with known geology, spectral endmembers collected in the field, and national vegetation maps. As part of the exercise, we used the National Geochemical Survey of Australia (NGSA) dataset containing spectra from surface soil samples30. These samples were homogenised from the upper 10 cm of soil and dried prior to being analysed in the laboratory using spectroscopy and then resampled to Landsat sensor band specifications. We compared the spectra from these soil samples (n = 1059) to the 25 m2 resolution spectra obtained from the Barest Earth mosaic at each location. We show the improvement in spectral similarity between the Barest Earth spectra and the NGSA soil sample spectra across Australia compared with a clear Landsat observation exhibiting a high vegetation signal (Fig. 5). We see little improvement in the arid area of Australia (<450 mm of average annual rainfall) due to the sparse amount of vegetation present. However, in the non-arid area which has a high density of permanent and seasonal vegetation, we see significant improvement in matching the soil spectra (Fig. 6).

Discussion

Our Barest Earth products show a significant reduction of the influence of vegetation in the multi-spectral bands with the result of enhancing our ability to observe responses pertaining to variations in surface mineralogy. The reduced influence in vegetation is most striking in landscapes with seasonal variations in ground cover (Fig. 7) and in more arid fire prone landscapes (Fig. 4). This allows us to overcome in many landscapes the largest challenges (vegetation masking and burn scars) in mapping soil and rock from space.

At one extreme, we have dense vegetation cover, corresponding to closed forest and closed shrub lands. In these areas, it is unlikely that exposed ground will be seen unless the vegetation is removed by land clearing. In areas of less dense vegetation cover (open woodlands where the crowns of trees do not touch, open shrub lands and grasslands), we have a much higher probability of estimating the bare spectral response through the time series. Periods of increased bareness commonly relate to seasonal drying, drought, the phenological cycle, fires, erosion and land clearing. For example, in agricultural landscapes where there is a pressing requirement to map and monitor soil health31 we see a dramatic increase in bareness an consequently our ability to map soils directly (Fig. 7). A less obvious, but apparent at the 25 m2 resolution, is a reduction in vegetation response in areas of relatively low vegetation density corresponding to the central arid and semi-arid zone (Fig. 6). These arid zones cover ~70% of the Australian continent and, on average, have vegetation densities of the order of 10–30%, with local areas up to 60% being common13.

From a soil and geological perspective, especially in areas where rocks and soil are exposed, multi-spectral remote sensing with Earth observation satellites has proven effective in mapping lithologies (e.g. individual or groups of minerals), geochemistry, bedrock structure and the nature of the regolith (i.e. weathered bedrock and sediments)9,17,32,33,34. Subtle geochemical alteration patterns associated with mineral deposits have been also mapped using satellite imagery9,35,36,37. Mineral spectra associated with soils have been used to identify different soil types38 and provide proxies to infer soil properties, such as cation exchange capacity39. However, these remote sensing approaches have always been limited by vegetation cover and burn scars in the landscape. Our continental-scale Barest Earth products now provide the ability to more accurately map soil and rock spectra over large areas that were previously masked by vegetation. Importantly, due to the multivariate nature of our technique, the spectral integrity (e.g. relationships between bands) is maintained across all wavelengths. This makes possible the extraction of mineral features based on their spectral signatures using well-established band ratios and principal component analysis11,17,40, or to apply machine learning algorithms for automated classification of mineralogy in the observed landscape. An advantage of our approach is that methods previously only effective on individual clear scenes with sparse vegetation cover can now be holistically calibrated and applied at a continental scale. This includes edge detection, band ratioing, and several statistical and spectral analysis techniques that are well-known for further separating and classifying vegetation and mineral spectra (e.g., directed principle component analysis18, spectral unmixing41, decorrelation stretching and saturation enhancement17). For example, see Fig. 8.

Our approach has produced the first Barest Earth mosaics for the Australian continent. They have broad application in mineral exploration, agriculture and in understanding the nature and processes operating within the life-sustaining critical zone between tree tops and groundwater aquifers42. These thematic maps will be powerful covariates in machine-learning algorithms used for predictive mapping of soil properties43 and surface geochemistry44 and will improve the detection of often subtle mineral alteration patterns associated with mineral deposits. These enhanced products will be integrated with other geological datasets that capture information from the deeper crust and upper mantle, as part of a national and holistic research initiative called UNCOVER45 to support mineral exploration in Australia. Furthermore, as our method can be applied over any epoch of time, for example annually, it may be used to monitor bareness through time or deviations from its long-term observations can be assessed with direct applications in establishing environmental baselines, assessing impacts of climate change, and monitoring soil health and land degradation.

Methods

Weighted geometric median

We consider our space-time stack of observations over the Australian continent as a collection of time series, where $${\Bbb X}_{ij}$$ is the time series of observations at pixel location (i, j) and $$n: = n_{ij}$$ is the number of observations at that location. This is possible because all pixels are aligned through time due to the ortho-rectification and registration of all images in our archive. The observations in the (i, j)’th p-dimensional time series are written

$${\Bbb X}_{ij} = \left[ {{\boldsymbol{x}}_{ij}^{\left( 1 \right)},{\boldsymbol{x}}_{ij}^{\left( 2 \right)}, \ldots ,{\boldsymbol{x}}_{ij}^{\left( n \right)}} \right]^T.$$

This means that $${\Bbb X}_{ij}$$ can be viewed as a data matrix with dimension n × p, wherein p is the number of bands (so in the case of Landsat data, p = 6).

As our approach does not consider neighbouring values or the spatial context around each pixel, we may describe the model for a single time series without loss of generality, with the understanding that an identical process is carried out for each time series in our dataset. As such, we omit the “ij” subscript and write $${\Bbb X}$$ for the time series (or data matrix) and x(t) for the t-th pixel observation in $${\Bbb X}$$ to avoid notational clutter. Given $${\Bbb X},$$ we compute a weighted geometric median (WGM) of the component observations by solving the following optimisation problem

$${\mathrm{m}}_w: = {\mathrm{argmin}}_{x \in {\Bbb R}}p\mathop {\sum }\limits_{t = 1}^n w_t\left\| {x - x^{(t)}} \right\|$$

where $$\left\| \cdot \right\|$$ is the Euclidean norm on $${\Bbb R}^p$$ and $$w_t \in {\Bbb R}^p$$ is the weight of the t-th observation. This is a generalisation of the geometric median (GM), which is obtained from the WGM when wt = 1 for every t. The resulting quantity mw is a p-dimensional vector and, by following the mathematical proofs in Kemperman46, it can be shown that mw always exists for a set of observations. The weight function w that assigns the weight wt for observation t can be thought of as an unsupervised classifier that allocates a higher value to the biophysical state that we are interested in and a lower value to other states.

Designing the weighting scheme

The choice of weight $$w_t \in \left[ {0,1} \right]$$ allows us to penalise or accentuate certain observations through time at every pixel location (i, j). Since the weight function can be viewed as an unsupervised classifier of the biophysical state, this implies that even though our overall methodology can be applied to any sensor, the weight function will need to be redesigned for each case. In this paper, we consider the case of Landsat data; thus, we take p = 6 and so for a fixed spatial location (i, j), the t-th observation, $$x^{\left( t \right)} = {\Bbb X}$$, in $${\Bbb X}$$, is a vector of surface reflectance values relating to each sensor, i.e. $${\Bbb X} = [x_{{\mathrm{BLUE}}}x_{{\mathrm{GREEN}}},x_{{\mathrm{RED}}},x_{{\mathrm{NIR}}},x_{{\mathrm{SWIR}}1},x_{{\mathrm{SWIR}}2}]^T.$$ Our aim is then to identify a suitable, data dependent, weighting scheme that maps $${\Bbb X}$$ to a value between [0, 1]. The weighting scheme consists of two steps: firstly the choice of an (unnormalised) weighting function $$\tilde w-{\Bbb R}^p \to {\Bbb R}$$, which is applied to each of the n observations, and secondly a step that takes these transformed values $$\tilde w\left( {x^{\left( 1 \right)}} \right), \ldots ,\tilde w\left( {x^{\left( n \right)}} \right)$$ and normalises them relative to each other, so that the final weights through time $$w_1, \ldots ,w_n$$ take values between 0 and 1, giving a weight vector $${\Bbb W} = [w_1, \ldots ,w_n]^T,$$ where wt is the weight assigned to the t-th observation in the definition of the WGM. We note that a simple way to normalise values is to identify the minimum wmin, and maximum wmax elements of $$\tilde w\left( {x^{\left( 1 \right)}} \right), \ldots ,\tilde w\left( {x^{\left( n \right)}} \right)$$ and to calculate $$w_t = (w_t - w_{{\mathrm{min}}})/(w_{{\mathrm{max}}} - w_{{\mathrm{min}}})$$ to ensure that weights are between 0 and 1; however, this does not lead to the most optimal result.

The first step is the choice of a weighting function $$\tilde w$$ that penalises or accentuates each observation based on its biophysical class at that point in time. We started by generating simple features with which to separate various biophysical classes, based on the spectral information in the pixel $${\Bbb X} = [ {x_1,x_2, \ldots ,x_p} ]^T$$. In the most general case, these features have the functional form

$$f\left( {\Bbb X} \right) = \left( {\frac{{x_i - \mu x_j + \zeta }}{{\nu x_i + x_j + \xi }}} \right)\left( {\frac{{\alpha _1\left( {x_1 - \gamma _1} \right) + \cdots + \alpha _p\left( {x_p - \gamma _p} \right) + \rho }}{{\beta _1\left( {x_1 - \sigma _1} \right) + \cdots + \beta _p\left( {x_p - \sigma _p} \right) + \delta }}} \right)$$
(1)

for some choice of i, j with $$i\, \ne\, j$$ and for vectors $$\alpha = ( {\alpha _1, \ldots ,\alpha _p} )^T$$, $$\gamma = ( {\gamma _1, \ldots ,\gamma _p} )^T$$, $$\beta = ( {\beta _1, \ldots ,\beta _p} )^T$$, $$\sigma = ( {\sigma _1, \ldots ,\sigma _p} )^T$$ and constants ρ, δ, v, ζ, ξ, μ. Features of this form are often called ratio or normalised difference transformations of the bands and are commonly used in remote sensing. Classic examples include the Normalised Difference Vegetation Index $${\mathrm{NDVI}} = ({\mathrm{x}}_{{\mathrm{NIR}}} - x_{{\mathrm{RED}}})/({\mathrm{x}}_{{\mathrm{NIR}}} + x_{{\mathrm{RED}}})$$, the Green Normalised Difference Vegetation Index $${\mathrm{GDVI}} = (x_{{\mathrm{NIR}}} - x_{{\mathrm{GREEN}}})/(x_{{\mathrm{NIR}}} + x_{{\mathrm{GREEN}}})$$, the Green Soil Adjusted Vegetation Index $${\mathrm{GSAVI}} = ((x_{\mathrm{NIR}} - x_{\mathrm{GREEN}}))/(x_{\mathrm{NIR}} + x_{\mathrm{GREEN}} + 0.5)\;(1 + 0.5)$$, the Ratio Vegetation Index $${\mathrm{RVI}} = x_{\mathrm{NIR}}/x_{\mathrm{RED}}$$ and the Soil Adjusted Vegetation Index $${\mathrm{SAVI}} = (x_{{\mathrm{NIR}}} - x_{{\mathrm{RED}}})/(x_{{\mathrm{NIR}}} + x_{{\mathrm{RED}}} + 0.5)(1 + 0.5)$$. We also considered more recent contributions to the literature on separating soil classes. The Ratio Index for Bright Soil (RIBS)47 is a composite band ratio designed to assign lower weights to bright soil land cover. It is given by $${\mathrm{RIBS}} = {x}_{\mathrm{NDSI}}/{\mathrm{NTC}}1$$ where48 $${\mathrm{NDSI}} = (x_{{\mathrm{GREEN}}} - x_{{\mathrm{SWIR}}1})/(x_{{\mathrm{GREEN}}} + x_{{\mathrm{SWIR}}1})$$ and NTC1 is the normalised first Taselled Cap Transformation49. The Product Index for Dark Soil47 (PIDS) is designed to assign lower weight to darker soils and is given by PIDS = TCl × NDVI. We can then define our (unnormalised) weight function as

$$\tilde w\left( {\Bbb X} \right) = c_1f_1\left( {\Bbb X} \right) + \cdots + c_mf_m({\Bbb X})$$
(2)

where $$c_1, \ldots ,c_m$$ are the constants to be determined and $$f_1\left( {\Bbb X} \right), \ldots ,f_m\left( {\Bbb X} \right)$$ are features (as defined above). The simplest cases are when $$c_k = 1$$ or $$c_k = - 1$$ and $$c_i = 0$$ for $$i\, \ne\, k$$. Once the unnormalised weights $$\tilde w( {{\Bbb X}^{(1)}}), \ldots ,\tilde w( {{\Bbb X}^{(n)}})$$ have been assigned to all the observations through time, we then normalise them using the softmax function, which gives

$$w_t = \frac{{e^{\tilde w\left( {{\Bbb X}^{\left( t \right)}} \right)}}}{{\mathop {\sum }\nolimits_{i = 1}^n e^{\tilde w\left( {{\Bbb X}^{\left( i \right)}} \right)}}},\quad t \in \{ 1,2, \ldots ,n\}.$$
(3)

This forces each wt to take values in the range [0, 1] and also results in all of the weights $$w_1, \ldots ,w_n$$ summing to 1, so that (Eq. 2) is a convex combination of features. Figure 9 shows the effect of some model choices on the resulting pixel composite mosaic across an area to the west of Canberra (Australia).

Fitting an optimal model

Given m features $$f_1, \ldots ,f_m$$ of the form (Eq. 1), a model for bareness is defined in terms of our (unnormalised) weight function $$\tilde w$$ given by Eq. (2). This means that weights $$c_1, \ldots ,c_m$$ need to be identified to obtain an optimal model. The difficulty is identifying a model that works well at a continental scale. Specifically, Australia has a large geographical size and extremely varied climate. The south-east and south-west corners have a temperate climate and moderately fertile soil. The northern part of the country has a tropical climate, varying between tropical rainforests, grasslands and desert. This means that a model that works well in some areas may not be optimal in others. Moreover, due to the enormous range of soil types across Australia, developing a model that classifies bare soil can be difficult. Our model circumvents this ranking (instead of classifying) the observations through time at each pixel based on the relative presence of vegetation.

To reduce the potential of overfitting our weight model, we propose an approach that can be viewed as fitting an ensemble of regression models based on spectral features from which we derive a single model from this ensemble through a loss function that minimises average NDVI and variability over our training sites. We now describe this more precisely.

We fit an optimal model given features $$f_1, \ldots ,f_m$$, in order to fine tune our approach to specific (smaller) regions or as a way to determine the optimal model for a continental-scale output. The fitting approach starts by identifying spatial areas that exhibit a large amount of spectral change through time. At the i′th area, obtain the spatial-spectral-temporal stack of observations $${\Bbb A}_i$$. For simplicity, we explain our approach under the assumption that we only have one area $${\Bbb A}$$ of dimensions $$( {n_y,n_x,p,n_t} )$$, where p is the number of bands, nt is the number of observations through time and $$(n_y,n_x)$$ denote the number of spatial pixels.

Let $$\theta = (c_1, \ldots ,c_m)$$ be the parameter vector for our choice of model $$\tilde w$$. We define our loss function, which we are trying to minimise, as

$$\ell \left( {\theta ;{\Bbb A}} \right) = - \frac{{{\mathrm{mean}}_{y,x}{\Bbb B}}}{{{\mathrm{stddev}}_{y,x}{\Bbb B}}}$$

where $${\Bbb B} = - {\mathrm{NDVI}}\left( {{\mathrm{BARE}}\left( {{\Bbb A};{\uptheta}} \right)} \right)$$. The function BARE, given a spatial-spectral-temporal stack $${\Bbb A},$$ calculates the Barest Earth model using $$\tilde w,$$ given parameters θ (as described in the previous section) and outputs a $$( {n_y,n_x,p})$$-dimensional pixel composite mosaic (PCM). The function NDVI, given a $$( {n_y,n_x,p})$$-dimensional image, calculates the Normalised Difference Vegetation Index $${\mathrm{NDVI}} = (x_{{\mathrm{NIR}}} - x_{{\mathrm{RED}}})/(x_{{\mathrm{NIR}}} + x_{{\mathrm{RED}}})$$ and returns a $$( {n_y,n_x} )$$ dimensional image. The functions $${\mathrm{mean}}_{y,x}$$ and $${\mathrm{stddev}}_{y,x}$$ calculate the mean and standard deviation across the (y, x) dimensions and returns a number. As NDVI takes values in between [−1, 1] and increases as vegetation increases, $${\Bbb B}$$ will be an $$( {n_y,n_x} )$$ grayscale image with larger values at each pixel, meaning less vegetation (as we take the negative NDVI). This means that the $${\mathrm{mean}}_{y,x}$$ will increase as the PCM has reduced vegetation response. However, we also want to penalise outputs that appear more pixelated, so we divide by $${\mathrm{stddev}}_{y,x}$$. The term $${\mathrm{mean}}_{y,x}{\Bbb B}/{\mathrm{stddev}}_{y,x}{\Bbb B}$$ is our reward or utility function, but we multiply this by −1 to obtain a loss function.

In the simple case where m = 1, we have θ = c1 and only one choice of feature f1. Supplementary Fig. 1 shows the loss function $$\ell$$ for various choices of simple one-feature models. Remarkably, we observe a very smooth relationship between the parameter θ = c1 and our loss function $$\ell$$. This makes it very easy to identify the optimal θ that minimises the loss. The case of m > 1 is more complicated and involves minimising $$\ell \left( \theta \right)$$ over a m-dimensional surface $$\theta = \left( {c_1, \ldots ,c_m} \right)$$. As we have (empirically) observed that $$\ell$$ is smooth in each coordinate of θ, an efficient approach for minimising $$\ell$$ can be achieved through the use of Bayesian optimisation using a Gaussian process prior. Fitting over N training areas simultaneously can be easily achieved by defining the loss function as $$\ell \left( {\Bbb A} \right) = \ell \left( {\theta ,{\Bbb A}_1} \right) + \cdots + \ell \left( {\theta ;{\Bbb A}_N} \right)$$ where $${\Bbb A}_1, \ldots ,{\Bbb A}_N$$ are the spatial-spectral-temporal stacks of observations for these N training areas.

Differences between the Barest Earth products

Comparison of histograms between the two continental-scale products highlight some interesting differences. Landsat-8’s OLI sensor provides improved signal-to-noise radiometric (SNR) performance quantised over a 12-bit dynamic range compared to the 8-bit dynamic range of Landsat-5 and Landsat-7 data. This results in 4096 potential grey levels in each band compared with only 256 in the 8-bit instruments on Landsat-5 and Landsat-7. This means there is trade-off. The improved SNR performance in Landsat-8 results in more accurate separation of vegetation and bare spectral signatures through time and the resulting Barest Earth mosaic maintains the improved SNR performance. Whereas, the Landsat 30+ product has worse SNR performance but a greater capacity to find barest ground due to the greater temporal depth.

Model robustness

Our approach for estimating the Barest Earth spectra is robust in a variety of ways. First, our features within our unnormalized weightings are robust to parallel shifts of the spectra due to the use of normalised difference ratios between the bands. This is important so that we are robust to atmospheric effects and residual thin cloud. Second, we recall that the breakdown point of an estimator is the proportion of incorrect observations (e.g. arbitrarily large observations) an estimator can handle before giving an incorrect (e.g., arbitrarily large) result. The geometric median is well-known to have a breakdown point50 of 50%. We have conducted a simulation study of our model, which is a combination of a weighting and the geometric median, to understand how it performs under a variety of different noisy perturbations. We compare its robustness against a model that uses a weighted mean instead of a weighted geometric median and find that our approach is significantly more robust (see Supplementary Fig. 2). Finally, the (weighted) geometric median is translation equivariant and orthogonal equivariant50 which means that the relationship between the bands is maintained under orthogonal transformations and shifts (a mathematical fact that follows easily due to the invariance of Euclidean norm under such transformations). This is not true, for example, if the coordinatewise median is used (i.e., a univariate median in each band separately). This final property is particularly important for machine learning due to the extensive use of Principal Component Analysis (PCA) in the analysis of satellite imagery as PCA is a statistical and data reduction procedure that uses an orthogonal transformation to convert observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components.

Data availability

Both the Landsat-8 Barest Earth mosaic and the Landsat 30+ mosaic are freely available through the web viewer https://nationalmap.gov.au. Alternatively, these mosaics (sized around 200GB each, in Australian Albers projection, tiled, and in GeoTIFF format) can be obtained by contacting Geoscience Australia.

Code availability

The code used to generate the Barest Earth mosaics from freely available Landsat data and the code to validate results against the spectra of the soil samples collected in the field is available through the GitHub repository of the first author: https://github.com/daleroberts

References

1. Siegal, B. S. & Goetz, A. F. Effect of vegetation on rock and soil type discrimination. Photogramm. Eng. Remote Sens. 43, 191–196 (1977).

2. Murphy, R. J. & Wadge, G. The effects of vegetation on the ability to map soils using imaging spectrometer data. Int. J. Remote Sens. 15, 63–86 (1994).

3. Hewson, R. et al. Using the Geoscience Australia-CSIRO ASTER maps and airborne geophysics to explore Australian geoscience. J. Spat. Sci. 60, 207–231 (2015).

4. Campbell, J. B. & Wynne, R. H. Introduction to Remote Sensing (Guilford Press, 2011).

5. Grebby, S., Cunningham, D., Tansey, K. & Naden, J. The impact of vegetation on lithological mapping using airborne multispectral data: a case study for the north troodos region, cyprus. Remote Sens. 6, 10860–10887 (2014).

6. Brooks, R. R. Geobotany and biogeochemistry in mineral exploration (Harper and Row Publishers, 1972).

7. Ustin, S. L., Smith, M., Jacquemoud, S., Verstraete, M. & Govaerts, Y. Remote Sensing for the Earth Sciences, Manual of Remote Sensing 3rd edn, Vol. 3 (ed Andrew N. Rencz) 189–248 (John Wiley and Sons, 1999).

8. Post, D. F. et al. Relations between soil color and Landsat reflectance on semiarid rangelands. Soil Sci. Soc. Am. J. 58, 1809–1816 (1994).

9. Rowan, L. C. & Mars, J. C. Lithologic mapping in the Mountain Pass, California area using advanced spaceborne thermal emission and reflection radiometer (ASTER) data. Remote Sens. Environ. 84, 350–366 (2003).

10. Hewson, R., Cudahy, T., Mizuhiko, S., Ueda, K. & Mauger, A. Seamless geological map generation using ASTER in the Broken Hill-Curnamona province of Australia. Remote Sens. Environ. 99, 159–172 (2005).

11. Cudahy, T. et al. Satellite-derived mineral mapping and monitoring of weathering, deposition and erosion. Sci. Rep. 6, 23702 (2016).

12. Rowan, L. C., Goetz, A. F. & Ashley, R. P. Discrimination of hydrothermally altered and unaltered rocks in visible and near infrared multispectral images. Geophysics 42, 522–535 (1977).

13. Simpson, C. Deep weathering, vegetation and fireburn Significant obstacles for geoscience remote sensing in Australia. Int. J. Remote Sens. 11, 2019–2034 (1990).

14. Demattê, J. A. M., Fongaro, C. T., Rizzo, R. & Safanelli, J. L. Geospatial Soil Sensing System (GEOS3): a powerful data mining procedure to retrieve soil spectral reflectance from satellite images. Remote Sens. Environ. 212, 161–175 (2018).

15. Rogge, D. et al. Building an exposed soil composite processor (SCMaP) for mapping spatial and temporal characteristics of soils with Landsat imagery (1984–2014). Remote Sens. Environ. 205, 1–17 (2018).

16. Fan, J., Han, F. & Liu, H. Challenges of Big Data analysis. Natl Sci. Rev. 1, 293–314 (2014).

17. Van der Meer, F., Van der Werff, H. & Van Ruitenbeek, F. Potential of ESA’s Sentinel-2 for geological applications. Remote Sens. Environ. 148, 124–133 (2014).

18. Fraser, S. & Green, A. A software defoliant for geological analysis of band ratios. Int. J. Remote Sens. 8, 525–532 (1987).

19. Lewis, A. et al. Rapid, high-resolution detection of environmental change over continental scales from satellite data–the Earth Observation Data Cube. Int. J. Digital Earth 9, 106–111 (2015).

20. Lewis, A. et al. The Australian Geoscience Data Cube — foundations and lessons learned. Remote Sens. Environ. 202, 276–292 (2017).

21. Li, F. et al. A physics-based atmospheric and BRDF correction for Landsat data over mountainous terrain. Remote Sens. Environ. 124, 756–770 (2012).

22. Edgeworth, F. Y. On a new method of reducing observations relating to several quantities. Lond. Edinb. Dublin Philos. Mag. J. Sci. 25, 184–191 (1888).

23. Weber, A. Theory of Location of Industries (translation of 1909 original). (University of Chicago Press, Chicago, 1929).

24. Minsker, S. Geometric median and robust estimation in Banach spaces. Bernoulli 21, 2308–2335 (2015).

25. Godichon-Baggioni, A. Estimating the geometric median in Hilbert spaces with stochastic gradient algorithms: L p and almost sure rates of convergence. J. Multivar. Anal. 146, 209–222 (2016).

26. Cardot, H., Cénac, P. & Zitt, P.-A. Efficient and fast estimation of the geometric median in Hilbert spaces with an averaged stochastic gradient algorithm. Bernoulli 19, 18–43 (2013).

27. Cardot, H., Cénac, P. & Godichon-Baggioni, A. Online estimation of the geometric median in Hilbert spaces: nonasymptotic confidence balls. Ann. Stat. 45, 591–614 (2017).

28. Small, C. G. A survey of multidimensional medians. Int. Stat. Rev. 58, 263–277 (1990).

29. Roberts, D., Mueller, N. & McIntyre, A. High-dimensional pixel composites from earth observation time series. IEEE Trans. Geosci. Remote Sens. 55, 6254–6264 (2017).

30. Lau, I. et al. National Geochemical Survey of Australia reflectance spectroscopy measurements. https://doi.org/10.25919/5cdba18939c29 (2016).

31. Field, D. J., Morgan, C. L. S. & McBratney, A. B. Global Soil Security (Springer International Publishing, 2016).

32. Drury, S. Image Interpretation In Geology (UK: Nelson Thornes 1987).

33. Mars, J. C. & Rowan, L. C. Spectral assessment of new ASTER SWIR surface reflectance data products for spectroscopic mapping of rocks and minerals. Remote Sens. Environ. 114, 2011–2025 (2010).

34. A Ali, E., O El Khidir, S., AA Babikir, I. & M Abdelrahman, E. Landsat ETM+ 7 digital image processing techniques for lithological and structural lineament enhancement: case study around Abidiya area, Sudan. Open Remote Sens. J. 5, 83–89 (2012).

35. Rowan, L. C., Goetz, A. F. & Ashley, R. P. Discrimination of hydrothermally altered and unaltered rocks in visible and near infrared multispectral images. Geophysics 42, 522–535 (1977).

36. Carranza, E. J. M. & Hale, M. Spatial association of mineral occurrences and curvilinear geological features. Math. Geol. 34, 203–221 (2002).

37. Carranza, E. & Hale, M. Mineral imaging with Landsat Thematic Mapper data for hydrothermal alteration mapping in heavily vegetated terrane. Int. J. Remote Sens. 23, 4827–4852 (2002).

38. Netto, J. M., Robbez-Masson, J.-M. & Martins, E. Visible–NIR hyperspectral imagery for discriminating soil types in the La Peyne watershed (France). Dev. Soil Sci. 31, 219–611 (2006).

39. Gomez, C., Lagacherie, P. & Coulouma, G. Regional predictions of eight common soil properties and their spatial structures from hyperspectral Vis–NIR data. Geoderma 189, 176–185 (2012).

40. Mwaniki, M. W., Moeller, M. S. & Schellmann, G. A comparison of Landsat 8 (OLI) and Landsat 7 (ETM+) in mapping geology and visualising lineaments: a case study of central region Kenya. ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences 897–903 (2015).

41. Bierwirth, P. Mineral mapping and vegetation removal via data-calibrated pixel unmixing, using multispectral images. Int. J. Remote Sens. 11, 1999–2017 (1990).

42. Giardino, J. R. & Houser, C. Principles and Dynamics of the Critical Zone. Vol. 19 (Elsevier, 2015).

43. McBratney, A. B., Santos, M. M. & Minasny, B. On digital soil mapping. Geoderma 117, 3–52 (2003).

44. Wilford, J., de Caritat, P. & Bui, E. Predictive geochemical mapping using environmental correlation. Appl. Geochem. 66, 275–288 (2016).

45. Australian Academy of Science. Searching the Deep Earth: A Vision for Exploration Geoscience in Australia (Australian Academy of Science, 2012).

46. Kemperman, J. The median of a finite measure on a Banach space. Statistical Data Analysis Based On The L1-norm And Related Methods 217–230 (Neuchâtel, Birkhäuser Basel 1987).

47. Qiu, B., Zhang, K., Tang, Z., Chen, C. & Wang, Z. Developing soil indices based on brightness, darkness, and greenness to improve land surface mapping accuracy. GISci. Remote Sens. 54.5, 759–777 (2017).

48. Salomonson, V. & Appel, I. Estimating fractional snow cover from MODIS using the normalized difference snow index. Remote Sens. Environ. 89, 351–360 (2004).

49. Zhang, X. et al. MODIS tasseled cap transformation and its utility. In Geoscience and Remote Sensing Symposium, 2002. IGARSS’02. 2002 IEEE International, Vol. 2 1063–1065 (IEEE, 2002).

50. Lopuhaa, H. P. & Rousseeuw, P. J. Breakdown points of affine equivariant estimators of multivariate location and covariance matrices. Ann. Stat. 19, 229–248 (1991).

Acknowledgements

D.R. and J.W. would like to thank Silvio Mezzomo, Chris Evenden and David Arnold at Geoscience Australia (GA) for graphical support, Richard Blewett (GA), Karol Czarnota (GA), Norman Mueller (GA), and Simon Oliver (GA), Geoff Fraser (GA) and Alexis McIntyre for scientific discussions and support, Patrice de Caritat (GA) and Ian Lau for the NGSA site data and spectra. GA’s Digital Earth Australia team for their ongoing infrastructure support and (in particular) David Gavin and Harshu Rampur for preparing and hosting the products on nationalmap.gov.au. This paper is published with the permission of the CEO, Geoscience Australia. The research is a product of the ‘Exploring for the Future’ project.

Author information

Authors

Contributions

J.W. posed the problem of obtaining a high-quality barest Earth mosaic of Australia. D.R. designed the mathematical approach to solve the problem, wrote the codes, ran the codes on the continental-scale satellite imagery dataset and performed the validation against the NGSA dataset. J.W. and D.R. jointly contributed to figure preparation, results interpretation and writing the final manuscript. O.G. helped with data collection for test areas, preparation of some figures and evaluating various weighting schemes over the test areas.

Corresponding authors

Correspondence to Dale Roberts, John Wilford or Omar Ghattas.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review information Nature Communications thanks Yichun Xie and the other, anonymous, reviewer for their contribution to the peer review of this work. Peer reviewer reports are available.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

Roberts, D., Wilford, J. & Ghattas, O. Exposed soil and mineral map of the Australian continent revealing the land at its barest. Nat Commun 10, 5297 (2019). https://doi.org/10.1038/s41467-019-13276-1

• Accepted:

• Published:

• DOI: https://doi.org/10.1038/s41467-019-13276-1

• Bare Earth’s Surface Spectra as a Proxy for Soil Resource Monitoring

• José A. M. Demattê
• José Lucas Safanelli
• Caroline Jardim da Silva Lisboa

Scientific Reports (2020)


text:
# Simple Socket Server in C for Windows

I have taken the code from a tutorial of Microsoft webpage. Of course, the example is for illustration only but I wanted to organize it and make it clear in order to understand it a little more.

``#include <stdio.h> #include <winsock2.h> #include <ws2tcpip.h>  #pragma comment (lib, "Ws2_32.lib")  #define DEFAULT_PORT "27016" #define DEFAULT_BUFLEN 512  int receiveBuffer(SOCKET *ClientSocket) {     char recvbuf[DEFAULT_BUFLEN];     int bytesReceived, iSendResult;     int recvbuflen = DEFAULT_BUFLEN;      // Receive until the peer shuts down the connection     do {         bytesReceived = recv(*ClientSocket, recvbuf, recvbuflen, 0);         if (bytesReceived > 0) {             printf("Bytes received: %d\n", bytesReceived);             // Echo the buffer back to the sender             iSendResult = send(*ClientSocket, recvbuf, bytesReceived, 0);             if (iSendResult == SOCKET_ERROR) {                 printf("send failed: %d\n", WSAGetLastError());                 return 1;             }             printf("Bytes sent: %d\n", iSendResult);         } else if (bytesReceived == 0)             printf("Connection closing...\n");         else {             printf("recv failed: %d\n", WSAGetLastError());                      return 1;         }     } while (bytesReceived > 0);      // shutdown the connection since we're done     bytesReceived = shutdown(*ClientSocket, SD_SEND);     if (bytesReceived == SOCKET_ERROR) {         printf("shutdown failed with error: %d\n", WSAGetLastError());                 return 1;     }     return 0; }  SOCKET acceptSocket(SOCKET *ListenSocket) {     // Accept a client socket     SOCKET ClientSocket = accept(*ListenSocket, NULL, NULL);      if (ClientSocket == INVALID_SOCKET) {         printf("accept failed with error: %d\n", WSAGetLastError());         //closesocket(ListenSocket);         //WSACleanup();         return INVALID_SOCKET;     }     return ClientSocket; }  int listenSocket(SOCKET *ListenSocket, int backlog) {     if ( listen( *ListenSocket, backlog ) == SOCKET_ERROR ) {         printf( "Listen failed with error: %ld\n", WSAGetLastError() );         //closesocket(ListenSocket);                 return SOCKET_ERROR;     }     return 0; }  SOCKET createSocket(struct addrinfo *result) {        SOCKET ListenSocket = INVALID_SOCKET;     ListenSocket = socket(result->ai_family, result->ai_socktype, result->ai_protocol);     if (ListenSocket == INVALID_SOCKET) {         printf("Error at socket(): %ld\n", WSAGetLastError());         //freeaddrinfo(result);              return INVALID_SOCKET;     }     return ListenSocket; }  int bindSocket(SOCKET *ListenSocket, struct addrinfo *result) {     int iResult = bind( *ListenSocket, result->ai_addr, (int)result->ai_addrlen);         if (iResult == SOCKET_ERROR) {         printf("bind failed with error: %d\n", WSAGetLastError());         return SOCKET_ERROR;     }     return 0; }  int initializeWinSock() {     WORD wVersionRequested;     WSADATA wsaData;     int err;     /* Use the MAKEWORD(lowbyte, highbyte) macro declared in Windef.h */     wVersionRequested = MAKEWORD(2, 2);     err = WSAStartup(wVersionRequested, &wsaData);     if (err != 0) {         /* Tell the user that we could not find a usable */         /* Winsock DLL.                                  */         printf("WSAStartup failed with error: %d\n", err);         return 1;     } }  int prepareSocket(SOCKET *ListenSocket, SOCKET *ClientSocket, struct addrinfo *result) {     initializeWinSock();      struct addrinfo hints;      int iResult;     int err;      ZeroMemory(&hints, sizeof (hints));     hints.ai_family = AF_INET;     hints.ai_socktype = SOCK_STREAM;     hints.ai_protocol = IPPROTO_TCP;     hints.ai_flags = AI_PASSIVE;      // Resolve the local address and port to be used by the server     iResult = getaddrinfo(NULL, DEFAULT_PORT, &hints, &result);     if (iResult != 0) {         freeaddrinfo(result);         printf("getaddrinfo failed: %d\n", iResult);                 return 1;     }      *ListenSocket = createSocket(result);     //SOCKET ClientSocket = INVALID_SOCKET;      err = bindSocket(ListenSocket, result);     freeaddrinfo(result);     if(err != 0) {         return 1;     }      err = listenSocket(ListenSocket, SOMAXCONN);     if(err != 0) {         return 1;     }      *ClientSocket = acceptSocket(ListenSocket);      if(*ClientSocket == INVALID_SOCKET) {         return 1;     }      err = receiveBuffer(ClientSocket);       if(err != 0) {         return 1;     } }  int main(int argc, char const *argv[]) {     struct addrinfo *result = NULL;     SOCKET ListenSocket = INVALID_SOCKET;     SOCKET ClientSocket = INVALID_SOCKET;      int err = prepareSocket(&ListenSocket, &ClientSocket, result);      WSACleanup();     closesocket(ListenSocket);     closesocket(ClientSocket);      if(err != 0) {         printf("There is an error, we're trying to solve it, goodbye\n");         return 1;     }      return 0; }  $$````$$ ``