text:
# geometry – X,Y,D,E collinear in simple incircle configuration with a right triangle

There is a right triangle $$ABC$$ with right angle at $$A$$. Let the incircle touch $$AB$$ and $$AC$$ at $$D$$ and $$E$$, respectively. $$X$$ and $$Y$$ are the arc midpoints of minor arc $$AB$$ and $$AC$$ on $$(ABC)$$. Show that $$X,Y,D,E$$ collinear.

The lemma above is a step in in proving a different result which won’t be provided here. I am sure there will be an easy proof to the above, but I appear to be having difficulty in showing that it is true. Sorry for the lack of information about what I have already tried, but indeed I have not made too much progress; thanks in advance.

format: plain


text:
## Define: nonblocking vs asynchronous

• blocking – does the call occupy the stack
• synchronous – how messages are delivered
• does the call deliver the message
• alternatively does the call achieve the effect

Common code for demonstration of possibilities of (non)blocking (a)synchronous behavior, do() is the studied call.

## Insights

• Blocking means current threadexecution container cannot progress with the routine’s code (considering the routine itself). Effectively it may be a long computing subroutine call or being unscheduled(e.g. blocking I/O).
• Multithreading Mutliple execution containers are quite orthogonal concept to blocking.
• You may use other execution containers to achieve non-blocking behavior even with time consuming subroutines.
• blocking, synchronous -> non-blocking, synchronous
• e.g. use other thread for calculation, assign a future to receiver
• Note that with preemptive scheduling you must synchronize access to shared data (receiver, queue).

## Asynchronous APIs

I think there are following asynchronous APIs:

• callbacks,
• blocking wait calls,
• polling
• and message passing?.

### Polling

On the line marked with XXX, we assume uninterrupted flow in order to consequently work with a valid result. This may be achieved by non-preemptive scheduling or using condition variables.

### Message passing

This approach requires some encapsulation of communication entities. FIXME it’s just a different view on previous approaches.

## Events

• Events form an intermediate link between caller and callee/sender and receiver.
• It allows implementing both synchronous and asynchronous behavior.
• Qt’s direct and queue connections (also sent and posted events).
• It may also allow runtime (dis)connection of senders/receivers.
• Generally we want to avoid blocking in receivers/callees as it blocks whole program or event loop (in async scenario).
• Partial remedy is calling processEvents (or similar) from within the blocking call.

### Events without loop (synchronous events)

Calling raise_event is synchronous and depending on the actual listeners, it may also be blocking.

Note that if a raised event raises another instance of itself it may lead to stack overflow.

### Events with loop (asynchronous events)

Calling raise_event is asynchronous and typically non-blocking (for blocking call, one would perform the computation inside raise_event and store result to the queue, which would be later assigned during process_event call).

This approach brings another level of versatility since each execution container can have it’s own queue (access to which needs to be correctly synchronized).

Shown only modification to the code above (the rest is same).

Or as Philip Roberts says it in his JsConf talk:

Look at the stack and the task queue. If the stack is empty, take task from the task queue.


text:
## College Algebra (10th Edition)

We know the following equation for an ellipse: $\frac{(x-h)^2}{a^2}+\frac{(y-k)^2}{b^2}=1$ Where h,k is the center of the ellipse. Thus, the center of this ellipse is: (0,0)


text:
# princomp¶

## Purpose¶

Computes principal components of a data matrix.

## Format¶

{ p, v, a } = princomp(x, j)
Parameters: x (NxK data matrix) – $$N > K$$, full rank. j (scalar) – number of principal components to be computed ($$j <= K$$). p (NxJ matrix) – the first j principal components of x in descending order of amount of variance explained. v (Jx1 vector) – fractions of variance explained. a (JxK matrix) – factor loadings such that: $x = p*a + error.$

## Remarks¶

Adapted from a program written by Mico Loretan.

The algorithm is based on Theil, Henri “Principles of Econometrics.” Wiley, NY, 1971, 46-56.

## Example¶

// Create matrix with percent return
// of 4 stocks over 11 time periods
pcnt_return = {  0.0646   1.2326   0.0508  -0.0346,
-0.1632   0.1806   0.1104   0.1276,
1.3477   1.3347   0.1424   0.0159,
-0.4465  -0.5691  -0.1524  -0.1719,
1.6232   1.4690  -0.0192   0.0979,
0.3381   0.5307   0.0610   0.0374,
-0.0383   0.2556   0.0370   0.0518,
0.4493   0.3140   0.0177   0.1001,
0.5896   0.0542   0.1991   0.2669,
-0.2218   0.3772   0.1189   0.1234,
1.1778  -0.0464  -0.1282   0.2171 };

// Compute: all 4 principal component vectors,
//         percent variance explained
{ p, v, a } = princomp(pcnt_return, 4);

After the code above:

p =   0.2662  -0.6077   0.0965  -0.2951     v = 0.8394     a = 2.4244   2.3264   0.1321   0.2227
0.0059  -0.1702  -0.3938  -0.1569         0.1436         0.9506  -0.9977  -0.1402   0.1566
0.5631  -0.0350   0.0953   0.6146         0.0144         0.0317   0.0153  -0.2757  -0.3420
-0.2170   0.0709   0.4012  -0.0219         0.0026         0.0208  -0.0188   0.1455  -0.1162
0.6491   0.0491   0.2359  -0.2269
0.1823  -0.1085  -0.0554   0.0444
0.0456  -0.1485  -0.1299  -0.1762
0.1624   0.0654  -0.1032  -0.1584
0.1445   0.2677  -0.6520   0.2598
0.0337  -0.3008  -0.3926  -0.2472
0.2447   0.6267  -0.0115  -0.5214

From the results above, we can see that approximately 83.9% of the variance in the pcnt_return is included in the first principal component vector and another 14.36% is included in the second principal component.


text:
# Function Definition¶

In Python, a function is a chunk of code that performs some operation that is meaningful for a person to think about as a whole unit. Once a function has been defined and you are satisfied that it does what it is supposed to do, you will start thinking about it in terms of the larger operation that it performs rather than the specific lines of code that make it work.

In this chapter you will learn about named functions, functions that can be referred to by name when you want to execute them.

The syntax for creating a named function, a function definition, is:

def name( parameters ):
statements

You can make up any names you want for the functions you create, except that you can’t use a name that is a Python keyword, and the names must follow the rules for legal identifiers that were given previously. The parameters specify what information, if any, you have to provide in order to use the new function. Another way to say this is that the parameters specify what the function needs to do it’s work.

There can be any number of statements inside the function, but they have to be indented from the def. In the examples in this book, we will use the standard indentation of four spaces. Function definitions are the third of several compound statements we will see, all of which have the same pattern:

1. A header line which begins with a keyword and ends with a colon.
2. A body consisting of one or more Python statements, each indented the same amount – 4 spaces is the Python standard – from the header line.

We’ve already seen the for statement which has the same structure, with an indented block of code, and the if, elif, and else statements that do so as well.

In a function definition, the keyword in the header is def, which is followed by the name of the function and some parameter names enclosed in parentheses. The parameter list may be empty, or it may contain any number of parameters separated from one another by commas. In either case, the parentheses are required.

We will come back to the parameters in a little while, but first let’s see what happens when a function is executed, using a function without any parameters to illustrate.

Here’s the definition of a simple function, hello.

docstrings

If the first thing after the function header is a string (some tools insist that it must be a triple-quoted string), it is called a docstring and gets special treatment in Python and in some of the programming tools.

Another way to retrieve this information is to use the interactive interpreter, and enter the expression <function_name>.__doc__, which will retrieve the docstring for the function. So the string you write as documentation at the start of a function is retrievable by python tools at runtime. This is different from comments in your code, which are completely eliminated when the program is parsed.

By convention, Python programmers use docstrings for the key documentation of their functions.

Next Section - Function Invocation


text:
# How to teach so that your student doesn't feel stupid?

I'll start by admitting I'm not a teacher. But I am a professional computer programmer with over 20 years of experience and a BS in computer science. I'm also a dad. My daughter is 9 years old and is about to finish 3rd grade.

She does very well in school, her favorite subject is math and for the last year she's participated in a computer club. She enjoys it immensely and is at the top of her class there -- but no thanks to me.

I've tried to talk to her about programming many times, but somehow it just doesn't work. I don't push it either. Recently she made her way into a programming competition final, and we tried to study the previous year's questions for practice. It was difficult. For some reason she kept getting discouraged.

I talked about this to my wife who is also a computer programmer (hey, it runs in the family!) and with whom I've also always had problems whenever I tried to explain her something programming-related. She helped me put the problem into words - at least somewhat.

The idea is that I somehow make it sound... too easy? Like it should be obvious. But when it doesn't immediately click, it makes the listener feel stupid. And that's when learning stops and inecurities kick in.

I do try to be as encouraging as I can. I'm never impatient, I don't chastise (or give any negative feedback at all) and if the first explanation doesn't click, I try to come up with another and another and another. But when you know the subject very well then it's hard to imagine what it's like for someone who doesn't, and to understand just what doesn't click -- especially if they can't explain it themselves.

Atr there any tricks or methods that can help me here? I'd love to teach the subject that we both love to my daughter, but this is an obstacle that I just don't see how to overcome.

Teaching someone you have a close relationship with can be difficult or impossible for many people. And it isn't just an issue on the "teacher" side of the relationship but is also evident in possible psychological effects on the "student" side. It is best to recognize this and may be necessary to avoid such attempts. I know that I've found similar difficulties and have given some thought to why they occur.

On the one hand, your relationship makes it difficult for you to see why your child doesn't get it - she is obviously very smart. Your kid after all and we always think our kids are best. It is hard to recognize that effect, even when it is very strong. Compounding this is that you do understand all of this stuff, but forget that there was a time when you were totally lost about how it all worked. Now that it all fits you don't remember a time when it was just a bunch of disjoint pieces. This too can be very strong. So you get frustrated. And while you are sure that you aren't expressing dissatisfaction, are you really sure you can control the small muscles in your face that others can easily interpret?

On the other hand, your "student" wants more than anything to please you, but can't and so also gets frustrated. The frustration leads to avoidance and, if you keep pushing, further frustration and avoidance. A vicious circle. Few kids like to feel that they are a "disappointment" to a parent. (The same is true for other kinds of relationships in many cases.)

A third element is that you may not completely understand that the mind of a nine year old is different (in kind) from that of an adult. In a few years this will change, and may have begun already. The studies of Piaget explain all of this. Kids learn differently than adults, with logic being much less important.

I can make a couple of suggestions that might help. I doubt that this is a complete list, of course.

The first is to just give it up and hire a tutor who is familiar with both the topic and the age group. You can tutor the tutor if you like if they are not quite as advanced as yourself.

The second idea, speculative at best, is to create a group of kids your child's age and teach them all. I'd suggest four, say, as the right size. Now you have a more formal, less fraught, relationship with the class and, if you can manage to treat them all the same, many of the psychological effects of teaching your child will be lessened (not completely relieved). I would, myself, make their learning group based, rather than individual, with lots of small group projects. At that age some non computer activities are likely appropriate. Use lots of analogies and metaphors in teaching, also. You could also, somehow, encourage student vs teacher mindset so that your kid is encouraged to separate psychologically from you and join with her "classmates." This might actually make it easier and less likely that she will feel that she is not disappointing you (a big deal for a kid) by not "getting it" immediately.

• Yay, she got 3rd place! :) Anyways, I get what you are saying. The idea of "being a disappointment" hadn't crossed my mind but it fits the bill. I've never pressured her and whenever I see that she becomes unhappy I just stop and move on to something else. So I don't think I've got a vicious circle yet -- but I'll keep this in mind for the future. As for your suggestions - as I said, she's already going to a group that has a good teacher, and I don't think that we need a private tutor, so, basically, I'll just stop trying to teach her unless she comes to me first. Thanks! :) – Vilx- May 18 at 17:21
• I'll see if something else comes up, and if it doesn't within a day or two, I'll accept your answer. – Vilx- May 18 at 17:25
• It is good to wait for a while before accepting answers. Some people who want to answer will also want to reflect a bit before actually writing. But note, also, that you can withdraw and re-assign acceptance if it comes to that. – Buffy May 18 at 19:43
• I know. But I also don't want to scare other people off, who might not answer if the question already has an accepted answer. :) – Vilx- May 18 at 21:45

The idea is that I somehow make it sound... too easy? Like it should be obvious. But when it doesn't immediately click, it makes the listener feel stupid. And that's when learning stops and inecurities kick in.

This reminds me of the first time I was teaching an intro to programming class and I was going through a live demo of how some code worked. I mistakenly had an error in my code and got a compilation error. One of my students gasped and said, "You get those too?!"

Previously, it hadn't occurred to me that students thought that people who are "good" at programming didn't get errors. They thought there was some inherent quality that I had that prevented me from making errors like they did.

I'd recommend modeling the problem-solving process. Try doing one of the problems yourself and just let her watch and ask questions. Think aloud and describe everything that goes through your head. Even try explaining different approaches you might consider in solving the problem. It helps demystify the problem solving process of programming when novices get to witness your thinking (and debugging) process.

• Actually, it is useful to intentionally make a few mistakes when programming in front of a class. Back in the day a useful student exercise was to write a program that generated every Fortran diagnostic. Another useful exercise was (and remains) giving students a program that had certain well designed errors (some syntactic, others deeper) with instructions to repair the program. Such a program is called a Fixer Upper in Pedagogical Patterns circles. – Buffy May 19 at 13:23
• @Buffy Is there, somewhere, a collection of exercises fitting this pattern? I find them maddeningly hard to design. – Ben I. May 19 at 15:53
• Hmm... interesting ideas for my metaphorical toolbox. While I doubt they alone will solve my problem, they are nonetheless very useful. – Vilx- May 19 at 17:16
• @BenI., actually they aren't that hard to create. I've used them as a first exercise to give a quick overview of a language and a program in the language. Take any program that you are proud of and that has great structure. Make it a bit longer (twice) than you would ask them to write at that stage. Throw in a couple of logic errors and a few syntax errors. In Java, make sure you have javadocced the intent of each part. Don't include basic structural errors at that point. Probably tell them how many errors it has. You want them to see a few compiler messages and you want them to get (cont) – Buffy May 19 at 18:55
• output that is not quite right. Off by one errors for example, or even run time errors if not too complex to fix. With six errors and ten minutes expected per error you have an exercise. It is fine if one of the errors is a bit tricky, but not all of them. – Buffy May 19 at 18:56

The idea is that I somehow make it sound... too easy? Like it should be obvious. But when it doesn't immediately click, it makes the listener feel stupid.

# Dismissive Language

It's common to want to put your student at-ease by saying things like this:

• "You just need to transmogrify the jabberwocky."
• "All you need to do is reverse the polarity on the tachyon emitters."
• "After you call the recursive stochastinator, the rest is easy."
• "Of course, the next step is to liquify the whatchamacallit."

Humans use this kind of language to reassure the person we're talking to. We're trying to say "don't worry, it's not as bad as you think it is!" - the problem is that this usually has the exact opposite effect.

By telling somebody that something should be easy, we're alienating them when it's not easy, which in computer science is quite often. "My teacher told me this should be easy, but I'm really confused, I guess I'm just really bad at it."

Mathematical Microaggressions discusses this language. It's from a math perspective, but it's a good primer.

To improve this, the first thing I'd do is monitor your own language. Maybe record yourself speaking and then listen back to it.

# The Process

As others have said, it's also important for you to demonstrate the process of solving a problem rather than jumping straight to the answer.

When teaching something, especially if you have more experience than the topic you're teaching, it can be tempting to lay it all out at once. "To get the total of an array of integers, you'd track the sum in a variable, then use a for loop to iterate over the array and increment the variable by the value at each index."

There are a couple problems here:

1. You haven't checked in with your student, so they're probably lost. "I stopped understanding what you were saying about 3 words in..."
2. You haven't shown them how you obtained this information in the first place. "How am I supposed to remember all of that?"
3. You haven't demonstrated what to do when things go awry, or when they encounter a slightly different problem. "Okay now I've memorized how to compute a sum, but what about an average?"

To improve this, I think it's important to show your student how you got this information, and what steps they should follow when they're facing something similar by themselves. This should also include steps like planning, getting error messages (which is a totally normal part of programming, but is often skipped over when teaching), and debugging.

"Let's start about by coming up with a goal. What do we want to do? Okay, how can we break that down into smaller steps? Now let's focus on the first step. Where can we find more information? Oh, we got an error message, that's totally normal! Let's read through it to figure out what it's saying..."

Shameless self-promotion: I wrote up a little meta-tutorial on this process here.

# Engage

I also think it's important to make sure you're meeting students where they live. Specifically, I would make sure that your student is interested in whatever topic you're discussing, or that you at least "brand" the topic with something personal to them. If you're talking about getting the sum of an array of integers, make that the total number of <insert thing your student is interested in>.

This can all feel pretty tedious, especially if you have more experience than your students. It's tempting to jump to the "interesting stuff" without making sure your students are along for the journey.

But teaching is a different skill than writing code. And like any skill, the best way to improve is through practice.

You could also take a step back and try to find a set of tutorials that you and your daughter can enjoy together, instead of putting all of the teaching on yourself. Coding Train has a bunch of beginner-friendly videos that might be fun.

• Nice! I'll try to remember all this the next time when/if get to teaching her. :) – Vilx- Jun 11 at 7:22

You got some good specific answers already. Here are a couple of small points that could be useful when teaching in general:

• Without seeing a transcript of one of your explain-the-programming sessions, I can't say for sure, but it's possible that your word choice is also a factor.

An example: Something I run into a lot is the word just, as in "What is a for-loop?" "You're just telling the program to do something a certain number of times." Just makes some people feel relieved ("oh, that's not too complicated, I can do that"), but it makes other people (or even the same people under different circumstances) feel discouraged ("why didn't I figure that out myself?" / "if it's so simple, why do I still not get it?"), even if your tone is not condescending or pitying.

(This is one of my bad habits. I've been trying to only use just when I feel it's a good choice for the student/situation rather than all the time.)

• I also can't say for sure whether this next point applies to you in particular, but I often find that when people (including me) make something sound easy or obvious, it's because we're explaining how to do something rather than guiding the student through their work. It's like giving a kid a sheet with advice and tips on how to build a big tower ("here are all the facts I've gathered over the years that you should know" / "these are beginner mistakes" / the idea that to be good, you can't make these mistakes) versus working through why their current tower keeps falling down (also gives you the opportunity to explain that you understand whatever the erroneous line of reasoning is and that maybe anyone could make that mistake).

That might not apply to you at all. I don't know.

• I think it might. Thanks! :) – Vilx- Jun 11 at 7:16

# Encourage and model mistakes

First see what Carol Dweck has to say on mistakes https://www.ted.com/talks/carol_dweck_the_power_of_believing_that_you_can_improve

Then model mistakes, encourage you daughter to make mistakes, give her space to experiment (give a learning-goal, but give her space to do it on her own).

(this is not a full answer, I am only adding one point to what is already here)

• I've heard about these ideas before, and I agree with them. I've even had a bit of a personal experience with this problem as well in my own youth. And I am trying to impart this wisdom to my daughter as well, but I don't think it's going too well. She's still fairly afraid to make mistakes and gets discouraged by them. Are there any practical tips for making people more confident about making mistakes and putting effort into improving? I'm kind of at a loss here, besides offering a lot of "don't worry, it's no biggie, you'll do better next time" kind of encouragement. – Vilx- May 21 at 8:42
• encourage strategy and effort (not outcome), encourage mistakes (If you don't make any mistakes, then what you are doing is too easy). – ctrl-alt-delor May 21 at 18:26

Good book by father & son team: "Hello World!", 2nd edition, Warren and Carter Sande, Manning , https://www.manning.com/books/hello-world-second-edition

• That's all very well and nice, but as far as programming itself is concerned, I already have that knowledge. How can this book help me to teach my daughter in a way she feels comfortable with? – Vilx- May 25 at 16:51
• It probably won't help you, but you can give her the book as a present. – Pascal May 26 at 9:13

I just simply give you one advice just don't put the burden of work on your students just give them as much work as they can do. And try to do the task step by step in lecture and give them free hand to approach you anytime.

According to me this is the best thing to do. In this way you get attached with your students even with the low ones.

• Did you read the question, or just the title? – Vilx- May 19 at 21:52


text:
User:

Documento sin título
 Pre-algebra Arithmetics Integers Divisibility Decimals Fractions Exponents Percentages Proportional reasoning Radical expressions Graphs Algebra Monomials Polynomials Factoring Linear Equations Graphs of linear equations Rectangular Coordinate System Midpoint Formula Definition of Slope Positive and negative slope Determine the slope of a line Equations of lines Equation of lines (from graph) Applications of linear equations Inequalities Quadratic equations Graphs of quadratic equations Absolute Value Radical expressions Exponential equations Logarithmic equations System of equations Graphs and functions Plotting points and naming quadrants Interpreting Graphs Relations and Functions Function Notation Writing a Linear Equation from a Table Writing a Linear Equation to describe a Graph Direct Variation Indirect Variation Domain and range Sequences and series Matrices Inverse of a matrix Determinants Inner product Geometry Triangles Polygons 2-D Shapes 3-D Shapes Areas Volume Pythagorean Theorem Angles Building Blocks Geometry Transformations Parallel, coincident and intersepting lines Distances in the plane Lines in space Plane in space Angles in the space Distances in the space Similarity Precalculus Sequences and series Graphs Graphs Definition of slope Positive or negative slope Determine the slope of a line Equation of a line (slope-intercept form) Equation of a line (point slope form) Equation of a line from graph Domain and range Quadratic function Limits (approaches a constant) Limits (approaches infinity) Asymptotes Continuity and discontinuities Parallel, coincident and intersepting lines Introduction to Functions Limits Continuity Asymptotes Trigonometry Trigonometric ratios The reciprocal trigonometric ratios Trigonometric ratios of related angles Trigonometric identities Solving right angles Law of sines Law of cosines Domain of trigonometric functions Statistics Mean Median Mode Quartiles Deciles Percentiles Mean deviation Variance Standard Deviation Coefficient of variation Skewness kurtosis Frequency distribution Graphing statistics & Data Factorial Variations without repetition Variations with repetition Permutations without repetition Permutation with repetition Circular permutation Binomial coefficient Combinations without repetition Combinations with repetition

Trigonometry
Trigonometric identities

The following equations are the most basic and important trigonometric identities. These equations are true for any angle. From them, countless additional identities can be formed.

Basic trigonometric identities:

 tanθ = sinθ cosθ cotθ = _1_ tanθ secθ = _1_ cosθ cscθ = _1_ sinθ

Pythagorean Identities:
a) sin2A + cos2 A = 1

b) 1 + tg2 A = sec2A
c) 1 + cot2 A = csc2A

You can use the trigonometric identities given above to find trigonometric ratios if you are given the quadrant an angle lies in and the value of one trigonometric ratio.

It will be useful to remember the sign of the trigonometric ratios in each quadrant:

If sinθ=$\frac{3}{5}$ and , find cosθ and tanθ.

cos2θ=1-sin2θ=1-($\frac{3}{5}$)2=$\frac{16}{25}$, then cosθ=-$\frac{4}{5}$ because

If the cotangent of an angle is -1.555 and the angle is in the fourth quadrant


text:
# American Institute of Mathematical Sciences

## Applications of mathematics to maritime search

 1 School of Computer and Computing, Zhejiang University City College, Hangzhou 310015, China 2 Department of investigation, Zhejiang Police College, Hangzhou 310053, China 3 School of Business, Zhejiang University City College, Hangzhou 310015, China 4 School of Business and Economics, Australian National University, Acton ACT 2061, Australia

* Corresponding author: Jinming Zhang

Received  October 2017 Revised  January 2018 Published  November 2018

The issue of searching missing aircraft is valuable after the event of MH370. This paper provides a global optimal model to foster the efficiency of maritime search. Firstly, the limited scope, a circle whose center is the last known position of the aircraft, should be estimated based on the historical data recorded before the disappearance of the aircraft. And Bayes' theorem is applied to calculate the probability that the plane falling in the region can be found. Secondly, the drift of aircraft debris under the influence of wind and current is considered via Finite Volume Community Ocean Model(FVCOM) and Monte Carlo Method(MC), which make the theory more reasonable. Finally, a global optimal model about vessel and aircraft quantitative constraints is established, which fully considers factors including the area of sea region to be searched, the maximum speed, search capabilities, initial distance of the vessels by introducing 0-1 decision variables.

Citation: Jinling Wei, Jinming Zhang, Meishuang Dong, Fan Zhang, Yunmo Chen, Sha Jin, Zhike Han. Applications of mathematics to maritime search. Discrete & Continuous Dynamical Systems - S, doi: 10.3934/dcdss.2019064
##### References:
 [1] A. Adcroft and J.-M. Campin, Rescaled height coordinates for accurate representation of free-surface flows in ocean circulation models, Ocean Modelling, 7 (2004), 269-284. [2] F. Balibrea, On problems of topological dynamics in non-autonomous discrete systems, Applied Mathematics and Nonlinear Sciences, 1 (2016), 391-404. [3] J.-M. Campin, A. Adcroft, C. Hill and J. Marshall, Conservation of properties in a free-surface model, Ocean Modelling, 6 (2004), 221-244. [4] W. Chapline, Estimating the drift of distressed small craft, New London (CT): Coast Guard Alumni Association Bulletin, US Coast Guard Academy, 22 (1960), 39-42. [5] C. Chen, G. Cowles and R. Beardsley, An unstructured grid, finite-volume coastal ocean model: Fvcom user manual, SMAST/UMASSD. [6] M. Fischetti and M. Monaci, Proximity search for 0-1 mixed-integer convex programming, Journal of Heuristics, 20 (2014), 709-731. [7] S. M. Griffies, Fundamentals of Ocean Climate Models, vol. 518, Princeton university Press, 2004. [8] U. C. Guard, Us coast guard addendum to the united states national search and rescue supplement to the international aeronautical and maritime search and rescue manual, United States Coast Guard Publication. [9] W. Guo, Z. Zhu and Y. Hou, Bayesian network based cooperative area coverage searching for uavs, in Frontiers in Computer Education, Springer, 2012, 611–618. [10] E. Hernández, M. Carreras, J. Antich, P. Ridao and A. Ortiz, A topologically guided path planner for an auv using homotopy classes, in Robotics and Automation (ICRA), 2011 IEEE International Conference on, IEEE, 2011, 2337–2343. [11] M. I. Karavelas, Guarding curvilinear art galleries with edge or mobile guards via 2-dominance of triangulation graphs, Computational Geometry, 44 (2011), 20-51. doi: 10.1016/j.comgeo.2010.07.002. [12] E. Klarreich, In search of bayesian inference, Communications of the ACM, 58 (2015), 21-24. [13] T. M. Kratzke, L. D. Stone and J. R. Frost, Search and rescue optimal planning system, in Information Fusion (FUSION), 2010 13th Conference on, IEEE, 2010, 1–8. [14] K. Li, L. Li, T. Tesfazghi and E. H.-M. Sha, Adaptive and cost-optimal parallel algorithm for the 0-1 knapsack problem, in Parallel, Distributed and Network-Based Processing (PDP), 2011 19th Euromicro International Conference on, IEEE, 2011, 537–544. [15] Y. Nakano, Quasi-monte carlo methods for choquet integrals, Journal of Computational and Applied Mathematics, 287 (2015), 63-66. doi: 10.1016/j.cam.2015.03.026. [16] M. H. Overmars and J. Van Leeuwen, Maintenance of configurations in the plane, Journal of computer and System Sciences, 23 (1981), 166-204. doi: 10.1016/0022-0000(81)90012-X. [17] M. M. Polycarpou, Y. Yang and K. M. Passino, A cooperative search framework for distributed agents, in Intelligent Control, 2001.(ISIC'01). Proceedings of the 2001 IEEE International Symposium on, IEEE, 2001, 1–6. [18] S. Rajasekaran, Efficient parallel hierarchical clustering algorithms, IEEE Transactions on Parallel & Distributed Systems, 497–502. [19] H. R. Richardson and L. D. Stone, Operations analysis during the underwater search for scorpions, Naval Research Logistics Quarterly, 18 (1971), 141-157. [20] M. Rosa and M. Gandarias, Multiplier method and exact solutions for a density dependent reaction-diffusion equation, Appl. Math. Nonlinear Sci, 1 (2016), 311-320. [21] C. A. A. Sanches, N. Y. Soma and H. H. Yanasse, Parallel time and space upper-bounds for the subset-sum problem, Theoretical Computer Science, 407 (2008), 342-348. doi: 10.1016/j.tcs.2008.06.051. [22] R. Smith, J. Dukowicz and R. Malone, Parallel ocean general circulation modeling, Physica D: Nonlinear Phenomena, 60 (1992), 38-61. [23] L. Štrubelj, I. Tiselj and B. Mavko, Simulations of free surface flows with implementation of surface tension and interface sharpening in the two-fluid model, International Journal of Heat and Fluid Flow, 30 (2009), 741-750. [24] B.-F. Wang, S.-C. Ku and K.-H. Shil, Cost-optimal parallel algorithms for the tree bisector and related problems, Parallel and Distributed Systems, IEEE Transactions on, 12 (2001), 888-898.

show all references

##### References:
 [1] A. Adcroft and J.-M. Campin, Rescaled height coordinates for accurate representation of free-surface flows in ocean circulation models, Ocean Modelling, 7 (2004), 269-284. [2] F. Balibrea, On problems of topological dynamics in non-autonomous discrete systems, Applied Mathematics and Nonlinear Sciences, 1 (2016), 391-404. [3] J.-M. Campin, A. Adcroft, C. Hill and J. Marshall, Conservation of properties in a free-surface model, Ocean Modelling, 6 (2004), 221-244. [4] W. Chapline, Estimating the drift of distressed small craft, New London (CT): Coast Guard Alumni Association Bulletin, US Coast Guard Academy, 22 (1960), 39-42. [5] C. Chen, G. Cowles and R. Beardsley, An unstructured grid, finite-volume coastal ocean model: Fvcom user manual, SMAST/UMASSD. [6] M. Fischetti and M. Monaci, Proximity search for 0-1 mixed-integer convex programming, Journal of Heuristics, 20 (2014), 709-731. [7] S. M. Griffies, Fundamentals of Ocean Climate Models, vol. 518, Princeton university Press, 2004. [8] U. C. Guard, Us coast guard addendum to the united states national search and rescue supplement to the international aeronautical and maritime search and rescue manual, United States Coast Guard Publication. [9] W. Guo, Z. Zhu and Y. Hou, Bayesian network based cooperative area coverage searching for uavs, in Frontiers in Computer Education, Springer, 2012, 611–618. [10] E. Hernández, M. Carreras, J. Antich, P. Ridao and A. Ortiz, A topologically guided path planner for an auv using homotopy classes, in Robotics and Automation (ICRA), 2011 IEEE International Conference on, IEEE, 2011, 2337–2343. [11] M. I. Karavelas, Guarding curvilinear art galleries with edge or mobile guards via 2-dominance of triangulation graphs, Computational Geometry, 44 (2011), 20-51. doi: 10.1016/j.comgeo.2010.07.002. [12] E. Klarreich, In search of bayesian inference, Communications of the ACM, 58 (2015), 21-24. [13] T. M. Kratzke, L. D. Stone and J. R. Frost, Search and rescue optimal planning system, in Information Fusion (FUSION), 2010 13th Conference on, IEEE, 2010, 1–8. [14] K. Li, L. Li, T. Tesfazghi and E. H.-M. Sha, Adaptive and cost-optimal parallel algorithm for the 0-1 knapsack problem, in Parallel, Distributed and Network-Based Processing (PDP), 2011 19th Euromicro International Conference on, IEEE, 2011, 537–544. [15] Y. Nakano, Quasi-monte carlo methods for choquet integrals, Journal of Computational and Applied Mathematics, 287 (2015), 63-66. doi: 10.1016/j.cam.2015.03.026. [16] M. H. Overmars and J. Van Leeuwen, Maintenance of configurations in the plane, Journal of computer and System Sciences, 23 (1981), 166-204. doi: 10.1016/0022-0000(81)90012-X. [17] M. M. Polycarpou, Y. Yang and K. M. Passino, A cooperative search framework for distributed agents, in Intelligent Control, 2001.(ISIC'01). Proceedings of the 2001 IEEE International Symposium on, IEEE, 2001, 1–6. [18] S. Rajasekaran, Efficient parallel hierarchical clustering algorithms, IEEE Transactions on Parallel & Distributed Systems, 497–502. [19] H. R. Richardson and L. D. Stone, Operations analysis during the underwater search for scorpions, Naval Research Logistics Quarterly, 18 (1971), 141-157. [20] M. Rosa and M. Gandarias, Multiplier method and exact solutions for a density dependent reaction-diffusion equation, Appl. Math. Nonlinear Sci, 1 (2016), 311-320. [21] C. A. A. Sanches, N. Y. Soma and H. H. Yanasse, Parallel time and space upper-bounds for the subset-sum problem, Theoretical Computer Science, 407 (2008), 342-348. doi: 10.1016/j.tcs.2008.06.051. [22] R. Smith, J. Dukowicz and R. Malone, Parallel ocean general circulation modeling, Physica D: Nonlinear Phenomena, 60 (1992), 38-61. [23] L. Štrubelj, I. Tiselj and B. Mavko, Simulations of free surface flows with implementation of surface tension and interface sharpening in the two-fluid model, International Journal of Heat and Fluid Flow, 30 (2009), 741-750. [24] B.-F. Wang, S.-C. Ku and K.-H. Shil, Cost-optimal parallel algorithms for the tree bisector and related problems, Parallel and Distributed Systems, IEEE Transactions on, 12 (2001), 888-898.
The bounding limit of aircraft in distress
Wind-induced drift component diagram
Ocean Surface Zonal Currents-Ocean Surface Meridional Currents(meter/sec)
Sketch of calculation process
Time taken by exhaustive method for optimal solution with quantitative constraint
 Problem size $n$ Number of package $S$ Number of package $t$ 30 $2^{30}$=1073741824 About 1s 50 $2^{50}$=1125899906842624 About 13 days 100 $2^{100}$ About $4\times{}10^{13}$ days
 Problem size $n$ Number of package $S$ Number of package $t$ 30 $2^{30}$=1073741824 About 1s 50 $2^{50}$=1125899906842624 About 13 days 100 $2^{100}$ About $4\times{}10^{13}$ days
Time taken by exhaustive method for optimal solution with quantitative constraint
 Problem size $n$ Number of package $S$ Computation time $t$ 30(20 ships, 10 planes) $\binom{20}{10}\cdot\binom{10}{5}=46558512$ About 0.05 s 50(40 ships, 10 planes) $\binom{40}{10}\cdot\binom{10}{5}=213610453056$ About 4 s 100(90 ships, 10 planes) $\binom{100}{10}\cdot\binom{10}{5}=1441602661439556$ About 400 hours
 Problem size $n$ Number of package $S$ Computation time $t$ 30(20 ships, 10 planes) $\binom{20}{10}\cdot\binom{10}{5}=46558512$ About 0.05 s 50(40 ships, 10 planes) $\binom{40}{10}\cdot\binom{10}{5}=213610453056$ About 4 s 100(90 ships, 10 planes) $\binom{100}{10}\cdot\binom{10}{5}=1441602661439556$ About 400 hours
 [1] Giacomo Dimarco. The moment guided Monte Carlo method for the Boltzmann equation. Kinetic & Related Models, 2013, 6 (2) : 291-315. doi: 10.3934/krm.2013.6.291 [2] Caterina Calgaro, Meriem Ezzoug, Ezzeddine Zahrouni. Stability and convergence of an hybrid finite volume-finite element method for a multiphasic incompressible fluid model. Communications on Pure & Applied Analysis, 2018, 17 (2) : 429-448. doi: 10.3934/cpaa.2018024 [3] Stefan Berres, Ricardo Ruiz-Baier, Hartmut Schwandt, Elmer M. Tory. An adaptive finite-volume method for a model of two-phase pedestrian flow. Networks & Heterogeneous Media, 2011, 6 (3) : 401-423. doi: 10.3934/nhm.2011.6.401 [4] Pavol Kútik, Karol Mikula. Diamond--cell finite volume scheme for the Heston model. Discrete & Continuous Dynamical Systems - S, 2015, 8 (5) : 913-931. doi: 10.3934/dcdss.2015.8.913 [5] Boris Andreianov, Mostafa Bendahmane, Kenneth H. Karlsen, Charles Pierre. Convergence of discrete duality finite volume schemes for the cardiac bidomain model. Networks & Heterogeneous Media, 2011, 6 (2) : 195-240. doi: 10.3934/nhm.2011.6.195 [6] Guillaume Bal, Ian Langmore, Youssef Marzouk. Bayesian inverse problems with Monte Carlo forward models. Inverse Problems & Imaging, 2013, 7 (1) : 81-105. doi: 10.3934/ipi.2013.7.81 [7] Lianzhang Bao, Wenjie Gao. Finite traveling wave solutions in a degenerate cross-diffusion model for bacterial colony with volume filling. Discrete & Continuous Dynamical Systems - B, 2017, 22 (7) : 2813-2829. doi: 10.3934/dcdsb.2017152 [8] Paola Goatin, Sheila Scialanga. Well-posedness and finite volume approximations of the LWR traffic flow model with non-local velocity. Networks & Heterogeneous Media, 2016, 11 (1) : 107-121. doi: 10.3934/nhm.2016.11.107 [9] Jiakou Wang, Margaret J. Slattery, Meghan Henty Hoskins, Shile Liang, Cheng Dong, Qiang Du. Monte carlo simulation of heterotypic cell aggregation in nonlinear shear flow. Mathematical Biosciences & Engineering, 2006, 3 (4) : 683-696. doi: 10.3934/mbe.2006.3.683 [10] Michael B. Giles, Kristian Debrabant, Andreas Rössler. Analysis of multilevel Monte Carlo path simulation using the Milstein discretisation. Discrete & Continuous Dynamical Systems - B, 2017, 22 (11) : 1-23. doi: 10.3934/dcdsb.2018335 [11] Elena Bonetti, Giovanna Bonfanti, Riccarda Rossi. Analysis of a model coupling volume and surface processes in thermoviscoelasticity. Discrete & Continuous Dynamical Systems - A, 2015, 35 (6) : 2349-2403. doi: 10.3934/dcds.2015.35.2349 [12] Christos V. Nikolopoulos, Georgios E. Zouraris. Numerical solution of a non-local elliptic problem modeling a thermistor with a finite element and a finite volume method. Conference Publications, 2007, 2007 (Special) : 768-778. doi: 10.3934/proc.2007.2007.768 [13] Yangang Chen, Justin W. L. Wan. Numerical method for image registration model based on optimal mass transport. Inverse Problems & Imaging, 2018, 12 (2) : 401-432. doi: 10.3934/ipi.2018018 [14] Joseph Nebus. The Dirichlet quotient of point vortex interactions on the surface of the sphere examined by Monte Carlo experiments. Discrete & Continuous Dynamical Systems - B, 2005, 5 (1) : 125-136. doi: 10.3934/dcdsb.2005.5.125 [15] Chjan C. Lim, Joseph Nebus, Syed M. Assad. Monte-Carlo and polyhedron-based simulations I: extremal states of the logarithmic N-body problem on a sphere. Discrete & Continuous Dynamical Systems - B, 2003, 3 (3) : 313-342. doi: 10.3934/dcdsb.2003.3.313 [16] Olli-Pekka Tossavainen, Daniel B. Work. Markov Chain Monte Carlo based inverse modeling of traffic flows using GPS data. Networks & Heterogeneous Media, 2013, 8 (3) : 803-824. doi: 10.3934/nhm.2013.8.803 [17] Mazyar Zahedi-Seresht, Gholam-Reza Jahanshahloo, Josef Jablonsky, Sedighe Asghariniya. A new Monte Carlo based procedure for complete ranking efficient units in DEA models. Numerical Algebra, Control & Optimization, 2017, 7 (4) : 403-416. doi: 10.3934/naco.2017025 [18] Yali Yang, Sanyi Tang, Xiaohong Ren, Huiwen Zhao, Chenping Guo. Global stability and optimal control for a tuberculosis model with vaccination and treatment. Discrete & Continuous Dynamical Systems - B, 2016, 21 (3) : 1009-1022. doi: 10.3934/dcdsb.2016.21.1009 [19] Yukie Goto, Danielle Hilhorst, Ehud Meron, Roger Temam. Existence theorem for a model of dryland vegetation. Discrete & Continuous Dynamical Systems - B, 2011, 16 (1) : 197-224. doi: 10.3934/dcdsb.2011.16.197 [20] Yi Shi, Kai Bao, Xiao-Ping Wang. 3D adaptive finite element method for a phase field model for the moving contact line problems. Inverse Problems & Imaging, 2013, 7 (3) : 947-959. doi: 10.3934/ipi.2013.7.947

2017 Impact Factor: 0.561


text:
## Algebra 1

20 $14-5\times3+3^2$
Use the order of operations to evaluate. $14+5\times3-3^2\longrightarrow$ resolve exponent $=14+5\times3-9\longrightarrow$ multiply $=14+15-9\longrightarrow$ add and subtract working left to right $=29-9\longrightarrow$ subtract $=20$ Changing the addition to subtraction and the subtraction to addition will give a result of 8. $14-5\times3+3^2\longrightarrow$ resolve exponent $=14-5\times3+9\longrightarrow$ multiply $=14-15+9\longrightarrow$ add and subtract working left to right $=-1+9\longrightarrow$ subtract $=8$


text:
Ribose Consensus Protocol
Ribose is a Delegated Proof of Stake (DPoS) consensus protocol. GeneChain uses Ribose to reach network consensus and avoid forking.
Note: This topic is still under construction.

# Brief

Ribose selects 21 validation nodes to produce blocks in turn.

# Concepts

## Validator

Anyone can sign up for an account as a validator by calling register method of the system contract, and only the validator account can accept mortgage in the system contract and run for active validators.

## Stake

Anyone can increase the mortgage weight for any validator by staking their RNA and ARM or one of them at any time. For the weight algorithm, see Mortgage Weight Algorithm. Users can also withdraw their mortgage to get their RNA and ARM back. However, the mortgage needs a lockup period to stabilize the network. It is tentatively set as 86,400 blocks, which is about 72 hours.

### Staking Power

Staking power is calculated as
$\begin{cases} RNA \times \ln{(ARM)},&ARM \geq e\\ RNA,&otherwise\end{cases}$
, where e is the Euler's number and 2.718281828459045236 is used here.

## Candidates

The system contract will record the top 50 validators with total mortgage weight as candidate validators. To ensure that the candidate validator remains active, a validator can be included in the candidate list only when its total mortgage weight changes. Hence, when a validator is removed from the candidate list, other validators must initiate a mortgage to win a place in candidate validators.

## Validator Set

Validator Set is a set of 21 addresses. GeneChain blocks can only be produced by nodes that are running in mining mode and binding to an account within these 21 addresses
Validator Set is updated every 200 blocks. Only top 21 accounts.


text:
Mara Boyd

2023-01-01

For what value of c is the relation a function: (2,8),(12,3), (c,4), (-1,8), (0,3)?

Yaretzi Huynh

Expert

Every x-value in a function has exactly one and only one y-value.
So, $\left\{c\mid c\ne 2,12,-1,0,c\in \mathbb{R}\right\}$.

Do you have a similar question?


text:
### About a complex operator resolvent

Vasiliy I. Fomin
Published: 1 January 2022

Abstract: A normed algebra of bounded linear complex operators acting in a complex normed space consisting of elements of the Cartesian square of a real Banach space is constructed. In this algebra, it is singled out a set of operators for each of which the real and imaginary parts commute with each other. It is proved that in this set, any operator for which the sum of squares of its real and imaginary parts is a continuously invertible operator, is invertible itself; a formula for the inverse operator is found. For an operator from the indicated set, the form of its regular points is investigated: conditions under which a complex number is a regular point of the given operator are found; a formula for the resolvent of a complex operator is obtained. The set of unbounded linear complex operators acting in the above complex normed space is considered. In this set, a subset of those operators for each of which the domains of the real and imaginary parts coincide is distinguished. For an operator from the specified subset, conditions on a complex number under which this number belongs to the resolvent set of the given operator are found; a formula for the resolvent of the operator is obtained. The concept of a semi-bounded complex operator as an operator in which one component is a bounded and the other is an unbounded operator is introduced. It is noted that the first and second resolvent identities for complex operators can be proved similarly to the case of real operators.
Keywords: complex operators / algebra / unbounded / imaginary / formula / normed / resolvent of the operator

#### Scifeed alert for new publications

Never miss any articles matching your research from any publisher


text:
# All Questions

18,380 questions
2 views

### Digital Signature Algorithm for large files - bottle-necked by hash function?

I am new to studying digital signatures like ECDSA and EdDSA, and was curious about something. Lets say hypothetically I want to create a digital signatures of millions of files, with a total size of ...
13 views

### Big-O Encryption Algorithm

I am currently doing a research paper on the Blowfish encryption algorithm and one of the components that I need to include is time and space complexity. I have tried reading academic articles and ...
122 views

### Is GMW different from Secret Sharing?

Is GMW (How to play ANY mental game) protocol different from Secret Sharing?? If so, how is it different from Secret sharing?
21 views

### Is there a standard definition of non-malleability for the encryption schemes?

I find some different definitions of non-malleability for the encryption schemes. They may be equivalent, but I am not sure which one is better or if there is a standard definition. I give two ...
17 views

### PKCS#10 why keeping the Subject Public Key in the X.509 Certificate

In the pkcs#10 workflow to obtain the x.509 certificate used in digital signature, in the Certificate Request Information, itself in the CSR, we have the Subject Public Key. That Subject Public Key ...
164 views

### Generate PGP keys very very fast

Is there a way to generate thousands of PGP keys at a very very fast rate? I am generating thousands of 3072-bit PGP keys looking like part of a personal research project. Obviously, this is a very ...
89 views

### RSA key problem

So I have the cryptographic task. There is a problem with the encryption, and I need to find a way to generate private keys out of the public. There is what I have. The private key generated by ...
82 views

### Low entropy - how does it compromise the crypto

One of the often pointed out mistake in dealing with crypto is not having enough entropy in the system. Without enough entropy the random numbers generated are not random enough. But what I am not ...
152 views

### What are some real world examples where one time pad encryption is used or can be used?

I understood that One-Time Pad (OTP) encryption ensures perfect secrecy. However, I couldn't find any real-world examples where the OTP is used. Also, which are some real-world examples where it won'...
20 views

### Twist on Elliptic Curve defined over an extension field [on hold]

I am trying to compute the quadratic twist for an example of an Elliptic curve defined over a $GF(p^8)$ Field: With p=...
31 views

### BGN encryption scheme with unbounded message space

In Evaluating 2-DNF Formulas on Ciphertexts stated that decryption in this system takes polynomial time in the size of the message space T. Therefore, the system as described above can only be ...
30 views

### Division using Fully Homomorphic Encryption

When I am not allowed to do interaction with server, then I use Taylor approximation for division in non-interactive process. Let's say that I am allowed to do interactions with server. I have ...
57 views

### NIST randomness test p values

I have a binary sequence of length $2^{20}$. I am using the NIST statistical test, assess. What should be taken for ./assess and ...
31 views

### Subtracting one pederssen commitment from another

I am building a range proof to prove that a secret number x lies between a specific range a prover commits to a values a<x<b prover generates a pedersen ...
15 views

### what is the relationship between epsilon and sensitivity in the Differential-Privacy?

In some Differential-Privacy(DP) papers, they use epsilon as the x-axis in the figures of the experiments' result while other papers use the sensitivity. What is the relationship between epsilon and ...
29 views

### How secure is a book derived key and what can be done to strenghten it?

If you have a manual encryption system where the keying is done by extracting a key from a book, a major weakness is that an adversary can search trough all existing books to find a perfect Match and ...
44 views

### How to apply lagrange interpolation on bilinear pairings?

I have seen in some places applications of Shamir secret sharing and lagrange interpolation mixed with bilinear pairings, however I fail to understand how this works. For instance, here I find the ...
12 views

### Django Web API hosted in IIS - shared file acces [on hold]

I have create Django rest api and deployed in IIS.. I am trying to access a shared folder on another server.. And has give access to a domain user which is the Site anonymous user and app pool ...
55 views

### If OAEP is introducing randomness to encryption how could be ciphertext decrypted?

I was studying OAEP and see we introduce a random value $r$ when computing the padding. See the figure below from Wikipedia So if we are adding randomness to encryption, how could this ciphertext be ...
2k views

### 80-bit collision resistence because of 80-bit x87 registers?

This is just a curious question, and it probably doesn't belong here anyway, and I'm just being bold asking it here. 80-bit used to be considered an adequate level of security, Skipjack and SHA1 ...
33 views

### Purpose of the ZKPoPK in the original SPDZ protocol

What is the purpose of ZKPoPK in the SPDZ paper? And will the following can detect the active adversaries in 2 party spdz excluding ZKPoPK step? Each party sends a commitment on his MAC key ∆_i ...
28 views

### Is there any point to encrypting a file if it's transferred via ssh?

Suppose I have two machines which can ssh into each other. Is there very little point in using gpg with a password protected symmetric cipher to transfer a ...
36 views

### What is chain voting?

I read The ThreeBallot Voting System by Rivest. This paper-based voting system can be attacked with chain voting. But can't find any description of what "Chain voting" is mentioned in section 4.9. ...
33 views

### trapdoor commitment from lattice-based assumptions?

I'm wondering that is there any equivocal commitment scheme (i.e., trapdoor commitment) can be constructed from lattice-based assumptions? I know there are a lot of commitment schemes from lattices as ...
28 views

### What is the security model of the FHE system introduced in Fully Homomorphic Encryption Using Ideal Lattices?

How would one construct a security model to play against the adversary, and define the security of the overall scheme? This is in reference to the scheme introduced in "Fully Homomorphic Encryption ...
33 views

### Is computing majority element while doing secure two party computation for inner products ( input of the two parties ) an attack on the protocol?

This was told to be an attack. However, we can create a simulator in the ideal world which sends all 1s to the trusted party and receives the number of ones of the honest party. Is this simulation ...
62 views

### How was the 5 digit random number in the VIC cipher generated?

Knowing that VIC was a "spy cipher" it is unlikely that the agents used a cryptographic device to genreate the 5 digit number but how did they do it?
2k views

### Are encryption algorithms with fixed-point free permutations inherently flawed?

Flaw in Enigma One of the Enigma machine's flaw was the derangement (fixed-point free permutation) of the produced ciphertext, or simply put: No plaintext-letter can be enciphered to itself. See this ...
90 views

### Fermat's factorization method on weak RSA modulus

Given a public key for RSA, I have extracted the modulus which looks like this: ...
486 views

### Truncating the output of AES-128

I'd like to use the output of a pseudorandom function (PRF) as a one time pad. I want to use AES-128 (as it's commonly used as a PRF). As we know AES-128 output 128 bits. Question: Would be secure ...
33 views

### Time-based decryption service

i am thinking about building a public crypto service, yet at the same time, I am still quite new to crypto and therefore prone to snake-oil inventions I guess. My plan is to build a website which ...
69 views

### Elliptic curve ed25519 vs ed448 - Differences

Other than key size, What are some differences between the Elliptic curve ed25519 and ed448?
59 views

### What is the time complexity of computing a cryptographic hash function/random oracle?

I'm wondering what's the computational complexity of computing a hash function/random oracle when doing complexity analysis. For example, what's the computational complexity of computing $H(b\|r)$? ...
49 views

### Does SHA1 require public keys? [on hold]

I want to know if they do, and if they don't which ones don't require public keys.
55 views

### PRG in a stream cipher [on hold]

Alice wants to encrypt a message for Bob using a stream cipher built from two candidate PRGs, call them G1 and G2. The goal is that if either is a secure PRG, the resulting cipher is a secure stream ...
20 views

### Brute atack force affine cipher [on hold]

help me please, I really need the code Program (Pascal or C) for encryption and decryption, the cipher affine whith brute force attack
79 views

### What is the difference between a Vigenère cipher and a one time pad cipher?

I'm doing a decryption competition at a nearby college. We are allowed one cheat sheet so I'm trying to make one. I realized that the Vigenère cipher was very similar to the one time pad and wanted ...
68 views

### What are known metrics for TRNG?

I was wondering what the known metrics are for studying the randomness of TRNG (besides NIST tests). For example, for PUFs, there are known metrics such as uniformity, uniqueness, BER, etc.
31 views

### Which 'info' and 'salt' parameters to use in HKDF for TLS 1.3?

all the following discussion is for HKDF in TLS1.3: Regarding the 'salt' parameter in - HKDF-Extract(salt, IKM) as written in RFC 5869: HKDF without having to protect the secrecy of the salt. In ...
35 views

### UC-framework: Simulating specific cases (proof in PVW)

I am struggling to understand the proof of security in the UC-framework of the OT by PVW. The cases I don't understand are the cases where: i) Both parties are corrupted. In this case, they say ...
60 views

### In cryptocurrency they use Secp256k curve. Can someone generate private key with different curve?

A lot of cryptocurrencies is using Secp256k1 (for example: https://en.bitcoin.it/wiki/Secp256k1). Every cryptocurrency library comes with its own redundant implementation of Secp256k1, ECDSA, ...
63 views

### Definition of symmetric cipher and definition of asymmetric cipher

So I'm currently taking a cryptography class right now and my professor has told the class that she can't even define symmetric cipher for us without getting into asymmetric ciphers and the textbook ...
651 views

### Can you recognize a low private exponent from a public key?

In the RSA cryptosystem, Wiener, Boneh and Durfee showed that low private exponents can be efficiently recovered. Is it possible to see from a public key alone whether the private exponent (d) is ...
34 views

### what type of the cryptographi operation is done? [on hold]

I was reading a cryptography article recently and faced the section which I could not understand it is there anyone to describe it completely thanks
11 views

### Standard deviation of gaussian noise in FHEW scheme

I've got two questions regarding the paper FHEW: Bootstrapping Homomorphic Encryption in less than a second. First, the final error of a ciphertext after the refresh procedure is stated as following ...
61 views

### Hash for use in embedded HMAC application

I developed a GNSS logger (http://flysight.ca) which has become popular for scoring skydiving competitions. At the moment, there is little protection against competitors falsifying tracks, but I would ...
91 views

### Why did MD4 replace MD2?

MD2 was a hash function based on swapping bytes in a state array permutation, much like the RC4 stream cipher, whereas MD4 was a novel construction. MD4 replaced MD2 despite the fact that MD4 is more ...
29 views

### EC non-shared cryptosystems - different group for every party

Efficient Identity Based Parameter Selection for Elliptic Curve Cryptosystems by Arjen K. Lenstra contains a proposal for a non-shared elliptic curve cryptosystem. Every party chooses its own field ...


text:
Encoding categorical features: likelihood, one-hot, and feature selection

This post describes techniques used to encode high cardinality categorical features in a supervised learning problem.

In particular, since these values cannot be ordered, the features are nominal. Specifically, I am working with the Kaggle competition here. The problem with this dataset is that some features (e.g. types of cell phone operating systems) are categorical and has hundreds of values.

The problem occurs in how to fit these features in our model. Nominal features work fine with decision trees (random forests), Naive Bayes (use count to estimate pmf). But for other models, e.g. neural networks, logistic regression, the input needs to be numbers.

Before introducing likelihood encoding, we can go over other methods in handling such situations.

Likelihood encoding

Likelihood encoding is a way of representing the values according to their relationships with the target variable. The goal is finding a meaningful numeric encoding for a categorical feature. Meaningful in this case means as much related to the output/target as possible.

How do we do this? A simple way is 1) first group the training set by this particular categorical feature and 2) representing each value by within group mean of that value. For example, a categorical feature might be gender. Suppose the target is height. Then, we might have the average height for male is 1.70m, while the average height for female is 1.60m. We then change ‘male’ to 1.70, while ‘female’ to 1.60.

Perhaps we should also add some noise to this mean to prevent overfitting to training data. This can be done by :

• add Gaussian noise to the mean. Credit to Owen Zhang :
• use the idea of “cross-validation”. So here, instead of using the grand group mean, we use the cross-validation mean. (Not very clear on this point at the moment. Need to examine the idea of cross-validation. Will write in the next post.) Some people propose on Kaggle about using two levels of cross-validation: https://www.kaggle.com/tnarik/likelihood-encoding-of-categorical-features

One hot vector

This idea is similar to the dummy variable in statistics. Basically, each possible value is being transformed into its own columns. Each of these columns will be a 1 if the original feature equals this value, or 0 if the original feature does not equal this value.

An example is for natural language processing models, the first step is usually 1) tokenize the sentence and 2) constructing a vocabulary and 3) map every token to an index (a number, or a nominal value, basically). After that, we do 4) one hot encoding and 5) a linear transformation in the form of a linear layer in a neural network (basically transform high-dim one hot

After that, we do 4) one hot encoding and 5) a linear transformation in the form of a linear layer in a neural network (basically transform high-dim one hot vectors into low dim vectors). In this way, we are basically representing every symbol in a low dimensional vector. The exact form of the vector is learned. What is happening here is actually dimension reduction.  So, after learning the weighting matrix, other methods, like PCA, can potentially work here as well!

Hashing

A classmate named Lee Tanenbaum told me about this idea. This is an extension on the one-hot encoding idea. Suppose there can be n values in the feature. Basically, we use two hash functions, hash the possible values into two variables. The first hash all values into \sqrt(n) number of baskets,basketh baskent there are \sqrt(n) number of feature values. All feature values in the same busket is going to be the same for variable A. Then, we use a second hash function, that carefully hash the values into another busket variable B. We want to make sure the combination of A and B can fully represent every possible value in the target feature. We then learn a low-dim representation for both A and B, and concantenate them together.

My opinion on this is, this is still one-hot encoding + weight learning. However, we are forcing certain structures onto the weight matrix.

Feature selection

Still based on one-hot encoding. However, instead of compressing everything into a tiny low-d vector, we discard some dummy variables based on their importance. In fact, LASSO is exactly used for this! L1 usually drives the coefficient of some features to zero, due to the diamond shape of the constraint. Source:

• On why l1 gives sparsity: video here :  https://www.youtube.com/watch?v=14MKVkhvMus&feature=youtu.be
• Course here : https://onlinecourses.science.psu.edu/stat857/book/export/html/137 Statoverflow answer here: https://stats.stackexchange.com/questions/74542/why-does-the-lasso-provide-variable-selection

Domain specific methods

These models exploit the relationship between these symbols in the training set, and learn a vector representation of every symbol (e.g. word2vec). This is certainly another way of vectorizing the words. Probably I will write more about this after learning more on representation learning!

Python generators

This short post describes what is a generator in Python.

A function with yield in it is a function. However, when called the function, it returns a generator object. Generators allow you to pause a function and return an intermediate result. The function saves its execution context and can be resumed later if necessary.

def fibonacci():
a, b = 0, 1
while True:
yield b
a, b = b, a + b

g = fibonacci()

[next (g) for i in range(10)]

This will return [1, 1, 2, 3, 5, 8, 13, 21, 34, 55].

When call the list comprehension again, it will return:

[next (g) for i in range(10)]

[89, 144, 233, 377, 610, 987, 1597, 2584, 4181, 6765]

Here, note the function is like a machine that can generate what you want. It will also remember its last state. This is different from a function that returns a list, which will not remember its last state..

Python generators can also interact with the code called with the next method. yield becomes an expression, and a value can be passed along with a new method called sendHere is an example piece of code:

def psychologist():
while True:
answer = (yield) # note the usage of yield here
if answer is not None:
if answer.endswith("?"): # note it's endSwith, the s there
print("Don't ask yourself too many questions.")
elif "good" in answer:
print("A that's good, go on. ")
print("Don't be so negative.")

This defines a function that can return a generator.

free = psychologist()

type(free)

This will return “generator”

next(free)

This will return “Please tell me your problem.”

free.send("what?")

This will return “Don’t ask yourself too many questions.”

free.send("I'm feeling bad.")

This will return “Don’t be so negative.”

free.send("But she is feeling good!")

This will return, “A that’s good, go on.”

send acts like next, but makes yield return the value passed. The function can, therefore, change its behavior depending on the client code.

Should I use an IDE, or should I use Vim?

This problem has been bugging me for a while so I decide to write it out even though it’s just a short piece.  This post compares tools for Python programming using :

• Jupyter Notebook
• IDEs like PyCharm
• Text editors like Vim

Jupyter Notebook:

• The pro is it’s easily visualizable. When you want a graph, you can see a graph immediately. The comments are also beautifully formatted.
• Another pro is it can be connected to a Linux server like Dumbo.
• The con is, it’s not a program. A notebook is it’s own file and although it can be downloaded as a .py file, the file is usually too long, with lots of comments like typesetting parameters.

When to use it?

• Data exploration. because of the visualization and analysis nature.
• Big data. Because it can be connected to a server, that makes running large amount of data possible.

PyCharm

• The pro is it’s suited for Python development. I have not learnt the functionalities entirely, but e.g. search and replace are easily doable in PyCharm. Debugging also seems to be easy?
• The con is it’s usually not available on a server.
• Another con is need extra finger movement when switching from terminal to Pycharm.

When to use it?

• Debugging complicated programs. e.g. NLP programs.
• No need to run on a server.

Vim

• The pro is it’s everywhere. Thus, whenever you write on your own machine, or on the server,   it feels the same.
• Another pro is it can be used for anything. like python, C++, markdown, bash… So there is no need to switch to other places when ssh to the server.
• The con is it’s not that easy to use. e.g. search and replace… hard to do this. Adjust tab? also not immediately doable.
• Another con is it’s not that easy to debug. have to manually print out variables… This makes it particularly difficult when the program is large.

When to use it?

• When need to connect to a server. e.g. big data size.

Chi-square and two-sample t-test

This post explains a basic question I encountered, and the statistical concepts behind it.

The real-life problem

Someone asks me to construct data to prove that a treatment is useful for 1) kindergarten and 2) elementary school kids in preventing winter cold.

Chi-square and student’s t test

First, decide how to formulate the problem using statistical tests. This includes deciding the quantity and statistic to compare.

Basically, I need to compare two groups. Two tests come to mind: Pearson’s chi-square test, and two-sample t-test. This article summarizes main difference between the two tests, in terms of Null Hypothesis, Types of Data, Variations and Conclusions. The following section is largely based on that article.

Null Hypothesis

• Pearson’s chi-square test: test the relationship between two variables, or whether something has effects on the other thing (?). e.g. men and women are equally likely to vote for Republican, Democrat, Others, or Not at all. Here the two variables are “gender” and “voting choice”. The null is “gender does not affect voting choice”.
• Two-sample t-test : whether two sample have the same mean. Mathematically, this means $\mu_1 = \mu_2$ or $\mu_1 - \mu_2 = 0$. e.g. boys and girls have the same height.

Types of Data

• Pearson’s chi-square test: usually requires two variables. Each is categorical and can have many number of levels. e.g. one variable is “gender”, the other is “voting choice”.
• two sample t-test: requires two variables. One variable has exactly two levels (two-sample), the other is quantitively calculatable. e.g. in the example above, one variable is gender, the other is height.

Variations

• Pearson’s chi-square test: variations can be when the two variables are ordinal instead of categorical.
• two-sample t-test: variations can be that the two samples are paired instead of independent.

Transform the real-life problem into a statistical problem

Using chi-square test

Variable 1 is “using treatment”. Two levels: use or not.

Variable 2 is “getting winter cold”. Two levels: get cold or not.

For kindergarten kids and for pre-school kids, I thus have two 2 * 2 tables.

(question: can I do a chi-square test on three variables? The third one being “age”.)

Using two-sample t-test

Variable 1 is “using treatment”. Two levels: use or not

Variable 2 is supposed to be a numerical variable —- here use disease rate. But then there is no enough number of samples.

Thus, conclude that Chi-square test should be used here.

Brief explanation of statistical learning framework

This post explains what is a statistical learning framework, and common results under this framework.

Problem

We have a random variable X, another random variable Y. Now we want to determine the relationship between X and Y.

We define the relationship by a prediction function f(x). For each x, this function produces an “action” a in the action space.

Now how do we get the predictive function f? We use a loss function l(a, y), that for each a and y, we produce a “loss”. Note since X is a random variable, f(x) is a transformation, so a is a random variable, too.

Also, l(a, y) is a transformation of (a, y), so l(a, y) is a random variable too. It’s distribution is based on both X and Y.

We then calculate f by minimizing the expectation of the loss, which is called “risk”. Since the distribution of l(a, y) is based both on the distribution of X and Y, to get this expectation, we need to do integration both on X and on Y. In the case of discrete variables, we do summation based on the pmf of (x, y).

The above are about theoretical properties of Y, X, loss function and prediction function. But we usually do not know the distribution of (X, Y). Thus, we choose to minimize empirical risk instead. We calculate empirical risk by summing all the empirical loss together, divided by m. (q: does this resemble Monte Carlo method? is this about computational statistics? Need a review.)

Results

In the case of square-loss, we have the result, a = E(y|x).

In the case of 0-1 loss, we have the result, a = arg max P(y|x)

Example:

We want to predict a student’s mid-term grade (Y). We want to know the relationship between predicted value, and whether she is hard-working (X).

We use square-loss for this continuous variable Y. Since we know that to minize square loss, for any random variable we should predict the mean value of the variable (c.f. regression analysis, in OLS scenerio we calculate the MSE — but need further connection to this framework).

Now we just observed unfortunately the student is not hard-working.

We know for a not-hardworking student the expectation of mid-term grade is 40.

We then predict the grade to be 40, as a way to minimize square-loss.

Probability, statistics, frequentist and Bayesian

This post is a review of basic concepts in probability and statistics.

Useful reference: https://cims.nyu.edu/~cfgranda/pages/DSGA1002_fall15/notes.html

https://ocw.mit.edu/courses/mathematics/18-05-introduction-to-probability-and-statistics-spring-2014/

Probability

It’s a tool to mathematically measure uncertainty.

Formal definition involving $\sigma-algebra$:

A probability space isa triple $(\Omega, F, P)$ consisting of :

• A sample space $\Omega$
• A set of events F – which will be $\sigma-algebra$
• A probability measure P that assigns probabbilites to the events in F.

Example: We have a fair coin. Now we toss it 1000 times, what’s the probability of getting 600 heads or more?

Statistics

The goal of statistics is to 1) draw conclusion from data (e.g. reject Null Hypothesis) and 2) evaluate the uncertainty of this information (e.g. p-value, confidence interval, or posterier distribution).

At the bottem, statistical statement is also about probability. Because it applies probability to draw conclusions from data.

Example: We would like to know whether the probability of raining tomorrow is 0.99. Then tomorrow comes, and it does not rain. Do we conclude that P(rain) = 0.99 is true?

Example 2: We would like to decide if a coin is fair. (Data) Toss the coin 1000 times, and 809 times it’s a head. Do we conclude the coin is fair?

Note : probability is logically self-contained. There are a few rules, and the answers follow from the rules. Statistics can be messy, because it involves draw conclusion from data – much art than science.

Frequentist vs Bayesian

Two schools of statistics. They are different in their interpretation of probability.

Frequentist interpret probability to be the frequencies of events in repeating experiments. E.g. P(head) = 0.6. Then if we toss a coin 1000 times, we will have 600 heads.

Bayesian interprets probability to be a state of knowledge, or a state of belief, about a preposition. E.g. P(head) = 0.6, means we are fairly certain (around 60% certain!) that a coin will be tossed head.

In practice though, Bayesian seldom use a single value to characterize such belief. Rather, it uses a distribution.

Frequentists are used in social science, biology, medicine, public health. We see two sample t-tests, p-values. Bayesian is used in computer science, “big data”.

Core difference between Frequentists and Bayesian

Bayesian considers the results from previous experiments, in the form of a prior.

See this comic for an illustration.

What does it mean?

A frequentist and a Bayesian are making a bet about whether the sun has exploded.

It’s night, so they can not observe.

They ask some expert whether the sun has gone Nova.

They also know that this expert will toss two coins. If both get 6, she will lie. Else, she won’t. (Data generation process)

Now they ask the expert, who tells them yes, the sun has gone Nova.

Frequent conclude that since the probability of getting two 6’s is 1/36 = 0.0027 <0.05 (p < 0.05), it’s very unlikely the expert has lied. Thus, she concludes the expert did not lie. Thus, she concludes that the sun has exploded.

Bayesian, however, has a strong belief that the sun has not exploded (or else they will be dead already). The prior distribution is

• P(sun has not exploded) = 0.99999999999999999,
• P(sun has exploded) = 0.00000000000000001.

Now the data generation process is essentially the following distribution:

• P(expert says sun exploded |Sun not exploded) =  1/36.
• P(expert says sun exploded |Sun exploded) =  35/36.
• P(expert says sun not exploded |Sun exploded) =  1/36.
• P(expert says sun not exploded |Sun not exploded) =  35/36.

The observed data is “expert says sun exploded”. We want to know

• P( Sun exploded | expert says sun exploded ) = P( expert says sun exploded | Sun exploded) * P( Sun exploded) / P(expert says sun exploded)

Since P(Sun exploded) is extremely small compared to other probabilities, P( Sun exploded | expert says sun exploded ) is also extremely small.

Thus although the expert is unlikely to lie (p = 0.0027), the sun is much more unlikely to have exploded. Thus, the expert most likely lied, and the sun has not exploded.

Debugging larger pyspark ML programs

This post describes my learning experience in developing larger programs, especially those :

• Takes a long time to run – due to big data sets and computationally intensive algorithms
• Requires developing locally and on HPC. That is cannot be solved in a Python IDE

The take away is:

• To save time, try writing scripts in one place only.
• Do not develop interactive and then paste everything to an editor!

The problem

I found myself spending excessive time (~ 4 days) developing a program that should be simple in its logic. Basically, I just need to call Pyspark API functions to do classification on 20Newsgroup dataset. It should be a simple program!

How did I spend my time?

• First day, I found a script doing a similar task. When I tried to use it I came into the following problems:
• Read files on HDFS. Spark only works with files on HDFS not local file system. This mistakes took me some time, as I thought the problem was with syntax in reading nested folders.
• The script does not clean text – remove headers, numbers, punctuations. To do this, I had to understand the .trans function. This also works differently in Python2 and Python3, which took me some time to realize.
• The script use Pyspark SQL, thus took some time to learn DataFrame as well.
• By day 2 the data is read into Spark DataFrame.
• I then had problem calling functions from MLLib, because I didn’t realize ML and MLlib are two libraries and have different data structures. I then came into problem when using an MLlib function to ML data.
• I also tried to convert the data structure back and forth, from RDD to Labeled Point to ML.
• To inspect what is in the data, I also spent time calling the wrong functions, or transform everything into an RDD and call map functions.
• By day 3 I intend to use Spark-submit on HPC. The main task is to learn to use editor.
• Because someone told me I should be using editor instead of debugging interactively, or else I cannot see code structure, I began to learn vim. That took a morning or so (!).
• By day 4, I am trying to clean up the code and write functions.
• This creates another level of complexity. One of the bug is I forget to update the function argument

Trouble-shooting:

• I type every line of code three to four times. First on my workstation, then on the server’s pyspark shell. Finally I copy the code into a script. This not only creates space for mistakes, but also is inefficient.
• I did this because I am not comfortable writing scripts on HPC yet.
• Also because I am not comfortable debugging with a script and an editor, without using IDE.
• I input every line of code at least three to four times.
• First in interactive mode.
• Then, copying the line into my editor. Since the project spans across days, every time I start again I need to re-read the files!
• Also, I then creates a file on a small dataset on HPC and run that.
• After that, I run the file again on a larger dataset.
• Since I am debugging at multiple places, I need to do version control as well.
• I remember scp back and forth sending files along. Whenever I edited something, I remove the older version of the program at the other place.
• I am not familiar with data structure and functions on Pyspark. This also leads to waste of time.
• Interactive mode of Spark is slow. I once forgot to cut the dataset smaller, and running one command (e.g. transform) on the whole dataset takes 10 minues! If there are 20 commands like this that would be ~3 hours.

Solution

• The above problems can be summarized as :
• I need to write script on multiple places: server and local.
• Solution: learn to use editor in the server. e.g. Vim
• Submit program interactively vs in batch. I do not know how to debug with a script so I have to use interactive mode to make sure I know what I am doing. But debugging interactively means double the amount of typing because every command needs to go through the terminal and the editor. That’s also running the program twice.
• Solution: learn to run the program from the script. The con of this is need to laod data multiple times. Also, use a smaller dataset so data loading will not be a pain.
• Time spent on start-up. The data file is huge, thus re-loading it takes time. If every time load the dataset is 3 minutes, load it 20 times would be 60 minutes.
• Solution: use a smaller sample dataset for developing.

Moral: try to write only one set of programs in a single place.

• For pyspark, I can only use HPC. So I just write on HPC.
• Pyspark also cannot use pdb.
• For Python, I should test everything locally first. There are two final programs
• A script that can run locally and on HPC.
• A script to be submitted to the cluster.

Debugging philosophy

• “Bugs” are not bugs but errors. The responsibility lies with the programmer. A program that has errors is simple wrong, because 1) the programmer is not really familiar with the rules and grammars of the library. Used the wrong data structure, used the wrong function call. etc.
• Debugging is a learning experience.  Why does debugging takes lots of time? Because the programmer is learning something new, thus need time to try and make mistakes. There seems to be a wishful thinking that a perfect program will magically began working by itself, and thus time will be well spent. It wont’, because learning always takes time!

Pyspark ML vs MLLib

A post that summarizes main difference between Pyspakr ML and MLlib. This is based on Spark 2.2.0 and Python 3.

Data Structure

• pyspark.mllib is the older library for machine learning. It can only use RDD labeled point. But then more features than ML
• pyspark.ml newer library for machine learning. It can only use sql DataFrame. But easier to construct a ML pipeline.
• It is recommended to use DataFrame APIs (i.e. ML) because it is more stable. Also this is newer.

Features:

The two libraries seem to be similar in terms of feature selection APIs.

Models

• pyspark.ml implements Pipeline workflow, which is based on dataFrame and can be used to “quickly assemble and configure practical machine learning pipelines”.

Classification algorithms provided by MLlib:

Classification algorithm by ML:

Seems that ML has multilayerPerceptron which is not in MLLib.

Evaluation

• pyspark ml implements pyspark.ml.tuning.CrossValidator while pyspark.mllib does not
• However Pyspark ML’s evaluation function is difficult to use.  The docs are not clear. Thus, changing back to MLlib to use this.

When to use which?

• Efficiency: MLLib uses Labelpoint RDD, while ML uses structured DataFrame. Thus, “if the data is already structured DataFrames, then ML confer some performance benefits over RDD’s, this is apparently drastic as the complexity of your operations increase.”
• Resource: DataFrames consume far less memory when caching than RDDs. Thus lower level operations RDD’s are great but high level operations, viewing and typing with other API’s use DataFrames. (Above all quote Stackoverflow user Grr)
• It is recommended to use ML and DataFrame. However if want to use Gridsearch, seems still need to revert to Labeled point!

Reference :

A stackoverflow question: https://stackoverflow.com/questions/43240539/pyspark-mllib-versus-pyspark-ml-packages

MLlib doc: https://spark.apache.org/docs/latest/api/python/pyspark.mllib.html

ML doc: https://spark.apache.org/docs/latest/api/python/pyspark.ml.html

Comparison of data structures: https://databricks.com/blog/2016/07/14/a-tale-of-three-apache-spark-apis-rdds-dataframes-and-datasets.html

https://www.quora.com/Why-are-there-two-ML-implementations-in-Spark-ML-and-MLlib-and-what-are-their-different-features

Clean Text in Python3

A pain in the ass. This post summarizes “best” approaches to clean text data in Python3.

It will not cover depreciated syntax in Python2. For example string.maketans has a different usage in python2 — it is not discussed here.

Is that a string or a Unicode?

Reference here.

When you see a string in python2, there are two possibilities:

• ASCII strings: Every character in a string is a byte. Look up the hexadecimal value in the ASCII table.
• Unicode strings: every character in a string is one or more than one byte. Look up the hexiadecimal value in the Unicode table – there are many of them. The most popular one is UTF-8.
• Example: 猫 is represened in three bytes in Unicode. when Python2 reads this, it gots it wrong – it thinks there are 5 characters but there is in fact just three… As python2 use ascii to decode.
• To produce the correct representation, use x.decode(‘utf-8’)

String vs sequence of bytes in Python2

• String is a sequence of Unicode codepoints… they are abstract concepts and cannot be stored on disk. They are a sequence of characters.
• bytes are actual numbers… they can be stored on disk.
• Anything has to be mapped to a byte to be stored on a computer
• To map a codepoint to a byte, use Unicode encoding
• To convert a byte to a string, use decoding .

String vs sequences of bytes in Python3

• In sum, in Python 3, str represents a Unicode string, while the bytes type is a sequence of bytes. This naming scheme makes a lot more intuitive sense.

Encode vs Decode

• To represent a unicode string as a string of bytes is known as encoding

Remove punctuation

The best answer (I think) from Stackoverflow :

import string

# Thanks to Martijn Pieters for this improved version

# This uses the 3-argument version of str.maketrans
# with arguments (x, y, z) where 'x' and 'y'
# must be equal-length strings and characters in 'x'
# are replaced by characters in 'y'. 'z'
# is a string (string.punctuation here)
# where each character in the string is mapped
# to None
translator = str.maketrans('', '', string.punctuation)

# This is an alternative that creates a dictionary mapping
# of every character from string.punctuation to None (this will
# also work)
#translator = str.maketrans(dict.fromkeys(string.punctuation))

s = 'string with "punctuation" inside of it! Does this work? I hope so.'

# pass the translator to the string's translate method.
print(s.translate(translator))

The code above removes punctuation and just delete it.

Replace punctuation with a blank

This method uses regular expression… I find it to be better than using a translator!

import re
re.sub(r'[^\w]', ' ', text)

Dealing with all kinds of blanks

Again borrowing from this awesome stackoverflow post…

# Remove leading and ending spaces, use str.strip():
sentence = ' hello  apple'
sentence.strip()
>>>'hello  apple'

# Remove all spaces, use str.replace():

sentence = ' hello  apple'
sentence.replace(" ", "")
>>> 'helloapple'

# Remove duplicated spaces, use str.split(), then join the words together

sentence = ' hello  apple'
" ".join(sentence.split())
>>>'hello apple'

Summary of workflow:

1. Decide if that’s a string or a unicode, or a sequence of bytes. This will decide whether need to encode or not. Ultimately we want a string, i.e. “str” type.
2. Import re module. MHOP this is the most convenient one to remove punctuations.
1. re.sub can replace all numbers and punctuations with blanks. Thus, will not join two unrelated words together if they are connected by punctuations.
3. Use s.lower() to change a string to lower case… Note here s is a string! not a unicode object.
4. Use s.strip() to strip out the excessive blanks!
5. Use . join to join the list of words together with only one blank between them, if needed.

Machine Learning in Spark

This post describes Spark’s architecture for building and running machine learning algos.

Machine learning algorithms can be found by Spark’s MLlib API.

The data structure used are DataFrame. With regard the the last post, this means we need to import spark SQL context to use as well.

There are five key concepts:

• DataFrame: the data structure. Each column can be text, labels, features, or numbers…
• Transformer: an object to transform a DataFrame to another one. A machine learning algo is represented as a transformer that takes an input DataFrame with features and outputs a DataFrame with the predictions. A transformer is called by a transform() method on a DataFrame.
• Estimator: an algorithm can be fit to a DataFrame and produces a transformer.  e.g. a ML algo can be fit to the training set and produce a model. An estimator is called by a fit() method on a DataFrame.
• Pipeline: a collection of transformers and estimators that eventually consists of a trained model.
• Parameters: parameters of the transformer and the estimator.

Now go over the example on Spark Website for understanding of how to use machine learning Pipeline.

Step one: read in data

First create the spark SQL context, and use that to create a SparkDataFrame. This assumes a spark context is already created as sc.

sqlContext = SQLContext(sc)

training = sqlContext.createDataFrame([
(0, "a b c d e spark", 1.0),
(1, "b d", 0.0),
(2, "spark f g h", 1.0),
(3, "hadoop mapreduce", 0.0)
], ["id", "text", "label"])

The results of this step:

training.show()
+---+----------------+-----+
| id|            text|label|
+---+----------------+-----+
|  0| a b c d e spark|  1.0|
|  1|             b d|  0.0|
|  2|     spark f g h|  1.0|
|  3|hadoop mapreduce|  0.0|
+---+----------------+-----+

Step two: preprocess + training = ML pipeline

Now, preprocess data by constructing a ML pipeline.

• Tokenizer here tokenize every text into words ….
• hashingTF: this one is more tricky. As per the document,

Our implementation of term frequency utilizes the hashing trick. A raw feature is mapped into an index (term) by applying a hash function. Then term frequencies are calculated based on the mapped indices. This approach avoids the need to compute a global term-to-index map, which can be expensive for a large corpus, but it suffers from potential hash collisions, where different raw features may become the same term after hashing. To reduce the chance of collision, we can increase the target feature dimension, i.e., the number of buckets of the hash table. The default feature dimension is 220=1,048,576

My understanding on hashTF (hash term frequency?) is that :

• Hash functions map data of arbitrary size to a fixed size. It maps every word to a unique position, hopefully. But there will be collisions, which we ignore…
• Next we just count the number of items at each position.
• This saves the step of storing a map of word – index. Instead we just need to store a function.

# Configure an ML pipeline, which consists of three stages: tokenizer, hashingTF, and lr.
tokenizer = Tokenizer(inputCol="text", outputCol="words")
hashingTF = HashingTF(inputCol=tokenizer.getOutputCol(), outputCol="features")
lr = LogisticRegression(maxIter=10, regParam=0.001)
pipeline = Pipeline(stages=[tokenizer, hashingTF, lr])

# Fit the pipeline to training documents.
model = pipeline.fit(training)

Now let’s go over the output of each step. First, check the results of the tokenizer….

p1 = Pipeline(stages = [tokenizer])
p1.fit(training).transform(training).show()

+---+----------------+-----+--------------------+
| id|            text|label|               words|
+---+----------------+-----+--------------------+
|  0| a b c d e spark|  1.0|[a, b, c, d, e, s...|
|  1|             b d|  0.0|              [b, d]|
|  2|     spark f g h|  1.0|    [spark, f, g, h]|
+---+----------------+-----+--------------------+

Then, check the results of the HashingTF….

hashingTF = HashingTF(inputCol=tokenizer.getOutputCol(), outputCol="features")
p2 = Pipeline(stages = [tokenizer, hashingTF])
p2.fit(training).transform(training).show()

+---+----------------+-----+--------------------+--------------------+
| id|            text|label|               words|            features|
+---+----------------+-----+--------------------+--------------------+
|  0| a b c d e spark|  1.0|[a, b, c, d, e, s...|(262144,[97,98,99...|
|  1|             b d|  0.0|              [b, d]|(262144,[98,100],...|
|  2|     spark f g h|  1.0|    [spark, f, g, h]|(262144,[102,103,...|
+---+----------------+-----+--------------------+--------------------+

Third, check the output of the logistic regression model:

lr = LogisticRegression(maxIter=10, regParam=0.001)
p3 = Pipeline(stages=[tokenizer, hashingTF, lr])
p3.fit(training).transform(training).columns

['id', 'text', 'label', 'words', 'features', 'rawPrediction', 'probability', 'prediction']

In our original data, we only have id, text and label.

• ‘words’ are added by the tokenizer,
• ‘features’ are added by the hashingTF,
• ‘rawPrediction’ is created by logistic regression. It is the “score” of each class labels. It varies algo by algo. For example, in a neural network it can be interpreted as the last layer passed to the softmax.
• prediction: the predicted class label. the argmax of raw prediction.
• probability : the “real” probability given raw prediction… Looks like passing rawprediction to a softmax for multi-class classification problems.

Step three: making predictions

After building the model now we can start to make predictions by building and testing on a toy test set.

First build test set — it is similar to the training set, except without labels

test = sqlContext.createDataFrame([
(4, "spark i j k"),
(5, "l m n"),
(6, "spark hadoop spark"),
], ["id", "text"])

test.show()
+---+------------------+
| id|              text|
+---+------------------+
|  4|       spark i j k|
|  5|             l m n|
|  6|spark hadoop spark|
|  7|     apache hadoop|
+---+------------------+

Make predictions…

prediction = model.transform(test)

DataFrame[id: bigint, text: string, words: array<string>, features: vector, rawPrediction: vector, probability: vector, prediction: double]

Here prediction is of the same format as the transformed train dataset. The reason is, fitting machine learning algorithm is equivalent to transform the test data to add other columns…

Finally, printing out the results we care: prediction

selected = prediction.select("id", "text", "probability", "prediction")
for row in selected.collect():
rid, text, prob, prediction = row
print("(%d, %s) --> prob=%s, prediction=%f" % (rid, text, str(prob), prediction))

(4, spark i j k) --> prob=[0.159640773879,0.840359226121], prediction=1.000000
(5, l m n) --> prob=[0.837832568548,0.162167431452], prediction=0.000000
(6, spark hadoop spark) --> prob=[0.0692663313298,0.93073366867], prediction=1.000000
(7, apache hadoop) --> prob=[0.982157533344,0.0178424666556], prediction=0.000000

Question: what will the output be if some of the classes are not in the training set?

Reference: https://spark.apache.org/docs/latest/ml-pipeline.html#main-concepts-in-pipelines