text:
# Generating music in the waveform domain

## March 24, 2020

In November last year, I co-presented a tutorial on waveform-based music processing with deep learning with Jordi Pons and Jongpil Lee at ISMIR 2019. Jongpil and Jordi talked about music classification and source separation respectively, and I presented the last part of the tutorial, on music generation in the waveform domain. It was very well received, so I’ve decided to write it up in the form of a blog post.

ISMIR used to be my home conference when I was a PhD student working on music information retrieval, so it was great to be back for the first time in five years. With about 450 attendees (the largest edition yet), it made for a very different experience than what I’m used to with machine learning conferences like ICML, NeurIPS and ICLR, whose audiences tend to number in the thousands these days.

Our tutorial on the first day of the conference gave rise to plenty of interesting questions and discussions throughout, which inspired me to write some of these things down and hopefully provide a basis to continue these discussions online. Note that I will only be covering music generation in this post, but Jordi and Jongpil are working on blog posts about their respective parts. I will share them here when they are published. In the meantime, the slide deck we used includes all three parts and is now available on Zenodo (PDF) and on Google slides. I’ve also added a few things to this post that I’ve thought of since giving the tutorial, and some new work that has come out since.

This is also an excellent opportunity to revive my blog, which has lain dormant for the past four years. I have taken the time to update the blog software, so if anything looks odd, that may be why. Please let me know so I can fix it!

## Overview

This blog post is divided into a few different sections. I’ll try to motivate why modelling music in the waveform domain is an interesting problem. Then I’ll give an overview of generative models, the various flavours that exist, and some important ways in which they differ from each other. In the next two sections I’ll attempt to cover the state of the art in both likelihood-based and adversarial models of raw music audio. Finally, I’ll raise some observations and discussion points. If you want to skip ahead, just click the section title below to go there.

Note that this blog post is not intended to provide an exhaustive overview of all the published research in this domain – I have tried to make a selection and I’ve inevitably left out some great work. Please don’t hesitate to suggest relevant work in the comments section!

## Motivation

### Why audio?

Music generation has traditionally been studied in the symbolic domain: the output of the generative process could be a musical score, a sequence of MIDI events, a simple melody, a sequence of chords, a textual representation1 or some other higher-level representation. The physical process through which sound is produced is abstracted away. This dramatically reduces the amount of information that the models are required to produce, which makes the modelling problem more tractable and allows for lower-capacity models to be used effectively.

A very popular representation is the so-called piano roll, which dates back to the player pianos of the early 20th century. Holes were punched into a roll of paper to indicate which notes should be played at which time. This representation survives in digital form today and is commonly used in music production. Much of the work on music generation using machine learning has made use of (some variant of) this representation, because it allows for capturing performance-specific aspects of the music without having to model the sound.

Piano rolls are great for piano performances, because they are able to exactly capture the timing, pitch and velocity (i.e. how hard a piano key is pressed, which is correlated with loudness, but not equivalent to it) of the notes. They are able to very accurately represent piano music, because they cover all the “degrees of freedom” that a performer has at their disposal. However, most other instruments have many more degrees of freedom: think about all the various ways you can play a note on the guitar, for example. You can decide which string to use, where to pick, whether to bend the string or not, play vibrato, … you could even play harmonics, or use two-hand tapping. Such a vast array of different playing techniques endows the performer with a lot more freedom to vary the sound that the instrument produces, and coming up with a high-level representation that can accurately capture all this variety is much more challenging. In practice, a lot of this detail is ignored and a simpler representation is often used when generating music for these instruments.

Modelling the sound that an instrument produces is much more difficult than modelling (some of) the parameters that are controlled by the performer, but it frees us from having to manually design high-level representations that accurately capture all these parameters. Furthermore, it allows our models to capture variability that is beyond the performer’s control: the idiosyncracies of individual instruments, for example (no two violins sound exactly the same!), or the parameters of the recording setup used to obtain the training data for our models. It also makes it possible to model ensembles of instruments, or other sound sources altogether, without having to fundamentally change anything about the model apart from the data it is trained on.

Digital audio representations require a reasonably high bit rate to achieve acceptable fidelity however, and modelling all these bits comes with a cost. Music audio models will necessarily have to have a much higher capacity than their symbolic counterparts, which implies higher computational requirements for model training.

### Why waveforms?

Digital representations of sound come in many shapes and forms. For reproduction, sound is usually stored by encoding the shape of the waveform as it changes over time. For analysis however, we often make use of spectrograms, both for computational methods and for visual inspection by humans. A spectrogram can be obtained from a waveform by computing the Fourier transform of overlapping windows of the signal, and stacking the results into a 2D array. This shows the local frequency content of the signal over time.

Spectrograms are complex-valued: they represent both the amplitude and the phase of different frequency components at each point in time. Below is a visualisation of a magnitude spectrogram and its corresponding phase spectrogram. While the magnitude spectrogram clearly exhibits a lot of structure, with sustained frequencies manifesting as horizontal lines and harmonics showing up as parallel horizontal lines, the phase spectrogram looks a lot more random.

When extracting information from audio signals, it turns out that we can often just discard the phase component, because it is not informative for most of the things we could be interested in. In fact, this is why the magnitude spectrogram is often referred to simply as “the spectrogram”. When generating sound however, phase is very important because it meaningfully affects our perception. Listen below to an original excerpt of a piano piece, and a corresponding excerpt where the original phase has been replaced by random uniform phase information. Note how the harmony is preserved, but the timbre changes completely.

The phase component of a spectrogram is tricky to model for a number of reasons:

• it is an angle: $$\phi \in [0, 2 \pi)$$ and it wraps around;
• it becomes effectively random as the magnitude tends towards 0, because noise starts to dominate;
• absolute phase is less meaningful, but relative phase differences over time matter perceptually.

If we model waveforms directly, we are implicitly modelling their phase as well, but we don’t run into these issues that make modelling phase so cumbersome. There are other strategies to avoid these issues, some of which I will discuss later, but waveform modelling currently seems to be the dominant approach in the generative setting. This is particularly interesting because magnitude spectrograms are by far the most common representation used for discriminative models of audio.

### Discretising waveforms

When representing a waveform digitally, we need to discretise it in both time and amplitude. This is referred to as pulse code modulation (PCM). Because audio waveforms are effectively band-limited (humans cannot perceive frequencies above ~20 kHz), the sampling theorem tells us that we can discretise the waveform in time without any loss of information, as long as the sample rate is high enough (twice the highest frequency). This is why CD quality audio has a sample rate of 44.1 kHz. Much lower sample rates result in an audible loss of fidelity, but since the resulting discrete sequences also end up being much shorter, a compromise is often struck in the context of generative modelling to reduce computational requirements. Most models from literature use sample rates of 16 or 24 kHz.

When we also quantise the amplitude, some loss of fidelity is inevitable. CD quality uses 16 bits per sample, representing 216 equally spaced quantisation levels. If we want to use fewer bits, we can use logarithmically spaced quantisation levels instead to account for our nonlinear perception of loudness. This “mu-law companding” will result in a smaller perceived loss of fidelity than if the levels were equally spaced.

## Generative models

Given a dataset $$X$$ of examples $$x \in X$$, which we assume to have been drawn independently from some underlying distribution $$p_X(x)$$, a generative model can learn to approximate this distribution $$p_X(x)$$. Such a model could be used to generate new samples that look like they could have been part of the original dataset. We distinguish implicit and explicit generative models: an implicit model can produce new samples $$x \sim p_X(x)$$, but cannot be used to infer the likelihood of an example (i.e. we cannot tractably compute $$p_X(x)$$ given $$x$$). If we have an explicit model, we can do this, though sometimes only up to an unknown normalising constant.

### Conditional generative models

Generative models become more practically useful when we can exert some influence over the samples we draw from them. We can do this by providing a conditioning signal $$c$$, which contains side information about the kind of samples we want to generate. The model is then fit to the conditional distribution $$p_X(x \vert c)$$ instead of $$p_X(x)$$.

Conditioning signals can take many shapes or forms, and it is useful to distinguish different levels of information content. The generative modelling problem becomes easier if the conditioning signal $$c$$ is richer, because it reduces uncertainty about $$x$$. We will refer to conditioning signals with low information content as sparse conditioning, and those with high information content as dense conditioning. Examples of conditioning signals in the image domain and the music audio domain are shown below, ordered according to density.

Note that the density of a conditioning signal is often correlated with its level of abstraction: high-level side information tends to be more sparse. Low-level side information isn’t necessarily dense, though. For example, we could condition a generative model of music audio on a low-dimensional vector that captures the overall timbre of an instrument. This is a low-level aspect of the audio signal, but it constitutes a sparse conditioning signal.

### Likelihood-based models

Likelihood-based models directly parameterise $$p_X(x)$$. The parameters $$\theta$$ are then fit by maximising the likelihood of the data under the model:

$\mathcal{L}_\theta(x) = \sum_{x \in X} \log p_X(x|\theta) \quad \quad \theta^* = \arg \max_\theta \mathcal{L}_\theta(x) .$

Note that this is typically done in the log-domain because it simplifies computations and improves numerical stability. Because the model directly parameterises $$p_X(x)$$, we can easily infer the likelihood of any $$x$$, so we get an explicit model. Three popular flavours of likelihood-based models are autoregressive models, flow-based models and variational autoencoders. The following three subsections provide a brief overview of each.

### Autoregressive models

In an autoregressive model, we assume that our examples $$x \in X$$ can be treated as sequences $$\{x_i\}$$. We then factorise the distribution into a product of conditionals, using the chain rule of probability:

$p_X(x) = \prod_i p(x_i \vert x_{<i}) .$

These conditional distributions are typically scalar-valued and much easier to model. Because we further assume that the distribution of the sequence elements is stationary, we can share parameters and use the same model for all the factors in this product.

For audio signals, this is a very natural thing to do, but we can also do this for other types of structured data by arbitrarily choosing an order (e.g. raster scan order for images, as in PixelRNN2 and PixelCNN3).

Autoregressive models are attractive because they are able to accurately capture correlations between the different elements $$x_i$$ in a sequence, and they allow for fast inference (i.e. computing $$p_X(x)$$ given $$x$$). Unfortunately they tend to be slow to sample from, because samples need to be drawn sequentially from the conditionals for each position in the sequence.

### Flow-based models

Another strategy for constructing a likelihood-based model is to use the change of variables theorem to transform $$p_X(x)$$ into a simple, factorised distribution $$p_Z(z)$$ (standard Gaussian is a popular choice) using an invertible mapping $$x = g(z)$$:

$p_X(x) = p_Z(z) \cdot |\det J|^{-1} \quad \quad J = \frac{dg(z)}{dz}.$

Here, $$J$$ is the Jacobian of $$g(z)$$. Models that use this approach are referred to as normalising flows or flow-based models45. They are fast both for inference and sampling, but the requirement for $$g(z)$$ to be invertible significantly constrains the model architecture, and it makes them less parameter-efficient. In other words: flow-based models need to be quite large to be effective.

For an in-depth treatment of flow-based models, I recommend Eric Jang’s two-part blog post on the subject, and Papamakarios et al.’s excellent review paper.

### Variational autoencoders (VAEs)

By far the most popular class of likelihood-based generative models, I can’t avoid mentioning variational6 autoencoders7 – but in the context of waveform modelling, they are probably the least popular approach. In a VAE, we jointly learn two neural networks: an inference network $$q(z \vert x)$$ learns to probabilistically map examples $$x$$ into a latent space, and a generative network $$p(x \vert z)$$ learns the distribution of the data conditioned on a latent representation $$z$$. These are trained to maximise a lower bound on $$p_X(x)$$, called the ELBO (Evidence Lower BOund), because computing $$p_X(x)$$ given $$x$$ (exact inference) is not tractable.

Typical VAEs assume a factorised distribution for $$p(x \vert z)$$, which limits the extent to which they can capture dependencies in the data. While this is often an acceptable trade-off, in the case of waveform modelling it turns out to be a problematic restriction in practice. I believe this is why not a lot of work has been published that takes this approach (if you know of any, please point me to it). VAEs can also have more powerful decoders with fewer assumptions (autoregressive decoders, for example), but this may introduce other issues such as posterior collapse8.

Generative Adversarial Networks9 (GANs) take a very different approach to capturing the data distribution. Two networks are trained simultaneously: a generator $$G$$ attempts to produce examples according to the data distribution $$p_X(x)$$, given latent vectors $$z$$, while a discriminator $$D$$ attempts to tell apart generated examples and real examples. In doing so, the discriminator provides a learning signal for the generator which enables it to better match the data distribution. In the original formulation, the loss function is as follows:

$\mathcal{L}(x) = \mathbb{E}_x[\log D(x)] + \mathbb{E}_z[log(1 - D(G(z)))] .$

The generator is trained to minimise this loss, whereas the discriminator attempts to maximise it. This means the training procedure is a two-player minimax game, rather than an optimisation process, as it is for most machine learning models. Balancing this game and keeping training stable has been one of the main challenges for this class of models. Many alternative formulations have been proposed to address this.

While adversarial and likelihood-based models are both ultimately trying to model $$p_X(x)$$, they approach this target from very different angles. As a result, GANs tend to be better at producing realistic examples, but worse at capturing the full diversity of the data distribution, compared to likelihood-based models.

### More exotic flavours

Many other strategies to learn models of complicated distributions have been proposed in literature. While research on waveform generation has chiefly focused on the two dominant paradigms of likelihood-based and adversarial models, some of these alternatives may hold promise in this area as well, so I want to mention a few that I’ve come across.

• Energy-based models measure the “energy” of examples, and are trained by fitting the model parameters so that examples coming from the dataset have low energy, whereas all other configurations of inputs have high energy. This amounts to fitting an unnormalised density. A nice recent example is the work by Du & Mordatch at OpenAI10. Energy-based models have been around for a very long time though, and one could argue that likelihood-based models are a special case.

• Optimal transport is another approach to measure the discrepancy between probability distributions, which has served as inspiration for new variants of generative adversarial networks11 and autoencoders12.

• Autoregressive implicit quantile networks13 use a similar network architecture as likelihood-based autoregressive models, but they are trained using the quantile regression loss, rather than maximimum likelihood.

• Two continuous distributions can be matched by minimising the L2 distance between the gradients of the density functions with respect to their inputs: $$\mathcal{L}(x) = \mathbb{E} [\vert\vert \nabla_x \log p_X(x) - \nabla_y \log p_Y(y) \vert\vert ^2]$$. This is called score matching14 and some recent works have revisited this idea for density estimation15 and generative modelling16.

• Please share any others that I haven’t mentioned in the comments!

### Mode-covering vs. mode-seeking behaviour

An important consideration when determining which type of generative model is appropriate for a particular application, is the degree to which it is mode-covering or mode-seeking. When a model does not have enough capacity to capture all the variability in the data, different compromises can be made. If all examples should be reasonably likely under the model, it will have to overgeneralise and put probability mass on interpolations of examples that may not be meaningful (mode-covering). If there is no such requirement, the probability mass can be focused on a subset of examples, but then some parts of the distribution will be ignored by the model (mode-seeking).

Likelihood-based models are usually mode-covering. This is a consequence of the fact that they are fit by maximising the joint likelihood of the data. Adversarial models on the other hand are typically mode-seeking. A lot of ongoing research is focused on making it possible to control the trade-off between these two behaviours directly, without necessarily having to switch the class of models that are used.

In general, mode-covering behaviour is desirable in sparsely conditioned applications, where we want diversity or we expect a certain degree of “creativity” from the model. Mode-seeking behaviour is more useful in densely-conditioned settings, where most of the variability we care about is captured in the conditioning signal, and we favour realism of the generated output over diversity.

## Likelihood-based models of waveforms

In this section, I’ll try to summarise some of the key results from the past four years obtained with likelihood-based models of waveforms. While this blog post is supposed to be about music, note that many of these developments were initially targeted at generating speech, so inevitably I will also be talking about some work in the text-to-speech (TTS) domain. I recommend reading the associated papers and/or blog posts to find out more about each of these works.

### WaveNet & SampleRNN

WaveNet17 and SampleRNN18 are autoregressive models of raw waveforms. While WaveNet is a convolutional neural network, SampleRNN uses a stack of recurrent neural networks. Both papers appeared on arXiv in late 2016 with only a few months in between, signalling that autoregressive waveform-based audio modelling was an idea whose time had come. Before then, this idea had not been seriously considered, as modelling long-term correlations in sequences across thousands of timesteps did not seem feasible with the tools that were available at that point. Furthermore, discriminative models of audio all used spectral input representations, with only a few works investigating the use of raw waveforms in this setting (and usually with worse results).

Although these models have their flaws (including slow sampling due to autoregressivity, and a lack of interpretability w.r.t. what actually happens inside the network), I think they constituted an important existence proof that encouraged further research into waveform-based models.

WaveNet’s strategy to deal with long-term correlations is to use dilated convolutions: successive convolutional layers use filters with gaps between their inputs, so that the connectivity pattern across many layers forms a tree structure (see figure above). This enables rapid growth of the receptive field, which means that a WaveNet with only a few layers can learn dependencies across many timesteps. Note that the convolutions used in WaveNet are causal (no connectivity from future to past), which forces the model to learn to predict what values the signal could take at each position in time.

SampleRNN’s strategy is a bit different: multiple RNNs are stacked on top of each other, with each running at a different frequency. Higher-level RNNs update less frequently, which means they can more easily capture long-range correlations and learn high-level features.

Both models demonstrated excellent text-to-speech results, surpassing the state of the art at the time (concatenative synthesis, for most languages) in terms of naturalness. Both models were also applied to (piano) music generation, which constituted a nice demonstration of the promise of music generation in the waveform domain, but they were clearly limited in their ability to capture longer-term musical structure.

WaveNet: paper - blog post
SampleRNN: paper - samples

### Parallel WaveNet & ClariNet

Sampling from autoregressive models of raw audio can be quite slow and impractical. To address this issue, Parallel WaveNet19 uses probability density distillation to train a model from which samples can be drawn in a single feed-forward pass. This requires a trained autoregressive WaveNet, which functions as a teacher, and an inverse autoregressive flow (IAF) model which acts as the student and learns to mimic the teacher’s predictions.

While an autoregressive model is slow to sample from, inferring the likelihood of a given example (and thus, maximum-likelihood training) can be done in parallel. For an inverse autoregressive flow, it’s the other way around: sampling is fast, but inference is slow. Since most practical applications rely on sampling rather than inference, such a model is often better suited. IAFs are hard to train from scratch though (because that requires inference), and the probability density distillation approach makes training them tractable.

Due to the nature of the probability density distillation objective, the student will end up matching the teacher’s predictions in a way that minimises the reverse KL divergence. This is quite unusual: likelihood-based models are typically trained to minimise the forward KL divergence instead, which is equivalent to maximising the likelihood (and minimising the reverse KL is usually intractable). While minimising the forward KL leads to mode-covering behaviour, minimising the reverse KL will instead lead to mode-seeking behaviour, which means that the model may end up ignoring certain modes in the data distribution.

In the text-to-speech (TTS) setting, this may actually be exactly what we want: given an excerpt of text, we want the model to generate a realistic utterance corresponding to that excerpt, but we aren’t particularly fussed about being able to generate every possible variation – one good-sounding utterance will do. This is a setting where realism is clearly more important than diversity, because all the diversity that we care about is already captured in the conditioning signal that we provide. This is usually the setting where adversarial models excel, because of their inherent mode-seeking behaviour, but using probability density distillation we can also train likelihood-based models this way.

To prevent the model from collapsing, parallel WaveNet uses a few additional loss terms to encourage the produced waveforms to resemble speech (such as a loss on the average power spectrum).

If we want to do music generation, we will typically care more about diversity because the conditioning signals we provide to the model are weaker. I believe this is why we haven’t really seen the Parallel WaveNet approach catch on outside of TTS.

ClariNet20 was introduced as a variant of Parallel WaveNet which uses a Gaussian inverse autoregressive flow. The Gaussian assumption makes it possible to compute the reverse KL in closed form, rather than having to approximate it by sampling, which stabilises training.

Parallel WaveNet: paper - blog post 1 - blog post 2
ClariNet: paper - samples

### Flow-based models: WaveGlow, FloWaveNet, WaveFlow, Blow

Training an IAF with probability density distillation isn’t the only way to train a flow-based model: most can be trained by maximum likelihood instead. In that case, the models will be encouraged to capture all the modes of the data distribution. This, in combination with their relatively low parameter efficiency (due to the invertibility requirement), means that they might need to be a bit larger to be effective. On the other hand, they allow for very fast sampling because all timesteps can be generated in parallel, so while the computational cost may be higher, sampling will still be faster in practice. Another advantage is that no additional loss terms are required to prevent collapse.

WaveGlow21 and FloWaveNet22, both originally published in late 2018, are flow-based models of raw audio conditioned on mel-spectrograms, which means they can be used as vocoders. Because of the limited parameter efficiency of flow-based models, I suspect that it would be difficult to use them for music generation in the waveform domain, where conditioning signals are much more sparse – but they could of course be used to render mel-spectrograms generated by some other model into waveforms (more on that later).

WaveFlow23 (with an F instead of a G) is a more recent model that improves parameter efficiency by combining the flow-based modelling approach with partial autoregressivity to model local signal structure. This allows for a trade-off between sampling speed and model size. Blow24 is a flow-based model of waveforms for non-parallel voice conversion.

WaveGlow: paper - code - samples
FloWaveNet: paper - code - samples
WaveFlow: paper - samples
Blow: paper - code - samples

### Hierarchical WaveNets

For the purpose of music generation, WaveNet is limited by its ability to capture longer-term signal structure, as previously stated. In other words: while it is clearly able to capture local signal structure very well (i.e. the timbre of an instrument), it isn’t able to model the evolution of chord progressions and melodies over longer time periods. This makes the outputs produced by this model sound rather improvisational, to put it nicely.

This may seem counterintuitive at first: the tree structure of the connectivity between the layers of the model should allow for a very rapid growth of its receptive field. So if you have a WaveNet model that captures up to a second of audio at a time (more than sufficient for TTS), stacking a few more dilated convolutional layers on top should suffice to grow the receptive field by several orders of magnitude (up to many minutes). At that point, the model should be able to capture any kind of meaningful musical structure.

In practice, however, we need to train models on excerpts of audio that are at least as long as the longest-range correlations that we want to model. So while the depth of the model has to grow only logarithmically as we increase the desired receptive field, the computational and memory requirements for training do in fact grow linearly. If we want to train a model that can learn about musical structure across tens of seconds, that will necessarily be an order of magnitude more expensive – and WaveNets that generate music already have to be quite large as it is, even with a receptive field of just one second, because music is harder to model than speech. Note also that one second of audio corresponds to a sequence of 16000 timesteps at 16 kHz, so even at a scale of seconds, we are already modelling very long sequences.

In 10 years, the hardware we would need to train a WaveNet with a receptive field of 30 seconds (or almost half a million timesteps at 16 kHz) may just fit in a desktop computer, so we could just wait until then to give it a try. But if we want to train such models today, we need a different strategy. If we could train separate models to capture structure at different timescales, we could have a dedicated model that focuses on capturing longer-range correlations, without having to also model local signal structure. This seems feasible, seeing as models of high-level representations of music (i.e. scores or MIDI) clearly do a much better job of capturing long-range musical structure already.

We can approach this as a representation learning problem: to decouple learning of local and large-scale structure, we need to extract a more compact, high-level representation $$h$$ from the audio signals $$x$$, that makes abstraction of local detail and has a much lower sample rate. Ideally, we would learn a model $$h = f(x)$$ to extract such a representation from data (although using existing high-level representations like MIDI is also possible, as we’ll discuss later).

Then we can split up the task by training two separate models: a WaveNet that models the high-level representation: $$p_H(h)$$, and another that models the local signal structure, conditioned on the high-level representation: $$p_{X \vert H}(x \vert h)$$. The former model can focus on learning about long-range correlations, as local signal structure is not present in the representation it operates on. The latter model, on the other hand, can focus on learning about local signal structure, as relevant information about large-scale structure is readily available in its conditioning signal. Combined together, these models can be used to sample new audio signals by first sampling $$\hat{h} \sim p_H(h)$$ and then $$\hat{x} \sim p_{X \vert H}(x \vert \hat{h})$$.

We can learn both $$f(x)$$ and $$p_{X \vert H}(x \vert h)$$ together by training an autoencoder: $$f(x)$$ is the encoder, a feed-forward neural network, and $$p_{X \vert H}(x \vert h)$$ is the decoder, a conditional WaveNet. Learning these jointly will enable $$f(x)$$ to adapt to the WaveNet, so that it extracts information that the WaveNet cannot easily model itself.

To make the subsequent modelling of $$h = f(x)$$ with another WaveNet easier, we use a VQ-VAE25: an autoencoder with a discrete bottleneck. This has two important consequences:

• Autoregressive models seem to be more effective on discrete sequences than on continuous ones. Making the high-level representation discrete makes the hierarchical modelling task much easier, as we don’t need to adapt the WaveNet model to work with continuous data.
• The discreteness of the representation also limits its information capacity, forcing the autoencoder to encode only the most important information in $$h$$, and to use the autoregressive connections in the WaveNet decoder to capture any local structure that wasn’t encoded in $$h$$.

To split the task into more than two parts, we can apply this procedure again to the high-level representation $$h$$ produced by the first application, and repeat this until we get a hierarchy with as many levels as desired. Higher levels in the hierarchy make abstraction of more and more of the low-level details of the signal, and have progressively lower sample rates (yielding shorter sequences). a three-level hierarchy is shown in the diagram below. Note that each level can be trained separately and in sequence, thus greatly reducing the computational requirements of training a model with a very large receptive field.

My colleagues and I explored this idea and trained hierachical WaveNet models on piano music26. We found that there was a trade-off between audio fidelity and long-range coherence of the generated samples. When more model capacity was repurposed to focus on long-range correlations, this reduced the capability of the model to capture local structure, resulting in lower perceived audio quality. We also conducted a human evaluation study where we asked several listeners to rate both the fidelity and the musicality of some generated samples, to demonstrate that hierarchical models produce samples which sound more musical.

Hierarchical WaveNet: paper - samples

### Wave2Midi2Wave and the MAESTRO dataset

As alluded to earlier, rather than learning high-level representations of music audio from data, we could also use existing high-level representations such as MIDI to construct a hierarchical model. We can use a powerful language model to model music in the symbolic domain, and also construct a conditional WaveNet model that generates audio, given a MIDI representation. Together with my colleagues from the Magenta team at Google AI, we trained such models on a new dataset called MAESTRO, which features 172 hours of virtuosic piano performances, captured with fine alignment between note labels and audio waveforms27. This dataset is available to download for research purposes.

Compared to hierarchical WaveNets with learnt intermediate representations, this approach yields much better samples in terms of musical structure, but it is limited to instruments and styles of music that MIDI can accurately represent. Manzelli et al. have demonstrated this approach for a few instruments other than piano28, but the lack of available aligned data could pose a problem.

Wave2Midi2Wave: paper - blog post - samples - dataset
Manzelli et al. model: paper - samples

### Sparse transformers

OpenAI introduced the Sparse Transformer model29, a large transformer30 with a sparse attention mechanism that scales better to long sequences than traditional attention (which is quadratic in the length of the modelled sequence). They demonstrated impressive results autoregressively modelling language, images, and music audio using this architecture, with sparse attention enabling their model to cope with waveforms of up to 65k timesteps (about 5 seconds at 12 kHz). The sparse attention mechanism seems like a good alternative to the stacked dilated convolutions of WaveNets, provided that an efficient implementation is available.

Sparse Transformer: paper - blog post - samples

### Universal music translation network

An interesting conditional waveform modelling problem is that of “music translation” or “music style transfer”: given a waveform, render a new waveform where the same music is played by a different instrument. The Universal Music Translation Network31 tackles this by training an autoencoder with multiple WaveNet decoders, where the encoded representation is encouraged to be agnostic to the instrument of the input (using an adversarial loss). A separate decoder is trained for each target instrument, so once this representation is extracted from a waveform, it can be synthesised in an instrument of choice. The separation is not perfect, but it works surprisingly well in practice. I think this is a nice example of a model that combines ideas from both likelihood-based models and the adversarial learning paradigm.

Universal music translation network: paper - code - samples

Dadabots are a researcher / artist duo who have trained SampleRNN models on various albums (primarily metal) in order to produce more music in the same vein. These models aren’t great at capturing long-range correlations, so it works best for artists whose style is naturally a bit disjointed. Below is a 24 hour livestream they’ve set up with a model generating infinite technical death metal in the style of ‘Relentless Mutation’ by Archspire.

Adversarial modelling of audio has only recently started to see some successes, which is why this section is going to be a lot shorter than the previous one on likelihood-based models. The adversarial paradigm has been extremely successful in the image domain, but researchers have had a harder time translating that success to other domains and modalities, compared to likelihood-based models. As a result, published work so far has primarily focused on speech generation and the generation of individual notes or very short clips of music. As a field, we are still very much in the process of figuring out how to make GANs work well for audio at scale.

### WaveGAN

One of the first works to attempt using GANs for modelling raw audio signals is WaveGAN32. They trained a GAN on single-word speech recordings, bird vocalisations, individual drum hits and short excerpts of piano music. They also compared their raw audio-based model with a spectrogram-level model called SpecGAN. Although the fidelity of the resulting samples is far from perfect in some cases, this work undoubtedly inspired a lot of researchers to take audio modelling with GANs more seriously.

WaveGAN: paper - code - samples - demo - colab

### GANSynth

So far in this blog post, we have focused on generating audio waveforms directly. However, I don’t want to omit GANSynth33, even though technically speaking it does not operate directly in the waveform domain. This is because the spectral representation it uses is exactly invertible – no other models or phase reconstruction algorithms are used to turn the spectograms it generates into waveforms, which means it shares a lot of the advantages of models that operate directly in the waveform domain.

As discussed before, modelling the phase component of a complex spectrogram is challenging, because the phase of real audio signals can seem essentially random. However, using some of its unique characteristics, we can transform the phase into a quantity that is easier to model and reason about: the instantaneous frequency. This is obtained by computing the temporal difference of the unwrapped phase between subsequent frames. “Unwrapping” means that we shift the phase component by a multiple of $$2 \pi$$ for each frame as needed to make it monotonic over time, as shown in the diagram below (because phase is an angle, all values modulo $$2 \pi$$ are equivalent).

The instantaneous frequency captures how much the phase of a signal moves from one spectrogram frame to the next. For harmonic sounds, this quantity is expected to be constant over time, as the phase rotates at a constant velocity. This makes this representation particularly suitable to model musical sounds, which have a lot of harmonic content (and in fact, it might also make the representation less suitable for modelling more general classes of audio signals, though I don’t know if anyone has tried). For harmonic sounds, the instantaneous frequency is almost trivial to predict.

GANSynth is an adversarial model trained to produce the magnitude and instantaneous frequency spectrograms of recordings of individual musical notes. The trained model is also able to generalise to sequences of notes to some degree. Check out the blog post for sound examples and more information.

GANSynth: paper - code - samples - blog post - colab

### MelGAN & GAN-TTS

Two recent papers demonstrate excellent results using GANs for text-to-speech: MelGAN34 and GAN-TTS35. The former also includes some music synthesis results, although fidelity is still an issue in that domain. The focus of MelGAN is inversion of magnitude spectrograms (potentially generated by other models), whereas as GAN-TTS is conditioned on the same “linguistic features” as the original WaveNet for TTS.

The architectures of both models share some interesting similarities, which shed light on the right inductive biases for raw waveform discriminators. Both models use multiple discriminators at different scales, each of which operates on a random window of audio extracted from the full sequence produced by the generator. This is similar to the patch-based discriminators that have occasionally been used in GANs for image generation. This windowing strategy seems to dramatically improve the capability of the generator to correctly model high frequency content in the audio signals, which is much more crucial to get right for audio than for images because it more strongly affects perceptual quality. The fact that both models benefited from this particular discriminator design indicates that we may be on the way to figuring out how to best design discriminator architectures for raw audio.

There are also some interesting differences: where GAN-TTS uses a combination of conditional and unconditional discriminators, MelGAN uses only unconditional discriminators and instead encourages the generator output to match the ground truth audio by adding an additional feature matching loss: the L1 distance between discriminator feature maps of real and generated audio. Both approaches seem to be effective.

Adversarial waveform synthesis is particularly useful for TTS, because it enables the use of highly parallelisable feed-forward models, which tend to have relatively low capacity requirements because they are trained with a mode-seeking loss. This means the models can more easily be deployed on low-power hardware while still performing audio synthesis in real-time, compared to autoregressive or flow-based models.

MelGAN: paper - code - samples
GAN-TTS: paper - code (FDSD) - sample

## Discussion

To wrap up this blog post, I want to summarise a few thoughts about the current state of this area of research, and where things could be moving next.

### Why the emphasis on likelihood in music modelling?

Clearly, the dominant paradigm for generative models of music in the waveform domain is likelihood-based. This stands in stark contrast to the image domain, where adversarial approaches greatly outnumber likelihood-based ones. I suspect there are a few reasons for this (let me know if you think of any others):

• Compared to likelihood-based models, it seems like it has been harder to translate the successes of adversarial models in the image domain to other domains, and to the audio domain in particular. I think this is because in a GAN, the discriminator fulfills the role of a domain-specific loss function, and important prior knowledge that guides learning is encoded in its architecture. We have known about good architectural priors for images for a long time (stacks of convolutions), as evidenced by work on e.g. style transfer36 and the deep image prior37. For other modalities, we don’t know as much yet. It seems we are now starting to figure out what kind of architectures work for waveforms (see MelGAN and GAN-TTS, some relevant work has also been done in the discriminative setting38).

• Adversarial losses are mode-seeking, which makes them more suitable for settings where realism is more important than diversity (for example, because the conditioning signal contains most of the required diversity, as in TTS). In music generation, which is primarily a creative application, diversity is very important. Improving diversity of GAN samples is the subject of intense study right now, but I think it could be a while before they catch up with likelihood-based models in this sense.

• The current disparity could also simply be a consequence of the fact that likelihood-based models got a head start in waveform modelling, with WaveNet and SampleRNN appearing on the scene in 2016 and WaveGAN in 2018.

Another domain where likelihood-based models dominate is language modelling. I believe the underlying reasons for this might be a bit different though: language is inherently discrete, and extending GANs to modelling discrete data at scale is very much a work in progress. This is also more likely to be the reason why likelihood-based models are dominant for symbolic music generation as well: most symbolic representations of music are discrete.

### Alternatives to modelling waveforms directly

Instead of modelling music in the waveform domain, there are many possible alternative approaches. We could model other representations of audio signals, such as spectrograms, as long as we have a way to obtain waveforms from such representations. We have quite a few options for this:

• We could use invertible spectrograms (i.e. phase information is not discarded), but in this case modelling the phase poses a considerable challenge. There are ways to make this easier, such as the instantaneous frequency representation used by GANSynth.

• We could also use magnitude spectrograms (as is typically done in discriminative models of audio), and then use a phase reconstruction algorithm such as the Griffin-Lim algorithm39 to infer a plausible phase component, based only on the generated magnitude. This approach was used for the original Tacotron model for TTS40, and for MelNet41, which models music audio autoregressively in the spectrogram domain.

• Instead of a traditional phase reconstruction algorithm, we could also use a vocoder to go from spectrograms to waveforms. A vocoder, in this context, is simply a generative model in the waveform domain, conditioned on spectrograms. Vocoding is a densely conditioned generation task, and many of the models discussed before can and have been used as vocoders (e.g. WaveNet in Tacotron 242, flow-based models of waveforms, or MelGAN). This approach has some advantages: generated magnitude spectrograms are often imperfect, and vocoder models can learn to account for these imperfections. Vocoders can also work with inherently lossy spectrogram representations such as mel-spectrograms and constant-Q spectrograms43.

• If we are generating audio conditioned on an existing audio signal, we could also simply reuse the phase of the input signal, rather than reconstructing or generating it. This is commonly done in source separation, and the approach could also be used for music style transfer.

That said, modelling spectrograms isn’t always easier than modelling waveforms. Although spectrograms have a much lower temporal resolution, they contain much more information per timestep. In autoregressive models of spectrograms, one would have to condition along both the time and frequency axes to capture all dependencies, which means we end up with roughly as many sequential sampling steps as in the raw waveform case. This is the approach taken by MelNet.

An alternative is to make an assumption of independence between different frequency bands at each timestep, given previous timesteps. This enables autoregressive models to produce entire spectrogram frames at a time. This partial independence assumption turns out to be an acceptable compromise in the text-to-speech domain, and is used in Tacotron and Tacotron 2. Vocoder models are particularly useful here as they can attempt to fix the imperfections resulting from this simplification of the model. I’m not sure if anybody has tried, but I would suspect that this independence assumption would cause more problems for music generation.

An interesting new approach combining traditional signal processing ideas with neural networks is Differentiable Digital Signal Processing (DDSP)44. By creating learnable versions of existing DSP components and incorporating them directly into neural networks, these models are endowed with much stronger inductive biases about sound and music, and can learn to produce realistic audio with fewer trainable parameters, while also being more interpretable. I suspect that this research direction may gain a lot of traction in the near future, not in the least because the authors have made their code publicly available, and also because of its modularity and lower computational requirements.

Finally, we could train symbolic models of music instead: for many instruments, we already have realistic synthesisers, and we can even train them given enough data (see Wave2Midi2Wave). If we are able to craft symbolic representations that capture the aspects of music we care about, then this is an attractive approach as it is much less computationally intensive. Magenta’s Music Transformer45 and OpenAI’s MuseNet are two models that have recently shown impressive results in this domain, and it is likely that other ideas from the language modelling community could bring further improvements.

DDSP: paper - code - samples - blog post - colab
Music Transformer: paper - blog post
MuseNet: blog post

### What’s next?

Generative models of music in the waveform domain have seen substantial progress over the past few years, but the best results so far are still relatively easy to distinguish from real recordings, even at fairly short time scales. There is still a lot of room for improvement, but I believe a lot of this will be driven by better availability of computational resources, and not necessarily by radical innovation on the modelling front – we have great tools already, they are simply a bit expensive to use due to substantial computational requirements. As time goes on and computers get faster, hopefully this task will garner interest as it becomes accessible to more researchers.

One interesting question is whether adversarial models are going to catch up with likelihood-based models in this domain. I think it is quite likely that GANs, having recently made in-roads in the densely conditioned setting, will gradually be made to work for more sparsely conditioned audio generation tasks as well. Fully unconditional generation with long-term coherence seems very challenging however, and I suspect that the mode-seeking behaviour of the adversarial loss will make this much harder to achieve. A hybrid model, where a GAN captures local signal structure and another model with a different objective function captures high-level structure and long-term correlations, seems like a sensible thing to build.

Hierarchy is a very important prior for music (and, come to think of it, for pretty much anything else we like to model), so models that explicitly incorporate this are going to have a leg up on models that don’t – at the cost of some additional complexity. Whether this additional complexity will always be worth it remains to be seen, but at the moment, this definitely seems to be the case.

At any rate, splitting up the problem into multiple stages that can be solved separately has been fruitful, and I think it will continue to be. So far, hierarchical models (with learnt or handcrafted intermediate representations) and spectrogram-based models with vocoders have worked well, but perhaps there are other ways to “divide and conquer”. A nice example of a different kind of split in the image domain is the one used in Subscale Pixel Networks46, where separate networks model the most and least significant bits of the image data.

## Conclusion

If you made it to the end of this post, congratulations! I hope I’ve convinced you that music modelling in the waveform domain is an interesting research problem. It is also very far from a solved problem, so there are lots of opportunities for interesting new work. I have probably missed a lot of relevant references, especially when it comes to more recent work. If you know about relevant work that isn’t discussed here, feel free to share it in the comments! Questions about this blog post and this line of research are very welcome as well.

## References

1. Sturm, Santos, Ben-Tal and Korshunova, “Music transcription modelling and composition using deep learning”, Proc. 1st Conf. Computer Simulation of Musical Creativity, Huddersfield, UK, July 2016. folkrnn.org

2. Van den Oord, Kalchbrenner and Kavukcuoglu, “Pixel recurrent neural networks”, International Conference on Machine Learning, 2016.

3. Van den Oord, Kalchbrenner, Espeholt, Vinyals and Graves, “Conditional image generation with pixelcnn decoders”, Advances in neural information processing systems 29 (NeurIPS), 2016.

4. Dinh, Krueger and Bengio, “NICE: Non-linear Independent Components Estimation”, arXiv, 2014.

5. Dinh, Sohl-Dickstein and Bengio, “Density estimation using Real NVP”, arXiv, 2016.

6. Rezende, Mohamed and Wierstra, “Stochastic Backpropagation and Approximate Inference in Deep Generative Models”, International Conference on Machine Learning, 2014.

7. Kingma and Welling, “Auto-Encoding Variational Bayes”, International Conference on Learning Representations, 2014.

8. Bowman, Vilnis, Vinyals, Dai, Jozefowicz and Bengio, “Generating Sentences from a Continuous Space”, 20th SIGNLL Conference on Computational Natural Language Learning, 2016.

9. Goodfellow, Pouget-Abadie, Mirza, Xu, Warde-Farley, Ozair, Courville and Bengio, “Generative Adversarial Nets”, Advances in neural information processing systems 27 (NeurIPS), 2014.

10. Du and Mordatch, “https://arxiv.org/abs/1903.08689”, arXiv, 2019.

11. Arjovsky, Chintala and Bottou, “Wasserstein GAN”, arXiv, 2017.

12. Kolouri, Pope, Martin and Rohde, “Sliced-Wasserstein Autoencoder: An Embarrassingly Simple Generative Model”, arXiv, 2018.

13. Ostrovski, Dabney and Munos, “Autoregressive Quantile Networks for Generative Modeling”, International Conference on Machine Learning, 2018.

14. Hyvärinen, “Estimation of Non-Normalized Statistical Models by Score Matching”, Journal of Machine Learning Research, 2005.

15. Song, Garg, Shi and Ermon, “Sliced Score Matching: A Scalable Approach to Density and Score Estimation”, UAI, 2019.

16. Song and Ermon, “Generative Modeling by Estimating Gradients of the Data Distribution”, Advances in neural information processing systems 32 (NeurIPS), 2019.

17. Van den Oord, Dieleman, Zen, Simonyan, Vinyals, Graves, Kalchbrenner, Senior and Kavukcuoglu, “WaveNet: A Generative Model for Raw Audio”, arXiv, 2016.

18. Mehri, Kumar, Gulrajani, Kumar, Jain, Sotelo, Courville and Bengio, “SampleRNN: An Unconditional End-to-End Neural Audio Generation Model”, International Conference on Learning Representations, 2017.

19. Van den Oord, Li, Babuschkin, Simonyan, Vinyals, Kavukcuoglu, van den Driessche, Lockhart, Cobo, Stimberg, Casagrande, Grewe, Noury, Dieleman, Elsen, Kalchbrenner, Zen, Graves, King, Walters, Belov and Hassabis, “Parallel WaveNet: Fast High-Fidelity Speech Synthesis”, International Conference on Machine Learning, 2018.

20. Ping, Peng and Chen, “ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech”, International Conference on Learning Representations, 2019.

21. Prenger, Valle and Catanzaro, “WaveGlow: A Flow-based Generative Network for Speech Synthesis”, International Conference on Acoustics, Speech, and Signal Procesing, 2019

22. Kim, Lee, Song, Kim and Yoon, “FloWaveNet : A Generative Flow for Raw Audio”, International Conference on Machine Learning, 2019.

23. Ping, Peng, Zhao and Song, “WaveFlow: A Compact Flow-based Model for Raw Audio”, ArXiv, 2019.

24. Serrà, Pascual and Segura, “Blow: a single-scale hyperconditioned flow for non-parallel raw-audio voice conversion”, Advances in neural information processing systems 32 (NeurIPS), 2019.

25. Van den Oord, Vinyals and Kavukcuoglu, “Neural Discrete Representation Learning”, Advances in neural information processing systems 30 (NeurIPS), 2017.

26. Dieleman, Van den Oord and Simonyan, “The challenge of realistic music generation: modelling raw audio at scale”, Advances in neural information processing systems 31 (NeurIPS), 2018.

27. Hawthorne, Stasyuk, Roberts, Simon, Huang, Dieleman, Elsen, Engel and Eck, “Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset”, International Conference on Learning Representations, 2019.

28. Manzelli, Thakkar, Siahkamari and Kulis, “Conditioning Deep Generative Raw Audio Models for Structured Automatic Music”, International Society for Music Information Retrieval Conference, 2018.

29. Child, Gray, Radford and Sutskever, “Generating Long Sequences with Sparse Transformers”, Arxiv, 2019.

30. Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin, “Attention is All you Need”, Advances in neural information processing systems 30 (NeurIPS), 2017.

31. Mor, Wolf, Polyak and Taigman, “A Universal Music Translation Network”, International Conference on Learning Representations, 2019.

32. Donahue, McAuley and Puckette, “Adversarial Audio Synthesis”, International Conference on Learning Representations, 2019.

33. Engel, Agrawal, Chen, Gulrajani, Donahue and Roberts, “GANSynth: Adversarial Neural Audio Synthesis”, International Conference on Learning Representations, 2019.

34. Kumar, Kumar, de Boissiere, Gestin, Teoh, Sotelo, de Brébisson, Bengio and Courville, “MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis”, Advances in neural information processing systems 32 (NeurIPS), 2019.

35. Bińkowski, Donahue, Dieleman, Clark, Elsen, Casagrande, Cobo and Simonyan, “High Fidelity Speech Synthesis with Adversarial Networks”, International Conference on Learning Representations, 2020.

36. Gatys, Ecker and Bethge, “Image Style Transfer Using Convolutional Neural Networks”, IEEE Conference on Computer Vision and Pattern Recognition, 2016.

37. Ulyanov, Vedaldi and Lempitsky, “Deep Image Prior”, IEEE Conference on Computer Vision and Pattern Recognition, 2018.

38. Pons and Serra, “Randomly weighted CNNs for (music) audio classification”, IEEE International Conference on Acoustics, Speech and Signal Processing, 2019.

39. Griffin and Lim, “Signal estimation from modified short-time Fourier transform”, IEEE Transactions on Acoustics, Speech and Signal Processing, 1984.

40. Wang, Skerry-Ryan, Stanton, Wu, Weiss, Jaitly, Yang, Xiao, Chen, Bengio, Le, Agiomyrgiannakis, Clark and Saurous, “Tacotron: Towards end-to-end speech synthesis”, Interspeech, 2017.

41. Vasquez and Lewis, “Melnet: A generative model for audio in the frequency domain”, ArXiv, 2019.

42. Shen, Pang, Weiss, Schuster, Jaitly, Yang, Chen, Zhang, Wang, Skerry-Ryan, Saurous, Agiomyrgiannakis, Wu, “Natural TTS synthesis by conditioning wavenet on mel spectrogram predictions”, IEEE International Conference on Acoustics, Speech and Signal Processing, 2018.

43. Schörkhuber and Klapuri, “Constant-Q transform toolbox for music processing”, Sound and Music Computing Conference, 2010.

44. Engel, Hantrakul, Gu and Roberts, “DDSP: Differentiable Digital Signal Processing”, International Conference on Learning Representations, 2020.

45. Huang, Vaswani, Uszkoreit, Simon, Hawthorne, Shazeer, Dai, Hoffman, Dinculescu and Eck, “Music Transformer: Generating Music with Long-Term Structure ”, International Conference on Learning Representations, 2019.

46. Menick and Kalchbrenner, “Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling”, International Conference on Learning Representations, 2019.

### Musings on typicality

A summary of my current thoughts on typicality, and its relevance to likelihood-based generative models. Continue reading

#### Addendum: quantifying our flawed intuitions

Published on September 01, 2020

#### New Lasagne feature: arbitrary expressions as layer parameters

Published on November 10, 2015


text:
# Decision Tree Classifier to classify values based on values of other columns

I have data with multiple labels, for example

My X set is fromt second to third column, and I want to classify either first column or the last column, so I made my Y the last column.

The goal is so that if I would classify Vios it would return me Car or 0 in other words it can find its way to the first row.

Classification use case:

classify("poodle") #just pretend this is a working function

returns: Pets

How I did it in attempt to train my model:

from sklearn.feature_extraction.text import TfidfVectorizer
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 72)
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(X_train)
tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)
clf3 = RandomForestClassifier().fit(X_train_tfidf, y_train)

I'm using a guide from somewhere on the net that works a bit the same with it, but at the end im getting returned:

ValueError: Found input variables with inconsistent numbers of samples: [5, 4156]

I knew immediately i was doing it wrong. How do I train a model so that it achieves my goal? Any relevant guides or techniques I should be following intead of this? I dont even know the correct way to use vectors on this case.

Couple of things: Cant replicate the problem exactly but if you follow these steps you are not exposing yourself

TfidfVectorizer is CountVectorizer + TfidfTransformer, you are exposing yourself to unnecessary complexity and potential errors

Use pipelines, cant stress this enough, its a compact way to pack all the sklearn transformers together AND THEN use fit, predict methods...

I would advise you to follow something like that, or find similiar problem here

• thabks, but do you think my method of vectorizing it in order to classify it would work in this kind of problem in the first place? – user87205 Dec 20 '19 at 14:16
• I would try Countvectorizer and Tf-idf sperately and they should give good results – vienna_kaggling Dec 20 '19 at 14:19
• Im a bit confused when only using vectorizer, heres my data: prnt.sc/qdm2iz Vectorized it: prnt.sc/qdm2rs, when correlating im getting this error: prnt.sc/qdm2y7 as such I cant fit it to any of my models: prnt.sc/qdm38s – user87205 Dec 20 '19 at 14:59
• I have already categorized it by id: prnt.sc/qdm3v6, so whats left is to train a model with the 'combined' column, that is vectorized, then i thought id be able to predict to which category it belongs to. – user87205 Dec 20 '19 at 15:00
• This is why I thought having a countvectorizer would do the trick, I think this is a factor of how I arrange my data? Im using bagging method to classify, if a vector belong from 1 to 19 categories – user87205 Dec 20 '19 at 15:04

format: plain


text:
# Homework Help: Infinite limit as X tends to infinity

1. Mar 30, 2010

### IdanH14

1. The problem statement, all variables and given/known data
I am required to express in $$\varepsilon - \delta$$ way what I'm suppose to prove in case $$lim_\below{(x \rightarrow \infty)} f(x) = \infty$$

2. Relevant equations
None.

3. The attempt at a solution
So first, intuitively I thought that what this means is that $$f(x)$$ is bigger than any arbitrary number when $$x$$ is bigger than any arbitrary number. So I attempted to combine the $$\varepsilon - \delta$$ definitions of when $$x$$ tends to infinity and when limit $$f(x)$$ tends to infinity.

I came up with this:
$$lim_\below{(x \rightarrow \infty)} f(x) = \infty$$ if for every $$M>0$$ there exists $$N>0$$ so that for every $$x>M$$, $$f(x)>N$$.

I am unsure of whether it's the correct definition. Anyone can verify that?

2. Mar 30, 2010

### tiny-tim

Welcome to PF!

Hi IdanH14! Welcome to PF!

(have a delta: δ and an epsilon: ε and an infinity: ∞ )
Yes , except it's the other way round …

no matter how large N is, we can find an M above which f(x) > N.

(you get the same result if you use 1/f, 1/δ, and 1/ε)

3. Mar 30, 2010

### IdanH14

Thanks! :)
Let me summarize it to see if I got it. It should be
For every N>0 there exists M>0, so that for every x>M, f(x)>N
Right?

Last edited: Mar 30, 2010
4. Mar 30, 2010

### tiny-tim

Right!

5. Mar 30, 2010

### IdanH14

I like you usage of icons. I think I'll adopt it. ;)

Now, I'm trying to solve an exercise in my math book with this principle. Despite the fact I already learned infinite limits arithmetics, I'm required to prove that $$lim_\below{x\rightarrow \infty}) x cos\frac{1}{x} = \infty$$ in this cumbersome way. So I think I have a solution, but again, my insecurities creep in.

So I noticed that the bigger x gets, the closer $$cos\frac{1}{x}$$ gets to 1. If I'll choose $$M=N$$ then, unless they $$N=\infty$$ I'll get something that's smaller than N, because X is multiplied by something which is close to 1, but doesn't equal 1. But if $$M=2N$$ then problem eliminated. Almost.

Why almost? Because if $$\frac{1}{x}>1$$ then $$cos\frac{1}{x}$$ is potentially getting farther from 1. So, what I was thinking was to say $$M=max(1,2N)$$ and then problem solved.

I hope I'm clear enough. Is this a valid proof? Did I find for every N>0 an M>0 that meet the requirements?

Thanks :)

Last edited: Mar 30, 2010
6. Mar 30, 2010

### tiny-tim

That's correct.

I can work out why you're then using M = 2N (because of course M = N doesn't quite work), but you haven't actually specified why M = 2N does work.

(for example, does it work for any N?)


text:
# Find the Longest Repeat in a String solved by 321

March 18, 2014, 8:26 p.m. by Rosalind Team

Although the suffix tree decreases memory requirements from O(|Text|2) to O(|Text|), on average it still requires about 20 times as much memory as Text. In the case of a 3 GB human genome, 60 GB of RAM still presents a memory challenge for most machines. This reveals a dark secret of big-O notation, which is that it ignores constant factors. For long strings such as the human genome, we will need to pay attention to this constant factor, since the expression O(|Text|) applies to both an algorithm with 2 · |Text| memory and an algorithm with 1000 · |Text| memory.

Yet before seeing how we can further reduce the memory needed for multiple pattern matching, we ask you to solve three problems showing how suffix trees can be applied to other pattern matching challenges. The first such problem is the Longest Repeat Problem.

## Longest Repeat Problem

Find the longest repeat in a string.

Given: A string Text.

Return: A longest substring of Text that appears in Text more than once. (Multiple solutions may exist, in which case you may return any one.)

## Sample Dataset

ATATCGTTTTATCGTT

## Sample Output

TATCGTT


text:
Unity 5 Animation attaching

I have created the walking and idle animation for my fps player. I've added a JavaScript script to access it, but when I run it and press button(w), it stops and says that the animation is not attached to the object.

When I drag the walking/idle animation and drop to that object, it creates an animation controller. Again I run it, still same error. I can't find where to attach the animation

Here's my script:

function Update ()
{
if(Input.GetKeyDown("w"))
{
GetComponent.<Animation>().Play("walk_anim2", PlayMode.StopAll);
}
if(Input.GetKeyUp("w"))
{
GetComponent.<Animation>().Play("idle_anim2", PlayMode.StopAll);
}
}

Where do I need to attach the animation?

• Can you screencap the hierarchy where the relevant GameObject is and whatever the inspector is showing with this player selected? – Foggzie Dec 22 '15 at 22:14
• Is the animation controller that you are attaching the one with the animations in it? I have had this issue before when adding an animation controller it just creates an empty on instead. If that is the case simply copy your animations into that controller that is attached to your game object. – Shaun Dec 23 '15 at 7:51
• i've added the screenshot.... i can't understand.. hlp pls !!! – Rajesh Kumar Das Rinku Jan 4 '16 at 17:24
• You have an Animator component attached but your script is asking for an Animation component. These are two different things. – DMGregory Jan 4 '16 at 17:28
• i've added the animation component too. but, that was also not working. anyway, what exactly i have to do right now? – Rajesh Kumar Das Rinku Jan 4 '16 at 17:40

You are using an Animator component but you are referencing Animation in your script. You can see in the error it says MissingComponent. You should use Animator component.


text:
# Find the Equation of the Plane Through the Line of Intersection of the Planes X + Y + Z = 1 and 2x + 3y + 4z = 5 Which is Perpendicular to the Plane X − Y + Z = 0. Also Find the Distance of the Plane, Obtained Above, from the Origi - Mathematics

Find the equation of the plane through the line of intersection of the planes x + y + z = 1 and 2x + 3y + 4z = 5 which is perpendicular to the plane x − y + z = 0. Also find the distance of the plane, obtained above, from the origin.

#### Solution

Equation of the plane through the line of intersection of the two planes is

 (x+y+z−1)+λ(2x+3y+4z−5)=0⇒(1+2λ)x+(1+3λ)y+(1+4λ)z−1−5λ=0                        .........(1)

Normal vector of the required plane is

vecN =(1+2λ)hati+(1+3λ)hatj+(1+4λ)hatk

Since this plane is perpendicular to x-y+z=0, their normals are also perpendicular.

⇒vecN_1.vecN_2=0

⇒((1+2λ)hati+(1+3λ)hatj+(1+4λ)hatk).(hati−hatj+hatk)=0

⇒1+2λ−1−3λ+1+4λ=0

⇒λ=−1/3

Substituting the value of λ in equation (1), to obtain the required equation of the plane

(1+2(−1/3))x+(1+3(−1/3))y+(1+4(−1/3))z−1−5(−1/3)

=0

xz+2=0

To find out the distance of this plane from the origin we need to convert this equation to the standard form:

lx+my+nz=d

Dividing both sides of the equation by
sqrt2

, we get:

x/sqrt2−z/sqrt2=−2/sqrt2

x/sqrt2−z/sqrt2=−sqrt2

Hence, the distance of the plane from the origin is
sqrt2  units.

Concept: Distance of a Point from a Plane
Is there an error in this question or solution?

#### APPEARS IN

NCERT Class 12 Maths
Chapter 11 Three Dimensional Geometry
Q 11 | Page 493


text:
0
0
Entering edit mode
pillailab1 • 0
@94e404f4
Last seen 3 days ago
United States

Hello, I would like to generate Aligned read tracks where the reads are organized in this following fashion: one read per line only. Arranged by reads 3' end Color reads based on their tags (spliced vs. unspliced etc).

The figure is from the following manuscript: https://www.cell.com/molecular-cell/fulltext/S1097-2765(20)30937-0#secsectitle0070

The authors generated the figures from IGV but with a lot of manual manipulation of the svg files. Anyway we can recreate all or any of the features in these images in gviz AlignedTrackReads starting from an alignment file ? Thank you, Manoj `

Gviz • 70 views


text:
# Сonvergence of the sum

This is a problem from my exam on functional analysis. I did only trivial case and now I'm just curious about another cases.

Let $T : H \rightarrow H$ is a linear continuous unitary ($T^*=T^{-1}$) operator, $H$ is a Hilbert space (not necessary). $T_n$ $-$ a sequence of linear operators $T_n \underset{n\rightarrow \infty}{\longrightarrow} T$, $|| T_n - T||\rightarrow 0$.

Suppose that for some subspace $H_1 \subset H$, $\forall h_1 \in H_1 \Rightarrow Th_1=h_1$ and we have a sum on it where $n \rightarrow \infty$ :

$$S_n= \frac{1}{n}\left( T_1 h_1 + T_1 T_2 h_1 + \dots + T_1 \dots T_n h_1 \right), \ \ h_1 \in H_1.$$

The question was:

Find such sufficient сonditions on $T_k$ and $\ T$ that (Limit exist):

$$\underset{n\rightarrow \infty} \lim S_n = h^* \in H$$

I could construct only trivial example when for all $T_k$ we have $T_k h_1 = h_1$.

Conditions $|| T_k|| < 1$ or $||T_1 T_2 ... T_k || < 1$ not sufficient. My question is about examples of some others sufficient conditions (may be even necessary and sufficient)?


text:
Logistic regression is one of the most commonly used algorithms for machine learning and is a building block for neural networks. It is really important to understand the concepts and the derivations of logistic regression. In this post we will explore the fundamentals of logistic regression and also concepts like maximum likelihood function, cross-entropy is covered briefly. In the end, we have provided codes for Logistic Regression in Python from scratch.

Previously, we have covered linear classifier models and we have described the concept of decision boundaries. We have taken some features $$x_i$$, chosen some random weights $$w_i$$ and calculated weighted sums. Then we used a simple sign function to determine if the value is positive or negative. : \begin{aligned} &\text{Model: } \hat{y}_i = \text{sign(Score(}\textbf{x}_i)) \\ &\text{Score(}\textbf{x}_i) = w_1 \textbf{x}_1+ … + w_n \textbf{ x}_n + b \end{aligned} If the value is postive we have given a postive prediction and if the value is negative we have given a negative prediction. But there are problems with these models. The error function we have used in linear classifiers is a simple sign function that gives us output depending on the sign of the values and it is a discrete value.

Using linear classifiers, numbers like .0000001 and 10 are going to have the same predictions. But what if we want a model in which we can predict and at the same time we can tell how confident we are about a prediction. Here’s where probability comes in. Using probability we can tell how confident we are about a prediction. Also in order to use gradient descent, our error function needs to be both continuous and differentiable and we need to move from discrete predictions to continuous predictions.

## Class Probabilities

In logistic regression, we don’t just use discrete predictions like $$+1$$ or $$-1$$, we actually predict a number generally between 0 and 1 which we will consider a probability. These probabilities are extremely useful because they give us an indication of how sure we are about the predictions we make. However, a model can not predict correctly every time. Using probability we can actually predict how confident we are about a prediction.

So, we want to use a continuous error function and we want to use probability. Question is, how can we do that? The way we address both of these problems is by changing our step function from a discrete step function to a continuous step function.

This function is called the sigmoid function and its equation is: $$\sigma(x) = \frac{1}{1+e^{-x}}$$

It gives output between 0 and 1 and it is a continuous function. Now, we need an algorithm that will help us find the best model. The best model is the model that gives a higher probability of the events that actually happen to us. The method is called the Maximum likelihood in which we pick the model that gives the existing labels the highest probability. Thus by maximizing probability, we can pick the best possible model.

## Maximum Likelihood

Maximum likelihood is a quite simple but interesting concept. Suppose, we have some probabilities of events $$p_1, p_2, …, p_n$$. In order to obtain the probabilities of the whole arrangement, we have to multiply these probabilities $$p_1 * p_2 * …* p_n$$. The bigger the individual probabilities will be the resulting value will get bigger as well. Now, all we need to do is to maximize this probability. The equation of maximum likelihood function: $$l(w) = \prod_{i=1}^NP(y_i | \textbf{x}_i, \textbf{w})$$

The maximum likelihood is the product of numbers and the product has got some problems. If we have a big data-set with thousands of probabilities and if these probabilities $$p_1, p_2, …, p_n$$ get really small then the resulting multiplication will become really tiny since the value of the probability is always between 0 and 1. Also, if we have a product of thousands of numbers and we change one of them the product will change drastically. So, we want to avoid products and one alternative is to use sums. One easy way to do this is to use logarithms and the concept is called the cross-entropy.

## Cross-Entropy

In cross-entropy, we take products and we take the logarithms of them. We know: \begin{aligned}ln (a*b) = ln (a) + ln (b)\end{aligned} So, if we take the logarithms of the probabilities we get: $$ln(p_1 * p_2 * …* p_n) = ln(p_1) + ln(p_2)+ … +ln(p_n)$$ Which is a good solution. But there’s still a problem. Every $$ln(p_i)$$ will be a negative number because logarithms of numbers between 0 and 1 is always a negative number and $$ln(1) = 0$$. But if we take the negative of the logarithms of the probabilities we get positive numbers. \begin{aligned}\text{CE} &= -ln(p_1) – ln(p_2) … – ln(p_n) \\ &= – \sum ln(p_i)\end{aligned} This value is called cross-entropy which is a very important concept and will tell us how good our model is. What cross-entropy says is if we have some events and some corresponding probabilities how likely are those events going to happen based on the probabilities. If it is very likely then we have a small cross-entropy, if it is unlikely cross-entropy will be large. It is important to notice that, our goal has changed from maximizing the probability to minimizing the cross-entropy. Now, we need a general formula for cross-entropy.

Let’s say we are going to do an assignment and our supervisor has suggested three books to look at to get some help. Suppose, the probability of getting something useful from these books are $$p_1 = 0.9$$, $$p_2 = 0.7$$ and $$p_3 = 0.2$$ respectively and suppose these are independent events. So, the probability of these events to be happening is 0.9 for the first book, 0.7 for the second book and 0.8 for not having useful content in the third book. We know the cross-entropy will be the sum of the negative logarithms of the probabilities $$-ln(p_1) – ln(p_2) – ln(1-p_3) = – ln(0.9) – ln (0.7) – ln(0.8)$$ Let’s there be another variable called $$y_i$$ which will be 1 if there is something useful in the book and 0 if it’s not. Again suppose, for the first book $$y_1 = 1$$, for the second book $$y_2 = 1$$ and for the third book $$y_3 = 0$$. If we put all this information together, we can find a formula: $$\text{Cross-Entropy} = – \sum_{i=1}^m y_{i}ln(p_i) + (1- y_i)ln(1-p_i)$$

If we look closely, we will notice if there is something useful in the books $$y_i$$ will be 1 and the second term will be zero and if there is not something useful the $$y_i = 0$$ and the first term becomes zero. That is the beauty of this formula and it really gives the sums of the negatives of the logarithms of the probabilities which we have defined as cross-entropy. If we calculate the cross-entropy of the pair: $$\text{CE }[(1,1,0),(0.9,0.7,0.2)] = 0.69$$ which is low since the event is likely. On the other hand, if we calculate $$\text{CE }[(0,0,1),(0.9,0.7,0.2)] = 5.12$$ the cross-entropy is high. By convention we actually use average instead of using the sum: $$\text{Cross-Entropy} = – \frac{1}{m}\sum_{i=1}^m y_{i}ln(p_i) + (1- y_i)ln(1-p_i)$$

In this example, we only had two classes. Having something useful in the book and not having something useful in the book. What if wwe have more than two classes? The concept is called Multi-Class Cross Entropy.

## Multi-Class Cross-Entropy

Suppose again, we need to do an assignment and our supervisor has suggested looking at three different books. But, in this case we have three different topics: Science, Technology, and Sports. For these cases, we use a formula which is more general. $$\text{Cross-Entropy} = -\sum_{i=1}^n\sum_{j=1}^m y_{ij} ln(p_{ij})$$

The above formula is actually the same as the formal we have used for two classes.So, now that we have our error function we can use gradient descent to minimize the function.

Let’s recall that we have n points labelled $$x_1, x_2, \ldots, x_n$$ and we know the error formula and the prediction is given by: \begin{aligned}E &= -\frac{1}{n} \sum_{i=1}^n \left( y_i \ln(\hat{y_i}) + (1-y_i) \ln (1-\hat{y_i}) \right) \\ \hat{y_i} &= \sigma(Wx_i + b) \end{aligned} Where the prediction is given by $$\hat{y_i} = \sigma(Wx^{(i)} + b)$$ and we can get the derivative of the Sigmoid:

\begin{aligned} \sigma(x) &= \frac{1}{1+e^{-x}} \\ \sigma(x)^{’} & = \frac{d}{dx} \frac{1}{1+e^{-x}} \\ &= \frac{e^{-x}}{({1+e^{-x}})^2} = \frac{1}{1+e^{-x}}. \frac{e^{-x}}{({1+e^{-x}})}\\ &= \sigma(x) . (1 – \sigma(x)) \end{aligned}

We can use the above formula to derive: \begin{aligned} \frac{\partial}{\partial w_j}\hat{y} &= \frac{\partial}{dw_j} \sigma(Wx+b)\\ &= \sigma(Wx+b) (1 – \sigma(Wx+b)) . \frac{\partial}{\partial w_j}(Wx+b)\\ &= \hat{y}(1-\hat{y}) . \frac{\partial}{\partial w_j}(Wx+b) \\ &= \hat{y}(1-\hat{y}) . \frac{\partial}{\partial w_j} (w_1x_1 + … + w_jx_j+ … + w_nx_n + b ) \\ &= \hat{y}(1-\hat{y}) . x_j \end{aligned}

Our goal is to calculate the gradient of E at a point $$x = (x_1, \ldots, x_n)$$ given by the partial derivatives $$\nabla E =\left(\frac{\partial}{\partial w_1}E, \cdots, \frac{\partial}{\partial w_n}E, \frac{\partial}{\partial b}E \right)$$

To simplify our calculations, we’ll actually think of the error that each point produces, and calculate the derivative of this error. The total error, then, is the average of the errors at all the points. Now, we can go ahead and calculate the derivative of the error E at a point x with respect to the weight $$w_j$$: \begin{aligned} \frac{\partial}{\partial w_j} E &= \frac{\partial}{\partial w_j} \left[- y \ln(\hat{y}) – (1-y) \ln (1-\hat{y}) \right] \\ &= – y \frac{\partial}{\partial w_j} \ln(\hat{y}) – (1-y) \frac{\partial}{\partial w_j} \ln (1-\hat{y}) \\ &= -y . \frac{1}{\hat{y}}. \frac{\partial}{\partial w_j} \hat{y} – (1-y) . \frac{1}{1-\hat{y}} . \frac{\partial}{\partial w_j} (1 – \hat{y}) \\ &= -y(1-\hat{y}).x_j + (1- y) \hat{y}. x_j \\ &= -(y-\hat{y}) x_j\end{aligned} A similar calculation will show us that $$\frac{\partial}{\partial b} E = – (y – \hat{y})$$

This actually tells us something very important. For a point with coordinates $$(x_1, \ldots, x_n)$$, label y, and prediction $$\hat{y}$$, the gradient of the error function at that point is
$$\left(-(y – \hat{y})x_1, \cdots, -(y – \hat{y})x_n, -(y – \hat{y}) \right)$$ In summary, the gradient is $$\nabla E = -(y – \hat{y}) (x_1, \ldots, x_n, 1)$$

## Logistic Regression Algorithm

We now have all the tools we need to describe the Logistic Regression algorithm. Here is the pseuodocode for the Logistic Regression algorithm:


text:
Hydrodynamic Stability
Get Hydrodynamic Stability essential facts below. View Videos or join the Hydrodynamic Stability discussion. Add Hydrodynamic Stability to your Like2do.com topic list for future reference or share this resource on social media.
Hydrodynamic Stability
A simple diagram of the transition from a stable flow to a turbulent flow. a) stable, b) turbulent

In fluid dynamics, hydrodynamic stability is the field which analyses the stability and the onset of instability of fluid flows. The study of hydrodynamic stability aims to find out if a given flow is stable or unstable, and if so, how these instabilities will cause the development of turbulence.[1] The foundations of hydrodynamic stability, both theoretical and experimental, were laid most notably by Helmholtz, Kelvin, Rayleigh and Reynolds during the nineteenth century.[1] These foundations have given many useful tools to study hydrodynamic stability. These include Reynolds number, the Euler equations, and the Navier-Stokes equations. When studying flow stability it is useful to understand more simplistic systems, e.g. incompressible and inviscid fluids which can then be developed further onto more complex flows.[1] Since the 1980s, more computational methods are being used to model and analyse the more complex flows.

## Stable and unstable flows

To distinguish between the different states of fluid flow one must consider how the fluid reacts to a disturbance in the initial state.[2] These disturbances will relate to the initial properties of the system, such as velocity, pressure, and density. James Clerk Maxwell expressed the qualitative concept of stable and unstable flow nicely when he said:[1]

"when an infinitely small variation of the present state will alter only by an infinitely small quantity the state at some future time, the condition of the system, whether at rest or in motion, is said to be stable but when an infinitely small variation in the present state may bring about a finite difference in the state of the system in a finite time, the system is said to be unstable."

That means that for a stable flow, any infinitely small variation, which is considered a disturbance, will not have any noticeable affect on the initial state of the system and will eventually die down in time.[2] For a fluid flow to be considered stable it must be stable with respect to every possible disturbance. This implies that there exists no mode of disturbance for which it is unstable.[1]

On the other hand, for an unstable flow, any variations will have some noticeable affect on the state of the system which would then cause the disturbance to grow in amplitude in such a way that the system progressively departs from the initial state and never returns to it.[2] This means that there is at least one mode of disturbance with respect to which the flow is unstable, and the disturbance will therefore distort the existing force equilibrium.[3]

## Determining flow stability

### Reynolds number

A key tool used to determine the stability of a flow is the Reynolds number (Re), first put forward by George Gabriel Stokes at the start of the 1850s. Associated with Osborne Reynolds who further developed the idea in the early 1880s, this dimensionless number gives the ratio of inertial terms and viscous terms.[4] In a physical sense, this number is a ratio of the forces which are due to the momentum of the fluid (inertial terms), and the forces which arise from the relative motion of the different layers of a flowing fluid (viscous terms). The equation for this is[2]

${\displaystyle R_{e}={\frac {\text{inertial}}{\text{viscous}}}={\frac {\rho u^{2}}{\frac {\mu u}{L}}}={\frac {\rho uL}{\mu }}={\frac {uL}{\nu }}}$

where

${\displaystyle \rho ={\text{density}}}$
${\displaystyle {\text{u}}={\text{velocity of the fluid flow}}}$
${\displaystyle \mu ={\text{dynamic viscosity}}}$ - measures the fluids resistance to shearing flows
${\displaystyle {\text{L}}={\text{characteristic length}}}$
${\displaystyle \nu ={\text{kinematic viscosity}}={\frac {\mu }{\rho }}}$ - measures ratio of dynamic viscosity to the density of the fluid

The Reynolds number is useful because it can provide cut off points for when flow is stable or unstable, namely the Critical Reynolds number ${\displaystyle R_{c}}$. As it increases, the amplitude of a disturbance which could then lead to instability gets smaller.[1] At high Reynolds numbers it is agreed that fluid flows will be unstable. High Reynolds number can be achieved in several ways, e.g. if ${\displaystyle \mu }$ is a small value or if ${\displaystyle \rho }$ and ${\displaystyle {\text{u}}}$ are high values.[2] This means that instabilities will arise almost immediately and the flow will become unstable or turbulent.[1]

### Navier-Stokes equation and the continuity equation

In order to analytically find the stability of fluid flows, it is useful to note that hydrodynamic stability has a lot in common with stability in other fields, such as magnetohydrodynamics, plasma physics and elasticity; although the physics is different in each case, the mathematics and the techniques used are similar. The essential problem is modeled by nonlinear partial differential equations and the stability of known steady and unsteady solutions are examined.[1] The governing equations for almost all hydrodynamic stability problems are the Navier-Stokes equation and the continuity equation. The Navier-Stokes equation is given by:[1]

${\displaystyle {\frac {\partial \mathbf {u} }{\partial t}}+(\mathbf {u} \cdot \nabla )\mathbf {u} -\nu \,\nabla ^{2}\mathbf {u} =-\nabla p_{0}+\mathbf {b} .}$

where

• ${\displaystyle \mathbf {u} ={\text{velocity field of fluid}}}$
• ${\displaystyle p_{0}={\text{pressure of fluid}}}$
• ${\displaystyle \mathbf {b} ={\text{body force acting on fluid e.g gravity}}}$
• ${\displaystyle \nu ={\text{kinematic viscosity}}}$
• ${\displaystyle {\frac {\partial \mathbf {u} }{\partial t}}={\text{partial derivative of the velocity field with respect to time}}}$
• ${\displaystyle \nabla =\left({\frac {\partial }{\partial x}},{\frac {\partial }{\partial y}},{\frac {\partial }{\partial z}}\right)}$

Here ${\displaystyle \nabla }$ is being used as an operator acting on the velocity field on the left hand side of the equation and then acting on the pressure on the right hand side.

and the continuity equation is given by:

${\displaystyle {\frac {D\mathbf {\rho } }{Dt}}+\rho \,\nabla \cdot \mathbf {u} =0}$

where

• ${\displaystyle {\frac {D\mathbf {\rho } }{Dt}}={\text{material derivative of the density}}}$

Once again ${\displaystyle \nabla }$ is being used as an operator on ${\displaystyle \mathbf {u} }$ and is calculating the divergence of the velocity.

but if the fluid being considered is incompressible, which means the density is constant, then ${\displaystyle {\frac {D\mathbf {\rho } }{Dt}}=0}$ and hence:

${\displaystyle \nabla \cdot \mathbf {u} =0}$

The assumption that a flow is incompressible is a good one and applies to most fluids travelling at most speeds. It is assumptions of this form that will help to simplify the Navier-Stokes equation into differential equations, like Euler's equation, which are easier to work with.

### Euler's equation

If one considers a flow which is inviscid, this is where the viscous forces are small and can therefore be neglected in the calculations, then one arrives at Euler's equations:

${\displaystyle {\frac {\partial \mathbf {u} }{\partial t}}+(\mathbf {u} \cdot \nabla )\mathbf {u} =-\nabla p_{0}}$

Although in this case we have assumed an inviscid fluid this assumption does not hold for flows where there is a boundary. The presence of a boundary causes some viscosity at the boundary layer which cannot be neglected and one arrives back at the Navier-Stokes equation. Finding the solutions to these governing equations under different circumstances and determining their stability is the fundamental principle in determining the stability of the fluid flow itself.

### Linear stability analysis

To determine whether the flow is stable or unstable, one often employs the method of linear stability analysis. In this type of analysis, the governing equations and boundary conditions are linearized. This is based on the fact that the concept of 'stable' or 'unstable' is based on an infinitely small disturbance. For such disturbances, it is reasonable to assume that disturbances of different wavelengths evolve independently. (A nonlinear governing equation will allow disturbances of different wavelengths to interact with each other.)

## Analysing flow stability

### Bifurcation theory

This is a useful way to study the stability of a given flow bifurcation theory with the changes that occur in the structure of a given system, in the case of hydrodynamic stability this is a series of differential equations and their solutions. A bifurcation occurs when a small change in the parameters of the system causes a qualitative change in its behaviour,[1] the parameter that is being changed in the case of hydrodynamic stability is the Reynolds number. It can be shown that the occurrence of bifurcations falls in line with the occurrence of instabilities.[1]

### Laboratory and computational experiments

Laboratory experiments are a very useful way of gaining information about a given flow without having to use more complex mathematical techniques. Sometimes physically seeing the change in the flow over time is just as useful as a numerical approach and any findings from these experiments can be related back to the underlying theory. Experimental analysis is also useful because it allows one to vary the governing parameters very easily and their effects will be visible.

When dealing with more complicated mathematical theories such as Bifurcation theory and Weakly nonlinear theory, numerically solving such problems becomes very difficult and time consuming but with the help of computers this process becomes much easier and quicker. Since the 1980s computational analysis has become more and more useful, the improvement of algorithms which can solve the governing equations, such as the Navier-Stokes equation, means that they can be integrated more accurately for various types of flow.

## Applications

### Kelvin-Helmholtz instability

This is an image, captured in San Francisco, which shows the "ocean wave" like pattern associated with the Kelvin-Helmholtz instability forming in clouds.

The Kelvin-Helmholtz instability (KHI) is an application of hydrodynamic stability that can be seen in nature. It occurs when there are two fluids flowing at different velocities. The difference in velocity of the fluids causes a shear velocity at the interface of the two layers.[3] The shear velocity of one fluid moving induces a shear stress on the other which, if greater than the restraining surface tension, then results in an instability along the interface between them.[3] This motion causes the appearance of a series of overturning ocean waves, a characteristic of the Kelvin-Helmholtz instability. Indeed, the apparent ocean wave-like nature is an example of vortex formation, which are formed when a fluid is rotating about some axis, and is often associated with this phenomenon.

The Kelvin-Helmholtz instability can be seen in the bands in planetary atmospheres such as Saturn and Jupiter, for example in the giant red spot vortex. In the atmosphere surrounding the giant red spot there is the biggest example of KHI that is known of and is caused by the shear force at the interface of the different layers of Jupiter's atmosphere. There have been many images captured where the ocean-wave like characteristics discussed earlier can be seen clearly, with as many as 4 shear layers visible.[5]

Weather satellites take advantage of this instability to measure wind speeds over large bodies of water. Waves are generated by the wind, which shears the water at the interface between it and the surrounding air. The computers on board the satellites determine the roughness of the ocean by measuring the wave height. This is done by using radar, where a radio signal is transmitted to the surface and the delay from the reflected signal is recorded, known as the "time of flight". From this meteorologists are able to understand the movement of clouds and the expected air turbulence near them.

### Rayleigh-Taylor instability

This is a 2D model of the Rayleigh-Taylor instability occurring between two fluids. In this model the red fluid - initially on top, and afterwards below - represents a more dense fluid and the blue fluid represents one which is less dense.

The Rayleigh-Taylor instability is another application of hydrodynamic stability and also occurs between two fluids but this time the densities of the fluids are different.[6] Due to the difference in densities, the two fluids will try to reduce their combined potential energy.[7] The less dense fluid will do this by trying to force its way upwards, and the more dense fluid will try to force its way downwards.[6] Therefore, there are two possibilities: if the lighter fluid is on top the interface is said to be stable, but if the heavier fluid is on top, then the equilibrium of the system is unstable to any disturbances of the interface. If this is the case then both fluids will begin to mix.[6] Once a small amount of heavier fluid is displaced downwards with an equal volume of lighter fluid upwards, the potential energy is now lower than the initial state,[7] therefore the disturbance will grow and lead to the turbulent flow associated with Rayleigh-Taylor instabilities.[6]

This phenomenon can be seen in interstellar gas, such as the Crab Nebula. It is pushed out of the Galactic plane by magnetic fields and cosmic rays and then becomes Rayleigh-Taylor unstable if it is pushed past its normal scale height.[6] This instability also explains the mushroom cloud which forms in processes such as volcanic eruptions and atomic bombs.

Rayleigh-Taylor instability has a big effect on the Earth's climate. Winds that come from the coast of Greenland and Iceland cause evaporation of the ocean surface over which they pass, increasing the salinity of the ocean water near the surface, and making the water near the surface denser. This then generates plumes which drive the ocean currents. This process acts as a heat pump, transporting warm equatorial water North. Without the ocean overturning, Northern Europe would likely face drastic drops in temperature.[6]

## Notes

1. See Drazin (2002), Introduction to hydrodynamic stability
2. See Chandrasekhar (1961) "Hydrodynamic and Hydromagnetic stability"
3. ^ a b c See V.Shankar - Department of Chemical Engineering IIT Kanpur (2014), "Introduction to hydrodynamic stability"
4. ^ See J.Happel, H.Brenner (2009, 2nd edition) "Low Reynolds number hydrodynamics"
5. ^ See the Astrophysical journal letters, volume 729, no. 1 (2009), "Magnetic Kelvin-Helmholtz instability at the Sun"
6. See J.Oakley (2004), "Rayleigh-Taylor instability notes"
7. ^ a b See A.W.Cook, D.Youngs (2009), "Rayleigh-Taylor instability and mixing"

## References

• Drazin, P.G. (2002), Introduction to hydrodynamic stability, Cambridge University Press, ISBN 0-521-00965-0
• Chandrasekhar, S. (1961), Hydrodynamic and hydromagnetic stability, Dover, ISBN 0-486-64071-X
• Charru, F. (2011), Hydrodynamic instabilities, Cambridge University Press, ISBN 1139500546
• Godreche, C.; Manneville, P., eds. (1998), Hydrodynamics and nonlinear instabilities, Cambridge University Press, ISBN 0521455030
• Lin, C.C. (1966), The theory of hydrodynamic stability (corrected ed.), Cambridge University Press, OCLC 952854
• Swinney, H.L.; Gollub, J.P. (1985), Hydrodynamic instabilities and the transition to turbulence (2nd ed.), Springer, ISBN 978-3-540-13319-3
• Happel, J.; Brenner, H. (2009), Low Reynolds number hydrodynamics (2nd ed.), ISBN 9024728770
• Foias, C.; Manley, O.; Rosa, R.; Teman, R. (2001), Navier-Stokes equations and turbulence, Cambridge University Press, ISBN 978-8126509430
• Panton, R.L. (2006), Incompressible Flow (3rd ed.), Wiley India, ISBN 8126509430
• Johnson, Jay R.; Wing, Simon; Delamere, Peter A. (2014), "Kelvin-Helmholtz instability in planetary magnetospheres", Space Science Reviews, 184 (1-4): 1-31, Bibcode:2014SSRv..184....1J, doi:10.1007/s11214-014-0085-z