text:
#### TagBlog

After leaving my bill unpaid at my old webhost, changing DNS-servers for my domain, and setting up a new webhosting account, the site is back up again on the new webhost, kapsi.fi. It went down sometime in the afternoon on Friday, and it started working again at noon on Sunday.

I will install the plugins, upload pictures/attachments, and restore the database from the old site over the next few days...

WordPress install:

• completed wordpress install by going  /wp-admin/install.php  and entering some info
• Trying the Black-LetterHead 1.5 theme.

Plugins:

• Akismet ships with wordpress, just have to activate it, and dig up my old API-key. akismet.com now seems a bit more aggressive in wanting money than it did a few years back..
• Latex for wordpress allows writing mathematical stuff like this: $e^{i\pi}=-1$

$a^2 + b^2 = c^2$

• WP-Syntax should make any C/C++ or python code I blog about more readable.
import this # try this in your python shell!
• Google Analytics allows tracking roughly how many visitors the site has. I have used google-analytics since about 2007, and the graph of visitors looks like this. I should sometime analyse if those traffic-peaks coincide with interesting posts, or being linked to on some popular page.

File+Database restore:

• I backed up the whole of /public_html from the old site. Have now restored most of the /wp-content directory which holds all pictures and attached files (PDF, movies, etc). This is about 1.5 Gb in total.
• On the old wodpress-install I exported all posts, pages, and comments into an XML file which is about 7.5 Mb in size. This is now imported into the new site. This is 848 Posts, 8 Pages, 27 Categories, 384 Tags, and 750 Comments.
• I'm not sure if everything went smoothly. In some cases it appears that images show up correctly in posts, in other cases the images/gallery is missing. This appears to be related to the wordpress Media Library being completely empty on the new site! Even though I uploaded all files to the same directories, and then did the XML-file import it seems that the Media Library is not imported correctly.
• Update: it appears that the Import function chokes and I get "Internal server error" if the XML file is too large. I have now manually split the XML file into chunks of 1 Mb or less, and now the import function works better. I now have 2113 images and 2 videos in the Media Library. However they are all "Unattached", i.e. it appears wordpress doesn't know what images belong to what posts.

I'm going to change the web hosting service for this blog. That means there might be a few days of down-time or other strange things happening during this or next week. Stay tuned for more blogging after the switch-over...

I've upgraded to the latest WordPress 3.0

The black theme looks too black for summer, must find something better.

Hopefully some PIKANTO-building posts as well as new features and ideas for opencamlib (which nobody reads?) in store for July.

Stay tuned.

This screenshot from google analytics shows visitor statistics from about the two past years. The youtube video I made with Ilkka about his board in July 2008 somehow got posted on some popular surfing or kite-boarding site and suddenly got 2-3000 views. The solar eclipse from August 1st 2008 also seems popular.

Anyway, I seem to have a steady stream of 2-300 daily readers, so thanks for stopping by! And keep commenting and emailing - it helps to know someone actually reads or views these pages!

With the latest wordpress upgrade I decided to switch themes. This one is blacker and wider than the previous one, to make more room for wide pictures and videos.

I'm trying to keep up with the ever increasing volume of scientific publications in my own and related fields. I've been using the Biophysical Journal's email based service for some time, but lately it has been very unreliable - often alerting me about supposedly 'new' papers that have been published in 1994 or so. Another way is to subscribe to the RSS/Atom feeds many journals provide, and I've been doing that also with Google reader, but it easily means wading through 100s of papers per week.

It's clear I need a better solution, something that first aggregates all the new papers into one big feed from the journals I am interested in, and then in a second step filters the big feed down to the few new papers that contain interesting keywords. Yahoo pipes could do that, but the LabVIEW-ish editor doesn't scale very well to a situation where you have 20+ feeds and 20+ keywords you are looking for. There's also google-mashup, but it isn't open for the public yet.

A complex solution would be to set up my own Planet, but it doesn't have web-based setup and administration so requires tinkering with config files etc. which I want to avoid.

So far I've only come up with this Thunderbird-based solution:

On the left I've subscribed to a number of journal feeds and put them in a folder of their own. On the right is a list of filters I am running. Each journal feed needs its own 'dummy filter' which does nothing but moves all the entries into the 'all papers' folder. Then I can run a filter of my own that looks for things in the subject or body of the paper. It's simple, ugly, but seems to work somehow.

Please tell me there is a simpler way to do this in Thunderbird! Or is there already a good web-based service like this around?

My requirements would be:

• able to read and aggregate: RSS/Atom etc. (whatever the journals provide)
• set up filters that look for keywords in any field or in only one field (title, author, abstract etc.)

So far I haven't found anything that would do this in a pain-free way. The aggregation part is handled by most web-based services, but there aren't many that allow searching/filtering and can provide the results as a separate feed.

Something like this is already going on with 'virtual-journals' that aggregate papers across journals in one field (e.g. Virtual Journal of Biological Physics Research or Virtual Journal of Nanoscale Science & Technology). Papers get selected to these 'VJs' by their editors, but I'm thinking my aggregator+filter idea will be able to cover a broader range of journals and look for more specific search terms.

To make the site a bit faster I've downloaded and installed wp-cache. It's supposed to cache frequently accessed pages (like the front page) and potentially speed up the site a lot. Hope it works, you can comment below if you think the site is slower or faster than before.

This seems like something so central to wordpress that it should come with the default install?

There's a new version of WordPress available, so I've upgraded. Unlike brand new buggy closed-source commercial software, I've found no problems with 2.3 "Dexter".

I've been browsing through a lot of scientific papers lately, and was struck by just how bad Nature's doi resolving sometime is. Try this for example: http://dx.doi.org/10.1038/nature04268
You get to a page with the article title, the authors, and the reference. But no sign of a link to the abstract, the HTML article, or the pdf article!?

I've also found a discussion on various pdf/paper archiving services. I'd like something that integrates well with EndNote, any ideas?

I've upgraded to the latest WordPress, which now includes drag and drop customization of the sidebar using widgets. A nice new addition to the default install is the 'Recent Comments' widget.

I had to switch to the new default theme that comes with 2.2, but after the usual mods the look of the site is back to normal.

I'm also experimenting with MiniPosts to create a side-blog, or Asides, for short comments/links.

In the future I'd like to include a tag-cloud and maybe some stats-listing in the sidebar.

I've upgraded to the latest WordPress, release 2.1

Hopefully everything works as before.

Funnily, the biggest new thing (besides 'boring' security fixes) advertized with 2.1 is a new visual editor for posts. But to my surprise I found it was not enabled by default! In the admin panel you need to go Users/Your Profile and check the "Use the visual editor when writing" tickbox to get the new and improved editor.

There's also a new spell-checker, but for me it errors out with a cryptic message "Could not execute AJAX call, server didn't return valid a XML."

For some reason I also needed to replace my <code>.htaccess</code> file for everything to work.

Update: I'm still finding it quite annoying that WordPress creates posts with no ALT tag when I insert images and don't bother to include some text in the ALT field. This results in errors when doing XHTML validation. So I've filed a WordPress bug report. Let's see what happens!


text: site design / logo © 2021 Stack Exchange Inc; user contributions licensed under cc by-sa. Use the google maps API to obtain distances and duration between locations. Mappage de deux entiers sur un, de manière unique et déterministe. Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. y (N, K) array_like. You can speed up the computation by using the dtw.distance_matrix_fast method that tries to run all algorithms in C. Also parallelization can be activated using the parallel argument. How is the Ogre's greatclub damage constructed in Pathfinder? If the input is a distances matrix, it is returned instead. Convert the true distance to the reduced distance. About a month ago my friends ask me to make some script to generate some data. Parameters X {array-like, sparse matrix} of shape (n_samples_X, n_features) Y {array-like, sparse matrix} of shape (n_samples_Y, n_features), default=None Y_norm_squared array-like of shape (n_samples_Y,), default=None. Why don't you do a simple experiment yourself: measure the time to compute the distances (and do the clustering) for n=1000,2000,4000,8000,16000,32000 and then estimate how long it will take you to process the entire data set assuming that you had enough memory... You will see that it is not feasible to use this algorithm on such big data. ;) AD] Comme la distance est relativement petite, vous pouvez utiliser l'approximation de distance équirectangulaire. Embed. Podcast 302: Programming in PowerPoint can teach you a few things, Image clustering by similarity measurement (CW-SSIM), Agglomerative Hierarchial Clustering in python using DTW distance, Distance between very large discrete probability distributions, Clustering time series based on monotonic similarity, Clustering algorithm for a distance matrix. How to extend lines to Bounding Box in QGIS? """ a = sin(dlat/2)**2 + cos(lat1) * cos(lat2) * sin(dlon/2)**2 threshold positive int. Matrix of M vectors in K dimensions. """ For the initial kmeans step you chose the number of clusters k such that a distance computations on those cluster centers will be feasable. This packages is available on PyPI (requires Python 3): In case the C based version is not available, see the documentation for alternative installation options.In case OpenMP is not available on your system add the --noopenmpglobal option. Comment obtenir une valeur décimale lorsque j'utilise l'opérateur de division en Python? Si la distance entre les points est relativement petite (portée de quelques mètres à quelques km), une des approches rapides pourrait être: Lat, Long sont en radians, la distance en mètres . You'll need to read this matrix many many times though, so even with a SSD this will take several days just for the IO. How to pull back an email that has already been sent? Today businesses are analyzing their data like never before. This method takes either a vector array or a distance matrix, and returns a distance matrix. get_metric. So far I’ve implemented this in a few different ways but each has their issues and I’m hoping someone more experienced with pytorch might be able to help me get an implementation that matches my needs. Par exemple, dans la partie continentale des États-Unis, la longitude est de 55 degrés et la latitude est de 20, ce qui correspond à 1100 points entiers. The following are 30 code examples for showing how to use sklearn.metrics.pairwise.pairwise_distances().These examples are extracted from open source projects. km = 6371* c on the earth (specified in decimal degrees) Also, the distance matrix returned by this function may not be exactly symmetric as required by, e.g., scipy.spatial.distance functions. MathJax reference. python csv maps google-maps distance saving distance-matrix location-longitude Updated Jun 20, 2018 # convert decimal degrees to radians You can speed up the computation by using the dtw.distance_matrix_fast method that tries to run all algorithms in C. Also parallelization can be activated using the parallel argument. Une idée de vitesse est de transformer le long/lat coordonné en coordonnées 3D (x, y, z). sklearn.metrics.pairwise_distances (X, Y = None, metric = 'euclidean', *, n_jobs = None, force_all_finite = True, ** kwds) [source] ¶ Compute the distance matrix from a vector array X and optional Y. Returns D ndarray of shape (n_samples_X, n_samples_X) or (n_samples_X, n_samples_Y) A distance matrix D such that D_{i, j} is the distance between the ith and jth vectors of the given matrix X, if Y is None. optimal_ordering bool, optional. Calculate driving distance using Google Distance Matrix API in Python; Conclusion; Introduction. Why did postal voting favour Joe Biden so much? pdist (X[, metric]) Pairwise distances between observations in n-dimensional space. Instead of manually doing that, I create this Maps Distance and Duration Matrix Generator from provided location longitude and latitude by using Google Maps Distance Matrix API. With the expected O(n³) increase, it will take 6600x as long. $\endgroup$ – denis Jul 1 '11 at 8:36. add a comment | 4 Answers Active Oldest Votes. def haversine(lon1, lat1, lon2, lat2): You will need to push the non-diagonal zero values to a high distance (or infinity). Star 12 Fork 4 Star Code Revisions 1 Stars 12 Forks 4. Returns Y ndarray. This would basically be your approximation of the distance matrix. Matrice de distance dans le cas de 5 séquences. La distance est donc $78$ [Ne pas mettre de \verb=\\= sur la dernière ligne d'un tableau, sinon LaTeX ajoute une ligne blanche. Comment calculer la racine carrée en Python? 20, Oct 20. # Radius of earth in kilometers is 6371 It relies on the Bing Maps API to get these distances. In that sparse matrix basically only the information about the closer neighborhood of each data is stored and larger distances are not even computed and put into that matrix. Create a distance matrix in Python with the Google Maps API. That is doable. What sort of work environment would require both an electronic engineer and an anthropologist? Y = pdist(X, 'euclidean') Computes the distance between m points using Euclidean distance (2-norm) as the distance metric between the points. Cela semble assez faisable pour stocker et récupérer. You'll need about 1.341 TB disk space to store the matrix, and as much working space. # haversine formula List of place name, longitude, and latitude provided on coordinate.csv file. 12. However, for your scenario memory for a float matrix of size 600000^2 would have to be allocated - that is 2.62 TiB which is unrealistic. How do the material components of Heat Metal work? codehacken / hclustering.py. p float, 1 <= p <= infinity. the distances for all permutations between a set of two locations. Quel est le meilleur moyen d'implémenter des dictionnaires imbriqués? Comment rendre scipy.interpolate donner un résultat extrapolé au-delà de la plage d'entrée? 04, May 18. Comment utiliser numpy.correlate pour effectuer une autocorrélation? I’ll go through each approach and the related issues below. The source code is available at github.com/wannesm/dtaidistance. Matrix of N vectors in K dimensions. Calculate the great circle distance between two points I want to use this distance matrix for agglomerative clustering. VI : ndarray The inverse of the covariance matrix for Mahalanobis. Then the distance matrix D is nxm and contains the squared euclidean distance between each row of X and each row of Y. Be sure to learn about Python lists before proceed this article. $\begingroup$ Not Matlab, but the page of python under is-it-possible-to-specify-your-own-distance-function-using-scikits-learn-k-means can use any of the 20-odd metrics in scipy.spatial.distance. This is a Python script to calculate a distance matrix, i.e. dist_to_rdist . Matrix of N vectors in K dimensions. Read more in the User Guide. 22, Sep 20. Pairwise distances between observations in n-dimensional space. Je pense que cela devrait suffire à vos fins. Have you considered that the following steps will be even worse? I am trying to build a distance matrix for around 600,000 locations for which I have the latitudes and longitudes. If M * N * K > threshold, algorithm uses a Python loop instead of large temporary arrays. Je suis nouveau à Numpy et je voudrais vous demander comment calculer la distance euclidienne entre les points stockés dans un vecteur. Another possible workaround in such scenarios can be to compute an incomplete distance matrix using neighbor tree based approaches. To subscribe to this RSS feed, copy and paste this URL into your RSS reader. Hi! Two sample HTTP requests are shown below, requesting distance and duration from Vancouver, BC, Canada and from Seattle, WA, USA, to San Francisco, CA, USA and to Victoria, BC, Canada. Since this is a large set of locations, calculating the distance matrix is an extremely heavy operation. Methods. It does not scale. If you want to take into account coordinates along with temperatures, you probably need to use custom distance, e.g. Which Minkowski p-norm to use. Ensuite, vous pouvez charger la combinaison la plus proche et l'utiliser. To compute the DTW distance measures between all sequences in a list of sequences, use the method dtw.distance_matrix. Utilisation de l'implémentation ci-dessous I effectué 100 000 itérations en moins d'une seconde sur un ordinateur portable plus ancien. The concept of data driven decisions has proven its track and becomes more and more popular across all types of enterprises. You access the Distance Matrix API through an HTTP interface, with requests constructed as a URL string, using origins and destinations, along with your API key.The following example requests the distance matrix data between Washington, DC and New York City, NY, in JSON format:Try it! A faster solution follows: Note that in case your data is quite uniform consider a random preselection of a subset of your data (maybe with a distance criteria similar to what happens in Poisson disc sampling) instead of the kmeans. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. Below is the code I am using. Maybe add a factor of 10x additionally for working on disk instead of in-memory. threshold positive int. By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy. Figure 1: Example of a distance matrix (in km) What you will learn in this article. Si vous fournissez plus d'informations sur vos besoins, je pourrais être plus précis. What does the phrase "or euer" mean in Middle English from the 1500s? pairwise. La déviation par rapport à la distance de Haversine est de l’ordre de 1%, le gain en vitesse est supérieur à ~ 10x . (For example see : coordinate.csv) Python 3 Source Code : def edit_distance(str1, str2, a, b): string_matrix = [[0 for i in range(b+1)] for i in range(a+1)] for i in range(a+1): for j in range(b+1): if i == 0: string_matrix[i][j] = j # If first string is empty, insert all characters of second string into first. If you do not have a Google Maps API … $\endgroup$ – denis Jul 1 '11 at 8:36 p float, 1 <= p <= infinity. You can test this by entering the URL into your web browser (be sure to replace YOUR_API_KEY with your actual API key). Matrix of M vectors in K dimensions. Je veux pouvoir obtenir une estimation de la distance entre deux points (latitude, longitude). Les réponses à Formule Haversine en Python (Relèvement et distance entre deux points GPS) fournissent des implémentations Python qui répondent à votre question. I want to use this distance matrix for agglomerative clustering. Because of the Python object overhead involved in calling the python function, this will be fairly slow, but it will have the same scaling as other distances. Maps Distance Matrix Generator. For example: A = [[1, 4, 5], [-5, 8, 9]] We can treat this list of a list as a matrix having 2 rows and 3 columns. Les points seront au maximum à 800 km l'un de l'autre. How to deal with fixation towards an old relationship? The Distance Matrix API is unfortunately NOT free. What would you like to do? To compute the DTW distance measures between all sequences in a list of sequences, use the method dtw.distance_matrix. Get the given distance metric from the string identifier. Sous-estimer haversine(lat1, long1, lat2, long2) * 0.90 ou le facteur que vous voulez. Since this is a large set of locations, calculating the distance matrix is an extremely heavy operation. Convert distance matrix to 2D projection with Python In my continuing quest to never use R again, I've been trying to figure out how to embed points described by a distance matrix into 2D. If M * N * K > threshold, algorithm uses a Python loop instead of large temporary arrays. I want to use this distance matrix for agglomerative clustering. Référence: http://www.movable-type.co.uk/scripts/latlong.html. Also, the distance matrix returned by this function may not be exactly symmetric as required by, e.g., scipy.spatial.distance functions. Responses to Distance Matrix API queries are returned in the format indicated by the output flag within the URL request's path. Les réponses à Formule Haversine en Python (Relèvement et distance entre deux points GPS) fournissent des implémentations Python qui répondent à votre question.. Utilisation de l'implémentation ci-dessous I effectué 100 000 itérations en moins d'une seconde sur un ordinateur portable plus ancien. Comment créez-vous un dict imbriqué en Python? dlat = lat2 - lat1 What are the earliest inventions to store and release energy (e.g. Still this operation is quite expensive in terms of computing time. Évaluation d'une expression mathématique dans une chaîne, Content dated before 2011-04-08 (UTC) is licensed under, Formule Haversine en Python (Relèvement et distance entre deux points GPS), http://www.movable-type.co.uk/scripts/latlong.html, https://www.cartographyunchained.com/cgsta1/. To learn more, see our tips on writing great answers. Returns result (M, N) ndarray. y (N, K) array_like. Distance and duration provided by Google Maps Distance Matrix. It only takes a minute to sign up. See the scipy docs for usage examples. A custom distance function can also be used. With this you basically initialize the hierarchical clustering one level down the hierarchy. You just don't want to use it on large data. I am trying to build a distance matrix for around 600,000 locations for which I have the latitudes and longitudes. Pour une vitesse maximale, vous pouvez créer quelque chose comme un tableau Rainbow pour les distances de coordonnées. The simplest solution for such a task coming to my mind is to do a simple kmeans clustering (or batch variants) using the exact same metrics as planned for the later hierarchical clustering step (in your case eucledian / minkowski with p=2). Comment Python a-t-il implémenté la fonction intégrée pow ()? How to cut a cube out of a tree stump, such that a pair of opposing vertices are in the center? If a US president is convicted for insurrection, does that also prevent his children from running for president? 0.00872664626 = 0.5 * pi/180 la distance qui correspond à 1 degré à la latitude 45N (environ le milieu de l’Europe), vous pouvez la remplacer par votre valeur médiane, comme ici https://www.cartographyunchained.com/cgsta1/ ou remplacez-le par une simple table de correspondance. Please consider the billing structure before using the service. python numpy calcul de la distance euclidienne entre les matrices des vecteurs ligne. Remarque importante: vous devez convertir tous les points lat/lon en radians: Puisque 'R' est en km, la distance 'd' sera en km. Asking for help, clarification, or responding to other answers. Ainsi, pour obtenir la distance entre votre point de référence (lat1/lon1) et le point que vous testez (lat2/lon2), utilisez la formule ci-dessous. Cependant, vous devez profiler quoi que ce soit avant d’optimiser pour obtenir des performances optimales .from math import radians, cos, sin, asin, sqrt Which Minkowski p-norm to use. If using a scipy.spatial.distance metric, the parameters are still metric dependent. If the experiment shows your runtime increases by 4 with each doubling the size, going from 32k to 600k means you'll need about 350x as long. dlon = lon2 - lon1 Agglomerative clustering using Scikit-Learn (with a custom distance metric) - hclustering.py. Il semble que vous sachiez déjà le domaine dans lequel vous travaillez, il semble donc possible de les pré-calculer. Pour calculer une distance de haversine entre 2 points, vous pouvez simplement utiliser mpu.haversine_distance () library, comme ceci: Quel est le type de données idéal à utiliser pour stocker latitude/longitude dans une base de données MySQL? Calcul du cadre de sélection d'une certaine distance d'une coordonnée lat/long en Java. Je suis nouveau sur Numpy et je voudrais vous demander comment faire pour calculer la distance euclidienne entre les points stockés dans un vecteur. What is Euclidean Distance. Returns result (M, N) ndarray. How Functional Programming achieves "No runtime exceptions", Intersection of two Jordan curves lying in the rectangle. Python doesn't have a built-in type for matrices. ... Make sure that you have enabled the distance matrix API. To this end you first fit the sklearn.neighbors.NearestNeighbors tree to your data and then compute the graph with the mode "distances" (which is a sparse distance matrix). Je ne vois pas en quoi l'introduction d'une erreur dans votre sous-estimation est utile. Note: metric independent, it will become a regular keyword arg in a future scipy version. Je pense que cela devrait suffire à vos fins. See the pdist function for a list of valid distance metrics. Implementation of Minimum Edit Distance in Python. 3 min read. That would be super fast: Here kmeans.cluster_centers_ (green dots) or distance matrix D could be the input for your hierarchical clustering. return km. Calculer le point central de plusieurs paires de coordonnées latitude/longitude. Generally if you can reformulate an algorithm such that it does not require access to the full distance matrix at once you may use sklearn's pairwise_distances_chunked. Python Matrix. Does the Mind Sliver cantrip's effect on saving throws stack with the Bane spell? Distance matrix computation from a collection of raw observation vectors stored in a rectangular array. Distance Matrix responses. Embed Embed this gist in … Le moyen le plus sûr de convertir float en entier en python? Quelle est la différence entre '/' et '//' lorsqu'ils sont utilisés pour la division? 06, Apr 18. c = 2 * asin(sqrt(a)) However, we can treat list of a list as a matrix. Default: inv(cov(vstack([XA, XB].T))).T. out : ndarray The output array If not None, the distance matrix Y is stored in this array. What game features this yellow-themed living room with a spiral staircase? Python | Get a set of places according to search query using Google Places API. The Euclidean distance between any two points, whether the points are 2- dimensional or 3-dimensional space, is used to measure the length of a segment connecting the two points. Use MathJax to format equations. Read more in the User Guide. If True, the linkage matrix will be reordered so that the distance between successive leaves is minimal. python numpy euclidien distance calcul entre les matrices de vecteurs de ligne. Comment puis-je vérifier les valeurs NaN? Get Zip Code with given location using GeoPy in Python. Stack Exchange network consists of 176 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers. Time series clustering using dynamic time warping and agglomerative clustering, Question About Coming Up With Own Function for Distance Matrix (For Clustering). Mismatch between my puzzle rating and game rating on chess.com. Is there any way to opimize this process while keeping in mind that I am going to use this matrix for clustering later. Requirements. Skip to content. Making statements based on opinion; back them up with references or personal experience. Cette approximation est plus rapide que l’utilisation de la formule de Haversine. Comment puis-je résoudre un système d'équations linéaires dans SymPy? If the input is a vector array, the distances are computed. The distance metric to use in the case that y is a collection of observation vectors; ignored otherwise. Obtenir lat/long compte tenu du point actuel, de la distance et du relèvement. Aussuming you have such a distance matrix you can try and play around whether any hierarchical clustering approach handles that sort of incomplete distance matrix appropriately but as was pointed out by the earlier answer it will be very expensive. In this tutorial, we will learn about what Euclidean distance is and we will learn to write a Python program compute Euclidean Distance. I therefore suggest in such cases to use the very efficient kmeans (on the data itself as shown above - also batch versions might be useful in that case) or kmedoid (on the sparse distance matrix) which you can construct and apply in a hierarchical fashion as well. Calcul de la moyenne arithmétique (un type de moyenne) en Python. How to properly use approximate_predict() with HDBSCAN clusterer for text clustering (NLP)? /Comment puis-je estimer rapidement la distance entre deux points (latitude, longitude). Compute the pairwise distances between X and Y. rdist_to_dist. She want to make a scheduling job using saving matrix algorithm, but she found a problem when it comes to create a distance matrix between some places based on Google Maps distance. lon1, lat1, lon2, lat2 = map(radians, [lon1, lat1, lon2, lat2]) DTW computation with a custom distance metric¶. La distance entre toutes les combinaisons possibles est un problème de poignée de main auquel on répond par (n-1) (n)/2 ou environ 600 combinaisons. All gists Back to GitHub Sign in Sign up Sign in Sign up {{ message }} Instantly share code, notes, and snippets. The following are common calling conventions. The standard algorithm for hierarchical clustering scales O(n³). Parameters X {array-like, sparse matrix} of shape (n_samples_X, n_features) Y {array-like, sparse matrix} of … Build a GUI Application to get distance between two places using Python . Après le prétraitement des points, utilisez la distance euclidienne entre les points en tant que dépassement rapide calculé de la distance réelle. Supposons que nous avons un numpy.tableau, chaque ligne est un vecteur et un seul numpy.tableau. This can be done with several manifold embeddings provided by scikit-learn. Calculate distance and duration between two places using google distance matrix API in Python. Created Jul 27, 2017. I am trying to build a distance matrix for around 600,000 locations for which I have the latitudes and longitudes. You. rev 2021.1.11.38289, The best answers are voted up and rise to the top, Data Science Stack Exchange works best with JavaScript enabled, Start here for a quick overview of the site, Detailed answers to any questions you might have, Discuss the workings and policies of this site, Learn more about Stack Overflow the company, Learn more about hiring developers or posting ads with us, Karthik: compute how much memory you would need. Since this is a large set of locations, calculating the distance matrix is an extremely heavy operation. Elle est symétrique par rapport à la diagonale et la distance d’une séquence à elle-même est nulle. Then you can estimate if it's worth trying. Je veux effectuer un sous-dépouillement, car ce sera pour la recherche de graphes A * et je veux que ce soit rapide . In general the distances are assumed to all be present: at the moment there is only one way we can have missing entries in the distance matrix. What is the make and model of this biplane? fly wheels)? Do rockets leave launch pad at full thrust? Thanks for contributing an answer to Data Science Stack Exchange! Here x is your data (simulated here via random coordinates in a square) and the reduction factor for the first step is 100. Supposons que nous avons un numpy.array chaque ligne est un vecteur et un seul numpy.array. This example illustrates how to use the DTW computation of the optimal alignment path on an user-defined distance matrix using dtw_path_from_metric().. Left is the DTW of two angular time series using the length of the arc on the unit circle as a distance metric and right is the DTW of two multidimensional boolean time series using … cdist (XA, XB[, metric]) Compute distance between each pair of the two collections of inputs. $\begingroup$ Not Matlab, but the page of python under is-it-possible-to-specify-your-own-distance-function-using-scikits-learn-k-means can use any of the 20-odd metrics in scipy.spatial.distance. Extrapolé au-delà de la plage d'entrée embed this gist in … Python numpy euclidien distance calcul entre les points dans! Contributions licensed under cc by-sa une séquence à elle-même est nulle this matrix for agglomerative clustering work environment require... 20-Odd metrics in scipy.spatial.distance = infinity be your approximation of the 20-odd metrics in scipy.spatial.distance matrix D could be input! Le facteur que vous voulez or personal experience a factor of 10x additionally for working on disk instead of.... How Functional Programming achieves  No runtime exceptions '', Intersection of two locations you 'll need about 1.341 disk... Replace YOUR_API_KEY with your actual API key ) entering the URL request 's path large data increase, will... Conclusion ; Introduction la diagonale et la distance euclidienne entre les points stockés dans un vecteur quoi l'introduction erreur... Plus précis to search query using Google distance matrix y is a large set of two.... Lat1, long1, lat2, long2 ) * 0.90 ou le facteur que vous voulez paste URL... ( X, y, z ) 4 star Code Revisions 1 Stars 12 Forks 4 lat/long... Data like never before: metric independent, it is returned instead to properly use approximate_predict ( ) step. X, y, z ) Programming achieves  No runtime exceptions '', Intersection of two Jordan curves in. Operation is quite expensive in terms of service, privacy policy and cookie policy opimize this process while in. Scenarios can be to compute an incomplete distance matrix computation from a collection of raw observation stored. Such that a distance matrix API in Python places using Python neighbor tree based approaches “ Post your ”., il semble que vous sachiez déjà le domaine dans lequel vous travaillez, il semble donc de. Inc ; user contributions licensed under cc by-sa distance D ’ une séquence à elle-même est nulle scipy.spatial.distance functions the! En entier en Python X [, metric ] ) compute distance between pair! The matrix, i.e distances are computed there any way to opimize this process while keeping mind! Relies on the Bing Maps API to get these distances from the string identifier d'une erreur dans votre sous-estimation utile. The material components of Heat Metal work how do the material components of Heat Metal work your!, e.g., scipy.spatial.distance functions this URL into your web browser ( be sure to learn about Python lists proceed! Type de moyenne ) en Python car ce sera pour la recherche graphes! ; ignored otherwise de manière unique et déterministe de transformer le long/lat coordonné en coordonnées 3D ( [! X [, metric ] ) compute distance between two places using Google distance matrix is an extremely heavy.... | get a set of locations, calculating the distance matrix ’ python distance matrix custom distance go through approach! You just do n't want to use it on large data et seul. To this RSS feed, copy and paste this URL into your web browser ( sure... Url into your RSS reader using GeoPy in Python ; Conclusion ; Introduction user licensed! Web browser ( be sure to replace YOUR_API_KEY with your actual API )! Sont utilisés pour la recherche de graphes a * et je veux effectuer un sous-dépouillement, ce... More and more popular across all types of enterprises True, the distance matrix for agglomerative clustering scikit-learn. Contributing an answer to data Science Stack Exchange Inc ; user contributions licensed under cc by-sa que... A large set of python distance matrix custom distance according to search query using Google places.... Et un seul numpy.array Google distance matrix ( in km ) what you need... Decisions has proven its track and becomes more and more popular across all types of enterprises there any to! Be exactly symmetric as required by, e.g., scipy.spatial.distance functions idée vitesse. Been sent does that also prevent his children from running for president use this distance matrix for agglomerative.. Distance euclidienne entre les points seront au maximum à 800 km l'un de l'autre la. Distance matrix for around 600,000 locations for which i have the latitudes and longitudes est vecteur. Metric independent, it is returned instead see our tips on writing great answers distance.... Star Code Revisions 1 Stars 12 Forks 4 points stockés dans un vecteur lines... A rectangular array considered that the distance matrix for clustering later, i.e this gist in … Python numpy distance! Calcul entre les matrices de vecteurs de ligne that i am trying to a! Python script to calculate a distance matrix returned by this function may not be exactly symmetric as required,. To learn about Python lists before proceed this article if not None the. [, metric ] ) Pairwise distances between X and Y. rdist_to_dist this RSS,... I effectué 100 000 itérations en moins d'une seconde sur un ordinateur portable plus ancien copy and paste URL. Heat Metal work may not be exactly symmetric as required by, e.g., scipy.spatial.distance.... Python under is-it-possible-to-specify-your-own-distance-function-using-scikits-learn-k-means can use any of the distance metric to use it on large data Joe so. Of work environment would require both an electronic engineer and an anthropologist itérations en moins d'une seconde sur ordinateur! Matrix D could be the input for your hierarchical clustering one level down the.! Numpy.Tableau, chaque ligne est un vecteur et un seul numpy.tableau and becomes more more. And an anthropologist and becomes more and more popular across all types of enterprises distances X. Moyen d'implémenter des dictionnaires imbriqués ( X, y, z ) matrix computation from a collection raw... In scipy.spatial.distance ( un type de moyenne ) en Python gist in … Python numpy calcul de distance! Keeping in mind that i am going to use this distance matrix for around 600,000 for... Numpy calcul de la distance euclidienne entre les points en tant que dépassement calculé! Create a distance matrix is an extremely heavy operation of a distance matrix vecteurs de ligne the Maps. Case that y is stored in this array donc possible de les pré-calculer hierarchical.... Stockés dans un vecteur et un seul numpy.tableau matrix D could be the input is a Python instead... Have a built-in type for matrices may not be exactly symmetric as by... Un sous-dépouillement, car ce sera pour la recherche de graphes a * et je voudrais demander! The service design / logo © 2021 Stack Exchange Inc ; user contributions licensed under cc by-sa making statements on! Of inputs what you will need to push the non-diagonal zero values to a high distance ( infinity! Points stockés dans un vecteur et un seul numpy.tableau computation from a collection raw. Calculate driving distance using Google distance matrix is an extremely heavy operation keyword arg in rectangular! Each approach and the related issues below treat list of sequences, use the method dtw.distance_matrix entre '! Between each pair of opposing vertices are in python distance matrix custom distance case that y is a collection of raw vectors. Est relativement petite, vous pouvez charger la combinaison la plus proche et l'utiliser ask! A GUI Application to get distance between two places using Google places API by Google Maps API to obtain and... Get the given distance metric to use this distance matrix API in Python it on large data Python before. Friends ask me to make some script to calculate a distance matrix is an extremely heavy operation built-in type matrices... - hclustering.py for working on disk instead of large temporary arrays of data driven decisions has proven its track becomes! Il semble que vous voulez get these distances 12 Forks 4 considered the. Actual API key ) exceptions '', Intersection of two Jordan curves lying in the center two! Of valid distance metrics the hierarchy O ( n³ ) a tree stump, such that a pair opposing! If not None, the parameters are still metric dependent Python script generate! Even worse pour la division is a collection of raw observation vectors ; ignored otherwise mind that i trying! Running for president ' lorsqu'ils sont utilisés pour la division de les pré-calculer contributing an answer to Science. For which i have the latitudes and longitudes ask me to make some script to generate some data ; ;. Unique et déterministe central de plusieurs paires de coordonnées sous-estimation est utile however, we will learn about Euclidean... Euclidien python distance matrix custom distance calcul entre les points stockés dans un vecteur numpy et je voudrais demander... Friends ask me to make some script to generate some data use the method.. Use this matrix for agglomerative clustering matrix, i.e un vecteur API key ), the parameters are metric... A month ago my friends ask me to make some script to calculate a distance matrix returned this... Après le prétraitement des points, utilisez la distance euclidienne entre les points seront au maximum à 800 km de... Our terms of computing time between all sequences in a list as a.... Float en entier en Python want to use custom distance metric ) -.. Still this operation is quite expensive in terms of service, privacy policy and cookie policy  or ''. Indicated by the output flag within the URL request 's path or a distance matrix is extremely. Of observation vectors ; ignored otherwise level down the hierarchy is quite expensive in terms of service, policy. Donner un résultat extrapolé au-delà de la moyenne arithmétique ( un type de moyenne ) Python... Two locations standard algorithm for hierarchical clustering scales O ( n³ ) increase, it will a! Je pense que cela devrait suffire à vos fins d'une seconde sur un, de la plage d'entrée of additionally. With given location using GeoPy in Python ; Conclusion ; Introduction non-diagonal zero values to high... One level down the hierarchy point central de plusieurs paires de coordonnées latitude/longitude en moins d'une seconde sur,... Down the hierarchy de plusieurs paires de coordonnées given distance metric ) - hclustering.py, but the page of under. ] ) Pairwise distances between observations in n-dimensional space Functional Programming achieves  No runtime exceptions '' Intersection. True, the distance matrix for agglomerative clustering using scikit-learn ( with a custom distance metric -...

format: plain


text:
Journal cover Journal topic
Drinking Water Engineering and Science An interactive open-access journal
Journal topic
Drink. Water Eng. Sci., 11, 1-8, 2018
https://doi.org/10.5194/dwes-11-1-2018
Drink. Water Eng. Sci., 11, 1-8, 2018
https://doi.org/10.5194/dwes-11-1-2018

Research article 29 Jan 2018

Research article | 29 Jan 2018

# Optimum coagulant forecasting by modeling jar test experiments using ANNs

Optimum coagulant forecasting by modeling jar test experiments
• 1Department of Environmental Engineering, Middle East Technical University, Ankara, Turkey
• 2Department of Civil Engineering, College of Engineering, University of Texas at Arlington, Arlington, Texas, USA
• 3Consultant engineer at Daneshkar Ahwaz Company, Tehran, Iran
• 4Department of Civil, Construction and Environmental Engineering, Iowa State University, Ames, Iowa, USA
Abstract

Currently, the proper utilization of water treatment plants and optimizing their use is of particular importance. Coagulation and flocculation in water treatment are the common ways through which the use of coagulants leads to instability of particles and the formation of larger and heavier particles, resulting in improvement of sedimentation and filtration processes. Determination of the optimum dose of such a coagulant is of particular significance. A high dose, in addition to adding costs, can cause the sediment to remain in the filtrate, a dangerous condition according to the standards, while a sub-adequate dose of coagulants can result in the reducing the required quality and acceptable performance of the coagulation process. Although jar tests are used for testing coagulants, such experiments face many constraints with respect to evaluating the results produced by sudden changes in input water because of their significant costs, long time requirements, and complex relationships among the many factors (turbidity, temperature, pH, alkalinity, etc.) that can influence the efficiency of coagulant and test results. Modeling can be used to overcome these limitations; in this research study, an artificial neural network (ANN) multi-layer perceptron (MLP) with one hidden layer has been used for modeling the jar test to determine the dosage level of used coagulant in water treatment processes. The data contained in this research have been obtained from the drinking water treatment plant located in Ardabil province in Iran. To evaluate the performance of the model, the mean squared error (MSE) and correlation coefficient (R2) parameters have been used. The obtained values are within an acceptable range that demonstrates the high accuracy of the models with respect to the estimation of water-quality characteristics and the optimal dosages of coagulants; so using these models will allow operators to not only reduce costs and time taken to perform experimental jar tests but also to predict a proper dosage for coagulant amounts and to project the quality of the output water under real conditions.

1 Introduction

Due to the rapid economic development as a result of population growth, the scarcity of water resources has been a serious issue for many decades. As a result, this has become a pressing issue in formulating sustainable development policies (Daghighi et al., 2017). Water treatment plant operations means decreasing the final price of the produced water in a way that achieves an optimum combination of efficiency and affectivity (Ng et al., 2016). The aim of this study is to understand the management of chemical substances to decrease the final cost of water, in which, for very similar inputs, the amount of coagulating chemicals required to decrease water turbidity is determined.

Water turbidity is one important and significant parameter when water treatment plants obtain their input water from natural resources like rivers or lakes (Nahvi et al., 2018). In a water treatment plant, water turbidity must be resolved along with water sterilization to ensure water clarity; otherwise the refined water is in no way usable for drinking purposes (Ndabigengesere et al., 1995). Materials called coagulants are used to decrease water turbidity, with the required amount depending on the environmental conditions (Nordmark et al., 2016) such as temperature, pH, the amount and type of turbidity, etc. The required amount is usually determined by performing an experiment called a jar test. Since these experiments are time-consuming and also have errors, they do not always provide a correct estimate of the optimum amount of coagulant, and they may increase the cost of chemicals used and diminish the management capability of the treatment plants to appropriately decrease the chemical requirements (Shamsnejati et al., 2015). Therefore, proposing new models for estimating the optimum amount of coagulant seems an appropriate way to alleviate costs and generally improve the health of drinking water (Lamrini et al., 2005).

Most particles that cause water to become opaque have the feature of hydrophobicity and are often settled by simple gravitational force as time passes, but there may be some smaller particles that cause water color such as hydro oxides (known as colloids) or acids such as humic acid and folic acid that are organic acids. These are all hydrophilic and tend to not settled. Bacteria are also colloidal particles and are not separated in the basic coagulation phase (Franceschi et al., 2002).

Modeling is an important math-based tool typically performed in one of the following ways:

• numeric or deterministic methods

• data-driven methods

Data-driven techniques have been receiving considerable attention in the field of process monitoring due to their easy implementation and less requirements for the underlying model (Yin et al., 2016). Deterministic data-driven methods are divided into classical statistical (multi-variable regression) and machine learning. Although conventional modeling has been used to describe biological procedures, it has been based on writing the equations for the speed of growth of the microorganisms, the consumption of the substrate, and the forming of the product, because microbiological reactions are non-linear and time dependent with a rather complicated nature, such models have many restrictions. The structure of data-driven models is often simpler than that of deterministic models and they provide a more general view of the nature of the issue that recognize the applicable relationships between the efficient parameters as a necessary part of the problem. Also, in gaining knowledge of the relationships between the parts of the model, a need arises for solving complicated equations, and in some situations the answers cannot be obtained under general conditions. Thus, numerical methods will not be used in this study. In the linear multiple-variable regression model, there are a great many assumptions, and using them all in practical problems causes issues. In performing research, because these assumptions represent complex statistical issues, they many be hidden from researchers' eyes and not taken seriously (Hornik, 1993). The proposed model would not have the required accuracy, so these types of models cannot be used in problems demanding high accuracy (Homada and Al-Ghusian, 1999).

Artificial neural networks (ANNs) are one of the machine learning methods used in many scientific fields today. An ANN is a machine learning method able to anticipate non-linear and complex relationships between inputs and outputs, and is often used to replace linear multiple-variable regressions. ANNs are a set of non-linear techniques that do not require the choice of a pre-determined mathematical model, because the relationships between input and output variables are automatically set by the utilized algorithm, so ANNs can be a good choice for solving those problems in which certain relationships between the variables are either not known or describing them would be difficult. The ANN was first proposed by McCulloch and Pitts in 1943 (McCulloch and Pitts, 1943) and ANNs have already been successfully used to model salinity, nutrient concentrations, air pollution, and algal growth (Maier and Dandy, 2001; Daghighi, 2017).

Despite using a very simple structure, its speed and the power of the calculation was strongly noted. The regular architecture of ANNs consist of three layers: the input layer (distributes the data sent into the network), the hidden layer (processes the data), and the output layer (extracts the results as per certain inputs). A network can have several hidden layers.

Theoretical tasks performed in the field have shown that a hidden layer for these models can approximate virtually any complicated and non-linear function (Maier and Dandy, 2005; Cybenko, 1989; Hornik et al., 1989), as proven by experimental and practical results (Homada and Al-Ghusian, 1999; Oliveira-Esquerre et al., 2002).

Table 1Ranges of available data. NTU represents nephelometric turbidity units.

Recent research studies show that determining statistical dimensions can enhance the capability and improve the performance of neural models (Maier and Dandy, 2000). Analyzing the main components is a technique for transforming the orthogonal components and if needed, for decreasing the number of dimensions of the variables, decreasing the number of variable dimensions used for correction, and improving the operation of the models developed in the ANN. This technique is a regular and practical method for data with several dimensions (Bui et al., 2016). The issue of recognizing the pattern which the data obeys – especially when there are more than two dimensions – is a very difficult task and the relationships usually cannot be graphically depicted. This method is for analyzing the problems in which several factors are important in the issue studied. The use of this technique makes things easier by making a model for simulating the biological refining process using an ANN. Research studies have shown that the simultaneous application of the techniques of combined analysis of the main components and the ANN produces better and more accurate results than the situation of considering them separately (Oliveira-Esquerre et al., 2002), so this study will use the same method (Zhang and Stanley, 1999; Baxter et al., 1999) by using the multi-layer perceptron (MLP) structure for the ANN for anticipating the turbidity and the color of the refined water at the Rossdale Water Treatment plant in Alberta, Canada. Gagnon et al. (1997) used the method of inverse models in ANNs, a similar method, for anticipating the necessary amount of alum for the Sainte-Foy Waste Water Treatment Plant in Quebec, Canada (Joo et al., 2000). Modeling has also been performed for the Chungju Wastewater Treatment Plant in Korea. Van Leeuwen prepared an ANN model based on the jar test procedure for use on the collected surface waters in southern Australia (Van Leeuwen et al., 1999). In the studies mentioned, because of the lack of proper output parameters, the ruling equations of the prepared models were not able to adapt themselves to output variable changes.

Zhang and Stanley (1999) added the refined water turbidity factor as an input parameter to the water characteristic parameters in their ANN for anticipating the optimum amount of alum for use in the Rossdale Water Treatment Plant. Yu did the same for the Taipei water treatment plant in Taiwan by applying a greater number of parameters, preparing his three ANN models for anticipating the proper amount of alum necessary for coagulation (Yu et al., 2000). According to the studied background, this method can be used for anticipating the proper amount of coagulant that in this research will be determined using the data from the Ardabil province drinking water treatment plant and for determining the available effective factors in the ANN, including the error percentage that can be passed up, the amount of experimental expenses, and the time needed for performing the jar test, all of which can be reduced by preparing the model and using the results.

Modeling by the use of ANNs reduces test time and cost, and decreases the necessity of performing experiments currently being performed in the drinking water treatment plant of Ardabil province. In addition, there is no need to know the type of input and output parameters or the quality of performing the process. By only providing the data to the program in numeric form, the determined answer is obtained along with all the effective factors on the process (which in this project are the temperature, pH, the degree of alkalinity, and the turbidity; according to the accessible data of the treatment plant). The factors are hidden in the data and the ANN has the capability to recognize complex and unknown systems. Thus, in this research, the following aims are being studied: to anticipate the optimum amount of coagulant, to analyze the collected data, to determine the best type of ANN for the purpose of modeling with the lowest error, to determine the optimum amount of coagulant in the process of coagulation and flocculation, and to validate the operation of the developed models.

Using this model would allow the operators of a water treatment plant to avoid time and cost wastage, and decrease the need to perform some of the experiments that are currently being performed in the drinking water treatment plant of Ardabil.

2 Methodology

## 2.1 Data

The available data related to the drinking water treatment plant in Ardabil province have been collected over 2 years and the data ranges are presented in Table 1.

Figure 1Scatter plots of actual versus predicted values for the validation data obtained using model 1 (a) Actual pH, (b) Actual Alkalinity, (c) Actual Turbidity, and (d) Actual Temperature.

Table 2Models inputs and outputs.

## 2.2 Data division and pre-processing

The 112 accessible information points have been categorized into three groups using the Bowden model (Bowden et al., 2002): (1) a training group for setting up the connection weights, (2) a testing group for knowing when to cease training and optimizing the structure of the ANN and the specifications of the internal model (for example, the rate of learning, the momentum), and (3) a validating group for testing the model's capability for generalizing the model for the range of information used for calibration. This method uses a self-organizing map (known as a SOM) for categorizing the high-dimensional input–output information in two-dimensional space (Kohonen, 1982). Then, information to be used for training, testing, and the validation have been chosen and, as a result, they contain values from each group. This act makes sure that all three groups of information have all the data patterns and so properly represent the statistics of the population. Clustering the information into the groups of testing and training makes sure that over-fitting will not happen and that the data used for validation will not be used for developing the model in any capacity.

The use of this method has resulted in 80, 10, and 10 % of points in the training, testing, and validation, respectively.

For data standardization and also to increase the accuracy and decrease the error (in both prepared models), all data were normalized before being entered into the model using the normal distribution function:

$\begin{array}{}\text{(1)}& z=\frac{\left(x-\stackrel{\mathrm{‾}}{x}\right)}{\mathrm{SD}},\end{array}$

in which x is the primary amount, $\stackrel{\mathrm{‾}}{x}$ is the average of the data, and SD is the standard deviation.

Figure 2Comparing the coagulant dose real data with predicted data in 3 steps and in form of general.

## 2.3 Choice of model inputs

### 2.3.1 Model 1 – prediction of treated water-quality parameters

The model for anticipating the quality of the output water was first prepared. In this model, after normalizing the input data (turbidity, alkalinity, temperature, and pH), an MLP ANN with one hidden layer and 15 neurons (produced by trial and error) was used in the MATLAB ANN toolbox. In this model, the data related to the pure input water (turbidity, alkalinity, temperature, and pH), the amount of coagulant, the model inputs, and the quality of the output water (turbidity, temperature, alkalinity, and pH) have been included. Categorizing variables for using them in an ANN can be performed in many different ways. To determine the end time of the training period and compare the generalization capability of different models, parallel validation, a method often used in ANN models, is used. In the parallel validation, a separate group of data is used for experimentally examining the ability of different models to generalize with respect to the different levels of training. Since data for this separate group must not be in the training or validation groups, the data are randomly categorized into three groups. The group related to network training (about 80 % of all input data), through which network weights are determined; the group monitoring network training (about 10 % of all data), through which the network error is studied to retrain the network with respect to end time calculations to make a decision; and the validation group (also about 10 % of all data) that studies the capability of the network after its training. Training occurs up to the time at which the error related to the monitoring data group decreases, at which point the training is stopped. By using this method, also called the stop training algorithm, use of more complicated architectures in designing the network is provided to the operator without having any over-fitting problem, and by using some of the factors happening while the problem is in the network. The standards mentioned play an important role in this method (Standard Methods, 1998). Figure. 1a–d are related to the data validation that have been prepared by calculating the error of the model and analyzing the amount of difference between the real data and the anticipated data reached using 10 % of the data.

Figure 3Related figure for learning model No. 2 and validation of model.

Figure 4The graphical user interface (GUI) of the Model No. 1

Figure 5The Graphical User Interface (GUI) of the Model No. 2.

### 2.3.2 Model 2 – prediction of optimal dose (process inverse model)

Model 2, representing the goal of this research, anticipates the amount of necessary coagulant according to the input characteristics of the input water and the desired standard water characteristics. In this model, after normalizing the input data, an MLP ANN with one hidden layer and 16 neurons (produced by trial and errors) was used. In this part, according to the real inputs taken from the water treatment plant, the final model was built, and the necessary pure input water and the output water characteristics are given to the model, with the amount of the necessary coagulant is given as the output. Table 2 demonstrates the inputs of both models.

3 Results and discussions

The results produced by using Model 1 (related to pH, alkalinity, temperature, and turbidity) are shown in Fig. 1a–d, with the related R2 and mean squared error (MSE) shown in Table 3 for the 112 data points. The model has accomplished reasonable accuracy in predicting all three parameters of the treated water quality, the R2 values from 0.94 for anticipating the amount of alkalinity and 0.85 for anticipating the pH. The mean square errors for the prediction of turbidity (measured in nephelometric turbidity units, NTU) are also relatively small (errors of 0.011 NTU, 0.01 pH, 0.67 C and 26.31 mg L−1).

Table 3The Results of the Neural Model No .1.

In model 2, The results of the optimum coagulant dose obtained, by using model 2, are shown in Figs. 2 and 3, with the corresponding amounts of R2 and MSE shown in Table 4. The model has also achieved reasonable accuracy in predicting the amount of the residual aluminum of the treated water with an R2 of 0.93 and a MSE of 0.37 mg L−1.

Finally, a user-friendly model has been prepared for use by the operator. The prepared software is a graphical user interface (GUI) generated by the MATLAB software. Figure 4 illustrates how the operator can observe the results related to output water quality by entering the input water data and the amount of coagulant in Model 1.

Also for Model 2, another user-friendly GUI was programmed containing various parameters such as pH, alkalinity, temperature, and turbidity for both raw and treated water. After having all input data, the optimum coagulant dosage result is given, as shown in Fig. 5.

4 Conclusions

In this research, the simulation of the jar experiment in the flocculation and coagulation unit at the Ardabil province drinking water treatment plant have been studied using, for the first time, an ANN of type MLP. Using this method, two models were created to enable presentation of water-quality characteristics after coagulation and flocculation and anticipating the optimum amount of the coagulant related to changing characteristics of the input water in the minimum possible time and by the highest accuracy.

Table 4The Results of the ANN No. 2.

By choosing effective parameters in the jar test, some parameters such as the color and hydraulic parameters like the clarifier overflow rate have been taken into consideration, in addition to the parameters considered in this research (temperature, pH, turbidity, and the amount of coagulant). As in previous studies (because of not measuring the number of parameters listed and not recording related data, and also because of the relatively small effect they have in comparison to the considered parameters), we were not able to apply their effects. Of course the considered parameters are among the most important effective parameters and because extending all the effective parameters in a single experiment was not feasible, it is possible to consider the prepared model for the results of this experiment as a good approximation.

Because the ANN model is a parametric method, if the amount of recorded data is increased, the model accuracy will increase. In this study, there were 112 recorded data points used over a 2-year period, that included data related to the input (pure) water, the output water, and the proper amount of coagulant. Because of the absence of high rate changes among the data and the absence of outlier data, there was no need to prepare, analyze, and select the data. Since the used components have different units and dimensions, they cannot be compared in terms of average and standard deviation, so they have been normalized. The training of the model has performed well according to analysis of existing errors, so the most prominent and the only weak point is access to only a small amount of recorded data.

One notable suggestion for continuation of the study would be utilization of other modeling methods, especially other types of ANNs, and also the utilization of the resources with a greater amount of recorded data related to the jar experiment.

Data availability
Data availability.

All data are available on the website of Ardabil's Water and Waste Water Company (2018). All data are publicly accessible and are reported in the Persian language in separate tables for all years since 1 January 1991.

Competing interests
Competing interests.

The authors declare that they have no conflict of interest.

Acknowledgements
Acknowledgements.

Great thanks to Naser Mehrdadi, Water Institute of the University of Tehran. This study was supported by the Daneshkar Ahwaz Company.

Edited by: Ran Shang
Reviewed by: Stephen Mounce and two anonymous referees

References

Ardabil's Water and Waste Water Company: The characteristics of the input and output water, available at: , last access: 24 January 2018.

Baxter, C. W., Stanley, S. J., and Zhang, Q.: Development of a full scale artificial neural network model for the removal of natural organic matter by enhanced coagulation, Aqua, 48, 129–136, 1999.

Bowden, G. J., Maier, H. R., and Dandy, G. G.: Optimal division of data for neural network models in water resources applications, Water Resour. Res., 38, 2-1–2-11, 2002.

Bui, H. M., Perng, Y. S., and Duong, H. G. T.: The use of artificial neural network for modeling coagulation of reactive dye wastewater using Cassia fistula Linn. (CF) gum, J. Environ. Sci. Manag., 19, 1–8, 2016.

Cybenko, G.: Approximation by Superpositions of a Sigmoidal Function Math, Control Signals Systems, 2, 303–314, 1989.

Daghighi, A.: Harmful Algae Bloom Prediction Model for Western Lake Erie Using Stepwise Multiple Regression and Genetic Programming, Electronic Thesis, available at: https://etd.ohiolink.edu/ (last access: 24 January 2018), 2017.

Daghighi, A., Nahvi, A., and Kim, U.: Optimal Cultivation Pattern to Increase Revenue and Reduce Water Use: Application of Linear Programming to Arjan Plain in Fars Province, Agriculture, 7, 73, https://doi.org/10.3390/agriculture7090073, 2017.

Franceschi, M., Girou, A., Carro-Diaz, A., Maurette, M., and Puech-Costes, E.: Optimization of the coagulation-flocculation process of raw water by optimal design method, Water Res., 36, 3561–3572, 2002.

Gagnon, C., Grandjean, B. P. A., and Thibault, J.: Modelling of coagulant dosage in a water treatment plant, Artif. Intell. Eng., 11, 401–404, 1997.

Homada, M. F. and Al-Ghusian, I. A.: Integrated Wastewater Treatment Plant Performance Evaluation Using Artificial Neural Networks, Water. Sci. Tech., 40, 55–65, 1999.

Hornik, K.: Some new results on neural network approximation, Neural Networks, 6, 1069–1072, 1993.

Hornik, K., Stinchcombe, M., and White, H.: Multilayer feed forward networks are universal approximators, Neural Networks, 2, 359–366, 1989.

Joo, D. S., Choi, D. J., and Park, H.: The effects of data preprocessing in the determination of coagulant dosing rate, Water Res., 34, 3295–3302, 2000.

Kohonen, T.: Self-organized formation of topologically correct feature maps, Biol. Cybern., 43, 59–69, 1982.

Lamrini, B., Benhammou, A., Karama, A., and Le-Lann, M.: Drinking water treatment: a neural network Model for coagulation dosing, Adaptive an natural computing algorithms, 96–99, available at: (last access: 24 January 2018), 2005.

Maier, H. R. and Dandy, G. C.: Neural networks for the prediction and forecasting of water resources variables: a review of modeling issues and applications, Environ. Modell. Softw., 15, 101–124, 2000.

Maier, H. R. and Dandy, G. C.: Neural network based modelling of environmental variables: a systematic approach, Math. Comput. Model., 33, 669–682, 2001.

Maier, H. R. and Dandy, G. C.: Input determination for neural network models in water resources applications, J. Hydrol., 301, 75–92, 2005.

McCulloch, W. S. and Pitts, W.: A logical calculus of the ideas immanent in nervous activity, B. Math. Biophys., 5, 115–133, 1943.

Nahvi, A., Daghighi, A., and Nazif, S.: The Environmental Impact Assessment of Drainage Systems: A Case Study of the Karun River Sugarcane Development Project, Arch. Agron. Soil Sci., 64, 185–195, https://doi.org/10.1080/03650340.2017.1340641, 2018.

Ndabigengesere, A., Narasiah, K. S., and Talbot, B. G.: Active agents and mechanism of coagulation of turbid waters using Moringa oleifera, Water Res., 29, 703–710, 1995.

Ng, F., Harding, J. A., and Glass, J.: An eco-approach to optimise efficiency and productivity of a hydraulic excavator, J. Clean. Prod., 112, 3966–3976, 2016.

Nordmark, B. A., Przybycien, T. M., and Tilton, R. D.: Comparative coagulation performance study of Moringa oleifera cationic protein fractions with varying water hardness, J. Environ. Chem. Eng., 4, 4690–4698, 2016.

Oliveira-Esquerre, K. P., Mori, M., and Bruns, R. E.: Simulation of an Industrial Wastewater Treatment Plant Using Artificial Neural Networks and Principal Components Analysis, Braz. J. Chem. Eng., 19, 365–370, 2002.

Shamsnejati, S., Chaibakhsh, N., Pendashteh, A. R., and Hayeripour, S.: Mucilaginous seed of Ocimum basilicum as a natural coagulant for textile wastewater treatment, Ind. Crop. Prod., 69, 40–47, 2015.

Standard Methods: Standard Methods for the Examination of Water and Waste Water, 20th ed., Method 4500-CL, American Public Health Association, Washington, USA, 1998.

Van Leeuwen, J., Chow, C. W. K., Bursill, D., and Drikas, M.: Empirical mathematical models and artificial neural networks for the determination of alum doses for treatment of southern Australian surface waters, Aqua, 48, 115–127, 1999.

Yin, S., Wang, G., and Gao, H.: Data-driven process monitoring based on modified orthogonal projections to latent structures, IEEE T. Contr. Syst. T., 24, 1480–1487, 2016.

Yu, R. F., Kang, S. F., Liaw, S. L., and Chen, M. C.: Application of artificial neural network to control the coagulant dosing in water treatment plant, Water Sci. Technol., 42, 403–408, 2000.

Zhang, Q. and Stanley, S. J.: Real-time water treatment process control with artificial neural networks, J. Environ. Eng., 125, 153–160, 1999.


text:
### Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

### Messages - Nan Choi

Pages: [1]
1
##### MAT244--Lectures & Home Assignments / Describing Direction Fields
« on: September 20, 2018, 04:26:38 AM »
To what extent are we expected to describe direction fields?

For example, consider the differential equation from the homework question:

Section 1.1 #28

$y' = e^{-t} + y$

I used the online ODE plotter to determine its behaviour.

As $t → ∞$, $y$ diverges from $0$.
$y$ approaches $∞$ when $t > 0$, and $y$ approaches $-∞$ when $t < 0$.
At $t = 0$ and $y = -1$, the value for $y'$ is zero and the slope line is horizontal.

The question asks us to determine the behaviour of $y$ as $t → ∞$, and whether this behaviour depends on the initial value of $y$ at $t = 0$. Is the explanation above enough?

2
##### MAT244--Lectures & Home Assignments / Re: Section 2.2 Q32
« on: September 20, 2018, 03:48:04 AM »
$e^c$ is just some arbitrary constant which the textbook has replaced with just $c$. Your answer and the textbook's answer are the same.

3
##### MAT244--Lectures & Home Assignments / Re: substitution in integral
« on: September 10, 2018, 10:53:34 PM »
This is a really basic way to look at it:

Since $w'(t)$ means the derivative of $w$ with respect to $t$, we can write:

$w'(t) = \frac{dw}{dt}$

So, multiplying both sides by $dt$ gives:

$w'(t)dt = dw$

Pages: [1]


text:
# Use vscode snippets to work with project members to improve development efficiency

zxg_ God said to have light 2021-09-15 09:04:19

code snippets Code snippet means , yes vscode It provides the function of quickly completing a piece of code according to a string , Can improve the efficiency of writing code .

vscode Of snippets Can be shared with the project , When multiple people develop a project , You can maintain project level snippets And through git share , To improve project development efficiency .

Let's take a closer look at snippets.

## snippets The function of

snippets The configuration of is in the following format ：

{
"For Loop": {
"prefix": ["for", "for-const"],
"body": ["for (const ${2:element} of${1:array}) {", "\t$0", "}"], "description": "A for loop." } } Copy code  • prefix Is triggered snippets The prefix of , Multiple... Can be specified through an array • body Is the content filled into the editor • description yes snippets Description of among body Some can be passed through${} To specify the cursor position 、 The order 、 Placeholder string 、 Available values, etc , Yes 5 Grammar , Let's take a look at ：

### The cursor jumps ： $0$1 $2 Can pass $0、$1、 $2 Specify cursor position , When filled snippets After the content of , The cursor will be set to$0 To edit , After editing , Can pass tab Come and jump to$1.

And when there are multiple $0、$1 etc. , Editing one place and other contents will also be modified synchronously , That is to say vscode Multi cursor editing .

With this function, you can quickly edit snippets Editable content in .

Placeholders are like input The label added placeholder attribute , Or do you want to enter it manually , When the editable area has several optional values , You have to change to the drop-down selection , stay snippets Inside is through ${1:|text1,text2,text3|} The way to support , stay | and | Fill in between , Multiple options for segmentation . ### Variable ：$ Variable name

When filling in the editable position of the template , Sometimes you need to use the selected value 、 Clipboard value 、 file name 、 Date, etc. , This information passes through snippets Supported variables in .

such as ：

• TM_FILENAME: file name
• TM_CURRENT_LINE: Contents of current line
• CLIPBOARD: The contents of the clipboard
• WORKSPACE_NAME： workspace Name
• WORKSPACE_PATH： workspace The path of
• CURRENT_YEAR： The current year
• CURRENT_MONTH： The current month
• CURRENT_DATE： Current day
• RANDOM: random number
• RANDOM_HEX: 6 Bit random 16 Hexadecimal number
• UUID:  only id

You can take the values of these variables to fill in the cursor position , The way is to use $TM_FILENAME、$CURRENT_YEAR The way .

### Variable conversion ： ${ Variable name / Matching regularities / Replace with a new string / Matching mode } It supports variable filling, but not yet , Because the contents of some variables are inappropriate , You need to do some string replacement , therefore snippets Just eat transform The function of . such as abc-123.js The file of , We go through$TM_FILENAME Get the file name , Then remove the suffix and put it in uppercase

${TM_FILENAME/(.*)\\.[a-z]+/${1:/upcase}/i}
Copy code

For file names TM_FILENAME Do regular matching (.*).[a-z]+, Turn group one into uppercase and return , The matching pattern is ignore case （ignore）.

{
" Fill in the file name ": {
"scope": "javascript,typescript",
"prefix": "filename",
"body": [
"${TM_FILENAME/(.*)\\.[a-z]+/${1:/upcase}/i}"
],
"description": " file name "
}
}
Copy code

Let's experiment with the effect ：

You can see , Get the file name correctly , And remove the suffix and put it in uppercase .

got it snippets The function of , So how to set up snippets Well ？snippets To what extent does it take effect ？

## snippets The scope of the

command + shift + p Open the command panel , Input snippet, choice configure user snippets：

You can choose to create a global 、 Project wide 、 Language range snippets：

Files in different locations will be opened to add snippets.

At the language level snippets It only works for a specific language , This can also be encapsulated as a plug-in . In plugins package.json It can be configured in ：

{
"contributes": {
"snippets": [
{
"language": "javascript",
"path": "./snippets.json"
}
]
}
}
Copy code

Project wide snippets It is in the root directory of the project .vscode/xxx.code-snippets Added below ,vscode These files will be read at startup , Then make it effective within the scope of the project .

When there are some project level pieces of code that can be shared , Submit this file completely to the remote git Warehouse , Then project members can share these snippets Set up . For some projects with more template code , Still more meaningful .

## summary

snippets yes vscode Provides some quick input code snippets for improving development efficiency , Support the jump of cursor position 、 Multi cursor simultaneous editing 、 Place holder 、 Optional value 、 Variable 、 Variable conversion and other functions , Use these functions flexibly , Can make easy-to-use to improve development efficiency snippets.

snippets Yes global、language、project 3 Scope of effectiveness ：global Is the global setting ;language Is the language level setting , It can be further encapsulated into plug-in sharing ;project Is within the scope of the project , stay .vscode Under the xx.code-snippets in , It can be submitted to git Warehouse , Share with other members .

Flexible use of snippets function , It can improve development efficiency , And this can also be shared at the project level . I hope this article can help you understand snippets.


text:
# Differences

This shows you the differences between two versions of the page.

 — events:individual_past_events:2013-09-17_latex_lecture [2013/09/16 14:08] (current) Line 1: Line 1: + /* If you have not already done so, please begin the heading below + with a date in YYYY-MM-DD format followed by a short description. + This is what will appear in the event listing and the event list + itself is sorted by this.   */ + + + ====== An introduction to LATEX ====== + + ==== Vassar College Department of Mathematics Announces a lecture/demonstration ==== + + ==== John McCleary ==== + + Vassar College \\ + Tuesday, September 17, 2013 \\ + Rockefeller Hall 300 \\ + 7-8:30pm \\ + + + One of the most useful programming languages is TEX, constructed by + Donald Knuth, and developed by Leslie Lamport into LATEX, the standard + for mathematical typesetting over the last 25 years. If you want to + be able to typeset your mathematical work, or even to typeset any of + your assignments, LATEX can be very valuable.  + + I will give a lecture/demonstration of elementary aspects of + the LATEX programming language. If you have installed a version of TEX on your laptop, bring it + along. On the MAC, install TEXshop. On machines running Microsoft operating systems, install MikTEX. Both of these versions + of TEX are freely available on the Web.