- whats is data science
- why learn vba
- importance of data visualization
- excel tanh function
- excel lognorm dist function
- excel logest function
- excel linest function
- excel large function
- excel kurt function
- excel intercept function
- excel hypgeom dist function
- excel harmean function
- excel growth function
- excel gauss function
- excel gammaln precise function
- excel gammaln function
- excel gamma inv function
- excel gamma dist function
- excel gamma function
- excel forecast linear function
- excel forecast ets stat function
- excel forecast ets seasonality function
- excel forecast ets confint function
- excel forecast ets function
- excel forecast function
- excel fisherinv function
- excel fisher function
- excel finv function
- excel f test function
- excel f inv rt function
- excel f inv function
- excel f dist rt function
- excel f dist function
- excel expon dist function
- excel devsq function
- excel covariance s function
- excel covariance p function
- excel countifs function
- excel countif function
- excel countblank function
- excel counta function
- excel count function
- excel correl function
- excel confidence t function
- excel confidence norm function
- excel chisq test function
- excel chisq inv rt function
- excel chisq inv function
- excel chisq dist rt function
- excel chisq dist function
- excel binom inv function
- excel binom dist range function
- excel binom dist function
- excel beta inv function
- excel beta dist function
- excel averageifs function
- excel averageif function
- excel averagea function
- excel average function
- excel avedev function
- excel yearfrac function
- excel year function
- excel workday intl function
- excel workday function
- excel weeknum function
- excel weekday function
- excel today function
- excel timevalue function
- excel time function
- excel second function
- excel now function
- excel networkdays intl function
- excel networkdays function
- excel month function
- excel minute function
- excel isoweeknum function
- excel hour function
- excel eomonth function
- excel edate function
- excel days360 function
- excel days function
- excel day function
- excel datevalue function
- excel datedif function
- excel date function
- excel webservice function
- excel filterxml function
- excel encodeurl function
- excel value function
- excel upper function
- excel unicode function
- excel unichar function
- excel trim function
- excel textjoin function
- excel text function
- excel substitute function
- excel search function
- excel right function
- excel rept function
- excel replace function
- excel proper function
- excel phonetic function
- excel numbervalue function
- excel mid function
- excel lower function
- excel len function
- excel left function
- excel jis function
- excel fixed function
- excel find function
- excel exact function
- excel dollar function
- excel dbcs function
- excel concatenate function
- excel concat function
- excel code function
- excel clean function
- excel char function
- excel bahttext function
- excel asc function
- excel vlookup function
- excel unique function
- excel transpose function
- excel sortby function
- excel sort function
- excel single function
- excel rtd function
- excel rows function
- excel row function
- excel offset function
- excel match function
- excel lookup function
- excel indirect function
- excel index function
- excel hyperlink function
- excel hlookup function
- excel getpivotdata function
- excel formulatext function
- excel filter function
- excel columns function
- excel column function
- excel choose function
- excel areas function
- excel address function
- excel xor function
- excel true function
- excel switch function
- excel or function
- excel not function
- excel ifs function
- excel ifna function
- excel iferror function
- excel if function
- excel false function
- excel and function
- excel sheets function
- excel sheet function
- excel na function
- excel istext function
- excel isref function
- excel isodd function
- microsoft excel pivot table
- vba array
- vba operators
- create vba function
- automate excel vba
- mongodb gui access
- ranges in excel vba
- regex code syntax guide
- probability data science step by step week2 3
- descriptive statistics week1
- data science learning path
- human being a machine learning experience
- data preparation dbms
- vba codes practise sub commandnametoday
- business analytics
- challenges in data analytics
- probability short course data analyst
- become data driven organization
- category of analytics
- become data scientist
- why monkidea blog
- free books data analytics
- 10 fun facts about analytics
- summary of monkidea com till this post
- data visualization summary table mosaic chart
- observational and second experimental studies
- relative standard deviation coefficient of variation
- sampling types statistics
- population and sample statistics
- data transformation statistics
- variability vs diversity statistical spread
- data visualization box plot
- data visualization histogram
- data visualization bar pie chart
- data visualization scatter plot
- data exploration introduction bias types
- sql queries for practice oracle 11g
- creating your own schema oracle 11g xe
- dml insert update delete in sql
- creating the other schema objects oracle 11g sql
- learning constraints sql
- ddl data defination language a note
- sql as a set oriented language union union all minus intersect
- subqueries sql
- plsql basics an introduction
- an introduction to sql functions with examples
- sql select statement an introduction
- sql operators
- schema datatypes constraints
- first step toward oracle database xe
- sql introduction dbms interfaces
- 1st post on oracle 11g sql monkidea
- rdbms components
- indexing yet to be updated
- naming conventions data integrity rdbms
- normalization rdbms
- data model design rdmbs
- removing inconsistencies in designing rdbms
- ddlc database development life cycle
- rdbms an introduction
- data in a dataset set theory
- data types
- origin or sources or top generators of data for analytics
- data definition label dbms
- big data analytics an introduction
- statistics tests a summary
- why every business analyst needs to learn r
- tools for analytics
- use of analytics w r t industry domains
- analytics as a process
- top view of analytics big picture
- emergence evolution of analytics
- terms and definition used in analytics
- why do we need analytics
- analytics overview
TOOLS FOR ANALYTICS: A perspective on analytics tools and comparison of analytical tool software. An idea on what clicks to whom, not everyone is a programmer. Sometime being analytical savvy is good to go..
I have used R programming and VBA programming in my other pet projects at home/office as they provide more of learning and exposure to working on the algorithms.
Weka only to explore how this works ( But not used any for my projects).
For a budding data scientist with day to day prospective three machine learning algorithms are more important and must if your work requires little bit category creation for segmenting the data into meaningful chunks of data (we could use them in orange and weka to start with due to GUI interface) :
1. Logistic Regression : best for small variable or better 2 ... does not handle larger variation
2. Decision trees -- but as we have more of variations in our categorical variables hence better - random forest(best for variation automation) : filling the missing and unknown brands , models etc...
1. SVM- for small data sets .. large data set it's computationally more intensive
Open Refine The second level of Tool for Cleaning data : easy to use and could be customized to perform automation also.
OpenRefine: Formerly GoogleRefine, OpenRefine is a data cleaning software that allows you to get everything ready for analysis. Example. Recently, I was cleaning up a database that included a lot of variation in company names and noticed that rows had different spellings, capitalization, spaces, etc that made it very difficult for creating report and present to the client. Open refine come very handy that time and become a part of regular analytic work.Its also have lot of features like:
- Import data in various formats xlsx,csv, json etc...
- Explore datasets in a matter of seconds.. Categories ,numbers,
Apply basic and advanced cell transformations: distance matching
- Deal with cells that contain multiple values
- Create instantaneous links between datasets
- Filter and partition your data easily with regular expressions
- Use named-entity extraction on full-text fields to automatically identify topics
- Perform advanced data operations with the General Refine Expression Language
The analytic software ranges from simple statistical tools like spreadsheets (Excel) to statistical software packages like (SPSS) to business intelligence suites like (SAS, JMP, Oracle, IBM among the big players).
Open source tools like R which are free and cost-effective to learn and could be implemented on large scale too. Companies are also developing in-house tools designed for specific purposes like financial Account analysis, AOP planning, CDR analysis for billing audit etc...
MS Excel: This is the most common and widely used application in business know as MS Office suite and Excel. Excel is an excellent reporting and dashboarding tool.
Almost all analyst agree that it’s bread and butter of their life at one moment in their career. e.g. I start my work with the Oracle SQL to extract required data and done analysis using the other software but after all, is done we generally use excel application to finish up the reporting and presentations as graphs easy to make and automate to an extent. A small amount of analysis could be done on the summary tables also. Excel 2007 onward can handle tables with up to 10 Lakh rows making it a powerful yet versatile tool.
Excel + VBA, SAS, R tools are sufficient to work on any analysis problem well at least 80% of the time. Remaining 20% is actually for actuarial science & theory.
SPSS is now used very less and lots of other tools are the same… as people have learned programming and what I believe one-day programming will become necessity…
Programming has given a better view to automate and formulate the logic… the day is not far when programming language will take the place for analytics… but only reason difficult to fit into the equation are the people who have the combination of the skills set required mainly constituting these three “Programming + Statistics Basic & Advance+ Business Process “.
Note: Excel, SPSS & SAS are paid and off-limits to some people who love open source.
Analytical Open Source comparison
I used the table below as a resource to showcase analytical tools which I feel are better and could be used by anyone or with just little training and examples. Help, books, and Documentation; user interface and graphics; how stable the package is; ease of learning; programming and; how many machines learning algorithms that are available:
Help & Doc.’s
The most popular data mining packages in the industry are SAS and SPSS, but But they come at a price hands only the large corporates can afford the Scalable solutions like SAS.
In reality, only 20% off data mining business is actually been implemented in SAS just because they have a good amount of services and support. hence many of the corporates are inclined towards SAS. In reality, the actual 70% of the work is being performed in Excel and eventually, even at server level lot of data mining and data manipulation are been performed.
If you want to learn and practice the data science there are many free options available which could be used to produce the same results locally on your system.
And, especially in a developing country like India, there is a huge segment of industry which keeps remaining untouched and that is the small medium enterprises where these corporates want to leverage the benefit of Data Analytics but can't implement highly priced statistical software.
Open source analytical software perform a wonderful job and this is where one should focus upon. Orange, R, RapidMiner, Statistica, and WEKA all can be used for doing real data mining work. While some of them are unpolished.
Generally, it took around 1 week to 2 weeks to understand the software interfaces and how to use them to get desired results. The only thing one has to remember while performing or using this software is to make sure you know what exactly you want to perform.
My major focus in terms of analytics has always into text mining and natural language processing so most of my work actually goes in categorizing the text content into different segments. Hence the below descriptions of open Analytics software you will find are more inclined towards the text mining domain instead of covering entire machine learning.
Let me summarize what I learned so far:
I find orange, python libraries are good for working on my problems. R was a little hard to get started and WEKA was less polished yet I finished the experimentation on using logistic regression on my dataset.
R has lot of support and examples to begin with which really helped me kept going.
Statistica and RapidMiner had several function and features and were well groomed. But spent very less time on these.
Let's begin with Orange:
Orange is an open source data mining package build on Python, NumPy, wrapped C, C++ and Qt. Works both as a script and with an ETL work flow GUI. Shortest script for doing training, cross-validation, algorithms comparison, and prediction.
I found Orange the easiest tool to learn.
I found orange very easy to use and well placed flowcharts for completing the analysis. If you want quick understanding on the data science and want to observe the input and output of a data model. Orange is the software for you.
A few Python libs deserve to be mentioned here: scikit, NumPy, SciPy ,Pandas, mlpy, NLTK,Matplotlib & seaborn.
- Python is best programming language one should learn if you are into the development of the applications and data models.
- New release of python has become easy to use because of easy syntax and evolution of community .
- The libraries are huge in numbers for ML and are self contained.
- Just select the library and start customization as per requirement.
- The machine learning is NLTK is very elegant if you have a text mining or NLP problem.
R is an open source statistical and data mining package and programming language in others words it’s an integrated suite of application for data manipulation, statistical calculation and graphics designs.
- Very extensive statistical library.
- Best part it handles arrays as well as matrices a big plus for data science enthusiast
- Logistic regression code was made with few lines .
- Huge numbers of packages to select from.
- Statistical graphs are easy yet to actually make them good looking needs a lot of work.
R vs. Orange/Python
Python and R have a lot in common: they are both elegant, minimal, interpreted languages with good numeric libraries.
As my problem was to identify the brands or diving the rows into different categories orange was very handy and easy to operate with. But a simple, csv file import it took lot of time to identify the problem of control characters.
Whereas in Import and export of data from spreadsheet is easier in R, read.csv or read.table a simple syntax helps have the spreadsheet stored in a data frames that the different machine learning algorithms are operating on. Programming in R really is very different, you are working on a higher abstraction level, but you do lose control over the details.
Rapidminer seems to be commercial offering an end to end solution which have example of python and R scripts.
WEKA is an open source statistical and data mining library written in Java.
- A lot of machine learning algorithms.
- Easy to learn and use.
- Good GUI.
- Platform independent.
- Worse connectivity to Excel spreadsheet and non Java based databases. It takes a lot of correction and back-and-forth to import an excel file into the weka.
- CSV reader not as robust as in RapidMiner.
- Not as polished
Selecting of tools is just a way to express your emotion about data science. More the skills easier it becomes to select the correct tools. Higher the understanding, more you will love to be at the core.