Is it gonna rain today? Should I take my umbrella to the office or not? To know the answer to such questions we will just take out our phone and check the weather forecast. How is this done? There are computer models which use statistics to compare weather conditions from the past with the current conditions to predict future weather conditions. From studying the amount of fluoride that is safe in our toothpaste to predicting the future stock rates, everything requires statistics. Data is everything in statistics. Calculating the range, median, and mode of the data set is all a part of descriptive statistics.
Data representation, manipulation, and visualization are key components in statistics. You can read about it here.
The next important step is analyzing the data, which can be done using both descriptive and inferential statistics. Both descriptive and inferential statistics are used to analyze results and draw conclusions in most of the research studies conducted on groups of people.
Through this article, we will learn descriptive statistics using python.
Descriptive statistics describe the basic and important features of data. Descriptive statistics help simplify and summarize large amounts of data in a sensible manner. For instance, consider the Cumulative Grade Point Index (CGPI), which is used to describe the general performance of a student across a wide range of course experiences.
Descriptive statistics involve evaluating measures of center(centrality measures) and measures of dispersion(spread).
Centrality measures give us an estimate of the center of a distribution. It gives us a sense of a typical value we would expect to see. The three major measures of center include the mean, median, and mode.
…the average of the given values. To compute mean, sum all the values and divide the sum by the number of values.
\(\displaystyle\overline{x}=\frac{1}{n}\sum_{i=1}^n x_i\)
Consider a class of 7 students. Suppose they appear for midterm exams and score the following marks out of 20:
| Name | Subject 1 | Subject 2 | Subject 3 | Subject 4 | Total Marks |
|---|---|---|---|---|---|
| Student 1 | 10 | 8 | 15 | 11.5 | 44.5 |
| Student 2 | 14 | 9 | 7.5 | 11 | 41.5 |
| Student 3 | 11 | 17 | 11.5 | 9 | 48.5 |
| Student 4 | 7 | 14.5 | 10 | 12 | 43.5 |
| Student 5 | 9.5 | 12 | 10.5 | 14 | 46 |
| Student 6 | 15 | 18 | 7 | 12 | 52 |
| Student 7 | 19 | 15.5 | 11 | 7.5 | 53 |
Mean with python
There are various libraries in python such as pandas, numpy, statistics (Python version 3.4) that support mean calculation.
numpy.mean(a, axis=None, dtype=None)
a: array containing numbers whose mean is required
axis: axis or axes along which the means are computed, default is to compute the mean of the flattened array
dtype: type of data to be used in calculations
>>> import numpy as np
>>> A = np.array([[10,14,11,7,9.5,15,19],[8,9,17,14.5,12,18,15.5],
[15,7.5,11.5,10,10.5,7,11],[11.5,11,9,12,14,12,7.5]])
>>> B = A.T
>>> print B
[[ 10. 8. 15. 11.5]
[ 14. 9. 7.5 11. ]
[ 11. 17. 11.5 9. ]
[ 7. 14.5 10. 12. ]
[ 9.5 12. 10.5 14. ]
[ 15. 18. 7. 12. ]
[ 19. 15.5 11. 7.5]]
>>> print(np.mean(B))
11.75
>>> print(np.mean(B,axis=0))
[ 12.21428571 13.42857143 10.35714286 11. ]
>>> print(np.mean(A,axis=1))
[ 11.125 10.375 12.125 10.875 11.5 13. 13.25 ]
In the above code, axis=0 calculates the mean along every column and axis=1 calculates it along every row of the array.
Why mean?
Now that we have learned how to calculate mean manually as well as by using python, let’s understand its importance. Mean represents the typical value that acts as a yardstick for all observations. For instance, in our example above, average marks of the class will help the teacher to judge the performance of an individual relative to others.
…the value where the upper half of the data lies above it and lower half lies below it. In other words, it is the middle value of a data set.
To calculate the median, arrange the data points in the increasing order and the middle value is the median. It is easy to find out the middle value if there is an odd number of data points, say, we want to find the median for marks of all students for Subject 1. When marks are arranged in the increasing order, we get {7,9.5,10,11,14,15,19}. Clearly, the middle value is 11; therefore, the median is 11.
If Student 7 did not write the exam, we will have marks as {7,9.5,10,11,14,15}. This time there is no clear middle value. Then, take the mean of the third and fourth values, which is (10+11)/2=10.5, so the median in this case is 10.5.
Median with python
Median can be calculated using numpy, pandas and statistics(version 3.4) libraries in python.
numpy.median(a, axis=None, out=None)
a: array containing numbers whose median is required
axis: axis or axes along which the median is computed, default is to compute the median of the flattened array
out: alternative output array to place the result, must have the same shape and buffer length as the expected output.
>>> import numpy as np >>> A=np.array([[10,14,11,7,9.5,15,19],[8,9,17,14.5,12,18,15.5], [15,7.5,11.5,10,10.5,7,11],[11.5,11,9,12,14,12,7.5]]) >>> B=A.T >>> a=np.median(B, axis=0) >>> b=np.median(B, axis=1) >>> print(a,b) [ 11. 14.5 10.5 11.5] [ 10.75 10. 11.25 11. 11.25 13.5 13.25]
Median wins over mean when…
The median is a better choice when the indicator can be affected by some outliers. Looking at the picture below, we can see that Student 9 and Student 10 scored much more than the rest and their scores were included in the calculation of mean, making it less representative of the typical observation. On the other hand, median is the middle value which is not affected by these outliers. It gives us a better estimate of the typical score. However, this means that presence of Student 9’s and Student 10’s relatively higher scores are hidden.
Mean wins over median when…
The mean is a better choice when there are no extreme values that can affect it. It is a better summary because the information from every observation is included rather than median, which is just the middle value.
We can also derive the sum of all the observations, for example, the total marks scored by all students, when the number of observations is provided.
…the value that occurs the most number of times in our data set. Suppose there are 15 students appearing for an exam and following is the result:
When the mode is not unique, we say that the data set is bimodal, while a data set with more than two modes is multimodal.
Mode with python
Similar to the mean and the median, we can calculate mode using numpy(scipy), pandas, and statistics.
scipy.stats.mstats.mode(a, axis=0)
a: array containing numbers whose mode is required
axis: axis or axes along which the mode is computed, default is to compute the mode of the flattened array
It returns (mode: array of modal values, count: array of counts for each mode).
>>> import numpy as np >>> from scipy import stats >>> A=np.array([[10,14,11,7,9.5,15,19],[8,9,17,14.5,12,18,15.5], [15,7.5,11.5,10,10.5,7,11],[11.5,11,9,12,14,12,7.5]]) >>> B=A.T >>> a=stats.mode(B,axis=0) >>> b=stats.mode(B,axis=1) >>> print(a) ModeResult(mode=array([[ 7., 8., 7., 12.]]), count=array([[1, 1, 1, 2]])) >>> print(b) ModeResult(mode=array([[ 8. ], [ 7.5], [ 9. ], [ 7. ], [ 9.5], [ 7. ], [ 7.5]]), count=array([[1], [1], [1], [1], [1], [1], [1]]))
Why mode?
Mode also makes sense when we do not have a numeric-valued data set which is required in case of the mean and the median. For instance, if a company wants to figure out the most common place their employees belong to, then the mode will be a city (which can’t be done in the case of mean and median).
Why not mode?
Mode is not useful when our distribution is flat; i.e., the frequencies of all groups are similar, for example, in midterm exam for Subject 1 case, the distribution is flat as there is no particular number which is appearing more than once. In such cases, the mode does not provide any useful information. Also, at times mode can appear at the tail of the distribution which is not necessarily at or near the center of a distribution.
Measures of dispersion are values that describe how the data varies. It gives us a sense of how much the data tends to diverge from the typical value, while central measures give us an idea about the typical value of the distribution.
…one of the simplest dispersion measures. It is the difference between the maximum and minimum values in the distribution. For instance:
Range with python
We use numpy.ptp() function to calculate range in python. There are other functions to calculate minimum and maximum such as numpy.amin() and numpy.amax(). ‘ptp’ stands for ‘peak to peak’.
numpy.ptp(a, axis=None, out=None)
a: array containing numbers whose range is required
axis: axis or axes along which the range is computed, default is to compute the range of the flattened array. It returns a new array with the result.
>>> import numpy as np
>>> A=np.array([[10,14,11,7,9.5,15,19],[8,9,17,14.5,12,18,15.5],
[15,7.5,11.5,10,10.5,7,11],[11.5,11,9,12,14,12,7.5]])
>>> B=A.T
>>> a=np.ptp(B, axis=0)
>>> b=np.ptp(B,axis=1)
>>> print(a)
[ 12. 10. 8. 6.5]
>>> print(b)
[ 7. 6.5 8. 7.5 4.5 11. 11.5] Why range?
The range gives a quick sense of the spread of the distribution to those who require only a rough indication of the data.
Why not range?
There are some disadvantages of using the range as a measure of spread. One being it does not give any information of the data in between maximum and minimum. There can be two data sets with the same range, but the values(hence the distribution) may differ significantly as shown below.
Also, the range is very sensitive to extreme values as it is the difference between the maximum and minimum values.
The distribution above has range 18, but it can be clearly seen that the data is clustered around 8 to 12, suggesting a range of 4. This does not give a good indication of the observations in the data set.
…a measure which indicates the value below which a given percentage of points in a dataset fall. For instance, the 35th percentile(\(P_{35}\)) is the score below which 35% of the data points may be found. We can observe that median represents the 50th percentile. Similarly, we can have 0th percentile representing the minimum and 100th percentile representing the maximum of all data points.
There are various methods of calculation of quartiles and percentiles, but we will stick to the one below. To calculate \(k^{th}\) percentile(\(P_{k}\)) for a data set of \(N\) observations which is arranged in increasing order, go through the following steps:
Suppose we want to calculate \(P_{27}\) for the marks of students in Subject 2. Let us first arrange the data in increasing order which results in the following dataset {8,9,12,14.5,15.5,17,18}.
Following the steps above,
Step 1: \(\displaystyle i=\frac{27}{100}\times 7 = 1.89\)
Step 2: Not applicable here as 1.89 is not a whole number, so let us move to step 3
Step 3: Rounding up 1.89 gives 2, hence 27th percentile is the value of second observation, i.e., 9
Therefore, 9 is \(27^{th}\) percentile which means that 27% of the students have scored below 9.
Quartiles are…
…the three points that split the data set into four equal parts such that each group consists of one-fourth of the data. We also call 25th percentile the first quartile(\(Q_1\)), 50th percentile the second quartile(\(Q_2\)), and 75th percentile the third quartile(\(Q_3\)).
Percentiles and quartiles with python
numpy.percentile(a, q, axis=None,iterpolation=’linear’)
a: array containing numbers whose range is required
q: percentile to compute(must be between 0 and 100)
axis: axis or axes along which the range is computed, default is to compute the range of the flattened array
interpolation: it can take the values as ‘linear’, ‘lower’, ‘higher’, ‘midpoint’or ‘nearest’. This parameter specifies the method which is to be used when the desired quartile lies between two data points, say i and j.
numpy.percentile() agrees with the manual calculation of percentiles (as shown above) only when interpolation is set as ‘lower’.
>>> import numpy as np >>> A=np.array([[10,14,11,7,9.5,15,19],[8,9,17,14.5,12,18,15.5], [15,7.5,11.5,10,10.5,7,11],[11.5,11,9,12,14,12,7.5]]) >>> B=A.T >>> a=np.percentile(B,27,axis=0, interpolation='lower') >>> b=np.percentile(B,25,axis=1, interpolation='lower') >>> c=np.percentile(B,75,axis=0, interpolation='lower') >>> d=np.percentile(B,50,axis=0, interpolation='lower') >>> print(a) [ 9.5 9. 7.5 9. ] >>> print(b) [ 8. 7.5 9. 7. 9.5 7. 7.5] >>> print(c) [ 14. 15.5 11. 12. ] >>> print(d) [ 11. 14.5 10.5 11.5]
Why percentiles?
Percentile gives the relative position of a particular value within the dataset. If we are interested in relative positions, then mean and standard deviations does not make sense. In the case of exam scores, we do not know if it might have been a difficult exam and 7 points out of 20 was an amazing score. In this case, personal scores in itself are meaningless, but the percentile would reflect everything. For example, GRE and GMAT scores are all in terms of percentiles.
Another good property of percentiles is that it has a universal interpretation; i.e., it does not depend on whether we are looking at exam scores or the height of the players across a few basketball teams. 55th percentile would always mean that 55 % would always be found below the value and other 45% would be above it. It helps in comparing the data sets which have different means and deviations.
…the difference between the third quartile and the first quartile.
\(IQR=Q_3-Q_1\)
Interquartile range with python
scipy.stats.iqr(a, axis=0, interpolation=’linear’)
>>> import numpy as np >>> from scipy.stats import iqr >>> A=np.array([[10,14,11,7,9.5,15,19],[8,9,17,14.5,12,18,15.5] [15,7.5,11.5,10,10.5,7,11],[11.5,11,9,12,14,12,7.5]]) >>> B=A.T >>> a=iqr(B, axis=0 , rng=(25, 75), interpolation='lower') >>> b=iqr(B, axis=1 , rng=(25, 75), interpolation='lower') >>> print(a,b) [ 4.5 6.5 3.5 3. ] [ 3.5 3.5 2.5 5. 2.5 8. 8. ]
Why IQR?
The interquartile range is a better option than range because it is not affected by outliers. It removes the outliers by just focusing on the distance within the middle 50% of the data.
…the average of the squared differences from the mean. For a dataset, \(X=\{a_1,a_2,\ldots,a_n\) with the mean as \(\overline{x}\), variance is
\(\displaystyle Var(X)=\frac{1}{n}\sum_{i=1}^n(a_i-\overline{x})^2\)
Variance with Python
Variance can be calculated in python using different libraries like numpy, pandas, and statistics.
numpy.var(a, axis=None, dtype=None, ddof=0)
Parameters are the same as numpy.mean except
ddof : int, optional(ddof stands for delta degrees of freedom. It is the divisor used in the calculation, which is N – ddof, where N is the number of elements. The default value of ddof is 0)
>>> import numpy as np >>> A=np.array([[10,14,11,7,9.5,15,19],[8,9,17,14.5,12,18,15.5], [15,7.5,11.5,10,10.5,7,11],[11.5,11,9,12,14,12,7.5]]) >>> B=A.T >>> a = np.var(B,axis=0) >>> b = np.var(B,axis=1) >>> print(a) [ 13.98979592 12.8877551 6.12244898 3.92857143] >>> print(b) [ 6.546875 5.921875 8.796875 7.546875 2.875 16.5 19.0625 ]
Why variance?
It is an important measure in descriptive statistics because it allows us to measure the spread of a data set around its mean. The observations may or may not be meaningful if observations in data sets are highly spread.
…the square root of variance.
Standard deviation with Python
It can be calculated in python using different libraries like numpy, pandas, and statistics.
numpy.std(a, axis=None, dtype=None, ddof=0)
Parameters are the same as numpy.var().
>>> import numpy as np >>> A=np.array([[10,14,11,7,9.5,15,19],[8,9,17,14.5,12,18,15.5], [15,7.5,11.5,10,10.5,7,11],[11.5,11,9,12,14,12,7.5]]) >>> B=A.T >>> a = np.std(B,axis=0) >>> b = np.std(B,axis=1) >>> print(a) [ 3.74029356 3.58995196 2.4743583 1.98206242] >>> print(b) [ 2.55868619 2.43349029 2.96595263 2.74715762 1.6955825 4.0620192 4.3660623 ]
Variance versus standard deviation
The only similarity between variance and standard deviation is that they are both non-negative. The most important difference is that standard deviation is on the same scale as the values in the data set. Therefore, it is expressed in the same units, whereas variance is scaled larger. So it is not expressed in the same units as the values.
… whether the distribution has a longer tail on one side or the other or has left-right symmetry. There have been different skewness coefficients proposed over the years. The most common way to calculate is by taking the mean of the cubes of differences of each point from the mean and then dividing it by the cube of the standard deviation. This gives a coefficient that is independent of the units of the observations.
It can be positive(representing right skewed distribution), negative(representing left skewed distribution), or zero(representing unskewed distribution).
Skewness with python
scipy.stats.skew(a, axis=0)
>>> import numpy as np
>>> import scipy
>>> A=np.array([[10,14,11,7,9.5,15,19],[8,9,17,14.5,12,18,15.5],
[15,7.5,11.5,10,10.5,7,11],[11.5,11,9,12,14,12,7.5]])
>>> B=A.T
>>> a=scipy.stats.skew(B,axis=0)
>>> print(a)
[ 0.45143419 -0.30426514 0.38321624 -0.39903339]
…whether the shape of a distribution mat
Kurtosis with python
scipy.stats.kurtosis(a, axis=0, fisher=True)
The parameters remain the same except fisher
fisher: if True then Fisher’s definition is used and if False, Pearson’s definition is used. Default is True
>>> import numpy as np
>>> import scipy
>>> A=np.array([[10,14,11,7,9.5,15,19],[8,9,17,14.5,12,18,15.5],[15,7.5,11.5,10,10.5,7,11],
[11.5,11,9,12,14,12,7.5]])
>>> B=A.T
>>> a=scipy.stats.kurtosis(B,axis=0,fisher=False) #Pearson Kurtosis
>>> b=scipy.stats.kurtosis(B,axis=1) #Fisher's Kurtosis
>>> print(a,b)
[ 2.19732518 1.6138826 2.516175 2.30595041] [-1.11918934 -1.25539366
-0.86157952 -1.24277613 -1.30245747 -1.22038567 -1.46061811] Why skewness and kurtosis?
Skewness and kurtosis are used to describe some aspects of the symmetry and shape of the distribution of the observations in a data set.
Descriptive statistics measures are limited in the way that we can only make the summary about the people or objects that are actually measured. The data cannot be used to generalize to other people or objects. For example, if we have recorded the marks of the students for the past few years and would want to predict the marks for next exam, we cannot do that only relying on descriptive statistics; inferential statistics would help. Descriptive statistics can often be difficult when we are dealing with a large dataset.
Next, we will learn about inferential statistics that can help gain insights into the population from which the data was collected.
Organizations of all industries struggle with employee turnover. The high turnover rates cause increased hiring…
Virtual hiring events are becoming vital for modern recruitment, and the hiring world is changing…
The competition for talent today is intense, and this makes it very important for organizations…
Hiring trends are continuously evolving over the ages to keep pace with the latest technological…
Hiring practices have changed significantly over the past 30 years. Technological advancements and changing workforce…
In the current world, where the hiring process is ever-evolving, it has become crucial to…