Posted by By Dripto Mukhopadhyay
Yesterday I read a nice and enlightening article on “ranking” by my ex-colleague and friend Dr. Sanjib Pohit (Article link – https://www.dailypioneer.com/2019/columnists/it—s-not-just-about-the-ranking.html). Sanjib’s article is very timely as well as relevant from methodology practice point of view, especially in India’s context. In recent years ranking exercise has become extremely common in every domain of activities. Rankings are applied to announce awards for various state governments, departments, different initiatives and what not.
I have worked on more than 60/70 studies that involved certain indexation and ranking based on certain purpose-based parameters. This involved studies for Government departments to top end media. Ranges from ranking banks to cities to states to districts to specific locations or initiatives for awards like “Safaigiri” where the Prime Minister gives award to stakeholders. Also these studies span from IT domain to entrepreneurship.
Why I found Sanjib’s article relevant is because of the fact that in many cases I see that neither the researcher nor other stakeholders are well aware of how to address the problem and implications of the methodology. That is the reason I thought of extending Sanjib’s though a bit further and address a few other issues that he did not touch upon in the mentioned article.
Before going into good bad ugly, lets understand why a ranking exercise is important from academic as well as from action point of view. As we do understand and have experienced in our day to day life directly or indirectly, ranking is done to compare more than one entity from any particular reference purpose. That purpose can vary from comparing students appeared in an examination to understand quality of services hotels of a destination to ranking states on how they are performing in terms of income generation. In fact, the purposes can be numerous since human activities have expanded many folds and ranking helps us to understand relative positions among entities to compare.
Here I introduce the concept of relative position. This is extremely critical to understand rest of the argument I am going to put forward. In view of ranking what to be done and how to be done depends completely on how we want to view this relative position. Keeping in mind that the only constant in the world is “change”, this becomes even more important. Any ranking exercise, from professional perspective, should be actionable. This means unless we are able to use the ranking exercise meaningfully for a specific productive purpose, the entire effort is pointless.
It may not always be true that we require a benchmark while working on any ranking exercise. In fact, a benchmark should be used prior to ranking exercise to filter out irrelevant entities from purpose point of view. But, if the purpose is defined precisely and with complete clarity, we may overcome the need for filtering the irrelevant outlier. However, need for Apple to Apple comparison is no denying.
Now, its time that we think of how to go for a ranking and why indexation is used for this. Indexation helps in bringing information or data in a more understandable platform within a comparable scale. Its easier to comprehend and to interpretable and actionable. Indexation can be done in many ways. It can be with the help of a single variable or including multiple relevant variables. In general, unless in a very unique situation, no unobserved phenomenon can be captured truly and realistically with a single variable. That’s the reason, most of the time ranking through indexation involves multiple variable.
When multiple variables come into picture, to arrive at a composite index, the most important question is to whether to involve any weighting scheme or not. This depends on the problem in hand and its nature. Based on this it is decided whether to go for a statistical model like factor analysis or the similar one or to go for a non-model based indexation. Many a times arbitrary weights are used basis perception of the researcher. The robustness of indexation depends immensely on choice of variables, method used for indexation and also researchers understanding of the model.
In most of the cases if the indexation methodology chose right variables and right method, it automatically takes care of irrelevancy of entities that are not fir into a particular scheme of problem. Here comes the relevance of “relative position” concept. One can use two ways to create distinctions among entities. One is through ranking, which is more of a crude way. The reason being even for an insignificant different between entities, their positions can be significantly different. This does not serve the purpose realistically.
Therefore, more realistic and actionable way is to categorise entities based on certain principles, be it statistical or simple difference in values. with Most of the cases a static scenario is used. More precisely, entities are compared for a specific time point and their relation position through ranking or categorisation. This is to me a half-cooked meal. The entities should be compared to other entities in one time point, but also should be seen how it has changed compared to itself in past years. Very few studies do this, but this can play critical role in actionable decision making.
To sum up, ranking and usage of indexation methodology can and should play important role in decision making at any front. However, the impact expected out of such action completely depends on methodological robustness used for such purpose. Though apparently it looks simple, but the applicability and actionability can die down completely unless some key concerns are taken care of.