Skew in response rates by household incomes repaired

by Mahesh Vyas

It is instructive to know the outcome of a survey to see if the composition of the respondents reflects, broadly, the composition of the sample. In this context, here we study outcomes with respect to household incomes. We have no control on this attribute during survey execution. An additional complication of the distribution of income is that it is not necessarily stable over time. So, changes in the composition of respondents over time could be a reflection of the structural changes in the economy and not necessarily of changes in the CPHS sample or execution.

We create five income groups. These are in terms of nominal annual household income. The five groups are:

  1. Annual household income less than or equal to Rs.100,000
  2. Annual household income greater than Rs.100,000 but not greater than Rs.200,000
  3. Annual household income greater than Rs.200,000 but not greater than Rs.500,000
  4. Annual household income greater than Rs.500,000 but not greater than Rs.1,000,000
  5. Annual household income greater than Rs.1,000,000.

The income group ranges have cut-offs that are convenient rounded figures. These cut-offs and therefore the groups do not reflect the distribution of the households. They are arbitrary cut-offs based on a heuristic understanding of the distribution of household incomes. For example, the average household income is around Rs.200,000 in India. This average anchors the broad range of household incomes to be addressed by the income groups. An important element in deciding these cut-offs is that they are relatable to everyday life.

Another idiosyncrasy in this study is that the income groups are kept constant over time. They are not adjusted for inflation or for the possible changes in the distribution of the income of Indian households. This is because the underlying data is expressed in nominal terms and is not adjusted for inflation and so, the scale also needs to be expressed in nominal terms.

Here are a few highlights from the distribution of households in the sample since the first wave in January-April 2014.

  1. The share of households that earn less than Rs.100,000 declined from 31.5 per cent in Wave 1 to 6.1 in Wave 19 which was conducted during January-April 2020, just as the first lockdown hit India. Then, during the lockdown, their share rose to 8 per cent in Wave 20, then 9.6 in Wave 21 and 10.1 in Wave 22. Apparently, more households in the sample that had moved out of the lower income bracket have returned.
  2. The share of households that earn more than Rs. 1 million rose from 0.6 per cent in Wave 1 to 1.9 per cent in Wave 17 of May-August 2019. Then, it fell marginally to 1.8 and 1.7 per cent, respectively in the following two waves. But, the fall real came in Wave 20 when the share of the rich households declined to 1.5 per cent and then to 0.95 and 0.92 per cent, respectively in Waves 21 and 22. More households in the sample have moved out of the richer income bracket.
  3. There is a similar fall in households in the upper income bracket of Rs.500,000 to Rs.1 million.

The above shift in the composition of the sample is good to know but the pertinent question here is the distribution of the responses and the distribution of the response rates in particular.

The average response rate between Wave 3 and Wave 18 was around 84 per cent, ranging from 81 to 87 per cent. We exclude the first two waves as outliers. In the following paras we explain how the distribution of responses was disturbed by the lockdowns.

Wave 19 was the first to be impacted by the lockdown. It saw the response rate drop to 64.4 per cent. But, the distribution of response rates was quite uniform across income groups. The lowest response rate was from the lowest income group’s those with incomes less than Rs.100,000. The response rate for this group was 55.7 per cent. The response rate for the richest group’s annual household income of more than Rs.1 million was much higher at 62.8 per cent. And, the response rate for the middle three groups between Rs.100,000 and Rs. 1 million was between 65.3 and 66.4 per cent.

The biggest hit to the response rate was in Wave 20. It fell to 43.8 per cent. In this wave there is a clear inverse relationship between the response rate and the annual household income. While the average response rate was 43.8 per cent, the response rate from households at the lower end of the income spectrum, those with annual income of less than Rs.100,000, was 52.1 per cent. Then, the response rate falls to 45.5 per cent for households with incomes between Rs.100,000 and Rs.200,000 per annum. It falls further to 43.7 per cent for household with incomes between Rs.200,000 and Rs.500,000 per annum and then further to 37.9 per cent for households with annual incomes between Rs.500,000 and Rs.1 million. The response rate was the lowest at 31.9 per cent in case of households with income of more than Rs.1 million.

In Waves 21 and 22, better execution improved the response rates and also the skew in the response rates by household income groups. The range of response rates narrowed from 20 percentage points in Wave 19 to 6 percentage points in Wave 21 and 7.5 percentage points in Wave 22. In Wave 21, the average response rate bounced back to 70.6 per cent with a range of 66.1 per cent response rate for the richest households to 72.6 per cent for the mid-range income groups. In Wave 22, the average response rate was 73.4 per cent with a range of 69 per cent for the poorest households to 77.3 per cent for the upper middle class households.

Lockdowns were never lifted entirely. Mobility restrictions continue to pose challenges to the execution but a large the responses and the distribution across income groups has also improved.