Measuring inclusion.
Inclusion is an essential, and often overlooked tenet of building a diverse workforce. To see the whole picture, we explore a range of approaches to measure, and track inclusion.
Approaches to measuring inclusion
In our last post we provided an update on our workforce diversity — data we believe is important to share, but that fails to paint the full picture of the experience once talent is through the door. At Square, we place the “I” before the “D” as we believe inclusion is an often overlooked tenet of building a diverse workforce. For this post, we’ll unpack approaches for using data to assess inclusion. While companies are increasingly talking about the importance of inclusion (which is great!), there is a less robust discussion around how to measure inclusion quantitatively. In truth, inclusion is trickier to measure and track than diversity. Some approaches include surveying employees to measure sentiments like belonging, or tracking turnover rates to understand how retention compares for employees of different demographic groups. But as we discuss below, each of these approaches has their limitations. When it comes to measuring inclusion, one metric doesn’t do it — a multifaceted approach is needed to see the whole picture.
Tracking employee sentiment
One key tool that we use to track inclusion is our biannual employee engagement survey, which we call Pulse. This survey includes a host of questions related to inclusion — questions about belonging, decision-making, growth opportunity, fairness, recognition, and more — allowing us to track sentiments about these topics over time and compare them across teams, locations, and tenure groups. Furthermore, we are able to explore the correlation of survey responses to outcomes such as employee attrition.
Pulse provides us with a useful overview of how the company and different teams are feeling on matters of inclusion, but it doesn’t immediately provide an understanding of how members of underrepresented groups are feeling in comparison to others. This angle is critical to explore, as differing experiences for underrepresented groups may be masked within team averages. In the past we ran a stand-alone Inclusion Survey to assess group differences in inclusion, but today, in an effort to bring an inclusion and diversity (I&D) lens to everything we do, we have shifted to make a deeper inclusion analysis a standard part of our Pulse process.
Now, as part of every Pulse cycle, we run an additional inclusion analysis of Pulse results based on the answers to inclusion-related questions. We look for any meaningful gaps in aggregated survey scores along the lines of gender, race/ethnicity, and age (self-reported by employees). To protect employee privacy, we only run the analysis for teams of 25 or more people where there are at least 5 members of each demographic group being compared. Findings are aggregated to further protect individual privacy. We surface our findings in an inclusion report customized for each org’s leader, showing which (if any) questions have meaningful gaps in average group scores.
An illustrative example of a report with fabricated data
What we like about running a statistical inclusion analysis as part of our standing employee engagement survey is that it establishes considerations of inclusion and diversity as a standard part of how leaders assess the health and performance of their team. The analysis provides leaders with data to understand how well they are building inclusive teams, and more importantly, to identify opportunities to do better. The hope is that this analysis, rooted in employee sentiment, uncovers inclusion issues that can be acted upon before they are reflected in outcomes such as attrition. Understanding employee retention, however, is also critically important.
Measuring employee retention
As a complement to survey data, we look at employee retention and turnover to monitor whether any demographic groups are more likely to leave the company than others. Higher voluntary turnover among some groups not only can signal potential inclusion concerns, but also can undo the work of building diverse teams. Below is an overview of some retention metrics to consider.
Turnover rate comparison
The simplest approach to measuring retention is to compare turnover rates for various groups. Turnover can be calculated by dividing the number of employees who left by the average headcount across a given time period. For example, if 8 women software engineers left over the past year, out of a population of 100, that would be an 8% turnover rate. You could compare that to the turnover rate for software engineers who identify as men — let’s say 30 people left, out of a population of 250, giving you a turnover rate of 12%. From those numbers, you might conclude that women in engineering are less likely to leave your company. Given that women are underrepresented in engineering compared to the overall population, you’d probably see this as an indication that women are not facing significant inclusion issues.
However, because there are a number of factors likely to impact turnover (notably, tenure, seniority, and role type), comparing turnover rates doesn’t always tell the whole story. In the above example, if the women engineers had a lower tenure on average than men engineers (making the women engineers less likely to leave), the lower turnover rate for women may largely reflect this discrepancy in average tenure, rather than indicate a particularly inclusive environment.
Because high-level turnover rate comparisons don’t account for the influence of these other factors, they may incorrectly show (or conceal) group differences in retention. Accordingly, we focus turnover comparisons on groups that are similar on factors that we know generally influence turnover likelihood. For example, when comparing turnover between demographic groups, we look at employees within the same role type, level of seniority, or range of tenure.
Advanced retention risk modeling
Beyond turnover rate comparisons, we can also leverage more advanced statistical modeling techniques to understand our employee retention and proactively monitor for gaps. For one, logistic regression can uncover how a demographic (like gender, race / ethnicity, or age) influences an employee’s likelihood of leaving, while accounting for other employee characteristics (like location, tenure, role type, or job level). This allows for an apples-to-apples comparison between demographic groups.
A survival analysis is another statistical approach that helps to identify long-term retention trends. Survival analysis takes tenure into account by design, starting everyone at zero tenure and modeling retention over time. This analysis allows us to readily compare retention among demographic groups within specific cohorts (e.g. team, function, hiring cohort). While this analysis is more complex than simple turnover rates, the output is fairly intuitive, providing a clear visual representation of any discrepancies in retention by groups. Because survival analysis requires more data over a longer period of time, these approaches are less useful for detecting recent changes and best used to keep an eye on longer-term trends.
Across both the simple and advanced retention measurement approaches, these analyses still don’t paint a full picture of why employees leave or what aspects of a company or team culture aren’t inclusive. To understand what may be driving any notable differences in retention between groups, it’s necessary to review quantitative and qualitative data (like comments from employee engagement and exit surveys, and conversations with employee resource groups) to uncover specific issues facing underrepresented groups.
At the end of the day, inclusion is a feeling, making it quite challenging to measure quantitatively. Nonetheless, we see immense value in strengthening the ways in which we use data to better understand inclusion across the company and to communicate with our leaders about its importance. Developing an approach to measuring and tracking inclusion is a process of inventing and learning, and one that won’t stop here for us at Square.