Civil Service People Survey

The Civil Service People Survey is the annual employee attitudes survey of the UK Civil Service, and has a focus on measuring levels of employee engagement. I was responsible for the initial development of the survey’s data collection and analysis methodology (2009-2010) and subsequently lead the delivery team (2011-2015). More recently, I have gone on to support the team with automation, data engineering and reproducibility (2019-2022).

Developing an analytical framework for employee engagement in the Civil Service

Prior to the Civil Service People Survey’s inception in 2009 each government department and agency was responsible for running its own employee attitudes survey. This meant surveys were run using different question sets, a different times of the year, with differing frequency, and with different analytical techniques. This hindered efforts to make comparisons between organisations; whether to better facilitate sharing of good practice across government or to hold senior officials to account for the management and leadership of government departments and agencies.

The principal aim of the CSPS was to unify the collection of employee attitudes from the 100+ organisations that make up the UK Civil Service. Many institutions already ran their own survey, but with their own questions and run by different suppliers, therefore creating a challenge to develop a harmonised questionnaire that integrated aspects from these different pre-existing questionnaires.

The project also had an aim to introduce the concept of employee engagement to the UK Civil Service, influenced in part by the Engaging for Success report published in July 2009 (commissioned by the Department for Business, Innovation and Skills, written by David MacLeod and Nita Clarke). Engaging for Success did not introduce a specific set of questions for measuring “employee engagement” or for including in a staff survey. As a result there was a need to a specific measure for the CSPS.

Prior to the existence of the CSPS most departments and agencies outsourced all aspects of survey delivery, including questionnaire design, to external suppliers. While fieldwork and general reporting would continue to be outsourced, following experience delivering HMRC’s staff survey I joined the Cabinet Office’s programme team as a social research specialist to develop a distinct methodology for the CSPS.

Some external suppliers had already started to measure employee engagement in the employee surveys they offered to departments and agencies. However, there were concerns from stakeholders about consultancy language in the questions such as “going the extra mile” or “discretionary effort”, there was also conceptual mixing where some questions were measuring emotional affective aspects while others were measuring behavioural intentions. I developed a revised set of questions that had improved relevance for the public sector.

In addition to defining employee engagement there was a need to explore and identify the relationship between it and the other aspects of the questionnaire. This was achieved by using factor analysis, firstly to confirm employee engagement as a distinct concept that is separate from other measures, but also to group the remaining questions into themes. Regression analysis, presented to staff and managers as “key driver analysis”, was then conducted to review the correlation between the “engagement index” with each of the factors¹².

This was communicated to managers and staff through a flow diagram that showing illustrating a conceptual framework for employee engagement that showed the “driver” relationship between the survey factors/themes and the engagement index as well as the consequential relationship between employee engagement and outcomes such as organisational performance and employee wellbeing.

In addition to “simple” numerical representation of the engagement index, an “engagement segmentation” was developed to provide an alternative way to think about employee engagement by considering the different patterns of how people respond to the engagement index questions. This segmentation was developed using cluster analysis, identifying 7 different groups ranging from “disengaged” respondents that answered all 5 questions negatively through to “engaged” respondents that answered all 5 questions positively. This segmentation was based on data from the first results of the 2009 survey, I then developed a set of algorithms to allocate respondents to one of the 7 segments in future.

Analysing and reporting results

Each year the CSPS is able to produce reports for over 10,000 teams across the Civil Service, providing an overview of the survey results for leaders, managers and staff across all levels of the organisation. I oversaw two iterations of the design of these reports, ensuring that these presented a large volume of information – question results, aggregate scores (engagement index, and theme scores), time-series and benchmark comparison, driver analysis results – in simple, coherent and understandable ways³.

I was responsible for cross-government results analysis and briefing these results to the Civil Service’s most senior leaders. This analysis was both to summarise overall results but also focussed analysis for specific cross-government issues or areas of interest, for example:

• descriptive analysis of results for particular groups of civil servants such as for the Permanent Secretary disability diversity champion or the Government Economic Service;
• correlation and time-series analysis of units headed by Directors General to support career management and progression of top civil servants;
• correlation analysis of the relationship between engagement levels of Senior Civil Servants and junior staff;
• regression analysis of the relationship between engagement levels and sickness absence statistics;
• qualitative analysis of open-text comments for specific groups of respondents, such as disabled civil servants or Senior Civil Servants that said they wanted to leave their organisation within the next 12 months;
• natural language processing (NLP) modelling of open-text comments to develop topic and thematic structures of survey comments to inform policy makers looking at civil service and organisational reform.

From 2010 through to 2014⁴, I also produced an externally published report each year summarising both the headline results as well as key analytical findings that had been developed for senior leader briefings – read the 2010, 2011, 2012, 2013 and 2014 reports⁵, and the associated technical guide (2014 version⁵).

Automation, data engineering and reproducibility

My most recent work with the Civil Service People Survey (from 2019-2022⁴) has focussed on supporting the survey’s coordination and analytical team in the Cabinet Office to adopt automated approaches to their analysis of the survey. In these activities I worked with and mentored a reproducible analytical pipelines specialist that I had helped the team recruit.

The first automation project was to validate the PDF reports produced by an external supplier and check for errors. This involved developing processes to extract figures from the reports (taking account of subtle variations in the layout of reports based on the hierarchy level of the unit). The extracted figures were then compared to separate figures that had been computed independently of the supplier, and any discrepancies flagged. Comprising both direct survey results for the relevant group of respondents but also comparative figures each PDF report contains hundreds of figures. The time taken to develop the extraction process and automate it to run all 10,000+ reports was quicker than the previous approach that manually checked just a small sample of reports.

In 2020 the People Survey team switched suppliers away from a service delivered by a market research agency to a service provided by a cloud survey platform. A by-product of this switch resulted in a significant amount of post-survey data processing moving in-house to the Cabinet Office team. Together with the team’s reproducibility specialist, we developed an approach to clean and process the survey data to ensure no loss in fidelity in the information available between years, this included:

• cleaning data from the ~10,000 paper survey responses that had been input by a third-party data capture agency, as the cloud platform could not process questionnaires completed on paper;
• calculating derived variables that relied on responses from multiple questions, which could not be done in the platform, such as:
  • combining routed questions on detailed ethnic groups, locations and occupations into a single variable, or the National Statistics Socio-Economic Classification;
  • calculating variables such as the engagement index and theme score which aggregate numerical values from input questions;
  • mapping values from the organisation specific grade/payband questions into a the common cross-Civil Service grade structure;
• developing disclosure control routines to apply secondary suppression and thus minimise the residual risks that an individual respondent can be identified in the cloud provider’s online dashboard tools.

This data cleaning process was revised and improved in the course of the delivery of the 2021 and 2022 surveys. For example, ahead of the 2021 survey a process was developed to derive additional variables on gender identity following the alignment of the survey with the Census 2021 questions on sex and gender.

The cloud platform did not provide a mechanism to easily replicate the previous approach to driver analysis, using linear regression, to give individual teams (of an appropriate size) their own driver analysis⁶. I explored the feasibility of an “offline” process to calculate these models that could be run by the central team and produce one-page drive analysis reports for each team, or figures that could be uploaded to the cloud platform’s dashboard visualisations. I also reviewed the feasibility of reintroducing the segmentation analysis and uploading this to the cloud platform. Unfortunately, while both the driver analysis and segmentation could have been run by the central team with little additional overhead, and even integrated into the cloud platform’s dashboards, this proposal was rejected by senior managers.

I helped the team improve the quality and reproducibility of their central reporting to senior stakeholders such as briefings to Permanent Secretaries and HR Directors or reports for diversity and professional networks. This work developed into {deckhand}, an R package I have written. I also developed a prototype of an interactive website for the team, based on the govukhugo project.