Table of Contents
Key Findings
- Research points to long-standing ambiguity over the notion of skills and how they are defined. Skills shortages have traditionally been measured by estimating the supply and demand shifts of certain “skill proxies,” such as educational qualifications, fields of study, and occupations; however, this approach does not adequately depict the diverse and rapidly changing skill needs of the Canadian economy.
- Technical advances in data analysis (i.e., machine learning, natural language processing, and web scraping) facilitate the use of data from online job postings for gaining insight into skills shortages. Observing the frequency, content, and duration of these postings, as well as job seeker profiles provides an opportunity to improve our understanding of the changing nature of jobs.
- The availability of online data, combined with a Canadian skills and competency taxonomy, creates an opportunity to map skills to the Canadian National Occupational Classification (NOC) system, which will contribute to ensuring that Canadians have the right skills to succeed.
Introduction
Current and future skills shortages in Canada have been a major concern of policy makers for a number of years (see, for example, LMI Insights no. 3 and the 2019 federal budget). The persistent call to action has been to identify, with some precision, the skill needs of employers and employees. As our recent public opinion research has shown, “skill requirements” is a highly sought after labour market information need by individuals (second only to wages) and “finding qualified or skilled workers” is a major human resource challenge for Canadian firms (second only to retaining employees).
This edition of LMI Insights builds on the groundwork laid down in LMI Insights no. 3 and examines the topic of skills and skills shortages in greater detail. First, we discuss the conceptual and empirical challenges of defining “skills” themselves. Then, we review two approaches for identifying and measuring skills and related shortages
Defining Skills: Perspective is Everything
Definitions of skills vary widely between different sources and disciplines. Standard economics, for example, treats “skills” as the set of knowledge and other worker qualities that influence one’s labour productivity (Frazis & Loewenstein, 2007).1 The concept of skills in economics, therefore, is closely related to the general idea of “human capital.” In psychology, the term “skill” is far more specific. It typically refers to the application of some innate “ability” to the execution of a given task (Nickols, 2011; Krathwohl, 2002). Moreover — and in contrast to the standard economic approach — skills are regarded as distinct from the related concepts of “knowledge,” “ability,” and “tasks.”2 In addition, the approach in psychology is to express skills in relation to a given task. This makes separating a skill from its task and the conditions under which it is executed both conceptually and empirically challenging.
When Skills Aren’t Really Skills
As noted in LMI Insights no. 3, Statistics Canada (STC). and Employment and Social DevelopmentCanada (ESDC) maintain the National Occupational Classification (NOC) system using “skill level” and “skill type” as classification criteria. These criteria, however, do not refer to actual skills but to educational qualifications and the type of work performed, respectively. Moreover, “skill level,” often used in empirical work to study the supply and demand of skills, is widely recognized as a poor approximation for skills (Pellizzari & Fichen, 2017; Levels, van der Velden, & Allen, 2014; see also section below on the proxy approach to skills measurement).
To improve the link between skills and work, ESDC created the Essential Skills framework in the 1990s, identifying nine skills deemed “essential for learning, work and life.”3 This framework was later used to develop 372 Essential Skills Profiles describing how workers use each of the nine Essential Skills for 361 NOC occupational groups.
Recognizing the need to move beyond these nine categories, however, STC and ESDC, along with other federal agencies, are currently reviewing possibilities for the development of a more comprehensive and precise taxonomy of skills and competencies and their association to NOCs. In that context, skills are being defined as “developed capacities that an individual must demonstrate to be effective in a job, role, function, task, or duty.” These efforts are essential to finding common ground in the definition, and ultimately the measurement, of skills in Canada.
Measuring Skills: Direct and Indirect Methods
Currently, the most direct approach to measuring skills is through psychometric and skills testing. Psychometric tests capture general aptitude and behavioral characteristics, and can include personality profiling tests; logical, numerical, and verbal reasoning assessments; and situational judgement tests. They are sometimes used by employers to predict whether a candidate is able to perform a job or possesses the behavioral traits needed to be successful in a particular role (Shingal, 2019; Dullu, 2017). Skill tests, on the other hand, are tied to occupational competencies and measure specific abilities (Shingal, 2019). For example, applicants for software or programming jobs may have to pass a programming test as part of the recruitment process.
Using psychometric or skill tests to measure skills, however, has a number of limitations. First, psychometric tests do not measure discrete skills per se; rather, they estimate whether candidates have the knowledge, ability, and temperament to succeed in the workplace. They are predominantly cognitive in nature (i.e., skills in information processing, learning, and problem solving), detached from any particular job-specific task. Second, although skill tests offer more discrete skill measurement, they lack the theoretical and statistical basis of psychometric tests (Shingal, 2019; Dullu, 2017), and are more limited in scope. Finally, both types of assessment tools can be very costly, and, at best, only provide estimates of skill supply, not skill demand.
Two other methods of skill measurement are also available: skills by proxy and self-reporting. Although both take an indirect approach to measuring skills, they estimate both the supply and demand of skills — albeit to varying degrees — and therefore uncover shortages. Each method has other benefits and limitations, further elaborated in the following sections.
Measuring Skills Shortages
A proxy approach
The fundamental challenge of identifying and measuring skills shortages lies in the lack of any observable markets for individual skills. To address this gap, the proxy approach attempts to measure skills indirectly by using other, observable labour market measures to capture the demand and supply of skills.
The starting point to this approach (see Figure 1) is to find suitable and observable proxies for skills, of which the most common are educational qualifications (i.e., skill level), occupation, and field of study. The choice of proxy is guided by assumptions about the relationship between the proxy and the skills. When using educational qualifications, for example, one assumes all holders of a certain qualification (i.e., a bachelor’s degree) to possess the same skill set. Skill proxies based on occupations or fields of study face similar problems — namely, treating large groups of people as having identical skills, and jobs as having identical skill requirements.4
Once a suitable proxy has been chosen, an appropriate indicator (or a combination of indicators) is selected that, when analyzed and interpreted, signals changes in the supply and demand of that proxy. The presence or absence of a skills shortage is then determined based on these prevailing signals. In using educational qualifications as a proxy, one approach is to divide the labour force into high and low skill categories based on skill level. Specifically, highskilled workers are those working in occupations classified as professional or managerial (skill level A), or technical and skilled (skill level B). Next, if we accept hourly wages as an indicator of the latent value of the skills possessed by workers, we might infer a skills shortage if wages were to rise faster than, say, 3% per annum (wages are typically expected to grow at an average of 3% annually: 2% for inflation plus 1% for labour productivity growth).5
Figure 1: Overview of the proxy approach to measuring skills shortages
The choice of proxy is at the core of this indirect approach to measuring skills shortages. While some proxies are more directly tied to skills than others, the use of a proxy comes with notable drawbacks. As previously mentioned, it assumes that all individuals within a particular “skill group” possess the same set of skills. While empirically expedient, this is not realistic since there is great heterogeneity among individuals holding the
same qualification or credential due to differences in program quality (Hanushek & Kimko, 2000), individual ability, and a host of other reasons. In addition, skills acquired outside of the educational program (e.g., on-the-job training) or lost over time (McGuinness, Pouliakas, & Redmond, 2018) are not accounted for. Even if one uses more granular skill proxies, such as occupation or field of study, the same limitations persist.
Second, estimating a skills shortage relies on insights from economic theory, which suggests that certain indicators, like wages, respond to labour market pressures in predictable ways. But indicators do not necessarily react in the ways one would expect. For example, economists have noted the influence of regional and sectoral factors in preventing wages in certain industries from increasing despite shortages (Wilkins, 2019). Insufficient capital investment, monopsonistic market conditions (buyer power among employers), and the gig economy are also potential reasons for the failure of wages to rise. Furthermore, in using indicators, one assumes a sufficient level of granularity to be able to assess what is happening. Often, these indicators are available only at an aggregated level, obscuring local labour market conditions.
Of course, another major drawback to this approach is that it does not account for individual skill. This may be obvious given that it is a proxy, but this is particularly problematic in a labour market that is constantly changing and evolving. Clearly, information on specific skills and possible shortages has become essential. One way to close this particular gap is to use a skills-to-occupation mapping that lists the skills required by occupation (see Box 1). Such a mapping adds granularity to the skills proxy approach.
Box 1: Mapping Skills to Occupations
The most comprehensive and widely used skillsto-occupation mapping that exists belongs to the United States’ Occupational Information Network (O*NET) program. The program defines skills as “procedures for working with knowledge,” and includes a taxonomy of 35 skills. Occupational analysts rate the importance of each skill on a scale from 1 to 5 (1 being “not important” and 5 being “extremely important”) for each occupation of the Standard Occupational Classification (the US equivalent of NOC). For skills rated 2 or higher, analysts also assign a level of importance between 1 and 7. The result is a list of skills associated with each occupation of the classification.
The European Skills/Competences,qualifications and Occupations (ESCO) system defines skills as “the ability to apply knowledge and use know-how to complete tasks and solve problems.” The system contains 13,485 skills/ competencies. Here, skills are classified as either “essential” or “optional” for each occupation of the International Standard Classification ofOccupations (ISCO) system.
A self-reporting approach with big data
The second approach to measuring skills and related shortages makes use of self-reported skills data, which can be pulled directly from job seeker profiles (supply of skills) and/or employer job postings (demand for skills).6 Datasets with millions of self-reported skills supplied and demanded are available from firms specializing in web and database scraping, such as Vicinity Jobs Network, or directly from job portals, such as Indeed, Job Bank, and LinkedIn.
The first step is to compile information from job postings and job seeker profiles in a reliable and useable format (Figure 2). With respect to job postings, the data must be “cleaned” to remove duplicate and fake or unreliable job postings. This cleaning is often achieved using a variety of natural language processing (NLP) techniques. Similarly, text not related to skill requirements is removed. User profiles should be cleaned in a similar manner, retaining only profiles of individuals actively seeking employment in Canada, and extracting only those text features useful for analyzing the skills supplied by the individual.
The final, and perhaps most complex phase, involves organizing and categorizing skills in a consistent and coherent manner. Here, organizing skills into predefined categories and hierarchies is done differently by each entity processing the raw data. In other words, there is no common skills taxonomy. This makes comparison across data sources extremely difficult, if not impossible.
Figure 2: Sorting and defining skills
After organizing the skills into predefined categories, the next step is to estimate the extent to which a shortage exists for the identified skills. The challenge here is that self-reported skills data represent information before a market transaction is completed, which means we cannot use the traditional shortage indicators, such as wages or vacancy rates. But we can and do observe specific skills across a wide range of geographies and job types. The challenge is to convert these observations into well-founded measures of skills shortages. Several approaches are possible, but the applicability of each varies according the data available (see Table 1).
One approach relies on the frequency with which bundles of skills appear in online job postings. A rapid increase in the number of times a given skill or bundle of skills is listed could indicate an increase in the demand for said kills. Since skill acquirement is usually expected to lag behind demand, a sudden increase in the demand for a particular skill may denote a shortage. Similarly, one might interpret a rapid increase in the number of resumes in which a given skill appears as an increase in the skill supply (and potentially a reduction of a skills shortage).
One drawback to this approach is that there is no certain way to know if demand has been met. For example, online job postings may remain active despite having been filled, as some may be set to expire automatically. In these circumstances, an employer may let the posting expire on its own rather than take it down. Similarly, there is no way to determine whether an individual truly has acquired the skills stipulated in their resume; nevertheless, the mere presence of these skills could indicate a demand for them. The ideal data source would not only report the outstanding demand for skills via job postings, but also which individual and their skill set fills that role.
Table 1: Skill Shortage Indicators
Several concerns arise when analyzing skills shortages using specific skills or bundles of skills. First, the different ways of defining and describing skills makes consistent and accurate measurements difficult. Terminologies and naming conventions can differ across sectors, industries, and even employers. Moreover, skills data is often conflated with other worker characteristics, such as ability, interest, and work style. This makes it challenging to measure skills demands accurately and to separate out non-skill data.
Second, the wealth of individualized data from job postings and job seeker profiles introduces its own data gaps. Perhaps most obviously, job seekers have an incentive to overstate (within reason) the skills they possess. If we assume the degree of overstatement remains roughly constant over time and between skill levels and skill types, however, this is not so problematic. The trend in skills over time is the most informative. A potentially larger data gap is the unstated skills sought by employers. For example, many jobs require verbal communication as a skill but do not specifically mention it in the posting. Thus, it is very likely that the demand is understated.
Third, if both job posting data (skill demand) and job seeker data (skill supply) are available, it is possible to measure the relative balance of supply and demand at any point in time for a specific skill. Nevertheless, one is unable to know whether the change in skills supply is driven by a true change in the underlying skill sets of job seekers or if people are simply reporting their skills differently. This general problem of self-reported skills could be mitigated, in part, if the data included information on which job seekers filled which positions. With such matching information, one could conceivably extract the common skills demanded and supplied for a more refined picture of the latent markets for skills.7
Finally, this approach lacks some statistical validation. Online job postings likely skew toward higher-paying, urban job markets — though even this is difficult to determine concretely.8 Moreover, this bias may not be symmetric between job seekers and employers. For example, one can imagine online job postings to be more common for high-skilled jobs for certain sectors than others. While the large number of data points (e.g., over 200,000 job postings per month in Canada) helps reduce concerns about skewed data, persistent underrepresentation of certain segments of the job market — and hence of skills — is a reality.
The Way Forward
How we define skills can be confusing, which limits our ability to measure skills and skills shortages. There is an urgent need for a common, credible, and open skills taxonomy. This would enable stakeholders — job seekers, employers, and educators — to speak the same language with respect to skills.
The development of a common taxonomy would also create an opportunity to map these skills to the Canadian National Occupational Classification (NOC) system, a key step towards improving our understanding of the changing nature of jobs. The two prevailing methods of measuring skills and related shortages discussed here — each with its benefits and drawbacks — are proxying skills and self-reporting. These and other methods must be evaluated in detail regarding how they might contribute to ensuring that, in our changing and dynamic world of work, Canadians have the right skills to succeed and employers find the right talent to grow their businesses.
Acknowledgements
This edition of LMI Insights was prepared by Anthony Mantione of LMIC. We would like to thank LMIC’s federal, provincial, and territorial partners, as well as our National Stakeholder Advisory Panel and Labour Market Information Experts Panel, for their comments and suggestions for this edition. In particular, the team would like to acknowledge the valuable input of David Chaundy, Robert Carlyle, Ana Ferrer, Vickie Francisco, Sareena Hopkins, Naomi Pope, Ernie Stokes, Arthur Sweetman, and Nicolas Tremblay. We would also like to thank David Hartnett (ESDC), Strac Ivanov (Vicinity Jobs), Santo Milasi (European Commission), Matthias Oschinski and Asa Motha-Pollock (MaRS), and Marieke Vandeweyer (OECD), as well as the Skills and Employment Branch at Employment and Social Development Canada (ESDC) for their comments.
For more information about this edition of LMI Insights or other LMIC activities, please contact Anthony Mantione, Economist, at anthony.mantione@lmic-cimt.ca or Tony Bonen, Director of Research, Data and Analytics, at tony.bonen@lmic-cimt.ca.
References
Blake, N., Dods, J., & Griffiths, S. (2000). Employers skill survey: Existing survey evidence and its use in the analysis of skill deficiencies. Business Strategies. Retrieved from https://dera.ioe.ac.uk/15176/1/Employers%20skill%20survey%20-%20existing%20survey%20evidence.pdf
Cappelli, P. (2015). Skill gaps, skill shortages, and skill mismatches: Evidence and arguments for the United States. ILR Review, 68(2), 251–290. https://doi.org/10.1177/0019793914564961
Counter, R. (2018, September 4). Want a job in Canadian tech? Don’t worry about that university degree. Canadian Business. Retrieved from https://www.canadianbusiness.com/work/want-a-job-in-canadian-tech-dont-worry-about-that-university-degree/
Cross, P. (2014). Do labour shortages exist in Canada? Reconciling the views of employers and economists. Fraser Institute. Retrieved from https://www.fraserinstitute.org/sites/default/files/do-labour-shortages-exist-in-canada.pdf
Dullu, S. D. (2017). The top five psychometric tests used in recruitment. INDVSTRVS. Retrieved from https://indvstrvs.com/psychometric-tests/
European Centre for the Development of Vocational Training. (2013). Piloting a European employer survey on skill needs, no. 36. Retrieved from http://www.cedefop.europa.eu/files/5536_en.pdf
Frazis, H., & Loewenstein, M. A. (2007). On-the-job-training. Foundations and Trends® in Microeconomics, 2(5), 363–440. http://dx.doi.org/10.1561/0700000008
Green, F. (2011). What is skill? An inter-disciplinary synthesis. Centre for Learning and Life Chances in Knowledge Economies and Societies. Retrieved from https://www.researchgate.net/publication/264874441_What_is_Skill_An_Inter-Disciplinary_Synthesis
Hanushek, E., & Kimko, D. (2000). Schooling, labour-force quality, and the growth of nations. American Economic Review, 90(5), 1184–1208. Retrieved from https://www.aeaweb.org/articles?id=10.1257/aer.90.5.1184
Krathwohl, D. R. (2002). A revision of Bloom’s taxonomy: An overview. Theory into Practice, 41(4), 212–218. https://doi.org/10.1207/s15430421tip4104_2
Labour Market Information Council. (2018, 3 October). What’s in a name? Labour shortages, skills shortages, and skills mismatches. LMIC Insights, no. 3. Retrieved from https://buff.ly/2CDiBNo
Levels, M., van der Velden, R., & Allen, J. (2014). Educational mismatches and skills: New empirical tests of old hypotheses. Oxford Economic Papers, 66, 959–982. http://repec.economicsofeducation.com/2014valencia/09-34.pdf
McGowan, A., & Andrews, D. (2015). Labour market mismatch and labour productivity: Evidence from PIAAC data. OECD Economics Department Working Papers, no. 1209. Paris: OECD Publishing. https://doi.org/10.1787/18151973
McGuinness, S., Pouliakas, K., & Redmond, P. (2018). Skills mismatch: Concepts, measurement and policy approaches. Journal of Economic Surveys, 32(4), 985–1085. https://doi.org/10.1111/joes.12254
Montt, G. (2015). The causes and consequences of field-of-study mismatch: An analysis using PIAAC. OECD Social, Employment and Migration Working Papers, no. 167. Paris: OECD Publishing. https://www.oecd-ilibrary.org/social-issues-migration-health/the-causes-and-consequences-of-field-of-study-mismatch_5jrxm4dhv9r2-en
National Research Council. (2010). A database for a changing economy: Review of the occupational information network (O*NET). Washington, DC: The National Academies Press. https://www.nap.edu/catalog/12814/a-database-for-achanging-economy-review-of-the-occupational
Nickols, F. (2011). Task versus skill training: A matter of practice. Training Journal. Retrieved from https://nickols.us/TJ_TaskvsSkill.pdf
OECD. (2017). Getting skills right: Skills for jobs indicators. Paris: OECD Publishing. https://doi.org/10.1787/25206125
Pellizzari, M., & Fichen, A. (2017). A new measure of skill mismatch: Theory and evidence from PIAAC. IZA Journal of Labour Economics, 6(1), 1–30. https://doi.org/10.1186/s40172-016-0051-y
Psychometric Institute. (n.d.). The comprehensive psychometric testing guide: Aptitude and personality testing. Retrieved from https://www.psychometricinstitute.com.au/Psychometric-Guide.html
Quintini, G. (2011). Over-qualified or under-skilled: A review of existing literature. OECD Social, Employment and Migration Working Papers, no. 121. Paris: OECD Publishing. https://www.oecd-ilibrary.org/social-issues-migration-health/over-qualified-or-under-skilled_5kg58j9d7b6d-en
Royal Bank of Canada. (2018). Humans wanted: How Canadian youth can thrive in the age of disruption. Retrieved from http://www.rbc.com/newsroom/_assets-custom/pdf/03-2018-rbc-future-skills-report.pdf
Sattinger, M. (2012). Assignment models and quantitative mismatches. University at Albany, SUNY. Retrieved from https://www.albany.edu/economics/research/seminar/files/Michael%20Sattinger.pdf
Shingal, T. (2019). The difference between psychometric tests and skill tests. Retrieved from https://blog.mettl.com/ alent-hub/mettl-psychometric-tests-vs-skill-tests
Wilkins, C. A. (2019, January 31). A look under the hood of Canada’s job market. Bank of Canada. Retrieved from https://www.bankofcanada.ca/2019/01/look-under-hood-canadas-job-market/
End Notes
- These skills are often further divided into two broad categories: those deemed transferable or generalizable (“generic skills”) versus those directly tied to an employer or a firm (“firm-specific skills”).
- More information on the differences between abilities, skills, knowledge, and tasks as defined by the US Occupational Information Network (O*NET) program can be found here.
- The nine essential skills are as follows: reading, writing, oral communication, numeracy, document use, thinking, computer use, working with others, and continuous learning.
- For example, when using education as a proxy, one makes the implicit assumption that all jobs require the same educational requirements. This is changing, however, especially in the tech industry. See, for example, Counter (2018).
- This is the expectation when the labour market is roughly near full employment, when worker productivity is the strongest determinant of wages.
- Another possible source for self-reported skills data comes from survey tools. Although, at this time, Canada does not have any national surveys that specifically ask about skills, some European countries do. In the UK, for instance, two such surveys are the Extent, Causes, and Implications of Skills Deficiencies Survey and the UK Commission for Employment and Skills’ (UKES) Employer Skills Survey. Similarly, the European Centre for the Development of Vocational Training (CEDEFOP) piloted its Europen employer survey on skill needs in 2013.
- Strictly speaking, having matching information helps determine whether the company perceived the applicant to have the demanded skill set rather than the applicant’s true skill set.
- Some measurement of bias in the data can be extrapolated from Statistics Canada’s quarterly JVWS survey. Specifically, the answers that employers provide to the question about recruitment strategies (indicating percentage of jobs advertised online, broken down by province and 1-digit NOC) may be used to approximate the distribution of jobs posted online.
