Labour Market Information
Best Practices Guide
The Labour Market Information Best Practices Guide aims to provide best practices in the generation, analysis and dissemination of labour market information throughout the pan-Canadian ecosystem.
Table of Contents
Introduction
In 2018, the Labour Market Information Council (LMIC) launched two expansive information-gathering projects:
1
An environmental scan documenting the prevailing gaps identified in the labour market information (LMI) ecosystem in Canada.
2
A pan-Canadian public opinion research project surveying over 20,000 individuals and organizations.
From these two initiatives, we learned that LMI is difficult to access, use and understand for policy makers—including governments, organized unions and other organizations—job seekers, employers and career practitioners alike. This document outlines two LMIC work streams aimed to address key LMI gaps in Canada: producing quality LMI and enhancing its accessibility and use.
Overview of Labour Market Information Council
The Labour Market Information Council (LMIC) is a not-for-profit, pan-Canadian organization composed of a diverse group of individuals with expertise in the use of LMI. Our aim at LMIC is to empower Canadians to make informed decisions by enabling access to quality, relevant, comprehensive data and insights across the pan-Canadian LMI ecosystem. This includes conducting research, collecting and analyzing data related to the job market, and generating insights intended to enhance the quality (e.g., accuracy, precision, comparability) of LMI. LMIC also makes every effort to improve access to LMI by sharing and disseminating LMI (e.g., Insight Reports, online interactive dashboards) in a way that addresses the diversity of user needs. We also provide recommendations for clear and consistent terminology, data collection and reporting to guide the production and consumption of LMI.
Producing Timely, Reliable and Accessible Labour Market Information
Overview of LMIC’s approach to generating LMI
Background
Canada’s job market and workplaces are continually evolving. Drivers such as technological advancement, an aging population, changing global trading patterns and climate change create uncertainty around employment, job quality, skill requirements, and the ability to train and maintain a globally competitive workforce. This uncertainty has been intensified by the recent large-scale employment losses from the COVID-19 health crisis, underscoring the need for quality labour market information that addresses the workplace, career, training and educational concerns most salient to Canadians today and tomorrow.
Needs assessment
As one of our first tasks, LMIC conducted an environmental scan of the state of LMI in Canada. We reviewed a wide range of reports, including the Advisory Panel on Labour Market Information’s (APLMI) seminal 2009 report; Drummond and Halliwell’s 2016 report; Aon Hewitt’s 2016 survey of large private-sector employers; and the Canada West Foundation’s 2017 report, among several others.
In examining these reports, we identified three recurring inadequacies that greatly limit the usefulness and impact of prevailing LMI:
LMI is not tailored to users’ needs.
First, labour market information is not tailored to users' needs. It is not easily accessible by individual Canadians and is rarely provided in a format that lends itself to informed decision-making. Moreover, there is a lack of labour market insights for certain segments of the population.
LMI is not specific or granular enough.
Second, most LMI is not specific or granular enough to truly impact users’ decisions. Greater levels of detail are needed commensurate with the needs of users.
Confusion exists over LMI terminology.
Finally, much confusion exists over basic terms and definitions, primarily caused by a lack of standardized, consistent language and terminology, as well as non-transparent methodologies related to the production of LMI.
As a result, labour market professionals have raised concerns about its overall quality, challenges identifying the appropriate data to use, limits to the insights that can be drawn from the data, and obstacles to clear and effective distribution and delivery.
Baseline criteria for producing LMI
The three common themes identified in the needs assessment (see Needs Assessment) constitute the foundation of a theoretically sound, evidence-based approach for the generation and delivery of quality labour market information, which is summarized in Table 1.
Table 1: Three guiding principles for producing and delivering quality LMI.
| Theme/Issue | Guiding Principle | Related Attributes* | |
|---|---|---|---|
| 1. | LMI not tailored to user needs | Design LMI for end users. When producing LMI, follow a client-centred approach in terms of the data itself, its organization and its presentation. The resulting information is relevant and easily accessible in a format that best lends itself to the user’s decision-making process. | Scoped (Targeted, Relevant, User-specific) |
| 2. | Missing local granular data | Improve local granular data. When producing LMI, the level of aggregation and geography should be as transparent as possible. To the extent possible, data generated should be collected and made available at the most disaggregated level possible. This parallels the nature of decisions made by individuals, policy makers, organizations and others. | Local, Timely, Granular |
| 3. | Lack of standardized language; lack of open and transparent methodologies; inferior LMI | Enhance our knowledge and understanding of complex issues. Quality LMI should be underpinned by careful and thorough research and validated by experts in the field. This process helps to refine the information, garner support and approval among stakeholders, and prevent confusion. | Understandable, Meaningful, Consistent, Evidence-based |
* Complete definitions for related attributes are available in Appendix A.
Additional data quality attributes
Several additional attributes ensure that labour market information is accurate, accessible, interpretable and coherent. These attributes, informed in part by Statistics Canada’s Quality Assurance Framework, are summarized in Figure 1.
Figure 1: Six additional data quality attributes
Quality Attributes
Open
To the extent possible, labour market information should be cost-free, publicly available and easily accessible to all users. As a rule, data and related metadata should accompany any publication in a downloadable, structured format consistent with industry standards and be both human and machine readable. This contributes to transparency and reliability, as well as supports and encourages use of and reference to the collected data.
Metadata includes variable names, survey populations, reference periods, data collection methodologies, classifications/taxonomies used to prepare the data, data dictionaries and, if applicable, measures of the data’s accuracy. If methodologies cannot be released due to trade secrets, this should be noted; every attempt should be made to explain as much detail as is possible.
Comparable and viable
To the extent possible, and where appropriate, official standards for data reporting should be used (e.g., when reporting on occupations, the National Occupation Classification system should be used).
In cases where official standards are not appropriate (e.g. more granularity is needed to address client needs), the reason for departing from official standards should be noted. Classifications should be clearly identified, documented and justified; any significant changes from previous releases should be explained. This includes, but is not limited to, changes to variables, reporting units or the values themselves.
Metadata should adhere to recognized standards (click to download document) and follow suggested naming conventions and formats, which ensures comparability across data sources.
Protective of sensitive information
Data shared publicly should align with organizational and/or jurisdictional rules on confidentiality. Those accessing sensitive data should undergo mandatory confidentiality training. This is particularly important especially now, in the era of rampant cyber attacks.
Different methods can be employed to anonymize the data (e.g. making use of unique identifiers and suppression rules). These methods should be clearly and directly explained to users.
Statistically sound
If it has been determined that new data is required to address users’ LMI needs, data generation should adhere to evidence-based, statistically sound methodology (see, for example, Statistics Canada’s Quality Guidelines and Survey Methods and Practices). Using statistically sound methodology can improve the reliability, reproducibility and comparability of information.
Transparent
All limitations and caveats of the data and subsequent insights should be thoroughly communicated to the user. No data source is perfect, but it is important to be direct about its known limitations. This will determine if the data fits the user’s needs and contributes to a well-informed decision-making process.
Contactable
Producers of labour market information must provide a method of contact, as well as a mechanism for revising and updating their data or information drawn from this data. Producers should be willing and available to discuss the data with (potential) users. For example, users could be invited to direct any questions seeking additional information, clarification or suggestions to a particular company email address.
Methodology
Overview
In practice, LMIC engages in a six-phase approach based on the three guiding principles and additional LMI quality attributes (discussed above) to generate timely, reliable, accessible labour market information. We use the following steps:
Detailed approach
1
identify and refine the issue through research
The first phase of the LMI generation process involves several, often iterative, steps, beginning with the identification of the LMI user group and their needs. Users are the individuals or organizations that will use the information to make decisions regarding their career, training, educational pathways and so on. Their informational needs may be facts, figures, charts, graphs or insights about those areas of the job market relevant to their decision.
For example, one growing concern among various stakeholders—from policy makers and educators to job seekers and students—is labour and skills shortages. Recognizing the need for better information and clarity related to both the identification and measurement of shortages for these various user groups motivated the publication of our LMI Insight Report no. 3: What’s in a Name? Labour Shortages, Skills Shortages, and Skills Mismatches.
Once the preliminary user and LMI need has been specified, both are refined through careful, thorough research. The research process includes a review of all relevant academic literature, White Papers, Sector Council reports, media articles and professional blogs; interviews with stakeholders; and consultations with experts. Only through research can one be sure that information will be targeted, relevant and efficient.
When preparing LMI Insight Report no. 3, we began by conducting an academic literature review to understand the theoretical and conceptual notion of shortages. Next, we expanded our review using keyword searches, such as “shortages,” “labour shortages,” “skills,” “skills shortages,” “skills issues,” “skills mismatches,” “skills gap” and others. We then collected a wide variety of documents: White Papers, such as the 2013 Parliament and House of Commons labour and skills shortages report; media reports, such as Dangerfield’s 2017 Global News article; sector council reports, such as the Canadian Agricultural Human Resource Council’s 2016 report; and literature from other professional organizations like the Network for international polices and cooperation in education and training (NORRAG).
During the research process, one also identifies the requisite data needed and the possible sources to be consulted and then documents their relevant features. Table 2 provides an overview of the analysis we conducted with respect to data derived from online job postings. Only after achieving a complete understanding of the data, source, methodology and limitations is one able to use that data to generate accurate, reliable labour market information. It is possible, however, that no data exist to address the informational need. In this case, it is valid to consider collecting the data oneself (see C. "Statistically sound”).
Table 2: Examples of data features that should be documented in Phase 1
| Data features | Example: LMIC Online Job Postings Dashboard |
|---|---|
| All definitions and descriptions of the data and source (including metadata) | In creating our Canadian Job Trends Dashboard, a team of LMIC economists defined key labour market terms, descriptions of the data and the data sources used. |
| All variables available within a given data set | All variables were clearly defined and distinguished to avoid confusion. For example, the underlying information used is job postings data collected by Vicinity Jobs. These contain information on job openings posted by employers and their work requirements. LMIC economists defined each variable within work requirements, disambiguating between skills, qualifications, etc. |
| The data collection method, including frequency of data and levels of granularity and localness | The dashboard contains information on the data collection methodology, which is scraping online job postings with machine learning algorithms. The raw data is grouped by geography (CMA and CA) to improve the localness and granularity of the data. |
| Accessibility of the data | The dashboard is publicly available and is updated weekly. The data is easily accessible and can be downloaded in a useful format (spreadsheets). |
| Advantages and limitations of the data | The key benefits and limitations of the job postings data are clearly defined in the accompanying documents (methodology documentation and FAQs). |
| Data assumptions | Data assumptions are identified and discussed in detail in the methodology document. |
| Context of use | How the data can be used and interpreted is explained in detail in the methodology document of the dashboard. |
2
Generate labour market information
The next phase of the process involves turning the data into actionable insights that clients can use to inform their career- or workplace-related decisions. As with Phase 1, the LMI generation phase can entail multiple stages (summarized in Table 3).
Table 3: Three steps to generating LMI
| Generating LMI | Description/Example | |
|---|---|---|
| 1. | Determine the product and method of delivery | Products: Charts, graphs, infographics, insights, interactive tables, research reports.
Delivery: Newsletters, company websites, online dashboards, application programming interfaces (APIs). |
| 2. | Working with the data | The data is cleaned and analyzed at an appropriate level of granularity to be meaningful to the user (e.g., 4-digit NOC by economic region). |
| 3. | Drafting the final product | In the case of a dashboard, we create a staging site that can be shared to elicit feedback from our stakeholders. Revisions can then be made before making the site public. |
To begin, the nature of the product to be created should be determined. For example, the product might be a chart or figure, an insight or infographic, a dashboard or a full research report, to name a few. Regardless, the final product and its delivery should be driven by end users and their needs.
While it may be the case that a final product and delivery format were chosen prior to starting the research process, the product and its delivery should be revisited after conducting (or even during) research since the initial choice may need to be revised. When writing LMI Insight No. 3, for instance, we originally set out to analyze the methods for identifying and measuring skills shortages. As we engaged in the research process, however, it became evident that, conceptually, skills, labour shortages and skills shortages have not been consistently defined, resulting in much confusion. As a result, we decided to first write a report that clarified these labour market issues and recommended definitions.
The next step during this phase usually involves working with data to generate insights, as summarized in Table 4.
Table 4: Five steps of working with data
| 1. | Access data | What data are you using? How can it be accessed (e.g., API or web scraping, Databases, online)? |
| 2. | Exploratory data analysis (EDA) | What is in the data set? What are the variables? What types of values exist? Are there missing or inconsistent values? Are there particular segments of the data? Do duplicate values exist? |
| 3. | Data preparation and cleaning | Data cleaning is the process of correcting missing or inaccurate records, changing data types, filtering rows, creating new variables and conducting conditional processing. |
| 4. | Data analysis and reporting | Extracting information and insight from the data, understanding the data and deriving conclusions. |
| 5. | Exporting results | Export your analysis and conclusions into a format that is fit for publication. |
The last step of this phase involves formalizing the results and formatting them according to the specifications defined in step one. For example, if the final product is to be a short report, such as an LMI Insight Report, this step would include creating a draft of the report to submit for review. It is important to note that although these steps are described sequentially, in practice, they may occur simultaneously and proceed iteratively.
3
Submit for internal peer review
Once a working draft has been constructed, it is subjected to an internal peer review process by several members of the Research, Data and Analytics (RDA) team. This team is composed of six economists, each with expertise in one or more areas related to the job market. The draft may undergo several revisions based on the recommendations and feedback of the RDA team.
Submit for external peer review
After the product has been reviewed internally, it is passed out for external review. We typically consult two groups of external reviewers.
One is our National Stakeholder Advisory Panel (NSAP), composed of non-governmental stakeholders with substantial insight and expertise in labour market information. These individuals may be sector council leaders, career development practitioners, economists, business leaders, HR professionals, education professionals, etc.
The second group is our Labour Market Information Experts Panel, which consists of LMI experts including university professors and other academics and researchers across Canada.
As with internal peer review, the LMI product may undergo several revisions based on the recommendations and feedback of the external peer reviewers.
5
Publish and request feedback
After the product has been published and is accessible through our website and social media, stakeholders and readers can provide us with feedback, comments and suggestions through social media channels and a monitored inbox. Although changes may not be made to the product, the feedback is collected and consulted when developing future initiatives to ensure that Canadians can access the LMI that they need and want.
6
Update as necessary
Part of our mandate is to ensure that the LMI we deliver is relevant and up to date. To adhere to that, many of our publications are updated regularly. Our Future of Work Annotated Bibliography, for example, is updated whenever new reports are published. Similarly, our Canadian Job Trends Dashboard is updated every week when new postings have been collected. Updates to our products are then shared through our website, social media and newsletter.
Products that require regular updates are monitored and updated accordingly, some more frequently based on level of importance and relevance. The decision about which products are more relevant is determined by what is currently influencing the labour market and topics frequently discussed by labour market organizations, economists and our stakeholders.
Interpreting and Applying LMI in the Decision-Making Context
Background
The production of quality LMI is the first step towards facilitating career, education and workplace decision-making for all Canadians. The second is to ensure that this information is used appropriately and in the right context. To accomplish this, LMIC has committed to three ongoing, evergreen projects:
1
WorkWords, designed to enhance the overall awareness, knowledge access, use and understanding of LMI and key related terms.
2
Background documentation, which accompanies all of our LMI tools and plainly outlines the data source(s), data collection and methodology, as well as special considerations, such as data limitations, to be aware of.
3
A series of detailed LMI “user guides” that provide explanations, context and examples of the appropriate use of LMI, especially in a career-counselling framework.
WorkWords: An online labour market encyclopedia for LMI
WorkWords is an online labour market encyclopedia that provides definitions of key labour market terms, data and concepts. It is designed to bring clarity and understanding of complex labour market subjects that will improve the use, application and interpretation of LMI. Each entry contains plain language definitions, links to assist the user in accessing data sources, advantages and limitations to be aware of, as well as guidance on how to use the information to draw insights to assist in decision making.
Consider occupation outlooks, a widely used but often misunderstood type of labour market information. Many different outlooks are produced in Canada by the federal government, some provincial and territorial governments, as well as various research-oriented organizations (e.g., Conference Board of Canada, Stokes Economics) and sector councils. Sometimes details about their construction can be vague, if they exist at all. This makes evaluating their reliability and usefulness challenging.
This is where WorkWords can help:
Definitions and Sources
In general, our entries begin with a section entitled “Definitions and Sources,” which provides plain language definitions and alternative terminologies—if applicable—as well as descriptions of sources. The occupation outlooks entry, for example, begins by describing what they are and what other names they go by (i.e., “labour market outlooks,” “occupational demand and supply outlooks,” and “job outlooks”). This is followed by a brief description of what they are used for—to identify potential future labour market imbalances. Next, we provide an overview of the different entities producing occupation outlooks. This section concludes with a detailed explanation of the general approach used to create these outlooks, including key assumptions, and an in-depth comparison of the different approaches used across Canada.
Data access
The next section common to all WorkWords entries is “Data Access.” Here readers will find direct links to data sources. Federal sources, such as Statistics Canada surveys, are usually listed first, followed by provincial and private sources where applicable. For example, the occupation outlooks entry provides links to occupation outlook reports produced by different organizations in Canada, as well as their detailed individual methodologies.
Applications
Finally, all entries conclude with an “Applications” section that documents how the information is used in Canada. The occupational outlooks entry, for instance, describes how outlooks can be used by individuals, industry, educational organizations and governments in decision making. Employers, for example, use occupation outlooks to inform Labour Market Impact Assessments when submitting requests to hire foreign workers under the Temporary Foreign Worker Program.
By providing clear definitions and explanations for labour market concepts, as well as detailing the methodologies, availability and access to LMI, WorkWords serves as an important tool for improving LMI knowledge and literacy in Canada. As an evergreen product, WorkWords will continue to add new entries relevant to today’s job market and meet the needs of all decision-makers. Existing entries will also be updated regularly to ensure accuracy.
Background documentation
Whereas WorkWords provides explanations for general labour market concepts, users also need guidance on the appropriate use and application of the LMI products produced, prepared or distributed by LMIC. This need is partly addressed by the inclusion of the background documentation that accompanies each of our products, from dashboards to Insight Reports. Our Canadian Job Trends Dashboard, for example, provides information and insights on the work requirements of occupations in Canada. It comes with documentation that explains how the tool should be used, outlines the information available, describes data limitations, and presents the methodology behind the data collection and analysis for full transparency.
LMI user guides
To further ensure that LMI is used appropriately and in the right context, LMIC is partnering with career development professionals to create a series of user guides for each type of LMI. Whereas WorkWords contains general information on a variety of labour market topics, user guides will cover specific types of LMI (e.g., occupation outlooks, salaries/wages, skill requirements) with the goal of providing explanations and examples of how best to incorporate each type of LMI into the decision-making process. While the information will be useful to a broad range of stakeholders, the primary purpose is to provide career development practitioners with a toolkit of how to use and interpret specific LMI effectively in their work with their clients.
Conclusion and Request for Feedback
Considerable progress has been made to improve the quality of labour market information (LMI) in Canada, but several important gaps persist, such as the absence of local, granular data. These guidelines were created, in collaboration with stakeholders, to provide directions to improve the accessibility, use and clarity of LMI. We encourage producers to apply these guidelines to help ensure that new LMI is generated in formats and styles that are useful to all Canadians. We encourage consumers to apply these guidelines when consulting LMI to ensure that they are using the right information for their needs.
As with our other LMI products, we are dedicated to ensuring that these guidelines are up to date and relevant to all Canadians. As a result, these guidelines and any products discussed in them are subject to review and revision. Your suggestions and feedback are critical to this process. We invite you to share your input and views by emailing us at info@lmic-cimt.ca. You can also follow us on Twitter and/or LinkedIn and subscribe to our newsletter.
Appendices
Appendix A – Definitions of related attributes
Scoped
(targeted or relevant
or user-specific)
LMI is created to address specific concerns of the decision-maker relevant to the scope of the project. In other words, LMI should be targeted to the end user’s needs and provided in formats or styles that promote use and engagement. For example, LMI producers generating pan-Canadian wage information must be aware that information on occupations varies by location. To ensure that the data generated meshes with the project scope, the data collected must be granular (i.e. grouped by province) and presented in multiple formats (e.g., annual wages, biweekly wages, monthly wages). Doing this will ensure that the data generated is targeted, relevant and useful to the end user.
Local
LMI should describe the smallest geographical area possible. Consider, for example, the unemployment rate. We can calculate the unemployment rate for all of Canada, but it is also available at more local levels, such as by province, by economic region and by city. The narrower the information, the more local it is said to be.
Timely
Strictly speaking, timeliness refers to the lag between data collection and publication. LMI that is available for public consumption soon after collection is said to be timely. In many cases, timeliness also encompasses the idea of frequency, which is how often the data are renewed (e.g., monthly, annually). Here, we use timely to refer to both concepts.
Granular
This refers to the number of categories used to group the information. Consider, for example, the unemployment rate. We might want the unemployment rate for youth or for Indigenous persons. We might also want unemployment information by occupation, immigration status or education level. The greater the number of categories, the more granular the information.
Understandable
Information should be communicated in plain language and be free of ambiguity. Jargon, acronyms and other specialized language should be avoided.
Meaningful
Information should be useful to the end user. How the data is generated, and the limitations of the methodology, should be clearly communicated. This can help the end user to make informed decisions about the relevance of the data.
Consistent
Concepts and words should refer to the same ideas in all circumstances where possible. For example, the term “skills gap” has been used to reference three different problems in Canada: 1) a deficit of basic skills acquired through K–12 education; 2) a firm’s existing staff not having the skills needed to do their jobs efficiently; and 3) the lack of required skills among applicants for a job. All three situations are distinct and require different interventions. Building a common language where a term is clearly defined and then used consistently is important.
Evidence-based
Information is based on the best available current research. LMI generated must be created with sound and proven methodology to ensure that it is reproducible and comparable.
Appendix B – Producing quality LMI: A guide for producers
Quality LMI Attributes Checklist
The Quality LMI Attributes Checklist can help ensure that Labour Market Information (LMI) produced in Canada is meaningful to all Canadians.
The questions below cover key LMI challenges identified through environmental scans and public opinion research. Satisfying the requirements below can ensure that Canadian LMI is easy to access and use, and is understandable by all.
By answering “yes” to the Quality LMI Attributes question, you are ensuring the Canadian LMI requirements under gaps addressed are being fulfilled.
| www.lmic-cimt.ca | ||||
|---|---|---|---|---|
| Quality LMI Attributes | Yes | No | Comments | If yes, gaps addressed are: |
| Is the information designed for the end user? | Use/ Understand | |||
| Is it local, timely and granular? | Access/ Use | |||
| Is it understandable, meaningful and evidence-based? |
Access/ Use | |||
| Are the insights user-specific? | Use | |||
| Is the data cost-free and publicly available? | Access | |||
| Is the metadata included? | Use/ Understand | |||
| Are all limitations and caveats reported? | Use/ Understand | |||
| Does it align to national standards for reporting? If not, is the reason clearly noted? | Use/ Understand | |||
| If collecting own data, does the experimental design adhere to evidence-based statistical best-practices? | Access/ Use/ Understand | |||
| Do you provide a way for users to contact the LMI provider for follow-up? | Use/ Understand | |||
| Has sensitive information been anonymized properly? | N/A | |||
| Is the data open? | Access | |||
| Is the data statistically sound? | Use/ Understand | |||
| Is the methodology transparent and understandable? |
Access/ Use/ Understand | |||
| Is there contact information available? | Use | |||