Workplace Digital Essential Skills in Rural Small Businesses Pilot Project

SDRC – Digital Essential Skills Assessment

Orientation Session, October 30, 2013, Campbellton, New Brunswick

Preliminary Steps

Identifying the occupations where participants would be trained in

Identifying the digital tasks participants would be trained to perform

Targeted Occupations -1

Identified occupations where there were would be large numbers of lower literacy workers who use Information and Communications Technologies

Based to some extent on the National

Occupational Classification system and the OLES website – identifying required skills relating to Information and Communication Technologies

Confirmed in the pre-ONA conducted in the summer that these were the “right” occupations3

Targeted Occupations - 2

Administrative (all sectors): general office, administrative, accounting, purchasing, and production clerks

Production (manufacturing sector): machine operators, assemblers, inspectors, material handlers, labourers, cleaners

Operations (service sector): cleaners, housekeepers, cooks, material handlers, labourers, patient service associates

Client service (all sectors): customer service clerks, receptionists, client service assistance, sales associates

4

Digital Tasks – Desired Attributes Examined past research – external validity

Advised by Satya Brink, an expert in the field

Derived a framework: a continuum of tasks covering the range of digital activities workers perform in their job

Tasks are common across, but contextualized to,

occupational groups

Confirmed in the pre-ONA with participating organizations5

Digital Tasks

Use1. Consult/read documents to the Web or other digital media2. Use documents/databases on the web or other digital

media3. Complete forms on the Web or other digital media

Understand4. Perform online commercial transactions 5. Search databases on the Web or other digital media6. Seek, find choose information on the Web

Communicate/Create/Do7. Use a calendar on the Web or other digital media8. Communicate electronically with co-workers, suppliers,

clients to coordinate workplace activities6

Assessments - Purpose

Administrative purpose (pre-training assessment): to determine which digital tasks an employee lacks the skills to perform and therefore needs digital skills training for

Employer indicates what tasks the employee does and thus needs to be assessed in

Results contribute to “skills map” of the organization shown to employer

Research purpose (with post-training assessment): to measure the effectiveness of training in increasing digital skills levels

by comparing pre- to post-training levels7

Assessments - Considerations Pre & post training assessments should be very

similar Reason: to ensure measured gains in skills between

pre- and post-training are due to the training and not the assessment instrument (easier or harder)

Risks:1. potential for participant to remember the pre-test2. can’t be totally certain skills gains are due to only the

training (may have been other causes) Common to all training evaluations where there is

no control group (individuals who were not trained) Should not “teach to the test” - assessment should

not have exactly same content as the training otherwise would be testing memory, rather than ability

8

Creating Assessments

Creating an assessment for each of the 8 digital tasks, with some tasks having up to 3 sub-tasks

This means a lot of assessments: for each occupational group each with 4 or 5 component detailed occupations, there are up to 80 cells for which assessments must be produced

However, once a “generic” assessment question set is produced for a digital task, it can fairly easily be contextualized across detailed occupations

By adapting positions and documents applicable to the particular occupation

9

Contextualizing to OccupationsDigital Tasks Administration Production/Operations Client Service1. Refer to documents on the Internet or by using other digital media

notes, letters, procedures manuals, schedules

notes, letters, service or user manuals

notes, customer info sheets, delivery schedules

2. Use documents and databases on the Internet or by other digital media

address books, inventories, suppliers lists

lists, parts catalogues, user manuals

client lists, price lists, competitor catalogues

3. Complete forms on the Internet or by other digital media

invoices, purchase orders, shipping slips (or waybills), time sheets

production reports, purchase orders, time sheets

client complaints forms, sales reports

4. Perform online commercial transactions

order, pay order, pay order, pay

5. Search databases (…) on the Internet or other digital media to find useful information (…)

supplier lists, supplies lists prices, quantities, contact info

suppliers’ part/product lists. dimensions, quantities, contact info

client and competitor lists addresses, telephone numbers, email addresses

6. Seek, find and choose information on the Internet

prices, government policies, regulations

equipment operation or repair specifications

competitor prices and products

7. Use a calendar to schedule … on the Web or other digital media

activities, meetings, orders production, delivery schedules sales, delivery

8. Communicate electronically with co-workers, suppliers and clients to coordinate workplace activities

nothing specific to this occupation nothing specific to this occupation nothing specific to this occupation

Digital Skills Score

To measure skills gains, need to measure gradations of skill not just whether or not one has a skill

Skills gains = difference between pre and post-training skills scores

Skill score depends on 3 dimensions of complexity: closed/open and well/un-defined, and complexity of sub-tasks required for each task

Complexity – 2 Dimensions

Closed/open: ◦ Closed problem has limited solutions: “Identify the number of

employees that left the company in April” ◦ Open problem has unlimited solutions: “Improve the system for

waste disposal that meets municipal green policies”

Well/not (or un)-defined: ◦ Well-defined problem is directive and clearly tells employee

what they are supposed to do: “Find and download government form x”.

◦ Undefined problem is less clear about what to do but more clear about the situation and what the employer needs: “Schedule the delivery of products to Calgary within the next 48 hours.”

Complexity of sub-tasks

Complexity of sub-tasks in different domains required to successfully carry out the task:◦ Prescriptiveness of instructions◦ Clarity of expected outcomes◦ Sophistication of technological applications◦ Complexity of digital info and needed info processing skills◦ Preciseness of task setting

Still working on details to formulate a skill score, but an example follows

Complexity Scale – Illustration

14

  1 2 3

1. USE

 

Task: Use Digital Information for consultation purposes only simple instructions,

repetitive actions a few simple identified

technological applications within a precise task setting

Task: Use Digital Information to perform a simple task more complex instructions but

fixed actions using several identified simple

technological applications within a defined task setting

Task: Use Digital Information to perform multiple, undetermined and complex tasks several complex instructions or no

instructions and unidentified actions using unlimited and undetermined

complex technological applications (selection)

within a familiar task setting

1. UNDERSTAND

(ORGANIZE)

Task: Understand Digital Information for one specific, simple purpose clear instructions and

requirements a few simple well-known

technological applications within a less precise task

setting:

Task: Understand and Organize Digital Information for simple tasks more complex instructions and

requirements using several potential

technological applications (selection)

within a partially defined task setting

Task: Understand and Organize Digital Information to perform multiple, undetermined and complex tasks several complex instructions or no

instructions and requirements using unlimited and undetermined

technological application (selection) within an less determined task setting

1. CREATE,

COMMUNICATE

AND

DO

 

Task: Create Digital Information for one set or fixed outcomes simple instructions and a

simple process using a few simple

technological applications (selection)

within an imprecise task setting

:

Task: Create and Analyse Digital Information for several expected outcomes with complex instructions and

multiple yet simple processes (selection)

multiple identified potential technological applications (selection)

within undefined task setting:

Task: Create, analyse and communicate Digital Information for several self-selected potential outcomes several complex or no instructions using multiple undetermined and

unlimited processes and technological applications

within an open undefined task setting (selection)

CLOSED OPEN

NO

T D

EFIN

ED

DEF

INED

1 2 3

2 3 4

3 4 5

Example of an Assessment: Task 8 (Q1 & Q2), Coordinating activities with colleagues by email

Question 1: You received an email from the bookkeeper asking about the arrival of some merchandise. Forward the email to your colleague Michel Laurent to ask him if he received the merchandise. Insert a subject and copy (CC) the bookkeeper on the email. The message should say:

Hello

Please let me know if you remember receiving the merchandise.

(your name)

 Question 2: You received an email from the production logistics coordinator asking you to forward an attached notice about overtime policy change to two colleagues, Eric Bernard and Gabriel Robert, and to notify them about the upcoming meeting (to discuss the new policy). You need to CC the coordinator on the email, and BCC the director general, Charles Dubois. Insert a subject and write this message:

Hello

Please read the attached policy on overtime hours. Note that there is a meeting about the policy on Thursday the 14th at 12:30PM.

(your name)

Example of an Assessment: Task 8 (Q3 & Q4), Coordinating activities with colleagues by email

Question 3: You received an email from the production logistics coordinator asking you to send your colleagues, Simon Fremont and Luc Thompson, the attached notice about the water service interruption. Forward the notice as an attachment to your colleagues. Then CC the sender, and BCC the director general, Charles Dubois. Indicate that the email is urgent. Insert a subject and write this email:

Hello

Please read the attached notice about the water service interruption. Make arrangements to avoid a delay in production.

(your name)

Archive both the received and forwarded emails in the Management folder of your email application. Question 4: You received an email from the production logistics coordinator asking you to send an attached pamphlet about a new piece of production equipment. You must forward the attachment to your colleagues, Jean-Claude Petit and Yves Lefebvre, and notify them there is an information session about the machine on Wednesday at noon. Make sure you CC the email sender and BCC the director general Charles Dubois. Also, indicate the message is urgent, and archive both emails in the Management folder.

Example of How an Assessment Might be Scored: Task 8, Q1 and Q2Question 1 Complexity Score: 3 (Closed/Open: 1, Defined/Not defined: 3) Task description (domains): Simple and precise instructions (1),

defined outcome (1), defined technological application (1) and information (1), and defined task setting (1), routine task with definite steps.

Question 2 Complexity Score: 4 (Closed/Open: 2; Defined/Not defined axis: 3) Task description (domains): Simple task with more complex

instructions (2), defined outcome (1), moderately complex technological applications (2), defined pieces of digital information (2), and defined task setting (1), routine task with definite steps.

Example of How an Assessment Might be Scored: Task 8, Q3 and Q4Question 3 Complexity Score: 4 (Closed/Open: 2, Defined/Not defined: 3) Task description (domains): Simple task with more complex

instructions (2), less precise outcome (2), moderately complex technological applications (2), a few defined and more complex pieces of digital information (2), and defined task setting (1), less routine task with less definite steps

Question 4 Complexity Score: 4 (Closed/Open: 2, Defined/Not defined axis: 3) Task description (domains): Simple task with more complex

instructions (2), less precise outcome (2), moderately complex technological applications (2), a few defined and more complex pieces of digital information (2), and less defined task setting (2), less routine task with less definite steps.

SRDC contacts

Norm Leckie: nleckie@srdc.org

Social Research and Demonstration Corporation: www.srdc.org