Computing

Student Understanding of the Computing Disciplines

! is Not a Rock Band

This presentation reports the initial findings of a multi-year study that is surveying major and non-major students’ understanding of the different computing disciplines.

about !

the team

Faith-Michael UzokaComputer Science & Information Systems

!

Randy ConnollyComputer Science & Information Systems

Marc SchroederComputer Science & Information Systems

Namrata Khemka-DolanComputer Science & Information Systems

Janet MillerCounselling

01 Background to this presentation

02 The context of our study

03 How we did our study

04 What our study found.

05 What it all means

06 Some problems and conclusions

Related Work

Methodology

Introduction Results

Discussion

Limitations and Conclusions

outline !

Introductionto the study

One of the most important achievements in computing education has been the recognition and elaboration of the five different computing disciplines.

introduction !

1 ComputerEngineering 2 Computer

Science 3 InformationSystems

4 InformationTechnology 5 Software

Engineering

The title of this paper refers to thethe fact that the computing disciplines should be understood to be quite unlike the distinct roles in a typical rock band.

introduction !

The computing disciplines have considerable overlap between them.

!

Despite this overlap, universities have to offer distinct computing degrees that typically do not blend curricula between the different disciplines.

For students, their initial understanding of the different computing disciplines may play a large role in how they decide which (if any) computing program to register in.

!

workrelated

This study is an extension of work by Courte and Bishop-Clark (C&BC) and then validated in a subsequent study by Battig and Shariq. Courte, J. and Bishop-Clark, C. 2009. Do students differentiate between computing disciplines?In Proceedings of the 40th ACM technical symposium on Computer science education (SIGCSE '09).

Battig, M. and Shariq, M. 2011. A Validation Study of Student Differentiation Between Computing Disciplines. In Information Systems Education Journal. 9, 5 (October 2011).

related work !

In their (C&BC) study, computing and non-computing students were askedto associate job task descriptions with the best disciplinary fit.

!

C&BC’s results suggest that students do not always have a clear understanding of disciplinary scopes (especially SE and IT).

!

Like the C&BC study, this study examinesstudent knowledge of the five different computing disciplines.

methodology !

Unlike the C&BC study, this study tried tocapture the overlap between the computing disciplines in the design of its survey.

!

In the C&BC study, students were given 15 task descriptions and for each task they had to indicate which of the five disciplines was the best fit for that task.

!

Designs hardware to implement communication systems

CE CS IS IT SE

⃝ ⃝ ⃝ ⃝ ⃝

The main drawback to the prior studies was that the students had to choose a single discipline for a task …

!

which does not capture the possibility of overlap between the disciplines.

To address that drawback, our study allowed the participants to choose how much each task fit with each of the five disciplines.

!

X

X

XX

X

Our questionnaire had demographic-related questions and then 31 discipline/task questions.

!

!

1 Designs hardware to implement communication systems2 Uses new theories to create cutting edge software3 Builds hardware devices such as iPods4 Is business oriented5 Focuses on large-scale systems development6 Integrates computer hardware and software7 Troubleshoots and designs practical technical applications8 Focuses on the theoretical aspects of technology9 Combines knowledge of business and technology10 Applies technology to solve practical problems11 Designs testing procedures for large-scale systems12 Selects computer systems to improve business processes13 Applies technical knowledge for product support14 Utilizes theory to research and design software solutions15 Manages large scale technological projects

Our first 15 questionswere the same as the earlier C&BC study:

!plus 15 new tasks added by the authors

16 Develops software systems that are maintainable, reliable, efficient, and satisfy customer requirements

17 Focuses on information, and views technology as a tool for generating, processing and distributing it

18 Utilizes sound engineering practices to create computer applications

19 Provides a support role, within an organization, to help others make the best use of its technical and information resources

20 Uses a wide range of foundational knowledge to adapt to new technologies and ideas

21 Uses technology to give a business a competitive advantage

22 Develops devices that have hardware and software in them

23 Applies mathematical and theoretical knowledge in order to compare and produce computational solutions and choose the best one

25 Understands both technology and business, but with a focus more on the technical side

26 Uses programming skills to create or modify business solutions

27 Develops or maintains web sites

28 Manages a team of software developers

29 Manages a company’s computing department

30 Evaluates and improves the usability (user experience) of computing systems

31 Works with an organization’s data assets

!… and an additional task that is not typically associated with the computing field

24Focuses exclusively on hardware design, including digital electronics, with little or no involvement in software design

The intent of the study was to find out if relatively-inexperienced students understood the tasks associated with different computing disciplines, prior to enrolment in computing courses/program.

!

timeline

Questionnaires were provided across ten sections of six introductory computing classes at Mount Royal University in the Fall 2012 semester.

Note that there are two computing programs at MRU: a computer science program and a blended IS/IT program.

SEPTOf 250 questionnaires that were distributed, 199 questionnaires were properly filled and coded for analysis.

JANRank ordering analysis along with standard statistical analysis.

!2012 20132013

MAY

re tus sl

results !

79%

response rate

25%

IS/IT

!

23%

CS

52%

Non-Major

results program of study

59%

17%

12%

year 2

year 1

year 3

9%

4%

other

year 4

results !year of study

72%Male

results !gender

28%Female

81%

11%

5%

< 2 Years

None

2-5 Years

3%

More than 5

results !computing experience

Rank order analysis was utilized to determine the students’ ranking of the disciplinary tasks relative to the five computing disciplines.

!

A further analysis was carried out to determine the levels of match between students’ task rankings and the disciplinary best fit.

results rank order analysis

!example rank ordering

Don’t Know % Level of Fit CE CS IS IT SE

15.6%

0 (No Answer) 19.6% 26.1% 25.1% 25.1% 24.1%

1 (No Fit) 2.0% 7.0% 9.5% 9.5% 20.6%

2 1.0% 10.6% 18.6% 15.1% 11.1%

3 7.5% 23.6% 21.1% 16.1% 11.1%

4 20.1% 23.1% 14.1% 20.1% 13.6%

5 (Best Fit) 49.7% 9.5% 11.6% 14.1% 19.6%

Mean 3.56 2.39 2.24 2.39 2.28

Median 4.00 3.00 2.00 3.00 2.00

Mode 5 0 0 0 0

Rank 1 3 5 2 4

Question #1 Designs hardware to implement communication systems

!discipline match distributions

Match Level CE CS IS IT SE

Very Accurate (5) 4 (100%) 3 (60%) 7 (78%) 5 (50%) 3 (38%)

Accurate (4) 0 (0%) 2 (40%) 1 (11%) 2 (20%) 2 (25%)

Ok (3) 0 (0%) 0 (0%) 0 (0%) 2 (20%) 2 (25%)

Fair (2) 0 (0%) 0 (0%) 1(11%) 1(10%) 1 (12%)

Poor (1) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%)

Total Tasks 4 5 9 10 8

Limitation: not every discipline had the same number of matching questions

Analyses were conducted to determineif there were any clear differences between the three groups of respondents, that is, between IS/IT, CS, and NM (non-major) students.

!

Responses to individual tasks were analyzed using one-way ANOVAs.

results differences between groups

Statistically significant differences (p < 0.05)

were found between program groups on 19 of the 31 questions …

!

Most of these differences occurred between the IS/IT and NM groups.

In general, the IS/IT students were more likely to rank the real-world tasks as better fits to the IS and IT disciplines …

!

A significant percentage of respondents either answered “Don’t Know” for a task or didn’t provide a response for a discipline on a task.

!results student uncertainty

!results student uncertainty

Don’t Know % Level of Fit CE CS IS IT SE

15.6%

0 (No Answer) 19.6% 26.1% 25.1% 25.1% 24.1%

1 (No Fit) 2.0% 7.0% 9.5% 9.5% 20.6%

2 1.0% 10.6% 18.6% 15.1% 11.1%

3 7.5% 23.6% 21.1% 16.1% 11.1%

4 20.1% 23.1% 14.1% 20.1% 13.6%

5 (Best Fit) 49.7% 9.5% 11.6% 14.1% 19.6%

Question #1 Designs hardware to implement communication systems

X

XX

A significant percentage of respondents either answered “Don’t Know” for a task or didn’t provide a response for a discipline on a task.

!results student uncertainty

Discipline Non-Responses Percentage

Don’t Know % CE CS IS IT SE

20.0% 31.4% 30.9% 31.2% 30.9% 31.6%

Non-major students answered DK more frequently, but based on our ANOVA cutoff this difference between groups was not significant.

!

Nonetheless, the finding that all our respondents were uncertain with one out over every five tasks is a significant finding.

results !

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

CE CS IS SE IT

IS/IT CS NM

discipline cluster scores

Since each disciple was associatedwith specific tasks, these scores were combined and averaged to create Discipline Cluster Scores.

!

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

CE CS IS SE IT

IS/IT CS NM

Due to the high uncertainty percentages discussed earlier, we also compared the cluster scores with the non-responses removed.

The data was subjected to a one-way analysis of variance that revealed only three significant differences between groups.

!

The IS/IT students made significantly better matches on three of the cluster scores compared to their non-major peers – for CS, IS and IT tasks.

Since Task 24 was not associated with any of the computer disciplines under study, all Task 24 answers were combined and averaged for each program group.

!results non-computing task performance

IS/IT CS Non-Majors

Task 24 Average Scores 2.08 2.10 1.89

When analyzed using a one-way analysis of variance, no significant differences between groups were discovered (in task 24 performance).

Results showed that Task 24 responses were significantly lower than those for the other tasks.

Discussion

By focusing on students who were taking an introductory computing course, we tried to provide insight into whether students who enroll in computing programs have a clear understanding of the disciplinary outcomes of the respective programs.

!discussion

Based on our rank-order analysisthe best matching occurred for tasks that related to CE, followed by IS (78%), CS (60%), IT (50%) and SE (38%).

!

These results are reasonably close to those encountered by the C&BC study.

Results showed statistically significant differences between the three program of study groups (IS/IT, CS, and NM) on 19 of the 31 items.

!

Non-major students tended to rank tasksas having a lower fit with each discipline.

!

IS/IT students tended to rank tasks as having a higher fit with each discipline.

At best this means the IS/IT students were more likely to be correct in their matchings.

!

At worse it might suggest a response bias where IS/IT students were more likely to assume that there is a higher fit for all tasks.

However, if there had been a response bias in place, we would have seen similar scores associated with Task 24, and this was actually not the case.

Like the earlier C&BC study, our results show that students are not always clear about the disciplinary “fit” of different computing tasks.

!

However, by allowing students to specify a degree of disciplinary fit, our study showed that by and large students are able to get their discipline matches close despite being inexperienced with computing.

This could be construed as a more encouraging result than that reported in C&BC.

!

Another important result was the lower likelihood that students would correctly identify the IS, SE, and IT tasks.

!also

This highlights how important it is for faculty in these fields to better articulate what these fields encompass, and to better communicate this information to prospective and current students alike.

Our data showed that the most confusion about what discipline a task belonged to were those tasks connected to real-world tasks.

!alsoUses programming skills to create or modify business solutions

Manages large scale technological projects

Develops or maintains web sites

Evaluates and improves the usability (user experience) of computing systems

Given that these larger projects often involve a variety of different skills and abilities, this uncertainty could even be construed as a positive sign.

There is a rain cloud in this sunny picture The very significant percentages of task and discipline uncertainty across all five sub-disciplines does indicate that all three student groups (IS/IT,CS,NM) have large gaps in their knowledge about the disciplines.

!however

!thusThis ignorance was likely masked in the C&BC approach since it did not provide an option for specifying uncertainty.

Designs hardware to implement communication systems

CE CS IS IT SE

⃝ ⃝ ⃝ ⃝ ⃝

versus

limitations

The main limitation of our study is similar to that of the C&BC study that inspired it: namely, if the task descriptions were too clear or too vague, then this would compromise the statistics and any conclusions drawn from them.

!limitations

As well, the five disciplines did not have the same number of tasks for which the discipline was the best fit.

The other key limitation of our study is that we did not have any CE or SE students in our study due to our university not having a CE or SE program.

!limitations

This limitation could conceivably be addressed in the future if data was obtained from universities that have a CE and SE programs.

conclusion

Over the last two decades, computing has undergone a reasonable level of differentiation into five sub-disciplines.

!conclusion

This has generated some ambiguity about the computing subdisciplines which resides in the minds of students, faculty, and even employers.

Other disciplines (such as engineering) also grapple with task/skill understanding by students and educators.

Our study adds to the literature on disciplinary task/skill identification via the enhancement of the C&BC instrument …

!conclusion

by identifying an additional 15 skills from the ACM sub-discipline descriptions and by allowing participants to specify a degree of disciplinary fit.

Our results show that students are not always clear about the disciplinary “fit” of different computing tasks …

!

But, when students provided an opinion about fit, major and non-major students alike were actually often close to correctly identifying the correct discipline.

This result is a new finding and a by-product of our revised survey design.

Our study also showed that IS/IT students had a better task understanding that those enrolled in the computer science program.

!finally