Assessing Student Learning in Engineering … filecontext and approach, and then outline the assessment methods used. The final part of the paper will The final part of the paper will

Assessing Student Learning in Engineering Entrepreneurship Education

1 | P a g e


FULL PAPER – DRAFT

Advances in Engineering Education

Elena Rodriguez-Falcon

Director of Enterprise Education and Faculty Director of Women in Engineering Mechanical Engineering, University of Sheffield

Mappin Street, Sheffield, S1 3JD +44 (0) 114 2227797

[email protected]

AND

Dr. Luke Pittaway William A. Freeman Distinguished Chair in Free Enterprise

Director, Center for Entrepreneurial Learning and Leadership College of Business Administration

Georgia Southern University P.O. Box 8154, Statesboro, GA, 30460

912-478-5321 [email protected]

AND

Dr. Alaster Yoxall

Principal Research Fellow Art and Design Research Centre

Sheffield Hallam University Arundel Street, Sheffield, S1 2NU, UK

+44(0) 114 2256754 [email protected]

Key words Engineering education Experiential learning Entrepreneurship education

Assessment practice Formative assessment

mailto:[email protected]




2 | P a g e

Abstract

This paper focuses on assessment practice in engineering entrepreneurship education. It examines the

value of formative assessment and explores the role that assessment methods have in student learning.

The paper features a nationally recognized and award winning course in entrepreneurship education in

the Faculty of Engineering at the University of Sheffield in the United Kingdom (UK) and uses the course

and data from the course to explain how assessment practices affect the student learning process. The

paper begins by outlining current issues and debates in the wider educational literature about

assessment practice in higher education (Pittaway and Cope, 2007a; Pittaway et al., 2009). It then

focuses on current research in entrepreneurship education exploring assessment practice (Bilen et al.,

2005; Penaluna and Penaluna, 2009; Carey and Matlay, 2010; 2011; Pittaway and Edwards, 2012) and

discusses the literature on entrepreneurship education in engineering programs (McMullen and Gillin,

1998; Chau, 2005; Collet and Wyatt, 2005; Dohn et al. 2005; Hamilton et al., 2005). The study then

provides a case study from the University of Sheffield. The paper describes the course and its context; it

outlines the assessment practices that have been used and explores how these practices have changed

over time. A variety of data are used to explain how different types of formative assessment impact on

student learning. The paper concludes by outlining the value of considering carefully assessment

methods, as they relate to the learning outcomes desired within engineering education.

Introduction

Assessment practice in entrepreneurship education has recently become the focus for studies in

the subject (Carey and Matlay, 2010; 2011; Penaluna and Penaluna, 2009; Pittaway et al., 2009;

Pittaway and Edwards, 2012). This paper seeks to build on this understanding of assessment practice by

outlining an innovative form of entrepreneurship education within engineering and exploring how

student assessment is used by the course to enable effective learning. The paper focuses principally on


3 | P a g e

student assessment practices and puts forward the case for expanding formative assessment in

engineering education. Its contribution is to showcase an award winning course and to explain how

certain methods of student assessment enhance student learning, particularly where they are formative

in nature. The paper will begin by explaining the current status of knowledge on assessment practice in

entrepreneurship education. Within this part we will explain the key issues and debates in the

assessment literature and explain how assessment has typically been handled within the

entrepreneurship education domain. We will conclude the section by exploring entrepreneurship

education in engineering and arguing for more research on assessment practice. The paper will then

progress to explain the University of Sheffield case study. Here we will introduce the course, explain the

context and approach, and then outline the assessment methods used. The final part of the paper will

investigate how the assessment methods used have enhanced student learning and finally, we will

conclude the paper by explaining why it is important in entrepreneurship education to be both

innovative in pedagogic design and in the assessment methods used.

Assessment Practice in Entrepreneurship Education

In research on entrepreneurship education the investigation of assessment practice has

generally been overlooked (Pittaway and Cope, 2007a; Pittaway et al., 2009). This oversight is

unfortunate given the emphasis placed on assessment practices by government agencies and

accreditors. Indeed, it could be argued that the assessment of students’ learning is part of the key

function of a university and that universities main role in society is to produce an officially documented

judgment of students’ work, in addition to, encouraging student learning. The scholarship of

assessment defined as “…sophisticated thinking about assessment” (Banta, 2002, p.242), comes in

several forms, including: institutional assessment; teacher assessment and student assessment (Banta,

2002). While this paper focuses on student assessment the other two forms are valid objects of concern


4 | P a g e

for researchers. Student assessment has particularly grown in importance for educational researchers

as the pressure from accrediting agencies, who seek to measure institutional performance, has

increased (Banta, 1999). Despite these pressures student assessment should be considered as being

integral to the educational process and it is obviously important in helping educators judge whether

intended educational outcomes are met (Martell, 2007). Concerns in educational research about

student assessment practice can be defined into two groups “political concerns” and “educational

concerns” (Pittaway et al., 2009). Political concerns focus on the role of external accreditation systems

on the independence of educators and typically question a system that has driven outcome-based,

credit-based and modular forms of education enhancing “credentialization” and summative assessment

above other forms (Ecclestone and Swann, 1999; Cassidy, 2006). In contrast educational concerns have

tended to focus on how assessment itself impacts on learning and have sought to understand how

assessment design can assist student learning (Dunne et al., 1997). Although, within the engineering

education domain, debate about the impact of accreditation processes on learning design and

assessment are important, this paper focuses on the latter area of interest and seeks to understand how

assessment design can enhance and support student learning.

Within the subject of student assessment when focused on educational concerns there are

several critical areas of discussion that regularly emerge in the literature (Ecclestone and Swann, 1999).

These include discussion about the correct balance between formative and summative forms of

assessment and feedback; the coherence of assessment policy across departments and institutions; the

role of second marking; the role and design of assessment criteria; the productivity of assessment

strategies; and the impact of formative assessment on subsequent student behavior (Ecclestone and

Swann, 1999; Pittaway et al., 2009). In addition, other areas that seem important in the literature have

included: the role of self and peer assessment (Falchikov and Goldfinch, 2000); the challenges of

assessing innovative learning designs (Gijbels et al., 2005); and, finally some consideration of the tension


5 | P a g e

between norm-referenced assessment and criterion-based assessment (Broadfoot, 1996; Ecclestone,

1996). Ultimately, educational research on assessment practice has concluded, based on these

discussions and subsequent surveys of assessment, that assessment practice varies considerably and

that in most subjects there is considerable diversity about what is assessed, how it is assessed, when

and where it is assessed and who is doing the assessing (Topping et al. 2000). Such tensions in

educational research between the “political concerns’ of assessment and the shear diversity of practice

has led to some uneasiness about the state of assessment practice in higher education (Boud, 1995;

Ramsden, 1992; Brown et al. 1997; Race, 2003). The problems identified are not insignificant, such as, a

common mismatch between learning outcomes articulated, learning tasks and assessment methods

(Rust, 2002) and assessment processes that encourage ‘surface learning’ rather than ‘deeper learning’

(Yorke, 1998). Some conclude that most educators’ assessment practices do not meet the objectives of

good assessment (valid, reliable and transparent) because they fail to measure what they claim to

measure, they apply personal judgments that may not be reliable and fail to be transparent about their

expectations with students (Race, 2003).

Given the nature and depth of concerns expressed by educational researchers about the

limitations of current assessment practices in higher education all subject disciplines need to consider

carefully assessment practice. The subject of assessment practice has consequently become a feature of

discussion in entrepreneurship education. In this disciplinary area several domain specific

considerations have arisen. First, much of the assessment discussion in educational research has

assumed that educators know what educational outcomes they seek. When applied to

entrepreneurship education this assumption is problematic. Entrepreneurship education has some

difficulty here in defining what ‘enterprise’ and ‘entrepreneurship’ actually are, and when applied to

educational practice this can lead to multiple forms of educational approach (Gibb, 2002; Pittaway and

Cope, 2007a). Secondly, there are also debates about ‘how best to learn it’ and so entrepreneurship


6 | P a g e

educators disagree over what students should learn and how best they can learn it (Pittaway et al.,

2009). Further issues arise when considering ‘conventional’ assessment practices. There are many

forms of conventional assessment practice in higher education (e.g. exams and essays) and these are

often adopted without thought, when entrepreneurship education requires domain specific assessment

techniques (e.g. business plans and elevator pitches) that are aligned to the educational designs

employed and the ‘community of practice’ of entrepreneurs (Pittaway and Edwards, 2012). Some of the

subsequent research in entrepreneurship education on assessment practice explored what educators

considered to be innovative (Pittaway et al., 2009) and explored how lessons from assessing creativity in

design subjects could be employed within entrepreneurship education (Penaluna and Penaluna, 2009;

Carey and Matlay, 2010). The conclusion of this work has been that assessment practice needs to

become more innovative but also that there is a need to explore what entrepreneurship educators

actually “do” with regard to assessment practice (Pittaway et al., 2009; Penaluna and Penaluna, 2009).

In addition, the research concluded that observations of practice would need to be sophisticated

recognizing the different forms of entrepreneurship education: ‘about’, ‘for’, ‘through’ and ‘embedded’

(Gibb, 2002; Pittaway and Cope, 2007a); and recognize variation in: the thematic focus of the course;

learning outcomes desired; disciplinary differences; and, differences in the educational philosophy of

the educator (Pittaway and Edwards, 2012). A subsequent study further developed the research on the

subject by surveying entrepreneurship courses and analyzing the current status of assessment practices

used (Pittaway and Edwards, 2012). This research made several conclusions relevant to this paper.

Initially, it observed that educational practice in entrepreneurship education remains somewhat

‘traditional’ in that it has typically focused on “about” forms of educational practice and that business

schools continue to dominate educational practice in the subject. Further, it a reported a lack of

innovative assessment practice and a domination of summative methods in ‘about’ forms of educational

design while more evidence of formative methods in other forms (Pittaway and Edwards, 2012).


7 | P a g e

Likewise, it was evident that the engagement of ‘others’ in the assessment process beyond the

professor leading the course was much lower than anticipated. Concluding the research authors

highlight a need to examine in more depth particular forms of assessment practice, that research

needed to be undertaken on the role of the assessor and other stakeholders and finally, “that there is a

need for further research that explores assessment practice in entrepreneurship education in disciplines

outside of business schools” (Pittaway and Edwards, 2012, p. 794). Based on the above status of

research in the subject of assessment practice in entrepreneurship education we aim to add a

contribution to the discussion by specifically exploring formative forms of assessment practice and

within the context of an engineering entrepreneurship education course. Next we will expand our

discussion to explore the nature of formative assessment and explore briefly prior research on

entrepreneurship education in engineering.

Formative assessment differs from summative assessment because it seeks to provide a

diagnostic test of learning rather than undertake a test of the educational outcomes (Ecclestone and

Swann, 1999). As such, it includes both formal and informal assessment methods which are used during

the learning process to change, monitor or adapt the teaching and learning process in order to improve

the quality of student learning. Although often loosely defined, assessment methods that are

considered to be ‘formative’ include qualitative feedback that enable students to understand the status

of their development and provide some direction on how to improve. In this sense, it is not the method

itself that is considered formative but how the information from the assessment is used (Ainsworth and

Viegut, 2006). In formative assessment data are used to help improve learning whereas in summative

assessment data are used to assess what learning or knowledge has been retained. Educational

researchers have carried out a significant number of studies on the value of formative assessment for

student learning, for example, a meta-analysis carried out on the subject in 1998 (Black and William,

1998) reviewed over 250 studies and there is considered to be a good base of data supporting its impact


8 | P a g e

on learning (Crooks, 1988). Within the entrepreneurship education field formative assessment has not

been considered in detail as studies have tended towards focusing on pedagogic design rather than

assessment practice (Pittaway and Cope, 2007a). Research in entrepreneurship education in

engineering has likewise tended towards a focus on pedagogy, on the design of programs, rather than

on assessment practices used (Boggs et al., 2004; Chau, 2005; Dohn et al., 2005; Nichols and Armstrong,

2003). Studies have identified the value of problem-based learning (Wee, 2004), have illustrated how

entrepreneurship education can positively support the recruitment and retention of students in

engineering subjects (Peterman and Kennedy, 2003) and have focused on educational outcomes for

students (Ohland et al., 2004; Nichols and Armstrong, 2003). This lack of attention on assessment

practice, and on formative assessment practice in particular, is unfortunate given the value placed on

the subject by educational researchers. One paper that did explore assessment in engineering

entrepreneurship education is written by Bilen et al. (2005) but it is principally focused on the program

and uses assessment data to explore the impact of the program rather than focusing on assessment

practice as a topic in its own right. In the parts of the paper that follow, therefore, we seek to

contribute to enhancing discussion in this area by presenting a case study of an innovative engineering

entrepreneurship education program and focusing principally on how formative assessment has added

value to the student learning experience. In the first part we introduce the program and its design and

in the second part we discuss the assessment practices and how they have influenced the learning

outcomes for the students.

University of Sheffield Case Study

The Technology Strategy and Business Planning, award winning module at the University of

Sheffield in the UK, is designed to introduce the ‘concept, strategy, and techniques behind a business

plan based on the exploitation or development of identified technological opportunities’. The course is


9 | P a g e

taken by final year students from across the engineering school, either as an elective or a mandatory

requirement, depending on their degree specialism. Since its inception in 2002, the number of students

enrolled has gradually increased from 13 to 157 in 2011/12. Essential to the course is the concept that

the students work in teams to find solutions to a real problem provided by a real customer. Typically,

the customer will be a member or members of the community (e.g. children with disabilities or

terminally ill people). The challenge is for the student teams to identify a technical solution to the

customer problem, which is defined as making their lives easier or better. Once the product concept is

identified teams must also develop business plans to explain how the concept may be exploited. The

main aim of the course is to give students an opportunity to solve a real problem while combining

engineering knowhow within the commercial constraints that would need to be met to enable business

success from their technical solutions. Students also have the opportunity to develop presentation skills

via the assessment techniques employed. The students are expected to: develop the analytical and

critical skills of final year engineering students; appreciate the importance of having a structured

business plan in the development of a commercial venture; develop a business plan for the exploitation

of a technical solution; and explore the components (resources, financial, marketing, IPR, etc…) of the

planning process for a real product.

In seeking to achieve the aforementioned learning objectives student teams are asked to

develop solutions and an accompanying business plan for real commercial problems. In recent years,

project briefs have focused on product concepts that are designed to improve the lives of real

individuals or groups living locally. For example, in 2007, the project ‘client’ was a 6-year old boy with

cerebral palsy who encountered a number of practical difficulties in his day-to-day life (see Image 1 and

2). Students were given a very open project brief of ‘making life easier’ for Kieron, and they developed

solutions ranging from writing aids to novel clothing. Each year, a different ‘client’ is selected, and this

individual/s and their families work closely with the student teams during this semester-long course.


10 | P a g e

The close involvement of these real ‘clients’, with students and with the potential for making a positive

contribution to their quality of life, often act as strong motivators for the students involved and can lead

to creative engineering solutions and viable commercially attractive products. Over several years clients

have included: Sheffield Children’s Hospital, St Luke’s Hospice and the Woolley Wood School.

[Insert Image 1 and Image 2]

During the process, as students develop product concepts and explore their commercial potential, they

are supported by external contributors including: business angels, bankers, marketers, business

advisors, manufacturers and others. On average, around 15 contributors are involved in the course

every year (see image 3).

[Insert Image 3]

External speakers, judges and sponsors provide crucial contributions to the course, their expertise, their

advice and often their financial resources make of the course a distinctive learning opportunity within

the engineering program at the University of Sheffield. Feedback from contributors has included:

"The business planning module has demonstrated its value by the approach the students take to investigating and questioning the business process. As an external supplier of business development to the university's students, I have found a refreshing openness when presenting business related topics to groups. Their understanding of the principles of business give them an edge that employers are crying out for." Richard Campos, MD RC Business Support Ltd

"The University of Sheffield Department of Mechanical Engineering's Business Planning Module is in my view a shining example of how to give students an awareness of the way that academic knowledge needs to be combined with practical application and a wider business knowledge in order to achieve success in the demanding 21st century marketplace. The course also deserves credit for the way in which it promotes and encourages entrepreneurialism whilst also benefitting the worthy causes around which the annual projects are based". Peter Crawford- Taylor&Emmet

As part of the assessment process and at the end of the course, teams pitch their ideas to an invited

audience and judges from a mixed background (technical, commercial and legal) and prizes are awarded

to the best presentations/solutions (see images 4 and 5).


11 | P a g e

[Insert Image 4 and 5]

Some of the students’ ideas have been further developed into prototypes and functional products. One

successful example is a walking frame for children with brittle bones, which is now being used by the

Sheffield Children’s Hospital. Another example is the ‘scribbler’, which is a writing aid device that was

prototyped for Kieron Norton in 2007 (see image 6 and 7)

[Insert Image 6 and Image 7]

Students invariably find the course to be a distinct form of learning from which they gain much value

and typical feedback includes many comments such as those highlighted next.

“I enjoyed the course very much and I am glad I decided to opt for this module. It gave me a great understanding about business planning but above all, helping others to lead a better life with our innovations and designs. This is my best module so far.” Student 2008-Sheffield Children’s Hospital Project

“At the start of this module, this was just another paper to score marks, but when these children came to the university, that’s when I realised that they needed all the help they could get. Their disabilities are not their fault and God has given us enough strength so I feel it’s our duty to make their life better. In simple terms I will do anything to see a smile on their faces.” Student 2010- Woolley Wood School Project

Clients likewise have valued the course and its approach. For example, a member of staff of the local

hospice who, after the module had finished, contacted the department with sad news of the death of

the patient who had contributed to the course, who for the purpose of this will be named George and

his wife Esther.

“Just to let you know that George died last Wednesday. Unfortunately, over the last few weeks George had become increasingly muddled and Esther was unable to manage him at home. After a long stay in St Lukes, George died in a nursing home. It was George’s funeral yesterday and Esther wanted everyone to know about George taking part in the University project and how much this meant to both of them particularly coming to the University and answering the student’s questions. Esther could not quite believe they had wanted to know about her as well! Esther was happy for me to let you know about George and I wanted you to know how much they both valued taking part in the project and the memory you left Esther of that time.”

A quote from a teacher in Woolley Wood School summarizes their view of the 2010 projects:


12 | P a g e

“The Mechanical Engineering students did a fantastic job in identifying pupil need. All the teams tackled the issues and challenges facing pupils and their families with a genuine keenness and sensitivity. We were extremely impressed with the creative thinking that went into developing the ideas. Every team's work is worthy of further development and the creation of prototypes, in order to trial and further enhance their marketing potential. We feel extremely proud that Woolley Wood has had this fantastic opportunity and look forward to future collaboration with the University and their students.”

The combination of the business-orientated product development with a real and visible social impact

has also created significant media interest. Such media coverage is relatively uncommon in curricular

engineering education and is a valuable asset to the overall program profile and recruitment of students

and future clients. Examples of this include articles in the Yorkshire Post, The Sheffield Star, the BBC

Website and BBC Look Northi (see image 8).

[Insert Image 8]

The course has been recognized as an exemplary example of engineering entrepreneurship education in

the UK and has won a number of national awards including from the HEA Engineering Subject Centre

and the Royal Academy of Engineering and has also been chosen as an exemplar for a white paper

sponsored by the by Bernard M. Gordon‐MIT Engineering Leadership Program. In the next section we

explore in more detail how the program engages in assessment practice and explore how this

contributes to the learning outcomes for the students.

Assessment Practice and Student Learning Outcomes

The learning methodologies used aim to enable students to work in multidisciplinary and

multicultural teams, conduct self-directive learning and inquiry based-learning. The assessment methods

applied are also used as learning activities through the application of formative assessment, as defined

earlier in this paper. The figure below shows the learning inputs, the assessment methods and the

formative learning opportunities that the students experience.


13 | P a g e

[Insert Figure 1]

The problem is initially defined by the customer and developed by the students with support

from engineering and design experts. The first real opportunity, however, that students have to further

develop and consolidate their research question and receive formative feedback is at an interview

session with the clients. This session has to be carefully set up to allow for an honest but caring learning

opportunity. Issues of confidentiality and ethics have to also be established in advance of the session.

As mentioned earlier, external experts provide a very valuable input to the students’ learning

experience. Each expert delivers a session which has a direct impact to the development of the business

plan. The fact that these practitioners provide a realistic account of what makes a successful business

plan allows students to translate this into the assessment, which is ultimately their business plan. The

final external input is provided at the poster presentation when judges ask questions to each team in

order to ‘assess’ their knowledge. The dialogue, however, between teams and judges often enable the

judges to provide their expertise and enrich what in fact is an official assessment day (see Image 9).

[Insert Image 9]

Students in teams of six or seven will tackle the problem given to them and are expected to produce the

following outputs throughout the course.

1. Initial solution and business model- Group (10%). Three pages long not including cover sheet. One

page covers the engineering solution (sketches annotated were ok at this stage), the second page

explains what needs the product met, target market and foreseen competitive advantages, and the

last page describes the business model students intended to adopt. Formative/qualitative feedback

is provided for this piece of assessment.

2. Business plan report –Group (40%). The business plan is expected to contain a sound structure, and

be written in a business-like manner. It includes an executive summary and a cash flow projection


14 | P a g e

for the first year of business. Every report is expected to be unique, as every team should have

approached the project differently and the length of the report is left up to each team. It is

recommended, however, that the actual plan does not exceed 15 pages, with no limit on the

appendices. The authors have to maintain a clear and straightforward written style, which enables

readers to gain an easy and quick understanding of the report. Students are provided with

marking/grading criteria and summative feedback is given at the end of the course.

3. Poster presentation and elevator pitch- Group (40%). Each team produces an A1 poster with the

information they considered to be the most relevant or eye catching in order to obtain support from

stakeholders, bankers, staff or whomever the plan was intended for. At the end of the project the

posters are presented to a panel of academics, business people and the customer who assess the

quality of the poster and its content. Team members are questioned by the panel. It is expected

that students would be able to answer questions and comment upon their business plan and

technical solution (see Image 10). A representative member of the team delivers an elevator pitch

not longer than 80 seconds. Again marking/grading criteria are provided to the students

[Insert Image 10]

4. 10% Discretionary points: points are only given for exceptional contributions (summative and

formative feedback is provided). Discretionary points have been awarded only when the team

provided work not stated as part of the project and which demonstrated a clear level of innovation

and creativity. For example, prototypes as seen in Images 11 and 12.

[Insert Image 11 and 12]

5. The individual effort from each team member is assessed though peer review. The course uses

WebPA to provide summative peer assessment. WebPA is an online peer moderated marking

system. Each student in the group grades their team members and own performance. The grades


15 | P a g e

given are then used to weight the individual’s grade which is associated with the overall group

grade.

Students are asked to consider what type of ‘human resource’ they will need to take their project to

successful completion and then to group themselves in teams of seven to ten. Students then have 11

weeks to complete the project.

Formative learning via assessment is an essential part of the course. Each activity is intended to

provide opportunities for the students to learn, whether via the feedback provided or the assessment

methods used. The initial interview with the customer provides students with a starting point, which is

translated into initial ideas for their technical solution and business model. These initial ideas have a

low weight in terms of the final grade (10%) because the actual purpose of the assessment is to provide

formative feedback. Business people, engineers and the customer give feedback to teams, which

students incorporate into their project. Throughout the course, students have the opportunity to meet

with assigned mentors who are able to provide feedback. The poster presentation provides a hugely

valuable formative learning experience, despite being a key aspect of the assessment technique used.

Judges and attendees, including peers, engage in priceless conversations about the teams’ solutions and

ideas. It is not uncommon to hear students say that “this (poster presentation) did not feel like an exam

at all, it was fun and informative”. The final piece of feedback that students receive is for their actual

business plan. This is at the end of the course, which does not inform further developments in the great

majority and is in the main summative in nature. There have been a few cases, however, where

students have decided to start their own companies and used the business plan as their starting point.

In these cases summative assessment has provide to be a formative spring-board into actual venture

creation. The most recent success is the ‘Talking Glove’ that works by converting simple finger

movements into gestures which control speech through a synthesised voice. The user wears the glove


16 | P a g e

and moves their fingers in order to articulate words or commands. The team of students who invented

the glove came up with the idea in 2010 when they met one of the Woolley Wood pupils who had a

speech impairment. After winning the first prize at the poster presentation, the students launched

Ecofriendly Technologies, developing this assistive technology further (see Image 13).

[Insert Image 13]

Discussion

The course has a series of formal assessment milestones, such as the elevator pitch, business

plan and poster. These elements were designed to provide methods that would aid the business

understanding of the students whilst providing mechanisms for formal comparative assessment within

standard HEI practice. The assessment design has formative assessment mechanisms aligned with

recommended practice for the ‘for’ form of entrepreneurship education (Pittaway and Edwards, 2012).

Notable aspects include: the involvement of external assessors and mentors from the entrepreneurial

community; client/community engagement and feedback; peer assessment; and, the simulation of

practice associated with entrepreneurial contexts, such as, elevator pitches, poster presentations and

business plans (Pittaway et al., 2009). All of these components are formative, as defined by Ecclestone

and Swann (1999), and engage in qualitative feedback mechanisms that progressively enable students to

move through the venture creation process, from initial idea, to developed prototype, and finally, to a

fully considered business proposition. Even the summative components of the assessment technique

had proven to have formative outcomes in particular circumstances. The question that needs to be

asked and explored though, is how did these different components impact on the student learning

process and the learning outcomes acquired?

In 2012 a questionnaire was undertaken to begin to answer this question, to explore the

assessment practices and how they impacted on student learning outcomes. The students were asked


17 | P a g e

to give their feedback on the various assessment methods used in the course along with their more

general thoughts surrounding its design and implementation. Approximately, a third of the cohort

responded and the results are shown in Table1 below.

[Insert Table 1]

In examining the results, items that scored highly (defined as 'Life Changing') were; disability awareness,

meeting the clients, researching the issues surrounding the project and working to solve them

collaboratively (shown diagrammatically below, Figure 2).

[Insert Figure 2]

These elements indicate that the move towards a more socially engaged project has created a wider and

deeper engagement for the students with the elements on disability and background research scoring

strongly in terms of ‘life changing’ learning outcomes. Here the students can be seen to be gaining in

multiple ways via formative assessment and pedagogic design (Emrf and Yoxall, 2010). The pedagogic

design allowing for ‘clients’ and unique clients that have significant social needs motivates the students

to utilize their engineering skills to find practical solutions to ‘real-life’ problems. The challenge

immediately embeds the course in the engineering discipline providing a unique challenge engineering

students are best prepared to consider (Handscombe et al., 2007). The real-time qualitative feedback

the students receive via the assessment, from the clients and from mentors, ensures the initial ideas

pursued are adapted both conceptually and commercially in an iterative way somewhat simulating the

venture creation process (Pittaway and Cope, 2007b). For a course to be rated a ‘life-changing’ learning

experience is also significant, few would consider even the best courses to be such, and so it is

important to recognize that the ‘doing good’ aspect of the design is a central feature in what appears to

be transformative learning (Cope, 2005), formative assessment enables, but it is perhaps the emotional

aspects that play the greater role (Cope, 2011) and to a certain extent mirror entrepreneurial learning


18 | P a g e

(Pittaway and Cope, 2007b). Similar course design elsewhere in engineering has been seen to provide

such a platform for enhancing student recruitment, engagement and retention in the subject (Peterman

and Kennedy, 2003). From an assessment point of view the poster presentation scored highly compared

to the writing of the business plan and the elevator pitch. Poster presentations were seen to be

significant learning events in their own right despite being a critical aspect of the assessment process.

Once again here it is evident that the need to articulate and defend a visual presentation with ‘members

of the target community’ of entrepreneurs, investors and business professionals is in itself a formative

learning experience valued by the students. There are perhaps a number of elements at play. Certainly,

‘vicarious learning’ is occurring, the knowledge and participation of experienced professionals, is

providing a key counter-point to the students’ ideas about possible products and commercialization

strategies and the students are learning through sharing experience with the community of practice

(Taylor and Thorpe, 2004). Also it might be argued that a poster session provides a ‘cumulative

experience’, a point at which all the efforts expended on the project come to a close and so the students

have a heightened sense of learning bought about by the intensity of the event (Pittaway and Cope,

2007b). Such intensity can ensure increased retention of the learning at the time.

The written business plan and the elevator pitch were scored less highly than the poster session

as contributors to the learning experience. Interestingly, despite being highly formative in nature, the

elevator pitch scores badly as an effective learning opportunity, which may reflect on its inclusion to

simulate the entrepreneurial aspects of the course. The pitch assessment was first implemented to

deliver a business context and to provide the judges with information prior to the poster session. As

engineering students they may have been taken outside of their comfort zone, as the elevator pitch is

not something they have done before. Or perhaps, there are process improvements required in the

course design and the students’ preparation for conducting elevator pitches. Either way, this does raise

the question with respect to the use and engagement of innovative methods of assessment, in that care


19 | P a g e

must be taken as to the reasons for inclusion and some thought given to how the methods used link to

the broader student learning experience, both within the course and within the discipline (Handscombe

et al., 2007). Lack of familiarity with an assessment technique may require teachers to spend more time

than normal outlining to students how to engage with it and how to do well, since student’s grades are

often important to them, proper preparation when engaging in innovative assessment seems essential.

For example, a formative assessment, even when designed well, provides summative outcomes and can

be treated as such by students.

Intriguingly, the move to embed the course in a social learning context, via the use of compelling

clients and engagement with the community of practice, has improved the overall pass rate; even while

student numbers have increased and learning outcomes have been broadened. It is difficult to

speculate over the exact reasons but enhanced emotive engagement with the problem must play a part,

students are more engaged and more motivated to solve a real-life problem that can change a person’s

life forever. Alongside this, enhanced formative assessment has played a crucial role in allowing

students to grow and develop their approach as they gain qualitative feedback from the client, from

their mentors and from the community of practice. Assessment that seems to enhance learning within

this context seems to be principally social and embedded both in the context of the problem at hand

and with the professional context of commercial and entrepreneurial endeavor. It seems evident that

this approach supports recent arguments to enhance the engagement of ‘others’ in the assessment

approach (Pittaway and Edwards, 2012). Truly effective formative assessment, it seems, must also be

social and drawing from the community of practice and so entrepreneurship educators can perhaps

enhance assessment practice by looking to self, peer and external assessment more regularly (Pittaway

et al., 2009).


20 | P a g e

Conclusions

In this paper we have explored how assessment practice enhances student learning within the

context of entrepreneurship education in engineering. In particular the paper focused on the value of

formative assessment. The paper first explores current debates in assessment practice and shows that

educational researchers are concerned with many aspects of assessment including many practical

concerns such as the nature of the assessment methods used, the value of qualitative feedback in the

assessment process and considerations over ‘who’ is engaged with assessment (Ecclestone and Swann,

1999). We then argued that research into assessment practice has been somewhat neglected in

entrepreneurship education where the focus has too often been descriptions of courses and course

designs and their efficacy (Pittaway and Edwards, 2012). Likewise our review of entrepreneurship

education in engineering also illustrated a desire to focus on the value of course designs and the impact

of educational practice on student learning rather than on the role of assessment in the learning process

(Boggs et al., 2004; Chau, 2005; Dohn, 2005; Nichols and Armstrong, 2003). From this we conclude that

there is much scope for expanding research in engineering entrepreneurship education into all aspects

of assessment practice. There is, for example, an interesting question about the role of accreditation

agencies in engineering and how their expectations and ‘credentialization’ of learning impact on the

learning process in engineering and aid or prevent the more innovative forms of learning and

assessment as those proposed by entrepreneurship educators. In addition, all aspects associated with

the practical process of assessing student learning seem under-researched. What role does the balance

between summative and formative assessment play in a more general sense? Who should be engaged

and how should educators expand the involvement of the ‘community of practice’ and expand the use

of self and peer assessment? What role do key educational debates play, such as, the role of second

marking; the role and design of assessment criteria; and, the productivity of assessment strategies?

There are certainly many questions to consider within the assessment research domain and we argue


21 | P a g e

that educators in entrepreneurship education, more broadly, and educators in engineering

entrepreneurship education, more specifically, should be considering these issues more.

Our contribution in this paper has been to explore a nationally recognized and award winning

engineering entrepreneurship education program in the UK and specifically consider how formative

assessment has enhanced its success and value in terms of student learning. We can conclude from this

that formative assessment has been critical to the success of the program in a number of key ways.

First, it has enabled ‘milestones’ to be set that allow for regular qualitative feedback loops that draw in

feedback from the client and the community of practice, enabling students to learn as they go, to

improve and enhance the development of their solutions in a way that simulates entrepreneurial

endeavor (Pittaway and Cope, 2007b). We have also shown the important role of external assessors and

peer assessment in the process and how these aspects enhance the value of the formative assessment

carried out (Falchikov and Goldfinch, 2000). The most significant contribution the paper makes is to

show how ‘life-changing’ learning can be achieved by using assessment that presents real-life problems

that have significant implications for the lives of others. The heightened engagement that students gain

from both, interaction with the clients and with the emotional exposure experienced, accelerates the

learning process and to some extent simulates entrepreneurial learning (Pittaway et al., 2011).

Formative assessment, receiving ongoing feedback from the client, from mentors and from

entrepreneurs is a critical aspect behind the success of the program. Without that engagement and

ongoing qualitative feedback on the development of the ideas, designs and commercialization strategies

the course would be devoid of value and so formative assessment is shown here to be an instrumental

component of successful engineering entrepreneurship education.


22 | P a g e

References Ainsworth, L. and Viegut, D. (2006). Common formative assessments. Thousand Oaks, CA: Corwin Press.

Banta, T. W. (1999). Assessment essentials: planning, implementing and improving assessment in Higher Education, San Francisco, CA: Jossey-Bass.

Banta, T. W. (2002). Building a scholarship of assessment, San Francisco, CA: Jossey-Bass/John Wiley & Sons.

Bilen, S., Kisenwether, E., Rzasa, S. and Wise, J. (2005). Developing and assessing students’ entrepreneurial skills and mind-set, Journal of Engineering Education, 94(2): 233-243.

Black, P. and Wiliam, D. (2003). In praise of educational research: formative assessment, British Educational Research Journal, 29(5): 623–637.

Boggs, J.W., Williams, R.C., Mattila, K.G., Kennedy, W.A. and Dewey, G.R. (2004). The pavement enterprise at Michigan Technological University, Journal of Professional Issues in Engineering Education and Practice, 130(3), 197-204.

Boud, D. (1995). Assessment and learning: contradictory or complementary? In Knight, P. (Ed) Assessment for learning in Higher Education, London: Kogan Page.

Broadfoot, P. (1996). Education, assessment and society: a sociological analysis, Buckingham: Open University Press.

Brown, G., Bull, J. and Pendlebury, M. (1997). Assessing student learning, London: Routledge.

Carey, C. and Matlay, H. (2010). Creative disciplines education: a model for assessing ideas in entrepeneurship education?’, Education and Training, 52(8/9): 694-709.

Carey, C. and Matlay, H. (2011). Emergent issues in enterprise education: the educator’s perspective, Industry and Higher Education, 25(6): 1-10.

Cassidy, S. (2006). Developing employability skills: peer assessment in Higher Education, Education & Training, 48(7): 508-510.

Chau, K.W. (2005). Problem-based learning approach in accomplishing innovation and entrepreneurship of civil engineering undergraduates, International Journal of Engineering Education, 21(2): 228-232.

Collet, C. and Wyatt, D. (2005). Bioneering: teaching biotechnology entrepreneurship at the undergraduate level, Education + Training, 47(6): 408-421.

Cope, J. (2005). Toward a dynamic learning perspective of entrepreneurship, Entrepreneurship: Theory and Practice, 29(3): 373-398.

Cope, J. (2011). Entrepreneurial learning from failure: An interpretative phenomenological analysis, Journal of Business Venturing, 26(6): 604–23.

Crooks, T. (2001). The validity of formative assessments, British Educational Research Association Annual Conference, University of Leeds, September 13–15, 2001.

Dohn, J., Pepper, D.W. and Sandgren, E. (2005). Creating innovative curricula: developing new programs with new paradigms, International Journal of Engineering Education, 21(2): 233-238.


23 | P a g e

Dunne, E., Bennett, N., and Carré, C. (1997). Higher education: core skills in a learning society, Journal of Education Policy, 12(6): 511-525.

Ecclestone, K. (1996). How to assess the vocational curriculum, London: Kogan Page.

Ecclestone, K. and Swann, J. (1999). Litigation and learning: tensions in improving university lecturers’ assessment practice, Assessment in Education, 6(3): 377-389.

(Emrf and Yoxall, 2010).

Falchikov, N. and Goldfinch, J., (2000). Student peer assessment in Higher Education: a meta-analysis comparing peer and teacher marks, Review of Educational Research, 70(3): 287-322.

Gibb, A., (2002), In pursuit of a new enterprise and entrepreneurship paradigm for learning: creative destruction, new values, new ways of doing things and new combinations of knowledge, International Journal of Management Reviews, 4(3): 213-232.

Gijbels, D., Dochy, F., Van de Bossche, P. and Segers, M., (2005). Effects of problem-based learning: a meta-analysis from the angle of assessment, Review of Educational Research, 75(1): 27-61.

Hamilton, C., Crawford, G.P. and Suuberg, E.M. (2005). A Technology-Based Entrepreneurship Course, International Journal of Engineering Education, 21(2): 239-256.

Handscombe, R., Kothari, S., Rodriguez-Falcon, E. and Patterso, E. (2007). Embedding Enterprise in Science and Engineering Departments, NCGE Working Paper, No. 025/2007: National Council for Graduate Entrepreneurship.

Martell, K., (2007). Assessing student learning: are Business Schools making the grade, Journal of Education for Business, 82(4): 189-196.

Mcmullan, W.Ed. and Gillin, L.M. (1998). Developing Technological Start-up Entrepreneurs: a Case Study of a Graduate Entrepreneurship Programme at Swinburne University, Technovation, 18(4): 275-286.

Nichols, S.P. and Armstrong, N.E. (2003). Engineering entrepreneurship: does entrepreneurship have a role in engineering education?, Ieee Antennas and Propagation Magazine, 45(1): 134-138.

Ohland, M.W., Frillman, S.A., Zhang, G.L., Brawner, C.E. and Miller, T.K. (2004). The effect of an entrepreneurship program on Gpa and retention, Journal of Engineering Education, 93(4): 293-301.

Penaluna, A. and Penaluna, K. (2009). Assessing creativity: drawing from the experience of the UK’s creative design educators, Education and Training, 51(8/9): 718-732.

Peterman, N.E. and Kennedy, J. (2003). Enterprise education: influencing students' perceptions of entrepreneurship. Entrepreneurship Theory and Practice, 28(2): 129-144.

Pittaway, L. and Cope, J., (2007a). Entrepreneurship education: a systematic review of the evidence, International Small Business Journal, 25(5): 477-506.

Pittaway, L. and Cope, J., (2007b). Simulating entrepreneurial learning: assessing the utility of experiential learning designs, Management Learning, 38(2): 211-233.

Pittaway, L. and Edwards, C., (2012). Assessment: examining practice in entrepreneurship education, Education and Training, 54(8/9): 778-800.

Pittaway, L., Hannon, P., Gibb, A. and Thompson, J., (2009). Assessment in enterprise education, International Journal of Entrepreneurial Behaviour and Research, 15(1): 71-93.


24 | P a g e

Race, P. (2003). Why do we need to ‘repair’ our assessment processes? A discussion paper, Journal of Science Education, 4(2): 73-76.

Ramsden, P. (1992). Learning to teach in Higher Education, London: Routledge.

Rust, C. (2002). The impact of assessment on student learning, Active Learning in Higher Education, 3(2): 145-158.

Taylor, D.W., and R. Thorpe. (2004). Entrepreneurial learning: A process of co-participation. Journal of Small Business and Enterprise Development, 11(2): 203–11.

Topping, K. J., Smith, E. F., Swanson, I. and Elliot, A. (2000). Formative peer assessment of academic writing between postgraduate students, Assessment and Evaluation in Higher Education, 25(2): 149-166.

Wee, K.N.L. (2004). A problem-based learning approach in entrepreneurship education: promoting authentic entrepreneurial learning. International Journal of Technology Management, 28(7/8): 685-701

Yorke, M. (1998). The management of assessment in Higher Education, Assessment and Evaluation in Higher Education, 23(2): 101-114.


25 | P a g e

Table 1: How effective were the following learning opportunities for your learning experience?

Answer Options

Not effective at all

%

Reasonably effective

%

Very effective

%

Life changing

%

The disability awareness session 0.0 22.6 61.3 16.1

Question and Answer session with the customers 0.0 16.1 61.3 22.6

Lectures with external experts 3.2 32.3 54.8 9.7

Meetings/emails with your mentors (Rob or Anne) 6.5 45.2 41.9 6.5

Feedback of your initial ideas/business model 0.0 29.0 61.3 9.7

Doing the research for your project 0.0 25.8 51.6 22.6

Writing your business plan 3.2 16.1 64.5 16.1

The elevator pitch 6.5 22.6 64.5 6.5

Poster presentation (questions from judges and customers) 0.0 22.6 61.3 16.1

Feedback received at the end of the module 3.2 22.6 67.7 6.5

Working with your teammates 0.0 16.1 67.7 16.1


26 | P a g e

Figure 1: Assessment model.

Students define problem

Students do market and user design

research

Students produce initial ideas and business model

Students produce final solution, business plan, elevator pitch, poster

and discretionary work

Competition: Students

demonstrate competence

Costumer and designer expert

• Marketeers

• Business advisors

• IP Lawyers • Bankers

• Accountants

• Entrepreneurs

• Manufacturers

Judges as above and various others

Questions and answer session with

customers

Feedback from mentors

Feedback from customer, designer,

manufacturer, business advisors and mentors

Feedback closely interlinked with

competition

Feedback from judges

INPUT ASSESSMENT METHODS FORMATIVE LEARNIING


27 | P a g e

Figure 2:


28 | P a g e

Image 1: Students meeting Kieron and his family 2007

Image 2: Students meeting Kieron and his family 2007


29 | P a g e

Image 3: External contributors 2009

Image 4: Poster presentation 2009

Image 5: Awards ceremony 2009


30 | P a g e

Image 6: Prototype of walking frame (2008)

Image 7: Prototype of the ‘scribbler’ (2007)

Image 8:


31 | P a g e

Image 9: Dr Paul Thomas (Entrepreneur) and Richard Campos (Business consultant) judging at the

poster presentation.

Image 10: Winning team with customer


32 | P a g e

Image 11: Example of prototypes from 2010

Image 12: Example of prototypes from 2011

Image 13: 79 year old William Broad, who had a stroke 2 years ago, using the glove

ihttp://www.bbc.co.uk/mediaselector/check/player/nol/newsid_6670000/newsid_6670000?redirect=6670013.stm&news=1&nbram=1&bbram=1&bbwm=1&nbwm=1

http://www.bbc.co.uk/mediaselector/check/player/nol/newsid_6670000/newsid_6670000?redirect=6670013.stm&news=1&nbram=1&bbram=1&bbwm=1&nbwm=1

http://www.bbc.co.uk/mediaselector/check/player/nol/newsid_6670000/newsid_6670000?redirect=6670013.stm&news=1&nbram=1&bbram=1&bbwm=1&nbwm=1

Documents

Assessing Student Learning in Engineering … filecontext and approach, and then outline the assessment methods used. The final part of the paper will The final part of the paper will