Big Data Analysis on Student Learning & Course Evaluation in Waseda University, Japan

YAMAGISHI, Naoji Center for Higher Education Studies (CHES) Waseda University, Tokyo, Japan

May 20-21, 2016

@ Asia University, Taichung, Taiwan

1

Outline 1. Accountability and its Backgrounds in Japan

• Grand Transformation in Japanese Higher Education • Japan’s Policy Issues • H.Ed.’s Responses

2. Waseda’s Strategies: CHES and its IR Functions • Roles of CHES • CHES’s IR Functions • CHES has done and will do

3. A Case of Data Analysis 4. Issues: Chance to Open a Dialogue

Yamagishi

１. Accountability: Grand Transformation in Japanese H.Ed.

2

Global Competition & Knowledge-Based Economy

• Ask me my three main priorities for government…education, education and education (Blair, Oct. 1, 1996)

H.Ed. has become the key for economic success for nations, organizations, & individuals.

Quality of H.Ed. has become EVERYONE’s stakes.

Japan’s Specific Background : declining in the # of 18 years old and increasing the # of institutions falling below their quota

• 18-year-old population was about 2million in 1992, but declined to about 1.2million in 2011.

• the # of private institutions was 372 in 1990 → 605 in 2012

• among private 4-year-colleges, about 45% falls below their quota

• seating capacity (# of enrolled/ # of applicants) = 92%

• about 46% were admitted without admission tests (e.g.) recommendation

Knock, and the (college) door will be opened to you (Matthew 7:7-8 ).

Larger investment on H.Ed., but ambiguous quality →Are college graduates sufficiently educated?

Yamagishi

１. Accountability: Grand Transformation in Japanese H.Ed.

3 Yamagishi

１. Accountability: Grand Transformation in Japanese H.Ed.

4 Yamagishi

The # of applicants (thousand)

100% & beyond (left axis)

The # of private institutions

• Applicant-to-seat ratio and the number of applicants (private inst.)

The # of applicants (right axis)

Above 50% below 100% (left axis)

Below 50% (left axis)

Source:https://www.jri.co.jp/MediaLibrary/file/seminar/110602_359/handout_110602_359_01.pdf

１. Accountability: Japan’s Policy Stance

5

Deregulation of University Act (1991) :

• allowed H.Ed. Inst. to have considerable discretion over curriculum design, degree-grating criteria, faculty placement, etc.

Quality Assurance through Accountability on Student Learning

• Self-assessment & evaluation was stipulated as task for each inst. (University Act, 1991)

• Third party evaluation became compulsory for all H.Ed. Inst. (2004)

Institutional as well as program recognition

• Central Council for Education’s report to the government

Focus on student learning (2008)

Establish institution-wide academic management and PDCA cycle (2012)

Annually open results of assessment of student learning to the public (2012)

• Enforcement Regulations for the School Education Law (2011)

Mandatorily disclose 9 indicators of institution-wide information

• Public financial support (by MEXT): total budget ￥20 billion（NTD6 billion) for 300 institutions for improving education quality that includes establishing institution-wide academic management and utilizing student course evaluation

Yamagishi

１. Accountability: H.Ed. Responses

6

National Survey 1 (MEXT, 2014) (n=766 institutions <response rate 100%>)

• carry out student course evaluation: 80% institutions（2008）→95％（2012)

• open the aggregated results to the public: 18%

• open the aggregated results to students and faculty only: 39%

• not open at all (1%)

National Survey 2 (The University of Tokyo, 2014) (n=555 institutions <response rate

71%>)

• institution-wide IR office (30%)

• IR related office (25%)

• function of IR: provide information to the university executive office (66%), get ready for the third party evaluation (53%), analyze student course evaluation (42%) , analyze GPA (39%), evaluate faculty development program (25%)

• Who can access to data:

student course evaluation data: executive office (20%), IRer (14%), each school/department (93%)

student related data (e.g. GPA): executive office (18%), IRer (14%), each school/department (98%)

Yamagishi

CHES

2．Waseda’s Strategies: Roles of CHES

7

Vision 150 In 2012, Waseda University established “Waseda Vision 150”, displaying a road map to be followed until its 150th anniversary in 2032.

13 Core Strategies encompass the entire range of topics upon which universities are operated, including the admissions process, education, research, international initiatives, pioneering new fields, and university management.

The Center for Higher Education Studies (CHES) was established in February 2014, in order to promote the autonomous and sustainable reform of Waseda that contributes to its improving the quality of education, research and management.

Center for Higher Education & Institutional Research (CHEIR)

Center for Teaching, Learning, & Technology (CTLT)

Yamagishi

KEY ACTIVITIES

2．Waseda’s Strategies: Roles of CHES

8

• CHES’s director is the university’s provost.

• CHES reports to its management committee.

• CHES develops various

projects around 5 key areas that are connected to 4 of 13 Core Strategies.

Realizes Waseda Vision 150

Strengthening IR Functions

Higher Education Research

Supporting Teaching

and Learning

Developing Education Methods

Disserminating/ Developing

Good Practice

Yamagishi

2. Waseda’s Strategies: CHES’s IR has done & will do

has done….

CHES

IR

Admissions Center

Educational Affairs

Management Planning

Center for Research Strategy

Board

Executive Office

22 Schools

Research Institutes, Centers, Others

• built up Integrated Data Ware House (IDWH) with IT Strategies Division (March, 2016)

• formulated the guidelines for managing IDWH (March,2016)

• organizes IRer liaison committee (every other month) including Admissions Center, Educational Affairs Division, etc.)

Yamagishi

• support/consult with the schools and centers on data analysis

• open the results to the university community and the public

• provide frameworks for data analysis

• educate IRers at each school & center

• and so on, so on….

will do….

9

2. Waseda’s Strategies: CHES’s IR has done & will do

Four Purposes & Roles of IR (Volkwein, 1999)

A: To describe the inst. – IR as information authority B: To analyze alternatives – IR as policy analyst C: To present the best case – IR as spin doctor D: To supply impartial evidence of effectiveness – IR as scholar and researcher

Yamagishi

• CHES IR has done A, and will also intend to take roles of B, C, D

10

Purposes & Audiences

Formative & Internal – for improvement

Summative & External – for accountability

Organizational Role & Culture

Administrative & Institutional

Α C

Academic & Professional

B D

5 Developing Stages of IR as a change agent (Swing, 2009) 1. Build Awareness 2. Develop Focus 3. Increase Knowledge 4. Resolve to Change 5. Incorporate or Replace

CHES IR

3. A Case of Data Analysis

11

OBJECTIVS: The target audience was the internal stakeholders.

• internally publicized functions of IR & IR’s possible contributions to evidence based decision making

DOR: CHES & IT Strategies Division cooperatively worked together.

• Data Cleansing: IT Strategies Division

• Data Mining & Analysis: CHES

• NEC Corporation provided technical support.

DATA: was retrieved from IDWH & Waseda-net Course N@vi ®.

• IDWH includes student-related data (e.g., ID, age, sex, course grade, GPA) and course-related data (e.g., class-sizes, types, subjects, compulsory/elective requirements, designated year)

• Waseda-net Course N@vi ® includes student course evaluation data for each student (ID).

Course N@vi is Waseda’s original LMS introduced in 2007

The registered # of courses was over 13,000 (82%) , and the rate of use by faculty was 99% in 2015.

Yamagishi

3. A Case of Data Analysis

Distribution （%） of time spent in studying outside class w/n each letter grade • Data from one of the schools • n=9,002 (2012～2014）

Yamagishi

0%

10%

20%

30%

40%

50%

60%

70%＜

30m

in

30m

in～

1hou

r

1～3h

ours

3hou

rs ＜

＜30

min

30m

in～

1hou

r

1～3h

ours

3hou

rs ＜

＜30

min

30m

in～

1hou

r

1～3h

ours

3hou

rs ＜

＜30

min

30m

in～

1hou

r

1～3h

ours

3hou

rs ＜

＜30

min

30m

in～

1hou

r

1～3h

ours

3hou

rs ＜

A+ A B C F

12

4. Issues: Chance to Open a Dialogue

13

Open a dialogue with external stakeholders

• serious concern for quality of college education

push for more rigorous evaluation of academic achievement (GPA)

push for increase in time for studying out of class

• First of all, what makes people invest time in studying?

Do college grades worth something in Japan?

Espoused Theory vs. Theory-in-Action (Argyris and Schon, 1978; Argyris, 1999)

Espoused Theory: H.Ed. can educate people.

Theory-in-Action: H.Ed. is merely a device for screening people.

Open a dialogue within H.Ed. Community

• What is the espoused theory and theory-in-action in H.Ed.

• Does H.Ed. really function as a place for education?

• Does H.Ed. seriously check-in what, how, how much students learn?

• H.Ed. needs institution-wide, community-wide benchmarking data about student learning.

Much to work for IRer as an analyst, reporter, advocate, and communicator!!

Yamagishi

Bibliography

14

Argyris，C．and Schon，D．A．(1978) Organizational learning: A theory of action perspective．

Addison-Wesley．

Argyris，C. (1999) On organizational learning. (2nd ed)． Wiley-Blackwell．

Central Council for Education. (2008) Gakushikatei Toshin no kouchikunimukete

(Toshin)［Toward establishing undergraduate education (a report of an inquiry)］.

Central Council for Education. (2012) Aratana mirai wo kizukutame no daigakukyouiku no

shitsu teki tenkann ni mukete (Toshin)［Toward quality transformation of college education

for establishing a new future (a report of an inquiry)］. Ministry of Education, Culture, Sports, Science and Technology. (2014) Daigaku ni okeru kyouiku naiyo to no kaikaku jyoukyou nitsuite［Regarding collegeeducation reform］. Oki, K. (2011) Nihon no shiritsu daigaku ni okeru institutional research (IR) no dokou [The trend of institutional research in Japanese private universities]. Daigaku Hyoka Kenkyu, 10, 37-45, 2011. Swing, R. L. (2009) Institutional researchers as change agents. New Direction for Institutional Research, 143, 5-16. The University of Tokyo. (2014). Daigaku ni okeru IR no genjyo to arikata ni kansuru

cyousakenkyu [Investigation & research regarding the present and future of college IR]. Volkwein, J. F. (1999) The four faces of institutional research. New Direction for Institutional Research, 104, 9-19.

Yamagishi