6613 -2

Assignment 2-Analysis Report.docx

Table of ContentsOverview.3Goals Analysis4Figure 1: Goal Statement & Goals Analysis Diagram.....4Subordinate Skills Analysis..5Figure 2: Subordinate Skills Analysis for Step 6 in Public Program Evaluation ....6Learner Analysis....6Table 1: Learner Analysis for PAD6327: Public Program Evaluation..7Context Analysis...12 Performance Context Analysis.12Table 2: Analysis of Performance Context for PAD6327: Public Program Evaluation... 13Learner Context Analysis16Table 3: Analysis of Learner Context Analysis for PAD6327: Public Program Evaluation.16Appendix A: PAD6327 Student Interview Questions..19Appendix B: PAD6327 Student Interview Questions Part II......20Appendix C: PAD6327: End of Course Evaluation Survey....21

Overview

PAD6327 Public Program Evaluation is a Graduate course offered at the University of Central Florida. During this course students will concentrate on concepts and issues of public programs. This course focuses on developing an evaluation process and designing questions that will aid in the plan and growth of the program. Students will apply principles to assess the needs of the platform, discover issues that affect the outcome of the process, analyze questions, and evaluate ethical issues. Students will focus on the evaluation process and the related results.This course is one of the major elective courses for the Master of Public Administration program and is one of the core courses for the Graduate Certificate in Nonprofit Management program.PAD6327 will be delivered on the World Wide Web. Students will be required to contribute to discussions and complete assignments as scheduled. This is a 16 week course.Below is an Analysis Report, which summarizes the goals of the course along with the subordinate skills analysis for evaluation issues and questions.

Goals AnalysisStudents in PAD6327 will explain major concepts in program evaluation and apply the concepts in developing a practical evaluation proposal (Figure 1). Additionally, students will apply principles of research design to evaluation questions. Students will also identify useful, feasible, ethical, accurate, and culturally sensitive methods of conducting process and outcome evaluation for research administration students. The goals statement was identified by Dr. Chen and the subject matter expert (SME) for this course, Dr. Jo Smith of the Department of Public Administration. It states that: Working with a service-learning partner, Research Administration graduate students will formulate an action plan to evaluate a public or nonprofit organization using a logic model following established research methods.The statement was classified as an Intellectual Skill based on Gagnes taxonomy. According to the Dick and Carey model, intellectual skills require learners to learn concepts, follow rules and solve problems in order to master goals (2009, p.44). The goals statement identifies eight major steps that are necessary for students to successfully meet the course goal statement. The subsequent sections of this analysis report will focus on Step 6: Formulate Evaluation Questions, where the focus will be on creating evaluation questions for Public Program Evaluation Research. Figure 1: Goal Statement & Goals Analysis Diagram

Subordinate SkillsFigure 2 depicts the subordinate skills and procedural tasks for the sixth step: Formulate Evaluation Questions. To begin the process, learners will need to identify who the decision makers are in regards to current evaluation issues at hand. Once the decision makers are identified, the purpose of the evaluation needs to be determined by asking the decision makers what they feel is the purpose of the evaluation. After the purpose of the evaluation has been determined, a program logic model will need to be constructed. Following the construction of the program logic model, summary questions for the evaluation should be drafted. The learner should develop both formative and summative questions. Examples of some formative questions are: What is the definition and scope of the problem or issue? Where or how serious is the problem? How should the program or technology be delivered to address the problem? How well is the program or technology delivered?Examples of summative questions include: What type of evaluation is feasible? What is the effectiveness of the problem or technology? What is the net impact of the program?

Once those questions have been drafted, the last step is to verify that the evaluation questions meet the required criteria to be included in the evaluation plan. It should be noted that the entry skills required for this step are the completion of steps 1-5 of the Goals Analysis Diagram (see Figure 1) prior to beginning step 6. Since these skills apply to the entire unit or course goal, it does not have any direct connection with any specific subordinate skill.

Figure 2: Subordinate Skills Analysis for Step 6 in Public Program Evaluation in Research Administration

Learner Analysis One significant reason analysis of learners is very important is to ensure that design and implementation of instruction is geared to the specific needs of the intended target group. It is significant to note that Instructional Analysis and Learner Analysis are parallel steps, not sequential, in systematic design. The size and complexity of a goal will frequently depend on any number of characteristics of those who will perform the tasks IDs identify and describe. Below is a table that explains across nine separate categories, the data sources, learner characteristics and implications for design, delivery and evaluation of the target audience for PAD 6327. Data was collected from four major sources: a verbal interview with Dr. Jo Smith, the professor for the course and the SME, the UCF Graduate schools current catalog description of the MRA masters program, a written interview with April Heyward (graduated from the UCF MRA program in August 2013 and was enrolled in PAD 6327 during the Summer 2013 semester), and finally anonymous responses to a PAD 6327 Summer 2013 course evaluation filled out by three students. The last two items can be found in Appendices A and B respectively.Table 1. Learner Analysis for PAD6327: Public Program EvaluationInformationCategoriesDataSourcesLearnerCharacteristicsImplications: Design, Delivery and/or Evaluation

EntrySkills and knowledgeInterviewed:Dr. Jo SmithRecords:UCF Graduate Catalog 2013-14, Research Administration MRAInterviewed:April Heyward, a student enrolled in PAD6327 in 2013

Relative to course content:

PAD6327 is the culminating course for the UCF Master Program in Research Administration (MRA), as such, learners will normally have already taken 33 hours in:

PAD 6742 Introduction to Research Administration

PAD 6XXX Governance and Regulatory Issues for Sponsored Programs

PAD 5850 Grant and Contact Management

PAD 6743 Leadership and Organization Models in Research Administration

PAD 6744 Financial Management in Research Administration

PAD 6417 Human Resource Management

PAD 6745 Contracting for Sponsored Programs

PAD 6335 Strategic Planning and Management

PAD 6746 Intellectual Property, Technology Transfer and Commercialization

PAD 6747 Audits in Research Administration

PAD 6237 Research Integrity for Research AdministratorsWhile most students have are highly experienced in the RA field by the time they enter PAD 6327, they are not familiar with the course content.

Ms. Heyward reported that she was exposed to some concepts pertinent to PAD 6327, while taking the course PAD 6335- Strategic Planning and Management. In particular, she mentioned that she learned Formative evaluations monitor the progress of the strategic plan implementation on an ongoing basis and provide feedback for improvements. That same student also reported that an undergraduate course in Quantitative Methods which taught such skills as probability, tests of significance, linear programming, and Markov analysis, queuing theory, inventory models, basic calculus with business applications, was very helpful in preparing her for PAD 6327.

Relative to Delivery System:

The MRA is given completely online indicating that by the time they are taking PAD6327 students are sophisticated users of technology, experienced practitioners of internet research and excellent independent workers who are highly self-motivated (students must maintain a B average to stay in the program).Relative to course content:

PAD6327 is the last in a sequence of 12 courses required to earn the MRA. It may be necessary to provide a brief review of information covered in the earliest courses, particularly for those skills necessary for successful completion of Public Program Evaluation.

Given that there is variability in the entry skills of incoming students (due to undergraduate or work for certifications, prior to being accepted into the UCF MRA) the professor may want to administer a pre-test or even do a brief survey of all students in order to introduce any concepts that are basic building blocks for the course.

Relative to Delivery System:

The MRA is given entirely online, therefore students are thoroughly fluent in all skills needed for a positive learning experience. Additionally, UCF has numerous tools readily available to assist students who are taking courses online such as videos, tutorials, and technicians are available by phone and email. Last, the professor is available for help with any glitches that may be creating difficulty for the student.

Prior knowledge of topic areaInterviewed:Dr. Jo SmithInterviewed:April Heyward, a student enrolled in PAD6327 in 2013

The interview with Dr. Smith revealed that students entering this course have not been introduced to the concepts in Public Program Evaluation in any prior MRA courses. Given the newness of the subject, and the fact that it is known in relatively small academic circles, learners prior knowledge is likely little to none.

Ms. Heywards responses reinforced information found in the previous category: that there is a wide variation of prior knowledge in students taking PAD6327, due to classes taken and certifications earned as undergraduates before entering the UCF MRA program.Given that entering students are unfamiliar with the concepts necessary for an understanding of what Public Program Evaluation is, designers should start with more basic information than they otherwise would with such an intellectually sophisticated group. Given the caliber of students, their ability to learn will quickly make the learning curve less steep. It is likely that they will rapidly gain momentum and be able to demonstrate the pragmatic use of the concepts, rules and problem-solving skills in the field.

Despite the fact that some students may have already acquired skills helpful in performing public program evaluations, the professor is only safe in assuming those concepts and problem solving methods taught in the first eleven courses of the MRA program.

Attitude toward contentInterviewed:Dr. Jo Smith

Survey:3 Students taking PAD6327 during Summer 2013.

Interviewed:April Heyward, a student enrolled in PAD6327 in 2013

Dr. Smith, as the instructor, felt that the unfamiliarity with the topic led to an initial struggling with the content and thus, a somewhat negative attitude toward content. Once they were assured of its relevancy, and fact that these skills will be expected of them if they want to advance in their careers, their attitudes improved.

Ms. Heyward reported that, the course content is beneficial for, but not limited to:

Developing a proposal for funding

Developing & evaluating a new or existing program/service/technology

Working with faculty and staff principal investigators in developing their proposals for funding

Reviewing proposals for funding as a Peer Reviewer

There were eight questions in the course evaluation dealing with matters of course content that were relevant to Implications: Design, Delivery and/or Evaluation. Those questions were 2, 11, 12, 14, 18, 19, 21 and 22. These answers can be found in Appendix B and are described below.

The most positive aspect to be identified in the course evaluations was the students appreciation of Power Point presentations and video lectures that explained in a practical way how to apply concepts to their evaluation plans.

On the negative end, 1 or 2 students (of the 3 who submitted their evaluations) mentioned they did not find the course contents relevant, useful in understanding Research Administration, nor were satisfied with it. Also, 1 student found the textbook to be intense and stodgy, a lot to read and not easy to conceptualize the information in the way we would need it for our own plans.

Finally, one student found the course to be moderately easy and two found it to be not easy at all.Being aware of the students intimidation by these particular aspects of research methodology is important for the instructional designer. Once they learn that concepts such as quasi-experimental, meta-analysis, statistical effect sizes (powers and errors) are similar in difficulty level than other concepts they have learned, students proceed without unusual difficulty. It would be important to introduce them to the lingo of their field at this time, possibly in an instructor provided glossary or perhaps in one created by the students as part of an assignment. Rapid improvement in attitude toward content can then be comfortably anticipated.IDs should inform their work with the knowledge that students learn most readily when they have videos or presentations that remain stable and can be reviewed as many times as possible for understanding, and that they have a stronger preference for electronic media over printed materials.

Only one student felt negatively enough about the textbook to mention it in the survey. It would be difficult to know from that one response if it is worth the time to investigate other texts.

Attitude toward delivery systemInterviewed:Dr. Jo SmithRecords:UCF Graduate Catalog 2013-14, Research Administration MRA

Interviewed:April Heyward, a student enrolled in PAD6327 in 2013The MRA program is given completely online. The students are employed in full-time positions as research administrators and many of them have families. The online system, while often challenging, makes it possible for these students to earn their degrees. As has been stated previously, this is a culminating course in the program and as such students are familiar with the technology necessary to do well in an online course. Given the above, their attitudes toward the delivery system are extremely positive.

Ms. Heyward concurs with the above adding that the Canvas-Instructure System which allowed me to take the required courses in the program. The online system was my general learning preference while she was taking her degree courses. She elaborated, It [Canvas] allows you to work collaboratively with fellow cohort members around the country. You can post and view videos developed by faculty and fellow cohort members. The system notifies you of your progress during each course.

The completely online delivery system of the UCF MRA attracts many of its students precisely because it is extremely beneficial for them to work at their own pace, place and time.

Instructional designers can take full advantage of the fact that students are so familiar with having their course completely supplied to them through technological means. The IDs can use their expertise with all areas of online learning and vary instructional delivery through the use of videos, podcasts, and information accessible by devices other than computers such as tablets, phones and PDAs.

It is very important for IDs to keep in mind while developing curriculum and designing instruction that they should take advantage of the ease with which classmates can work collaboratively. This is especially imperative because research administrators generally work in teams on the job.

Academic motivationInterviewed:Dr. Jo SmithRecords:UCF Graduate Catalog 2013-14, Research Administration MRA


UCF requires that a student, in order to remain in the program, cannot earn lower than a B in any of the 12 MRA courses. This alone motivates students to achieve to the best of their ability.The caliber of professionalism and independence required to prepare for and enter the field of Research Administration is significant enough to assure Instructional Designers that they also possess considerable degrees of self-generated incentive to consistently work in a diligent and effective manner.

It is safe to say that Ms. Heyward spoke for herself and her classmates when she said My primary motivation during the semester, the last semester of graduate school, when I took PAD 6327 was graduating with my MRA degree on Saturday, August 3, 2013 at the University of Central Florida.Instructional designers need to take full advantage of the fact that all students are currently employed in Research Administration. Their work-circumstances augment their academic motivation, which provides them with the context in which to do their best work. It is reasonable for IDs to expect that these students can process sophisticated material efficiently as they design curriculum.

Educational and ability levelsInterviewed:Dr. Jo SmithRecords:UCF Graduate Catalog 2013-14, Research Administration MRASurveyed:Students who enter the MRA program are already college graduates. At the time they take PAD6327 they have almost two additional years of post-secondary education.To quote the UCF graduate school catalog: The Master in Research Administration is a highly competitive program that admits only thirty (30) students once a year in the fall semester.In order to be accepted into the MRA program students must submit three letters of recommendation: letters of recommendation should be from professors or professional research administrators who can attest to the applicants ability to succeed in graduate coursework and his or her work ethic.Finally prospective students must accompany their applications with mission statements and sophisticated descriptions of why they want to be in this program, what their areas of special interest are, what their expectations are for this program and what they feel is special about the qualifications they will bring to the program.Based on the stringent qualifications for being accepted into the program and remaining in it, it is safe to say the educational and ability levels are fairly heterogeneous and unquestionably solid. Instructional design for such a group should keep this in mind when deciding how large individual goals should be and when they make decisions regarding how much detail this class of learners will need for each step.The ID can safely expect a high level of intuitive, abstract, critical and concrete thinking when creating learning materials for this course.

General learning preferencesInterviewed:Dr. Jo SmithSurveyed:According to Dr. Smith this is a heterogeneous group when it comes to learning preferences. A certain percentage prefers working in groups, others like to work individually and a third group is comfortable with both. Most students have a strong appreciation for instructional videos because they are valuable for reinforcement of clarifying initial confusion with new material.It is important to note that as these students progress in the field of Research Administration their need to work in groups will continue to grow.Instructional designers should provide material that supports students who are the most reluctant and have the greatest difficulty working in groups. Perhaps it would be helpful to make sure that every student rates their own and each group members work in order to allay concerns about grading fairness with group work.

Attitudes towards UCF and the organization evaluated during PAD 6327Interviewed:Dr. Jo Smith

Interviewed:April Heyward, a student enrolled in PAD6327 in 2013Dr. Smith explained that students are initially concerned about whether or not the MRA is accredited at UCF because it is such a new field and very few institutions offer majors. Once they are assured that accreditation is covered by UCFs university-wide umbrella accreditation, their attitude is one of satisfaction.Additionally, the students are appreciative that UCF was awarded a grant from the National Council of University Research Administrators to start a two-year Masters Program.Ms. Heyward concurred with the above stating that The University of Central Florida is the premier institution in the country and one of the few institutions that offered a Masters program in Research Administration. The MRA program is comprehensive and covers all areas of Research Administration.

Heyward was employed by the University of South Carolina during the MRA program (and is currently) thus, that is the organization she evaluated during PAD 6327. She stated that the University of Central Florida is more progressive regarding Research Administration as compared to the University of South Carolina such as technology, policies, procedures, etc.There are, fortunately, no negative attitudes that instructional designers need to address.

All of Ms. Heywards remarks support the observation that IDs do not need to adjust design, delivery or evaluation due to any negative attitudes on the part of students towards UCF.

While the students attitude towards the organization they evaluate for this course is relevant, because of the variation due to each student potentially working on a different evaluation, it is reasonable that ID cannot anticipate the needs of the individual learner in this regard. The professor may have to start with a checklist and work one on one with students hopefully giving each one reason to see positive aspects in situations that are not apparently encouraging.

GroupCharacteristicsInterviewed:Dr. Jo SmithRecords:UCF Graduate Catalog 2013-14, Research Administration MRA


The MRA is limited to 30 students. As a result of the competitiveness necessary to become one of the 30, the group as a whole consistently demonstrates strong motivation and superior diligence in independent work.

Ms. Heyward observed that the inaugural graduating class of the Master of Research Administration (MRA) program consisted of females mostly and three males. I believe one male had a Master degree prior to entering the MRA program at UCF. Everyone else had Bachelor degrees prior to entering the MRA program at UCF. The inaugural graduating class had diverse backgrounds and experiences relating to Research Administration. There was a mix of students in the MRA program that held management positions and non-management positions at their respective institutions.

Since there are only 30 students or less in this course, designers can capitalize on the fact that the instructor can gear her teaching to smaller groups and not have to deal with the impersonality of communicating with a larger population. She has the luxury of tailoring the course delivery to student needs more-so than if the group were larger. Since PAD 6327 is a culminating class, designers can take advantage of the fact that these MRA candidates are well-acquainted with the standard techniques and vocabulary of Research Administrators each having worked in the field for 3-5 years and having received a B average or higher in the first 11 courses of the program.

Additional Information RequirementsThe interviews with Dr. Jo Smith and Ms. Heyward provided substantial details for the purpose of the learner analysis, as did the pertinent descriptions of the MRA requirements in the graduate school catalog. Only the sparse number of students who filled out the class evaluations shows any promise from further investigation.Context AnalysisThe Context Analysis report is comprised of information pertaining to the performance and learning environment of students in the Public Program Evaluation course. The information provided is meant to enhance the learning experience for students as well as provide insight for students to enhance their motivation and apply their newly acquired skills in the workplace. Information was obtained through interviews conducted with the SME, Dr. Smith who is also the instructor of this course, and a survey sent to a former student, April Heyward, who completed this course previously.Performance Context AnalysisTable 2 has the results of four information categories relevant to performance context analysis. Data was contributed by the SME, Dr. Smith, and by a former student of PAD632, April Heyward.The following table contains four categories. The first category defines the level of managerial or supervisory provisions in the work setting. The second category analyzes the physical facets of the location; we collected information based on four areas: the setting, the equipment, the time and the resources. In the third category, we measured the social facets of the work setting that are often determined by working in the field of Public Evaluation. The last category in the performance context analysis shows how the skills learned in the instructional unit will be pertinent to the work environment. Based on feedback from April Heyward, (see the performance context box under Relevance of new skills to workplace) one sees how well prepared Dr. Smiths students were to function in the real world of Research Administration. This is an indication that former students are an excellent source of information that can impact future instructional design.

We measured the social facets of the work setting that are often determined by the spectrum of the research. The last deliberation in the performance context analysis is to guarantee the skills learned in the instructional unit will be pertinent to the work environment.Table 2: Analysis of Performance Context for PAD6327: Public Program EvaluationInformation CategoriesData SourcesPerformance ContextImplications Design, Delivery and/or Evaluation

Managerial/supervisory supportInterviewed: Dr. Smith

Interviewed: April Heyward a student enrolled in PAD6327 in 2013Although a limited number of students were able to complete the survey, students who have been employed in the Public Research Administration field will need to be examined in order to determine the level of managerial/supervisory support, for the purpose of this analysis, it is assumed that supervisory support will vary based on the companys environment and the supervisors personality.

According to Ms. Heyward, I selected to evaluate the USCeRA (University of South Carolina Electronic Research Administration) System. I met with the IT & Data Management Director who developed the USCeRA system in 2002. The IT & Data Management Director reports directly to the Vice President for Research. The stakeholders that I identified for the evaluation plan were: IT & Data Management Director Involved in USCeRA Operations Category Research Awards Budget Manager Involved in USCeRA Operations Category Senior Applications Analyst Involved in USCeRA Operations Category Research Awards Budget Assistant Involved in USCeRA Operations Category IT & Administrative Support Involved in USCeRA Operations Category USC Faculty Served or Affected by USCeRA Category USC Staff Served or Affected by USCeRA Category USC Students Served or Affected by USCeRA Category Vice President for Research Intended Users of Evaluation Findings Category Associate Vice President for Research Intended Users of Evaluation Findings Category

Instruction should cover complications and potential pushback from managers/supervisors that may arise during the evaluation process so students will be better equipped to handle various levels of resistance from managers/supervisors that might occur in their future work.

Ms. Heyward met with over 10 members of the USCeRA Operations team, and an unspecified number of USC staff and students, she availed herself of an optimal managerial and supervisory support experience. Making herself known to such a wide variety of workers that she would interact with gave her a massive advantage over any other Research Administrators that may have limited themselves to a select few.

The implications for design and delivery are huge. A worker who does the work upfront to learn as much as he/she can about all those who may have an impact on his/her work opens up avenues that can be accessed when necessary in order to create products that are the result of being able to reach the most ambitious and sophisticated goals of instruction.

Physical aspects of siteInterviewed: Dr. Smith


Facilities: Each facility will vary. It is assumed that each student will have access to an appropriate workspace and necessary equipment needed to complete job requirements.

Resources: It is assumed that students will be given access to the necessary data needed for completing program evaluations.

Time: While Ms. Heyward worked on her evaluation plan she was a student, not an employee of the University of South Carolina (which she now is). Students were given 15 weeks to complete their evaluation plan.

Equipment: It is assumed that most companies will have the necessary computer equipment and software required to complete program evaluations.Instruction will mirror real workplace settings as close as possible. Students will be required to complete service-learning hours to gain real world experience. Students will also have access to the necessary hardware and software required to complete a public evaluation.

Opportunities will be provided through the service-learning hours to practice writing public evaluations in a real world environment.

In the first part of Ms. Heywards student interview questions, she made the comments: The University of Central Florida is more progressive regarding Research Administration as compared to the University of South Carolina such as technology, policies, procedures, etc. For the purposes of design and delivery, one can deduce that there is not an unlimited amount of time available to accomplish an evaluation in real world at the University of South Carolina. Thus, the design and delivery of instruction can describe optimal conditions under which to produce an evaluation and give students options for the best way to cut corners when necessary.

Social aspects of the performance siteInterviewed: Dr. Smith

Interviewed: April Heyward, a student enrolled in PAD6327 in 2013The social aspects of a workplace have a significant impact on worker performance. IDs must determine whether the student will work alone or as part of a team when demonstrating what has been learned as a result of instruction.

Will the student ultimately be supervising a number of other workers; will she be alone in the performance of her work obligations; will she berequired to present to audiences - these are among the questions that need to be answered by instructional designers.

Ms. Heyward responded in her Interview Questions Part II that I assigned the following roles and responsibilities of the Evaluation Team Members: IT & Data Management Director Lead Evaluator Role Research Awards Budget Manager Data Collection and Data Analysis Role Senior Applications Analyst Data Collection and Data Analysis Role Research Awards Budget Assistant Data Collection and Data Analysis Role IT & Administrative Support Evaluation Support Role Vice President for Research Stakeholder/Advisor Role Associate Vice President for Research Stakeholder/Advisor RoleThe implication for design and delivery that come out of that response are huge. A public program evaluation is best served and created by someone who has been taught how to bring out the best and most cooperation from team members and others working on the same project. Instruction needs to be geared to the skills and tools that prepare students on how to interact effectively with a wide range of other workers.

Relevance of new skills to workplaceInterviewed: April Heyward, a student enrolled in PAD6327 in 2013

As research administrators, evaluation is important in order to measure work performance as it relates to efficiency, training program impact, and processes.

Ms. Heyward answered the question how did PAD 6327 prepare students for real world experiences? In the following manner:The Public Program Evaluation course content prepared me to be able to: Develop a full proposal for funding (i.e., project narrative, measurements, indicators, evaluation plan)

Develop and evaluate a new or existing program, service, or technology

Work with faculty and staff principal investigators in developing their proposals for funding

Review proposals for funding as a Peer Reviewer

Clearly the instructional designers of Ms. Heywards course well knew the implications for design, delivery and evaluation. Based on her response, her class was launched rather fully prepared with skills, tools and a sophisticated understanding of the requirements of her field.

For any future evaluations of the design and delivery of the course, it would be logical to continue surveying graduates and using their real world experiences as guides to any instructional changes.

Additional Information Requirements Additional information from students who are enrolled or have been previously enrolled in the Public Evaluation program is required to complete the Performance Context Analysis. Online surveys could be conducted to obtain a fuller depiction from the students perspective of what is required to complete an online course in Research Administration.

Learning ContextTable 3 illustrates the four major categories of learning context analysis. These categories include: number and nature of the sites, site compatibility with instructional needs, site compatibility with learner needs, and feasibility for simulating workplace. These categories provide information about the learning environment and the information obtained was compiled by interviews conducted with the SME, Dr. Smith as well as with a survey that was previously sent former student, April Heyward. Additionally implications for design, deliver and/or evaluation are discussed.

Table 3: Analysis of Learner Context Analysis for PAD6327: Public ProgramInformation CategoriesData SourcesPerformance ContextImplications Design, Delivery and/or Evaluation

Number and nature of the learning siteInterviewed: Dr. SmithClass size has never gone above 23 students in the class.

Facilities: This class is an online class. Student learning will take place on UCF Webcourses. Students can work at their speed. The learning process takes place through Canvas.

Equipment: It is an online class, so the only equipment required is a computer. Students must have the ability to work on the computer and have some information about Webcourses. They also need internet access, Microsoft office, Adobe, and different types of video and audio players. Students also need to have a Knights email account.Resources: One textbook is required. A full-time instructor monitors the students progress throughout the course semester. The instructor reviews weekly postings for student understanding and provides timely feedback in areas that students may be struggling in.

Constraints: Students may face issues accessing and downloading the materials. They may have some difficulties accessing Webcourses during maintenance time.

Facilities: Since this class is an online class, students need to be familiar with Canvas and online learning. Due dates and requirements are stated in the syllabus.

Equipment: Students should have the ability to work with the software needed for this class.

Resources: All required material will be posted on Canvas.

Constraints: The course syllabus should state where students can obtain access to the software required to successfully access and download course material. Additionally, 24-hour notice from Webcourses about maintenance scheduling would help the students avoid submitting late assignments.

Site compatibility with instructional needs Interviewed: Dr. SmithInstructional strategies: Canvas is compatible with many different instructional tools that are used. The course uses traditional modules, and videos.Group functionality provides students with access to a wiki doc function so students can share and work on projects together online.A discussion board is also required.

Delivery Approaches: Students have the ability to contact one another online using the conference area, course emails, or the discussion boards.Groups are also formed to provide instructional tools and resources for group projects. Most students are in the same profession and have been working together as a cohort over the past 16 months. Time: Students are given one semester to complete the evaluation assignment for the course.

Personnel: One instructor is available to present the course.Instructional strategies:Weekly postings are aligned with the sections of the Evaluation Plan paper due at the end of the course. Two exams are given in the middle and at the end of the course to measure student comprehension and understanding of the reading materials and concepts.

Delivery Approaches: Students should be able to exchange ideas online and use the appropriate method to manage their groups.

Time: The syllabus indicates the required timeframe for students to complete all coursework, assignments, and tests.

Personnel: Since there is just one instructor presenting the class, a clear syllabus needs to be provided.

Site compatibility with learner needsInterviewed: Dr. SmithLocation: The learning portion of this course occurs entirely through Canvas. All course materials are accessed online. The course is designed to serve students by being completely asynchronous.

Convenience: Learners can access Webcourses from anywhere they would like.

Space: Since it is an online class no physical space required.

Equipment: The campus has many computer labs for the students to access Webcourses.Students are provided with multiple ways to access the necessary materials required to complete this instructional unit as well as the course.

Feasibility for simulating workplaceInterviewed: Dr. Smith

Supervisory Characteristics:As research administrators, evaluation is important in order to measure work performance as it relates to efficiency, training program impact, and processes.

Physical Characteristics: This course is a service-learning course. Students are required to work directly with a real organization to develop an evaluation plan. They must interact with stakeholders within the organization to complete the final product.

Social Characteristics: Students are expected to maintain professionalism while working with stakeholders to complete their evaluations. Additionally, the course is designed for students to complete the evaluation process while working with an organization. Therefore, the service-learning portion of this course will prepare students for real-world applications.Supervisory Characteristics:Evaluation performance is often conducted in research administration offices annually and when special projects are to be reviewed. For example, training programs for administrators or faculty, and evaluating the impact of in-house grant programs to encourage external funding competitiveness for young faculty researchers.

Physical Characteristics: Going to a real organization to work will help the students to develop an idea about the evaluation plan. Further action from the instructor is required to give some advice from time to time.Social Characteristics:The social characteristics are easily simulated, as the students are required to work directly with an organization to complete their evaluation. Students are expected to maintain professionalism throughout the evaluation process while working with their selected organization, which simulates a workplace.

Additional Information RequirementsAdditional information from students who are enrolled or have been previously enrolled in the Public Evaluation program is required to complete the Learner Context Analysis. Online surveys could be conducted to obtain a fuller depiction from the students perspective of what is required to complete an online course in Public Evaluation.

Appendix A: PAD6327 Student Interview QuestionsThis questionnaire was administered via an interview to students who have already taken PAD6327 to gain a better understanding of student perceptions of the course.1. Prior to taking PAD6327 Public Program Evaluation, how would you describe your entry skills and knowledge of the topic - Public Program Evaluation; please address both course content and delivery system.2. Prior to taking PAD6327 how would you describe your educational and ability levels?3. Would you please describe your general learning preferences during the time you were taking PAD6327.4. Would you please describe your attitudes toward both UCF and the organization you evaluated during the semester you took PAD6327.5. Would you please describe group characteristics of your classmates during the semester you took PAD6327. This would include but not be limited to such areas as: Subgroups of audience members (rural/urban, grade/educational level, age and gender) Background Culture (social, economic) Educational expectations PAD6327 is the last in a sequence of 12 courses. Heterogeneity Overall impressions

6. Would you please describe your academic motivation during the semester you took PAD63277. Would you please describe your prior knowledge of public program evaluation at the beginning of the semester you took PAD6327.8. Would you please describe your attitude toward the content of PAD6327 before, during and after taking the class?9. Would you please describe your attitude toward the delivery system of PAD6327 before, during and after taking the class?

Appendix B: PAD6327 Student Interview Questions Part IIThis questionnaire was administered via an interview to students who have already taken PAD6327 to gain a better understanding of student perceptions of the course.1. In the workplace, how much time is given to complete a public research admin evaluation?

2. What are a Public Research Admin's job requirements?

3. Which employees do Public Research admin interact with while completing a program evaluation? (stakeholders, managers?)

4. How did PAD6327 prepare students for real world experiences?

5. How many service learning hours were you required to complete for PAD6327?

Appendix C: PAD6327 End of Course Evaluation SurveyThis survey was administered to students who completed PAD6327. The purpose of this survey was to provide feedback to the instructor about the overall course and instruction.

Q1 How well did your instructor meet the course objectives?Answ ered: 3 Skipped: 0

Completely met obj ectiv e

Met the obj ectiv e w ell

Just met the obj ectiv e

Barely met the obj ectiv e

DId not meet the obj ectiv e

0%20%40%60%80%100%

Students will be able to explain major c onc epts of program evaluation,

Studients will be to apply the c onc epts in developing a prac tic al evaluation proposal pape

Students w ill be able to explain maj or concepts of program ev aluation, Studients w ill be to apply the concepts in dev eloping a practical ev aluation proposal paper.Total Respondents

Completely100%100%

4

met objec tive22

Met the100%100%

2

objec tive well11

Just met the0%0%

0

objec tive00

Barely met the0%0%

0

objec tive00

DId not meet the objec tive0%00%0

0

2

Q2 How relevant was the course material?Answ ered: 3 Skipped: 0

Extremely w orthw hile

Very w orthw hile

Moderately w orthw hile

Slightly w orthw hile

Not at all w orthw hile

0%20%40%60%80%100%

Answ er ChoicesResponses

Extremely worthwhile33.33%1

Very worthwhile0%0

Moderately worthwhile66.67%2

Slightly worthwhile0%0

Not at all worthwhile0%0

Total3

Q3 How organized was the course content?Answ ered: 3 Skipped: 0

Extremely organized

Very organized

Moderately organized

Slightly organized

Not at all organized

0%20%40%60%80%100%


Extremely organized66.67%2

Very organized0%0

Moderately organized33.33%1

Slightly organized0%0

Not at all organized0%0

Total3

Q4 How knowledgeable in the course content was your instructor?Answ ered: 3 Skipped: 0

Extremely know ledgeable

Very know ledgeable

Moderately know ledgeable

Slightly know ledgeable

Not at all know ledgeable

0%20%40%60%80%100%


Extremely knowledgeable66.67%2

Very knowledgeable33.33%1

Moderately knowledgeable0%0

Slightly knowledgeable0%0

Not at all knowledgeable0%0

Total3

Q5 How clearly did your instructor explain difficult material?Answ ered: 3 Skipped: 0

Extremely clearly

Very clearly

Moderately clearly

Slightly clearly

Not at all clearly

0%20%40%60%80%100%


Extremely c learly66.67%2

Very c learly33.33%1

Moderately c learly0%0

Slightly c learly0%0

Not at all c learly0%0

Total3

Q6 How concerned was your instructor with how well students were learning?Answ ered: 3 Skipped: 0

Extremely concerned

Very concerned

Moderately concerned

Slightly concerned

Not at all concerned

0%20%40%60%80%100%


Extremely c onc erned100%3

Very c onc erned0%0

Moderately c onc erned0%0

Slightly c onc erned0%0

Not at all c onc erned0%0

Total3

Q7 How easy was it to contact and receive feedback from your instructor for clarification or questions?Answ ered: 3 Skipped: 0

Extremely easy

Very easy

Moderately easy

Slightly easy

Not at all easy

0%20%40%60%80%100%


Extremely easy100%3

Very easy0%0

Moderately easy0%0

Slightly easy0%0

Not at all easy0%0

Total3

Q8 How clearly did your instructor explain how students would be graded?Answ ered: 3 Skipped: 0

Extremely clearly

Very clearly

Moderately clearly

Slightly clearly

Not at all clearly

0%20%40%60%80%100%


Extremely c learly100%3

Very c learly0%0

Moderately c learly0%0

Slightly c learly0%0

Not at all c learly0%0

Total3

Q9 How fair was your instructor's grading?Answ ered: 3 Skipped: 0

Extremely fair

Very fair

Moderately fair

Slightly fair

Not at all fair

0%20%40%60%80%100%


Extremely fair100%3

Very fair0%0

Moderately fair0%0

Slightly fair0%0

Not at all fair0%0

Total3

Q10 Were you given too many assignments, too few assignments, or about the right amount?Answ ered: 3 Skipped: 0

Much too many

Slightly too many

About the right amount

Slightly too few

Much too few

0%20%40%60%80%100%


Muc h too many0%0

Somewhat too many0%0

Slightly too many0%0

About the right amount100%3

Slightly too few0%0

Somewhat too few0%0

Muc h too few0%0

Total3

Q11 How useful were the assignments in helping you understand research administration?Answ ered: 3 Skipped: 0

Extremely useful

Very useful

Moderately useful

Slightly useful

Not at all useful

0%20%40%60%80%100%


Extremely useful33.33%1

Very useful0%0

Moderately useful66.67%2

Slightly useful0%0

Not at all useful0%0

Total3

Q12 How easy were the assignments in this course?Answ ered: 3 Skipped: 0

Extremely easy

Very easy

Moderately easy

Slightly easy

Not at all easy

0%20%40%60%80%100%


Extremely easy0%0

Very easy0%0

Moderately easy33.33%1

Slightly easy0%0

Not at all easy66.67%2

Total3

Q13 How much did success in the course depend upon understanding ideas, rather than memorizing facts?Answ ered: 3 Skipped: 0

A great deal

A lot

A moderate amount

A little

None at all

0%20%40%60%80%100%


A great deal33.33%1

A lot33.33%1

A moderateamount33.33%1

A little0%0

None at all0%0

Total3

Q14 Were you satisfied with the course content, neither satisfied nor dissatisfied with it, or dissatisfied with it?Answ ered: 3 Skipped: 0

Extremely satisfied

Moderately satisfied

Slightly satisfied

Slightly dissatisfied

Moderately dissatisfied

Extremely dissatisfied

0%20%40%60%80%100%


Extremely satisfied33.33%1

Moderately satisfied0%0

Slightly satisfied66.67%2

Neither satisfied nor dissatisfied0%0

Slightly dissatisfied0%0

Moderately dissatisfied0%0

Extremely dissatisfied0%0

Total3

Q15 Were you satisfied with your instructor's teaching, neither satisfied nor dissatisfied with it, or dissatisfied with it?Answ ered: 3 Skipped: 0

Extremely satisfied

Moderately satisfied

Slightly satisfied

Slightly dissatisfied

Moderately dissatisfied

Extremely dissatisfied

0%20%40%60%80%100%


Extremely satisfied66.67%2

Moderately satisfied33.33%1

Slightly satisfied0%0

Neither satisfied nor dissatisfied0%0

Slightly dissatisfied0%0

Moderately dissatisfied0%0

Extremely dissatisfied0%0

Total3

Q16 Which assignments were most useful?Why?Answ ered: 1 Skipped: 2

#ResponsesDate

1It was helpful to work on the piec es of the evaluation planf8/8/2013 8:32 AM

Q17 Which assignments were least useful?Why?Answ ered: 0 Skipped: 3

#ResponsesDate

T here are no responses.

Q18 Which course materials were most useful? Why?Answ ered: 2 Skipped: 1

#ResponsesDate

1PowerPoint and video lec tures explaining the evaluation plan.8/8/2013 2:51 PM

2instruc tor video and powerpoint were most useful - they explained the c onc epts in a prac tic al way, as they needed to be applied to the evaluation plan8/8/2013 8:32 AM

Q19 Which course materials were least useful? Why?Answ ered: 1 Skipped: 2

#ResponsesDate

1T he textbook was intense and stodgy. It was a lot to read, and not easy to c onc eptualize the information in the way we would need it for our own plans8/8/2013 8:32 AM

Q20 What are the strengths of this course?Answ ered: 1 Skipped: 2

#ResponsesDate

1I do think understanding evaluations is important, whether you are pre-, post-, or other area of researc h administration. It is a key c onc ept to understand.8/8/2013 8:32 AM

Q21 How could the instructor improve this course?Answ ered: 1 Skipped: 2

#ResponsesDate

1use the modules to better explain the parts of the evaluation plan / give the video & powerpoint toward the beginning of the c ourse8/8/2013 8:32 AM

Q22 Other comments.Answ ered: 1 Skipped: 2

#ResponsesDate

1T his c ourse presented interesting material, but I think it will probably be the least useful c ourse for me in my role as a researc h administrator.8/8/2013 8:32 AM

Documents

6613 -2