CAEP Accountability Measures

Under Measure One, this item evaluates by observation or P-12 student and administration surveys how program completers apply the professional knowledge, skills, and dispositions acquired in their preparation programs effectively in their P-12 classrooms. Generally the EPP sends all completers an annual survey, grounded in the InTASC standards. The survey prompts a self-assessment regarding their effectiveness in the P-12 classroom. The EPP also surveys principals regarding completer performance and their P-12 learners' growth. Further, some EPP programs engage in observations of completers in the P-12 classroom. Unfortunately, during COVID the EPP was not permitted in-person access to teachers' classrooms and guests were not allowed in virtual classrooms, thereby limiting data collection for the 2020-21 academic year.  The EPP implemented a revised strategy in Spring 2021 and asked teachers to complete a self reflection table on P-12 growth as of mid-year 2021-22 and on progress on self-effectiveness goals. While some teachers reported not being able to meet their goals for effective teaching, others responded that they exceeded their goals in 2020-21. Moving forward, this new tool was used again late Fall 2021 to check in once again with classroom teachers to ask about P-12 mid-year growth in the classroom and how teachers are meeting their own effectiveness goals.  Teachers were able to respond via email or via Qualtrics anonymously.  This second data collection compares 2020-21 with mid-year 2021-22, thus allowing to compare previous and current years as well as support reliability and validity of this measure as it is used in concert with additional indicators of completer effectiveness. One aspect of this simpler tool is the personal details shared back to the EPP about the teachers' classrooms, what student growth has looked like over the past two years, and how they have impacted this growth. These rich responses have provided the EPP with excellent data to consider.

There were 16 survey responses across Elementary, Secondary, and Special Education teachers.  As a collective group, meeting personal goals for creating an effective learning environment, effectively teaching content, effectively using teaching strategies and effectively evaluating your teacher practice were just slightly higher by mid-2021 than they were for the AY 2020-21.  For example, in 2020-21, 14 counts teachers reported not meeting one or more effectiveness goals, but by mid-year 2021, that number had dropped to just seven.  In 2020-21, five counts teachers reported exceeding one or more effectiveness goals, but by mid-year 2021, that number had climbed to eight.

As documented  in the teaching effectiveness section above, teachers also reported percentages of their students who were exceeding, meeting, and not meeting district goals.  Again, reporting in the aggregate, teachers overall reported that P-12 learners met or exceeded district goals at a higher level during 2020-21 than at mid-year 2021-22.  This puzzled the EPP until reading some of the optional comments teacher provided.  During 2020-21, there were very low to no expectations from many of the districts represented so any and all growth was exceptional.  Expectations were in place for the 2021-22 school year and teachers felt they struggled with impacting student growth as the students themselves struggled with in-person classes and relearning classroom and school rules. We are currently reviewing data from teachers on student growth during the 2022-23 academic year.

Finally, the College has requested the latest copy of the PVAAS report from the Pennsylvania Department of Education.  This report shows value-added data for Pennsylvania educator preparation programs. Because K-12 testing was suspended during Covid, the latest PVAAS report available is for the 2021-22 academic year. The following areas had sufficient data for reporting:

In all the tables, statistical significance of the p-values is indicated by * for p < 0.10, ** for p < 0.05, and *** for p < .001.

Employer Satisfaction:

For the 2021-22 academic year, the EPP compared two years of aggregated survey responses from principals on where Penn State teachers need improvement to see how intentional changes in methods classes, dispositions, and student teaching seminars have made a difference.  In 2019, principals indicated inTASC 2 (diverse cultures), 5 (connect concepts), 6 (assessment), 7 (plan instruction) and 9 (professionalism) as areas that received scores below proficient.  In 2021, inTASC 2 (diverse cultures), 6 (assessment), 7 (plan instruction), 9 (professionalism)  and 10 (seek leadership roles) were areas that received scores below proficient.  Especially disappointing was the continued indication of additional work needed in diverse cultures, as the College has been implimenting anti-racism and equity into its curriculum, beginning in late 2020.  However the percentage of scores below proficient in almost every category decreased from 2019 to 2021, indicating that small changes have begun.  Puzzling was the added "seek leadership roles" as an area that needs improvement.  This has not been an area where Penn State teachers have typically faltered and the faculty questions if reduced field experiences in 2020 impacted new teacher confidence in seeking leadershp roles. This is an area that will receive additional scrutiny in the 2023 survey.

Also this year, in conjunction with field experience partner district visitations to discuss student teaching expectations and MOUs, the director of field experiences also asked principals and senior school personnel to share what they would like the program to know about first year teachers from Penn State, what the program is doing well and in what areas are the new teachers weak.  Overall, positive responses included the strengths and proficiencies that these teachers bring to their classrooms and buildings. Two common themes for improvement were management of courses and the idea to do the best they could do in unexpected moments - don't freeze.  As unexpected moments was also a theme that student teachers who took advantage of subbing experiences also reported, this topic will be discussed and reinforced during student teaching seminars.

Advanced Programs: Although the EPP has some of the same tracking completers into the field issues with the advanced preparation programs, there are some data for students completing Principal Certification programs. This program has begun surveying program completers to provide data for its SPA reporting and by looking at the question, "Did you change jobs after receiving your principal certification from Penn State?" the EPP can make judgments on the satisfaction of employers by the percentage of completers who were able to change positions. Of the 75 respondents who completed in 2016-2019, 17 of the 18 who changed jobs, did so into an administrative or supervisory role. Of the 58 completers who did not change positions, 15 were already in administrative or supervisory positions.

A second round of surveys were sent in 2022, and 20 of 37 responders indicated that receiving an advanced certification had a positive impact on their career including additional pay, additional responsibilities, promotions within the district, or new positions - all indications of Employer Satisfaction.

Because of the limited nature of the collected data, we developed a plan to collect additional data to demonstrate Employer Satisfaction. We modeled our Employer Satisfaction Survey from the Completer Survey administered and validated by institutions in the State of Mississippi. The instrument demonstrates sound psychometric properties as administered to completers of their advanced program. Aligned to CAEP key competencies and our Advanced Completer Survey, data from administration of the Employer Satisfaction Survey will inform advanced programs' continued improvement. As part of the 4.1 plan we demonstrate how we will assure the instrument meets standards for CAEP EPP-Created Assessments that provide data about our key competencies for Advanced Completers from all programs and allow for desegregated data and assessment of program-specific objectives. The 4.1 Plan includes a human subjects approved research study to further refine and test a measure that captures Employers' Satisfaction.

A test pilot of the survey was sent to two professional stakeholders, one for principal completers and one for superintendent completers.  These initial data show 100% Strongly Satisfied or Satisfied responses out of a rating scale of Strongly Satisfied, Satisfied, Dissatisfied, Strongly Dissatisfied.  Feedback for areas of improvement included the importance of developing, implementing, and measuring long-term strategic goals and measures, leading system-wide change processes including curriculum development for principals.  Feedback for superintendent development included high knowledge of personnel management, finance, contract negotiations, community relations, and personnel evaluations.  The EPP will review course curriculum for alignment with these feedback areas and discuss ways of strengthening instruction and assessment of these items. A full data collection is scheduled for June 2023.

While many institutions target a day for data and reflection to meet CAEP Standards, the EPP employs the use of the Data Roundtable series. Each session in the series targets a specific topic or issue. The EPP seeks feedback from district faculty and staff, EPP faculty, staff, and students, alumni completers, and alumni who serve in district leadership roles. These valuable sessions provide opportunity to share data with stakeholders, interrogate the data together, and listen to stakeholder feedback and suggestions for programs based upon these data.

Our Roundtable Talks for the 2023-24 year included:

Updating the Middle Level Grades 4-8 Program - Looking at a Two-Certification Model

Amended Public School Code of 1949 – Educator Pipeline Support Grant Program

Structured Literacies: A discussion with stakeholders on current practices, opportunities, and challenges

Our Roundtable Talks for the 2022-23 year included:

Developing and Supporting an Equitable Society through Education - ways to improve our preparation of educators through an equity lens

Recruiting Education Professionals into Shortage Areas Update - how are schools coping; feedback on new College initiatives

Mental Health in Our Schools – What is Working; What is Needed - what are schools doing; how can we prepare resilient teachers

Teacher Candidates as Substitute Teachers – One Year Later - feedback on a state law change that impacted student teachers

An additional example of intense data-examination for program review and potential revision was the need for all programs to undergo state review in 2021- 2022. Responses to PDE as a primary EPP stakeholder initiated with program faculty and PCCC and was informed by SPA data and non-SPA program data,  data derived through the QAS systematic use of data, and data from other stakeholders, including mentors, principals, and superintendents. Discussion of the PDE major review data was held at PCCC, and two retreats were held with program stakeholders to review and discuss essential data for their program reviews. PDE approved all CAEP and non-CAEP educator licensure programs at Penn State by July 1, 2022.

All students completing a teacher preparation program at Penn State must meet program course requirements by passing with a C grade or higher, maintain a 3.0 GPA overall in the program, and pass a PDE-required content exam. The following table of data presents three years of completers with Praxis or Pearson testing pass rates and the average pass score. For programs that have less than 5 students per year, the completer data have been aggregated into one or more blocks to preserve student privacy.

Program Completers with GPA and Certification Exam Pass Rates
AY 2019-20 through 2023-24

Advanced Completers

One of the measures on program outcome that the College of Education is required to list in its annual report to CAEP is the ability of our program completers to be hired into positions for which they have been prepared. The Pennsylvania Department of Education (PDE) provides a list of Penn State students who have been hired in Pennsylvania and the school districts that have hired them. However, because a significant number of completers choose to seek teaching positions in other states, the College has been working on ways to maintain connections with those students using emails and surveys, so as to include their data in our self assessments.

The following tables reports PDE data on the number of Penn State completers hired into Pennsylvania school districts for  five calendar years at the initial and at the advanced program levels. Please note that 2023 initial completers were new to the job search process, and any recent hires would not yet be loaded into the PDE system.

A longitudinal study was started in 2017, a year after the first cohort of the new dual degree CEAED PK- Grade 4 bachelor's program with a one year Special Education PK- Grade 8 Master's degree graduated. Seven completers were emailed asking for feedback on the following questions: Are you working in an elementary school, are you in a general or special education classroom, and do you believe that your master's degree helped you obtain your position or made you better qualified.  In 2021 26 completers were contacted for their feedback, to compare responses between co-horts and with additional completers..

Initially, six students (85.1%) responded, and all six reported that they were working in an elementary school. Three were working in general education and three were working in special education. All six reported that their second degree helped them obtain their position over the other candidates. Additional comments described satisfaction with their program, such as "I feel much more comfortable as a teacher" and "I thinking staying and getting my Master's was the best decision I could have made" and "I'm thankful to have had such a great education to prepare me!"

For the second survey, eight responses (30.8%) indicated that having the special education Master's  "made me WAY more prepared and WAY more marketable," and "In that short year, I learned so many strategies and techniques that I use in my classroom today."  

In Spring 2023, 50 Master's level completers from Fall 2020 to Summer 2022 were emailed the same questions to look at responses across years and to consider differences with the addition of other undergraduate majors such as music, agriculture, secondary and middle level education that had participated in the 4 + 1 model.  Five teachers responded: four with an initial degree in PK-4 education and one with  secondary education.  Two of the five teachers are teaching out of state. Four of the five are teaching in a special education classroom with varying levels of students.  All five teachers indicated that their master's degree assisted them in not only obtaining their position, but also made them better prepared for the challenges of their classrooms.

Finally, embedded within the master survey of teachers is a question that asks if teachers have changed classroom, schools, or districts since they began their career.  For the Spring 2023 survey, 84 teachers responded to this question: 39 were in the same classroom, five changed buildings within the district, 11 are in a different classroom, and 29 have changed school districts.  Since over 50% of the responders were able to obtain different employment, we consider this as additional evidence of employability as a qualified teacher.