CAEP Accountability Measures

The EPP examined teacher effectiveness through the lens of addressing systemic inequities as guided by the College’s Strategic Plan for 2023-24.  The survey question guiding the inquiry was “Do you think that your preparation program has provided you with foundational skills and classroom practice so you can effectively work with diverse P-12 students, their families, and/or others in your career field?  Why or why not?”  

Of the 40 teachers who responded to the overall survey, 26 chose to answer this question.  The responses ranged as follows:  Yes (50%), Yes but a qualifier (31%), No (15%), and Other (4%).  Examination of explanations given by the teachers’ responses provided additional information to guide program improvements.  Among affirmative responses, eight teachers referenced field experiences, course content, and foundational skills. Two teachers identified that it was the varied field experiences that provided opportunities with diverse students and two teachers explained they had requested  an inner-city school for their field experience. One teacher responded they had confidence with classroom students but not as much with families. Another teacher’s comment also referenced families, but it was to report the need for working with closed-minded or biased parents and school administrators who are trying not to offend them. Two of the ‘No’ responses reported they had sought additional professional development to increase effectiveness. The Other response simply commented that the EPP locale was not very diverse. Several teachers mentioned ELL learners in their comments, and the EPP will be intentional in tracking these data in subsequent annual surveys after the modification of the ESL certification course sequence. The new sequence enables more candidates to pursue this add-on certification while completing their primary area of preparation and without adding additional time to degree.

The EPP continues to seek data to inform student learning growth.  Additional data from the Pennsylvania Department of Education (PDE) collected post-pandemic provides some further insight into student growth by completers who have completed their degrees within the last five years and have one to three years of experience.

The following Table 1 presents data that demonstrate the landscape of growth across the Commonwealth for completers regardless of EPP institution.  Overall, these measures indicate an alarming lack of student learning growth across content areas.  Subsequent tables provide EPP specific data by content area.  Note that the largest number of EPP candidates are in PK-4 certification while growth is tracked in content areas.

In all the tables, statistical significance of the p-values is indicated by * for p < 0.10, ** for p < 0.05, and *** for p < .001.

The following table presents our completers’ student growth in Mathematics. Our EPP student growth data are markedly better than overall EPP student growth data across institutions. These data demonstrate completers are better prepared than those completers from other EPP institutions. One hallmark of EPP requirements is that secondary mathematics candidates enroll in extremely rigorous mathematics curricula and accompanying pedagogy coursework. This is in part due to the legacy of preparing McNair scholars. While comparatively, EPP completers are demonstrating better student growth, these data illustrate much room for improvement. 

The following table presents student growth data from our EPP Middle level English Language Arts completers.  Compared to student growth data from other State EPPs, EPP completers’ students demonstrate significantly better growth in ELA 4-8.

The following table presents student growth data from EPP Middle level mathematics completers.  Student growth data are weak and negative.  Compared to student growth data from other State EPPs, however, EPP completers’ students nonetheless demonstrate significantly better growth in mathematics 4-8.

The largest number of EPP completers are certified in PK-4. Given teacher shortages, these candidates often teach in 4-8 classrooms. Some of candidates test for certification in 5-6 grades and may represent those in the following table, which demonstrates student growth by 4-8 content areas.  Others may be teaching in these classrooms with emergency certification.  Interestingly, despite limited student growth, these completers’ students’ growth across ELA, Mathematics, and Science, is considerably stronger than growth overall as presented in Table 1. 

As indicated in this AFI, the EPP has been challenged to measure student growth. These PDE data provide additional insight into completers’ student growth.  While these expanded data are only recently available, they represent three cycles of data across which there is notable stability.  Despite demonstrated strength in comparison to other EPP completer’s growth data, these data illustrate room for additional growth. The EPP considers varied measures of student growth and recognizes that student growth data are confounded by many variables. Some such contextual variables in Pennsylvania include discrepancies among per pupil funding, large variance in teacher salaries, and inconsistent induction programs. The EPP considers these PDE data an important part of understanding completer impact, while recognizing limitations given contextual variables within Pennsylvania, and that many EPP completers teach outside of the state.  Additional details about the growth measures reported by PDE are necessary to adequately assess the reliability and validity of these overall data.  Further, the number of completers represented across these data is unclear. Additional interrogation of these data is necessary for adequate interpretation.

Other student growth measures have also been employed. The EPP continues to administer the longitudinal completer teacher survey that provides insight into how practicing teachers measure growth in their students and classrooms. Recent concerted efforts to follow completers into the field, such as AI assisted completer searches, and intensive support from alumni relations, should result in increased participation rates and better student growth data.

Employer Satisfaction:

For the 2021-22 academic year, the EPP compared two years of aggregated survey responses from principals on where Penn State teachers need improvement to see how intentional changes in methods classes, dispositions, and student teaching seminars have made a difference.  In 2019, principals indicated inTASC 2 (diverse cultures), 5 (connect concepts), 6 (assessment), 7 (plan instruction) and 9 (professionalism) as areas that received scores below proficient.  In 2021, inTASC 2 (diverse cultures), 6 (assessment), 7 (plan instruction), 9 (professionalism)  and 10 (seek leadership roles) were areas that received scores below proficient.  Especially disappointing was the continued indication of additional work needed in diverse cultures, as the College has been implimenting anti-racism and equity into its curriculum, beginning in late 2020.  However the percentage of scores below proficient in almost every category decreased from 2019 to 2021, indicating that small changes have begun.  Puzzling was the added "seek leadership roles" as an area that needs improvement.  This has not been an area where Penn State teachers have typically faltered and the faculty questions if reduced field experiences in 2020 impacted new teacher confidence in seeking leadershp roles. This is an area that will receive additional scrutiny in the 2023 survey.

Also this year, in conjunction with field experience partner district visitations to discuss student teaching expectations and MOUs, the director of field experiences also asked principals and senior school personnel to share what they would like the program to know about first year teachers from Penn State, what the program is doing well and in what areas are the new teachers weak.  Overall, positive responses included the strengths and proficiencies that these teachers bring to their classrooms and buildings. Two common themes for improvement were management of courses and the idea to do the best they could do in unexpected moments - don't freeze.  As unexpected moments was also a theme that student teachers who took advantage of subbing experiences also reported, this topic will be discussed and reinforced during student teaching seminars.

Advanced Programs: Although the EPP has some of the same tracking completers into the field issues with the advanced preparation programs, there are some data for students completing Principal Certification programs. This program has begun surveying program completers to provide data for its SPA reporting and by looking at the question, "Did you change jobs after receiving your principal certification from Penn State?" the EPP can make judgments on the satisfaction of employers by the percentage of completers who were able to change positions. Of the 75 respondents who completed in 2016-2019, 17 of the 18 who changed jobs, did so into an administrative or supervisory role. Of the 58 completers who did not change positions, 15 were already in administrative or supervisory positions.

A second round of surveys were sent in 2022, and 20 of 37 responders indicated that receiving an advanced certification had a positive impact on their career including additional pay, additional responsibilities, promotions within the district, or new positions - all indications of Employer Satisfaction.

Because of the limited nature of the collected data, we developed a plan to collect additional data to demonstrate Employer Satisfaction. We modeled our Employer Satisfaction Survey from the Completer Survey administered and validated by institutions in the State of Mississippi. The instrument demonstrates sound psychometric properties as administered to completers of their advanced program. Aligned to CAEP key competencies and our Advanced Completer Survey, data from administration of the Employer Satisfaction Survey will inform advanced programs' continued improvement. As part of the 4.1 plan we demonstrate how we will assure the instrument meets standards for CAEP EPP-Created Assessments that provide data about our key competencies for Advanced Completers from all programs and allow for desegregated data and assessment of program-specific objectives. The 4.1 Plan includes a human subjects approved research study to further refine and test a measure that captures Employers' Satisfaction.

A test pilot of the survey was sent to two professional stakeholders, one for principal completers and one for superintendent completers.  These initial data show 100% Strongly Satisfied or Satisfied responses out of a rating scale of Strongly Satisfied, Satisfied, Dissatisfied, Strongly Dissatisfied.  Feedback for areas of improvement included the importance of developing, implementing, and measuring long-term strategic goals and measures, leading system-wide change processes including curriculum development for principals.  Feedback for superintendent development included high knowledge of personnel management, finance, contract negotiations, community relations, and personnel evaluations.  The EPP will review course curriculum for alignment with these feedback areas and discuss ways of strengthening instruction and assessment of these items. A full data collection is scheduled for June 2023.

While many institutions target a day for data and reflection to meet CAEP Standards, the EPP employs the use of the Data Roundtable series. Each session in the series targets a specific topic or issue. The EPP seeks feedback from district faculty and staff, EPP faculty, staff, and students, alumni completers, and alumni who serve in district leadership roles. These valuable sessions provide opportunity to share data with stakeholders, interrogate the data together, and listen to stakeholder feedback and suggestions for programs based upon these data.

Our Roundtable Talks for the 2024-25 year included:

Educator Pipeline Grant Revised

New Teachers - Today's Classrooms

New School Leaders - Today's Buildings and Districts

The last two talks of 2024-25 were driven by suggestions from the superintendent survey responses sent in early in the academic year.
 

Our Roundtable Talks for the 2023-24 year included:

Updating the Middle Level Grades 4-8 Program - Looking at a Two-Certification Model

Amended Public School Code of 1949 – Educator Pipeline Support Grant Program

Structured Literacies: A discussion with stakeholders on current practices, opportunities, and challenges

Our Roundtable Talks for the 2022-23 year included:

Developing and Supporting an Equitable Society through Education - ways to improve our preparation of educators through an equity lens

Recruiting Education Professionals into Shortage Areas Update - how are schools coping; feedback on new College initiatives

Mental Health in Our Schools – What is Working; What is Needed - what are schools doing; how can we prepare resilient teachers

Teacher Candidates as Substitute Teachers – One Year Later - feedback on a state law change that impacted student teachers

An additional example of intense data-examination for program review and potential revision was the need for all programs to undergo state review in 2021- 2022. Responses to PDE as a primary EPP stakeholder initiated with program faculty and PCCC and was informed by SPA data and non-SPA program data,  data derived through the QAS systematic use of data, and data from other stakeholders, including mentors, principals, and superintendents. Discussion of the PDE major review data was held at PCCC, and two retreats were held with program stakeholders to review and discuss essential data for their program reviews. PDE approved all CAEP and non-CAEP educator licensure programs at Penn State by July 1, 2022.

All students completing a teacher preparation program at Penn State must meet program course requirements by passing with a C grade or higher, maintain a 3.0 GPA overall in the program, and pass a PDE-required content exam. The following table of data presents three years of completers with Praxis or Pearson testing pass rates and the average pass score. For programs that have less than 5 students per year, the completer data have been aggregated into one or more blocks to preserve student privacy.

Program Completers with GPA and Certification Exam Pass Rates
AY 2019-20 through 2023-24

Advanced Completers

A fourth measure on program outcome that the College of Education is required to list in its annual report to CAEP is the ability of our program completers to be hired into positions for which they have been prepared. The Pennsylvania Department of Education (PDE) provides a list of Penn State students who have been hired in Pennsylvania and the school districts that have hired them. However, because a significant number of completers choose to seek teaching positions in other states, the College has been working on ways to maintain connections with those students using emails and surveys, so as to include their data in our self assessments.

The following tables reports PDE data on the number of Penn State completers hired into Pennsylvania school districts for  five calendar years at the initial and at the advanced program levels. Please note that 2024 initial completers were new to the job search process, and any recent hires would not yet be loaded into the PDE system.

A longitudinal study was started in 2017, a year after the first cohort of the new dual degree CEAED PK- Grade 4 bachelor's program with a one year Special Education PK- Grade 8 Master's degree graduated. Seven completers were emailed asking for feedback on the following questions: Are you working in an elementary school, are you in a general or special education classroom, and do you believe that your master's degree helped you obtain your position or made you better qualified.  In 2021 26 completers were contacted for their feedback, to compare responses between co-horts and with additional completers..

Initially, six students (85.1%) responded, and all six reported that they were working in an elementary school. Three were working in general education and three were working in special education. All six reported that their second degree helped them obtain their position over the other candidates. Additional comments described satisfaction with their program, such as "I feel much more comfortable as a teacher" and "I thinking staying and getting my Master's was the best decision I could have made" and "I'm thankful to have had such a great education to prepare me!"

For the second survey, eight responses (30.8%) indicated that having the special education Master's  "made me WAY more prepared and WAY more marketable," and "In that short year, I learned so many strategies and techniques that I use in my classroom today."  

In Spring 2023, 50 Master's level completers from Fall 2020 to Summer 2022 were emailed the same questions to look at responses across years and to consider differences with the addition of other undergraduate majors such as music, agriculture, secondary and middle level education that had participated in the 4 + 1 model.  Five teachers responded: four with an initial degree in PK-4 education and one with  secondary education.  Two of the five teachers are teaching out of state. Four of the five are teaching in a special education classroom with varying levels of students.  All five teachers indicated that their master's degree assisted them in not only obtaining their position, but also made them better prepared for the challenges of their classrooms.

Finally, embedded within the master survey of teachers is a question that asks if teachers have changed classroom, schools, or districts since they began their career.  For the Spring 2023 survey, 84 teachers responded to this question: 39 were in the same classroom, five changed buildings within the district, 11 are in a different classroom, and 29 have changed school districts.  Since over 50% of the responders were able to obtain different employment, we consider this as additional evidence of employability as a qualified teacher.