Chapman, Marketing SLOs

 

The Marketing of Student Learning Objectives (SLOs): 1999-2014 

Laura H. Chapman © 2014 Pre-publication Draft

For permission to duplicate or circulate please send an email to the author describing the intended use chapmanLH@aol.com

 

 

Abstract

The practice of using student learning objectives (SLOs) to rate teachers has taken root in K-12 public schools. SLOs are widely promoted as the major alternative to value-added models (VAM), a family of statistical methods for rating teachers based on their students’ scores on statewide tests. SLOs are said to be appropriate for rating the majority of teachers not covered by VAM, including teachers in the arts and other “untested” or “nontested” subjects.

The SLO process is a version of the 1950s business practice known as management-by-objectives modified with pseudo-scientific specifications intended to create an aura of objectivity in rating teachers. I briefly review four federally funded reports that portray the uses of SLOs. These reports point out the absence of evidence to support any use of SLOs other than securing teacher compliance with administrative mandates. I conclude with a critical commentary on the strategies employed in marketing SLOs as part of ”a new grammar and framework” for rating teacher performance.

Appendices explain how the structure of a typical SLO is now determined by a national system of data gathering funded by the Bill and Melinda Gates Foundation and U.S. Department of Education. I show how this system shapes SLOs and the guidance state officials offer to teachers’ “frequently asked questions” with Ohio as a case study. That guidance resembles the game known as “Mother/Father May I? with arbitrary and sometimes contradictory answers to questions.

Introduction

Many states and districts are rating teachers based on their students’ scores on statewide tests as required under several federal programs and state mandates. Among the federal programs are the No Child Left Behind Act of 2001 known as NCLB, the 2006 Teacher Incentive Fund, and Race to the Top (RttT) authorized under the American Recovery and Reinvestment Act of 2009 (P. L. 107-110, P. L. 111-5, P. L. 109-149).

These programs, along with NCLB waivers now require 43 states to evaluate the effectiveness of teachers, principals, and schools by using multiple and rigorous measurements, including “growth scores,” defined as gains in students’ test scores between two or more points in time (Fed. Reg., 2009, pp. 559751-52).

 

 

 

In order to meet this requirement, state officials typically rate teachers by using a family of statistical models that require a large number of test scores such as those produced by statewide tests for NCLB—reading and mathematics (grades 3-8, once in grades 10-12), along with science (at least once in grade spans 3-5, 6-9, and 10-12).

These value-added statistical models, dubbed VAM, are supposed to isolate the effect of an individual teacher on the scores of their students and exclude the influence of other factors. The algorithms in VAM (e.g., linear regressions) are known to produce wildly unstable scores for individual teachers. This instability is made worse by relying on this score to place teachers in one of four or five reductive categories such as highly effective, effective, developing, or ineffective (acronym is HEDI). This process is deeply flawed.

After more than fifteen years of reports on the problems with using VAM for rating teachers, the American Statistical Association finally issued a statement to notify policy makers and others that “value-added” statistical models should not to be used for rating individual teachers (American Statistical Association, 2014). Such ratings are unreliable, invalid, and unfair. So are the HEDI classifications.

Up to 90% the variation in students’ scores is likely to be explained by factors teachers cannot control (Schochet & Hanley S. Chiang, 2010, p. 35). Even so, the practice of using VAM continues because federal regulations are difficult to change. In addition, state legislators and other policy makers have responded to intensive marketing of VAM as: (a) an objective method for teacher evaluation, and (b) an essential measure for identifying effective teachers (e.g., Reform Support Network, 2013, July).

The attention given to VAM in teacher evaluation has overshadowed the fact that about 70% of teachers have job assignments for which statewide tests are not available. This fact has led state policymakers to adopt a comparable system for rating teachers based on their ability to meet measurable “student learning objectives” (SLOs). The use of SLOs as a means for rating teachers is part of a broader federal project to make pay-for-performance the national norm for teacher compensation (Note 1).

 

This preoccupation with ratings and other forms of measurement is one manifestation of what I have called the econometric turn in federal and state policies. The econometric turn is most evident in the treatment of educational issues as managerial problems and the reification of metrics, especially test scores, as if these are objective, trustworthy, and essential for making educational decisions (Chapman, 2013).

In addition to describing and commenting on characteristics of the SLO process of evaluation, I offer highlights from four recent reports on SLOs from the Institute of Education Sciences, an agency of the U.S. Department of Education (USDE). These reports are intended to: (a) portray the research literature on SLOs for teacher evaluation, (b) indicate state requirements for the use of SLOs, (c) summarize lessons learned from districts that were “early adopters” of this evaluation strategy, and (d) illustrate how states comply with RttT mandates. I conclude with comments about the marketing campaign for SLOs in teacher evaluation, especially in connection with efforts to transform teaching into a pay-for-performance occupation.

Options for Measures of

Achievement and “Growth”

States have three alternatives for rating teachers whose job assignments are not linked to statewide tests. The USDE-funded Reform Support Network (RSN) propagates these options (Note 2). RSN is a loose structure of agencies and sub-contactors who advise states on complying with federal policies (e.g., Southwest Comprehensive Center at WestEd, n.d.).

Option One: VAM with Alternative Tests. States may use VAM with alternative tests that meet the psychometric standards of the American Educational Research Association (2013). The most common alternatives are the Stanford Achievement Test, Terra Nova, and Iowa Test of Basic Skills and end-of-course tests that meet psychometric standards (e.g., California’s high school history and science tests).

Option Two: A Collective VAM. In this method, individual teachers are assigned the VAM score on a “collective measure.” In Florida, for example, teachers of art can be assigned the school’s average value-added score on the reading portion of the statewide test known as FCAT (Florida Department of Education, 2013). Moreover, this practice is legal, even if it is unfair. A U.S. district judge has ruled that Florida can disregard a teacher’s job assignment in evaluating his or her performance (Robinson, 2014).

Teacher ratings based on a collective measure are likely to increase in states where Common Core State Standards (CCSS) are adopted and tested. This strategy is a rapid and cost-effective way to meet requirements for teacher evaluation because all teachers must help students meet the CCSS standards for English language arts, literacy, and mathematics.

Option Three: Student Learning Objectives

Student learning objectives (SLOs) are a version of the business practice known as management-by-objectives (Drucker, 1954). In brief, lower-level managers identify measurable goals and “targets” to be met. A manager of higher rank approves the goals, targets, and measures. Lower-level managers can earn a bonus if they attain or exceed these targets (e.g., a 15% increase in sales within four months).

States and districts are using versions of this process for teacher evaluation and in pay-for-performance plans. SLOs are sometimes called “student learning targets,” “student learning goals,” “student growth targets (SGOs),” or “SMART goals”—Specific, Measurable, Achievable, Results-oriented and Relevant, and Time-bound. Although definitions and acronyms vary, all SLOs are a strategy for monitoring teacher productivity. SLOs facilitate administrative audits of the goals that teachers set, their methods of instruction, and the tests that teachers use to measure student achievement. Computer software facilitates this system of surveillance and workforce management.

 

Key Components of the

SLO Evaluation Process

The most widely copied SLO process originated in Denver Public Schools in 1999. Two regional foundations and the California-based Eli and Edythe Broad Foundation funded a four-year pay-for-performance initiative designed and managed by the Boston-based consulting firm, Community Training and Assistance Center (CTAC, 2004). CTAC placed student growth objectives at the center of this plan. According to William J. Slotnick, Founder and Director of CTAC, the intent was to “add science to the art of teaching” (Reform Support Network, 2011a, p. 1).

Here are the bare bones of the process. Early in the school year teachers identify learning targets for their classes and the tests they will use to measure learning. District officials may set a minimum threshold for an acceptable target (e.g., 100% of students who attend 85% of classes will score at or above 60 on the 75 item end-of-course test). A trained evaluator rates this plan using a four or five point scale from “high quality” to “unacceptable” or “incomplete.” Before the end of the year (or course), teachers are rated on the proportion of students whose test scores met the targets, failed to meet them, or exceeded expectations.

Other accoutrements of the process are evident in Table 1, an example for art teachers in Denver (Denver Public Schools, 2013). The eight categories in the Denver example are said to be essential for a complete SLO (American Institutes of Research, 2012, p. 20).

 

 

Table 1.

Denver Public Schools: Criteria for a high quality SGO for art teachers, 2013

Rationale. Gives details on how this objective is completely aligned with one or more district, school, and/or department goals.

Population. Identifies the number of students for whom the teacher provides direct instruction. Unless the teacher and principal agree otherwise, the population includes 100% of students who are expected to be present for the pre-and post assessments. Assumes 85% attendance.

Interval of Time. Clearly identifies the time period provided in the school schedule for the intended instruction. The time period is appropriate.

Baseline Data. Gives a detailed class profile, including a disaggregated summary of pretest data that shows the performance levels of subgroups.

Assessment/Alternative Measure. Uses three or four tests and/or alternative standardized measures such as a “body of evidence” recommended by the district, and agreed upon by the teacher and SGO approver. Pre-approval is required for any teacher-made test or collaborative performance assessment/project. Approved assessments/measurements rigorously measure intended growth, gain, or change identified in the objective for learning.

Expected Growth. Analyzes initial data for each student—test gains and other trends in past performances. Predicts gains in learning from the beginning to the end of the instructional period. Class averages may not be used. Identifies performance targets for each student based on these predictions. If tiers of expected performance are targeted (e.g., greater gains for students who scored low on the pretest) a gain must still be projected for 85% to 100% of the population. High quality SGOs target more than a one-year gain in proficiency. Targets must be corroborated by one or more external sources.

Learning Content. Clearly states what pupils will know and be able to do by the end of the interval for instruction. Expectations for learning are set using baseline data to identify and target the individual needs of students and the identified population. Content to be learned will include three or four grade-level Common Core Standards or four to six essential learning goals, and where appropriate 21st century skills such as cooperative learning, critical thinking, and problem solving.

Strategies. Identifies five to eight research-based teaching strategies that are differentiated for all learners, meet their needs for learning the content, are observable, and can be documented.

Note: I have lightly edited the Denver example to eliminate redundancy. Italics were in the original. Source: Denver Public Schools. (2013). Welcome to student growth objectives: New rubrics with ratings. Retrieved from http://sgoinfo.dpsk12.org/

 

 

The categories and criteria for a typical SLO forward the illusion that every step in the process is scientific. Thus, students in classes are dubbed a “population.” Records from prior grades become “baseline data” for profiling and grouping students. “Expected growth” is a prediction of posttest scores, but stripped free of any theory that might leverage reasoning about the expected outcomes. “Growth” is a euphemism for an expected increase in scores, pretest to posttest, even if scores are based on rubrics.

In effect, SLOs are framed and rated as if the teacher is documenting a one-group pretest-posttest experiment for the population named in the SLO, but with no control group, and with an arbitrary demand for multiple standards, measures, and research-based teaching strategies. Given all of these variables and criteria, no reliable and valid inferences can be made about the effect of the teacher on the posttest scores. None. In this respect, the use of SLOs to justify judgments about a teacher’s effectiveness is not only blatantly unscientific but also unethical (Berliner, 2002; Popham, 2014).

In a typical SLO process, raters look for evidence that a teacher sets a “high quality” and “rigorous” learning target by aiming for “more than a one-year gain in proficiency.” Achieving that outcome corresponds to the federal definition for a highly effective teacher (Fed. Reg. 2009, p. 59805). “More than a one-year gain in proficiency” is a calculation to determine whether the scores of students assigned to a given teacher exceeded the average rate of increase in scores produced by other teachers who have a job-alike assignment. This definition and calculation also means that the evaluation system is designed to produce ratings from data arrayed with “stretch,” much like a bell curve. Thus, about half of teachers typically attain a safe harbor rating and about half do not.

The apparent scientific precision of the SLO process is an illusion. The metrics are decoupled from reasoning about how and when a given proficiency may be attained, and why it may educationally significant (Ligon, 2009).

The Denver template for writing an SLO has variants in other states (e.g., Ohio State Department of Education, 2014). Reformers who promote SLOs seem to assume that for every subject, every grade level, and every standard there also exists a deep reservoir of teaching methods differentiated for groups and individual students—all backed by current empirical research. These assumptions are without any foundation in fact. The practice of requiring teachers to write measurable student objectives is not entirely new, but it has been fashioned into a blunt instrument to support test-driven teaching and an unparalleled effort to micro-manage the work of teachers.

Reports on the Use of SLOs for Teacher Evaluation

Four recent reports from agencies funded by USDE illustrate the use of SLOs for teacher evaluation. All of the reports are intended to inform the work of policymakers who are dealing with teacher evaluation and pay-for performance issues.

 Report 1: Using alternative student growth measures for evaluating teacher performance: What the literature says (Gill, Bruch & Booker, 2013). This literature review treats SLOs as one of several alternative growth measures. Appendix A describes the literature search and sources of citations. Appendix B summarizes information from citations bearing on alternative student growth measures. Appendix C is comprised of 37 citations that refer to SLOs—the most relevant topic for teachers of art and other untested subjects.

Appendix C is comprised of five spreadsheets that categorize the literature on SLOs according to the presumed interests of readers. These spreadsheets include: Statistical studies (Table C-1, seven citations); implementation lessons and surveys of participants (Table C-2, nine citations); where and how SLOs have been used (Table C-3, 12 citations); and types of data collected from participants (Tables C-4, nine citations). In Table C-5, all 37 citations for SLOs are called “studies.” Most citations are not analytical or empirical studies but descriptions of rating systems that include SLOs.

Empirical evidence about the use of SLOs is not abundant. It is dominated by reports on pay-for-performance plans. Most of these plans were jump-started by special funding from foundations. Only four citations included data from participants—surveys, focus groups, and interviews. Surveys had an average response rate of about 62%.  In this respect, evidence from participants in the SLO process is neither abundant nor rigorous.

A major theme in reports on pay-for-performance is that most teachers complied with the SLO process after three to four years of training. They learn to teach for the tests specified in their SLOs (Table C-1). Promoters of SLOs interpret this high level of teacher compliance as evidence of improved teaching achieved through a “best practice” (e.g., Reform Support Network, 2011a, p. 4).

Among the “lessons learned” from the use of SLOs, the most salient are not positive.  Participants reported that the software and templates for SLOs distorted the intentions of the teacher. The process was complex and time consuming; especially finding tests or creating new tests and updating data from tests. Teachers did not think the process changed their practice in significant ways, and many believed the process did not distinguish between effective and ineffective teachers. Perceived fairness was undermined by factors not under the control of teachers including student attendance, mobility, and dropouts (Table C-2).

Readers seeking empirical research on the use of SLOs will find little evidence in this report to forward that cause. Only two citations bearing on SLOs were published in peer-reviewed journals. Both reported on SLOs in connection with pay-for-performance plans, one in Denver, Colorado (Goldhaber & Walch, 2012) the other in Austin, Texas (Brodsky, DeCesare & Kramerwine, 2010).

This report clearly states: “…no studies of SLOs have looked at reliability” (p. ii). Not one. Nor has the validity of this process for teacher evaluation been established. Only three studies considered validity and only by seeking correlations with VAM and standardized tests (p. ii). Given that VAM ratings are unstable and not relevant to content and standards in “untested” subjects, it is clearly a mistake to think that correlations of VAM with SLOs are meaningful indicators of the validity of each other.

The authors have exactly one recommendation for policymakers: Only use SLOs for instructional planning, not for evaluation (p. ii). This recommendation cavalierly ignores the reality that states and districts are already using SLOs for high stakes decisions, including compensation.

The literature review omitted one of the most nuanced and thorough criticisms of the use of SLOs for teacher evaluation (Marion, DePascale, Domaleski, Gong & Diaz-Bilello, 2012), as well as issues in quality control for SLOs (e.g., American Institutes of Research, 2012; Prince, Schuermann, Guthrie, Witham, Milanowski & Thorn, 2009).

By design, this report provides a gloss of academic respectability for a deeply flawed teacher evaluation strategy. The basic function of SLOs in education today is not different from Drucker’s outdated management-by-objectives in business, combined with mid-twentieth century reforms that focused on measurable behavioral objectives, especially for programmed instruction (e.g., Gagné & Briggs, 1974; Gronlund, 1970; Mager, 1962). A more thoughtful literature review would acknowledge this legacy and its revival under the banner of reform  (Note 3).

Report 2: How states use student learning objectives in teacher evaluation systems: A review of state websites (Lacireno-Paquet, Morgan & Mello, 2014). This report is based on information about SLOs posted on the official education websites of 50 states and the District of Columbia. The researchers found only 30 websites with retrievable SLO information. In addition to that limitation, the report is poorly planned. For example, Appendix A is comprised of spreadsheets labeled A.1 to A.6, with no obvious analytical structure. The topics encompass the use of SLOs in teacher evaluation; definitions, features, and characteristics of SLOs; and brief notes on additional requirements, state-by-state.

Appendix B is a straightforward list of websites that display SLO templates, examples, and other guidance. The report ends with a list of five references. Only one has a focus on SLOs, and it was written to market SLOs for teacher evaluation (Race to the Top Technical Assistance Network, 2010).

I constructed Table 2 from data scattered in several appendices. It portrays the most common SLO policies in rank order of apparent salience in websites.

 

 

Table 2.

SLO policies cited in state education websites by number of states

26    SLOs required for individual teachers or for a job-alike team

21    All teachers must prepare SLOs

21    A principal or district leader/evaluator must approve SLOs

19    SLOs must identify measurable goals/targets for learning

17    Teachers must prepare their own SLOs

14    SLOs must apply to all of a teacher’s students

14    SLOs may use national or statewide tests of student learning

12    SLOs must include gains in test scores (growth measures)

12    SLOs may use district or schoolwide measures

12    SLOs may use classroom tests

10    Evaluators must review SLO outcomes with teachers

9    SLO goals are set by teacher collaboration or with evaluators

7    SLOs must include differentiated growth measures for subgroups

6    SLOs are only required for subgroups of students

5    Tests must be comparable across classrooms

3    Tests must be reliable and valid

3    SLOs are required only for teachers of “untested” subjects

3    SLOs must be schoolwide

2    Tests must be rigorous

2    Tests must be aligned with state standards

Data from Lacireno-Paquet, Morgan & Mello, (2014). How states use student learning objectives in teacher evaluation systems: A review of state websites.

 

 

Several trends are apparent in Table 2. First, most states appear to favor the use of SLOs as a management strategy for all teachers. Many states also call for approval of SLOs by the principal or a district evaluator. Only ten states appear to require a face-to-face review and discussion of the SLO outcomes with teachers. Second, there is no obvious consensus on policies about SLO tests. Third, only a few states appear to be complying with federal requirements that tests for a given grade and subject must be comparable across classrooms. This policy is intended to guarantee that test scores have sufficient ”spread” for creating a rank order of student and teacher performances.

Fourth, and perhaps most important, few states appear to be addressing a major concern identified by policy makers—the reliability, validity, and alignment of SLO tests with standards in diverse subjects (Reform Support Network, 2012a, 2012b).

These and other unresolved issues about the integrity of tests should raise questions about the legitimacy of the whole SLO process for teacher evaluation. It is possible that state websites do not have easy-to-retrieve information about test integrity. Perhaps states have placed the whole matter of test integrity on the back burner. This review of state education websites does not help us know which is the case.

This report does not add much to our understanding of SLOs as an alternative to VAM in teacher evaluation. Indeed, the authors actually conflate SLOs and VAM as if there is no major difference: “SLOs may involve student growth models, such as value-added models or student growth percentiles” (p. 1). This language is totally at odds with the most developed rationales for SLOs; namely, as an alternative to value-added and other statistical models.

Report 3: Alternative Student Growth Measures for Teacher Evaluation: Profiles of Early-Adopting Districts (Gill, English, Furgeson & McCullough, 2014). The districts selected for this report are called “early adopters” of teacher evaluations based on SLOs or alternatives tests suitable for VAM calculations. All of the districts received financial support from external agencies and foundations for initiating pay-for-performance plans. The amount and duration of external financing is not reported. Only four of the eight districts adopted the SLO process.

In the four SLO districts, all K-12 teachers were engaged in a similar process. Early in the school year, each teacher identified strengths and weaknesses of students using available data, including test scores from prior years or pretests in the current year. Using this baseline information, teachers identified learning goals aligned with standards, identified tests to measure student achievement, and set targets for gains in achievement from pretest to posttest. Principals reviewed and approved the SLOs. Teachers were rated on the gains in their students’ pretest to posttest scores.

In three districts, teachers and district staff were organized in “design teams” and initially guided by outside consultants. In three cases, a district official approved SLOs. Three districts treated SLOs as a means to improve instruction, not just teacher evaluation. Two of the districts required teachers to identify and justify their instructional strategies.

Links in this report provide more information. For example, researchers in the Austin Independent School District (AISD) have studied the AISD pay-for-performance plan. It is based on multiple measures, including SLOs. One study noted that teachers learn to select SLO content likely to produce high scores on tests and also qualify them for a bonus (Schmitt, Lamb, Cornetto & Courtemanche, 2014).

Report 4. The View from the States: A Brief on Nontested Grades and Subjects (Reform Support Network, 2013). This eight-page publication reports on a seminar convened by USDE in 2013 with representatives from three states for the purpose of exchanging information on RttT requirements for teacher evaluation. One state, Tennessee, is of special interest because it is modifying the SLO process in order to rate teachers of art, dance, music, and theater.

The Tennessee initiative received state approval in 2012. The SLO process includes a masked peer review of student work called “evidence collections.” Art teachers assemble these collections in a digital portfolio that also includes other documents for the evaluation. A dedicated online site facilitates the process.

Many of the criteria for submitting a portfolio are linked to the concept of “purposeful sampling.” For example, the teacher must select samples of student work from two points in time (comparable to a pretest and posttest) in order to represent student growth. A teacher must submit five evidence collections. Each collection must be coded to identify the specific state standards for arts learning addressed in the lessons or units. The evidence collections must include targets for learning in three of the four major domains in state arts standards: Perform, Create, Respond, and Connect.

The online template offers guidance for submitting portfolios and understanding how the scoring system works. In this system, the art teacher rates the evidence collections—a form of self-evaluation. This self-evaluation becomes part of the portfolio. Then two exemplary art teachers with job-alike experience independently rate the portfolios—a form of masked peer review. These raters have been trained to use rubrics for the evaluation. The final rating places the teacher into one of six levels of performance from “significantly below expectations” to “significantly above expectations.” A third rater may be enlisted to ensure the final rating has a consensus.

In Tennessee, the “student growth” measure counts for 35 percent of a teacher’s overall evaluation. By 2013, 1,500 art teachers had been evaluated by this method. The plan is still a work-in-progress. It began with an extraordinary collaboration among teachers in the arts, community leaders in local and state agencies, scholars, and other experts in the arts and evaluation (Tennessee Department of Education, 2013).

 

The Econometric Turn and

Accountability Imperative in Federal Policies.

Federal policies for education forward an economic concept of growth as a measurable gain in learning with test scores given priority as if these are objective and an imperative for systems of accountability. This “accountability imperative” is evident in key definitions within RttT legislation.

“Student achievement means (a) For tested grades and subjects: A student’s score on the State’s assessments under the ESEA; and, as appropriate, other measures of student learning…provided they are rigorous and comparable across classrooms. (b) For non-tested grades and subjects: Alternative measures of student learning and performance such as student scores on pre-tests and end-of-course tests; student performance on English language proficiency assessments; and other measures of student achievement that are rigorous and comparable across classrooms” (Fed. Reg., 2009, p. 59806). Tests that teachers create for use in their own classrooms are not acceptable.

“Student growth means the change in student achievement for an individual student between two or more points in time” (Fed. Reg., 2009, p. 59806). I constructed a line graph  (Figure 1) to illustrate how a “trajectory” of learning toward a growth target is implicitly envisioned as a tidy and continuous upward line suggesting “continuous improvement.” The graph fits the federal “Race to the Top” metaphor and fantasy.

Figure 1.

Points 1 and 2 represent the distance to be traveled; that is, the content to be covered and learned between the start and end of the course. Five groups of students (colored coded lines) must learn at different rates to meet or exceed the acceptable end-of-course cut score (here 70). The learning trajectory is steeper for students who begin with low scores. They must learn more, and at a faster pace, than other students. The SLO process, like the Race to the Top slogan, treats education as a “time, distance, and speed” problem. Trajectories for gains in learning of the kind in Figure 1 are also an essential feature of rating teachers “effective,” “highly effective,” or “ineffective.”

“Effective teacher means a teacher whose students achieve acceptable rates (e.g., at least one grade level in an academic year) of student growth.” “Highly effective teacher means a teacher whose students achieve high rates (e.g., one and one-half grade levels in an academic year) of student growth…” (Fed. Reg., 2009, p. 59805). The focus on rates of growth in scores is not different from asking whether profits in a business are increasing, decreasing, or about the same.

The importance attached to VAM and SLOs in teacher evaluation (especially for high stakes personnel decisions) cannot be justified by claims that these measures are reliable and valid. In addition to that serious flaw, the federal definitions of student achievement, student growth and effective teacher—based on corporate accounting and management principles—are so alien to the educational thought and practice that USDE has funded a full scale marketing program to secure compliance with these measures.

The marketing campaign on behalf of RttT

USDE’s marketing campaign began in late 2010 with a $43 million grant to IFC International, a for-profit consulting and public relations firm.

The grant was for two purposes: (a) to create the Reform Support Network (RSN) enabling Race to the Top grantees to learn from each other, and (b) to promote promising practices for comparable reforms nation-wide. The grant included $13 million for nine subcontractors, each with specialized skills for RSN’s marketing campaign.

I do not use the phrase “marketing campaign” lightly. RSNs publications and media productions ostensibly offer states and districts “technical assistance.” However, almost all RSN documents have this disclaimer: “This document was developed by the Reform Support Network with funding from the U.S. Department of Education under Contract No GS-23F-8182H. The content of this publication does not necessarily reflect the views or policies of the U.S. Department of Education.” Some have the additional disclaimer: “Inclusion of this information does not constitute an endorsement by the U.S. Department of Education of any products or services offered or views expressed, nor does the Department of Education control its accuracy, relevance, timeliness or completeness.”

In other words USDE has outsourced its communications about policies to contractors who: (a) gather case material from RttT grantees, (b) add their own content, and (c) obscure the actual source of those additions. For example, RSN publications rarely identify the authors who are responsible for the content. Scholarly citations are rare. They come from a limited array of sources and few are from peer-reviewed publications. A sophisticated graphic design (much like a brand) identifies over three dozen RSN publications and media productions, but RSN has no address other than USDE’s website (Note 4). In effect, USDE hosts and propagates interpretations of its policies, but USDE does not endorse these interpretations or require marketers to engage in checking facts..

Here is one example of RSNs work.  In December 2012, anonymous contract writers for RSN published a portfolio of suggestions for marketing key policies in RttT.  Engaging Educators, A Reform Support Network Guide for States and Districts: Toward a New Grammar and Framework for Educator Engagement is addressed to state and district officials. It offers guidance on how to persuade teachers and principals to comply with federal policies bearing on pay-for-performance plans. Such plans usually depend on ratings calculated from multiple measures, including so-called growth scores.

Engaging Educators begins with the premise that RttT policies do not need to be changed. The policies are just misunderstood, especially by teachers. The solution is to deliver knowledge about RttT in formats most likely to secure compliance. Engaging Educators then packs about 30 communication strategies, all portrayed as “knowledge development,” into four paragraphs about “message delivery options.” These include “op-eds, letters to the editor, blast messages, social media, press releases,” and regular in-house techniques (p. 4). RSN writers emphasize the need to “Get the Language Right,” meaning that communications should by-pass “gotcha” talk—the idea that teachers can lose their jobs—and also avoid excessive “happy talk.” Instead, messaging should focus on improving student learning (p. 6).

RSN writers recommend that officials improve other aspects of their “messaging” for teachers. Among the suggested techniques are teacher surveys, focus groups, websites with rapid response to frequently asked questions, graphic organizers integrated into professional development, websites, podcasts, webinars, teacher-made videos of their instruction (vetted for SLO compliance), and a catalog of evocative phrases tested in surveys and focus groups. These rhetorical devices help to maintain a consistent system of messaging. RSN writers also suggest that districts offer released time or pay for message delivery by “teacher SWAT teams that can be deployed at key junctures of the… redesign of evaluation systems” (p. 9).

Another recurring theme is the need to enlist union leaders and other “teacher voice groups” as advocates for the rating systems in pay-for-performance plans. A “teacher voice group” is RSNs name for a non-union advocacy collective funded by private foundations favoring pay-for-performance. Five voice groups are mentioned by name. All have received major funding from the Bill and Melinda Gates Foundation: Teach Plus ($9.5 million), Center for Teacher Quality ($6.3 million), Hope Street Group ($4.7 million), Educators for Excellence ($3.9 million), and Teachers United ($942, 000). Other foundations are supporting these groups. For example, Teach Plus receives “partner” grants from eight other foundations including the Broad, Carnegie Corporation of New York, Joyce and several investment firms.

The marketing campaign for SLOs

RSN’s most recent marketing effort is A Toolkit for Implementing High-quality Student Learning Objectives 2.0 (2014, May). This booklet is addressed to state officials responsible for teacher evaluation. The narrative includes case material from leaders in a dozen state departments of education augmented by ideas from anonymous marketing experts.

Like other publications from RSN, A Tookit frames the discussion of SLOs as if this evaluation process is not different from the evaluation systems of “many American industries” (p. 5-6). A Tookit omits the historically important information that SLOs are a version of the 1950s practice of management-by-objectives (MBO). The most successful CEOs and personnel managers abandoned MBO long ago in favor of practices that enhance an organization’s work environment. MBO failed to honor the essential front-line workers in the business. Instead, it rewarded workers who were the most competitive and those who gamed the system. In addition, MBO created a maze of paperwork that one expert dubbed a product of “bureaupathology” (West, 1977).

Writers for RSN publications are probably aware of this dismal history. In order to cast the most positive light on SLOs, they focus on four points in the hope of inducing compliance with the process.

1. The SLO process is collaborative. Collaborations are usually voluntary and directed toward shared goals and benefits. Teachers and principals do not volunteer for the SLO process. The process may require discussions among teachers and between the teachers and the principal or evaluator, but the evaluator is responsible for using that information in a high stakes performance appraisal.

States do not routinely call for face-to-face discussions of the outcomes from the SLO process. The “collaborative” marketing pitch is at best a marginal claim of benefit (pp. 5-7). This claim is unwarranted when teachers’ evaluations are churned out by computer software. The Tennessee portfolio evaluation is among the few examples of a collaborative effort to reframe the SLO process. Even so, that system retains annual ratings without face-to-face discussions between teachers and the raters.

2. The SLO process is adaptable. Within RSN’s marketing, the benefit of “adaptability” only refers to the prospect of rating all teachers, not just those with VAM scores. The SLO process is supposed to “promote parity in expectations and credibility” between ratings of teachers in nontested grades and subjects and teachers whose ratings include VAM (p. 5).

RSN acknowledges that adaptability—allowing teachers to create or choose assessments and set their own targets—will compromise the reliability, validity, and comparability of teacher ratings. In order to address those problems, writers of the Toolkit highlight the advantages of district-level standardization of the SLO process—extending to objectives, learning targets, assessments, and rating rubrics for SLOs.

RSN writers portray standardization at the district level as a way to reduce a teacher’s workload. Computer-based systems for data entry can also increase the consistency, credibility, and comparability of SLO ratings, especially if the district pre-populates parts of the template such as baseline data (Note 5). In effect, the policy preference is for district standardization of SLOs (pp. 7-8, 10-13, 27).

RSN’s preference for a less-than-flexible SLO policy is reinforced by a discussion of the concurrent “rollout of college-and career-ready standards and aligned standardized assessments” with new requirements for teacher evaluations (p. 8).

RSNs writers favor compliance with the CCSS and other measures praised in a report from the Aspen Institute (Weiner, 2013). The Aspen Institute report, linked to the Toolkit, forwards a model for teacher evaluations based on: (a) student scores on state tests (VAM with SLOs a proxy for VAM), (b) classroom observations, and (c) student surveys.

The writers for RSN not only support these three methods of teacher evaluation but also add an astonishing and totally unsupported claim that: “An evaluation with multiple measures (student growth on State tests, classroom observations and student surveys) can accurately predict a teacher’s effectiveness” (p. 8). The writers offer no evidence. There is none.

These three measures do not predict anything. They are simply used to define teacher “effectiveness” and by insular and circular reasoning (Rothstein & Mathis, 2013). This circular reasoning is intended to exclude other considerations from teacher evaluations especially the influences of experience and advanced degrees that are not mapped by annual evaluations and test scores including, for example, professional awards and unsolicited praise from parents, peers, former students (e.g., Gibson & Dembo, 1984).

3. The SLO process improves instruction. This claim is not based on empirical evidence. It is really an assertion about the proper role of a teacher; namely, a person whose instruction is driven by unambiguous and measurable objectives, aligned with standards, and tied to instructional strategies selected to ensure that students meet specific learning targets. A teacher who is SLO-compliant engages in constant monitoring of student progress toward learning targets with no delays in corrective action, thereby striving for “continuous improvement” (RSN, 2012, pp. 12-17).

The process of writing and scoring an SLO assumes that content and skills for a course or year should be fully mapped and mastered by all students. The process also assumes that intended learning during a year is part of a vertically aligned system of year-to-year prerequisites and learning progressions. Thus, students who have not mastered these prerequisites, identified by baseline data, must learn at a faster rate in order to keep up with others (Ligon, 2009; Chapman, 2012).

SLOs are intended to “ensure that every minute of instruction is moving students towards higher levels of achievement” (Indiana State Department of Education, 2012, p. 5). I think it is fair to say that the SLO process honors teachers who engage in direct instruction of the kind associated with training. Training may be an aspect of education—marked by clear standards and well-honed methods of securing mastery—but training is not the same as education.

The difference between training and education is not trivial. Education is about learning to address non-routine problems; learning to ask questions for which the answers may not be known, or may be ambiguous; and learning to initiate inquiries and projects. Education means students are learning to ask why something is worth doing and thinking about—what life offers and may require beyond completing assignments and taking tests in school.

4. The SLO process improves student learning. This claim should be stated as a hypothesis. The central problem is that “student learning” is routinely abridged to mean scores on tests and assigned tasks (including rubric-based scores) as if these limited indicators of achievement are necessary, sufficient, and irrefutable evidence of the academic strengths and weaknesses of students and essential for rating teachers.

The practice of using SLOs (and VAM) for teacher evaluation is wrong. That is not just my opinion. It is well known that a student’s performance in school is influenced by factors teachers cannot control far more than instruction in school. Among these factors are inherited conditions; pre-natal and infant care; parental education and income; congruence in language and dialects spoken at home and in school; “food security;” nurturing peer and adult interactions at home, in school, and beyond; access to timely medical care; a dedicated place to sleep in a vermin and lead-free environment (e.g., Coleman et.al., 1966; Alaimo, Olson & Frongillo, 2001; Berliner,  2012).

Achievement and learning are the result of a brew or stew of activities, mostly hidden-from-view. No algorithm can predict what may be salient, memorable, and intelligible for a given student. Teachers and students are constantly engaged in making inferences based on partial information.

It is not a self-evident truth that the SLOs process improves student learning at every grade and in every subject.  I have yet to find experimental research that compares this major intervention into the work of teachers with other possible interventions and comparable investments (money, time, training, technology, oversight).

CTAC (2004, 2008, 2013) is a major source of claims that SLOs improve instruction and student learning. The most comprehensive CTAC study (2013), funded by USDE, documents the hazards of any multi-year effort to initiate a pay-for-performance plan with SLOs. This study attempted to link gains in learning to the quality of thinking and writing evident in a teacher’s SLO.

CTAC developed a four-rubric system for grading SLOs from “excellent” (earning a four) to “too little.” The criteria in the rubrics focus the attention of raters on: (a) the content of the SLO, (b) the expectations that teachers set, (c) the completeness of the document, and (d) the general coherence of the SLO as a written assignment.

After three years of training and practice, some of the teachers who earned a higher rating for complying with the criteria for this writing assignment also had classes that produced higher test scores. In other words, teachers learned to comply with the details of the process, including the selection of targets and tests likely to indicate students are learning. The rubric for judging SLO quality is more accurately described as a measure of teacher compliance with a writing assignment and skill in playing the SLO game.

The researchers who conducted this study offered no information about student learning beyond limited and uneven gains in scores in mathematics and reading, and not in every grade (CTAC, 2013, pp. 5-7). Furthermore, these marginal gains cannot be attributed to the teachers or to the “quality” ratings assigned to their SLOs. No information was gathered on achievement in the subjects and grades for which SLOs are supposed to be most relevant, including studies in the arts.

The most prominent finding in this multi-year study is that SLO compliance and outcomes were thoroughly compromised due to poor communication, budget cuts, changes in administrators and staffing, changes in software, changes in scheduling, and so on.

Concluding Observations

In closing, I wish to highlight two of the many troubling dimensions of the SLO process. These bear on the effort to micromanage teachers and a corruption of language and concepts about education.

 Micromanaging the work of teachers. SLOs (like VAM) enable reductive thinking about educational problems. The main educational problem in portrayed as an “achievement gap.” Reducing the gap is simply a matter of managing teachers so they work more efficiently and effectively. Measuring efficiency (amount of learning per unit of time invested in teaching) and measuring effectiveness (gain in the amount of learning per unit of time) can be done “objectively.” Good teaching is effective, meaning cost-effective in time and resources.

The practical import of this view is evident in the inordinate amount of time and money invested in evaluating SLOs for accuracy and completeness, and in seeking a match between the details in a teacher’s proposed plan and the details an evaluator wishes to see. Under the banner of accountability, evaluators are determining the content and aims of instruction and demanding absurd levels of documentation for every student and every aspect of content and instructional strategy.

For example, an RSN website featuring expert evaluations of SLOs in art and other subjects faulted a teacher’s description of an exploratory art course for students in grades 9-12, then offered suggestions for an “improved” version. The suggested changes transformed an exploratory course into a course demanding mastery. The evaluator added insult to injury by suggesting the teacher acquire two prior years of “baseline data ” for each of 114 students in the course and for 15 descriptors of content—about 3500 entries (for non-existent data). The evaluator also suggested the teacher identify the instructional methods needed to ensure all students would master each concept (Note 6).

The preoccupation with minutia in SLOs has other ripple effects. It adds to the pressure on principals and other administrators, and shifts their role from that of an inspirational leader to being auditor-in-chief who seeks data in order to analyze metrics and “calibrate” instruction (as if machines are proper models for education). By design, SLOs and VAM also distract attention from: (a) the resources teachers and students may need to reach specific goals, and (b) the many student achievements that are not mapped in academic tests.

The marketing campaign on behalf of SLOs as a method of teacher evaluation has been carefully planned to forward its use as a proxy for, or in addition to, teacher ratings based on VAM. These deeply flawed measures play a major role in making pay-for-performance the national norm for all educators with a spillover effect that weakens teacher’s unions. There is little evidence that such plans are common in business or that pay incentives survive in public education where administrative changes and shifting budget priorities are commonplace (Note 7).

The SLO process is designed to assert what is educationally significant in the work of teachers and in the lives of children and teens in their role as students. The system does not honor teachers as professionals who are trustworthy and capable of making wise decisions in real time, without pre-approved written plans or surveillance by an evaluator who may not be an expert in subject. These policies and practices assume that teachers are unable or unwilling to take responsibility for the wellbeing of their students and their achievements. Under the banner of accountability.teachers are stripped of their identity as professionals

Corruption of language about teaching, learning, and education. Business and economic definitions of student learning—now marketed by the Reform Support Network as part of “a new language and grammar” for education—are serious misrepresentations of what counts as learning, how learning occurs, and why education is important.

Gains in scores on tests given at two points in time are not credible measures of student learning or the effectiveness of teachers. Proponents of SLOs are intent on stripping away the layers of educational meanings attached to the concepts of human growth, development, and learning. The SLO process distracts attention from at last five educationally important ideas.

First, SLOs divert attention from the educational import of student interests and concerns that are not documented in “base-line data.” In practice, baseline data are highly reductive categories of student characteristics, easy to code, and increasingly standardized alphanumeric descriptors for use in software programs.

The preferred entries for baseline data are test scores—including the range of scores, the percentage of students “proficient” on subtests, and significant differences in scores by identifiers such as gender, ethnicity, socio-economic status, special needs, and English language learners. I have yet to encounter any concern about profiling students—setting expectations based on limited or misleading information that happens to be available. Teachers need to look beyond easy-to-code “data” especially as they begin a new course or year.

Second, the SLO process does not encourage full spectrum discussions about concurrent, and interdependent influences on learning—physical, sensory, perceptual, cognitive, social, and emotional. SLOs reflect a studied indifference to the asynchronous and multifaceted character of human growth and development. A sixth grade student with excellent analytical skills in varied contexts may, at the same time be physically awkward, feel socially inadequate, and be under stress from a home environment in disarray.

Learning is not well represented by a quantity that increases, decreases, or stays unchanged. The forms of learning that can be forced into the SLO template are not necessarily the most meaningful, memorable, and momentous to students or teachers. Learning is a personal process of making sense from what William James called a “great booming and buzzing confusion” thrust upon us at birth (James, 1890, p. 488).

Third, proponents of SLOs are eager to use the terms “assessment” and “test score” interchangeably, as if they are synonyms and have the same practical import. These terms are not interchangeable in meaning or significance. Assessments are deliberative, qualitative, and they are evaluations. They are communications intended to discern and disclose meanings, understand actions, and evaluate (find value) in accomplishments, ideally in face-to-face discussion. As soon as an “assessment” is designed to be “comparable across classrooms” it has become a test—a “one-size-fits-many” test. In the context of an SLO, tests exist for rating students and teachers.

Fourth, SLOs call for teachers to think about the “academic strengths and weaknesses” of students as if transmitting academic knowledge is the only responsibility of educators that matters. By definition “academic” knowledge is based on the inquiry and scholarship of experts. It is largely book-centered, discipline-specific, and transmitted in well-honed structures. Knowledge derived from and embedded in the experiences of students is not routinely treated as content worthy of critical and creative thinking in school.

Opportunities for learning are lost by this exclusive focus on academic learning. The SLO process is not designed to honor student-initiated inquiries or collaborations on theme-based, problem-based inquiries. Fortuitous opportunities for learning are not counted. In this respect, SLOs produce evaluations of teachers that may have little fidelity to their best professional judgments on what to teach, when, to whom, how, and why.

Finally, if educational policies and recommended practices are sound and credible then elaborate “messaging systems” should not be needed to seek compliance with them The SLO process permits administrators and policymakers to hide behind numbers and dodge the difficult work of inspiring teachers and students by the example of their expertise, humanity, and ingenuity. The value teachers add to the lives of their students in not strictly academic or test-related, or limited to the “interval of instruction” in an SLO (Note 8)

There can be little doubt that the SLO process attempts to micro-manage teachers. There can be no doubt that the process rewards teaching to the test. Appraisals of teachers by SLO and by VAM are part of an evaluation system that makes no distinction between the performance of students on a limited array of indicators and the effectiveness of a teacher.

The impulse to micromanage teachers is just one part of a well-financed and multi-faced campaign to diminish the credibility of teachers as professionals. This campaign has been in process for more than three decades. The end game is the transformation of public education, including large segments of higher education, into a for-profit but tax-subsidized industry…but that is another story (e.g. Ravitch, 2013; Zhao, 2014).

 

Laura H. Chapman

Cincinnati, OH. November, 2014

 

NOTES

Note 1. SLOs are part of the drive to make pay-for-performance the norm for teacher compensation. Since 2006, USDE has awarded over 100 grants in 29 states and the District of Columbia in support of pay-for-performance plans authorized under the Teacher Incentive Fund (Center for Educator Compensation Reform, 2014)

Note 2. The advisors gathered to launch the Reform Support Network are identified in Great teachers and leaders: State considerations on building systems of educator effectiveness. All of the advisors were supporters of USDE policies, including reforms designed to expand market-based education.

Note 3. Today’s quest for “scientific management” is closely aligned with early twentieth century efforts to manage schools and train workers, including soldiers, by principles of efficiency associated with engineering, science, and mass manufacturing (e.g. Bobbitt, 1912; Callahan, 1964; Taylor, 1914).

During WWII these ideas were applied to training programs for large-scale military and industrial operations. Researchers in psychology and education were enlisted to create more sophisticated tests for recruits and more detailed specifications for ”train the trainer” instructional manuals with “behavioral objectives” for courses. After WW II many of these ideas migrated into commerce and education, including early work on computer aided programmed instruction (Herman, 1995). SLOs still retain the imprints of programs designed for military and industrial training.

Note 4. The sophistication of the marketing campaign is suggested by one of the largest subcontracts— $6.3 million to Education First. The founding partner is Jennifer Vranek, a former advocacy expert with the Bill and Melinda Gates Foundation. She and others working for Education First helped a number of states apply for the RttT competition. They have fashioned PR campaigns for the Common Core State Standards in many states. The firm’s website includes a sample of the firm’s communication and advocacy services: “Outreach and public-engagement strategies and activities; strategic communications planning; reports, white papers and articles designed to synthesize, explain and persuade; development of communications tools, including marketing materials, web copy, press releases, and social media content” (Education First, 2014).

Note 5. An enlarged presence for computer-based SLO evaluations is evident in patents for software. An example is U.S. patent 8696365 B1 dated April 15, 2014 for a “System for defining, tracking, and analyzing student growth over time,” One of the inventors is a school superintendent in Ohio whose business is Align, Assess, Achieve, LLC. This software is now marketed as part of McGraw Hill offerings in education. The patent citations for this system (in development with patent protections since 2012) refer to its use in Austin, Texas; Charlotte-Mecklenburg, NC; Colorado; Georgia; Louisiana; New York;  Rhode Island;  and Delaware among other venues.

Note 6. RSN has a library of SLOs organized by state, grade level, and subject. As of November 2014, five states had contributed examples of SLOs in the arts. All examples have annotations made by a “trained evaluator” whose comments are framed to make the teaching of art fit SLO criteria. It is clear from these examples that the arts evaluators have little understanding of the aims and methods of instruction in the arts or the conditions under which teachers in the arts work. The SLOs must fit the evaluators’ concepts of “proper education” in the arts. No teacher in this sample had a fully compliant SLO. https://rtt.grads360.org/#communities/slo/slo-library

Note 7. There is ample evidence that compensation matters to teachers, but that it is not the primary reason teachers enter and stay in the profession. In a 2013 USDE-funded experiment, 1, 514 teachers identified as highly effective (VAM estimates for reading and mathematics) were offered a $20,000 bonus—paid in installments for two years—if they transferred to a low performing school. Only 81 teachers applied. Of these, 75 stayed for two years. About 35 stayed beyond two years. The retention rates for teachers and improved student scores attributed to these teachers varied by district. After the end of the bonus plan there was no sustained improvement in scores.

Other experiments with teacher pay-for-performance show that few plans survive beyond six years due to funding and allocation issues in addition to a failure to make any consistent difference in achievement (Murnane, R. J. & Cohen, D. K. 1986; Glazerman, Protik, Teh, Bruch, & Max, 2013).

Note 8. Unfortunately, professionals in education and civic leaders have adopted the SLO process as if it had proven benefits. For example, on June 27, 2014 seven education agencies in the state of Maryland, including the Baltimore Teachers Union, signed a memorandum of understanding (MOU) endorsing the use of SLOs statewide (Maryland Public Schools, 2014).

The MOU was the result of a contract inviting CTAC (a major promoter of SLOs since 1999) to advise state officials. The MOU was approved by the Maryland State Board of Education, the Maryland State Department of Education, the Maryland State Education Association, the Public School Superintendents Association of Maryland, the Maryland Association of Boards of Education, the Maryland Association of Secondary School Principals, the Maryland Association of Elementary School Principals, and the Baltimore Teachers Union.”

The purpose of the agreement was to ensure “the advancement of professional development, common language, streamlined communication and implementation strategies for Student Learning Objectives (SLOs) statewide and in each of the Local School Systems (LLS). The parties agreed to the following statements:

1. The primary goal of evaluating teaching should be to improve effectiveness in the classroom, which will lead to student growth. The SLO process is an important component of effective instruction,” When collaboratively developed and implemented appropriately, the use of rigorous objectives coupled with multiple strategies, measured by multiple assessments, leads to academic success and growth on the part of students.
2. The parties shall coordinate resources and strategies to assist educators in the development of rigorous and measurable, but obtainable SLOs.
3. The parties shall effectively assist teachers and principals in fully understanding, utilizing, and embracing SLOs, by focusing on professional development that, minimally: a) Identifies the key elements of a rigorous SLO, utilizing common definitions and content to develop consistency across LSSs; b) Assists in setting measurable and obtainable benchmarks; c) Recognizes progress/growth and effective strategies for achieving SLOs.”

 

Appendix A:

Role of SLOs in Generating Big Data

This Appendix is intended to connect some dots between the demands of the SLO process and national data-gathering campaigns established in 2005 with two independent but coordinated funding sources: The Bill and Melinda Gates Foundation and USDE.

The vision of the Gates Foundation. Between 2005 and early 2011, the Gates’ Foundation invested $75 million in a major advocacy campaign for data-gathering, aided by the National Governor’s Association, the Council of Chief State School Officers, Achieve, and The Education Trust. During the same period, the Gates Foundation also awarded grants totaling $390,493,545 for projects to gather data and build systems for reporting on teacher effectiveness. This multi-faceted campaign envisions the link between teacher and student data serving eight purposes:

1. Determine which teachers help students become college-ready and successful, 2. Determine characteristics of effective educators, 3. Identify programs that prepare highly qualified and effective teachers, 4. Assess the value of non-traditional teacher preparation programs, 5. Evaluate professional development programs, 6. Determine variables that help or hinder student learning, 7. Plan effective assistance for teachers early in their career, and 8. Inform policy makers of best value practices, including

compensation (TSDL, 2011, “Use and Purpose”).

The TSDL system is intended to monitor the work of teachers in a manner that ensures all courses are based on standards, and that all responsibilities for learning are assigned to one or more “teachers of record” in charge of a student or class. A teacher of record is best understood as person who has a unique identifier (think barcode) for an entire career in teaching. A record is generated whenever a teacher of record has some specified proportion of responsibility for a student’s learning activities. Learning activities must be defined in terms of the performance measures for a particular standard, by subject and grade level.

The TSDL system requires period-by-period tracking of teachers and students every day; including “tests, quizzes, projects, homework, classroom participation, or other forms of day-to-day assessments and progress measures”—a level of surveillance that is said to be comparable to business practices (TSDL, 2011, “Key Components”).

The system will keep current and longitudinal data on the performance of teachers and individual students, as well schools, districts, states, and educators ranging from principals to higher education faculty. This data will then be used to determine the “best value” investments to make in education and to monitor improvements in outcomes, taking into account as many demographic factors as possible, including health records for preschoolers.

The vision of USDE. The Gates-funded TSDL campaign added significant resources to a parallel federal initiative. Since 2006, the U.S. Department of Education has invested over $700 million in the Statewide Longitudinal Data Systems (SLDS) Grant Program. More than forty states have received multi-year grants to standardize data on education. Operated by the Institute of Education Sciences, the SLDS program is: “designed to aid state education agencies in developing and implementing longitudinal data systems. These systems are intended to enhance the ability of States to efficiently and accurately manage, analyze, and use education data, including individual student records…to help States, districts, schools, and teachers make data-driven decisions to improve student learning, as well as facilitate research to increase student achievement and close achievement gaps” (USDE, 2011, Overview).

 

 

Appendix B:

Case Study of SLO Implementation in Ohio

The following excerpts from the Ohio Department of Education website Frequently Asked Questions (FAQs) illustrate how SLO requirements are framed to produce data for use in the TSDL and SLDS systems, often at the expense of coherent thinking about education.

The garbled and often evasive answers to FAQs reflect a pathological demand for data even if it is relevant to a teacher’s job assignment. The questions and answers resemble a game analogous to “Mother May I?” or “Father May I?” played between teachers and state officials. Teachers cannot win this game without distorting their work and the achievements of students. I have omitted some FAQs and edited some for length.

Frequently Asked Questions about SLOs

Do all teachers have to write SLOs? No. Teachers who receive VAM are exempt. All others use SLOs as part of their student growth measures per the district plan.

Does the Department approve the SLOs? No. The Department…recommends that an existing district or building committee become trained to review, provide feedback, and ultimately approve SLOs.

How many SLOs do I have to write? The Department requires a minimum of two and recommends no more than four, representative of your teaching schedule and student population.

Do I have to write a SLO for each course that I teach? When feasible, all of your students should be covered by a SLO. For example, for a self-contained 3rd grade teacher who instructs all four core subjects, a district may make a local decision to focus SLOs on reading and math only.

What is the average length of a SLO? When writing the SLO, you should use the provided checklist to ensure it contains all of the required information needed for approval at the local level.

How do I describe baseline and trend data? Avoid vague words such as “most,” “several,” and “struggled.” Include student performance data, such as a table showing the range and frequency of student scores.  You should make written observations based on the data in this section. However, there must be data included.

What if my data are unrelated to my subject or do not exist? All teachers can find data relevant to their course. Data may be from related subjects, be from a brief survey to assess students’ background, or a pre-assessment, or the first chapter or unit exam.

Should I use prior content or current content to create baseline data? You may use a pre-assessment for baseline data. It should contain the content and skills to be taught during the upcoming year. End-of-course assessments from the prior year can be used if they are good proxies for the current course.

What portion of my student population or roster do I include? At least one of your SLOs must cover all the students enrolled in a course except in rare cases when you may have a very large student population. An additional “focused” or “targeted” SLO may be created for a subgroup of students within the course who need targeted assistance.  

Can I exclude students with disabilities from my SLO? No.

What if a student joins my class late in the year or withdraws from my class early? All students entering a course throughout the year should receive a pre-assessment to determine any gaps in learning along with a post-assessment for each student still enrolled in the course.

How do I write a SLO as a teacher of 450 students? The district plan should strive for comparability and consistency across subject and grade levels regarding the total number of SLOs per teacher as well as the size of the student population.

Is the interval of instruction one curriculum unit or the entire school year?  Match the interval of instruction with the length of the course (e.g., a year, semester, trimester, or a quarter). Districts with intervals of instruction other than a typical school year need multiple approval periods for their SLOs.

My school year ends on June 1. Does this mean my interval of instruction for my yearlong course ends on June 1? No.  State law requires the completion of the evaluation process by May 1. You should administer your post-assessments on or around April 15 so you have time to score the assessments, complete the SLO Scoring Template, and submit the data to the evaluator by May 1.

Should SLOs be aligned primarily to course curriculum or Common Core State Standards? Align SLOs in the following order: Common Core State Standards, Ohio Academic Content Standards, National standards put forth by education organizations

Should SLOs cover multiple standards or just one? You must have at least one SLO that covers the overarching standards that represent the breadth of the course. Once this course-level SLO is in place, you may then choose to write a targeted SLO, in which you focus on a subgroup of students (e.g., the low-achieving) and also narrow the content to only those standards that these students have yet to master.

In assessment(s) what is stretch? An assessment must contain questions that vary in difficulty. It should contain both basic and advanced knowledge and skill questions so that both low-performing and high-performing students can demonstrate growth.

Is it the intent of the SLO process to use the same instrument for pre-assessment and post-assessment to accurately measure student growth? Using the same instrument as a pre- and post-assessment is not ideal. In fact, using the same assessment multiple times within the same year may decrease the validity of results since students will have seen the questions before. A well-written pre-assessment (used in conjunction with other forms of baseline data) can be a valuable source of data, because it should closely align with the post-assessment to measure growth. Pre-assessments should assess the same general content as the post-assessment, be comparable in rigor, and should be reviewed by content experts for validity and reliability.

Can I create the assessment for my SLO? The Department strongly advises against an individual teacher creating an assessment. In rare cases where a team of teachers cannot create an assessment, you should develop the assessment in conjunction with an instructional coach, curriculum supervisor, special education teacher, English Language Learner teacher, and administrator or other faculty member with assessment expertise.

What if the pre-assessment used in the submitted SLO is not very strong? Evaluators can suggest how to improve the pre-assessment for next year. The goal is to learn from the process in these early years. (Note: SLOs count for 50% of a teacher’s evaluation. “We are still learning” is an astonishing response to the high stakes attached to SLOs). Districts and schools should have clear expectations regarding assessments to ensure quality pre- and post-assessments.

Will all growth targets be tiered?  Instances may exist where one growth target may be acceptable for all students, but this is rare. Ultimately, every student will have a target within the established tiers.

If a student is well below proficiency level, is it appropriate to set a growth target of proficiency?  Targets should first be developmentally appropriate and then rigorous and attainable. Expecting a student to grow from below basic to proficient in one year may be very difficult. However, in some cases, more than a year’s worth of growth is necessary to close the achievement gap. You should consult with colleagues, curriculum directors, administrators, and instructional coaches when determining appropriate growth targets.

At what point can a teacher revise his or her growth targets? You cannot revise growth targets once the SLO has been approved. …In some extenuating circumstances, such as after a natural disaster, outbreak of serious illness, or an unplanned extended absence, you may be able to revise your SLO with district approval.

How will the Ohio Department of Education and districts ensure that growth targets are rigorous across schools?  The Department recommends those approving SLOs complete a calibration process to ensure all team members are upholding rigorous standards for every SLO within the district. The state will monitor the implementation of SLOs by conducting random audits.

I feel like I am repeating a lot of information when I attempt to complete the Rationale for Growth Targets section. Am I doing this wrong? Rationales must include strong justifications for why the growth targets are appropriate and achievable for the student population, and, therefore, must be based on student data and the content of the SLO. The rationale ties everything together, and, as a result, it touches on every component that came before it. Rationales explain why this learning is important by making connections to school and district goals.

I need more information on the Business Rules for Student Growth Measures. Where do I find that information? See next section.

Business Rules for Student Growth Measures

(11/05/2013) Edited for length

The state definitional framework for Teacher-Student Data Linkages (TSDL) should guide the implementation of student growth measures.

Teacher of Record.  An educator who is responsible for a significant portion of a student’s instructional time (based on enrollment) within a given subject or course that is aligned to a relevant assessment. A teacher’s roster should include students that he/she has provided instruction for. SLOs only require accurate rosters, not proportional splits of instructional attribution. In situations where teachers share instruction for a student or group of students, those students may appear on both teachers’ rosters.

The Link Roster Verification (not yet completed for SLOs) will not only identify which students a teacher provides instruction for, it also addresses proportional attribution of that instructional time.

Student enrollment and attendance. Students with forty-five or more excused or unexcused absences are excluded from growth measures

Interval of instruction for SLOs. SLOs should be designed on the maximum available interval of instruction. An SLO on a year-long course should use a year long interval of instruction. It is important to note, that the OTES timeline requires a somewhat shortened “year.” That is, the yearlong course will need to collect the second data point (before May 1) in order to meet the evaluation requirements. Likewise, a semester course should use a semester interval of instruction.

Minimum number of students. Growth measures must include a minimum effective n size of six students. There is no maximum number of students. SLOs may be written across grade bands to capture the minimum n size of six. The district plan should strive for comparability and consistency across subject and grade levels regarding the total number of SLOs per teacher as well as the size of the student population for each SLO.

 

References

Alaimo, K., Olson, C. M., & Frongillo, E. A. (2001). Food insufficiency and American school-aged children’s cognitive, academic, and psychosocial development. Pediatrics, 108(1), 44-53.

American Educational Research Association. (2013). Standards for educational and psychological testing. Washington, DC: Author.

American Institutes of Research. (2012, November). Student learning objectives as measures of educator effectiveness: The basics. Washington, DC: Author. Retrieved from http://educatortalent.org/inc/docs/SLOs_Measures_of_Educator_Effectiveness.pdf

American Statistical Association. (2014, April 8). ASA statement on using value-added models for educational assessment. Alexandria, VA: Author. Retrieved from https://www.amstat.org/policy/pdfs/ASA_VAM_Statement.pdf

Berliner, D. (2013). Effects of inequality and poverty vs. teachers and schooling on America’s youth. Teachers College Record, 115(12).

Berliner, D. C. (2002). Comment: Educational research: The hardest science of all. Educational researcher, 31(8), 18-20.

Bobbitt, J. F. (1912). The elimination of waste in education. The Elementary School Teacher, 259-271.

Brodsky, A., DeCesare, D., & Kramer-Wine, J. (2010). Design and implementation considerations for alternative teacher compensation programs. Theory Into Practice, 49(3), 213-222.

Callahan, R. E. (1964). Education and the cult of efficiency. University of Chicago Press.

Center for Educator Compensation Reform. (2014). Teacher Incentive Fund grantee profiles by state. Retrieved from http://www.cecr.ed.gov/ 

Chapman, L. H. (2012, November 8). The circular reasoning theory of school reform: Why it is wrong. Invited Super Session Paper presented at the Ohio Art Education, Association Conference, Newport, KY.

Chapman, L. H. (2013, October). Accountability gone wild: The econometric turn in education. Pre-publication Draft. Author.

Coleman, J. S., Kelly, D. L., Hobson, C. J., McPartland, J., Mood, A.M., Weinfeld, F.D. & York, R. L. (1966). Equality of educational opportunity. U.S. Department of Health, Education, and Welfare. Washington, DC: U.S. Government Printing Office.

Community Training and Assistance Center. (2004, January). Catalyst for change: Denver pay-for-performance final report. Boston, MA: Author. Retrieved from http://www. ctacusa.com/PDFs/Rpt-CatalystChangeFull-2004.pdf.

Community Training and Assistance Center. (2008). CMS Student Learning Objective Guide. Boston, MA: Author. Retrieved from http://gse.berkeley.edu/research/pace/reports/altcomp/Smith_Student_Learning_Guide.pdf.

Community Training and Assistance Center, (2013, February). It’s more than money: Teacher Incentive Fund—Leadership for educators’ advanced performance, Charlotte-Mecklenburg Schools. Boston, MA: Author.

Denver Public Schools. (2013). Welcome to student growth objectives: New rubrics with ratings. Retrieved from http://sgoinfo.dpsk12.org/

Drucker, P. (1954). The practice of management. NY: Harper.

Education First. (2014). Website. www.education-first.com

Federal Register, (2009, November 18). Rules and regulations Department of Education. Final Definitions, 74(221).

Florida Department of Education (2013, February). Frequently asked questions about value added model (VAM). Author. Retrieved from https://www.ocps.net/cs/services/accountability/Documents/FAQ_About_VAM.pdf

Gagné, R. M., Briggs, J., (1974). Principles of instructional design. NY: Holt, Rinehart and Winston.

Gibson, S. & Dembo, M. H. (1984). Teacher efficacy: A construct validation. Journal of Educational Psychology,76(4), 569-582.

Gill, B., Bruch, J., & Booker, K. (2013). Using alternative student growth measures for evaluating teacher performance: What the literature says. (REL 2013–002). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Mid-Atlantic. Retrieved from http://ies.ed.gov/pubsearch/pubsinfo.asp?pubid=REL2013002

Gill, B., English, B., Furgeson, J., & McCullough, M. (2014). Alternative student growth measures for teacher evaluation: Profiles of early-adopting districts. (REL 2014-016). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Mid-Atlantic. Retrieved from http://ies.ed.gov/ncee/edlabs/regions/midatlantic/pdf/REL_2014016.pdf

Glazerman, S., Protik, A., Teh, B., Bruch, J., & Max, J. (2013). Transfer incentives for high-performing teachers: Final results from a multisite experiment. (NCEE 2014-4003). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee.

Goldhaber, D., & Walch, J. (2012). Strategic pay reform: A student outcomes-based evaluation of Denver’s ProComp teacher pay initiative. Economics of Education Review, 31(6), 1067-1083.

Gronlund, N, (1970). Stating behavioral objectives for classroom instruction. NY: Macmillan.

Herman, E. (1995. The romance of American psychology: Political culture in the age of experts. Berkeley: University of California Press.

Indiana State Department of Education. (2012). Rise evaluation and development system: Student learning objectives handbook. Indianapolis, IN: Author.

James, W. (1890). The principles of psychology. Cambridge, MA: Harvard University Press. Reprint. (2011). Digireads. com Publishing.

Lacireno-Paquet, N., Morgan, C., & Mello, D. (2014). How states use student learning objectives in teacher evaluation systems: A review of state websites. (REL 2014-013). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Northeast & Islands. Retrieved from http://ies.ed.gov/ncee/edlabs

Ligon, G. D. (2009). Performing on grade level and making a year’s growth: Muddled definitions and expectations, Growth Model Series, Part III. Austin, TX: ESP Solutions Group. Retrieved from http://www.espsolutionsgroup.com/espweb/assets/files/ESP_Performing_on_Grade_Level_ORG.pdf

Mager, R. (1962). Preparing objectives for programmed instruction. Palo Alto, CA: Fearon-Pitrnan Publishers.

Marion, S., DePascale, C., Domaleski, C., Gong, B. & Diaz-Bilello, E. (2012, May). Considerations for analyzing educators contributions to student learning in non-tested subjects and grades with a focus on student learning objectives. Dover, NH: Center for Assessment. Retrieved from http://www.nciea.org/publications-2/

Maryland Public Schools (2014, June 27). Student learning objectives memorandum of understanding. Baltimore, MD: Author. Retrieved from http://marylandpublicschools.org/press/2013Press/MOU_on_TPE_062714.pdf

Murnane, R. J., & Cohen, D. K. (1986). Merit pay and the evaluation problem: Why most merit pay plans fail and a few survive. Harvard Educational Review, 56(1), 1-18.

Ohio Department of Education. (2012). Student learning objective (SLO) template checklist. Columbus, OH: Author.  Retrieved from https://education.ohio.gov/getattachment/Topics/Academic-Content-Standards/New-Learning-Standards/Student-Learning-Objective-Examples/080612_2497_SLO_Checklist_7_24_12-1.pdf.aspx

Ohio Department of Education (2014). FAQs. http://education.ohio.gov/Topics/Teaching/Educator-Evaluation-System/Ohio-s-Teacher-Evaluation-System/Student-Growth-Measures/Student-Learning-Objective-Examples/Student-Learning-Objectives-FAQs-1

Ohio Department of Education (2014). Business Rules. http://education.ohio.gov/getattachment/Topics/Teaching/Educator-Evaluation-System/Ohio-s-Teacher-Evaluation-System/Student-Growth-Measures/110513_Business-rules-for-SGM.pdf.aspx

Popham, W. J. (2014). The right test for the wrong reason. Phi Delta Kappan. 96(1). 47-52.

Prince, C. D., Schuermann, P. J., Guthrie, J. W., Witham, P. J., Milanowski, A. T. & Thorn, C. A. (2009). The other 69 percent: Fairly rewarding the performance of teachers of nontested subjects and grades (A guide to implementation: Resources for applied practice). Washington, DC: Center for Educator Compensation Reform. Retrieved from http://cecr.ed.gov/pdfs/guide/other69Percent.pdf.

Public Law 107-110, No Child Left Behind Act of 2001 (NCLB), 107^th Congress.

Public Law 109-149.  The Teacher Incentive Fund, 109^th Congress.

Public Law 111–5. American Recovery and Reinvestment Act of 2009, Section 14005-6, Title XIV, Race to the Top Fund (RttT), 111^th Congress.

Race to the Top Technical Assistance Network. (2010). Measuring student growth for teachers in non-tested grades and subjects: A primer. Washington, DC: ICF International.

Ravitch, D. (2013). Reign of error: The hoax of the privatization movement and the danger to America’s public schools. NY: Knopf.

Reform Support Network. (2011a). Great teachers and leaders: State considerations on building systems of educator effectiveness. Washington, DC: Author. Retrieved from http://www.google.com/#q=Anthony+Bryk+%2B+%22Reform+Support+Network%22

Reform Support Network. (2011b). Targeting growth using student learning objectives as a measure of educator effectiveness. Washington, DC: Author.

Reform Support Network. (2012a, December). A quality control toolkit for student learning objectives. Retrieved from http://www2.ed.gov/about/inits/ed/implementation-support-unit/tech-assist/slo-toolkit.pdf

Reform Support Network. (2012b, December). SLO evaluator protocol for establishing consistent quality. Washington, DC: Author. http://public.grads360.org/rsn/slo/rsn-slo-evaluator-protocol.pdf

Reform Support Network. (2012c, December). Engaging educators: A reform support network guide for states and districts. Washington, DC: Author. Retrieved from www2.ed.gov/about/inits/ed/implementation-support-unit/tech-assist/engaging-educators.pdf

Reform Support Network. (2013, August). The view from the states: A brief on non-tested grades and subjects. Washington, DC: Author. Retrieved from http://www2.ed.gov/about/inits/ed/implementation-support-unit/tech-assist/view-from-states.pdf

Reform Support Network. (2013, July). Measuring teaching matters: What different ways of looking at student results tell us about teacher effectiveness. Washington, DC: Author. Retrieved from http://www2.ed.gov/about/inits/ed/implementation-support-unit/tech-assist/measuring-teaching-matters.pdf

Reform Support Network. (2014, May). A toolkit for implementing high quality student learning objectives 2.0. Washington, DC: Author. Retrieved from http://www2.ed.gov/about/inits/ed/implementationsupportunit/techassist/resources.html

Robinson, S. (2014, May 6). FEA disappointed in federal judge’s ruling on state’s unfair teacher evaluation system. Washington, DC: National education Association. Retrieved from http://www.nea.org/home/59054.htm

Rothstein, J. & Mathis, W. J. (2013). Have we identified effective teachers? Culminating findings from the Measures of Effective Teaching project. (Review). Boulder, CO: National Education Policy Center. Retrieved from http://nepc.colorado.edu/thinktank/review-MET-final-2013.

Schmitt, L. N. T.; Lamb, L. M.; Cornetto, K. M. & Courtemanche, M. (2014, March). AISD REACH program update, 2012-2013: Student learning objectives publication 12.83B. Austin, TX. Austin Independent School District.

Schochet, P, Z. & Chiang, H. S.  (2010). Error rates in measuring teacher and school performance based on student test score gains (NCEE 2010-4004). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

Slotnick,W. Smith, M. et al. (2004). Catalyst for change: Pay for performance in Denver. Boston: MA: Community Training and Assistance Center.

Southwest Comprehensive Center at WestEd (n. d.). Measuring student growth in non-tested grades and subjects: A primer. Phoenix, AZ: Author.

Taylor, F. W. (1911). The principles of scientific management. NY: Harper and Row.

Teacher Student Data Link Project. (2014).  Website at http://www.tsdl.org/

Tennessee Department of Education. (2013). Tennessee fine arts student growth measures. Retrieved from https://www2.ed.gov/programs/racetothetop/…/tle2-tn-arts-system.pdf‎

U.S. Department of Education. (2011). Statewide longitudinal data systems (SLDS) grant program. Website http://nces.ed.gov/programs/slds/index.asp

Weiner, R. (2013, March). Teaching to the core: Integrating implementation of common core and teacher effectiveness policies. Washington, DC: The Aspen Institute.

West, G. E. (1977). Bureaupathology and the failure of MBO. Human Resource Management, 16(2), 33-40.

Zhao, Y. (2014). Who’s afraid of the big, bad dragon? Why China has the best (and the worst) schools in the world. San Francisco, CA: Jossey-Bass.

 

 

 

Laura H. Chapman, Ph.D. is an independent scholar and consultant on arts education based in Cincinnati, Ohio. She is known for her work on curriculum development; reports on trends and issues bearing on art education and teacher preparation, pre-K to 12; and research on federal policies including NCLB, the marketing of the Common Core State Standards, and current mandates for teacher evaluation.

A Distinguished Fellow of the National Art Education Association, Laura began teaching visual art in 1957. She has taught students in urban, rural, and suburban schools and in community centers as well as museums. Her teaching experience ranges from pre-K to doctoral programs. She has been a consultant for international, national, state, and local agencies including projects for the National Endowment for the Arts, the J. Paul Getty Trust, Education Testing Service, and National Assessment of Educational Progress.

Over the years she has served as senior editor of Studies in Art Education and editorial advisor for Art Education, School Arts, Review of Research in Visual Arts Education, Studies in Art Education, Journal of Aesthetic Education, Journal of Multi-cultural and Cross-cultural Research in Art Education and The Arts Education Review of Books. In addition to numerous peer-reviewed articles, her publications include forewords written for six books.

She is the author of Approaches to Art in Education (1978), Instant Art, Instant Culture: The Unspoken Policy for American Schools (1982)—selected a Book of the Century in Education by a national panel. Her student texts and teacher resources include A World of Images for grade 7 and for grade 8 Art: Images and Ideas (1992); and for grades 1-6, Discover Art (1989), and Adventures in Art (1993, 1999).

Selected books and articles have been translated into Modern Greek, Dutch, Arabic, Spanish, Malay, and Chinese. Her creative work (painting and documentary photographs) has appeared in juried and invited exhibitions in the South and Midwest.

She can be reached at 2444 Madison Road, Regency Unit 1501, Cincinnati, OH 45208-1256

chapmanLH@aol.com (email is more reliable than cell)  Cell 513-861-7118

anThe-Marketing-of-Student-Learning-Objectives-SLOs-1999-2014