Scoring Services
Handscoring
̽»¨ÍµÅÄ handscores tens of millions of student responses annually. Our unified handscoring system allows us to conduct all hiring, training, qualifying, scoring, monitoring, communicating, and reporting tasks remotely. At the heart of this system is our state-of-the-art Virtual Scoring Center (VSC), comprised of VSC Capture (a system for acquiring images and decoding response data from paper tests), VSC Train (a secure training and practice application for raters and scoring leadership), and VSC Score (a secure user management, scoring, and reporting application).
At the company's inception, ̽»¨ÍµÅÄ developed an outstanding training method for the scoring of student constructed responses which has become the industry model. On this foundation, we use our scoring technologies to monitor rater performance effectively and efficiently. In addition to traditional measures of rater accuracy and agreement, we employ a host of automated quality-assurance score verifications to ensure the most appropriate score has been assigned to each response.
̽»¨ÍµÅÄ's handscoring service offerings include conducting rangefinding proceedings, developing scoring tools and training materials, evaluating prompts and constructed-response items, recruiting and hiring scoring personnel, performing training activities, and supervising scoring efforts.
Automated Scoring
̽»¨ÍµÅÄ has led the field in automated scoring solutions since first adopted by schools, districts, and states in formative and summative contexts. ̽»¨ÍµÅÄ's Project Essay Grade (PEG) automated scoring engine currently provides nearly 10 million summative scores for students across the US.
PEG and the ̽»¨ÍµÅÄ team have dominated public competitions testing the state of the art of automated scoring. These contests have spanned essay scoring (the Hewlett Foundation-sponsored Automated Student Assessment Prize [ASAP] phase one), short constructed response English language arts and science scoring (ASAP phase two) and reading constructed response scoring (the National Center for Education Statistics [NCES]-sponsored National Assessment of Educational Progress [NAEP] Automated Scoring Challenge).
In most operational assessment program contexts we recommend a hybrid scoring approach, in which an automated scoring engine is used alongside human raters. This approach is designed to leverage the respective strengths of automated and hand-scoring while mitigating their respective limitations.
Psychometric & Assessment Services
We offer a full range of psychometric services in addition to comprehensive test development services.
Psychometric Services
Our psychometric team, comprised of psychometricians, data analysts, and software developers, offers extensive educational assessment experience and knowledge of best practices. Guided by the Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, and the National Council on Measurement in Education, 2014), our work is psychometrically sound and adheres to federal peer review requirements. We offer a full range of psychometric services, including:
- field test planning
- operational test design
- equating
- scaling
- classical item analysis
- IRT analysis
- differential item functioning analysis
- alignment studies
- reliability and validity analyses
- technical report production
- standard setting
Our psychometric team also designs, analyzes, and presents findings of wide-ranging psychometric research. See the research page for details.
Assessment Services
Item and Test Development Services
We offer comprehensive test development services for standards-based assessments. In addition to developing items and test forms that illuminate instruction, our capabilities include developing item and test specifications, performance and oral assessments, observation checklists, administration manuals, scoring criteria, and interpretation guides. Our test development staff spans content specialists, project directors, editors, graphic artists, and support staff, who are advised by our psychometric team.
Project and Program Management
̽»¨ÍµÅÄ believes in building and maintaining relationships with customers and employees alike through effective, honest, and ongoing communication. To this end, we employ a distributed leadership model—rather than a regimented, hierarchical structure—in which our project and program managers can lead proactively, make decisions independently, and guarantee we exceed client expectations. ̽»¨ÍµÅÄ's team of project and program managers has diverse knowledge around multiple aspects of assessment, including holding previous roles in scoring, test development, and psychometrics. In addition, some individuals also have prior classroom experience and hold graduate degrees or are PMP-certified. This expansive range of backgrounds has provided ample opportunities for several team members to work effectively on numerous state-wide programs. To promote continual growth and expand on their experience, ̽»¨ÍµÅÄ's project and program managers are supported by our program management office, which provides protocols, tools, feedback, and professional development. This approach ensures our project and program managers have both the skills and the resources necessary to consistently meet client needs. Finally, we recognize that all projects are dynamic and that they can require give and take when leading many aspects of successful program management.
Document Production, Distribution, Receipt, and Storage
̽»¨ÍµÅÄ is capable of producing millions of test booklets annually. In addition to our in-house digital printing equipment, we maintain relationships with several local printers.
We have developed a proprietary Order-Pack-Ship (OPS) application used to scan and track test materials. In addition to allowing us to efficiently manage the selection and shipping of orders, our OPS system is built around extensive quality control procedures to ensure that the correct numbers and types of materials are shipped to schools and districts.
We have similarly developed a sophisticated internal tracking system used for logging and processing all materials returned from the field. This system is capable of locating individual test documents at any time they are in our possession.
We maintain a host of high-volume scanners and imaging equipment and use several time-tested processes that ensure all necessary precautions are taken during scanner and document setup, scanner calibration, data validation, and data export. We utilize a double-blind data correction process to achieve the most accurate reporting of student information and test results.
In addition to secure electronic delivery of assessment results, we are capable of producing reports in-house—using a variety of high-speed printers—and shipping to schools or districts.
Evaluation & School Improvement Services
Program Evaluation
Our team of accomplished researchers assists organizations in all aspects of the research/evaluation process, including leveraging data for program improvement and sustainability. Whether it is evaluating new programs, conducting statewide survey research, or disseminating resources and information on diverse topics, we offer objective, accurate, and sound information that clients can trust.
Technical Assistance and Professional Development Services
We assist school districts with planning and implementing new
initiatives and developing their capacity to create and sustain
meaningful change. We assess the needs of our clients and
provide a continuum of services—tiered technical assistance—to
meet those needs. We deliver flexible and adaptable assistance
through site-based and remote consultation, workshops,
conferences, and various resources.
Our areas of expertise include:
- Special Education Systems and Practices
- Multi-tiered Systems of Support (RtI and PBIS)
- Early Literacy
- Leadership Development
- Bullying Prevention
- School Safety and Healthy School Climate
- Career & Technical Education
- STEM
- Student Support Services