Collaborative Assessment Development
Proola is a web app where you can learn about assessment while creating items, with feedback from peers. As a community of educators, we're building and curating an open item bank. Questions are released under a CC4- licence, and can be exported for free using a QTI format.
After studying the learning resources, and browsing the bank to see some examples, you're ready to start contributing. Here's how it works.
Planning for Assessment
Assessment development begins with an idea of
- What you're assessing - the target construct or variable that students possess to different degrees
- For whom - the target population of students
- Why you're assessing it - how results are going to be used
Assessments
The what, for whom, and why of an assessment or test are described within a statement of purpose. Low quality assessments often have purposes that are unclear, whereas effective ones are built around purposes that are clearly defined.
In Proola, assessments are collections of saved items that are organized around a common purpose. The purpose is defined when creating the assessment.
Learning Objectives
The what in an assessment needs to be broken down into smaller parts called learning objectives or instructional objectives. Every question should be aligned to an objective.
Proola lets you add and organize your own learning objectives, and link your items to group objectives and national standards like the Common Core. Contact us for information on adding a new state or national standard set.
Groups
Proola lets you team up with other educators to develop shared assessment materials. An administrator initiates the process by creating a group and adding members. You can then share items and learning objectives with other group members, and filter results by group.
Creating Items
Once you've created learning objectives, the peer-review process for writing items involves four main steps:
- Draft your item
- Get feedback
- Edit based on feedback
- Submit for review
1. Draft an Item
Use a simple interface to create a selected-response or contructed-response item. Give it a short but descriptive title, and choose from a variety of subjects, from Accounting to Writing, and a range of grade levels, Pre-K through College. Optionally add common core learning objectives, and share any other info in the comments.
Remember, an effective item targets a single learning objective while minimizing the influence of extraneous information.
By default, drafts are only visible to you. You also have the option of sharing within groups, or with everyone.
2. Get feedback from peers and assessment specialists
Once an item is saved as a draft, anyone you've shared it with can view and comment on it. Wait patiently for comments, or recruit peers in your grade level, subject area, school, district, and get them to sign up, join your group, and leave feedback.
3. Edit your item based on feedback
After input from peers, edit and improve your item. Focus on the item writing guidelines and your selected learning objective. Aim for the highest depth of knowledge. Double-check for clarity and correct spelling and grammar.
4. Submit for review to share with the community
After submitting an item, there's no turning back. You and others can still comment, but edits are locked until a formal peer review is complete.
Revise and Resubmit
When revisions are recommended on an item you've submitted, edits are made to a new version of the item. Previous versions are viewable but not editable.
Revise and repeat this process as needed up to four times. Once approved, your item can be saved, printed, and exported freely by the Proola community.
Commenting on Items
Take your item writing skills to the next level by contributing to others' work. When commenting, remember to
- Be constructive
- Reference the guidelines
- Target the learning objective
- Optimize depth of knowledge
As you review and comment on the items of your peers, you'll learn new strategies, like how to use applications to assess higher depth of knowledge. You'll also see approaches that should be avoided, like making an item difficult via trickery. Here are a few principles of effective commenting.
Be constructive
Comments will naturally flow toward the limitations of an item. Remember to highlight strengths as well as weaknesses, and always provide suggestions for improvement.
Reference the guidelines
Flawed items tend to miss one or more item writing guidelines. Organize and focus your comments by referring to the guidelines. A dropdown in the commenting area gives you easy access to the list.
Target objectives and DOK
Always check scope, that is, how well items address the intended learning objective at the appropriate depth of knowledge.
Reviewing Items
Help curate the item bank by formally reviewing items. Reviews provide structured feedback based on an analytic rubric.
A formal review is provided for all submitted items. When revisions are recommended, the item is sent back to the original author for editing. Once approved by an administrator, the item is publicly available for saving, printing, and exporting. Group restrictions no longer apply.
Reviews are given by Proola administrators who have experience authoring items and evaluating item quality. Some administrators are assessment specialists with training in test theory and design. Others are educators with on-the-job training in the assessment process.