Collaborative Assessment Development

Proola is a web app where you can learn about assessment while creating items, with feedback from peers. As a community of educators, we're building and curating an open item bank. Questions are released under a CC4- licence, and can be exported for free using a QTI format.

After studying the learning resources, and browsing the bank to see some examples, you're ready to start contributing. Here's how it works.

Planning for Assessment

Assessment development begins with an idea of

  1. What you're assessing - the target construct or variable that students possess to different degrees
  2. For whom - the target population of students
  3. Why you're assessing it - how results are going to be used

Assessments

The what, for whom, and why of an assessment or test are described within a statement of purpose. Low quality assessments often have purposes that are unclear, whereas effective ones are built around purposes that are clearly defined.

In Proola, assessments are collections of saved items that are organized around a common purpose. The purpose is defined when creating the assessment.

Learning Objectives

The what in an assessment needs to be broken down into smaller parts called learning objectives or instructional objectives. Every question should be aligned to an objective.

Proola lets you add and organize your own learning objectives, and link your items to group objectives and national standards like the Common Core. Contact us for information on adding a new state or national standard set.

Groups

Proola lets you team up with other educators to develop shared assessment materials. An administrator initiates the process by creating a group and adding members. You can then share items and learning objectives with other group members, and filter results by group.

Creating Items

Once you've created learning objectives, the peer-review process for writing items involves four main steps:

  1. Draft your item
  2. Get feedback
  3. Edit based on feedback
  4. Submit for review

1. Draft an Item

Use a simple interface to create a selected-response or contructed-response item. Give it a short but descriptive title, and choose from a variety of subjects, from Accounting to Writing, and a range of grade levels, Pre-K through College. Optionally add common core learning objectives, and share any other info in the comments.

Remember, an effective item targets a single learning objective while minimizing the influence of extraneous information.

By default, drafts are only visible to you. You also have the option of sharing within groups, or with everyone.

2. Get feedback from peers and assessment specialists

Once an item is saved as a draft, anyone you've shared it with can view and comment on it. Wait patiently for comments, or recruit peers in your grade level, subject area, school, district, and get them to sign up, join your group, and leave feedback.

3. Edit your item based on feedback

After input from peers, edit and improve your item. Focus on the item writing guidelines and your selected learning objective. Aim for the highest depth of knowledge. Double-check for clarity and correct spelling and grammar.

4. Submit for review to share with the community

After submitting an item, there's no turning back. You and others can still comment, but edits are locked until a formal peer review is complete.

Revise and Resubmit

When revisions are recommended on an item you've submitted, edits are made to a new version of the item. Previous versions are viewable but not editable.

Revise and repeat this process as needed up to four times. Once approved, your item can be saved, printed, and exported freely by the Proola community.

Commenting on Items

Take your item writing skills to the next level by contributing to others' work. When commenting, remember to

  1. Be constructive
  2. Reference the guidelines
  3. Target the learning objective
  4. Optimize depth of knowledge

As you review and comment on the items of your peers, you'll learn new strategies, like how to use applications to assess higher depth of knowledge. You'll also see approaches that should be avoided, like making an item difficult via trickery. Here are a few principles of effective commenting.

Be constructive

Comments will naturally flow toward the limitations of an item. Remember to highlight strengths as well as weaknesses, and always provide suggestions for improvement.

Reference the guidelines

Flawed items tend to miss one or more item writing guidelines. Organize and focus your comments by referring to the guidelines. A dropdown in the commenting area gives you easy access to the list.

Target objectives and DOK

Always check scope, that is, how well items address the intended learning objective at the appropriate depth of knowledge.

Reviewing Items

Help curate the item bank by formally reviewing items. Reviews provide structured feedback based on an analytic rubric.

A formal review is provided for all submitted items. When revisions are recommended, the item is sent back to the original author for editing. Once approved by an administrator, the item is publicly available for saving, printing, and exporting. Group restrictions no longer apply.

Reviews are given by Proola administrators who have experience authoring items and evaluating item quality. Some administrators are assessment specialists with training in test theory and design. Others are educators with on-the-job training in the assessment process.