Conducting an Item Review

What is expected from an item reviewer

Written by Aimée Gott
Updated over a week ago

You are responsible for helping us to keep a high standard in our assessments. Do not allow items to proceed if they don’t meet the requirements.

Three things you must review for every item:

  • Does it test the required objective?

  • Is it technically correct?

  • Can it be answered by reading only the context and stem?

Three things that would help us out:

  • Is it suitable for an international audience?

  • Does it use any topics that should be avoided?

  • Are all of the item writing guidelines being followed?

How does a review happen?

  • You will be assigned a review as a pull request submitted by another item writer

  • You should review the items listed in the pull request using the expert review link in the pull request

  • The linked task will tell you the requirements the items should meet

Review decision for each item:

  • Accepted: Meets all criteria and could be put live with no change required

  • Revision required: Meets most criteria but requires a small change to be accepted. If you would be happy for the change to be made with no re-review, this would be considered a small change.

  • Rejected: Does not meet enough criteria to be accepted without significant changes. This item will be immediately dropped.

Your feedback:

  • Should be specific

  • Should enable the item writer to revise the item so that it will be accepted (if needed)

  • Should make a clear case for rejection (if needed)

The following are examples of what we expect in your review comments:

  • [ ] `123456` Accepted

  • [ ] `abc123` Revisions Required. The item does not meet guidelines multiple choice 4. This item can be accepted if the options are put into numerical order. In addition, there is a typo in the context “teh” should be “the”.

  • [ ] `ligawjegb` Rejected. The item does not test the learning objective. The objective requires that we test the ability to fit a linear model but this item is testing random forest regression. Significant rework requiring further review would be required to resolve the issue.

Did this answer your question?