As well as writing items, you will need to provide reviews of items written by other item writers. Reviews will be allocated to you as pull requests in GitHub.

Assuming the process has been followed in Creating a Pull Request, you should have everything that you need in the pull request. You should not need to use Teach for reviews.

You will have a link to Expert Review. This is a tool that allows you to see all test items in a similar way to how a test taker will experience them. The main difference is that you will immediately be able to see whether your solution was considered correct or not.

The item writer should have listed the item keys for all of the items they have added and are therefore requesting be reviewed, along with the learning objective that item is intended to test. You will be able to match to items in expert review based on this item key.

For every item to be reviewed, you must review against the criteria in the guidelines. For each item you must give one of the three decisions described below.

  • Accepted: Meets all criteria and could be put live with no change required

  • Revision required: Meets most criteria but requires a small change to be accepted. If you would be happy for the change to be made with no re-review, this would be considered a small change.

  • Rejected: Does not meet enough criteria to be accepted without significant changes. This item will be immediately dropped.

Items may be accepted with no further explanation. However for both “Revisions Required” and “Rejected” you must give clear feedback on which criteria are not met. For revisions required you must give clear indications of what needs to be changed for the criteria to be met. If you do not provide clear feedback you will be asked to repeat the review process.

Item writers are not compensated for items that are rejected. For this reason we ask that you think carefully before making the decision to reject an item and give a clear case for the rejection. However, you are also responsible for helping us to ensure that we continue to keep a high standard in our assessments, so we do ask that you do not allow items to proceed if they don’t meet expectations.

The following are examples of what we expect in your review comments. To maintain traceability of the item writing/review process, please always add new comments, do not edit existing comments.

  • [ ] `123456` Accepted

  • [ ] `abc123` Revisions Required. The item does not meet guidelines multiple choice 4. This item can be accepted if the options are put into numerical order. In addition, there is a typo in the context “teh” should be “the”.

  • [ ] `ligawjegb` Rejected. The item does not test the learning objective. The LO requires that we test the ability to fit a linear model but this item is testing random forest regression. Significant rework requiring further review would be required to resolve the issue.

Did this answer your question?