Submission Correctness Tests (SCTs) are scripts to check that a student's solution matches the project solution. Once SCTs have been authored, you can check them by previewing your project from the Teach Editor and running the solution code. This will produce one of four outputs (see below).
How to approach SCTs
SCTs are designed to check any outputs that must be produced as per the project's Instructions. They should follow a general logic of:
Check that the variable a learner needs to create exists.
Check that the variable is the correct data type/class.
Check that the data is correct, e.g., if it's a pandas DataFrame then first check the column names, then the values, subsetting if necessary to enable granular feedback on where learners might have gone wrong.
When authoring SCTs you should put yourself into the learners' shoes, thinking about the possible ways they may try to solve your project. This will allow you to design tests that allow the project to be solved in various ways, as long as the end results are correct.
SCTs are not designed to check intermediate steps used to produce the final output. As an example, in this project, learners work with LAPD crime data and are required to find the hour of day with the largest volume of crime, saving as an integer variable called peak_crime_hour. In order to find this information they may choose to extract the hour of day from a column called "TIME OCC" (time that the crime occurred) and store as a new column in the DataFrame called "HOUR OCC". However, we haven't specified that they need to create this column, so we should not test for it's existence.
Writing SCTs
The way the SCTs are formatted in projects is a list of assertions written in the native language (Python for Python or SQL projects, R for R projects). The DataLab will evaluate each of the assertions in order until one fails. If a failure occurs then this error will be passed up to the project and displayed to the learner.
We encourage you to catch all possible errors and return these as custom messages using the assertions, for example, it would make sense to check that df is defined before checking how many columns it has. This technique will allow you to give the learner more meaningful feedback than generic error messages.
Conventions
We recommend to start with simple tests and increase in complexity. For example, first check the existence of a variable, then check the data type, then check the number of columns, then the names of columns, then the values in the columns, etc.
Use comments to help describe to future content developers what your tests are doing.
For example, # This test checks that the column names are correct.
Always return a string describing to the user what is wrong when the assertion fails.
For example, “The output should contain 4 columns: "col_one", "col_two", "col_three", and "col_four", please check your code and try again”.
Examples of effective tests
Python/SQL
Please use these Python tests, which are from this project, as inspiration for what is possible. Feel free to try things beyond this too.
import numpy as np # So peak_crime_hour can also be type np.int64
# Question 1
# Check peak_crime_hour exists
assert peak_crime_hour is not None, "Did you create a variable called peak_crime_hour?"
# Check peak_crime_hour exists
assert isinstance(peak_crime_hour, int) or isinstance(peak_crime_hour, np.int64), "Did you create an integer variable called peak_crime_hour."
# Check peak_crime_hour is correct
assert peak_crime_hour == 12, "Did you identify the hour of day with the largest volume of crime? You can get the hour of the crime by extracting the first two values from the 'TIME OCC' column and converting to integer. This number would represent the hour ahead, e.g., 11 covers 11-11:59am."
# Question 2
# Check peak_night_crime_location exists
assert peak_night_crime_location is not None, "Did you create a variable called peak_night_crime_location?"
# Check peak_night_crime_location is a string variable
assert type(peak_night_crime_location) == str, "Did you save peak_night_crime_location as a string?"
# Check peak_crime_hour equals "Central"
assert peak_night_crime_location == "Central", "Did you correctly identify the location with the most crime between 10pm and 4am? After you've extracted the hour of the crime you can group by 'AREA NAME' and count the number of occurrences, sorting to get the largest frequency."
# Question 3
# Check victim_ages exists
assert victim_ages is not None, "Did you create a variable called victim_ages?"
# Check victim_ages is a pandas Series
assert isinstance(victim_ages, pd.Series), "Did you save victim_ages as a pandas Series? Note that the .value_counts() method returns a pandas Series."
# Store the correct labels
victim_ages_labels = ['0-17', '18-25', '26-34', '35-44', '45-54', '55-64', '65+']
# Check labels exist in index
assert sorted(victim_ages.index.tolist()) == sorted(victim_ages_labels), "Did you split victim ages into groups aligned with the Instructions? Remember to exclude negative age values when creating bins, e.g., the lowest value should be 0."
# Correct number of crimes per age bracket
correct_victim_age_values = [4528, 28291, 47470, 42157, 28353, 20169, 14747]
# Check values exist in Series
assert sorted(victim_ages.values.tolist()) == sorted(correct_victim_age_values), "Expected a different count of crimes per age range in victim_ages. Did you use pd.cut() and provide the correct bin and label values?"
# Correct answer
correct_victim_ages = dict(zip(victim_ages_labels, correct_victim_age_values))
# Individually check the number of crimes per age bracket are correct
for key, val in correct_victim_ages.items():
assert victim_ages.loc[key] == val, f"Expected a different number of crimes for {key} age group, please recheck your calculation. The correct labels should be {victim_ages_labels}, and you should use pd.cut() to get the number of crimes per age group."kl
R Projects
Here is an example of R SCTs, which use the stopifnot() function.
# Check if the vector retained_features is defined stopifnot("You should have a character vector called retained_variables, please make sure you have assigned this." = exists("retained_features")) # Check if retained_features is a character vector stopifnot("The vector retained_features should be of type character. Please check it is." = is.character(retained_features)) # Check if retained_features has fewer than ten features. stopifnot("Expected fewer than ten features. See if there are additional variable transformations or variables of low importance that can be removed." = length(retained_features) < 10)
Types of outputs
Below are the types of outputs produced following execution of SCTs, which run when a learner clicks "Submit Project" (note that this button will trigger the running of all cells in the notebook, following by the execution of sct.py / sct.R).
1. Success
Triggered when none of the SCTs throw an error including an AssertionError (assertion doesn’t match).
Always returns exactly what’s shown above, and then triggers the end of the project.
2. AssertionError
Triggered when the SCTs don’t match and therefore one returns an AssertionError and message.
This one will return “Your solution doesn’t look quite right” followed by the exact message returned by your AssertionError.
3. NameError
Triggered when your SCT attempts to access a variable that is now defined. Such as asserting df = something, but the learner hasn’t created a variable called df yet.
This always returns the message above, with the variable name at the end. You may wish to assert if a variable exists and return a custom message instead.
4. Any other error
Triggered when any other error is returned in the SCTs or the learners code.
This message is not customizable.