Submission Correctness Tests (SCTs) are scripts to check that a student's solution matches the project solution. Once SCTs have been authored, you can check them by previewing your project from the Teach Editor and running the solution code. This will produce one of four outputs (see below).
How to approach SCTs
SCTs are designed to check any outputs that must be produced as per the project's Instructions. They should check that the variable(s) exist, they are in the correct format (if this is specified), and contain the correct values.
When authoring SCTs you should put yourself into the learners' shoes, thinking about the possible ways they may try to solve your project. This will allow you to design tests that allow the project to be solved in various ways, as long as the end results are correct.
SCTs are not designed to check intermediate steps used to produce the final output. As an example, in this project, learners work with LAPD crime data and are required to find the hour of day with the largest volume of crime, saving as an integer variable called peak_crime_hour. In order to find this information they may choose to extract the hour of day from a column called "TIME OCC" (time that the crime occurred) and store as a new column in the DataFrame called "HOUR OCC". However, we haven't specified that they need to create this column, so we should not test for it's existence.
Writing SCTs
The way the SCTs are formatted in projects is a list of assertions written in the native language (Python for Python or SQL projects, R for R projects). The Workspace will evaluate each of the assertions in order until one fails. If a failure occurs then this error will be passed up to the project and displayed to the learner.
We encourage you to catch all possible errors and return these as custom messages using the assertions, for example, it would make sense to check that df is defined before checking how many columns it has. This technique will allow you to give the learner more meaningful feedback than generic error messages.
Conventions
We recommend to start with simple tests and increase in complexity. For example, check the number of columns, then the names of columns, then the values in the columns.
Use comments to help describe to future content developers what your tests are doing.
For example, # This test checks that the column names are correct.
Always return a string describing to the user what is wrong when the assertion fails.
For example, “The output should contain 4 columns, please check your output and try again”.
Always try out your assertions in the workspace before adding them to a real project. You can do this by adding the assertion alongside your code in the workspace editor and running the cell to see the result. Hopefully, the new teach projects editor will help with this soon.
When the project is live, check the feedback on SCTs and hints to ensure it is helping. For example, you may see high instances of generic errors which are not helpful to the learner, or a high proportion of learners voting that the feedback is not helpful.
Lastly, don’t call out to external data sources, data bases or APIs in your tests.
Examples of effective tests
Python/SQL
Please use these Python tests, which are from this project, as inspiration for what is possible. Feel free to try things beyond this too.
# Question 1
# Check peak_crime_hour is correct
assert peak_crime_hour == 12, "Did you identify the hour of day with the largest volume of crime?"
# Question 2
assert peak_night_crime_location == "Central", "Did you find the location with the most crime between 10pm and 4am? Expected somewhere else."
# Question 3
# Correct labels
victim_ages_labels = sorted(['<18', '18-25', '26-34', '35-44', '45-54', '55-64', '65+'])
# Check labels exist in index
assert sorted(victim_ages.index.tolist()) == sorted(victim_ages_labels), "Did you split victim ages into groups aligned with the Instructions?"
# Correct number of crimes per age bracket
correct_victim_age_values = [4528, 28291, 47470, 42157, 28353, 20169, 14747]
# Check victim_ages is a pandas Series
assert isinstance(victim_ages, pd.Series), "Did you save victim_ages as a pandas Series?"
# Correct answer
correct_victim_ages = {'18-25': 28291,
'26-34': 47470,
'35-44': 42157,
'45-54': 28353,
'55-64': 20169,
'65+': 14747,
'<18': 4528}
# Individually check the number of crimes per age bracket are correct
for label, val in correct_victim_ages.items():
assert val in victim_ages.values.tolist(), f"Expected {label} to have {val} crimes in victim_ages, please recheck how you have grouped the ages into bins."
R Projects
Here is an example of R SCTs, which use the stopifnot() function.
# Check if the variable best_feature_df is defined
stopifnot("You should have a dataframe called best_feature_df, please make sure you have assigned this." = exists("best_feature_df"))
# Check if best_feature_df is the right data type
stopifnot("You should have a dataframe called best_feature_df, it looks like you have a different data type assigned to best_feature_df, please check and try again." = class(best_feature_df) == "data.frame")
# Check if best_feature_df contains one row and two columns
stopifnot("The output should contain one row and two columns, please recheck your best_feature_df." = all.equal(dim(best_feature_df), c(1, 2)))
# Check for the correct feature
stopifnot("Expected a different feature in best_feature_df, please recheck your model evaluation." = best_feature_df[1,1] == "driving_experience")
# Check accuracy score
stopifnot("Expected a different accuracy score in best_feature_df, please recheck your model evaluation." = round(best_feature_df[1,2], 1) == 0.8)
Types of outputs
Below are the types of outputs produced following execution of SCTs, which run when a learner clicks "Submit Project" (note that this button will trigger the running of all cells in the notebook, following by the execution of sct.py / sct.R).
1. Success
Triggered when none of the SCTs throw an error including an AssertionError (assertion doesn’t match).
Always returns exactly what’s shown above, and then triggers the end of the project.
2. AssertionError
Triggered when the SCTs don’t match and therefore one returns an AssertionError and message.
This one will return “Your solution doesn’t look quite right” followed by the exact message returned by your AssertionError.
3. NameError
Triggered when your SCT attempts to access a variable that is now defined. Such as asserting df = something, but the learner hasn’t created a variable called df yet.
This always returns the message above, with the variable name at the end. You may wish to assert if a variable exists and return a custom message instead.
4. Any other error
Triggered when any other error is returned in the SCTs or the learners code.
This message is not customizable.