Submission Correctness Tests (SCTs) are scripts to check that a student's solution matches the project solution. Once SCTs have been authored, you can check them by previewing your project from the Teach Editor and running the solution code. This will produce one of four outputs.
Types of output
1. Success
Triggered when none of the SCTs throw an error including an AssertionError (assertion doesn’t match).
Always returns exactly what’s shown above, and then triggers the end of the project.
2. AssertionError
Triggered when the SCTs don’t match and therefore one returns an AssertionError and message.
This one will return “Your solution doesn’t look quite right” followed by the exact message returned by your AssertionError.
3. NameError
Triggered when your SCT attempts to access a variable that is now defined. Such as asserting df = something, but the learner hasn’t created a variable called df yet.
This always returns the message above, with the variable name at the end. You may wish to assert if a variable exists and return a custom message instead.
4. Any other error
Triggered when any other error is returned in the SCTs or the learners code.
This message is not customizable.
Writing SCTs
The way the SCTs are formatted in projects is a list of assertions written in the native language (Python for Python or SQL projects). The workspace will evaluate each of the assertions in order until one fails, this error will be passed up to the project and returned as explained above.
We encourage you to catch all possible errors and return these as custom messages using the assertions, for example, it would make sense to check that df is defined before checking how many columns it has. This technique will allow you to give the user more meaningful feedback than generic error messages.
One limitation is that any error originating from inside the workspace that is not an AssertionError or NameError will result in a generic error response. It may be difficult for learners to figure out why their code is returning this. We may wish to provide a hint about this or otherwise guide them to resolving the error (maybe the new ChatGPT feature in workspace will help with this issue).
Conventions
We recommend to start with simple tests and increase in complexity. For example, check the number of columns, then the names of columns, then the values in the columns.
Use comments to help describe to future content developers what your tests are doing.
For example, # This test checks that the column names are correct.
Always return a string describing to the user what is wrong when the assertion fails.
For example, “The output should contain 4 columns, please check your output and try again”.
Always try out your assertions in the workspace before adding them to a real project. You can do this by adding the assertion alongside your code in the workspace editor and running the cell to see the result. Hopefully, the new teach projects editor will help with this soon.
When the project is live, check the feedback on SCTs and hints to ensure it is helping. For example, you may see high instances of generic errors which are not helpful to the learner, or a high proportion of learners voting that the feedback is not helpful.
Lastly, don’t call out to external data sources, data bases or APIs in your tests.
Examples of effective tests
Please use these Python tests as inspiration for what is possible. Feel free to try things beyond this too.
# Check the shape of the output
assert df.shape == (9, 4), "The output should contain nine rows and four columns, please recheck your query."
# Check the names of the columns
assert df.columns.tolist() == ['industry', 'year', 'num_unicorns', 'average_valuation_billions'], \
"Have you named the columns correctly? Remember that some of the columns require you to create an alias."
# Check the correct industries have been returned
assert ['E-commerce & direct-to-consumer', 'Fintech', 'Internet software & services'] in df["industry"].unique(), \
"""The industries in the output don't match the best performing industries in terms of number of unicorns and average valuations. Please recheck your query."""
# Check there are three rows for each year
assert dict(df["year"].value_counts()) == {2021: 3, 2020: 3, 2019: 3}, \ "There should be three rows for each of the years 2019, 2020, and 2021. Did you filter for these years?"
# Check industries for 2021
twenty_twenty_one = df[df["year"] == 2021]
twenty_twenty_one_industries = ['E-commerce & direct-to-consumer', 'Fintech', 'Internet software & services']
assert twenty_twenty_one[twenty_twenty_one["industry"].isin(twenty_twenty_one_industries)]["industry"].all(), \
"One of more of the industries for 2021 do not match what we expected, please recheck your query."
# Check industries for 2020
twenty_twenty = df[df["year"] == 2020]
twenty_twenty_industries = ['E-commerce & direct-to-consumer', 'Fintech', 'Internet software & services']
assert twenty_twenty[twenty_twenty["industry"].isin(twenty_twenty_industries)]["industry"].all(), \
"One of more of the industries for 2020 do not match what we expected, please recheck your query."
# Check industries for 2019
twenty_nineteen = df[df["year"] == 2019]
twenty_nineteen_industries = ['E-commerce & direct-to-consumer', 'Fintech', 'Internet software & services']
assert twenty_nineteen[twenty_nineteen["industry"].isin(twenty_nineteen_industries)]["industry"].values.tolist() == ['E-commerce & direct-to-consumer', 'Fintech', 'Internet software & services'], \
"One of more of the industries for 2019 do not match what we expected, please recheck your query."
# Check the number of unicorns for the top industries in 2021
assert twenty_twenty_one["num_unicorns"].isin([47, 138, 119]).all(), \
"Please recheck your calculation for the number of unicorns for the top three industries in 2021, we expected different values."
# Check the number of unicorns for the top industries in 2020
assert twenty_twenty["num_unicorns"].isin([16, 15, 20]).all(), \
"Please recheck your calculation for the number of unicorns for the top three industries in 2020, we expected different values."
# Check the number of unicorns for the top industries in 2019
assert twenty_nineteen["num_unicorns"].isin([12, 20, 13]).all(), \
"Please recheck your calculation for the number of unicorns for the top three industries in 2019, we expected different values."
# Check average_valuation_billions values for 2021
assert twenty_twenty_one["average_valuation_billions"].isin([2.47, 2.75, 2.15]).all(), \
"Have you calculated the average valuation in billions of dollars for unicorns in 2021, rounded to two decimal places?"
# Check average valuations for top industries in 2020
assert twenty_twenty["average_valuation_billions"].isin([4.00, 4.33, 4.35]).all(), \
"Have you calculated the average valuation in billions of dollars for unicorns in 2020, rounded to two decimal places?"
# Check average valuations for top industries in 2019
assert twenty_nineteen["average_valuation_billions"].isin([2.58, 6.8, 4.23]).all(), \
"Have you calculated the average valuation in billions of dollars for unicorns in 2019, rounded to two decimal places?"