In this article, we will give you an overview of how to write tests for R, Python, and SQL projects, both Guided and Unguided variants.

Writing Tests in R

Overview of testing in R

Testing for R projects is done using Hadley Wickham's testthat package. After loading in testthat and IRkernel.testthat in the first @tests cell, tests can be included without any additional code in the remaining @tests cells.

Examples of testing in R

Below are some examples of tests from our A Visual History of Nobel Prize Winners (R) Guided Project.

Test that a package has been loaded successfully:

test_that("Test that tidyverse is loaded", {
expect_true( "package:tidyverse" %in% search(),
info = "The tidyverse package should be loaded using library()."

Using .last_plot() to inspect the most recent plot and compare it to a correct one.

p <- last_plot()

correct_p <- ggplot(prop_usa_winners, aes(decade, proportion)) +
geom_line() + geom_point() +
scale_y_continuous(labels = scales::percent, limits = 0:1, expand = c(0,0))

test_that("The plot uses the correct dataset", {
expect_equivalent(p$data, correct_p$data,
info = "The plot should show the data in prop_usa_winners.")

There's a nice overview of how to use it under the "testthat package" heading in this "Writing tests" blog post. Here's a key extract from the post, documenting some (but not all) functions:

The testthat package includes a number of helper functions for checking whether results match expectation.

  • expect_error() if the code should throw an error.

  • expect_warning() if the code should throw a warning

  • expect_equal(a, b) if a and b should be the same (up to numeric tolerance)

  • expect_equivalent(a, b) if a and b should contain the same values but may have different attributes (e.g., names and dimnames)

For further information, the testing chapter in the R packages book gets into the details, along with workflow advice and concrete examples:

Writing Tests in Python

Overview of testing in Python

Testing in Python is done using the nose framework, which is a Python unit test framework. After the ipython_nose extension is loaded via the first cell using %load_ext ipython_nose, you simply need to begin each @tests cell with %%nose and then include your tests.

There are instructions in each notebook for installing nose and the ipython_nose extension so that you can run your tests locally.

Examples of testing in Python

Below are some examples of nose tests from our A Visual History of Nobel Prize Winners (Python) Guided Project.

Test a package has been loaded and aliased correctly:

def test_pandas_loaded():
assert pd.__name__ == 'pandas', \
"pandas should be imported as pd"

Compare the output of a student to the correct output:

last_value = _

correct_value = nobel['birth_country'].value_counts().head(10)

def test_last_value_correct():
assert last_value.equals(correct_value), \
"The number of prizes won by the top 10 nationalities doesn't seem correct... Maybe check the hint?"
  • Note that the correct output must be defined in the testing cell. You cannot refer to code within the solution cell when writing tests.

  • Student output is defined by by using an underscore to refer to the last value returned, which on will be the output of the student's code.

Test that the y-axis data from a plot is the correct data.

def test_y_axis():
assert all(ax.get_lines()[0].get_ydata() == prop_usa_winners.usa_born_winner), \
'The plot should be assigned to ax and have usa_born_winner on the y-axis'

Writing Tests in Python... for SQL Projects

Overview of testing in SQL Projects

Similar to tests in Python projects, testing in SQL projects is done in Python using the nose framework, which is a Python unit test framework. After the ipython_nose extension is loaded via the first cell using %load_ext ipython_nose, you simply need to begin each @tests cell with %%nose and then include your tests.

Examples of testing in SQL Projects

Testing that the student's output is a SQL ResultSet:

last_output = _

def test_output_type():
assert str(type(last_output)) == "<class ''>", \
"Please ensure an SQL ResultSet is the output of the code cell."

It can be helpful to include the above test after every task. Then, convert the SQL ResultSet into a DataFrame and test the DataFrame:

results = last_output.DataFrame()

def test_results():
assert results.shape == (10, 6), \
"The results should have six columns and ten rows."
assert results.columns.tolist() == ["game", "platform", "publisher", "developer", "games_sold", "year"], \
'The results should have columns named "game", "platform", "publisher", "developer", "games_sold", and "year".'
assert last_output.DataFrame().loc[0, 'games_sold'] == 82.9, \
"The top selling game should be Wii Sports with 82.9 million copies sold."

Because SQL ResultSets have different data types than you’d typically see in a pandas DataFrame, you may need to update the data type that a test is checking for. Floats, which are saved as Decimals in ResultSets, are a particular thing to look out for when testing. For example, this test does not pass because the 82.90 that the test is checking for is a float in the Python test code, but stored as a Decimal object in the DataFrame.

def test_results():
assert _.DataFrame().loc[0, 'games_sold'] == 82.90,\
"The top selling game should be Wii Sports with 82.90 million copies sold."

The solution is to import the Decimal object and update the test to check for a Decimal object rather than a float.

from decimal import Decimal as D

def test_results():
assert _.DataFrame().loc[0, 'games_sold'] == D('82.90'),\
"The top selling game should be Wii Sports with 82.90 million copies sold."

Testing for Guided Projects

For DataCamp projects as a whole, we don't need to and can't possibly test everything. We just need to give students some help along the way. It is difficult/impossible to test things not saved to objects and cells with multiple plots. And that's okay!

However, testing should be rigorous enough to ensure that interdependent cells still function correctly. If a student needs to use a DataFrame throughout a project, there should be sufficient testing to ensure that they don’t arrive at a later task with incorrect data.

A good rule of thumb is to have at least one test for each instruction. For example, a task that asks the learner to join two tables, rename a column, and then filter the results might test the number of columns, the names of the columns, and the number of rows. This way, the learner is tested not only on the end result of their code, but intermediate steps as well. This also means that test failure messages can guide the learner to the next step, so it's important that tests be ordered in the same order a typical learner would approach the task.

Testing for Unguided Projects

Testing for Unguided Projects is relatively the same as it is for Guided Projects (i.e. using nose and testthat), but as the focus on these projects is the outcome, rather than the process, they should be considerably shorter and less focused on the methodology.

However, it is also generally best practice to design Unguided Project tests with the following considerations:

  • Are there alternate solutions that are still correct, but will produce different answers? If so, the test should be written to accept additional viable answers.

  • Can you diagnose common errors/problems? For example, if missing a cleaning step will produce an incorrect answer, you can first compare the learner solution with that result (e.g. learner_solution != missed_cleaning), and include the relevant feedback message.

  • Are there additional checks you can perform on the answer to help steer students in the correct direction and prevent simple mistakes (e.g. data type checks, DataFrame size, etc).

  • Can you write the tests to be lenient enough to accept answers that are close enough? For example, when testing for a string answer, converting the student solution to lower-case prior to testing.

Testing your tests

Learn about testing your tests both locally and on in this article on checking your tests.

If you are stuck, don’t hesitate to reach out via Asana or on GitHub (@datacamp-contentquality)!

Did this answer your question?