6 QA framework

6.1 Unit testing

6.1.1 Create a test

You can use below code to write a test. Make sure you have testthat package installed and loaded.

test_that(description_of_test, code_of_test)

Example:
context("Tests")
# These tests should either return error as it's intended or be silent
test_that("User Data", {
  expect_error(test_data_user())
  expect_error(test_data_user(matrix()))
  expect_error(test_data_user(data.frame()))
  foo <- data.frame(1)
  names(foo) <- NULL
  expect_error(test_data_user(foo))
  expect_warning(test_data_user(data.frame(a = 1, a = 2, check.names = FALSE)))
  expect_silent(mtcars)
})

Now, we will look into what kind of tests are possible. Remember, below tests are like bricks, you can build anything with them. Idea should be to keep them simple and use as much as you like. ### Equality expectations

See if an object is equal to value.

expect_identical()

expect_equal()

expect_equivalent()

expect_reference()

6.1.2 Expect

Use it for writing custom expectations.

expect(ok, failure_message, info = NULL, srcref = NULL, trace = NULL)

6.1.3 Expect error

Check if code throws an error.

expect_error()

expect_condition()

6.1.4 Expect invisible

These are pretty cool tests as we can test the function which might return no visible output.

expect_invisible(call, label = NULL)

expect_visible(call, label = NULL)

6.1.5 Expect length

It can be helpful to check length of a vector. See if it is as per requirement of software design or function desgn.

expect_length(object, n)
# where n is length
# expect_length(2:20, 18)

6.1.6 Expect match

Check if a string matches to regular expression.

expect_match( object_to_test, regular_expression)

6.1.7 Expect message

This can come handy when you want to check if the code you have written produces warning or messages.

expect_message(object)

Example:
dummy_function <- function(a){
  if (a == 0){
    message("a is zero")
    return(a)
  }
}
expect_message(dummy_function(0), "a is zero")
expect_message(dummy_function(1), NA)

expect_warning(object)

Example:
dummy_function <- function(a){
  if (a == 0){
    warning("a is zero")
    return(a)
  }
}
expect_warning(dummy_function(0), "a is zero")
expect_warning(dummy_function(1), NA)

6.1.8 Expect named

Check if object have name(s). You can also check what it s and if there is no name at all.

# to check if object name is from character vector
expect_named(object, expected_names)
# to check if object name is absent
expect_named(object, NULL)

Example:
a <- c( x= 0, y= 1)
expect_named(a)
expect_named(a, c("x", "y"))
expect_named(a, NULL) #check absence of names

6.1.9 Expect null

expect_null(object)

Example:
a <- NULL
expect_null(a)
#below is another way to check for equal
expect_equal(a, NULL)
# also you can check type of 'a' object
expect_type(a, "NULL")

6.1.10 Expect output

How to check if your code print output to the console.

expect_output( object, regular_expression)

Example:

expect_output(str(mtcars),"11 variables")

6.1.11 expect setequal

You canuse this test to check if two vectors contain the same values. We have two variants, setequal and mapequal.

expect_setequal(vector_1, vector_2)

list_ex <- list(x =0, y =1)
expect_mapequal(list_ex, list(x =0, y =1))

6.1.12 Expect silent

If you are sure that a code should not output any kind of output, messages or warnings, you can use this test.

expect_silent(object)

6.1.13 Expect vector

You can check if an object has vector properties.

expect_vector(object)

Example:
if (requireNamespace("vctrs")) {
expect_vector(1:10, ptype = integer(), size = 10)
show_failure(expect_vector(1:10, ptype = integer(), size = 5))
show_failure(expect_vector(1:10, ptype = character(), size = 5))
}

6.1.14 Fail/ Success

What if you want to manually trigger failure or success, you can do that easily.

fail(message = "Failure has been forced", info = NULL)
succeed(message = "Success has been forced", info = NULL)

6.1.15 Fail reporter

This is a reporter. We will see few of them as we move forward. This throws an error if any test fails. This is usually combined with other reporters.

6.1.16 Check inheritance

If you want a way to check if an object is inherited from S3 or S4 class,you can use these tests.

expect_type(x,type)
expect_s3_class(x,class) #for s3
expect_s4_class(x,class) #for s4

Example:
  
show_success(expect_s4_class(a, "data.frame"))
expect_type(a$y, "integer")

6.1.17 Logical expectations

These are used when other tests are not according to your expectations(Haha, no pun intended.) These are not very informative.

expect_true(object)
expect_false(object)

Example:
a <- 1:999
expect_true( length(a) == 999)
expect_false( length(a) == 0)
expect_equal(length(a), 999)

6.1.18 Skip a test

Assume you are developing something and you think you need to skip some tests or getting false failures. You are 100% sure about it. In that case,you can use skip() to skip tests as per requirement.

skip(message)

If you want to include conditions, you can do that too.

skip_if_not(condition, message = deparse(substitute(condition)))
skip_if(condition, message = NULL)

To check if a package is installed. This can come handy for designing troubleshooting mechanism where packages are dependent on each other.

skip_if_not_installed(pkg, minimum_version = "specify version")

Skip if a host is offline. This is handy if a package depends on some website for a functionality and it is offline. Sometimes it’s not obvious where the failure occured, having tests on each step will save a lot of time.

skip_if_offline(host = "bdverse.org")

And lastly, these are service specific. If you are using CRAN, you may want to skip particular tests. Some

skip_on_cran()
skip_on_bioc() 

To skip on a particular operating system.Supported: “windows”, “mac”, “linux” and “solaris”.

skip_on_os(os)

To skip on CI/CD and other services in your project workflow.

skip_on_ci()
skip_on_travis()
skip_on_appveyor()
skip_on_covr()

For translation related.

skip_if_translated(msgid = "'%s' not found") #to check for translation

6.1.19 run all tests with custom location of tests

We may come across a case where you want to run few tests froma directory. test_dir helps to do just that.

test_dir(testthat_examples(), reporter = "minimal") # you can change reporter

Similarly, you have test_file to run tests from specific file.

# store path in path variable
test_file(path, reporter = "summary") # you can change reporter

What if you want to locate a particular file in a testing directory to do some troubleshooting work. You can use test_path for that. It helps you find any test file inside tests/testthat

test_path("path to file")

6.1.20 C++ unit testing

This will allow you to do C++ unit testing if needed.

use_catch(dir = getwd()) #directory of package

6.1.21 Regression test

We can do regresion test using verify_output() which s specifically designed for that. The output is intuitive. It can be used with git.

# The first argument would usually be `test_path("informative-name.txt"`)
# but that is not permitted in examples
path <- tempfile()
verify_output(path, {
head(mtcars)
log(-10)=
"a" * 3
})
writeLines(readLines(path))
# Use strings to create comments in the output
verify_output(tempfile(), {
"Print method"
head(mtcars)
"Warning"
log(-10)
"Error"
"a" * 3
})
# Use strings starting with # to create headings
verify_output(tempfile(), {
"# Base functions"
head(mtcars)
log(-10)
"a" * 3
})

6.1.22 Automatic testing

Well what if you want to run tests after everychanges instead of running them manually. After all, once your testing infrastructure is up and running, you need not test the codebase manually everytime something changes. here auto_test and auto_test_package comes handy.

When you are developing code. Run following.

auto_test(code_path,test_path, reporter = "summary")

And when you are working with a package, you can run following:

auto_test_package(pkg = "package_pa", reporter = default_reporter(), hash = TRUE)

These are pretty much every possible test with testthat. Stay tuned for more updates!

6.2 shinytest

bdverse uses shiny for various packages to build beautiful User Interfaces for different packages. Testing shiny applications is very easy and intuitive, Imagine it like recording a screen and then replaying it to make sure everything works as intended. You do not have to write manual tests either. It is that easy. Let’s begin.

Shiny tests are a way to test Shiny applications. You can install it using following commands.

6.2.1 Install shinytest

library(devtools)
install_github("rstudio/shinytest")

6.2.2 Record a test

Shiny is a little didiffrent when it comes to testing. Here, we record them and then run the tests.

To record tests, we run following command.

library(shinytest)
recordTest("/path_of_shinyapp")

This puts the shiny app in testing/recording mode. You can then run the application and test it the way you think will be helpful to the bdverse. The tests will be recoreded and saved.

6.2.3 Run the recorded test

You can now run the recorded test using following command.

testApp("/path_of_shinyapp")

Another way of doing it. Here the second argument will get tests from tests/ directory.

# testApp("myshinyapp", "mytest")

Note: “mytest” is just name of test script. You can change t n test event recorder if you want to.

If you want to compare the results without rerunning the tests, you can use following command.

snapshotCompare("path_of_shinyapp", "mytest")
#"mytest" s

You can visit here for in depth tests if the above ones are not sufficient for your need.

6.3 Defensive programing

Coming Soon!

6.4 CI

Work in progress. If you know that you want Travis CI to skip a build (e.g., you’ve just edited the ReadMe file), include [ci skip] or [skip ci] anywhere in the commit message.

You can skip to build in Travis CI by putting keywords [skip ci] or [ci skip] in your commit messages. This can be used when doing minor changes such as fixing a typo in documentation,etc.

6.4.1 tic

6.4.1.1 Installation with R Studio.

remotes::install_github("ropensci/tic")

then A GITHUB_PAT needs to be set to create the SSH key pair required for deployment on GitHub Actions. Please call usethis::browse_github_pat(), follow the instructions and then call use_tic() again.

run usethis::browse_github_pat() This will open your github tokens page,may ask for your password. Generate the token with default settings provided and copy the token. Now run usethis::edit_r_environ(). This will open .Renviron file of your project and now write GITHUB_PAT=you_token_key_here. Make sure you add an empty new line below that line. Save the file. Restart R for changes to take place.

6.4.1.2 for new user of tic

Run tic::use_tic()

If you already use tic and want to configure a new CI provider do the following

6.4.1.3 For GitHub Actions

tic::use_ghactions_deploy() # (optional for deployment)
tic::use_ghactions_yml() # optional: Change `type` arg to your liking
tic::use_tic_r("package", deploy_on = "ghactions")

(all of the above in one call)
tic::use_tic(wizard = FALSE, linux = "ghactions", mac = "ghactions",
windows = "ghactions", matrix = "ghactions", deploy = "ghactions")
tic::use_tic_badge("ghactions")
tic::use_update_tic()

You can use usethis::edit_file('.github/workflows/tic.yml') to edit the tic file and comment out platforms.