---
title: "Multiple solutions"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Multiple solutions}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

[learnr]: https://rstudio.github.io/learnr/

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  message = FALSE,
  comment = "#>"
)

library(gradethis)
fibonacci <- c(0, 1, 1, 2, 3, 5, 8, 13, 21, 34)

# stable feedback messages
set.seed(2022-04-29)

grade_html <- function(grade) {
  htmltools::tags$div(
    role = "alert",
    class = if (!length(grade$correct)) {
      "alert"
    } else if (isTRUE(grade$correct)) {
      "alert alert-success"
    } else {
      "alert alert-danger"
    },
    htmltools::HTML(commonmark::markdown_html(grade$message))
  )
}

chunk_code <- function(label) {
  paste(knitr::knit_code$get(label), collapse = "\n")
}

grade_example_factory <- function(solution_code, grader) {
  default_grader <- force(grader)
  function(label, grader = NULL) {
    if (is.null(grader)) grader <- default_grader
    user_code <- chunk_code(label)
    grade_html(grader(mock_this_exercise(!!user_code, !!solution_code)))
  }
}
```

```{css echo=FALSE, eval = !identical(Sys.getenv("IN_PKGDOWN", "false"), "true")}
.alert {
  padding: 0.5em 1em;
  margin: 1em 2em;
  border: 1px #eee solid;
  border-radius: 5px;
}
.alert.alert-success {
  background-color: #e4f2dc;
  border-color: #ddedcf;
}
.alert.alert-success,
.alert.alert-success code,
.alert.alert-success code a:not(.close):not(.btn) {
  color: #5c9653;
}
.alert.alert-danger {
  background-color: #f2e2e2;
  border-color: #eed9dc;
}
.alert.alert-danger,
.alert.alert-danger code,
.alert.alert-danger code a:not(.close):not(.btn) {
  color: #ad3532;
}
.alert code {
  background-color: #ffffff55;
}
```
```{css echo=FALSE}
.alert :last-child {
  margin-bottom: 0;
}
```

## Overview

Not all exercises have exactly one right answer:
there could be a few different ways to arrive at the same result,
or the exercise prompt could be open-ended and ask the student to pick from a selection of possibilities.
In either case, and with minimal changes to your learnr tutorial,
gradethis can accommodate exercises with multiple solutions!

To signal that your exercise has multiple solutions,
write all of the solutions in the exercise's `-solution` chunk,
separating the solutions with **code headers** —
a comment followed by at least four dashes.

```
# code header ----
```

You can think of the code headers as _labels_ for each solution variant.
Here's an example `-solution` chunk for an exercise with two possible solutions
(the [full exercise](#example-different-results) is presented below).

````markdown
```{r average-solution}`r ''`
`r chunk_code("average-solution")`
```
````

There are two types of exercises with multiple solutions:

1. **Multiple solutions, same result.**
   When the solutions describe multiple ways of reaching the same final result,
   gradethis provides code feedback hints based on the closest solution to the user's code.

   In most cases, you'll only need to write out the solutions in the `-solution` chunk, and gradethis will take care of the rest.
   Learn more in [Multiple solutions, same result](#same-result) below.

2. **Multiple solutions, different results.**
   Other exercises have more than one acceptable result.
   In these exercises, multiple solutions can be used both for grading students' results
   and for providing code feedback.

   In addition to writing out the solutions in the `-solution` chunk,
   you may also need to modify your grading code slightly.
   Learn more in [Multiple solutions, different results](#different-results) below.

In both cases,
multiple solutions are written in the same way in the `-solution` chunk,
and minimal changes are needed in your grading code.
For advanced grading requirements,
gradethis adds two objects into the grading environment:
[`.solution_code_all`](#solution-code-all) and [`.solution_all`](#solution-all).
We'll cover both straight-forward and advanced uses in the examples that follow.

## Multiple solutions, same result {#same-result}

If all your solutions have the same result,
you might not need to do anything special in your grading code,
as we'll see in the next example.

### Example

In this example,
students are asked to convert a numeric value to a whole number.
The most likely solution is to use the `round()` function,
but a creative student might call `floor()` or `as.integer()`.

Notice that we include all three variations in the `round-solution` chunk below,
but we've placed the answer we consider "most correct" as the last solution in the chunk.
The last solution in the chunk will be the default solution used by gradethis
when the student's code is not a good match for any of the solutions.

````markdown
```{r setup}`r ''`
library(learnr)
library(gradethis)
```

Convert 5.2 to a reasonable whole number.

```{r round, exercise = TRUE}`r ''`
____(5.2)
```

```{r round-solution}`r ''`
# floor ----
floor(5.2)

# as.integer ----
as.integer(5.2)

# round ----
round(5.2)
```

```{r round-check}`r ''`
grade_this({
  # provide code hints if submission is wrong
  fail_if_code_feedback()

  # pass if the value is correct
  pass_if_equal()

  # or fallback to a failing grade
  fail(hint = TRUE)
})
```
````

```{r echo = FALSE}
solution_code <- "
# floor ----
floor(5.2)

# as.integer ----
as.integer(5.2)

# round ----
round(5.2)"

grader <- grade_this({
  fail_if_code_feedback()
  pass_if_equal()
  fail(hint = TRUE)
})

grade_round_example <- grade_example_factory(solution_code, grader)
```

If a student enters any of the correct answers to arrive at the value `5`,
they receive positive feedback and encouragement.

```{r round_correct, eval = FALSE}
as.integer(5.2)
```
```{r round_correct_graded, echo=FALSE}
grade_round_example("round_correct")
```

If a student gives an incorrect answer using `round()`,
the code feedback hint provided by `fail(hint = TRUE)`
will nudge them towards the correct `round()` solution:

```{r round_wrong, eval = FALSE}
round(5.2, digits = 1)
```
```{r echo = FALSE}
grade_round_example("round_wrong")
```

And if a student gives an incorrect answer but their text is closer to one of the other solutions,
the code feedback will nudge the student toward the correct solution.

```{r floor_wrong, eval = FALSE}
flor(5.2)
```

```{r echo = FALSE}
grade_round_example("floor_wrong")
```

```{r as_integer_wrong, eval = FALSE}
as.numeric(5.2)
```

```{r echo = FALSE}
grade_round_example("as_integer_wrong")
```

### Providing different messages for different solutions {#solution-code-all}

Grading functions that use solution code —
like `fail_if_code_feedback()`, `code_feedback()` or `fail(hint = TRUE)` —
will automatically give feedback based on the closest solution to the student's input.
In some cases,
you may want to accept multiple solutions,
but give different feedback depending on the solution the student gave.

To enable advanced custom grading,
gradethis makes all of the solutions to an exercise available to grading code
as `.solution_code_all`,
a named list containing the code for each solution.

You can use code like the following template
to give a special message (`<message>`)
for user input that matches a particular solution
identified by the label used for the code heading above the solution (`<solution_label>`):

```{r eval = FALSE}
correctly_used_solution <-
  is.null(code_feedback(.user_code, .solution_code_all[["<solution_label>"]]))

pass_if(
  correctly_used_solution,
  message = "<message>"
)
```

Some things of note in this template:

1. You can obtain the code for a specific solution using the label in its code heading:
   e.g. `.solution_code_all[["floor"]]` in our example would return `"floor(5.2)"`.

1. You need to explicitly ask for `code_feedback()` for the solution of interest.
   Otherwise, by default, `code_feedback()` will first match the user's code
   with the most option among the set of solutions.

1. `code_feedback()` returns `NULL` when `user_code` matches `solution_code`.
   Testing that the feedback `is.null()` is a robust way to test that
   the student submitted code that matches a solution
   because it ignores minor differences like whitespace and pipe usage.

#### Example

Using the template above,
we can provide some feedback specifically to students who use the `floor()` solution.

````markdown
```{r round-check}`r ''`
grade_this({
  correctly_used_floor <-
    is.null(code_feedback(.user_code, .solution_code_all[["floor"]]))

  pass_if(
    correctly_used_floor,
    "Correct, but remember that `floor()` always rounds down.
    If you want to round to the nearest whole number, `round()` is usually safer."
  )
  pass_if_equal()
  fail(hint = TRUE)
})
```
````

```{r floor_right, eval = FALSE}
floor(5.2)
```

```{r echo = FALSE}
grader_floor_specific <- grade_this({
  pass_if(
    is.null(code_feedback(solution_code = .solution_code_all[["floor"]])),
    "Correct, but remember that `floor()` always rounds down.
    If you want to round to the closest whole number, `round()` is usually safer."
  )
  pass_if_equal()
  fail(hint = TRUE)
})

grade_round_example("floor_right", grader_floor_specific)
```

## Multiple solutions, different results {#different-results}

If your exercise has multiple acceptable results,
you'll need to slightly adapt your code to ensure students' inputs are tested against all acceptable results.

To compare the result of student input to each solution,
you should use `.solution_all` as the first argument in `pass_if_equal()`.
Multiple solution results are saved in an environment called `.solution_all`.
(Because `.solution_all` is an environment, it requires some special handling.
Check [Working with `.solution_all`](#solution-all) for more details.)

Within `pass_if_equal()` messages,
you can use the object `.solution_label`,
which corresponds to the code header of the solution that the student's code matched.

Functions that grade students' code can still be used without any changes.

### Example {#example-different-results}

````markdown
```{r setup}`r ''`
library(learnr)
library(gradethis)
fibonacci <- c(0, 1, 1, 2, 3, 5, 8, 13, 21, 34)
```

Use `mean()` or `median()` to find the average of `fibonacci`, a vector containing the first 10 numbers in the Fibonacci sequence.

```{r average, exercise = TRUE}`r ''`
____(fibonacci)
```

```{r average-solution}`r ''`
`r chunk_code("average-solution")`
```

```{r average-check}`r ''`
`r chunk_code("average-check")`
```
````

```{r average-solution, echo = FALSE, eval = FALSE}
# mean ----
mean(fibonacci)

# median ----
median(fibonacci)
```

```{r average-check, echo = FALSE, eval = FALSE}
grade_this({
  pass_if_equal(.solution_all, "You solved it with `{.solution_label}()`!")
  fail(hint = TRUE)
})
```

```{r average-grader, echo = FALSE}
grade_average_example <- grade_example_factory(
  chunk_code("average-solution"),
  rlang::eval_bare(rlang::parse_expr(chunk_code("average-check")))
)
```

If a student gives either correct solution,
`pass_if_equal()` will give them a passing grade:

```{r mean_right, eval = FALSE}
mean(fibonacci)
```

```{r echo = FALSE}
grade_average_example("mean_right")
```

```{r median_right, eval = FALSE}
median(fibonacci)
```

```{r echo = FALSE}
grade_average_example("median_right")
```

If the student's input doesn't match any of the solutions,
grading will proceed as normal:

```{r sum_wrong, eval = FALSE}
sum(fibonacci)
```

```{r echo = FALSE}
user_code <- paste(knitr::knit_code$get()$sum_wrong, collapse = "\n")
grade_html(grader(mock_this_exercise(!!user_code, !!solution_code)))
```

### Working with all solution results {#solution-all}

`.solution_all` is an environment, not a list.
This carries certain advantages.
Exercises can be graded faster because each solution can be evaluated as needed rather than all up front.

But it also comes with drawbacks.
The most important are:

1. environments can't be subset with numbers, only names, and
2. all elements of an environment need to have a unique name.

If you need to access the results of specific solutions,
make sure you give a unique label to every solution.

```{r eval = FALSE}
# mean ----
mean(fibonacci)

# median ----
median(fibonacci)
```

To reference a specific solution result in grading code,
use the solution label to access the result in the `.solution_all`:

```{r eval = FALSE}
grade_this({
  pass_if_equal(
    .solution_all[["mean"]],
    "That's right! Do you think the mean is skewed?"
  )
  pass_if_equal(.solution_all)
  fail_if_code_feedback()
  fail()
})
```

If you don't give your solutions unique names,
`gradethis` will generate unique names for you.
For example, these solutions

```{r nonunique_names, eval = FALSE}
# mean ----
mean(fibonacci)

# mean ----
mean(fibonacci, na.rm = TRUE)

# median ----
median(fibonacci)

# median ----
median(fibonacci, na.rm = TRUE)
```

have these names:

```{r echo = FALSE}
solution_code <- chunk_code("nonunique_names")
ex <- mock_this_exercise(.user_code = "", .solution_code = !!solution_code)
names(ex$.solution_all[[".solution_labels"]])
```

This means that if you need to access specific solution results in your grading code,
it's almost always better to give your solutions unique labels.
On the other hand,
if your passing message makes use of `.solution_label`,
you may want to duplicate the solution labels.

Recall that the `-check` chunk for the `average` exercise was:

````markdown
```{r average-check}`r ''`
`r chunk_code("average-check")`
```
````

By including the `mean()` and `median()` solution variants with `na.rm = TRUE`
we get an appropriate passing message even though the solution labels are duplicated:

```{r median_right_2, eval = FALSE}
median(fibonacci, na.rm = TRUE)
```

```{r median_right_2_feedback, echo = FALSE}
grade_median_2 <- grade_example_factory(
  solution_code = chunk_code("nonunique_names"),
  grader = rlang::eval_bare(rlang::parse_expr(chunk_code("average-check")))
)
grade_median_2("median_right_2")
```
