See the slides at https://ksaaskil.github.io/introduction-to-property-based-testing/.

Examples of property-based testing including targeted property-based testing written in Elixir with PropCheck.

The project was initialized using mix:

$ mix new elixir-propcheck --app pbt

mix comes with Elixir.

Example of stateful property-based testing from PropEr testing book.

Examples of property-based testing using the Hypothesis library:

• python-hypothesis/test_hypothesis.py: Examples from Hypothesis documentation
• python-hypothesis/test_gitlab_stateful.py: Stateful property-based testing for GitLab-like API

Examples of targeted property-based testing in Erlang. Project initialized with rebar3:

$ rebar3 new lib erlang-targeted-pbt

Some notes below for preparing to demo targeted PBT. More complete examples in Elixir can be found in ./elixir-propcheck/test.

• Targeted property-based testing, A. Löscher and K. Sagonas, 2017.

  "We introduce targeted property-based testing, an enhanced form of property-based testing that aims to make the input generation component of a property-based testing tool guided by a search strategy rather than being completely random"

• Automating targeted property-based testing, A. Löscher and K. Sagonas, 2018.

  "To use [targeted PBT], however, the user currently needs to specify a search strategy and also supply all ingredients that the search strategy requires. - [In this paper], we focus on simulated annealing, the default search strategy of our tool, and present a technique that automatically creates all the ingredients that targeted PBT requires starting from only a random generator."

• Targeted property-based testing with Applications in Sensor Networks, A. Löscher's PhD thesis, 2018.

  "This dissertation presents targeted property-based testing, an enhanced form of PBT where the input generation is guided by a search strategy instead of being random, thereby combining the strengths of QuickCheck-like and search-based testing techniques. It furthermore presents an automation for the simulated annealing search strategy that reduces the manual task of using targeted PBT."

• Andreas Löscher:

  • Google Scholar
  • LinkedIn

• Konstantinos Sagonas:

  • Google Scholar
  • Fascinating papers on PropEr

• Proper: ?FORALL_TARGETED
• PropCheck.TargetedPBT
• hypothesis.target

Also:

• QuickTheories has targeted search for coverage

• PBT relies on generators, functions producing data from given search space
  • Typically sample a small part of the full search space
  • Unguided: no feedback to generator if our samples are good or bad
• Targeted PBT: Give feedback to the generator
  • Couples test execution to data generation
  • "This is more like it, well done!"
  • "This is not a good sample, please try again."

• Complex data generators (recursive)
• Stateful tests
• Generator metrics
• Shrinking (at least partially)
• Variations in data

• Generates data made for the problem at hand
• Can generate data not found with traditional generators
• Can replace complex generators
  • Simplifies generating, for example, unbalanced trees

• Formulated as an optimization problem
  • Task is to maximize a given function
  • Generator produces data leading to larger values -> reward
  • Generator produces data leading to smaller values -> no reward
• Be careful of local optima
  • Short-term vs. long-term rewards -> Non-greedy algorithms

• Search space S
• Target function E: S -> R
  • Mapping from search space to real numbers
  • Also known as energy or utility function
• Task: Minimize E
  • Equivalent to maximizing -E

• All lists of integers with length below 1000
• All valid User objects with given ID
• All HTTP requests accepted by a server

Examples from Hypothesis documentation:

• Number of elements in a collection, or tasks in a queue
• Mean or maximum runtime of a task (or both, if you use label)
• Compression ratio for data (perhaps per-algorithm or per-level)
• Number of steps taken by a state machine

• Execution time

  • S = All lists of integers with length below 1000
  • E = (Negative) time to sort the list

• Response time

  • S = All HTTP requests accepted by the server
  • E = (Negative) server response time

This property searches for input data that maximizes the execution time and it indeed fails, finding examples of lists that take more than a second to sort.

property "targeted quick sort", [:verbose, :noshrink, search_steps: 500] do
    lists = list(integer())
    short_lists = such_that(l <- lists, when: length(l) < 100_000)

    forall_targeted l <- short_lists do
      t0 = :erlang.monotonic_time(:millisecond)
      quick_sort(l)
      t1 = :erlang.monotonic_time(:millisecond)
      spent = t1 - t0
      maximize(spent)
      spent < 1000
    end
  end

• Simulated annealing as optimization algorithm
  • Non-greedy: probabilistic algorithm that may trade short-term rewards for long-term benefits
  • Originates from physics

• Method for finding the global minimum of a function E(s) with respect to s

• Algorithm:

  0. Choose initial temperature T=T_0, initial state s=s_0 and compute e=E(s)
  1. Generate a candidate ("neighbor") s' and compute e'=E(s')
  2. With acceptance probability P(e, e', T), move to new state by assigning s=s', e=e'
  3. If done, exit. Otherwise, update T according to annealing schedule and move to 1.

• Depends on "temperature" T
• In the beginning of the search, T is large
• As search progresses, T -> 0 according to annealing schedule
• T large: Transitions to higher-energy states (e' > e) are likely
• T small: Transitions to higher-energy states are unlikely
• T = 0: Transitions allowed only to smaller-energy states ("greedy" algorithm)
• Example: P(e, e', T) = 1 if e' < e, otherwise P(e, e', T) = exp[-(e'-e) / T]

• Probabilistic, iterative algorithm to minimize given target function
• Requires
  • Candidate generator function neighbor()
  • Acceptance probability function P(e, e', T)
  • Annealing schedule
  • Initial guess s_0 and initial temperature T_0

• Efficient candidate generation requires that you don't "hop around" to random states like crazy: rather try moves to states with similar energy
  • Similar to Metropolis-Hastings
  • Be careful of local minima
  • Also occasional restarts may help if trapped in a bad environment

• Customizable via custom neighbor function (user_nf in propcheck)
• Instead of letting framework decide which neighbors to try, you can define your own neighbor function
• Neighbor function takes the previous data point and a tuple of current depth and temperature and returns the next value to try

# Always add steps right and down at the end of drawn path
def path_next() do
  fn prev_path, , {_depth, _temperature} ->
    let(
      next_steps <- list(oneof([:right, :down])),
      do: prev_path ++ next_steps
    )
  end
end

What are the values of l in the following case?

def list_next() do
  fn _prev_list, , {_depth, _temperature} ->
    [1, 2, 3]
  end
end

property "targeted list generation" do
  forall_targeted l <- user_nf(list(integer()), list_next()) do
    ...
  end
end

Answer: l is always [1, 2, 3].

What's the generated data like in the following case?

def list_next() do
  fn prev_list, , {_depth, _temperature} ->
    prev_list
  end
end

property "targeted list generation" do
  forall_targeted l <- user_nf(list(integer()), list_next()) do
    ...
  end
end

Answer: l is random but fixed, equal to the first randomly drawn list.

• With custom neighbor functions, all generated data is a variation of the first drawn value
• You can get more variation by wrapping targeted search in a forall block
• Test below executes the forall block five times and searches for 10 steps for each block

property "targeted path generation with variation", search_steps: 10, numtests: 5 do
  forall p <- path() do
    # Trick to make a generator from value
    p_gen = let(p_ <- p, do: p_)

    forall_targeted p2 <- user_nf(p_gen, path_next()) do
      {x, y} = List.foldl(p2, {0, 0}, fn v, acc -> move(v, acc) end)
      neg_loss = x - y
      IO.puts("Last point: {#{x}, #{y}}, negative loss: #{neg_loss}")
      maximize(neg_loss)
      true
    end
  end
end