Python module for evaluation of subjective truthfulness of peer prediction mechanisms.
Dependencies for the module are:
-
python >= 3.6
-
numpy >= 1.13.3
-
matplotlib >= 2.0.2
-
pyjags >= 1.2.2
Additional dependencies for pyjags might include:
-
JAGS >= 4.3.0
-
gcc >= 4.8.1
Performs evaluation of subjective truthfulness of Output Agreement mechanism.
| Parameters: | reports: numpy ndarray of integers |
| Report(s) of a single or multiple agents. Shape of the array should be N (number of agents) * M (number of tasks) for N >= 1 and M >= 1. Entry (i, j) shows report of agent i on task j. Each report can be either 0 or 1. | |
| beliefs: numpy ndarray of floats | |
| Belief(s) of a single or multiple agents on other agents observing signal 1. Shape of the array should be equivalent to shape of reports. Entry (i, j) shows belief of agent i on task j. Each belief can be between 0 and 1. | |
| sampling: boolean (default=False) | |
| Closed-form method used if sampling=False. Estimation method is used if sampling=True. | |
| num_samp: integer (default=1000) | |
| Number of peers to sample if estimation method is used (sampling=True). Should be a positive number. | |
| Attributes: | truthfulness_: numpy ndarray |
| Shape of this array is N * M. Entry (i, j) = 1 if mechanism is truthful for agent i on task j, (i, j) = 0 if mechanism is untruthful and (i, j) = 0.5 if truthful and untruthful reporting produce equal rewards. | |
| payments_: numpy ndarray of floats | |
| Shape of this array is N * M. Entry (i, j) shows payment to agent i for task j given her original report. | |
| payments_c_: numpy ndarray of floats | |
| Shape of this array is N * M. Entry (i, j) shows payment to agent i for task j given her complementary (opposite) report. |
Performs evaluation of subjective truthfulness of Shadowing Method.
| Parameters: | reports: numpy ndarray of integers |
| Report(s) of a single or multiple agents. Shape of the array should be N (number of agents) * M (number of tasks) for N >= 1 and M >= 2. Entry (i, j) shows report of agent i on task j. Each report can be either 0 or 1. | |
| beliefs: numpy ndarray of floats | |
| Belief(s) of a single or multiple agents on other agents observing signal 1. Shape of the array should be equivalent to shape of reports. Entry (i, j) shows belief of agent i on task j. Each belief can be between 0 and 1. | |
| sampling: boolean (default=False) | |
| Closed-form method used if sampling=False. Estimation method is used if sampling=True. | |
| num_samp: integer (default=1000) | |
| Number of peers to sample if estimation method is used (sampling=True). Should be a positive number. | |
| Attributes: | truthfulness_: numpy ndarray |
| Shape of this array is N * M. Entry (i, j) = 1 if mechanism is truthful for agent i on task j, (i, j) = 0 if mechanism is untruthful and (i, j) = 0.5 if truthful and untruthful reporting produce equal rewards. | |
| payments_: numpy ndarray of floats | |
| Shape of this array is N * M. Entry (i, j) shows payment to agent i for task j given her original report. | |
| payments_c_: numpy ndarray of floats | |
| Shape of this array is N * M. Entry (i, j) shows payment to agent i for task j given her complementary (opposite) report. |
Performs evaluation of subjective truthfulness of Dasgupta-Ghosh mechanism with symmetric strategies.
| Parameters: | reports: numpy ndarray of integers |
| Report(s) of a single or multiple agents. Shape of the array should be N (number of agents) * M (number of tasks) for N >= 1 and M >= 3. Entry (i, j) shows report of agent i on task j. Each report can be either 0 or 1. | |
| beliefs: numpy ndarray of floats | |
| Belief(s) of a single or multiple agents on other agents observing signal 1. Shape of the array should be equivalent to shape of reports. Entry (i, j) shows belief of agent i on task j. Each belief can be between 0 and 1. | |
| sampling: boolean (default=False) | |
| Closed-form method used if sampling=False. Estimation method is used if sampling=True. | |
| num_samp: integer (default=100000) | |
| Number of peers to sample if estimation method is used (sampling=True). Should be a positive number. | |
| Attributes: | truthfulness_: numpy ndarray |
| Shape of this array is M. Entry i = 1 if mechanism is truthful for agent i, i = 0 if mechanism is untruthful and i = 0.5 if truthful and untruthful reporting produce equal rewards. Assumes agent adopts same symmetric strategy across all the tasks. | |
| payments_: numpy ndarray of floats | |
| Shape of this array is M. Entry i shows payment to agent i for all tasks assuming agent adopts same symmetric strategy across all tasks given her original reports. | |
| payments_c_: numpy ndarray of floats | |
| Shape of this array is M. Entry i shows payment to agent i for all tasks assuming agent adopts same symmetric strategy across all tasks given her complementary (opposite) reports. |
Aggregation algorithm Spectral Meta-Learner. Main idea of aggregation utilizes eigendecomposition.
Methods:
- aggregate(reports): Start aggregation of reports of multiple agents into one ndarray of size M.
| Parameters: | reports: numpy ndarray of integers |
| Report(s) of multiple agents. Shape of the array should be N (number of agents) * M (number of tasks) for N >= 2 and M >= 2. Entry (i, j) shows report of agent i on task j. Each report can be either 0 or 1. | |
| Returns: | None |
- predict(): Returns ndarray of size M of aggregated reports.
| Parameters: | None |
| Returns: | pred: numpy ndarray of integers |
| Aggregated reports for each of the tasks. |
class peerprediction.aggregation.HGCM(adapt=1000, iterations=1000, chains=4, thin=1, progress_bar=True, threads=1)
Aggregation algorithm Hierarchical General Condorcet Model. Relies on Gibbs Sampling (pyjags) for performing aggregation.
| Parameters: | adapt: non-negative integer (default=1000) |
| Number of adaptation (burnout) iterations for Gibbs Sampler. | |
| iterations: non-negative integer (default=1000) | |
| Number of sampling iterations for Gibbs Sampler. | |
| chains: non-negative integer (default=4) | |
| Number of chains for Gibbs Sampler | |
| thin: non-negative integer (default=1) | |
| Thinning interval for Gibbs Sampler. | |
| progress_bar: boolean (default=True) | |
| Shows sampler’s progress bar if True. | |
| threads: non-negative integer (default=1) | |
| Number of threads used for sampling. One thread can sample from at most one chain. |
Methods:
- aggregate(reports): Start aggregation of reports of multiple agents into one ndarray of size M.
| Parameters: | reports: numpy ndarray of integers |
| Report(s) of multiple agents. Shape of the array should be N (number of agents) * M (number of tasks) for N >= 2 and M >= 2. Entry (i, j) shows report of agent i on task j. Each report can be either 0 or 1. | |
| Returns: | None |
- predict(): Returns ndarray of size M of aggregated reports.
| Parameters: | None |
| Returns: | pred: numpy ndarray of integers |
| Aggregated reports for each of the tasks. |
Aggregation algorithm TwoCoinModel. Employs classical EM algorithm for maximizing likelihood of aggregated reports.
Methods:
- aggregate(reports): Start aggregation of reports of multiple agents into one ndarray of size M.
| Parameters: | epsilon: non-negative float (default=0.01) |
| Convergence threshold for Expectation-Maximization algorithm. EM algorithm converges if 1-norm of diff is less than epsilon. diff is defined as (mut - mut-1), where mut is a vector of posterior estimates at current iteration, and mut-1 is a vector of posterior estimates at previous iteration. | |
| iterations: non-negative integer (default=10000) | |
| Number of iterations for EM algorithm. Algorithm may stop if convergence criterion epsilon is met before total number of iterations is done. If convergence criterion is not met after performing all iterations, warning is displayed. |
| Parameters: | reports: numpy ndarray of integers |
| Report(s) of multiple agents. Shape of the array should be N (number of agents) * M (number of tasks) for N >= 2 and M >= 2. Entry (i, j) shows report of agent i on task j. Each report can be either 0 or 1. | |
| Returns: | None |
- predict(): Returns ndarray of size M of aggregated reports.
| Parameters: | None |
| Returns: | pred: numpy ndarray of integers |
| Aggregated reports for each of the tasks. |
class peerprediction.aggregation.SubjectiveAggregator(reports, beliefs, agg, sampler='simple', n_sampled_peers=100, inner_loops=1, **kwargs)
Evaluates subjective truthfulness of peer prediction mechanisms using aggregation algorithms.
| Parameters: | reports: numpy ndarray of integers |
| Report(s) of a single or multiple agents. Shape of the array should be N (number of agents) * M (number of tasks) for N >= 1 and M >= 2. Entry (i, j) shows report of agent i on task j. Each report can be either 0 or 1. | |
| beliefs: numpy ndarray of floats | |
| Belief(s) of a single or multiple agents on other agents observing signal 1. Shape of the array should be equivalent to shape of reports. Entry (i, j) shows belief of agent i on task j. Each belief can be between 0 and 1. | |
| agg: python class performing aggregation | |
| This is the aggregation algorithm that will be used for evaluation. Possible choices are 3 previous classes of aggregation algorithms mentioned in this API. It is also possible to use any other aggregation algorithm, that has methods aggregate(reports) and predict(). | |
| sampler: string, can be ’simple’ or ’smart_v1’ (default=’simple’) | |
| Method of sampling virtual peers for current agent according to her beliefs. If ’simple’ then peers will be sampled i.i.d. from Bernoulli process. If ’smart_v1’ then original dataset of reports will be used to emulate peers. In ’smart_v1’ we expand the original dataset and flip as few reports as possible such that they converge to current agent’s belief. | |
| n_sampled_peers: non-negative integer (default=100) | |
| Number of virtual peers to sample for each of the actual agents. | |
| inner_loops: non-negative integer (default=1) | |
| Number of times sampling procedure should be repeated. If 1, then values against which current agent will be scored come from single aggregation of sampled peers. For any inner loops > 1 there will be more than one aggregation. In this case final values against which agent will be scored are defined by majority vote over all performed aggregations. | |
| kwargs: python dictionary | |
| Additional arguments required by agg class. |
Methods:
- run_computation(num processes=1): Initiates evaluation of subjective truthfulness.
| Parameters: | num_processes: non-negative integer (default=1) |
| Number of processes to use while running the computation. In order to speed-up computation recommended value for this parameter should be equal to number of available cores. | |
| Returns: | None |
- show_subjective_truthfulness(): Shows information on subjective truthfulness of the mechanism.
| Parameters: | None |
| Returns: | truthfulness: numpy ndarray of integers |
| Shape of this array is N * M. Entry (i, j) = 1 if mechanism is truthful for agent i on task j or 0 otherwise. |
Peer prediction mechanism using aggregation algorithms. Does not evaluate subjective truthfulness or require beliefs of the agents. Produces actual payments for each of the agents on each of the tasks.
| Parameters: | reports: numpy ndarray of integers |
| Reports of multiple agents. Shape of the array should be N (number of agents) * M (number of tasks) for N >= 3 and M >= 2. Entry (i, j) shows report of agent i on task j. Each report can be either 0 or 1. | |
| agg: python class performing aggregation | |
| This is the aggregation algorithm that will be used for evaluation. Possible choices are 3 previous classes of aggregation algorithms mentioned in this API documentation. It is also possible to use any other aggregation algorithm, that has methods aggregate(reports) and predict(). | |
| kwargs: python dictionary | |
| Additional arguments required by agg class. |
Methods:
- produce_payments(): Produces rewards for each agent on each task.
| Parameters: | None |
| Returns: | None |
- show_payments(): Returns array containing payments.
| Parameters: | None |
| Returns: | payments: numpy ndarray of integers |
| Shape of this array is N * M. Entry (i, j) corresponds to reward of agent i on task j. |
Methods:
- draw simplex(pay_mat, save=False, savename=’simplex’): Draws probability simplex with belief model constraints based on payment matrix corresponding to some mechanism. Works for environments with tree signals.
| Parameters: | pay_mat: numpy ndarray of shape 3 * 3 |
| Payment matrix, specifying the mechanism. Entry (i, j) shows which reward current agent gets if she reports i and her peer agent reports j. | |
| save: boolean (default=False) | |
| Shows probability simplex if False. Saves the image in *.png format if True. | |
| savename: string (default=’simplex’) | |
| Savename for *.png file (only if save=True). | |
| Returns: | None |