Statistics

MTurk provides a variety of statistics for assessing your own use of the service ("requester statistics") as well as for tracking worker performance ("worker statistics").

To see a complete list of such statistics, simply look at ListStatistics:

> ListStatistics()
                         Statistic ValueType                        Type
1       NumberAssignmentsAvailable      Long       GetRequesterStatistic
2        NumberAssignmentsAccepted      Long       GetRequesterStatistic
3         NumberAssignmentsPending      Long       GetRequesterStatistic
4        NumberAssignmentsApproved      Long       GetRequesterStatistic
5        NumberAssignmentsRejected      Long       GetRequesterStatistic
6        NumberAssignmentsReturned      Long       GetRequesterStatistic
7       NumberAssignmentsAbandoned      Long       GetRequesterStatistic
8                NumberHITsCreated      Long       GetRequesterStatistic
9              NumberHITsCompleted      Long       GetRequesterStatistic
10            NumberHITsAssignable      Long       GetRequesterStatistic
11            NumberHITsReviewable      Long       GetRequesterStatistic
12      PercentAssignmentsApproved    Double       GetRequesterStatistic
13      PercentAssignmentsRejected    Double       GetRequesterStatistic
14               TotalRewardPayout    Double       GetRequesterStatistic
15             AverageRewardAmount    Double       GetRequesterStatistic
16            TotalRewardFeePayout    Double       GetRequesterStatistic
17                  TotalFeePayout    Double       GetRequesterStatistic
18         TotalRewardAndFeePayout    Double       GetRequesterStatistic
19                TotalBonusPayout    Double       GetRequesterStatistic
20             TotalBonusFeePayout    Double       GetRequesterStatistic
21        EstimatedRewardLiability    Double       GetRequesterStatistic
22           EstimatedFeeLiability    Double       GetRequesterStatistic
23         EstimatedTotalLiability    Double       GetRequesterStatistic
24       NumberAssignmentsApproved      Long GetRequesterWorkerStatistic
25       NumberAssignmentsRejected      Long GetRequesterWorkerStatistic
26       NumberKnownAnswersCorrect      Long GetRequesterWorkerStatistic
27     NumberKnownAnswersIncorrect      Long GetRequesterWorkerStatistic
28     NumberKnownAnswersEvaluated      Long GetRequesterWorkerStatistic
29   NumberPluralityAnswersCorrect      Long GetRequesterWorkerStatistic
30 NumberPluralityAnswersIncorrect      Long GetRequesterWorkerStatistic
31 NumberPluralityAnswersEvaluated      Long GetRequesterWorkerStatistic
32      PercentAssignmentsApproved    Double GetRequesterWorkerStatistic
33      PercentAssignmentsRejected    Double GetRequesterWorkerStatistic
34      PercentKnownAnswersCorrect    Double GetRequesterWorkerStatistic
35  PercentPluralityAnswersCorrect    Double GetRequesterWorkerStatistic

The third column indicates which MTurk API operation can be used to retrieve the relevant statistic. In MTurkR, the GetRequesterStatistic operation is implemented by the GetStatistic function and the GetRequesterWorkerStatistics operation is implemented by the GetWorkerStatistic function.

While you can call GetStatistic and GetWorkerStatistic directly, it is often easier to use the convenience functions RequesterReport and WorkerReport, which retrieve all relevant statistics. For example, here's the output of RequesterReport (you need not specify any arguments):

> RequesterReport()
                    Statistic    Value
1  NumberAssignmentsAvailable     -119
2   NumberAssignmentsAccepted    16937
3    NumberAssignmentsPending        0
4   NumberAssignmentsApproved    15059
5   NumberAssignmentsRejected       57
6   NumberAssignmentsReturned     1541
7  NumberAssignmentsAbandoned      280
8           NumberHITsCreated      253
9         NumberHITsCompleted      251
10       NumberHITsAssignable        0
11       NumberHITsReviewable        2
12 PercentAssignmentsApproved   99.623
13 PercentAssignmentsRejected    0.377
14          TotalRewardPayout  8049.21
15        AverageRewardAmount    0.535
16       TotalRewardFeePayout  804.925
17             TotalFeePayout  900.035
18    TotalRewardAndFeePayout 8854.135
19           TotalBonusPayout   950.34
20        TotalBonusFeePayout    95.11
21   EstimatedRewardLiability        0
22      EstimatedFeeLiability        0
23    EstimatedTotalLiability        0

Similarly, WorkerReport returns all statistics for a worker (specified by their MTurk WorkerId):

> WorkerReport("EXAMPLEWORKER")
                         Statistic Value
1        NumberAssignmentsApproved     1
2        NumberAssignmentsRejected     0
3        NumberKnownAnswersCorrect     0
4      NumberKnownAnswersIncorrect     0
5      NumberKnownAnswersEvaluated     0
6    NumberPluralityAnswersCorrect     0
7  NumberPluralityAnswersIncorrect     0
8  NumberPluralityAnswersEvaluated     0
9       PercentAssignmentsApproved   100
10      PercentAssignmentsRejected     0
11      PercentKnownAnswersCorrect     0
12  PercentPluralityAnswersCorrect     0

These statistics might be particular useful for tracking workers who are frequently rejected (so they can be blocked, using BlockWorker), or who perform well and merit a bonus (using BonusWorker), or to assign qualifications (using AssignQualification). Indeed, AssignQualification can be called with a named worker statistic to be used in specifying a qualification (e.g., to use the results of previous Review Policies to assign qualifications for new HITs).