-
Notifications
You must be signed in to change notification settings - Fork 18
Circumventing Batch Pricing
In July 2015, Amazon increased the price of MTurk HITs that include 10 or more assignments. In short, MTurk charges an additional 20% commission on top of the base commission when a HIT contains 10 or more assignments. This disproportionately affects academic requesters doing large-n survey-experimental research. This tutorial explains how to post a sequence of HITs with 9 (or fewer) assignments in order to obtain a completed number of assignments greater than 10 without incurring this extra charge.
The basic strategy will be to create a series of HITs, each of which (except for the first) has a QualificationRequirement that prevents workers who have done one of the previous HITs from completing an assignment. The example below implements this as a repeat
loop in R that is fully self-contained. You may need to modify it for your purposes.
# set total number of desired assignments
total <- 1000
# create QualificationType
qual <- CreateQualificationType(name="Already completed HIT",
description="Already completed identical HIT before.",
status = "Active")
# generate "DoesNotExist" QualificationRequirement structure
qreq <- GenerateQualificationRequirement(qual1$QualificationTypeId, "DoesNotExist", "")
# create HITType w/o qualification requirement
hittype1 <- RegisterHITType(title = "10 Question Survey",
description = "Something something something",
reward = ".20",
duration = seconds(hours = 1),
auto.approval.delay = seconds(days = 1),
keywords = "survey, questionnaire")
# create HITType w/ qualification requirement
hittype2 <- RegisterHITType(title = "10 Question Survey",
description = "Something something something",
reward = ".20",
duration = seconds(hours = 1),
auto.approval.delay = seconds(days = 1),
keywords = "survey, questionnaire",
qual.req = qreq) # this blocks past workers
# create first HIT
eq <- GenerateExternalQuestion("https://www.example.com/","400")
hit <- CreateHIT(hit.type = hittype1$HITTypeId,
assignments = 9, # IMPORTANT THAT THIS IS <= 9
expiration = seconds(days = 4),
question = eq$string)
# variable to index number of completed assignments
completed <- 0
# list to store assignments into
allassigns <- list()
# start the loop
repeat {
g <- GetHIT(hit$HITId, response.group = "HITAssignmentSummary",
verbose = FALSE)$HITs$NumberOfAssignmentsCompleted
# check if all 9 assignments have been completed
if (as.numeric(g) == 9) {
# if yes, retrieve submitted assignments
w <- length(a) + 1
allassigns[[w]] <- GetAssignments(hit = hit$HITId)
# assign blocking qualification to workers who completed previous HIT
AssignQualification(qual$QualificationTypeId, allassigns[[w]]$WorkerId, verbose = FALSE)
# increment number of completed assignments
completed <- completed + 9
# check if enough assignments have been completed
if(completed < total) {
# if not, create another HIT
hit <- CreateHIT(hit.type = hittype2$HITTypeId,
assignments = 9,
expiration = seconds(days = 4),
question = eq$string)
# wait some time and check again
Sys.sleep(180)
} else {
# if total met, exit loop:
break
}
} else {
# wait some time and check again
Sys.sleep(30) # TIME (IN SECONDS) TO WAIT BETWEEN CHECKING FOR ASSIGNMENTS
}
}
# get all assignments for all HITs as a data.frame
m <- do.call("rbind", allassigns)
This will run by itself until the total number of assignments is reached.
.