link: https://github.com/mitll/StatSWAG

StatSWAG implements several statistical estimators that, given noisy categorical predictions (labels) from multiple labelers for a set of data samples, estimate both the accuracy of each individual labeler and the true label for each data instance.