Netflix patents method to filter out inconsistent raters in video quality tests

1. A computer-implemented method for mitigating scoring inaccuracies in subjective quality experiments, the method comprising: performing, via a propagation solver, a plurality of iterative operations to update a first plurality of parameters of a model that includes a plurality of individual scores assigned by a plurality of participants during a first experiment round to generate a first subjective factor set comprising a different participant inconsistency for each participant in the plurality of participants, wherein performing the plurality of iterative operations includes generating a maximum likelihood estimation formulation of the model and executing the propagation solver to automatically compute a solution for the maximum likelihood estimation formulation to generate the first subjective factor set comprising initiating an update rule generator to generate a set of update rules for updating the first plurality of parameters, and iteratively initiating a propagation engine to iteratively update the first plurality of parameters based on the set of update rules and a refresh rate until a stop threshold is satisfied to generate an optimized plurality of parameters for the model that comprises the solution for the maximum likelihood estimation formulation, wherein the first subjective factor set optimized plurality of parameters corresponds to the first subjective factor set; computing a ranking of the plurality of participants with respect to each other based on the first subjective factor set; and generating a participant evaluation report that includes information indicating a portion of the plurality of participants that, based on the ranking of the plurality of participants with respect to each other, meet a participant inconsistency criterion for exclusion from a second experiment round during which the plurality of participants less the portion of the plurality of participants assign one or more additional individual scores, wherein the participant inconsistency criterion specifies a predefined number of participants included in the plurality of participants that are associated with highest participant consistencies.