In this project, we plan to design and analyze new algorithms for sequentially classifying a stream of data points based on a set of costly, noisy, and potentially malicious annotators. Assuming the algorithm obtains noisy labels by adaptively querying selected annotators, we are interested in studying trade-offs between the classification accuracy and the money spent to obtain the labels. We consider scenarios where annotators have fixed costs as opposed to accepting variable payments, where the annotators' accuracy may depend on the features of the data points, and where some annotators may act strategically in order to maximize their profit while minimizing their annotating effort. Our goal is to prove bounds on the learning algorithm's classification error that depend on the (unknown) functions relating data point features and received payments to annotation accuracy.