In order to train Natural Language Processing models, annotated corpora are essential. Unfortunately, resources for training Machine Learning and Deep Learning algorithms are scarce due to the high cost, labor-intensive nature, and lengthy nature of complicated semantic annotation processes. Our work presents a framework for semiautomating difficult tasks based on the human-in-the-loop approach to address this challenge. used this process to produce a Spanish news reliability dataset in order to effectively counteract misinformation and false news.