Cleaning two decades of student-collected election coverage data

Students working for Professor Barbara Allen in Political Science had compiled data on statewide and presidential election coverage, from videos of news stations, for over two decades. It mixed numbers and categories: timestamps and dates alongside rankings of bias and partisan attitudes that needed qualitative judgment.

The collection methods changed from one cohort to the next, and the sources were added to, edited and altered over the years, leaving sheets of sparse data that were hard to understand. One file had more than 250 columns. Shaun Baron-Furayama ‘23 spent a little over a summer combining and cleaning them.

The write-up, by Aishwarya Varma, is built from an interview with Shaun about what data cleaning involves in practice: fixing or removing incorrect, duplicate and incomplete data, with many chances along the way for records to be duplicated or mislabeled. It is part of the same research that produced the election news scraping project.