[This post is based on Senna Lindner’s Information Science Master thesis]

Digital libraries contain millions of documents, scans, newspapers, and books. As these collections continue to grow, so does a less visible problem: near duplicates. These are files that are not exact copies, but are almost identical, for example because the same document has been scanned twice, or because a thesis has been downloaded again with only a different cover page. Detecting such duplicates is difficult, but important for maintaining large digital collections.

Example of near duplicate scans

In his Master’s thesis, Senna Lindner investigated how near duplicate documents can be detected in the collections of the Koninklijke Bibliotheek (KB), the Dutch National Library. The research compared four existing duplicate detection techniques: the text-based methods SimHash and MinHash, and the image-based methods pHash and dHash. Rather than evaluating these methods individually, the study also explored combinations of methods.

The results showed that there is no single best method for all types of documents. For text-rich PDF theses, MinHash alone achieved perfect detection performance. For scanned documents, however, the best results came from combining text and image-based approaches, achieving an F1 score of over 93%. These hybrid approaches were more accurate, although they also required more computing time.

To understand how such a system could be used in practice, interviews were conducted with staff members of the KB. While interviewees acknowledged that near duplicates exist, they emphasized that usability and workflow integration are as important as detection performance. Participants preferred a human-in-the-loop approach, where the system suggests possible duplicates and staff members verify the results before any action is taken. High recall, finding as many duplicates as possible, was generally considered more important than perfect precision.

The thesis demonstrates that combining multiple detection methods can significantly improve the identification of near duplicates in digital heritage collections. At the same time, successful adoption depends not only on algorithms, but also on designing tools that are transparent, efficient, and fit existing archival practices.

The Github link to the code and data used in Senna’s thesis: https://github.com/sennalindner/Near-duplicate-detection

Senna Lindner. Near Duplicate Detection in Digital Libraries. Master’s Thesis, Vrije Universiteit Amsterdam, in collaboration with the Koninklijke Bibliotheek, 2026.