Improving Fine-Grained Medication Recognition with DINOv2 Projection Head Retrieval

Medication errors continue to plague the healthcare field, stemming from the misidentification of medications in pharmacies and patient care settings. Medications that look alike are often difficult to distinguish from one another, as they share identical geometries and colors. Meanwhile, individual pills and tablets often contain asymmetric imprints and scoring marks that make medications with the same identity look visually different. To counter this, machine learning models have been utilized to predict the identities of medications using visual recognition of various attributes, including shape, color, texture, and size. However, many models fail to perform at a high level, especially given the challenges of the datasets on which they are trained. This paper introduces how attaching a projection head on top of a pre-trained Vision Transformer (DINOv2) and using a 120 epoch cosine-annealing schedule, a low learning rate, and a multi-loss function allowed for the projection head to learn a better retrieval embedding space. During training, the optimal model was selected based on the highest validation two-sided mean top-10 accuracy to reflect real-world grid interfaces and to blend the multi-dimensional vector of a medication’s front and back faces to eliminate noise. This model was thus transformed into a high-performing retrieval system with strong test accuracies including a top-10 accuracy surpassing 97%. These findings suggest that projection head attachment and training on top of the DINOv2 backbone can support a high-precision retrieval system whose end goal may be to potentially reduce cognitive fatigue for pharmacists and patients as a decision-support system.