EquineML: Machine learning methods for recognition of the pain expressions of horses

Recognition of pain in animals is important because pain compromises animal welfare and can be a manifestation of disease. Animals cannot verbalize their pain, and the use of standardized behavioral pain scales is moreover challenged by the fact that prey animals such as horses display subtle pain behavior. It is beneficial for a prey animal to appear healthy, to lower the interest from predators. This makes pain recognition for veterinarians and caretakers a difficult task, and an automated system has large potential to improve the consistency and efficiency of pain predictions.

Video recording is desirable for ethological studies because it interferes minimally with the animal, in contrast to more intrusive measurement techniques, such as accelerometers. Moreover, to be able to say something meaningful about animal behavior, the subject needs to be studied for longer than the exposure of single images. In deep learning, we have not come as far for video as we have for single images, and even more questions remain regarding what types of architectures should be used and what these models are actually learning.

Specifically, we propose a method for end-to-end equine pain recognition from video, finding, in particular, that the temporal modeling ability of the neural network is important to improve the classification. We surpass veterinarian experts on a dataset with horses under relatively unambiguous experimental pain. Next, we investigate domain transfer between two types of pain in horses: clean, experimental pain and noisier, low-grade orthopedic pain. We find that a smaller, recurrent video model is more robust to domain shift on a target dataset than a large, pre-trained, 3D CNN, having equal performance on a source dataset. We also discuss challenges with learning video features on real-world datasets.

The project was active January 2017 -- December 2021.

Consortium

	Robotics, Perception and Learning KTH Royal Institute of Technology, Sweden Project Director: Hedvig Kjellström

	Clinical Sciences Swedish University of Agricultural Sciences, Sweden Principal Investigator: Pia Haubro Andersen

News

Via forskning kan datorer se hästens 3D-form, Hästsverige, January, 2021.

Publications

Sofia Broomé, Katrina Ask, Maheen Rashid, Pia Haubro Andersen, and Hedvig Kjellström. Sharing pain: Using pain domain transfer for video recognition of low grade orthopedic pain in horses. PLOS ONE, 17(3):e0263854, 2022.
Computer vision

Maheen Rashid, Sofia Broomé, Katrina Ask, Elin Hernlund, Pia Haubro Andersen, Hedvig Kjellström, and Yong Jae Lee. Equine pain behavior classification via self-supervised disentangled pose representation. In IEEE Winter Conference on Applications of Computer Vision, 2022.
Computer vision

Pia Haubro Andersen, Sofia Broomé, Maheen Rashid, Johan Lundblad, Katrina Ask, Zhenghong Li, Elin Hernlund, Marie Rhodin, and Hedvig Kjellström. Towards machine recognition of facial expressions of pain in horses. Animals, doi: 10.3390/ani11061643, 2021.
Pain diagnostics

Zhenghong Li, Sofia Broomé, Pia Haubro Andersen, and Hedvig Kjellström. Automated detection of equine facial action units, arXiv:2102.08983, 2021.
Computer vision

F. M. Serra Bragança, S. Broomé, M. Rhodin, S. Björnsdóttir, V. Gunnarsson, J. P. Voskamp, E. Persson-Sjodin, W. Back, G. Lindgren, M. Novoa-Bravo, C. Roepstorff, B. J. van der Zwaag, P. R. Van Weeren, and E. Hernlund. Improving gait classification in horses by using inertial measurement unit (IMU) generated data and machine learning. Nature Scientific Reports 10, paper number 17785, 2020.
Gait analysis

Katrina Ask, Marie Rhodin, Lena-Mari Tamminen, Elin Hernlund, and Pia Haubro Andersen. Identification of body behaviors and facial expressions associated with induced orthopedic pain in four equine pain scales. Animals 10(11), 2155, 2020.
Pain diagnostics

Maheen Rashid, Hedvig Kjellström, and Yong Jae Lee. Action graphs: Weakly-supervised action localization with graph convolution networks. In IEEE Winter Conference on Applications of Computer Vision, 2020.
Computer vision

Sofia Broomé, Karina Bech Gleerup, Pia Haubro Andersen, and Hedvig Kjellström. Dynamics are important for the recognition of equine pain in video. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Computer vision

Pia Haubro Andersen, Karina B. Gleerup, Jennifer Wathan, Britt Coles, Hedvig Kjellström, Sofia Broomé, Yong Jae Lee, Maheen Rashid, Claudia Sonder, Erika Rosenberg, and Deborah Forster. Can a machine learn to see horse pain? An interdisciplinary approach towards automated decoding of facial expressions of pain in the horse. In International Conference on Methods and Techniques in Behavioral Research, 2018.
Pain diagnostics