‘Action recognition’ for surveillance and medical care

Many crowded places such as airports, train stations, and shopping malls are monitored by closed-circuit television cameras, but how can anomalies be noticed, such as an emergency taking place?

In nursing homes for the elderly and the disabled, how can we ensure that alarms to alert the nurses on duty are sounded as soon as possible, for example when a patient falls and loses consciousness?

The first session of the China Smart Manufacturing Big Data Innovation and Entrepreneurship Competition Final was held in Ningbo, where Dr Bailing Zhang (pictured below) from the Department of Computer Science and Software Engineering at Xi’an Jiao-tong Liverpool University lead a competing team.

Their project, ‘Action recognition based on convolutional neural networks’, won the recognition of the judging panel and was awarded third prize for its technical quality prospective applications in smart surveillance, medical care, and traffic safety.

Through rounds of selections, 13 teams from all around China made it to the final where they held a project roadshow and were questioned by experts.

The XJTLU project was accomplished cooperatively by PhD student Shiyang Yan (pictured below, right) and Fangyu Wu under the guidance of Dr Zhang.

“The recognition of human actions, a basic problem for the visual function of computers, is of important practical significance, and has extensive application prospects in fields such as smart surveillance, automatic image understanding, and self-driving cars,” said Dr Zhang. “We presented to the public our latest research results in this competition.”

The research by Dr Zhang’s team on action recognition and event detection can now be applied to practical operations, such as automatic annotation and surveillance in animated and other kinds of videos.

Team member Shiyang Yan explained that action recognition technology can effectively filter surveillance videos that are many hours in length (as many as 100 million frames) and locate where anomalies happened, saving a lot of time and effort.

In the scenario of medical care, after predefining certain dangerous actions (for example, ‘fall’) action recognition technology can pick out actions as they occur, decide what has happened, and promptly alert the nurses as necessary.

“Another specific application would be the recognition of drivers actions, which plays an essential role in monitoring driving safety and dangerous driver behaviour,” said Shiyang.

According to Dr Zhang, recognition accuracy of the team’s work has reached the leading level in field, and its research paper has been published in international journals. Its accuracy also won it the first prize in the Pascal Visual Object Classes (VOC) challenge, an internationally renowned contest.

Besides action recognition and even detection research, team member Fangyu Wu is involved in research into traffic scene recognition and vehicle re-identification.

In the past three years, Dr Zhang and his student team (including postgraduate students) have had more than 20 research papers published in the international journals and been invited to 25 international conferences where they delivered speeches.

Many of their research results attract attention from industry, with proceeding discussions over how to substantiate the technologies.

By Luyun Shi; photos by Haochen Wang; additional image from Shutterstock
Translated by Boqiang Xiao; edited by Danny Abbasi