New project
We propose a project for software and hardware tuning of smartphones for unobtrusive monitoring of the parameters of human physical and psychophysiological health.
The continuous upgrading of smartphones creates favorable conditions for accumulating big data for learning artificial intelligence to solve diagnostic problems. This will lay the basis for remote service for long-term and current prediction and control of the physical and psychophysiological health of a very large number of people in ordinary and extreme situations (stroke, arrhythmia, stress), as well as with chronic diseases.
Primary data will be recorded using measurement tools and procedures commonly used in human physiology and psychophysiology which were adapted for smartphones and gadgets based on our experience gained in the studies and developments for clinical, space and sports medicine.
Artificial intelligence will be trained using original algorithms that are based on game theory methods and have proven themselves in real practice.
For demonstration purposes, we have developed a simple application ("Fast test for fine motor skills", Play Market) which allows quantifying the ability of a person to make accurate proprioceptive and conscious movements at the current moment of time.
We describe below some subjects of our project.
1. The advent of smartphones, smart watches, fitness trackers and other mobile gadgets equipped with a variety of sensors has provided an opportunity to acquire data characterizing the person’s condition for assessing his/her abilities, in particular, health status. The most important task for the analysis of these data is to classify whether the person is healthy or sick, and if he/she is sick then diagnose the exact disease.
For solution of this problem, we propose to use the minimax principle, i.e. the variational principle underlying the game theory widely used in economic, military and other sciences.
2. We have developed this method of classification and used it for the solution of the problem of topical classification of the texts of information reports and procurement announcements on electronic trading platforms. As initial data we used histograms of the occurrence (multiplicity) of words in lemmatized texts, and the classification algorithm was a variant of artificial intelligence based on solving a matrix game constructed by a training sample.
Earlier, we developed original software and hardware solutions for the study of psychomotor reactions and the quality of human movements. To date these solutions have been implemented as software applications for smartphones.
3. The sensors built in smartphones and other gadgets are capable of creating a dataflow characterizing people's health and their psychophysiological condition. In particular, some methods of research adopted and verified in clinical practice, for example, quantitative analysis of movements (Example 1), analysis of cardiograms and heart rate (Example 2), as well as analysis of myograms, electroencephalograms, and other physiological characteristics can be reproduced with a higher information efficiency. With the classification method described above, such data can be used to diagnose various diseases and assess person’s abilities on the basis of histograms constructed by the determined physiological characteristics.
4. We have developed new high-efficiency methods for getting information, for example, sensorimotor tests (Example 4) to assess the state of the sensory sphere, diagnose various diseases, person’s ability and readiness to perform at the moment different complex actions, such as mental and operational work, well-coordinated movements, including as a team member, and others.
Our Android application Fast test for fine motor skills provides a number of visual-motor and stabilometric tests.
The methodology for topical classification of texts from archives and information flows has been thoroughly worked out and tested.
5. Priorities:
5.1. The implementation of mobile applications on the Android and iOS platforms providing for basic visual-motor, audio-motor and stabilometric tests and primary processing and visualization of their results.
5.2. The elaboration of a local server for storing and variational analysis of tabular, physiological, psychophysiological data and texts in major languages.
5.3. The development of a project for the formation of a knowledge base on the basis of documents and digital data of medical and other institutions.
6. The quality of the final product will be determined by comparing the estimates provided by artificial intelligence and expert opinions.
7. The intellectual property will be protected by the methods used by the developers of the information technology.
8. Our own original developments include hardware solutions and a mobile application for visual-motor and stabilometric measurements, as well as a new method for classifying digital objects based on artificial intelligence using game theory methods.
9. The proposed technology of remote medicine will be attractive for various participants in the market of medical and health services:
– for the end user
due to a reduction in the cost of maintaining health due to unobtrusive monitoring of the health status, allowing the user and/or his/her patrons to make a timely decision on the need for medical and other assistance in acute situations and provide long-term health control;
– for the provider
due to the expansion of the volume, content and quality of telemedicine services with a significant reduction in the cost of servicing one client;
– for insurance companies
due to a decrease in insurance payments as a result of improved health of clients.
Example 1. Quantitative analysis of movements
It is a widespread and well-verified method for assessing the state of a person with an extremely wide range of applications – in neurology, traumatology, rehabilitation, orthopedics, physiotherapy, otorhinolaryngology, ophthalmology, manual therapy, sports and special medicine, psychiatry, pharmacology, dentistry, etc.
Currently, electromechanical measuring devices — stabilographic platforms — are most often used to conduct these studies. They allow recording the trajectory of the center of pressure on the support (analog of the center of gravity) while a person is holding a vertical posture during testing which lasts 30-60 seconds.
The present day smartphones, tables and gadgets have built-in motion sensors –
gyroscopes, accelerometers and inclinometers, the signals from which are used to control the gadgets and some applications. Thus, stationary stabilographs can be replaced by mobile devices. Studying motion becomes available to all smartphone users. The range of the tested subjects is expanding and there arises an opportunity to use more accurate mathematical methods of analysis. At the same time, the diagnostic and prognostic value of the data increases.
References
D.B. Skvortcov. Stabilographic Research, 176 pp. Maska, Moscow, 2010. https://biomera.ru/upload/biblio/skvortcov_preprint.pdf
V.A. Antonets, et al. Accelerometer stabilography, orthopedics, traumatology and prosthetics, No. 1, p. 55-56, Medicine, Kharkov, 1991.
Example 2. Heart rate analysis
Under normal conditions, the heart contracts cyclically. The duration of the cycle is usually measured by the time interval between the beginning of two consecutive contractions. The beginning of the contraction is the appearance in the electrocardiogram of R tooth. Therefore, the duration of the heart cycle is frequently called the R-R interval.
The duration of the R-R interval usually varies from contraction to contraction because the heart is controlled by its internal regulatory systems and the central nervous system and biochemical agents, the concentration of which in the systems of the human body can vary depending on different circumstances, loads and time.
Thus, the changes in the duration of cardiac contractile cycles, i.e. in the rate of heart contractions carry information on the regulation not only of the heart itself, but also of the organism as a whole. Therefore, the analysis of the heart rate allows a fairly detailed assessment not only of the reserves of the cardiovascular system, but also of the state of the regulatory systems of the body as a whole, in particular, enables diagnosis of stress manifestations.
The rhythmogram, i.e. the duration of the heart cycle as a function of time is usually analysed. Recently, pulsometry data has also been used to build rhythmograms. The rhythm is analyzed in more detail using scatterograms, which are a set of points on the plane with (Rj,Rj+1) coordinates, where Rj are the durations of the R-R intervals.
Rhythmograms
Scatterograms
There are many gadgets for controlling heart rate, for example, Zephyr heart rate monitor paired with smartphones. Smart watches and fitness bracelets, embracing over 150 million people worldwide, are also used for this purpose.
References
R.M. Baevskij, G.G. Ivanov, et al. Analysis of heart rate variability with the use of different electrocardiographic systems (part 1). Vestnik Aritmologii http://www.vestar.ru/article.jsp?id=1267
Ziep B., Е.О. Taratukhin. HEART RATE VARIABILITY ASSESSMENT AND ITS POTENTIAL. Russian Journal of Cardiology https://russjcardiol.elpub.ru/jour/article/view/1177/906
Example 3. Monitoring stroke
About half a million cases of stroke are registered annually in Russia. Mortality in the acute period of cerebrovascular accident reaches 35% and increases by 12-15% by the end of the first year; 44% of patients die within 5 years after the stroke. Therefore, monitoring the risks of stroke is of great importance.
The signs of stroke are well known:
Signs of stroke
An individual test for stroke signs can be designed on the basis of standard applications such as Selfie and Siri.
1. When taking photos in Selfie mode, parameters of user's hand movement can be recorded during frame formation. If the movements differ from the ones typical for the person, this is an unfavorable sign.
2. The face recognition program can determine whether the face looks as usual. If not, this is also an unfavorable sign.
3. The speech recognition programs such as Siri will notice speech distortions during speech test . E.g., the common test phrase in Russian practice is “The thirty-third artillery brigade”.
If there is a combination of unfavorable signs, the mobile application can inform about this the user or his/her relatives, the attending doctor, and others.
Example 4. Sensorimotor tests
Sensorimotor tests are designed to assess person's response to external stimuli. The most often evaluated parameters are the speed of response to the appearance of a significant stimulus and the number of errors. These data can be synchronized with data on the functioning of the human physiological (vegetative) systems – cardiovascular, respiratory, etc., which are recorded by means of gadgets (see Example 1). That is how the “physiological cost” of the test effort can be determined.
We have also developed a new method of psychophysiological testing based on measuring errors made by the user during manual contactless control of virtual audiovisual objects. An example of controlling sound (a) and visual (b) stimuli using a laboratory setup is presented in the figure below. Errors are measured by a special non-contact ultrasonic movement meter.
In the developed example of a mobile version of such a setup, control errors are measured by sensors available in all modern smartphones. Examples of studying color and parallelism perception are given in the figure. In the first case, by changing the position of the smartphone the subject under study must achieve maximum match of the colors of the model (bottom) and of the controlled images, and in the second case, of the angles of inclination of the axis of the model (bottom) and of the controlled figures. The capabilities of modern smartphones allow targeted development of tests for assessing the quality of perception of any desired stimulus.
Tests of this type make it possible to evaluate the psychophysiological potential of an unlimited number of users, i.e. conduct epidemiological studies taking into account regional, age, gender, social and other stratification of populations .
References
V.A. Antonets, et al. Quantification of human perception using the motor human-computer interface Hand Tracker, Modern Technologies in Medicine, vol. 11, No. 1, 2019, p. 141-149.
V.A. Antonets, М.A. Antonets, V.Yu. Pogodin, A.Yu. Kryukov, N.E. Ilyin. Laboratory and mobile versions of the hand-tracking method for studying human primary cognitive functions by their motor manifestations, Procedures and Methods of Experimental Psychological Research, Ed. by V.A. Barabanshchikov, Institute of Psychology RAS, Moscow, 2016, p. 639- 645.
М.A. Antonets. Estimation of samples relevance by their histograms. https://arxiv.org/abs/1701.08383v3