Citation
Exploring intertial measurement unit based position estimation method and deep learning predictive model for virtual reality stroke rehabilitation

Material Information

Title:
Exploring intertial measurement unit based position estimation method and deep learning predictive model for virtual reality stroke rehabilitation
Creator:
Sable, Anurag
Place of Publication:
Denver, CO
Publisher:
University of Colorado Denver
Publication Date:
Language:
English

Thesis/Dissertation Information

Degree:
Master's ( Master of science)
Degree Grantor:
University of Colorado Denver
Degree Divisions:
Department of Computer Science and Engineering, CU Denver
Degree Disciplines:
Computer science
Committee Chair:
Lin, Feng
Committee Members:
Miswas, Ashis Kumer

Record Information

Source Institution:
University of Colorado Denver
Holding Location:
Auraria Library
Rights Management:
Copyright Anurag Sanjay Sable. Permission granted to University of Colorado Denver to digitize and display this item for non-profit research and educational purposes. Any reuse of this item in excess of fair use or other copyright exemptions requires permission of the copyright holder.

Downloads

This item has the following downloads:


Full Text
EXPLORING INERTIAL MEASUREMENT UNIT BASED POSITION
ESTIMATION METHOD AND DEEP LEARNING PREDICTIVE MODEL FOR VIRTUAL REALITY STROKE REHABILITATION SYSTEM
By
Anurag Sable
Savitribai Phule Pune University, 2017 Bachelor of Engineering in Computer Science
A thesis submitted to the Faculty the Graduate School of the University of Colorado in partial fulfillment of the requirements for the degree of Master of Science
Computer Science


2019
©200x by All Rights Reserved
[This page should be included ONLY in theses that are copyrighted. If you include this page, change the page numbering for following pages to vi instead of v using the Insert/Page Numbers command.]


May 2019
University of Colorado Denver


Anurag Sanjay Sable
Exploring Inertial Measurement Unit based Position Estimation Method and Deep Learning Predictive Model for Virtual Reality Stroke Rehabilitation System
Thesis directed by Assistant Professor Feng Lin.
ABSTRACT
Stroke is the third largest cause of fatalities in the United States, also causing long-term mobility disabilities in many patients. Every year, around 795,000 people suffer from stroke, out of which 185,000 are recurrent attacks. Hence, as the numbers indicate, quality post-stroke treatment and rehabilitation are necessary to avoid recurrent attacks and permanent disabilities. In this thesis, we contributed towards developing and implementing IMU based Position Estimation algorithm and Deep Learning based Predictor for VRInsole, an intuitive, smart and innovative method for home-based poststroke rehabilitation, targeted at lower extremity mobility training. For this, we employ three key ideas: (i) Smartlnsole, which is a smart wearable sensor for collecting motion data including linear acceleration, angular rate and magnetic force which are inherently time-series, (ii) Position Estimation algorithm which processes the collected data to estimate user activity in real world and (iii) Deep Learning Predictor, in order to make the system more intuitive to user input by detecting what activity is being performed. Smartlnsole is an unobtrusive wearable sensor, which contains an onboard nine-axis Inertial Measurement Unit (IMU), which enables us to gather fine-grained information about user movement. For Deep Learning method, we designed a Long Short-Term Memory (LSTM) architecture, which is a class of Artificial Neural Networks, highly suitable for predicting time-series data. We demonstrate the performance of the proposed model by comparing its performance with state-of-the-art methods. Moreover, we developed an algorithm to estimate user activities based only on the displacement and orientation of their foot. We compared the results of this algorithm against real-world measurements to demonstrate higher estimation accuracy.
The form and content of this abstract are approved. I recommend its publication.
Approved: Feng Lin
IV


V


This thesis is approved for recommendation to the Graduate Committee.
Thesis Advisor: Feng Lin
MS Thesis Committee: Ashis Kumer Biswas Min-Hyung Choi
vi


DEDICATION
I dedicate this thesis to my parents, for their unconditional support and the faith they have shown in me. I would also like to thank all of my friends, who have helped me through difficult times. And last, but not the least, my eternal pillar of support Nikita.
Vll


TABLE OF CONTENTS
1. Introduction.................................................................1
1.1 Problem...................................................................1
1.2 Objective.................................................................2
1.3 Contribution..............................................................2
1.4 Organization of this Thesis...............................................3
1.5 Achievements..............................................................4
2. Background...................................................................5
2.1 Key Concepts..............................................................5
2.1.1 Inertial Measurement Unit.............................................5
2.1.2 Virtual Reality.......................................................6
2.1.3 Recurrent Neural Networks.............................................7
2.2 Literature Review.........................................................7
2.2.1 Virtual Reality Based Rehabilitation..................................8
2.2.2 Exergames.............................................................8
2.2.3 Home-based Rehabilitation Training....................................9
2.2.4 State-of-the-art Methods for Position Estimation......................9
2.2.5 Machine Learning based Methods for Activity Detection.................9
3. Proposed Architecture........................................................10
3.1 System Overview..........................................................10
3.2 Hardware Specification...................................................11
3.3 Position Estimation from Sensors.........................................12
vii


3.4 Deep Learning Predictor
14
3.4.1 LSTM Model Architecture.................................................15
3.4.2 Deep Learning Program...................................................17
3.5 Baseline Classifiers for Comparison.........................................20
3.5.1 K Nearest Neighbors.....................................................20
3.5.2 Random Forest Classifier................................................20
3.5.3 Support Vector Machines.................................................21
3.5.4 Naive Bayes Classifier..................................................21
3.6 Virtual Reality Design.....................................................22
3.7 Rehabilitation Task Design.................................................23
3.8 User Feedback Design.......................................................23
4. Results and Discussion.........................................................25
4.1 Experimental Setup..........................................................25
4.2 Evaluation.................................................................25
4.2.1 Position Estimation Results.............................................25
4.2.2 Orientation Estimation Results.........................................27
4.2.3 Deep Learning Predictor Results........................................28
4.2.4 Comparison with traditional Classifiers................................30
5. Conclusion.....................................................................31
5.1 Summary of Contributions....................................................31
5.2 Limitations.................................................................32
5.3 Future Work.................................................................33
viii
References
34


IX


LIST OF FIGURES
Figure 1: Oculus Rift HMD............................................................6
Figure 2: A vanilla Recurrent Neural Network Architecture............................7
Figure 3: Proposed System architecture..............................................10
Figure 4: Smartlnsole...............................................................11
Figure 6: Structure of an LSTM cell.................................................15
Figure 7: Constructing the LSTM network.............................................16
Figure 8: Pseudocode for the Deep Learning-based solution...........................19
Figure 5: Activities rendered in Virtual Reality platform...........................22
Figure 9: Position estimation plots.................................................26
Figure 10: Orientation estimation results...........................................27
Figure 11: Confusion matrix for the proposed model on the test dataset..............29
x


1. INTRODUCTION
1.1 Problem
Stroke is one of the major causes of fatalities around the world, with over 140,000 fatalities in the United States every year. Every year around 795,000 people suffer from stroke, out of which 185,000 are recurrent attacks [2], Recurrent attacks are most likely to cause fatalities and permanent disability and a leading cause for recurrent attacks is the lack of post-stroke treatment and rehabilitation. Stroke attacks cause a major loss in the mobility of the patient, which is difficult to regain without clinical treatment and can lead to a permanent state of partial or complete loss mobility.
Even though the importance of post-stroke rehabilitation is certainly known, statistics show very low numbers for rehabilitation treatment follow-up. The main causes for this phenomenon are attributed to the low accessibility, high cost and monotonous nature of the process. The rehabilitation process requires the patient to perform regular exercise routines at home with or without assistance depending upon the level of mobility. Research has shown that the home-based treatment is ineffective in a lot of patients because of lack of motivation and encouragement for the patient as well as the caregiver[3]. This also has a negative effect on the mental health of the patient, further reducing the effectiveness of the treatment.
To tackle the aforementioned issues many improvements have been introduced to the rehabilitation process, which includes audio-visual aids and reward-based routines. Some have tried using modem technology like Virtual Reality systems[4], and the results are definitely positive.
1


1.2 Objective
In this thesis, we include and introduce improvements to VRInsole[6], which is a smart wearable sensor, Virtual Reality assisted and Deep Learning based system for lower extremity mobility training. This system aims to solve issues with existing systems and introduce improvements to make home-based rehabilitation training intuitive, immersive and smart. In this thesis, we introduce, discuss and evaluate three important components of VRInsole[6], which is a smart wearable sensor, Virtual Reality assisted system for lower extremity mobility training, namely,
I. Using SmartInsole[7] to collect user motion data which will be processed and used as an input to the system.
II. Develop and evaluate performance of Position Estimation algorithm which will try to calculate displacement and orientation of user motion, while minimizing IMU drift velocity issue.
III. Develop and evaluate performance of Deep Learning Predictor which is used to predict rehabilitation task being performed, acting as a complementary system to the Position Estimation Algorithm.
1.3 Contribution
Our work is comprised of three main contributions, which care enlisted as follows.
(1) Using SmartInsole[7], a smart wearable device developed in-house with an onboard 9-axis IMU sensor and Bluetooth to collect motion data for user activity. This data is preprocessed to remove noise using bandpass filter.
2


(2) The filtered data is then again processed to convert the inertial frame of reference from body coordinates and adjusted for sensitivity of the sensors. Once this is done, we use the Position Estimation algorithm, which will estimate displacement and orientation of user’s activity, with minimum drift velocity issue. This will then be rendered in Virtual Reality assisted environment designed in Unity3D[8],
(3) The filtered data also serves as an input to the Deep Learning Predictor, which acts as a supplementary system to the Position Estimation algorithm. The Deep Learning Predictor uses a stacked LSTM approach to make prediction of user activity with good accuracy.
The Virtual Reality environment serves as a motivation for the user by rendering the activities in a game-like environment to make the experience motivational and engaging. The data acquired from Smartlnsole gives fine-grained information about user motion, which helps in accurate motion estimation and provides medical quality representation of the exercise routine. This allows us to provide an opportunity to analyze and produce quality medical feedback.
1.4 Organization of this Thesis
Chapter 1 is the introduction part. This chapter aims at highlighting how stroke is a major health issue and the importance of post-stroke rehabilitation process. Then we discuss the problems with currently used rehabilitation process and why VRInsole was developed. After that, we clearly outline our problem statement, which is to develop and evaluate Position Estimation and Deep Learning Predictor for the VRInsole system.
3


Chapter 2 talks about some preliminary concepts which required to understand the technical aspects of our works and then we analyze related works. Later we summarize the previously followed approaches and outline areas of improvement, which our proposed contributions might eliminate.
Chapter 3 gives a clear and in-depth explanation of our technical contribution, algorithms, and methods. First, we talk about the software and hardware used in our system, followed by details on the position estimation algorithm and proposed deep learning model. This chapter also provides a pseudocode representation of our Deep Learning model. Lastly, we discuss some baseline classifiers, which were used for comparison with our model and design of the feedback system. This section also mentions how the rehabilitation tasks for the experiment were chosen.
Chapter 4 starts with a description of our experimental setup, and then moves on to discuss the results of position estimation and Deep Learning model. Then we compare the results of our Deep Learning Model with selected baseline classifiers and talk about the statistical significance of our results.
Chapter 5 starts with a short discussion summarizing our contributions, moving on to the impact of our work, some limitations and concludes with some possibilities for future improvements.
1.5 Achievements
Our preliminary work on this project, titled “VRInsole: An Unobtrusive and Immersive Mobility Training System for Stroke Rehabilitation” [6] was published at 2018 IEEE 15th International Conference on Wearable and Implantable Body Sensor Networks (BSN), held in Las Vegas, Nevada between 4-7th of March 2018.
4


2. BACKGROUND
2.1 Key Concepts
In this section, we will discuss a few concepts, which will help us to understand the important concepts and issues addressed later on. This includes three main concepts, namely, Inertial Measurement Unit (IMU), Virtual Reality (VR), and Recurrent Neural Networks (RNN).
2.1.1 Inertial Measurement Unit
Inertial Measurement Unit (IMU) is an electromechanical device which is capable of measuring linear acceleration, angular rate and magnetic field, in the frame of reference of the body (or the device) on which the IMU has been mounted. This measurement is done by using accelerometer for linear acceleration, gyroscope for angular rate and magnetometer for magnetic field. The main purpose of using IMUs in any given system is for maneuvering and navigation. Each of the three sensors in an IMU grants it with three degrees of freedom (DOF) in 3D space. IMUs are used in combination with Global Positioning System (GPS), as a complementary system, like in the case where GPS signals are weak or unavailable.
The main issue with using only IMU for navigation purposes is accumulated measurement error, called drifting error. This error is usually corrected using GPS, which is an application of Kalman Fusion Filter.
5


2.1.2 Virtual Reality
Virtual Reality (VR) is an interactive computer-generated environment, achieved by providing the user with auxiliary devices, which can manipulate visual, auditory and tactile channels. This provides the user with a simulated virtual environment, which can be similar to a real-world scenario or completely fantastical, to achieve a high degree of immersion. There are various devices, which can be used to implement VR systems, but the most popular choice is using Head Mounted Display (HMD). These are compact devices, worn by the user as headgear, with integrated screens, speaker/headphones and some amount of processing power. Most widely used HMDs are Oculus Rift [9], HTC Vive and Samsung Gear VR (Figure 1). These devices also come with handheld controllers for interaction with objects in virtual reality and are capable of providing haptic feedback. The user’s movement is tracked and usually confined in the physical space using a camera and IMU based motion tracking.
There are various applications of VR systems, including, but not limited to, entertainment, media, education, simulation, scientific visualization, and healthcare.
Figure 1: Oculus Rift HMD
6


2.1.3 Recurrent Neural Networks
Recurrent neural networks (RNN) are a special class of artificial neural networks (ANN), which differ from traditional ANN by maintaining temporal memory of sequences, using directed connections between its nodes (neurons) along sequences (Figure 2). This allows RNNs to treat the input data as an ordered sequence of arbitrary length, which makes them very suitable for applications like time series prediction, handwriting recognition, speech recognition, and synthesis and language modeling.
2.2 Literature Review
During the time of writing this work, we reviewed some of the previous achievements by academia, similar to our work. We discuss various approaches to Virtual Reality based Rehabilitation systems, upper extremity mobility training systems using proprietary hardware, Augmented reality games for general exercise motivation and smartphone IMU based user activity detection.
7


2.2.1 Virtual Reality Based Rehabilitation
There are several systems, which deal with upper extremity rehabilitation using virtual reality. Most of these systems make use of off-the-shelf hardware like Nintendo Wii U VR[10] and Microsoft Kinect. The main reason to choose these systems is that they have their own camera-based user motion tracking, which makes it easier to generate corresponding gestures and mobility in the VR scene. However, this comes at a cost. Due to camera-based tracking, the user is always confined to the tracking area, restricting mobility. In addition, these systems are proprietary and closed source, which limits the customization of the VR application and the amount of raw data from the sensors.
Jack et al.[l 1] made use of Cyber-Glove, which serves as a motion input sensor to the VR environment. Frisoli et al.[12] present an upper extremity force-feedback exoskeleton for robot-assisted VR rehabilitation system. These methods make use of proprietary, custom-made sensors, which make use of multiple actuators, and sensors to get user motion data, which are not readily available and hard to replicate. In addition, no such system for lower extremity was found at the time of writing this work.
2.2.2 Exergames
A lot of work has been done in game-based exercise rehabilitation systems, such as SoccAR[13], This system uses Augmented Reality (AR) based approach to engage the user in a fast-paced motivational environment to perform exercise in a game like scenario. These systems are targeted at a general audience and do not tend to focus on specific medical benefits, hence, not suitable for health intensive training.
8


2.2.3 Home-based Rehabilitation Training
Clinical rehabilitation generally tends to be inaccessible, expensive and tedious for most of the patients, hence they tend to home-based methods. However, studies [14] have shown that such rehabilitation routines become monotonous overtime and sometimes have a negative effect on the mental health of both, the patient and the caregiver. In addition, there is no real guideline and monitoring for the patient’s progress, making it difficult to assess the quality and improvement of mobility.
2.2.4 State-of-the-art Methods for Position Estimation
The accumulating error in IMU measurements is very well known, as it is an inherent hardware limitation. Hence, there have been various attempts at compensating for this error. Many have used Kalman Fusion Filter [15], which uses GPS signals to cancel out the accumulating error. Other methods [16] use a much simpler method, where a step is detected by observing patterns in the accelerometer data, and calculate the distance traveled by multiplying the number of steps with average stride length. Such methods lose a large chunk of input data due to estimation.
2.2.5 Machine Learning based Methods for Activity Detection
There have been several attempts at employing ML based methods to predict human activity. Anguita et al.[17] used Support Vector Machines (SVM) based approach, using a smartphone IMU to get user activity data. They implement the SVM model over a moving, overlapped window of data, with a 17-feature vector input. Even though this approach performs better than existing static window methods, it suffers from lower accuracy rates due to losing temporal locality of time-series data.
9


3. PROPOSED ARCHITECTURE
3.1 System Overview
The system can be visualized as described in Figure 3. The first part of the system is responsible for collecting data from the sensor and processing it using our algorithm to obtain position and orientation of user activity. Next, we use the collected data to obtain a prediction of the user activity using a deep learning model, generated from the experiments performed during the training phase of the study. Lastly, the position estimation and prediction generated from the deep learning model will be used as an input to the Virtual Reality scene, where a simulation of the activity is presented to the user, with added animations to provide motivational feedback to the user. While the system does consist of several subsystems, the user will always interact with Virtual Reality scene, which will be the only interface provided to the end user.
Motion Estimation
V V
Figure 3: Proposed System architecture
10


3.2 Hardware Specification
For the initial experiments, Smartlnsole sensor was used, which was developed in collaboration with the team. Smartlnsole is a compact, wearable device and consists of a nine DOF Inertial Measurement Unit (IMU), a microcontroller unit, Bluetooth Low Energy (BLE) module for communication and a small Lithium (Li) battery pack for portability (Figure 4).
The IMU unit is an off the shelf Bosch BMX055 sensor [18], which has a 12-bit accelerometer, 16-bit gyroscope, and atri-axial magnetometer. This array of sensors helps us to gather linear acceleration, angular rate and magnetic deviation of user activity.
Figure 4: Smartlnsole
11


3.3 Position Estimation from Sensors
The process of converting the data collected from the sensors involves many steps, which can be summarized as follows.
The collected data from the sensors include accelerometer and gyroscope, which give the linear acceleration and angular rate of the user’s foot. The linear acceleration will be used to obtain the velocity and position, while the angular rate will give the rotation.
First, we obtain position from the linear acceleration, which technically should be calculated by integrating the acceleration twice over time, to get velocity and then position. However, this method does not work for standard accelerometers. The main reason behind this is the fact that the sensor is not always in motion, and the static periods induce “drift velocity”, which leads to massive errors in position estimation. Hence, we use a special algorithm to avoid that error.
i. All of the accelerometer data is studied for periods of actual motion, by setting a minimum threshold for acceleration (bandpass filter), a small amount of acceleration is observed even in stationary period due to marginal sensor error.
ii. Once such periods are identified, we mark the timestamps of such measurements for reference during velocity calculations.
iii. Then, we integrate acceleration over time to obtain velocity (refer to equation 1). Once velocity for all timestamps is identified, we reset the velocity to zero for the previously identified stationary timestamps.
iv. This rectified velocity is then integrated over time to obtain the final position.
12


Once we obtain velocity and position, we calculate the corresponding rotation. For this, we obtain the angular rotation along each of the axes (x, y, z) by integrating angular rate over time. This gives us the angular rate of change, which, after rectifying for drift velocity, gives the actual Quaternions and Euler Angles (refer equation 2). We chose the Quaternion representation over Euler Angles in order to eliminate traditional calculations like Gimbal Lock and Ambiguity. In addition, Quaternions are easier to compute, as evident from the following equations.
s* = Jf(a)dt ................(1)
Where, a is acceleration vector, v is velocity vector and ~s is displacement.
"0" 'atan2(2(g0 6 — asin(2(q0q2 - q^qi))
_atan2(2((jfo Where, quaternion *1 — t ?o Euler angles,

13


3.4 Deep Learning Predictor
For making our application more intuitive, we introduce a deep learning model to predict the user’s activity, to provide a more flexible environment and exercise regime. Deep Learning methods have been widely used for such purposes and really help in making the system intelligent and adaptive. Deep learning concepts try to emulate human brain function by mimicking small processing units, called “neurons”, and make use of several such units to create an artificial neural network (ANN). Each neuron is used to “sense” some part of the input and is assigned a “weight” through training, to ascertain the importance of that specific input. Thus, through multiple rounds, or “epochs”, the model tries to guess the best set of weights and use that to make predictions.
Here, we make use of a special kind of Deep Learning method, called Long Short-Term Memory or “LSTM” (Figure 6). LSTMs are special kind of ANN, which are very useful in classifying time-series data, like in our case. LSTMs consists of several units, each having its own cell (memory), input gate, output gate and forget gate. The cell is used to store the current state of the unit, which can be read by the next unit in the chain as a reference point in the sequence. The input gate gets the output from the previous cell and the output gates serve as an input to the next cell in the chain and decides if the output of the current cell should be visible. The forget gate is used to reset the memory of the cell and helps if the cell state needs to be reset. All of the input, output and forget gates use a sigmoid activation function, which produces smooth curves and makes the model differentiable.
14


ht
Legend:
Layer Pointwize op Copy
-i*
Figure 5: Structure of an LSTM cell
3.4.1 LSTM Model Architecture
After experimenting with various different architectures, the final model was selected for the best performance observed. The model consists of two fully connected layers, with 64 hidden units each, a rectified linear unit layer (‘relu’), two LSTM layers stacked on top of each other and then, a softmax layer. Usage of a stacked LSTM structure was found produce better results, as compared to a traditional LSTM or RNN, by virtue of having a longer “memory” of the sequences.
The code segment used to construct the LSTM network is shown in Figure 7. In lines 87-94, we constructed the 6x64 dimensional weight coefficients of the 64 LSTM
15


cells for the two LSTM layers, as well as the 64 bias coefficients are defined. Lines 102-105 defines the structure of the stacked LSTM layers. The program is written using Python 3.6 and utilizing TensorFlow 1.13.1 framework.
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99 100 101 102
103
104
105
106
107
108
109
110
N_CLASSES = 3
N_HIDDEN_UNITS =64
N FEATURES = 6
def create_LSTM_model(inputs): W = {
'hidden': 'output':
}
biases = {
'hidden': 'output':
tf.Variable(tf.random_normal([N_FEATURES, N_HIDDEN_UNITS])) , tf.Variable(tf.random_normal([N_HIDDEN_UNITS, N_CLASSES]))
tf.Variable(tf.random_normal([N_HIDDEN_UNITS], mean=l.0)), tf.Variable(tf.random_normal([N_CLASSES]))
}
X = tf.transpose(inputs, [1, 0, 2])
X = tf.reshape(X, [-1, N_FEATURES])
hidden = tf.nn.relu(tf.matmul(X, W['hidden']) + biases['hidden1]) hidden = tf.split(hidden, N_TIME_STEPS, 0)
# Stack 2 LSTM layers
lstm_layers = [tf.contrib.rnn.BasicLSTMCell(N_HIDDEN_UNITS, forget_bias=l.0) lstm_layers = tf.contrib.rnn.MultiRNNCell(lstm_layers)
outputs, _ = tf.contrib.rnn.static_rnn(lstm_layers, hidden, dtype=tf.float32)
# Get output for the last time step lstm_last_output = outputs[-1]
return tf.matmul(lstm last output, W['output']) + biases['output ]
for
in range(2)]
Figure 6: Constructing the LSTM network
16


3.4.2 Deep Learning Program
The architecture of the developed program can be explained using the pseudocode in Figure 8. The module “dataPreprocess” (Lines 1-4) does the preprocessing of the dataset and set the optimum values for the hyperparameters required by the LSTM network. The function “plotActivity” is used to plot the graphs to analyze and understand the data to find the optimum hyperparameters (lines 5-7). The “createSequence” function formats the time-series dataset into sequences of length specified by timesteps (lines 8-14). The “splitData” function splits the dataset randomly into training and test sets. The split size was set to 80% to the training and 20% to the test set (lines 15-16). In line 17, the placeholders for Tensorflow library are initialized which are used later by the subsequent modules. In lines 18-25 defines the function “learnModel” that learns the optimum weight coefficients for the proposed LSTM network. One “learnModel” returns, the function “generatePredictions” produces the predictions for the test set (line 26-28). The “resultMetrics” function in lines 29-31 reports the performance metrics for the generated predictions.
17


1 dataPrepocess (dataset_file):
2 OPEN dataset_file
3 CALL plotActivity graph_sorted_by_activity_labels
4 SET hyperparameters
5 plotActivity (activityjabel):
6 FOR record WHERE label == activity label:
7 RETURN record
8 createSequences ():
9 SET length_of_step_sequence
10 SET number_of_features
11 FOR every record in dataset_file:
12 Create sequence of length length_of_step_sequence,
13 WHERE features == number of features
14 RETURN sequences, labels
15 splitData (sequences, labels):
16 RETURN sequences_train, labels_train, sequences_test, labels_test
17 SET input_tensor_dimensions, output_tensor_dimensions
18


18 learningModel (sequences_train, labels_train):
19 FOR number_of_epochs:
20
fully-connected layerl
21
fully-connected Iayer2
22
ReLU layer
23
LSTM layerl
24
LSTM Iayer2
25
RETURN best_accuracy, loss, best_model
26 generatePredictions (best_model, sequences_test):
27 predictions = best_model.fit(sequences_test)
28 RETURN predictions
29 resultMetrics (predictions, labels_test):
30 generate accuracy, precision, recall, f1-score, confusionMatrix
31 PLOT metrics
Figure 7: Pseudocode for the Deep Learning-based solution
19


3.5 Baseline Classifiers for Comparison
To explain how we arrived at the choice of our chosen deep learning model, we explored many commonly used machine learning classifiers as well. Upon closely studying the nature of our dataset, it was evident that in order to learn time series data of repeating sequences, which are similar in pattern but not always the same, a recurrent neural network kind of approach, was best suited. Other classifiers tested on our dataset, are listed as follows.
3.5.1 K Nearest Neighbors
K Nearest Neighbors or KNN is a supervised learning algorithm, i.e., it requires a training label to learn and classify. It belongs to the “lazy” type of classifiers, as it does not necessarily require a training step. In addition, it does not require any other parameters for learning, except for “K”. This parameter is used to decide the target class for any given sample by matching its features with “K” number of closest neighbors.
3.5.2 Random Forest Classifier
Random Forest belongs to the class of supervised learning algorithms and includes creation of a “forest” of many decision trees, based on randomly selected features from the dataset. Once a forest of “n” decision trees is created, the classifier is ready to make predictions. The test sample is evaluated against each of the decision trees, and a target label is chosen. The final predicted label is decided by a majority vote from the result of all the decision trees in the forest. This helps the classifier to overcome the issue of overfitting and helps to identify the most important features.
20


3.5.3 Support Vector Machines
Support Vector Machine or SVM is a supervised learning algorithm. SVM uses a very simple approach for classification; it does so by trying to find a clear separation (support vector) between the classes represented in a given space. This support vector can be linear or hyperplane, depending upon the distribution of the points in a given dataset. This is done using various “kernels”, which are mathematical functions describing the support vector.
3.5.4 Naive Bayes Classifier
Naive Bayes classifier also belongs to the class of supervised learning algorithms. It makes use of Bayes theorem to calculate probabilities of each feature and then the label/target with the highest probability is selected as the prediction. This algorithm is called Naive as it does not consider the order and correlation between the features, and hence, loses a large chunk of information like repeating patterns and interdependent features.
21


3.6 Virtual Reality Design
The VR system acts as a viewport for the user, in a game like situation setting, which acts as both, a visual replay system and motivation to perform better. Every rehabilitation task is simulated in a game like immersive scenario, which can be further customized according to users need, with various characters and environments. This has been achieved using the Unity 3D, which is a cross-platform game development environment. This adds the possibility of making our system available on many VR compatible systems, including mobile devices, making the system portable.
Each of the rehabilitation tasks is set up in an interesting game-like situation, making the interaction enjoyable. For the kicking task, the user character is placed in a football field. For the leg lifting task, a modular indoor environment is presented, which reacts correspondingly to the level of user activity. For obstacle walking task, a series of identically sized obstacle is placed in an outdoor environment, which can be changed for adding multiple difficulty levels of activity. Figure 5 demonstrates the three activities rendered in a virtual reality environment.
Figure 8: Activities rendered in Virtual Reality platform
22


3.7 Rehabilitation Task Design
For our system consideration, we focus on lower extremity rehabilitation tasks for post-stroke patients, who have enough mobility, i.e., they have reasonable mobility without the assistance of any external devices, to narrow down our research area. The selected tasks focus on lower mobility training with focus areas of functional task, aerobic task and challenging walking task[19]. These tasks are selected in such a way that they can be comfortably performed in any indoor/outdoor scenario with ease and provide medical benefit to the patient by providing training for basic mobility to accelerate post-stroke recovery. The functional task selected is repetitive leg lifting, which involves lifting leg from a stationary position while taking support from a table like support to provide stability. This task provides core strength to the patient’s leg, which helps in basic motion. For further cardiovascular training, we chose kicking activity as aerobic task, which involves kicking motion from a stationary position[20]. Lastly, we chose obstacle walking as the challenging walking task, which involves stalking with stepping over similar sized obstacles, to provide complete mobility practice[13]. These three tasks provide a complete exercise regime to provide basic mobility to the patient post-stroke.
3.8 User Feedback Design
The system provides the user with various feedback options during and after rehabilitation tasks, which help in reviewing, tracking and improving the quality of poststroke recovery. A graphical representation of user activity is available after every session, which can be used to qualitatively analyze user motion, indicative of recovery progress.
23


This representation includes both, displacement and orientation graphs, showing user limb movement.
This information is also used by the VR system, to provide the user with a replay of his/her activity, which helps in easier understanding of mobility level and also provides motivation to the user for improvement.
24


4. RESULTS AND DISCUSSION
4.1 Experimental Setup
For testing our system and its effectiveness, we conducted some experiments, which try to simulate the intended rehabilitation purpose that this system is intended for. Data was collected from ten male and three female subjects, all aged between 22 and 30. All subjects were made to wear the Smartlnsole sensor, with their choice of footwear and performed each of the three rehabilitation tasks (kicking, leg lifting and obstacle walking), in random order over multiple data collection sessions. All subjects performed multiple repetitions of each task for a fixed time session of three minutes each, to ensure ample data collection.
4.2 Evaluation
4.2.1 Position Estimation Results
The gait estimation program can take motion data from every session and produce displacement and orientation graphs for corresponding linear and angular rate as shown in the figure below. For displacement analysis, we can observe multiple graphs for acceleration with stationary period markings, linear velocity, and displacement for each of the axes. This data can be visualized as a combined graph, which depicts the actual gait in the corresponding real-world scale (Figure 9). Errors in this gait estimation are minimal, within the bounds of our test conditions, due to inherent measurement issues with IMU devices.
25


This experiment was performed to evaluate the accuracy of our estimation algorithm, which aims to minimize the error from drift velocity. The results in Figure 9 are the output of the displacement estimation. To evaluate accuracy of this experiment, the estimated displacement was compared with actual distance travelled. This method for evaluation was chosen as there is no way to evaluate any empirical real-world quantity like distance travelled without proper equipment.
26


4.2.2 Orientation Estimation Results
For qualitative analysis of orientation calculations, we simulate accurate real-world rotation using a controlled servo and measure the corresponding orientation using our own algorithm. The orientation is compared to real-world vs device calculations to measure the accuracy of our algorithm. The results can be summarized in Figure 10.
Similar to previous experiment, measuring exact rotation of a body in a real world is not possible, as it is an empirical quantity. Hence, the simulation with calculated rotation of servo motor was chosen as the ground truth value. The orientation results from the estimation algorithm was compared with the actual servo rotation to evaluate accuracy, depicted in Figure 10.
27


4.2.3 Deep Learning Predictor Results
To evaluate the accuracy of the Deep Learning Predictor, the LSTM model was tested on 5 discrete, randomly selected chunks of the test dataset collected during previous experiments. The dataset was split into 80-20% sets, for training and testing, respectively. This was done to make our model robust within the scope of our research constraints.
The results for our LSTM model have been analyzed for standard metrics like accuracy, precision, recall, and F-l score. These metrics provide us with a good idea about the quality of the result and performance of the model. The results can be summarized as follows.
After running the prediction model for a sufficient number of times on randomly split dataset, we observe that the mean accuracy is 97.615 %, with a standard deviation of 0.66. Table 1 demonstrates the performance measures of the proposed model on the test dataset over a series of 5 runs of the training the model.
Table 1 : Performance of the proposed LSTM model
Metrics Testl Test2 Test3 Test4 Test5 Mean
Accuracy 98.1266 96.3934 97.8361 97.491 98.2295 97.6153 ±0.66
Precision 98.12 96.39 97.83 97.04 98.22 97.52 ±0.7
Recall 98.12 96.39 97.83 97.04 98.22 97.52 ±0.7
FI Score 98.12 96.39 97.83 97.04 98.22 97.52 ±0.7
28


Figure 11 shows the confusion matrix to provide a better view of the performance of the classification proposed classification system. It clearly shows the model is able to distinguish the three activities with high precision, recall and accuracy.
to
Q)
.a
to
— ^ 5 2
Confusion matrix
394 1 5
8 425 1
10 2 679
-600
•450
-300
150
Walk
Kick
Predicted label
Lift
Figure 11: Confusion matrix for the proposed model on the test dataset.
29


4.2.4 Comparison with traditional Classifiers
To ascertain the accuracy performance of our Deep Learning Predictor, we compared our results, four other commonly used methods used for similar purposes. These classifiers belong to the class of Machine Learning methods which treat each data point as a discrete event, as discussed earlier in Section 5. Hence, we expected our proposed method to perform better, which was observed in the results.
The results for other classifiers can be summarized below (Table 2). After analyzing the results, we can clearly see that KNN classifier performs better than the rest, but well below the performance of our classifier. The reasons for this observation will be discussed in the further sections.
Table 2: Performance comparison of the proposed system with baseline classifiers.
Classifier Accuracy Mean ± Deviation
Test 1 Test 2 Test 3
KNN 89.049 89.049 89.049 89.049 ± 0.00
Random Forest 88.839 88.875 88.856 88.856 ± 0.018
SVM 66.328 66.328 66.328 66.238 ±0.00
Naive Bayes 52.718 52.728 52.718 52.721 ± 0.005
Proposed system 98.13 96.39 97.83 97.44 ± 0.92
30


5. CONCLUSION
5.1 Summary of Contributions
To summarize our work discussed in this thesis, we achieved the following,
i. Successfully identified three key rehabilitation tasks for post-stroke patients to show the feasibility of our proof-of-concept system, “VRInsole”[6],
ii. Collected fine-grained human motion data using nine axis IMU embedded on our in-house developed Smartlnsole Sensor.
iii. The collected data was filtered, processed and analyzed to understand the representation of human motion in terms in linear and angular rate.
iv. Designed an algorithm to convert the acceleration data collected from the IMU sensor into position estimation, minimizing the drift velocity issue associated with IMUs, with good accuracy results.
v. Developed a Deep Learning based LSTM architecture to make predictions for the human activity being performed and proved its efficiency by comparing results with state-of-the-art classifier methods.
31


5.2 Limitations
Our contributions are few of the important components of a preliminary collaborative effort, VRInsole[6], which is work in the biomedical field. This work was aimed to be a feasibility study and designed to eliminate the shortcomings of similar works and also, improve home-based rehabilitation experience as studied in the related works.
This claim is certainly based on a few key assumptions, out of which the most important assumption is that we considered comparison with the studied related work as a good measure to prove the effectiveness of our system.
Also, all of the data collected during the experiments was collected from healthy individuals with no physical impairment, as we were unable to collect data from stroke patients. The reason for this research constraint was inability to obtain required permits for medical collaboration. We made an important assumption to account for this limitation, by assuming that if the estimation produces good accuracy for healthy patients, it should also work equally well for stroke patients albeit, with some modification to the parameters. For the Deep Learning Predictor similar assumption was made, training with dataset collected from stroke patients must produce similar results.
To summarize the point of this section, within the constraints of research we made some assumption, as the work described in this thesis and the project VRInsole[6], was a feasibility study only, which still needs to be evaluated for concrete medical benefits.
32


5.3 Future Work
Since the work done during the course of this thesis is developed to be a proof-of-concept system to prove the feasibility of our method, there are many possibilities that can be explored to expand the system in various aspects. Some of them are enlisted as follows,
i. The position estimation algorithm can be further improved by using magnetometer fusion with gyroscope to obtain corrections for orientation.
ii. Within the constraints of our study, we only used a single sensor to gather data from one of the many joints in the human body. Since stroke is an ailment that affects the entire human body, we can include an array of sensors for multiple joints and scale the position estimation algorithm to full body motion capture/estimation.
iii. Using more complex Deep Learning methods, it may be possible to estimate complimentary joint motion using motion data from a single joint. This will greatly reduce the number of sensors required and make the system even more comfortable and unobtrusive.
iv. Since the VR environment is designed in Unity3D, which supports crossplatform development, our system can be deployed on mobile systems. This will make the system portable and open to newer features like Augmented Reality.
33


REFERENCES
[1] A. Mirelman, P. Bonato, and E. Deutsch Judith, "Effects of Training With a Robot-Virtual Reality System Compared With a Robot Alone on the Gait of Individuals After Stroke," Stroke, vol. 40, no. 1, pp. 169-174, 2009/01/01 2009.
[2] C. f. D. C. a. Prevention. Centers for Disease Control and Prevention. [Online], Available: https://www.cdc.gov/stroke/facts.htm.
[3] C. P. M. Anderson, Cliona Ni PhD; Rubenach, Sally MSocSc; Clark, Michael PhD; Spencer, Carol BAppSc(OT); Winsor, Adrian FAFRM, "Home or Hospital for Stroke Rehabilitation? Results of a Randomized Controlled Trial: II: Cost Minimization Analysis at 6 Months," Stroke: Journal of the American Heart Association, vol. 31(5), no. May 2000, pp. 1032-1037, 2000.
[4] A. Henderson, N. Komer-Bitensky, and M. Levin, "Virtual Reality in Stroke Rehabilitation: A Systematic Review of its Effectiveness for Upper Limb Motor Recovery," Topics in Stroke Rehabilitation, vol. 14, no. 2, pp. 52-61, 2007/03/01 2007.
[5] S. H. Jang et al., "Cortical Reorganization and Associated Functional Motor Recovery After Virtual Reality in Patients With Chronic Stroke: An Experimenter-Blind Preliminary Study," Archives of Physical Medicine and Rehabilitation, vol. 86, no. 11, pp. 2218-2223, 2005/11/01/2005.
[6] H. Oagaz, A. Sable, M. Choi, W. Xu, and F. Lin, "VRInsole: An unobtrusive and immersive mobility training system for stroke rehabilitation," in 2018 IEEE 15th International Conference on Wearable and Implantable Body Sensor Networks (BSN), 2018, pp. 5-8.
[7] F. Lin, A. Wang, Y. Zhuang, M. R. Tomita, and W. Xu, "Smart Insole: A Wearable Sensor Device for Unobtrusive Gait Monitoring in Daily Life," IEEE Transactions on Industrial Informatics, vol. 12, no. 6, pp. 2281-2291, 2016.
[8] Unity. Unity 3D [Online], Available: https://store.unitv3d.com/.
[9] P. N. D. Parth Rajesh Desai, Komal Deepak Ajmera, Khushbu Mehta, 6 Aug 2014.
[10] Nintendo. Nintendo Wii U VR [Online], Available:
https ://www. nintendo. com/wiiu/what-i s-wiiu.
[11] D. Jack et al, "Virtual reality-enhanced stroke rehabilitation," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 9, no. 3, pp. 308-318, 2001.
34


[12] A. Frisoli, F. Salsedo, M. Bergamasco, B. Rossi, and M. C. Carboncini, "A Force-Feedback Exoskeleton for Upper-Limb Rehabilitation in Virtual Reality," Applied Bionics and Biomechanics, vol. 6, no. 2, 2009.
[13] J. J. Eng and P.-F. Tang, "Gait training strategies to optimize walking ability in people with stroke: a synthesis of the evidence," Expert Review of Neurotherapeutics, vol. 7, no. 10, pp. 1417-1436, 2007/10/01 2007.
[14] P. Duncan et al., "A Randomized, Controlled Pilot Study of a Home-Based Exercise Program for Individuals With Mild and Moderate Stroke," Stroke, vol. 29, no. 10, pp. 2055-2060, 1998/10/01 1998.
[15] Q. Gan and C. J. Harris, "Comparison of two measurement fusion methods for Kalman-filter-based multisensor data fusion," IEEE Transactions on Aerospace and Electronic Systems, vol. 37, no. 1, pp. 273-279, 2001.
[16] V. Renaudin, M. Susi, and G. Lachapelle, "Step Length Estimation Using Handheld Inertial Sensors," Sensors, vol. 12, no. 7, pp. 8507-8525, 2012.
[17] J.-L. Reyes-Ortiz, L. Oneto, A. Sama, X. Parra, and D. Anguita, "Transition-Aware Human Activity Recognition Using Smartphones," Neurocomputing, vol. 171, pp. 754-767, 2016/01/01/2016.
[18] B. Sensortech, "Bosch BMX055 Data Sheet."
[19] B. French et al., "Repetitive task training for improving functional ability after stroke," Cochrane Database of Systematic Reviews, no. 11, 2016.
[20] S. M. Linder, A. B. Rosenfeldt, M. Rasanow, and J. L. Alberts, "Forced Aerobic Exercise Enhances Motor Recovery After Stroke: A Case Report," American Journal of Occupational Therapy, vol. 69, no. 4, pp. 6904210010pl-6904210010p8, 2015.
[21] F. Lin, A. Wang, L. Cavuoto, and W. Xu, "Toward Unobtrusive Patient Handling Activity Recognition for Injury Reduction Among At-Risk Caregivers," IEEE Journal of Biomedical and Health Informatics, vol. 21, no. 3, pp. 682-695, 2017.
35


This is the final page of a Thesis and should be a blank page


Full Text