EHAAS: Energy Harvesters As A Sensor for Place Recognition on Wearables

03/20/2019 ∙ by Yoshinori Umetsu, et al. ∙ 0

A wearable based long-term lifelogging system is desirable for the purpose of reviewing and improving users' lifestyle habits. Energy harvesting (EH) is a promising means for realizing sustainable lifelogging. However, present EH technologies suffer from instability of the generated electricity caused by changes of environment, e.g., the output of a solar cell varies based on its material, light intensity, and light wavelength. In this paper, we leverage this instability of EH technologies for other purposes, in addition to its use as an energy source. Specifically, we propose to determine the variation of generated electricity as a sensor for recognizing "places" where the user visits, which is important information in the lifelogging system. First, we investigate the amount of generated electricity of selected energy harvesting elements in various environments. Second, we design a system called EHAAS (Energy Harvesters As A Sensor) where energy harvesting elements are used as a sensor. With EHAAS, we propose a place recognition method based on machine-learning and implement a prototype wearable system. Our prototype evaluation confirms that EHAAS achieves a place recognition accuracy of 88.5 F-value for nine different indoor and outdoor places. This result is better than the results of existing sensors (3-axis accelerometer and brightness). We also clarify that only two types of solar cells are required for recognizing a place with 86.2

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 3

page 4

page 5

page 7

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Reviewing and improving people’s lifestyle habits is essential to treat lifestyle-related diseases such as diabetes, hypertension, and mental health issues. Lifelogging system records the personal data generated by a user’s behavioral activities. It is attracting attention for the purpose of understanding life patterns and/or lifestyle habits. Nowadays, a lifelog is recorded automatically by a smartphone or wearable activity trackers. For example, the mobile application “Moves”111Moves - Activity Diary for iPhone and Android :
https://moves-app.com/
can track location, activities, and transportation. Sony’s lifelogging application “Lifelog”222Lifelog - innovative activity tracker Android app from Sony :
https://www.sonymobile.com/global-en/apps-services/lifelog/
tracks which applications are used and what their usage time is. It also logs music played, photos taken, etc. An activity tracker adds more detailed information such as steps, heart rate, and sleep status.

One piece of critical information in the lifelog is location. Recording an accurate position is not essential, but a room-level location (which we call place, hereafter) is required for assessing a user’s lifestyle habits or life pattern. Basically, the aforementioned applications including Moves, record location using GPS embedded in a smartphone. GPS is ubiquitous but it does not work indoors[15]. Therefore, it can only provide a building-level location. To record room-level indoor places such as an office room, meeting room, lecture room, or toilet, other localization technologies are required.

For indoor places, WiFi-based localization[19, 9, 27, 7, 6] is the most popular because WiFi access points have already been deployed widely and everyone has a WiFi-enabled smartphone. However, from the viewpoint of long-term lifelogging for lifestyle improvement, smartphone-based approaches are not suitable. The current mobile operating systems do not allow applications to keep tracking the location in the background because of battery conservation. Active/passive RFID-based indoor localization systems[3, 28] are also popular and able to achieve cm-level accuracy. However, they need some infrastructure (RFID tags/RFID readers) in environments and continuous/regular power charging of tags/readers for daily sensing. For our purpose, iBeacon333iBeacon :
https://developer.apple.com/ibeacon/
from Apple, where iOS keeps monitoring the Bluetooth signals in the operating system, might be the best solution. However, it is costly because it requires the deployment of new beacon tags into each place and the perpetual maintenance of their batteries. No wearables (including Apple watch) can keep sensing iBeacon signals continuously because that would drastically deplete the small battery of a wearable.

In this paper, we propose a wearable lifelogging system called EHAAS (Energy Harvesters As A Sensor), which utilizes a set of different energy harvesting elements as a sensor for recognizing places. Energy Harvesting (EH) has been expected to be used as a power source for running sensors and processors[24, 2]. It became a reality because the power consumption of those devices becomes smaller every year. In April 2018, a smartwatch called PowerWatch444Matrix PowerWatch :
 https://www.powerwatch.com/
, which uses thermoelectric EH, was released. It can count the number of steps by a low power accelerometer. However, the generated electricity is not enough to run other sensors or a network module for localization. Therefore, we believe that a fundamentally different approach is necessary to realize room-level localization with wearables.

Our key idea is to leverage a disadvantage of EH that the amount of generated electricity depends on the surrounding environment. Generated electricity of a solar panel varies based on the light conditions such as brightness, wavelength, and angle of the light source. Also, the generated electricity changes when a different material is used for a solar panel. Kinetic EH also has a disadvantage in that it only generates power when it moves physically. Leveraging the instability of EH elements as a feature of places, the variation of generated electricity could be used as a sensor for place recognition. If EH works as a sensor as well as a power supply, there will be a big potential for realizing a place recognition system with lower power consumption. The contributions of this paper are summarized as follows:

  • First, to investigate the feasibility of our idea, we evaluate the characteristics of various EH devices such as solar cells, peltier elements, and piezo elements. According to previous research, the characteristics of EH devices have not been studied well enough to evaluate the possibility of a sensor to recognize places. We show that EH devices are not only substitute for existing sensors, but can also be used as a sensor that is suitable for place recognition through the characteristic evaluation of EH devices.

  • Second, we evaluate the amount of generated electricity in various places to establish a relationship between places and generated electricity in those places. The target places in this paper comprise common places in our university such as a classroom and a restroom, aiming to apply the developed technology to an office environment. We show that each place generates a unique amount of electricity according to its environment or the movement of a user, and EH devices have a large potential for recognizing places.

  • Finally, we implement a prototype of a wearable device combining multiple EH elements as a sensor. Through experiments with the prototype, we confirm that EHAAS has achieved a place recognition accuracy of 88.5% F-value for nine different places including indoors and outdoors in the university. Additionally, we compare this result with the accuracy recognized by existing sensors such as accelerometers and luminometers. As a result, a higher accuracy is obtained with EHAAS. Furthermore, we clarify that this system with only two types of solar cells can recognize a place with 86.2% accuracy.

The rest of this paper is organized as follows. Section II reviews existing work and energy harvesting elements related to this paper. The preliminary measurements are provided in Section III, and Section IV presents our proposed place recognition system with energy harvesters. In Section V, we describe the performance evaluation and evaluation results. Finally, Section VI concludes this paper.

Ii Related work

As related work, we summarize existing studies of both indoor localization techniques and energy harvesting technologies.

Ii-a Indoor localization

Indoor localization techniques[29] have been widely studied. Sensor data and algorithms used for localization are diverse, depending on the target accuracy[20]. In this paper, we only consider the techniques that are feasible on ordinary devices and are for room-level localization (the target of this paper)[16]. Examples include a radio-based technique that utilizes WiFi or Bluetooth, a dead reckoning with inertial sensors[23], and acoustic fingerprinting[25].

WiFi-based localization methods usually adopt the technique called fingerprinting[5]

. This technique surveys a fingerprint of radio signals at various places in advance. Based on the similarity of fingerprints, it estimates the location. In the case of BLE beacons

[30, 16], the method is basically the same, and Kyritsis A.I., et al.[16] achieved 81.17% 100% accuracy (90.2% on average) with fingerprints at 63 different places in seven rooms. However, it is difficult to continuously scan WiFi or BLE signals for lifelogging on wearable devices that only have a small battery. In fact, most of the studies such as [16] use a laptop or a smartphone, not a smartwatch, for the experiment.

As a study of localization using a smartwatch, Lee et al. have proposed a place identification method for a home environment[18] that utilizes activity fingerprinting obtained by a motion sensor. However, since this research is based on the premise that the movement of an arm is different for each room, it cannot identify when the same work (PC or meeting) is done in different rooms, like in a university or office. Also, similar to the case with radio signal scanning, continuous use of the acceleration sensor consumes battery power and is not realistic for long-term lifelogging by a wearable device.

Ii-B Energy harvesting elements

Below, we introduce representative energy harvesting elements.

Photovoltaic (Solar cell): Solar cells, which use the photovoltaic effect, are the most popular energy harvesting elements because they are inexpensive and have high flexibility in size. There are several kinds of materials having a photovoltaic effect. The amount of generated electricity depends on the kind of material in addition to the brightness and wavelength. Representative materials are amorphous silicon, polycrystalline silicon, and organic thin films. Solar cells are widely used in calculators and traditional watches. Sometimes, they are attached to bags and backpacks. However, they are not used for the latest wearable devices because of the need for a small cell size. We can expect that, in the future, they will be embedded into clothes because RIKEN has released a novel solar cell that is launderable as well as pasteable using an iron.

Kinetic (Piezo element): A piezoelectric element utilizes the piezo effect that generates electricity by distortion. As an example of use as a power supply, the power generating floor produced by Soundpower Corporation555Soundpower corporation :
http://www.soundpower.co.jp/
in Japan is well-known. However, this element is usually used as a pressure sensor. We used it for sensing the posture of a worker on a chair[22] and for sensing the weight distribution on a glove[8]. H. Kalantarian et al. have used a piezo element as an insole pressure sensor and a power source [12]. Also, S. Khalifa et al. have used it as a motion sensor and a power source [14]. However, in all of these research studies, including our glove, electricity is not generated unless it is in force or in motion. Therefore, it cannot be used for recognizing places.

Thermoelectrics (Peltier element): The peltier element is an energy harvester using the peltier effect that generates electricity by a temperature difference. Usually, it is used for cooling a CPU by flowing current to the element. In April 2018, a smartwatch called PowerWatch using a peltier element as a power source was released. This watch is equipped with LCD, BLE, and a step counting function. Since the temperature difference is critical, it is a problem that electricity is not generated when the body temperature and the outside air temperature are similar.

Iii Preliminary measurement

Prior to designing the place recognition system, we investigate the characteristics of the following energy harvesting elements: solar cells (5 types), piezo element, and peltier element.

SC1 SC2 SC3 SC4 SC5
Picture
Manufacture SUNYOOO solar Limited. Fujikura VIMUN SHARP Panasonic
Type Polycrystalline silicon Organic thin film Amorphous silicon Polycrystalline silicon Thin amorphous silicon
Size () 8060 9160 3011 6740 11273
TABLE I: Investigated solar cells

Iii-a Generated-electricity measurement system

For the preliminary experiment, we developed a system that can measure the amount of generated electricity of energy harvesters (right figure in Fig. 1).

Fig. 1: Experimental environment for measuring the solar cells

We utilize an all-in-one Arduino-compatible and Bluetooth Low Energy (BLE) development board (Adafruit Feather nRF 52 Bluefruit LE) that has an ARM Cortex M4F processor clocked at 64MHz and eight analog inputs. Each energy harvesting element is connected to an analog input. The voltage value is acquired as a digital value by AD conversion in the board. The sampling frequency of the voltage value is 50 Hz, and data is transmitted to the laptop via BLE. On the laptop, Python scripts draw graphs and accumulate data.

In addition, we add SenStick [21] on the experimental board. SenStick is an ultra-tiny multi-sensing board having eight sensors: 9-axis (MPU-9250), temperature/humidity (SHT20), illuminance (BH1780GLI-E2), pressure (LPS25HBTR), and ultraviolet (VEML6070). We use it to collect the data for comparison.

Iii-B Solar cells

First, we investigate solar cells, which are widely used and easy to buy. There are several types of solar cells with different materials and manufacturing methods, and different characteristics against light conditions such as brightness and wavelength. In this paper, we prepared five small solar cells sold in Japan (Table I) and investigated their characteristics. The features of these solar cells are as follows.

  • SC1, SC4 (Polycrystalline silicon):
    Polycrystalline silicon is currently the most popular solar cell. SC1 has a glass coating on the surface, while SC4 has no glass coating.

  • SC2 (Organic thin film):
    Organic thin films are solar cells that can output high voltage even in a low light environment. They increase the light receiving sensitivity to visible light by adsorbing a dye on the surface of titanium oxide. From the viewpoint of manufacturing cost and design ability, it is expected that this solar cell will be widely applied.

  • SC3, SC5 (Amorphous silicon):
    Amorphous silicon is a kind of silicon with an irregular crystal structure. Although the energy conversion efficiency is lower than the polycrystalline silicon type solar cell, it can be thinner than polycrystalline silicon and can keep a high voltage output even at high temperature.

Iii-B1 Characteristic against the intensity of the light

Fig. 1 shows the experimental environment where Panasonic’s silica bulb LW100V 54W was used as a light source in a dark room. We investigated the characteristics against the intensity of the light by changing the distance between the experimental board and the light source.

Fig. 2: Generated electricity against the intensity of the light

Fig. 2 shows the amount of generated electricity of each solar cell. The generated electricity of all the solar cells increased linearly according to the light intensity. In particular, SC2 and SC3 generated more electricity than the others. This suggests the possibility that these solar cells can be used as illuminance sensors for distinguishing places.

Iii-B2 Characteristic against the wavelength of the light

Next, we investigated the characteristics against the wavelength of the light. In this experiment, we measure the quantum efficiency (QE), the fraction of photons incident on the solar cell that are converted into electrons without loss, because the energy conversion efficiency of solar cells depends on QE.

Fig. 3: A solar simulator for measuring quantum efficiency

We utilize a solar simulator (Bunko-keiki, CEP-2000RP), shown in Fig. 3, to measure QE under the condition that the intensity of the irradiation light is constant, the wavelength is set to 10 different values in the range 300 nm 1200 nm.

Fig. 4 shows the normalized quantum efficiency (QE) against the wavelength. Small peaks around 800 nm and 850 nm are waveforms generated when the measuring equipment switches light sources, so we ignore them.

We can observe that the sensitivity against wavelength is different among the types of solar cells. Therefore, this suggests that by utilizing multiple solar cells together, we can get information about the combination of wavelengths at a certain place. At a glance, there is no difference, but in recent offices, different kinds of lighting may often be used in different rooms, according to the purpose of room. This is because the condition of the lighting can affect workers’ mental state, such as emotion[11] and fatigue[13], and optimizing the lighting environment in workplaces is regarded as important[17]. Hence, multiple kinds of solar cells may distinguish different places by the characteristic wavelength distribution of each place.

Fig. 4: Normalized QE against wavelength

Iii-C Piezo element

To investigate the characteristics of piezo elements, we developed a measurement system (Fig. 5) equipped with a piezo element and SenStick, which can measure the generated-electricity of a piezo element and 9-axis motion simultaneously.

(a) View from above
(b) View from upper diagonal
Fig. 5: A measurement system of piezo and 9-axis motion

In this experiment, we installed LDT0-028K, a piezoelectric film of Measurement Specialties, Inc., parallel to the Y-Z plane in the measurement system. This means that the piezo element will curve in the X-axis direction according to the movement. Fig. 6(a) shows the result of acceleration values of the X, Y, Z axes. Fig. 6(b) shows the generated electricity of the piezo element, which includes raw data and the data smoothed by the moving average filter (time window is 0.2 sec).

(a) Acceleration
(b) Generated electricity
Fig. 6: Characteristic of the piezo element against the motion

This result suggests that the generated electricity of a piezo element has some correlation with the movement of the object and can be considered as a sensor for detecting motion.

Iii-D Peltier element

In the investigation of peltier elements, we observed the generated electricity by moving between indoors and outdoors. Fig. 7 shows the developed wearable system for this experiment, which equips a peltier element inside a band. The temperatures of the atmosphere and skin surface are measured by SenStick and Empatica’s E4 wristband666Real-time physiological signals — E4 EDA/GSR sensor :
https://www.empatica.com/research/e4/
, respectively.

(a) Overall view
(b) Wearing view
Fig. 7: A wearable measurement system for peltier element

Fig. 8(a) represents the transition in skin temperature and ambient temperature. Fig. 8(b) shows the generated electricity of the peltier element, the raw data, and the data smoothed by the moving average filter (2 sec of time window). We can observe a high correlation between the temperature change and the generated electricity. This suggests that the peltier element can be used as a sensor to distinguish indoors or outdoors if we assume that the skin temperature is almost constant.

(a) Temperature
(b) Generated electricity
Fig. 8: Characteristic of the peltier element against the temperature difference

Iii-E Summary of measurement results

We summarize the preliminary experiment as follows.

  • Solar Cells
    Since the characteristics against brightness and wavelength are different depending on the material, a combination of several types of SCs is expected to work well for place recognition.

  • Piezo element
    Since it works like accelerometer, a piezo element can be used to recognize the movement of a human. However, it may not be suitable for places where the user stays stationary.

  • Peltier element
    A peltier element can be used for indoor or outdoor detection. However, the peltier element has a usage restriction in that it must be in contact with the skin.

In this section, we investigated the capability of each EH element. In the following sections, we try to develop a place recognition system by building a sensor device and evaluating it through experiments. In particular, considering the use cases in offices or workplaces, we created a nameplate-like sensor device. Following the knowledge obtained in this section, we loaded only multiple solar cells and a piezo element on the device777The peltier element was excluded from the device because it has restrictions on hardware implementation with respect to the available environment and shape of devices. First, it does not work in a hot environment such as in summer, because it always needs a difference between the temperatures of the skin and the air to generate electricity. Furthermore, it also has to keep contact with the skin and suitable shapes of devices for continuous operation is limited for office use. These are unsuitable properties for a nameplate-like device.. Furthermore, we implemented all the EH elements except for the peltier element to find the best combination of EH elements.

Iv A place recognition system with energy harvesters

Based on the result of the preliminary measurements, we design a place recognition system where several types of energy harvesters are used as a sensor. We call it EHAAS, hereafter.

Iv-a Assumed lifelogging service

The goal of EHAAS is to realize long-term lifelogging composed of a time series of visited places in the working environment. The right side of Fig. 9 shows an image of the output of EHAAS.

Iv-B EHAAS architecture

EHAAS consists of a hardware component and a software component (algorithm). The former utilizes Bluefruit of Adafruit as the main board, to which multiple energy harvesting elements are connected as shown in the left side of Fig. 9. The electricity generated by the energy harvesting elements is used for both energy and the sensor.

Fig. 9: Proposed EHAAS architecture
Fig. 10: Example of collected data

In the main board, the following four modules are implemented.

Iv-B1 A/D converter

All the generated electricity is input into an A/D converter on the board. The resolution and sampling rate of the A/D converter on Adafruit Bluefruit are 10-bit (0…1023) and 16Hz respectively. The input voltage is adjusted to below 3.6V by the proper resistance. This means that every digit returned from the ADC is 3.515625mV (= 3600mV/1024).

Iv-B2 Feature extraction

We use several types of solar cells and a piezo element for extracting the features of places because the amount of generated electricity will depend on the environment such as light intensity, light wavelength, temperature, and the wearer’s movement. This module generates feature vectors composed of the generated electricity of each element every 2 seconds.

Iv-B3 Place recognition

This module classifies the places based on a movement feature and an environmental feature. Here, the movement feature is abstracted into the same length as the environmental feature and combined. Finally, the classifier generated by machine learning decides the most probable place. The details will be explained in the next subsection.

Iv-B4 Energy storage

All the generated electricity is charged into energy storage, such as a capacitor. However, the implementation of this part is beyond the scope of this paper.

Iv-C Place recognition model

We construct the model for place recognition by using machine learning. Through preliminary investigations, we employ Random Forests (RF)

[4] as the best among various machine learning algorithms.

Training data are needed for constructing the model. The feature used as the training data is the amount of generated electricity. This is obtained as the movement feature or the environmental features. As features, statistical values: average, variance, sum, median, maximum, minimum, difference between maximum and minimum, are calculated for every 2 second interval. Also, place labels are assigned to each interval as ground truth.

It is difficult to construct a model that can adapt to all possible places, because people stay/pass various places in daily life. However, one of the goals of this system is lifelogging in a building (e.g., workplace, school, etc.), and the number of places where a user often stays in such an indoor environment is limited. We assume the number of places visited by one user is no more than 10, such as an office, meeting room, toilet, stair, corridor, elevator, outdoor, cafeteria, and so on. Therefore, a training dataset could be prepared in each environment, and in this paper it was collected at representative places in our university.

Additionally, the training data should be collected considering various situations such as different time periods (morning, evening, night) and for different weather (sunny, cloudy, rainy, etc). Because our target is to recognize a place, not the behavior of each person, we do not need to develop each person’s model. Based on this, we believe that it is sufficient to collect 9 datasets that consider time and weather differences from no more than 10 places.

V Performance evaluation

We evaluate the usefulness and accuracy of our proposed system through experiment.

V-a Evaluation environment and scenario

As a target environment, we select our university campus where staff and students move among several buildings, rooms, and facilities within a stay. There are various kinds of places on the university campus, such as classrooms, laboratories, toilets, shops, and cafeterias. However, in such an environment, GPS cannot distinguish the room-level places. Even when walking outdoors between buildings, it is hard to receive GPS signals and calculate the position because the duration spent outdoors is too short. Fig. 10 shows a concrete movement scenario and the collected data. In this experiment, we target the following nine places: Laboratory, Seminar room, Toilet, Stairs, Elevator, Corridor (1st floor), Corridor (4th floor), Outdoors, and Cafeteria. In Fig.10 (a) and (b), we show that each place generates a unique amount of electricity according to its environment or the movement of a user, and EH devices have a big potential for recognizing places.

V-B Data collection

To collect the data, we developed an experimental board where a microprocessor and several energy harvesting elements are affixed on the surface, as shown in Fig. 11. Five different kinds of solar cells are attached to the board horizontally. A piezo element is set at the center. We adjust the weight attached to the tip of the piezo element to generate a moderate vibration (electricity) according to the body movement. We attached SenStick for measuring ground truth data of acceleration, gyro, magnetic, temperature, and light intensity.

Fig. 11: Prototype wearable device

We collected data in three days with different weather and at different times. The first day was rainy, the second day was cloudy, and the third day was sunny. The data was collected in different periods of time such as morning, afternoon, and evening including night. A data collector subject wearing this device visited nine places by walking along a predetermined route. Finally, we collected 10 sets of data in different situations.

We calculated seven features (average, variance, sum, median, maximum, minimum, difference between maximum and minimum) every two seconds from the collected data. The window size (two seconds) was decided empirically. As a result, a dataset with 42 features (seven features six elements) was created. Moreover, the ground truth label was applied by checking the video recorded in the experiment. Similarly to the features, we assigned place labels for every 2 seconds. We performed the same process for all the collected data and created ten case datasets.

V-C Evaluation method

We used a leave-one-case-out cross-validation for the evaluation. Out of the ten cases collected in different situations, one case was excluded as a test dataset for evaluation, and the remaining datasets were used for constructing a place recognition model by machine learning (Random Forest). We carried out this process for each of the 10 datasets and evaluated the accuracy of our proposed place recognition system, trying to avoid overfitting and minimize generalization errors. Also, we evaluated the effectiveness of this method by comparing the accuracy with that derived by a conventional method based on an illuminance sensor and accelerometer.

In addition, we compared the accuracy among different combinations of energy harvesters to investigate the possibility of reducing the number of elements. In particular, the combination of solar cells is important because it inhibits miniaturization. As we intend to apply EHAAS to a wearable system, a smaller number of elements will be better for implementation.

V-D Result

Precision Recall F-value Num. of data
Lab 0.969 0.944 0.957 7899
Seminar_Room 0.701 0.752 0.725 1767
Cafeteria 0.837 0.725 0.777 945
Corridor_4th 0.537 0.711 0.612 464
Toilet 0.955 0.823 0.884 434
Corridor_1st 0.635 0.738 0.683 427
Stairs 0.764 0.86 0.809 350
Outdoors 0.97 0.855 0.909 345
EV 0.721 0.913 0.805 161
avg / total 0.887 0.877 0.88 12792
TABLE II: place recognition accuracy of the proposed system (sensor: All energy harvesting elements)
Precision Racall F-value Num. of data
Lab 0.872 0.896 0.883 6867
Seminar_Room 0.418 0.350 0.381 1416
Cafeteria 0.402 0.367 0.384 1340
Corridor_4th 0.450 0.508 0.478 366
Toilet 0.483 0.413 0.445 516
Corridor_1st 0.435 0.477 0.455 384
Stairs 0.600 0.661 0.629 313
Outdoors 0.623 0.845 0.718 317
EV 0.410 0.423 0.416 182
avg / total 0.697 0.707 0.701 11701
TABLE III: place recognition accuracy of the conventional system (sensor: illuminance)
Precision Racall F-value Num. of data
Lab 0.911 0.869 0.889 7452
Seminar_Room 0.63 0.656 0.643 1534
Cafeteria 0.451 0.531 0.488 1516
Corridor_4th 0.584 0.602 0.593 405
Toilet 0.632 0.514 0.567 560
Corridor_1st 0.55 0.648 0.595 421
Stairs 0.834 0.856 0.845 353
Outdoors 0.88 0.872 0.876 344
EV 0.597 0.667 0.63 189
avg / total 0.781 0.769 0.774 12774
TABLE IV: place recognition accuracy of the conventional system (sensor: acceleration and illuminance)
Num. of EHs Combination Precision Recall F-value
All (6 types) 1,2,3,4,5,p 0.887 0.877 0.880
5 types Best 2,3,4,5,p 0.89 0.883 0.885
Worst 1,2,3,4,p 0.883 0.867 0.872
4 types Best 1,2,3,5 0.89 0.877 0.882
Worst 1,4,5,p 0.843 0.835 0.836
3 types Best 3,4,5 0.883 0.875 0.878
Worst 2,4,p 0.785 0.785 0.784
2 types Best 1,2 0.874 0.855 0.862
Worst 2,p 0.745 0.763 0.751
1 type Best 3 0.763 0.768 0.762
Worst p 0.644 0.686 0.655

*p: piezo element, Numerical number : type of solar cell. 1 means SC1.

TABLE V: Average F-value of each combination in the proposed system

First, in Table II, we show the result in the case that all the EH elements are used as a sensor. We see that our system could accurately recognize Lab with 95.9% of F-value and Outdoors with a 91.0% F-value. However, the F-value of both corridors on the 1st floor and 4th floor are a bit lower (71.8% and 61.4%) than other places because the light condition in corridors in the same building is similar. The average F-value for all of the nine places is 88.3%, which we believe is high enough for practical use, compared with the accuracy (90.2% in average) achieved by BLE-based room-level localization [16]. We also evaluated the place recognition accuracy of a conventional system[26] where an illuminance sensor is used as a sensor, in combination with an acceleration sensor for the recognition. Table III shows the result for the conventional system that utilizes an illuminance sensor (Photocell/Photodiode) for place recognition. It was able to recognize the place “Lab” with 88.3% accuracy. However, the accuracy of other places is low. Overall, the average F-value for all of the nine places is 70.1%. Table IV shows the result of the system where both an acceleration sensor and an illuminance sensor are used for place recognition. The average F-value for all of the nine places is 77.4%. Except for “Stairs”, where the place recognition accuracy is lower than our proposed system. We think that the information of the acceleration sensor worked effectively for detecting Stairs. However, it did not work well for recognizing other places.

Next, we compare the F-value for different combinations of EH elements to investigate the deterioration of the F-value when reducing the number of EH elements used. We investigated the performance of all the combinations of six EH elements, 63 patterns in total. Table V shows the results, where we pick up only the best result and the worst result for each number of EH elements used.

We see that the F-value is basically reduced when the number of elements is reduced. The best performance, 88.5%, was achieved by the combination of five EH elements (SC2, SC3, SC4, SC5, piezo). On the other hand, the worst performance, 65.5%, was achieved in the case where we used one piezo element. However, if we add one more EH element, the F-value rises to 86.2%, where SC1 and SC2 are used. The performance difference against the best combination (which requires 6 elements) is just 2.3%. This result is very convincing because we observed that SC1 (Polycrystalline silicon) and SC2 (Organic thin film) have different characteristics against the illuminance and wavelength in the preliminary experiment (see Fig. 2 and Fig. 4). From this result, we can say that the combination of a couple of EH elements having different characteristics have a high capability to be used as a sensor for place recognition. Furthermore, this result indicates the possibility of the miniaturization of devices for practical use of this system.

Table VI shows the detailed result of place recognition in the case where we used only SC1 and SC2 as a sensor. Compared with Table II, we see that the recognition accuracy of Corridor_1st and Corridor_4th are degraded. These places are also hard to distinguish for other combination of EH elements because of the similarity of the environment (light condition). However, the place recognition accuracy for other places are almost the same despite the number of EH elements being reduced from six to two. Hence, we consider that the most reasonable combination for wearables is SC1 and SC2.

Precision Recall F-value Num. of data
Lab 0.97 0.92 0.944 7899
Seminar_Room 0.635 0.761 0.692 1767
Toilet 0.944 0.783 0.856 434
Stairs 0.685 0.814 0.744 350
EV 0.649 0.814 0.722 161
Corridor_1st 0.638 0.677 0.657 427
Corridor_4th 0.452 0.651 0.534 464
Outdoors 0.925 0.861 0.892 345
Cafeteria 0.894 0.726 0.801 945
avg / total 0.874 0.855 0.862 12792
TABLE VI: place recognition accuracy of the proposed system (sensor: SC1 and SC2)

Finally, we discuss the effect of the piezo element on the place recognition performance. The result, 65.5% of average F-value, was not good in this experiment. We expected that the piezo element would help to improve the performance when it was used with other EH elements. Although we did not assume a no-light environment in this experiment, using a piezo element as a sensor will be useful for environments with no light where the solar cell does not generate electricity at all.

Vi Conclusion

In this paper, we have proposed a novel place recognition system for wearables called EHAAS (Energy Harvesters As A Sensor), where multiple energy harvesting elements are used as a sensor. First, we investigated various EH elements to clarify the characteristics of those elements against the environment. Second, we implemented a prototype wearable device and proposed a place recognition algorithm. Finally, we showed that our proposed EHAAS achieved 88.5% accuracy with five EH elements and 86.2% with just two EH elements. Furthermore, we confirmed that this system has better accuracy than an existing system consisting of an illuminance sensor and accelerometer, in place recognition in the daily life of indoor workers. This means that we can implement an EHAAS-based system on a smaller wearable such as a watch (Fig. 7).

In this paper, we did not use EH element as an energy source. As a next step, we will design a circuit that uses EH elements as both an energy source and a sensor. Also, we will investigate other combinations of EH elements including peltier elements and rectenna[10].

Acknowledgment

This work was supported by JSPS Grants-in-Aid for Scientific Research (KAKENHI) Grant Number 18H03233.

References

  • [1]
  • [2] Nagendra B Bharatula, Stijn Ossevoort, Mathias Stäger, and Gerhard Tröster. 2004. Towards wearable autonomous microsystems. In International Conference on Pervasive Computing. Springer, 225–237.
  • [3] M. Bouet and A. L. dos Santos. 2008. RFID tags: Positioning principles and localization techniques. In 2008 1st IFIP Wireless Days. 1–5. DOI:http://dx.doi.org/10.1109/WD.2008.4812905 
  • [4] Leo Breiman. 2001. Random forests. Machine learning 45, 1 (2001), 5–32.
  • [5] Mauro Brunato and Roberto Battiti. 2005. Statistical learning theory for location fingerprinting in wireless LANs. Computer Networks 47, 6 (2005), 825–845.
  • [6] Marc Ciurana, Sebastiano Cugno, and Francisco Barcelo-Arroyo. 2007. WLAN indoor positioning based on TOA with two reference points. In Positioning, Navigation and Communication, 2007. WPNC’07. 4th Workshop on. IEEE, 23–28.
  • [7] Matteo Cypriani, Frédéric Lassabe, Philippe Canalda, and François Spies. 2009. Open wireless positioning system: a Wi-Fi-based indoor positioning system. In Vehicular Technology Conference Fall (VTC 2009-Fall), 2009 IEEE 70th. IEEE, 1–5.
  • [8] A. H. Akpa Elder, Fujiwara Masashi, Arakawa Yutaka, Suwa Hirohiko, and Yasumoto Keiichi. 2018. GIFT: Glove for Indoor Fitness Tracking System. In Third IEEE PerCom Workshop on Pervasive Health Technologies(PerHealth2018). IEEE, 1–6.
  • [9] C. Feng, W. S. A. Au, S. Valaee, and Z. Tan. 2012. Received-Signal-Strength-Based Indoor Positioning Using Compressive Sensing. IEEE Transactions on Mobile Computing 11, 12 (Dec 2012), 1983–1993. DOI:http://dx.doi.org/10.1109/TMC.2011.216 
  • [10] A Georgiadis, G Vera Andia, and A Collado. 2010. Rectenna design and optimization using reciprocity theory and harmonic balance analysis for electromagnetic (EM) energy harvesting. IEEE Antennas and Wireless Propagation Letters 9 (2010), 444–446.
  • [11] Knez Igor and Kers Christina. 2000. Effects of Indoor Lighting, Gender, and Age on Mood and Cognitive Performance. Environment and Behavior 32, 6 (2000), 817–831.
  • [12] H. Kalantarian and M. Sarrafzadeh. 2016. Pedometers Without Batteries: An Energy Harvesting Shoe. IEEE Sensors Journal 16, 23 (Dec 2016), 8314–8321. DOI:http://dx.doi.org/10.1109/JSEN.2016.2591331 
  • [13] C.H.J.Smolders Karin and Kort Yvonne, A.W.de. 2014. Bright light and mental fatigue: Effects on alertness, vitality, performance and physiological arousal. Journal of Environmental Psychology 39 (2014), 77–91.
  • [14] Sara Khalifa, Guohao Lan, Mahbub Hassan, Aruna Seneviratne, and Sajal K Das. 2017. HARKE: Human Activity Recognition from Kinetic Energy Harvesting Data in Wearable Devices. IEEE Transactions on Mobile Computing (2017).
  • [15] Masakatsu Kourogi, Nobuchika Sakata, Takashi Okuma, and Takeshi Kurata. 2006. Indoor/outdoor pedestrian navigation with an embedded GPS/RFID/self-contained sensor system. In Advances in Artificial Reality and Tele-Existence. Springer, 1310–1321.
  • [16] Athanasios I Kyritsis, Panagiotis Kostopoulos, Michel Deriaz, and Dimitri Konstantas. 2016. A ble-based probabilistic room-level localization method. In Localization and GNSS (ICL-GNSS), 2016 International Conference on. IEEE, 1–6.
  • [17] Bellia L., Bisegna F., and Spada G. 2011. Lighting in indoor environments: Visual and non-visual effects of light sources with different spectral power distributions. Building and Environment 46, 10 (2011), 1984–1992.
  • [18] Seungwoo Lee, Yungeun Kim, Daye Ahn, Rhan Ha, Kyoungwoo Lee, and Hojung Cha. 2015. Non-obstructive room-level locating system in home environments using activity fingerprints from smartwatch. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 939–950.
  • [19] Hung-Huan Liu and Yu-Non Yang. 2011. WiFi-based indoor positioning for multi-floor Environment. In TENCON 2011 - 2011 IEEE Region 10 Conference. 597–601. DOI:http://dx.doi.org/10.1109/TENCON.2011.6129175 
  • [20] Rainer Mautz. 2012. Indoor positioning technologies. (2012).
  • [21] Yugo Nakamura, Yutaka Arakawa, Takuya Kanehira, Masashi Fujiwara, and Keiichi Yasumoto. 2017. SenStick: Comprehensive Sensing Platform with an Ultratiny All-In-One Sensor Board for IoT Research. Journal of Sensors 2017 (2017), 16.
  • [22] Yasuhiro Otoda, Teruhiro Mizumoto, Yutaka Arakawa, Chihiro Nakajima, Mitsuhiro Kohana, Motohiro Uenishi, and Keiichi Yasumoto. 2018. Census: Continuous posture sensing chair for office workers. In Consumer Electronics (ICCE), 2018 IEEE International Conference on. IEEE, 1–2.
  • [23] Deepak Pai, Inguva Sasi, Phani Shekhar Mantripragada, Mudit Malpani, and Nitin Aggarwal. 2012. Padati: A robust pedestrian dead reckoning system on smartphones. In Trust, Security and Privacy in Computing and Communications (TrustCom), 2012 IEEE 11th International Conference on. IEEE, 2000–2007.
  • [24] Joseph A Paradiso and Thad Starner. 2005. Energy scavenging for mobile and wireless electronics. IEEE Pervasive computing 1 (2005), 18–27.
  • [25] Stephen P Tarzia, Peter A Dinda, Robert P Dick, and Gokhan Memik. 2011. Indoor localization without infrastructure using the acoustic background spectrum. In Proceedings of the 9th international conference on Mobile systems, applications, and services. ACM, 155–168.
  • [26] Maurer Uwe, Rowe Anthony, Smailagic Asim, and Siewiorek Daniel. 2006. Location and Activity Recognition Using eWatch: A Wearable Sensor Platform. Ambient Intelligence in Everyday Life 3864 (2006), 86–102.
  • [27] Sunkyu Woo, Seongsu Jeong, Esmond Mok, Linyuan Xia, Changsu Choi, Muwook Pyeon, and Joon Heo. 2011. Application of WiFi-based indoor positioning system for labor tracking at construction sites: A case study in Guangzhou MTR. Automation in Construction 20, 1 (2011), 3 – 13. DOI:http://dx.doi.org/https://doi.org/10.1016/j.autcon.2010.07.009  Global convergence in construction.
  • [28] Fu Xiao, Zhongqin Wang, Ning Ye, Ruchuan Wang, and Xiang-Yang Li. 2018. One More Tag Enables Fine-Grained RFID Localization and Tracking. IEEE/ACM Trans. Netw. 26, 1 (Feb. 2018), 161–174. DOI:http://dx.doi.org/10.1109/TNET.2017.2766526 
  • [29] Faheem Zafari, Athanasios Gkelias, and Kin K. Leung. 2017. A Survey of Indoor Localization Systems and Technologies. CoRR abs/1709.01015 (2017). http://arxiv.org/abs/1709.01015
  • [30] Yuan Zhuang, Jun Yang, You Li, Longning Qi, and Naser El-Sheimy. 2016. Smartphone-based indoor localization with bluetooth low energy beacons. Sensors 16, 5 (2016), 596.