NOVIS — Yuhanxiao Ma's Portfolio Website

NOVIS: An Interactive Lifelogging and Notification System Based on Daily Activity Recognition

Roles: Conceptualization, Project Management, Software, Fabrication, User Studies, Writing
Keywords: Human-Computer Interaction, Context-Aware Computing, Activity Recognition, Lifelogging, Notifications, User Experience, Open-Source Data.

The Challenge

In our fast-paced lives, interruptions from tasks and notifications can hinder productivity and lead to forgetfulness. Traditional notification systems require constant user input and may not reliably reflect completed activities. This results in missed tasks, inefficient time management, and an overwhelming sense of chaos.

The Solution

Introducing NOVIS, a revolutionary interactive system designed to enhance your daily routines through automatic lifelogging and context-aware notifications.

Seamless Activity Recognition: NOVIS utilizes a lightweight, wrist-worn device that passively recognizes your daily activities in real-time without the need for cloud connectivity. It monitors your movements at an impressive rate of 50 times per second.
Automated Lifelogging: Forget about manually logging your tasks! NOVIS automatically ticks off your completed activities based on recognized actions, helping you keep an accurate record of your habits.
Personalized Notifications: Receive timely reminders tailored to your unique preferences and routines. Whether it’s time to take medication, turn off the gas, or wash your hands, NOVIS prompts you just when you need it.
Contextual Awareness: The system adapts to your daily life, learning from your habits and responding to your environment. For example, it may remind you to wear a mask before leaving home, enhancing your safety and well-being.

Hardware Component

Dimensions: 51m*19.11mm*12.61mm

The model features a 3D-printed case, a 110 mAh battery, and a printed circuit board (PCB). The PCB houses the microprogrammed control unit (MCU), a Bluetooth chip, and an inertial measurement unit (IMU) sensor.

System Design

The NOVIS system comprises three components: a wristband device, computer software, and mobile phone software.
- The wristband collects 6-dimensional data.
- The mobile phone software performs offline action recognition and facilitates user interactions.
- The computer software receives and enhances data, generates a deep learning model (CNN-based classification network using TensorFlow), and sends it to the mobile phone. (PC is only our design tool. Only the wristband and mobile phone are needed in daily use.)

Data Acquisition and Model Application

Data Overview

NOVIS has open-sourced a dataset with 13,675 granular entries representing 14 daily activities, aimed at helping designers create personalized services and assisting researchers in their studies. This initiative is the first of its kind to provide such a comprehensive open-source dataset focused on daily activities. The data is collected from both controlled laboratory settings and real-life contexts, capturing authentic user interactions—such as the movements involved in drinking water—while ensuring user anonymity and security, which is a significant advantage over image or video data.

NOVIS Dataset Composition

Original Data: 13,675 entries
Enhanced Data: 5,861 entries generated through data augmentation
Activity Labels: Each entry contains an activity label, participant ID, and six-dimensional sensor data from the wristband's IMU sensor.

Participants engaged in customizable activities, such as opening the door, walking, typing, washing hands, taking pills, wearing masks, smoking, reading, cooking, etc.
Participants selected at least three activities, with additional tasks assigned as necessary.

Data Collection Process

Device Distribution: Participants received identical wristbands with standardized software.
Guided Introduction: Comprehensive guidance was provided on data collection and expected activities.
Contextual Performance: Participants performed activities with consideration of context, resulting in a maximum of 800 frames of data per session.
Data Segmentation: Continuous datasets of 2000 frames were segmented into 800-frame datasets to preserve valid data for analysis.

This method produced data that reflects real-life scenarios better than traditional approaches, contributing to a more effective training model.

Activity Identification

Sliding window data stream

The recognition area for effective data stream

Participants were given a customized experience, choosing from activities such as walking, typing, and washing hands. Each participant received a wristband device with configured data collection software. To ensure high-quality data, we provided a guided introduction and required participants to consider possible activity combinations. They performed their chosen activities while wearing the device, generating continuous data that was segmented into 800-frame data sets, capturing the complexities of real-life scenarios.

Participant Recruitment & Timeline

Timeline for the user experience study and grouping of participants

We recruited 57 participants through advertisements on platforms like VeriMake and targeted emails to students and professionals. Participants were divided into two groups: data collection volunteers and evaluation participants. Among the 20 data collection volunteers, we ensured a diverse representation, including right-handed individuals, left-handed individuals, and those with hand tremors. The average age of volunteers was 31.7 years, with a gender ratio of 65% male to 35% female.

Screenshots of the Mobile Application