Deep Learning

Course by Udacity + Google.

My Master’s thesis in social robotics was an application of Machine Learning, a classification of emotions from voice signal features. With the undergraduate level Machine Learning knowledge, I was able to build on it through literature review, discussions with my grad school peers and experimentation. Further structured learning of Machine Learning was something that escaped me, especially now that I’m out of the academia environment and working full-time as an e-commerce software developer.

Pursuing the bleeding edge technology have always benefitted me, and as Wayne Gretzky says, “Skate to where the puck is going to be, not where it has been”, I need to learn more about the hottest tech of today without a doubt, Deep Learning. It’s a branch of Machine Learning and thankfully, I have the prerequisites to learn it and there’s no reason not to take advantage.

Here's the curriculrum:

The initial motivation given by Vincent Vanhoucke, Principal Scientist @ Google is certainly grasping and with my experience writing the motivation of my applied machine learning research in my Master’s thesis, I cannot agree with him more. In the future, coding skill will simply be a tool to learn the novelty of machine learning solution to problems in the world.

I loved his presentation of the need to optimize the loss function to basic logistic classification problems. Great review as it’s been a while and this seems promising! I will be making blog posts as I find well-explained concepts worth sharing.

Happy learning!

• Date: Nov 2017
• Category: Machine Learning, Software Engineering, Personal Development

Close Project

Product Management

10-week Bootcamp @ Brainstation.

My long-term goal is to transition towards management in the field of tech. But there are project management, technical management, site reliability management, etc... too many out there! To avoid spending too much time on contemplation, I decided to break and ice and just dive in, by taking a bootcamp course for PRODUCT Management! The Product Owners in my Scrum team at work are the ones who bring in the business needs in the form of developement tickets and I want to know how they came up with those. Learn more about the course curriculum here:
Product Management Fall 2017 @ Brainstation Course Package

Also, check out the mid-term presentation of our product, 2018 Worldcup Bars! We received many feedbacks, leading to great ideas for pivoting!

Midterm presentation

• Date: November 2017
• Category: Personal Development, Product Management

Close Project

Toastmasters

Public Speaking, Facilication, Leadership.

Public speaking was never my forte. There were many times when I had to do a presentations but I hated it. At the same time I had the biggest respect for those who are amazing at public speaking and was envious of them. My long-term goal of transition to management asks for outstanding leadership, communication and facilitation skills. It was time to conquer my fears and take action to improve my weakness. Toastmasters is a great club to join to work on public speaking, meeting facilitation and listening/evaluation skills. Every week I participate in Toastmaster meetings to play a role such as Toastmaster (MC), Tabletopic master, Speech Evaluator, Timer, Grammarian and participate in impromptu speeches. On 1-2 months basis I do a prepared speech by following the Competent Communicator (CC) track, thus far completing 5 projects.
I began to video record my performances after 6 months as a way to break through the plateau and face my own shortcomings. Currently, I am using these videos as a way of blogging my journey of development. Check it out here:
Yuma Tsuboi's Toastmasters Blog
And while you're here, check out my latest, CC5: Your body speaks!

• Date: Mar 2017 - Present
• Category: Public Speaking, Facilitation, Personal Development

Close Project

Emotion Recognition from Vocal Intonation

AI that understands how you feel.

The motivation behind this social robotics research is its value in assisting the aging population with dementia by providing social interaction. Handing the elderly with a robot for social engagement may still be hard to imagine. But the truth is that the number of social workers is declining as the population ages, and we are anticipating insufficiency in providing professional social interaction, which is an essential aspect of dementia treatment. Simply put, there will not be enough working population when the baby-boomer generation enters the age prone to dementia.

My research focused on enhancing the quality of interaction between a human and a robot. Interacting with robots today still feels artificial and unengaging. One way to make robot interaction less painful is for robots to have awareness of the other’s affect state. There are few ways us as humans execute emotional intelligence: facial expression, body gesture and vocal intonation. While there has been many research done to decipher facial expression, body gesture and vocal intonation has been a tricky subject. My thesis focuses on predicting emotion from vocal intonation. The overall research experience was a highly iterative process with a lot of bouncing back and forth on ideas, but it can be summarized in the following 3 steps: Obtaining Speech-to-Emotion data, Signal processing to obtain vocal intonation features, and training a Emotion-to-Vocal-Intonation prediction model. In the final stage, the resultant Vocal-Intonation-to-Emotion prediction model was combined with Emotion-to-Body-Gesture model developed by another Master student from the lab.

Speech-to-Emotion Data Collection There were several online database that maps speech recordings to emotions. After analyzing them, we were not satisfied the choice of language, age variance, and the 5 emotion classes (we want to classify 9 different emotions!). By scoping the target speakers to those of a certain age group, we can anticipate higher chance of building a successful classifier. Adding variances to model the real world should come afterwards. We recorded speeches with acted emotions by lab mates without any history of acting, and assumed that it represents how the average young adult conveys emotion.

Training Data Curation Now with the database of speeches labeled with corresponding emotions, the creation of training data was commenced. The main task for this step is to pick the best set of voice signal features that is a good indicator of emotion. In short, “what about the voice signal carries emotion info?” And this process is experimental, an iterative process of making an educated guess from signal processing perspective and putting it through various machine learning algorithms to see which gives the highest accuracy. After many, many trial and literature review, the best feature set, or feature vector, was made of 51 features including common attributes like amplitude and natural frequency to advanced signal processing attributes like mel-frequency cepstral coefficients! Using Matlab and Python, the recorded speeches were segmented and features were extracted, resulting in a training data of 994 feature vectors with length of 51!

Machine Learning! This is the validation stage of the selected feature set. I used Python to format the training data to something acceptable by Weka, the machine learning tool used for this research. Cross validation on the prediction model derived with neural network on the 994 vectors of 51 features yielded 82% accuracy in classifying a given speech to one of the 9 emotions. This model was used to build a real-time emotion classification system with Python and Weka, ready to be tested with human interaction!

BONUS: Multimodal Experimentation The Real-time Emotion Prediction via Vocal Intonation and via Body Gesture was integrated with additional machine learning work to answer the question of, “which is a better indicator of emotion in what situation?” It could be integrated with other modal of emotion, such as facial expression but this was out of our scope. Our integrated work, now the “AI that can detect emotion from vocal intonation and body gesture” in an interaction, was installed on a NAO robot developed by Softbank Robotics. The NAO robot was given dialogues to interact with people and the quality of the response was analyzed. The responses included consideration for the emotions detected, thus increasing the EQ of the robot!

• Date: Aug 2015 - Nov 2016
• Category: Social Robotics, Human-Robot Interaction, Machine Learning, Software Engineering

Close Project

Breathing Instructor app

Anxiety is inevitable.

The awareness for mindfulness was just on the rise and I was coming off fresh from working with clinicians from Centre for Addiction and Mental Health (CAMH) for my research project, “Effectiveness of Mobile Application in Smoking Cessation”, where I developed an Android app with professional insights on quitting smoking. For the Bachelor Thesis of my final year in undergrad, I worked with Professor Jonathan Rose of UofT to develop a mobile application to deliver guided breathing exercise and analyze its effectiveness in combating anxiety. Here are some of the engineering problems overcome:

Catching your breath: Signal processing The initial idea was to infer the breathing pattern of inhalation and exhalation based on the sound level captured with the smartphone’s microphone. Large sound level would be caused by breathing on the microphone, hence indicating inhalation and small sound level would be caused by no breath on the microphone, hence indicating exhalation. With a simple threshold calibration, I can easily classify the inhalation/exhalation of the app user! The truth couldn’t be any further from this, and the actual sound level I saw was noisy and jumping between “inhale” to “exhale” every couple milliseconds. After implementing signal processing technique to remove the noise and dictating the perfect distance to perform the breathing exercise from, successful classification was accomplished. But then a crucial problem arose: The phone had to be held in user’s face and no longer ergonomic to breath in. A headphone with microphone attachment proved to be a reasonable solution to this, being able to capture the breathing sound perfectly with the phone still in the user’s hand in a natural stance.

Guided breathing: UX design Learning Android development to build a functioning Android app was one thing, but I learned that making a nice-looking app that people would actually use was another animal. Beyond the general look of the app such as fonts and colors, the guidance of breathing had to be delivered visually and there was no way around creating some sort of animation. I implemented 4 user interfaces versions: sound level graphs of “target” and “user” (too science-y), flappy bird controlled with breathing, animation of inflating/deflating balloon with “target” outline, animation of inflating/deflating lung of a yoga instructor.

User testing: Survey The challenge of this project is that its impact was always going to be subjective. The two animation-based user interface was put under user-testing for final verdict. With the survey carefully designed to eliminate sample variances, 20 people were asked to try the exercise and answer questionnaires before and after the exercise. Here is an example questionnaire:

The verdict was that the balloon animation provided a more intuitive guidance to breathing and that users felt more calm after performing the breathing exercise!

• Date: April 2015
• Category: Android Development, UX, User testing

Close Project

Pipe Inspector Robot

Building a fully autonomous robot from scratch.

Oh, the memories of Engsci. If there’s one course that could sum up my undergraduate program, it would be the (infamous) AER201: Engineering Design.

In a team of three, we built a fully-autonomous robot from scratch as a pilot solution to inspect a industrial pipe for potential radioactive leakage. For testing, a white 10m-long pipe with circular cross-sectional area of 50cm diameter marked with black spots that represents problematic leakage was used. The pipe was elevated about 15 cm from the ground by two pillars and the ask was to perform the inspection with the body of the robot positioned on one side of the pipe throughout the process. The robot must also start by locating the pipe and returning to the starting position upon inspection completion and report the total number of markers found and their respective positions along the length and circumference of the cross-section.

As the circuit specialist of the team, I fabricated all the electrical circuits including the marker sensor, pillar sensor, motor control, signal conditioner, and the motherboard with microcontroller that connects the sensor to the actuators. This was integrated with the microcontroller programmed by the programmer member and the chassis built by the mechanic member. It was during the integration of these subcomponents when unexpected challenges arose and consequently turned me into a believer in ALWAYS allocating ample time for system integration.

Our complete robot features a main body that holds the input keypad, microcontroller and the circuitry hub supported by 2 motorized wheels and a following non-motorized tail wheel. There is an arm extending to its right side which suspends the circular ring of marker sensors. Here are some of the challenges and the corresponding design decisions, which fundamentally made our robot, Jackattack, unique from others.

Moving straight: Follow the pipe! Getting the robot to move in a straight line at first seems like an easy task, but it’s not. The range of motors available forced us to use two actuators, whose torque had to be sync’ed in terms of voltages energy sent and calibrated to manufacturing variances. This problem was compounded by the consideration of different surface types and highly asymmetrical chassis that resulted in uneven distribution of the weight and ultimately uneven friction made by the wheels. Our solution was to use a distance sensor and precision with pulse-width modulation (or PWM, a way to send power signal to actuators) to maintain certain space between the robot and pipe of interest to not just move straight, but to follow the pipe!

Overcoming the pillars: Springs! The ability to move along the pipe is great and all, but on the way, there are obstacles in the form of supporting pillars. Inspired by the idea of doors that closes itself, we made the bottom portion of the sensing ring a flap that can open and close by hitting the pillars by using a spring! Hydraulic-based springs would have decreased the compressive stress to the rest of the body and would’ve been a more faithful duplication of the damping effect in self-closing doors, but we had underestimated calibration at that point :)

Turning exactly 180 degrees: Felt strip! We finally made it through the pipe. Now we gotta go back past the starting line! Simply reversing with the pipe in the sensor ring and the tail wheel was infeasible. So we needed turn 180 degrees and trust the ability to drive somewhat straight. Turning 180 degrees consistently without any feedback signal was almost impossible. The tail wheel created highly variant friction that nullified all the calibration effort put into the motorized front wheels. And then an idea sparked: felt strips on the following tail wheel! The texture of felt strips made the variant friction negligible, which led to consistent amount of friction which we can calibrate for.

Markers in the shadow: Lights! Counting the markers wasn’t an easy task either. Particularly, when marker sensing seemed to be working, we noticed that it wasn’t reading the markers on the bottom along the pipe. We realized that it was poorly lit on the bottom and special calibration was need for the bottom sensors. But because the bottom sensors were on a moving component (i.e. the springed door), increasing sensitivity would cause false positives when going through the pillar. To avoid another calibration-as-a-solution, we coupled the light sensor with an LED. By bringing light to the shadow, we normalized the marker-to-pipe color ratio and the light sensors were able to pick up the bottom markers reliably.

Not enough microcontroller I/O pins: Code with circuits! At this point, there are 20 sensors and two actuators and one keypad for the microcontroller to manage, in addition to powering them! Luckily, we initially overestimated the number of components to be used and realized the insufficient number of I/O pins to connect them. We moved a lot of microcontroller logic to circuit level in order to decrease the number of signals required by microcontroller. For example, instead of sending 16 bits of signals from the 16 markers sensors on the sensor ring, the 16 signals were put through custom 16-to-4-bit binary encoder and series of OR gates, for signaling “marker found” interrupt and circumferential, we were able to cut down the required number of input pins from 16 to 5. Solving these logics at hardware level led to faster computation.

Find more on the AER201 website!

• Date: April 2012
• Category: Engineering Design, Robots, Circuits, Microcontroller

Close Project