Update: You can follow the discussion of this article around the web on HN, Adafruit, Hackaday, and Hackster.io!

This article explores the research and development journey behind my new sensor system, OptiGap, a key component of my PhD research. I’m writing this in a storytelling format to offer insights into my decision-making process and the evolution leading to the final implementation. It should hopefully provide a glimpse into the sometimes-shrouded world of PhD research and may appeal to those curious about the process. For a deeper dive into technical specifics, simulations, and existing research on this subject, my dissertation is available online here.

What does it do?

In very general terms, this sensor is basically a rope that if bent can tell you where along its length you bent it. The fancy term for that is “bend localization.”

OptiGap’s application is mainly within the realm of soft robotics, which typically involves compliant (or ‘squishy’) systems, where the use of traditional sensors is often not practical. The name OptiGap, a fusion of “optical” and “gap,” reflects its core principle of utilizing air gaps within flexible optical light pipes to generate coded patterns essential for bend localization.

How the OptiGap Sensor System Started

The idea for OptiGap came about while I was experimenting with light transmission through various light pipes (optical cables) for use as a bend detection sensor. I was initially trying to see how I could effectively “slow down” light through the fiber…a seemingly straightforward task, right?

During this process, I attached a section of clear 3D printer filament (1.75mm TPU) to a piece of tape measure for an experiment and incidentally discovered that when I bent the tape measure (and filament) at the spot where the electrical tape was attached, there was a significant drop in light transmission. I hypothesized that this was because the sticky residue of the electrical tape was causing the filament to stretch, which in turn reduced the light transmission.

To verify this hypothesis, I attached a longer piece of TPU to a tape measure and began bending it at various points to observe how light transmission would change.

I wrote a small Linux I2C driver for the VL53L0X ToF sensor to run on a Raspberry Pi and push the data to a socket using ZeroMQ. I then created a rough GUI in Python to pull the sensor data from the socket and visualize the light transmission data in realtime, shown in the GIF below, which very quickly validated my hypothesis. This validation marked the “Eureka!” moment that sparked the eventual development of the OptiGap sensor.

The OptiGap Realization

I realized that since I could control where the light was being attenuated, I could use this to encode information about the position of the bend on the sensor. Using electrical tape was not a practical solution, so I started looking for a more reliable and consistent way to create these attenuations. This led me to the idea of cutting the filament and then reattaching it together using a flexible rubber (silicone) sleeve, leaving a small air gap, as shown in the image below.

The main working principle of the air gap is that translation and/or rotation of one light pipe face relative to the other changes the fraction of light transmitted across the gap. The greater the bend angle, the more light escapes across the gap. The resulting change in intensity of the optical signal can then be correlated with known patterns for use as a sensor.

The Big Idea

I then proceeded to test this idea by creating multiple air gaps in a row and bending the filament to measure the attenuation.

As depicted in the GIF below, the optical intensity decreases at each air gap, with a more noticeable decrease as the bend angle increases. This initial experimentation served as proof of concept, demonstrating the feasibility of the idea. It led to the formulation of my final hypothesis of utilizing a pattern of these air gaps to encode information regarding the sensor’s bending and employing a naive Bayes classifier on a microcontroller to decode the bend location.

This concept resembles the functionality of a linear encoder. Linear encoders gauge an object’s linear movement, typically comprising a slider rail with a coded scale akin to a measuring ruler and a sensing head that moves across this scale to read it. Linear (absolute) encoders emit a distinct code at each position, ensuring consistent identification of displacement.

The OptiGap system, functioning like an absolute encoder, would encode absolute positions using patterns of bend-sensitive air gaps along parallel light pipes, effectively serving as a singular fiber optic sensor.

Encoding the Bend Location using Inverse Gray Code

Absolute encoders commonly employ Gray code, a binary system where two successive values differ in only one bit. This property allows for various applications, including error checking. However, Gray code isn’t optimal for the OptiGap sensor system. Here, we aim for consecutive values to differ by the maximum number of bits to facilitate easier differentiation. This necessity gave rise to Inverse Gray code.

Inverse Gray code is a binary code where two successive values differ by the maximum (n-1) number of bits. To implement this, I simply create cuts in the filament wherever there’s a “1” in the Inverse Gray code sequence. This approach can scale to any bit number. For the prototype, I utilized 3 bits, providing 8 possible positions.

Visualization of the OptiGap Sensor System

The illustration below depicts the signal patterns of the OptiGap sensor system for each bend position using three fibers. By employing a naive Bayes classifier, the sensor system can discern bend positions based on signal patterns. The third graph represents actual sensor data from the prototype system, utilized for training the classifier on the microcontroller.

The OptiGap Prototype

I proceeded to construct a prototype of the OptiGap sensor system, utilizing 3 strands of clear TPU 3D printer filament, each featuring a distinct pattern of air gaps. The image below showcases the filament just before cutting, with the cut pattern indicated on a piece of tape.

For the prototype, I employed a commercial 3:1 fiber optic coupler to merge the light from the 3 strands into a single fiber optic cable, resulting in the completion of the sensor prototype, as depicted below.

This marked the final phase of validating the hypothesis and operational theory behind the OptiGap sensor.

Reducing the Physical Size

The initial prototype proved to be large and bulky, primarily due to the size of the 3D printer filament used. Drawing from previous experience, I recognized that PMMA (plastic) optical fiber offered a smaller and more flexible alternative suitable for this application. Consequently, I assessed 500, 750, and 1000 micron unjacketed PMMA optical fibers from Industrial Fiber Optics, Inc. for the sensor strands, resulting in a significant reduction in sensor size.

I conducted tests on all three types of fibers to evaluate their light transmission and flexibility. Among them, the 500 micron fiber emerged as the optimal choice overall, although all three exhibited sufficient flexibility for this application.

Reducing the Optical Transceiver Complexity

I decided to switch from using the complex VL53L0X ToF sensor to a simple photodiode and IR LED setup to reduce the complexity of the system and to increase modularity. This also allowed me to use a  microcontroller to read the sensor data, which was a significant improvement over the initial prototype.

I then created a demo system for the sensor based around an STM32 microcontroller and a photodiode/IR LED setup.

Realtime Machine Learning on a Microcontroller

The final stage in developing the OptiGap sensor system involved integrating a naive Bayes classifier onto the STM32 microcontroller to decode the bend location from the sensor data. I opted for a naive Bayes classifier due to its efficiency compared to if-statements or lookup tables, its capability to handle new or previously unseen data, and its potential for increased accuracy by considering relationships between multiple input variables.

Implementing the naive Bayes classifier proved to be relatively straightforward. This classifier is a probabilistic model based on applying Bayes’ theorem to determine how a measurement can be assigned to a particular class, with the class representing the bend location in this context. I utilized the Arm CMSIS-DSP library for the classifier implementation.

Fitting the Sensor Data

The initial step in integrating the classifier was to fit the sensor data to a Gaussian distribution for each air gap pattern. To expedite this process, I developed a Python GUI for rapid labeling and fitting of the data using GNB (Gaussian Naive Bayes) from the scikit-learn library.

I later improved this UI to be more general and to allow for more complex data fitting.

The probabilities for each class were computed and saved as a header for use on the microcontroller.

Filtering the Sensor Data

To enhance the accuracy of the classifier, I implemented a two-stage filtering process on the STM32 . The initial stage involved a basic moving average filter, followed by a Kalman filter in the second stage.

The OptiGap Sensor System Demo

The GIFs provided below illustrate various stages of the OptiGap sensor system, encompassing assembly and the operational demonstration of the final sensor system.

System Overview

Assembly of an OptiGap Sensor using TPU Filament

Attenuation of Light through the OptiGap Sensor

Fitting of the Sensor Data

Segment Classification using PMMA Optical Fiber

Segment Classification using TPU Filament

Underwater Operation

OptiGap Design Specifications

Key Properties & Parameters

Material Recommendations

Next Steps

I’ve made significant progress on the OptiGap system beyond what’s documented here, including its integration into another modular actuation and sensing system I developed called EneGate.

This has involved custom PCB design and systems integration, detailed in my dissertation. Additionally, I’ve prototyped miniature PCB versions of the optics to interface with the PCBs for the EneGate system.

I’ve also validated OptiGap on a real-world soft robotic system, with full details set to be presented in an upcoming RoboSoft paper titled “Embedded Optical Waveguide Sensors for Dynamic Behavior Monitoring in Twisted-Beam Structures.“

Commercialization

There’s an ongoing commercialization aspect to this research as well. Feel free to reach out if you’re interested in further details.

That’s it for now!

I don’t want to make this too long so I’ll end here. I hope this provided some insight into the research and development process involved in something like this. If you have any questions or would like to learn more, don’t hesitate to contact me!

The post R&D Case Study: Developing the OptiGap Sensor System appeared first on Paul Bupe Jr, PhD.

Over the past couple of years we’ve been working on a rather cool new class of robots called Soft Curved Reconfigurable Anisotropic Mechanism(s), or SCRAM, under this National Science Foundation EFRI project. For my latest paper, Electronically Reconfigurable Virtual Joints by Shape Memory Alloy-Induced Buckling of Curved Sheets, I spent a lot of time working with shape memory alloy (SMA) wire while developing a SCRAM device. Much of this time was spent trying to figure out exactly how to solder to SMA wire, specifically a nickel (Ni) – titanium (Ti) alloy called nitinol. My conclusion after all that time? Soldering to SMA wire is really hard!

Visualizing an SMA Wire Solder Joint

Soldering to SMA wire is hard due to the tough oxide layer on the surface. Solder beads off the wire as if trying to solder to glass! This can be seen in the two SEM images below (shout-out to the research facilities at the University of Louisville). I prepared two samples by tying some stranded wire around a piece of SMA wire (with the oxide layer ground off) and a piece of standard hook-up wire (tin-coated copper). I then applied some non-acidic flux followed by a healthy blob of tin/silver solder. Finally, I ground down one surface of each wire in order to perform some cross-sectional imaging.

It’s apparent from the images that there is no bond to the SMA wire (right) while the regular wire (left) has a great bond!

Soldering to SMA wire

The “easy” solution is to use a really aggressive acidic flux to eat away the oxide layer when soldering. The main disadvantage of this is the need to thoroughly wash the joint with water after soldering. Acidic flux will keep eating away long after soldering so it must be completely washed off! Another option is to use an inert gas as shielding while soldering (as in MIG welding) after grinding away the oxide layer. This again is not practical for most applications.

Here are some tips for soldering to SMA wire without acidic flux based on my aforementioned soldering adventures:

• Grind off the oxide layer first to improve wetting.
• Crimping is king. If possible, join wires with a crimp and then apply solder. This produces the most reliable connections.
• “Tie” the wire once around the SMA first to create a mechanical hold then apply solder.
• Use a silver alloy solder to increase the chances of proper wetting.
• Use the lowest temperature possible for the solder since heating up the SMA too high will cause an oxide buildup.

These techniques for soldering to stainless steel actually apply to SMA wire as well so they are worth studying.

If you enjoy research and things like this interest you then take a look at our lab page for some potential opportunities!

The post Soldering to Shape Memory Alloy (SMA) Wire is Really Hard appeared first on Paul Bupe Jr, PhD.

Localization, in the context of robotics, is the process of determining the location of something within an environment. This article serves as a basic introduction to indoor localization, covering the commonly used techniques and technologies.

The most well known and widely used method of localization is GPS. Even with its widespread use, GPS is not suitable for all applications because it largely relies on line-of-sight (LOS) communication to GPS satellites. In indoor environments, these signals are greatly attenuated by walls and roofs, meaning that GPS often does not work indoors. Even when signals from at least four satellites are received indoors, the localization accuracy is too low to be useful. In lieu of GPS, a large number of indoor localization techniques and real-time locating systems (RTLS) have been developed. These can be grouped into two broad categories: radio-frequency (RF) based (wireless), and non-RF based (non-wireless).

Non-wireless techniques generally use cameras and sensors like Inertial Measurement Units (IMU) and laser distance finders (like LiDAR) for localization. Cameras can use markers or extract visual features from an environment to perform localization using various techniques including the very common technique known as simultaneous localization and mapping (SLAM). This technique involves building a map of an environment while at the same time localizing in that environment. Data from an IMU and laser scans can also be fused with the visual data to improve accuracy. These techniques are very computationally heavy, require specialized hardware, and are typically used in robotics or autonomous navigation applications (see Tesla, Boston Dynamics). This is the same type of localization technique I used in my last robot and I will cover this in more detail in a separate article.

Wireless techniques utilize technologies such as Wi-Fi, radio-frequency identification (RFID), Bluetooth, and ultra-wideband (UWB). These technologies are useful for indoor localization because they are already found in existing infrastructure and can be set up at very low cost. Wireless localization techniques can be classified as range-based, or range-free.

Ranged-based Indoor Localization

Range-based (or distance-based) techniques are more accurate than range-free and involve measuring the distance from the unknown node being localized to some fixed nodes with known locations, typically called anchors. Most range-based localization algorithms utilize one of the common measurement techniques which fall into two categories: angle-based and distance-based. The most common angle-based measurement technique is Angle of Arrival (AoA) and the main distance measurement techniques are Time of Arrival (ToA), Time Difference of Arrival (TDoA) and Received Signal Strength (RSS).

Angle-based Measurement

AoA measurement techniques calculate the angle (bearing) between the unlocalized node and a fixed anchor with a known location. These measurements are obtained using two main techniques that include (1) measuring the amplitude response of the receiving antenna and (2) measuring the phase response of the receiving antenna. The location of the unknown node is a line having a certain angle from an anchor node — this requires at least two nodes to calculate the position as shown in the figure below.

The accuracy of these measurements is affected by the directivity of the antenna and the environmental NLOS and multipath effects. Since AoA measures angles, it requires direct LOS between the receiver and transmitter because a reflected signal arriving at the receiver can be interpreted as coming from a completely different direction, which can result in very large errors in the measurement.

Distance-based Measurement

Time of Arrival

Time of Arrival (ToA) is a technique that calculates distance based on the measured time of arrival of a signal from a transmitting node to a receiving one. This is more formally referred to as a one way propagation time measurement. The primary drawback to this technique is that it requires perfect time synchronization between the clocks of the transmitter and receiver nodes; any difference between the two clocks can become a large error in the distance calculation. Assuming normal conditions (air as the medium and radio waves traveling at the speed of light) a small clock synchronization error of 1ns will relate to a distance measurement error of 0.3m.

One way of overcoming this issue is by measuring the round trip propagation time instead of the one way propagation. The first node sends a signal to a second node which in turn immediately sends that signal back to the first node and the distance is calculated using the round trip time. This removes the need for the transmitter and receiver to have synchronized clocks. The primary issue with this round trip method is the processing delay accrued from the second node receiving the signal then sending it back in turn. This delay is usually known and specified by the manufacturer (or during a calibration process) so it can be subtracted from the measurement at the first node.

Time Difference of Arrival

Time Difference of Arrival (TDoA) is another technique that measures propagation time but in this case the difference between the arrival time of a signal at two different fixed receivers is measured. This requires that the location of the two receivers are known and those two receivers also have synchronized clocks. Unlike ToA, there is no need for the clocks between the transmitter and receivers to be in perfect sync.

Received Signal Strength

There are two main methods of estimating distance using the Received Signal Strength (RSS): using the path loss log-normal shadowing model and RSS fingerprinting. Distance estimation using the path loss model is accomplished by measuring the signal attenuation as it propagates from the transmitting node to the receiving node. The relationship between distance and signal attenuation is heavily dependent on channel characteristics and as a result requires a very accurate propagation model in order to have acceptable results. The standard log-normal model used in this technique is as follows:

(1)  

with being the reference power at distance from the transmitter, being the path loss exponent measuring the rate at which the RSS decreases with distance, and being a zero mean Gaussian random variable with standard deviation which accounts for random showing effects.

Multilateration

Multilateration is a core technique for estimating the location of a node using the measured distances to multiple anchors. Traditionally, this is achieved using 3 anchors (for 2D localization) and is referred to as trilateration. The locations of the anchors is assumed to be known and the location of the unknown node is the intersection of the three circles with the center at the location of each anchor and radius equal to the measured distance to the unknown node.

An example of the relationship between the node and anchors is shown in the figure above. In practice the measurements aren’t always accurate so the circles don’t intersect at a single point, as shown below.

In this case with the locations of the anchors and the estimated distances between the anchors and node known, trilateration then becomes a problem of solving three nonlinear circle equations

(2)  

where , , and , are the coordinates of anchors 1, 2, and 3. These equations can be linearized into the form

(3)  

with

(4)  

(5)  

(6)  

which can be solved using the least squares method:

(7)  

Multilateration is not to be confused with triangulation which uses knowledge of the angles between the node and anchors to find the node-to-anchor distances using the law of sines.

Range-Free Indoor Localization

Range-free techniques use the relative positions of existing nodes, connectivity information, or detecting the proximity of the unknown node to fixed anchors with known locations (using RFID or Bluetooth beacons) to localize. They are simpler, cheaper, and more energy efficient than range-based algorithms at the cost of having low localization accuracy. Because of this, range-free algorithms are generally only useful when coarse locations are desired.

Connectivity-based

Connectivity-based localization works by checking if a node is connected to another node. If each node is aware of all the connected nodes then coarse locations can be determined by counting the hops between nodes using an algorithm DV-Hop. Another technique is simply detecting the presence of a node near a knwon beacon, typically using RFID or Bluetooth.

Profiling / Fingerprinting

Another type of range free localization is the use of RSS profiling or fingerprinting in order to overcome the inaccuracies of RSS-based distance measurements. Fingerprinting involves taking RSS measurements at various locations and building a map of those measurements and their position. Localization then becomes a matter of matching RSS measurements, not distances, to the map in order to localize. This type of localization is very well suited for machine learning.

Next Steps

This should hopefully have been a quick introduction to the concept of localization and the common techniques currently in use. The next article will include some code samples for implementing some of these techniques using Python.

The content of this article is based off the work I did for my Masters thesis.

The post Indoor Localization – An Introduction appeared first on Paul Bupe Jr, PhD.