All Projects

(Research) Multi-Medium Semantic SLAM  2/2022 - 9/2022

(this work has been submitted to ICRA 2023)

• Sparse environments with repetitive textures and features are an edge case for existing SLAM methods such as ORB-SLAM2

• ORB-SLAM2 drifts by up to 2 meters (20%) on this test!

• We utilize semantic segmentation and YOLO object detector networks to extract higher-level features that are more constant between frames

• As a result of incorporating both high-level and low-level features, we reduce tracking error by up to 70%!

• See full video submitted to ICRA here

Classic and Deep Learning Computer Vision Class 8/2022 - 12/2022

The RBE 549 Computer Vision by Prof Nitin Sanket included 5 hands-on projects spanning a variety of class and deep learning methods:

• Probability-of-Boundary Edge Detector

• Auto Panorama Stitching

• FaceSwap

• Auto Camera Calibration  (solve for camera params)

• NeRF and Structure from Motion    (3d model from 2d images)

• Visual-Inertial Odometry

• Semantic LiDAR Mapping

A few sample visuals are to the right, and I'd be happy to discuss any of these projects further in detail.

Off The Shelf Robotics (Former Startup) 5/2021 - 7/2022

I co-founded OTS Robotics to bring a new, simplified robotic fulfillment system to the online grocery space that exploded during the pandemic (but has since declined).

As CTO, I designed, manufactured, and programmed two robots to perform automated storage and retrieval system of grocery totes.

Our system is unique due to the fast setup time enabled by a vision-based guidance system that relies on vertically-mounted fiducials rather than a floor-based grid.

This video demonstrates our MVP system of robots working together to fully-autonomously retrieve a target tote container from the shelf.

Robotic Picking Using Deep Reinforcement Learning (Class Project): 3/2022 - 5/2022

• Robotics picking of items from bins is of growing importance for warehouses and factories

• In this project, we modified an existing deep RL network to improve bin picking in a simulated PyBullet environment

• Our network took inputs of an overhead RGBD camera, wrist RGBD camera, and the joint positions of the 7-DOF panda robot to produce an output of joint torque commands

Improved Monocular Depth Estimation using Semantic Information (Class Research Project): 3/2022 - 5/2022

• Monocular depth estimation is important in many applications including autonomous vehicles

• Depth estimation partially relies on internalizing object semantics to understand the expected shape & size

• Depth estimates from existing methods such as MonoDepth1 lack crisp boundaries between objects

• We created a multi-task network to simultaneously train on depth and semantics

• This resulted in much-improved object edges being more crisp and clear

Bounding Volume Hierarchy Trees for Minimum Distance Search

• This was an individual project in RBE595 Haptics class

• Bounding Volume Hierarchy Trees are recursively constructed for 3D objects from their respective triangular meshes 

• A real-time (1khz!) minimum distance search can then be performed between two objects using a simultaneous tree-walk algorithm to find the nearest points

•  Visuals were coded using PyQTGraph in Python to display each step of the process and algorithms