With Yann LeCun and Rob Fergus at the Computational and Biological Learning Lab at NYU, at NetScale Technologies in Northern NJ and with other Professors at the Courant Institute at NYU including Leslie Greengard and Paolo Barbano, I have worked on several projects which apply machine-learning techniques to real world data -- from Gigapixel images, to live video streams, to paint samples from the Metropolitan Museum to biological images of worms and mouse dendrites to 3D point-clouds of entire cities and robotics. I have worked in a collaboration between the Computer Science and Economics Departments at NYU and the NY Fed on real-estate transaction data. Most recently I have been working on a DARPA grant where we are seeking applications of Deep-Learning.
DARPA Deep-Learning
Deep-Learning, Gigapixel Photography, Situated video, Action Recognition
@ NYU
A DARPA program where we worked in collaboration with teams in Yoshua Bengio's lab at the University of Montreal and a group under Hans Peter Graf at NEC. We put together two demos to show applications of this exciting new technology.
Gigapixel Images
Type "man with a red hat" or "woman running" this system parses natural language and maps phrases directly to the most relevant image patches in huge gigapixel images which we took from New York City rooftops.
Live 360 Video
Clement's system can parse each image in a 360 degree video stream into 12-30 classes (car,person,streetlight, etc.) in near real-time plus he can offload the heavy computing to his custom architecture on a low power FPGA chip.
Collaborators
Yann LeCun, Rob Fergus, Yoshua Bengio, Urs Muller, Clement Farabet, Pierre Sermanet, Koray Kavukcuoglu, Li Wan, Antoine Bordes, Hugo Larochelle, Aaron Courville, Pascal Lamblin
Real-Estate Price Prediction
Large Datasets, Ensemble Methods, Machine Learning
@ NYU
A project run by Andrew Caplin, Professor of Economics at NYU. We build a price surface which can predict the price of any house at any time. Currently we are only looking at LA County in a project about Mortgage defaults sponsored by the NY Fed
Collaborators
DARPA URGENT
Large Datasets, 3D point-clouds, LIDAR, Sensor Fusion, Machine Learning
I was the main programmer for Net-Scale in the Net-Scale/HRL participation in the DARPA URGENT challenge. We built a stand-alone end-to-end system to which were input raw-point clouds and which produced polygonal classifications of large objects : trees, lawns, buildings, streets. The code was a C library which was linked from HRL's code base.
Collaborators
LAGR Program :: Videos
Autonomous Robotics, Vision, Machine Learning
Pierre Sermanet put together this video of some clips from the last few months which aims to explain most of the elements of the system which were working together in a pretty efficient end to end system
More Info
- Larger videos
- Yann LeCun's LAGR project page
- Raia Hadsell's page on LAGR and her collection of LAGR movies
Collaborators
Yann Lecun, Urs Muller, Pierre Sermanet, Raia Hadsell, Matt Grimes, Ayshe,
Biological Imaging
Biological Imaging, Vision, Machine Learning
@ NYU
Classified spine vs. dendrite. Collaboration initiated by Paolo Barbano with Paul Greengard's Lab at Rockefeller University
Labeled image for worm vs. egg vs. background classification. Collaboration initiated by Paolo Barbano with Fabio Piano's Lab at NYU
Collaborators
Yann Lecun, Paolo Barbano, Leslie Greengard, Fabio Piano
