Jonathan Tompson

Research Scientist
Google Scholar
Email: first dot last at gmail

I am a Senior Research Scientist for Google Brain in Mountain View, CA. My research background coves a wide range of topics: computer vision and graphics, robotics, computational fluid dynamics, reinforcement learning, unsupervised learning, hand and human body tracking and analog IC design.


Counting Out Time: Class Agnostic Video Repetition Counting in the Wild
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, Andrew Zisserman
CVPR 2020 [paper link coming soon!]
We present an end-to-end algorithm for detecting periodic motion in videos, and counting repeated actions in a sequence.

Imitation Learning via Off-Policy Distribution Matching
Ilya Kostrikov, Ofir Nachum, Jonathan Tompson
ICLR 2020 [paper link coming soon!]
Our ValueDice algorithm improves upon DAC, enabling similar performance in an entirely offline setting. This is done via a novel formulation of phrasing distributional matching as value function learning problem.

ADAIL: Adaptive Adversarial Imitation Learning
Yiren Lu, Jonathan Tompson
NeurIPS 19 workshop + Submission [paper]
The ADAIL tackles the problem of adaptive policies learning. We use an explicit latent dynamics encoder to improve policy robustness to unseen dynamics.

An Analysis of Object Representations in Deep Visual Trackers
Jonathan Tompson*, Ross Goroshin*, Debidatta Dwibedi (* equal)
Submission [paper]
In this work we performed an in-depth analysis of a popular tracking architecture and suggested a novel architecture to mitigate saliency biases.

Temporal Cycle-Consistency Learning
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, Andrew Zisserman
CVPR 2019 [paper] [webpage] [video]
A self-supervised representation learning method based on the task of temporal alignment between videos. In addition to robustly solving video alignment, these representations enable few-shot classification of video action phases.

Learning Latent Plans from Play
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, Pierre Sermanet
CORL 2019 [paper] [website]
A novel method for learning hierarchical robotic control policies from unstructured play data.

Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning
Ilya Kostrikov, Kumar Krishna Agrawal , Debidatta Dwibedi, Sergey Levine, Jonathan Tompson
ICLR 2019 [paper] [code]
Presented a SoTA Adversarial Imitation Learning method that utilizes an off-policy variant of the GAIL algorithm, as well as a novel mechanism for handling absorbing states.

PersonLab: Person Pose Estimation and Instance Segmentation
George Papandreou, Tyler Zhu, Liang-Chieh Chen, Spyros Gidaris, Jonathan Tompson, Kevin Murphy
ECCV 2018 [paper]
A box-free bottom-up approach for the tasks of pose estimation and instance segmentation of people in multi-person images using an efficient single-shot model.

Learning Actionable Representations from Visual Observations
Debidatta Dwibedi, Jonathan Tompson, Corey Lynch, Pierre Sermanet
IROS 2018 [paper] [website]
A novel framework for learning robust visual features for training robotic agents. Building upon the TCN framework, we show that our learned representations are as performant as policies trained from true state representations.

Temporal Reasoning in Videos using Convolutional Gated Recurrent Units
Debidatta Dwibedi, Jonathan Tompson, Pierre Sermanet
CVPR 2018 Workshop [paper]
An archiecture for video-based action recognition using a novel latent prediction loss to constrain and improve latent representations.

Discovery of Semantic 3D Keypoints via End-to-end Geometric Reasoning
Supasorn Suwajanakorn. Noah Snavely, Jonathan Tompson, Mohammad Norouzi
NIPS 2018 Oral [paper] [website]
A semi-supervised method to recover semantically consistent 3D keypoints from weakly labeled RGB data.

Learning Robotic Manipulation of Granular Media
Connor Schenck, Jonathan Tompson, Dieter Fox, Sergey Levine
CoRL 2017 [paper] [video]
This paper examines the problem of robotic manipulation of graunular media, where we learn predictive models of granular media dynamics to perform scooping and dumping actions.

Towards Accurate Multi-person Pose Estimation in the Wild
George Papandreou, Tyler Zhu, Nori Kanazawa, Alexander Toshev, Jonathan Tompson, Chris Bregler, Kevin Murphy
CVPR 2017 [paper] [slides]
State-of-the-art RGB human pose on MSCOCO using a 2-stage system for top-down detection.

Accelerating Eulerian Fluid Simulation With Convolutional Networks
Jonathan Tompson, Kristofer Schlachter, Pablo Sprechmann, Ken Perlin
ICML 2017 [paper] [video] [video presentation] [website]
A learning-based system for simulating Navier-Stokes Equations in real-time. We do so by reformulating the standard operator splitting method as an end-to-end network.

Inside-out Hand Tracking
Google Daydream 2016
Lead a project to enable high quality hand-tracking from a head-mounted camera. The ConvNet-based system ran in real time on embedded hardware. The system demonstrated robustness to occlusion and hand-shape variation. More details are unfortunately not yet public.

PhD Thesis: Localization of Humans in Images Using Convolutional Networks
NYU 2015 [thesis] [ppt]
My PhD thesis covers *most* of the human body tracking work I did while at NYU.

Efficient ConvNet-based Marker-less Motion Capture in General Scenes with a Low Number of Cameras
Ahmed Elhayek, Edilson De Aguiar, Arjun Jain, Jonathan Tompson, Leonid Pishchulin, Micha Andriluka, Christoph Bregler, Bernt Schiele, Christian Theobalt
CVPR 2015 [paper] [video] [website]
SoTA motion capture in arbitrary scenes from few cameras.

Efficient Object Localization Using Convolutional Networks
Jonathan Tompson, Ross Goroshin, Arjun Jain, Yann LeCun, Christoph Bregler
CVPR 2015 [paper] [predictions_flic] [predictions_mpii]
A novel cascaded architecture to help overcome the effects of MaxPooling and a modified dropout that works better in the presence of spatially-coherent activations. Achieved SoTA in human body tracking.
Turked MPII images containing one person: [data].

FLIC-plus Dataset
Jonathan Tompson, Arjun Jain, Christoph Bregler, Yann LeCun
NIPS 2014 [website]
Cleaned up an filtered the FLIC Human Pose dataset of Sapp et al. for fairer evaluation and higher quality labels.

Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation
Jonathan Tompson, Arjun Jain, Christoph Bregler, Yann LeCun
NIPS 2014 [paper] [predictions]
Following ICLR 2014 work, we substantially improved the architecture, incorporated the MRF into the ConvNet and significantly outperformed existing SoTA.

MoDeep: A Deep Learning Framework Using Motion Features for Human Pose Estimation
Arjun Jain, Jonathan Tompson, Yann LeCun, Christoph Bregler
ACCV 2014 [paper]
For ambiguous poses with poor image evidence (such as detecting the pose of camouflaged actors), we showed that motion flow features allow us to outperform state-of-the-art techniques.

Unsupervised Feature Learning from Temporal Data
Rostislav Goroshin, Joan Bruna, Jonathan Tompson, Arthur Szlam, David Eigen, Yann LeCun
ACCV 2014 [paper]
A sparse auto-encoder architecture to make use of temporal coherence. This formulation enables pre-training on unlabeled video data (of which there is a massive abundance), to improve ConvNet performance.

Learning Human Pose Estimation Features with Convolutional Networks
Arjun Jain, Jonathan Tompson, Mykhaylo Andriluka, Graham W. Taylor, Christoph Bregler
ICLR 2014 [paper]
It was a new architecture for human pose estimation using a ConvNet + MRF spatial model and it was the first paper to show that a variation of deep learning could outperform existing architectures.

NYU Hand Pose Dataset
Jonathan Tompson, Murphy Stein, Ken Perlin, Yann LeCun
[website] [code]
High quality hand pose dataset released. Was the primary hand pose evaluation dataset for the community for years.

Real-Time Continuous Pose Recovery of Human Hands Using Convolutional Networks
Jonathan Tompson, Murphy Stein, Ken Perlin, Yann LeCun
SIGGRAPH 2014 [paper] [video] [ppt] [code]
A novel method for real-time pose recovery of markerless complex articulable objects from a single depth image. We showed state-of-the-art results for real-time hand tracking.

Distributed Locking Protocol
Working at MongoDB Inc with the server kernel team (under Alberto Lerner): developed a new distributed lease protocol (for the sharding config server) using a heavily modified 2-phase commit with timeout mechanism.

Open-Source Randomized Decision Forests
[video] [code]
Early hand-tracking research: a randomized-decision-forest classifier trained to recognize hand pixels using Microsoft's Kinect. Provides an OSS implementation of Real-Time Human Pose Recognition in Parts from Single Depth Images

Jonathan Tompson, Ken Perlin, Murphy Stein, Charlie Hendee, Xiao Xiao Hiroshi Ishii
SIGGRAPH Realtime Live 2012 [video]
Group project with the MIT Media Lab and NYU Media Research Lab. ARCADE is a system that allows real-time video-based presentations that convey the illusion that presenters are directly manipulating holographic 3D objects with their hands.

Miscellaneous Open-Source Graphics Projects
Over the years I have open sourced implementations of multiple graphics algorithms, including:

IC Reserach and Development at Epoch Microelectronics
Designed a wide range of mixed signal RFICs including:

  • RF Modulators (passive and active).
  • Fractional and integer PLLs.
  • Wideband, low-noise LNAs.
  • Low-noise bandgaps and regulators.
  • LC and XTAL oscillators.
  • ΣΔ modulators.
  • FIR/IIR interpolation and decimation filters.
  • Anti-aliasing filters for data converters.
  • Continuous time, high linearity analog filters for base-band processing.
Worked with multiple telecommunication standards, including:
  • Terrestrial and Cable Television: DVB, ATSC, ISDB
  • Low-power standards: Bluetooth, Zigbee

2.6GHz RF Inductive Power Delivery for Contactless On-Wafer Characterization
Jonathan Tompson, Adam Dolin and Peter Kinget
ICMTS 2008 [paper]
Designed a contactless IC testing mechanism using inductive probing through custom devices.

Mismatch Characterization of Ring-Oscillators
Jonathan Tompson, Peter Kinget
2008 Research Associate
Investigated the matching of on-chip oscillators and compared statistics to theoretical estimates.

High-Speed, Chip-to-Chip Communcation
Jonathan Tompson, Gu-Yeon Wei
2006 Undergraduate Honors thesis
Designed a novel transformer-based communication system for high-speed digital systems.

Open Source Tools
I've written or contributed to many OSS tools over the years. This is a short list of some:

  • jtorch - Torch7 Utility Library for running models in OpenCL / C++
  • jcl - OpenCL Wrapper (to make OpenCL easier)
  • torchzlib - A utility library for zlib compression / decompression of Torch7 tensors
  • matlabnoise - Matlab procedural noise library
  • matlabobj - Matlab obj reader
  • torch7 - I'm a reasonably regular contributor to torch and it's various packages
  • icp - A C++ Iterative Closest Point library (with Matlab interface)
  • ik - A very simple inverse kinematics library (in C++)
  • ModelFit - Off-line fitting portion of the hand-tracking paper below
  • jzmq - A ZeroMQ Utility Library (C++)
  • There are probably others... See my github.