TouchFormer is a self-supervised Vision Transformer based foundation model designed for accurate 3D human pose and shape estimation from tactile pressure maps. Unlike vision-centric models that rely on optical sensors, TouchFormer processes non-optical physical pressure grids (64×27 force-resistive matrices from smart mattresses) to understand human body contact and pose.
This project bridges the gap between computer vision's foundation models success and the underexplored tactile perception domain, enabling contact-based human understanding for applications in healthcare, smart furniture, and embodied AI.
- Tactile perception has historically been restricted to task-specific models
- Pressure maps from contact sensors contain rich information about body shape, weight distribution, and pose
- Traditional CNNs lack the ability to capture complex, non-local pressure distributions
- No existing foundation models for tactile understanding comparable to vision transformers in CV
TouchFormer leverages:
- Self-supervised pretraining (MAE) with 75% masking to learn robust tactile features
- Transfer learning from ImageNet-pretrained ViT-B/16 for superior downstream performance
- Vision Transformers to capture long-range dependencies in pressure distributions
- Direct 3D pose regression from 2D pressure matrices (64×27 → 24 keypoints × 3D)
| Model Configuration | Backbone Init | Pretraining | MPJPE (cm) ↓ | PCK @ 15cm (%) ↑ |
|---|---|---|---|---|
| CNN Baseline (Clever et al., CVPR 2020) | Random | None | 11.18 | — |
| TouchFormer (Custom MAE - Exp B) | Random | MAE (Pressure Maps) | 6.91 | 90.8% |
| TouchFormer (ImageNet Transfer - Exp A) - SOTA | ImageNet-1k | None | 6.29 | 92.8% |
Key Finding: Transfer learning from ImageNet-pretrained weights outperforms task-specific self-supervised pretraining, achieving 43.7% error reduction over CNN baselines.
Phase 1: Self-Supervised MAE Pretraining
- Input: Raw 64×27 tactile pressure map
- Mask 75% of patches
- ViT-B/16 encoder processes visible patches only
- ViT decoder reconstructs all patches + masked regions
- Loss: MSE reconstruction on pressure values
Phase 2: Downstream Fine-Tuning & Inference
- Load pretrained ViT-B/16 encoder (from MAE or ImageNet)
- Add linear keypoint head (72-dim output: 24 keypoints × 3D)
- Fine-tune on PressurePose dataset
- Predict 3D skeleton directly from full pressure map
- Project Page: touchformer.onrender.com
- Backbone: Google Vision Transformer (ViT-B/16) - 12 layers, 768 hidden dims
- Input Patch Size: 16×16 patches on 64×27 pressure maps
- Pretraining Objective: Masked Autoencoder (MAE) with 75% masking ratio
- Fine-tuning Head: Linear regression head predicting 24 × 3D keypoints (72-dim output)
- Initialization Strategy: ImageNet-1k pretrained weights outperform random init
- PressurePose Corpus: 10GB of aligned pressure maps + 3D pose annotations
- Sensor Hardware: 64×27 force-resistive pressure matrices (smart mattress setup)
- Tasks: 3D human pose estimation, shape estimation, contact understanding
- MPJPE (Mean Per Joint Position Error): Distance error in cm (lower is better)
- PCK (Percentage of Correct Keypoints): % keypoints within 15cm threshold (higher is better)
Initializing with ImageNet-1k ViT-B/16 weights yields best SOTA results (6.29 cm MPJPE, 92.8% PCK), demonstrating that visual feature abstractions transfer effectively to tactile perception.
Custom MAE pretraining on pressure maps (20 epochs) achieves competitive results (6.91 cm MPJPE), showing that tactile-specific priors can be learned without expensive labeled data.
Despite academic compute constraints:
- Custom MAE model trained for only 20 epochs (not converged)
- ImageNet transfer learning requires zero additional pretraining
- Superior performance with minimal domain-specific training cost
| Component | Details |
|---|---|
| Frontend | HTML5, CSS3, vanilla JavaScript |
| Styling | Custom design with warm academic aesthetic (CSS Variables) |
| Fonts | Inter (sans-serif), Lora (serif) from Google Fonts |
| Architecture | SVG diagrams, responsive tables, feature cards |
| Hosting | Render (project page) |
- Sparsh (Meta AI, 2024): Optical tactile sensing; TouchFormer handles non-optical pressure grids
- Clever et al. (CVPR 2020): CNN baseline for pressure-based pose; TouchFormer achieves 43.7% error reduction
- Vision Transformers (ViT - Dosovitskiy et al., 2021): ImageNet pretraining; TouchFormer adapts ViT for tactile domain
- First foundation model for tactile understanding from non-optical pressure matrices
- Transfer learning analysis showing ImageNet pretrained ViT > custom tactile MAE
- Efficient tactile pretraining with 75% masking on 64×27 pressure grids
- Multi-task capable model for pose, shape, and contact understanding
If you use TouchFormer in your research, please cite:
@inproceedings{clever2020bodies,
title={Bodies at Rest: 3D Human Shape and Pose Estimation from a Pressure Image using Synthetic Data},
author={Clever, Henry M and Erickson, Zackory and Kapusta, Ariel and Turk, Greg and Liu, C Karen and Kemp, Charles C},
booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2020}
}- Project Page: touchformer.onrender.com — Full technical report, architecture diagrams, and evaluation results
- Code Repository: github.com/extremecoder-rgb/TouchGPT — Training code and model weights
- Baseline Paper: arXiv:2004.01166 — Clever et al. CNN baseline
Subhranil Mondal
Independent Researcher | Computer Vision & Tactile Perception
This project is open source. See repository for license details.
Built with:
- Google Vision Transformer (ViT-B/16) backbone
- ImageNet-1k pretrained weights
- Masked Autoencoder (MAE) pretraining methodology
- Inspired by Sparsh (Meta AI) and Clever et al. (CVPR 2020)
Hosted on Render.
Last Updated: July 2026
Repository: github.com/extremecoder-rgb/TouchFormer