# Haptic Grasp Dataset — first_success_run

In-hand haptic recordings of an **Allegro Hand (right)** covered with a **XELA uSkin
tactile skin** grasping YCB objects, mounted on a Franka Emika Panda. The dataset is
designed for **object recognition from haptic data alone**: each run provides the full
368-taxel, tri-axial skin stream at ~100 Hz together with hand proprioception, grasp-phase
annotations, and a time-aligned camera preview for inspection.

## Sessions & runs

Two recording sessions at Sorbonne University (ISIR), one grasp per object per session:

| Session | Date | Objects | Notes |
|---|---|---|---|
| `first_success_run` (s1) | 2024-04-25 | YCB 001, 002, 004, 005, 007 | reference-quality skin data (~1.8 % dropouts) |
| `session_2` (s2) | 2024-04-26 | YCB 001–009 | skin verified responsive on every run; runs 005/006 have one partial finger outage (30 dead taxels in the index-finger range, ~10 % dropout) |

14 runs total; objects YCB 001, 002, 004, 005, 007 have two independent grasps (different
days); objects 003, 006, 008, 009 one. The s1 tuna fish can (007) was recorded under a
`003_cracker_box` filename; its `object`/`ycb_id` were corrected in this release (the run's
`source_bag` attribute keeps the original bag name).
Object 010 (potted meat can) has no complete grasp cycle in any recording — the session
was interrupted during its handover. Every run passes an automated QC check
(`scripts/verify_release.py`): monotonic timestamps, expected shapes, video/haptic
duration match, and grasp response ≫ open-hand rest (65–135 vs 6–13 counts).

## Protocol

Each run follows the same scripted cycle (`record.py`), annotated live on `/string_state`:

1. **Handover** — an experimenter places the object into the open hand (test close/open).
2. **`start` marker**, then ~7 s of **open, unloaded hand** (used as the taxel baseline).
3. **Grasp** — the `four_finger` synergy closes on the object.
4. In-hand **arm poses** (object held throughout): `wrist_up` → `palm_up` → `top` in
   session 1; `wrist_up` → `thumbs_up` → `top` in session 2.
5. Hand opens (object released / removed).

## Sensor

XELA uSkin Allegro right-hand kit (model `XRALLEGRORIGHTCP`): **368 tri-axial magnetic
taxels** on 18 pads — 4 curved fingertips (30 taxels each), 11 phalanx pads (4×4),
3 palm pads (4×6). Each taxel reports x/y shear and z normal force as raw counts with a
resting level near 2¹⁵ ≈ 32 768 per axis; contact appears as a deviation from that level.
Taxels are indexed 0–367 in XELA server stream order (per-pad mapping is not part of the
server output). Values are uncalibrated counts, not newtons.

### Taxel → pad mapping (estimated)

`taxel_layout.json` maps stream indices to hand pads: index finger `[0,78)`, middle
`[78,156)`, ring `[156,234)` (each tip 30 + 3 phalanx pads, distal→proximal), thumb
`[234,296)` (tip + 2 phalanx pads), palm `[296,368)` (3 × 24). This mapping is **inferred**,
not vendor-documented: the pad sizes and per-finger structure follow the kit composition,
and the arrangement was selected by taxel-correlation block structure plus grasp-phase
evidence (palm-specific loading during the `palm_up` pose falls in `[296,368)`). Treat
finger identities and within-pad taxel order as a best estimate; corrections only require
editing `taxel_layout.json` and the `PAD_LAYOUT` block in `viewer/index.html`.

## Files per run

```
first_success_run/<run>/
  data.h5           full-rate data (see schema below)
  preview.mp4       640×360 @ 10 fps camera preview, time-aligned
  viewer_data.json  downsampled metadata for the HTML viewer
  viewer_haptic.bin int16-quantized taxel deviations for the viewer (20 Hz)
```

### HDF5 schema (`data.h5`)

| Dataset | Shape / dtype | Description |
|---|---|---|
| `haptic/t` | (T,) f64 | seconds since bag start (~100 Hz) |
| `haptic/values` | (T, 368, 3) f32 | raw counts, axes x/y/z, XELA taxel order |
| `haptic/valid` | (T, 368) bool | False where the reading was a transmission dropout (repaired by forward-fill) |
| `haptic/baseline` | (368, 3) f32 | per-taxel open-hand resting level (median over the rest window after `start`) |
| `hand/t`, `hand/position` / `velocity` / `effort` | (H,), (H, 16) f32 | Allegro joint states (~300 Hz), names in `hand/joint_names` |
| `annotations/t`, `annotations/label` | (A,) | protocol phase markers from `/string_state` |
| `camera/frame_t` | (F,) f64 | capture time of each `preview.mp4` frame |

Run metadata (object, YCB id, synergy, sensor model, provenance notes) is stored in the
file's root attributes.

### Loading example

```python
import h5py, numpy as np

with h5py.File("first_success_run/001_chips_can/data.h5") as f:
    t = f["haptic/t"][:]                       # (T,)
    contact = f["haptic/values"][:] - f["haptic/baseline"][:]  # (T, 368, 3)
    label = f.attrs["object"]                  # "chips_can"
```

`contact` is the recognition-relevant signal; `np.linalg.norm(contact, axis=2)` gives a
per-taxel contact magnitude. See `figures/object_signatures.png` — the mean in-hand taxel
signatures separate all five objects (pairwise cosine similarity 0.31–0.74, with the
highest values between the physically similar cylinder pair and box pair).

## Benchmark baseline (frame-level, grasp-level splits)

`scripts/benchmark_baseline.py` defines the reference protocol: each sample is a single
skin frame (baseline-subtracted 368×3 deviations, restricted to taxels alive in every run
of the protocol), and splits are **always by grasp/session, never by frame** — random
frame splits would leak near-duplicate neighbors and report meaningless ~100 % accuracy.
Results (`benchmark_results.json`):

| Protocol | Train → Test | Classes (chance) | Nearest-class-mean | Logistic regression |
|---|---|---|---|---|
| A | s2 → s1 (different day) | 9 → 5 (11 %) | **49.6 %** | 33.8 % |
| B | s1 → s2 (different day) | 5 → 5 (20 %) | **40.7 %** | 42.3 % |
| C | s2 wrist_up → top (pose transfer) | 9 (11 %) | 39.1 % | **41.3 %** |

Takeaway: single haptic frames carry object identity that transfers across days and hand
poses at 2–4.5× chance from **one** training grasp per object — evidence the modality
works, with ample headroom that more grasps per object would likely close. The simple
prototype classifier beating logistic regression cross-session is a classic
domain-shift signature (day-specific details do not transfer; contact patterns do).

### Pose-transfer matrices (%, NCM / logistic regression)

Every ordered pose pair within a session (same-pose pairs omitted — same-grasp frames
would leak). Caveat: pose pairs share the grasp instance, so this measures robustness to
load redistribution, not new-grasp generalization.

| s1 train \ test | wrist_up | palm_up | top |
|---|---|---|---|
| wrist_up | — | 100 / 100 | 90 / 92 |
| palm_up | 100 / 100 | — | 96 / 94 |
| top | 100 / 84 | 100 / 76 | — |

| s2 train \ test | wrist_up | thumbs_up | top |
|---|---|---|---|
| wrist_up | — | 96 / 96 | 39 / 41 |
| thumbs_up | 81 / 82 | — | 40 / 35 |
| top | 53 / 100 | 59 / 100 | — |

Transfers between upright poses are nearly saturated (the contact configuration
persists within a grasp); transfers **into** the inverted `top` pose are the hard
direction — flipping the hand reverses the load and reshapes the contact image.
Gravity direction is the dominant nuisance factor for frame-level recognition.

## Dataset viewer

Inspect camera footage next to the live skin response:

```
cd release
python3 -m http.server 8123
# open http://localhost:8123/viewer/
```

Pick a run, play the video or scrub the timeline; the taxel map updates in sync
(|Δ| magnitude or per-axis Δx/Δy/Δz). The **Hand** view places every taxel on a stylized
Allegro right hand (palm view, estimated pad mapping); the **Strip** view shows the raw
stream order. Deep-link with `?run=003&t=30`.

## Figures

`figures/<run>_overview.png` — whole-skin contact magnitude with phase bands plus a
taxel-vs-time heatmap. `figures/<run>_hand.png` — per-phase mean contact rendered on the
hand layout. `figures/object_signatures.png` — per-object mean taxel signatures and their
cosine-similarity matrix.

## Provenance & known issues

- Session 1 recordings were recovered from interrupted rosbag files (`.bag.active`,
  reindexed). The recorder's kill signal did not stop the previous recording, so each raw
  bag also contained the beginning of the *next* objects' trials; runs here are trimmed
  to the first grasp cycle, which corresponds to the labeled object.
- ~2 % of taxel readings arrived as (0,0,0) transmission dropouts. They are repaired by
  forward-filling the last valid sample; `haptic/valid` marks the original readings, so
  the raw stream is exactly recoverable. In `session_2` runs 005/006, 30 taxels (all
  within the index-finger index range) were offline for the whole run and remain zero.
- Two further passes exist in the raw data but are excluded: one with no skin messages
  at all, and one with an unstable sensor (whole-taxel outages for seconds at a time).
- The uSkin values are uncalibrated counts; no force calibration was performed.
- Faces in the preview videos are pixelated (YuNet detection with temporal persistence
  and generous margins; `scripts/blur_faces.py`). Unblurred videos can be regenerated
  from the raw bags with `scripts/extract_run.py` if ever needed.
- Extraction pipeline: `scripts/extract_run.py` in the accompanying repository, reusable
  for the remaining recordings of the session (`run_1`, top-level bags).

## License

Released under the Creative Commons Attribution 4.0 International License
(CC BY 4.0) — see `LICENSE.txt`. Please cite via the Zenodo DOI (added on
publication) and the accompanying paper where applicable.
