Force Curve Inference Pipeline

Predicting per-stroke force curves from video-only biomechanics. Two synchronized data streams converge during training; inference uses video alone. Click any stage to inspect data contracts.

Giacomo Cappelletto · giacomo.cappelletto@icloud.com
Boston, MA · Treviso, Italy ·

01 Data Sources
Video Capture ▶
Monocular rowing video → Sports2D / MotionBERT pose estimation

The sports2d_app pipeline processes monocular video through MotionBERT 3D lifting to extract H3.6M skeleton landmarks and compute joint kinematics frame-by-frame.

Sports2D pose overlay on erg video
Sports2D pose overlay: smooth keypoint trajectories under self-occlusion
Tracked handle and machine reference
Handle proxy (hand midpoint) and machine reference tracking
angles_h36m.csv
 frame_idx, time_s
 left/right_knee_deg, hip_deg, elbow_deg
 trunk_vs_horizontal_deg, spine_flexion_deg
 head_vs_trunk_deg

stroke_signal.csv
 frame_idx, time_s, handle_cx/cy_px
 relative_axis_px, velocity_axis_px_s
 stroke_idx, stroke_phase, is_drive
 is_catch, is_finish
RP3 Force Data ▶
Ergometer force exports → cleaned per-stroke force-at-distance bins

Raw RP3 exports are cleaned by inference_cli.py and expanded by expand_rp3_curve_data.py into spatially-sampled force bins at 2.2 cm intervals along the drive.

Per-stroke RP3 force curve reconstruction
Per-stroke force curves reconstructed from cleaned RP3 data (2.2 cm distance bins)
*-clean.csv (expanded)
 stroke_number, time, stroke_length
 peak_force, peak_force_pos, rel_peak_force_pos
 drive_time, recover_time
 force_at_2.2cm, force_at_4.4cm, … force_at_Ncm
02 Unilateral Standardization
Single-Side Biomechanics Mapping ▶
Camera-near side becomes the canonical limb chain; sign conventions normalized across sessions

Each session declares an active_side (left or right) and a rower_facing (left or right). The canonical mirror policy lives in inference/feature_contract.py and is shared by training and video-only inference: unsigned flexion angles (knee_active_deg, hip_active_deg, elbow_active_deg, spine_flexion_deg, head_vs_trunk_deg) are facing-invariant; only trunk_vs_horizontal_deg is flipped (180−θ) when facing left.

Canonical features (feature_contract.canonical_columns)
 knee_active_deg ← left or right_knee_deg
 hip_active_deg ← left or right_hip_deg
 elbow_active_deg ← left or right_elbow_deg
 trunk_vs_horizontal_deg (facing-flipped)
 spine_flexion_deg, head_vs_trunk_deg (central)
03 Stroke Segmentation
Video Stroke Segmentation ▶
Catch/finish boundary detection → isolate drive portions per stroke

Uses stroke_signal.csv flags: drive starts at catch, ends at finish. Only the drive portion is supervised against RP3 force curves.

Angle traces with stroke events
Joint angles, handle position, and stroke-phase markers from a single video
Per stroke
 catch_frame, finish_frame
 drive_frames[]: frame indices during drive
 feature_curves[]: angle values at drive frames
RP3 Stroke Structure ▶
Each RP3 row = one stroke; force sampled at 2.2 cm spatial bins along the drive

RP3 data is already per-stroke. Force values are sampled at fixed spatial intervals along the drive distance. The stroke length determines how many bins are valid.

Cleaned RP3 force curve data
Cleaned RP3 data: each sample at a fixed 2.2 cm step in stroke length
Per stroke (RP3 row)
 d_k = [2.2, 4.4, 6.6, …] cm
 F(d_k) = force at each distance bin
 stroke_length_cm → valid bin count
QC — stroke detection confidence, tracking completeness, physical plausibility, monotonic progression
04 Session Pairing & Synchronization
Cross-Modal Stroke Alignment ▶
Auto-pair from session registry → cross-correlation coarse anchor → DP fine sync → drift-gated pairing manifest

Auto-pair: inference/pair_session.auto_pair_run resolves active_side, rower_facing, athlete ID, RP3 clean CSV, and (optional) registry-provided anchor from session_registry.csv, so match_rp3_cli.py --auto-pair does not need per-run flags.

Coarse: coarse_sync_anchor cross-correlates video and RP3 cycle-duration sequences and proposes the RP3 row that pairs with a reference video stroke. This is the default anchor resolver when no explicit anchor is given (disable with --no-xcorr-anchor; trust gated by --xcorr-max-cost).

Fine: Dynamic-programming stroke match over catch/finish boundaries with the coarse anchor locked.

Drift gate: --reject-on-drift (default on) hard-fails pairings whose cumulative or mean per-stroke interval error exceeds --drift-reject-max-cum-err-s / --drift-reject-mean-interval-err-s — reject rather than emit silently-misaligned labels.

Alignment evolution
Before (position-max, 0.38s error)
Initial stroke matching with large drift
After (velocity-calibrated, 15ms error)
Calibrated stroke matching with angle overlay
pairing_manifest.json (emitted by match_rp3_cli)
 session_id, athlete_id, date, piece_id
 rower_facing, active_side
 coarse_sync: { offset_strokes, normalized_cost, mean_rate_diff_spm }
 drift: { cum_err_s, mean_interval_err_s, accepted }
 accepted_video_stroke_ranges[]
 qc_summary: { hard_drop_n, soft_flag_counts }
QC — stroke-rate consistency, pairing drift rejection, inter-stroke timing agreement
↳ Video + RP3 streams merge in normalized progress space
05 Drive-Domain Alignment
Normalized Progress Grid   s ∈ [0, 1] ▶
All signals mapped to normalized drive progress — not wall-clock time, not pixels

The master domain is normalized drive progress s. This decouples the representation from frame rate, stroke speed, and physical distance.

RP3 side: s_force = d_k / stroke_length_cm

Video side: s_video from drive phase progression signal.

Video features are interpolated from s_video to s_force grid. No pixel-to-cm calibration is needed in this iteration.

VIDEO (time domain) frame 0 frame k frame N uniform Δt, non-uniform in s RP3 (distance domain) 2.2cm d_k cm L cm uniform Δd = 2.2 cm s = progress(t) s = d / L COMMON GRID: s ∈ [0, 1] 0.0 0.5 1.0 X(s): angles, derivatives Y(s): force (training)
Per stroke, aligned
 s = [0.00, 0.02, 0.04, …, 1.00]
 X(s): kinematic features at each s
 Y(s): force at each s (training only)
06 Feature Engineering
Angle Curves & Derivatives ▶
Smoothed angles → dθ/ds via chain rule → optional d²θ/ds²

Smoothing: Fixed Savitzky-Golay kernel (small odd window, low polynomial order) applied to raw angle trajectories.

Chain rule: dθ/ds = (dθ/dt) / (ds/dt + ε)

Second derivatives included only if ablation shows clear gain. Strokes with ds/dt ≈ 0 or non-physiological spikes are rejected.

The per-stroke feature sequence builder (inference/segment_features.py) is shared by the training-side inference_cli.py and the video-only predict_force_cli.py. The no-RP3 variant build_pose_drive_segments emits features on the bundle's s_grid without needing any force data.

Smooth kinematic traces from Sports2D
High-precision kinematic traces: the raw angle signals before derivative computation
Primary features (6 canonical angles)
 knee(s), hip(s), elbow(s), trunk(s), spine(s), head_vs_trunk(s)
 d_knee/ds, d_hip/ds, d_elbow/ds, d_trunk/ds, d_spine/ds, d_head_vs_trunk/ds
 handle_velocity_px_s, handle_accel_px_s2
 [ablation] d²knee/ds², …
Coordination & Support ▶
Phase lags, angle ratios, handle kinematics; stroke rate & drive ratio as context

Coordination: Phase timing lags (legs → swing, swing → arms) and angle ratio features at key progress points.

Support: Handle velocity/acceleration projected onto the drive axis.

Context: Stroke rate, drive ratio as scalar conditioning variables.

Coordination
 leg_swing_lag, swing_arm_lag
 leg_trunk_ratio(s), trunk_arm_ratio(s)

Support & context
 handle_vel_axis(s), handle_acc_axis(s)
 stroke_rate, drive_ratio
QC Quality Control & Failure Modes
Six per-stroke flags → hard/soft classification → aggregate quality score
QC Gate Pipeline Each stroke passes through six independent checks producing flags and a composite score
STROKE ENTERS QC HARD-DROP GATES 1 qc_tracking_sparse HARD NaN gaps in angle tracking > 30% 2 qc_nonphysio_deriv HARD 600°/s tracking spike exceeds threshold 3 qc_duration_implausible HARD 0.4s plausible 2.0s drive or cycle outside physiological range SOFT FLAGS 4 qc_ds_dt_stall SOFT handle progress stalls mid-drive 5 qc_progress_nonmonotonic SOFT repaired raw >15% of frames needed repair 6 qc_weak_detection SOFT sharp mushy boundary velocity contrast < 15% of peak CLASSIFY FLAGS HARD DROP qc_excluded = True tracking | deriv | duration | alignment PASS retained + soft flags recorded ✓ stroke_quality_score nan_frac × max_deriv × mono_viol × detect_conf × drive_dur = 0.82 0 1 confidence signal for video-only inference (§13) — any single bad dimension drives score toward 0
Stroke flows top-to-bottom through six checks; hard flags reject, soft flags record; five sigmoid penalties multiply into the composite score
Progression Monotonicity ▶
Detects when handle progress required silent repair via maximum.accumulate

The normalized progress signal s_video should be monotonically increasing during the drive. When the raw signal has backwards segments (tracking glitches), np.maximum.accumulate forces monotonicity — but this distorts the progress axis and corrupts chain-rule derivatives.

Metric: progress_mono_violation_frac — fraction of drive frames where the raw value was below the running maximum.

Threshold: > 0.15 triggers qc_progress_nonmonotonic (soft flag).

Implementation
 raw_s = clip((dist - x0) / (x1 - x0), 0, 1)
 s_video = maximum.accumulate(raw_s)
 violations = sum(raw_s < s_video - 1e-9)
 frac = violations / len(raw_s)
Stroke Detection Confidence ▶
Measures boundary sharpness via velocity contrast at catch and finish

Strokes with mushy, ambiguous catch/finish boundaries have misplaced drive windows, directly corrupting the label alignment between video features and RP3 force curves.

Velocity contrast: computed as the difference in mean |slope| across a short window on each side of the boundary, normalized by peak drive velocity. High = sharp reversal, low = gradual transition.

Metric: detection_confidence = min(catch_contrast, finish_contrast)

Threshold: < 0.15 triggers qc_weak_detection (soft flag).

Three new fields on DriveEvent: catch_velocity_contrast, finish_velocity_contrast, drive_prominence.

Duration Plausibility ▶
Simple physiological range check on drive and cycle durations

If a detected drive is 0.15 s or 3.5 s, it cannot be a real rowing stroke — the detection failed. This trivial range check catches outliers that the velocity-based detector missed.

Drive range: [0.4, 2.0] s. Cycle range: [1.5, 4.0] s.

Classification: hard drop — implausible durations mean the label alignment is corrupted.

Constants (inference_cli.py)
 DRIVE_DURATION_MIN_S = 0.4
 DRIVE_DURATION_MAX_S = 2.0
 CYCLE_DURATION_MIN_S = 1.5
 CYCLE_DURATION_MAX_S = 4.0
Aggregate Quality Score ▶
Product of per-dimension sigmoid penalties → single [0, 1] confidence value

Rather than just binary flags, a continuous score enables soft weighting during training and gives end-users a trust signal during inference.

Aggregation: multiplicative — any single bad dimension drives the composite score toward zero (AND-logic, not averaging).

Five penalty dimensions each mapped through a smooth sigmoid from "good" (penalty ≈ 1) to "bad" (penalty ≈ 0):

_compute_quality_score()
 nan_frac → good < 0.05, bad > 0.3
 max_deriv_deg_s → good < 300, bad > 600
 mono_violation_frac → good < 0.05, bad > 0.15
 detection_confidence → good > 0.30, bad < 0.15
 drive_duration_s → good [0.6, 1.5], bad outside [0.4, 2.0]

score = ∏ sigmoid_penalty(dim_i)
modeling & prediction
07 Model Training

Model training page (full walkthrough: target, Stage 0, A, B, evaluation, summary)

Target representation Standard PCA or functional PCA (B-spline + PCA) — selected via --target-representation Same diagram on model training page
RAW FORCE CURVES F(s) s = 0.0 s = 1.0 N strokes × 64 bins on s-grid peak-norm PEAK-NORMALIZED F(s) / max(F) all curves scaled to [0, 1] PCA PCA DECOMPOSITION PC1 PC2 PC3 ... PC20 c1 c2 c3 .. c20 20 coefficients per stroke Reconstruction: F̂(s) = peak × Σ c_k · PC_k(s) Stage A predicts c_1 ... c_k   •   Stage B predicts F(s) directly

Force curve target encoding: raw curves → peak-normalized → PCA coefficients (20 components)

Stage 0 — Sanity Baselines ▶
Force reproducibility floor + metadata-only regression — the bar biomechanics models must clear

Reproducibility floor: Quantify within-condition variability of force curves (same athlete, similar stroke rate/length bins). Median pairwise curve distance, coefficient of variation.

Metadata baseline: Regress force-curve PCA coefficients from scalar metadata only (stroke rate, length, drive time). Reconstruct predicted curves. Every biomechanics model must beat this.

Inputs: stroke_rate, stroke_length, drive_time
Targets: PCA coefficients → F̂(s)
Gate: biomechanics must improve on this
Stage 0 Reproducibility floor and metadata-only baseline Same diagram on model training page
0a — FORCE REPRODUCIBILITY FLOOR Condition bin same athlete similar stroke rate similar stroke length ⇒ group of comparable strokes Pairwise comparison how much do they vary? Reproducibility metrics median pairwise curve distance coefficient of variation (peak force) coefficient of variation (impulse) ⇒ this is the irreducible variability floor 0b — METADATA-ONLY BASELINE Scalar metadata stroke_rate_spm stroke_length_cm drive_ratio no kinematics Linear regression Ridge / OLS PCA c_1 ... c_k predicted coeffs ↓ reconstruct curve Baseline prediction predicted vs actual GATE: all later models must beat 0b on curve error + rowing metrics

Stage 0: establish the reproducibility floor (0a) and metadata-only prediction baseline (0b)

Stage A — Interpretable Kinematic Models ▶
Stroke-level summary features + coordination scalars → regularized linear / tree ensembles → PCA or fPCA coefficients

Verifies that kinematic features add value beyond metadata. Identifies dominant explanatory biomechanics features through feature importance analysis. Target representation is selected per-run (--target-representation standard|fpca); both families can coexist in the same dataset directory.

Inputs: 6 canonical angle summaries (30) + coordination scalars (onsets, lags, range fractions, drive_ratio) + context (stroke rate, length)
Models: Ridge, Lasso, Random Forest, GBM
Targets: PCA or fPCA coefficients → reconstructed F̂(s)
Stage A Scalar features to PCA coefficients Same diagram on model training page
INPUT: SCALAR FEATURES (per stroke) Kinematic summary knee_active_deg_min knee_active_deg_max knee_active_deg_range knee_active_deg_mean knee_active_deg_s_at_max … ×6 angles = 30 features min, max, range, mean, s-position of peak build_training_dataset.py Coordination onset_knee_s onset_trunk_s onset_arms_s lag_knee_to_trunk_s lag_trunk_to_arms_s knee_range_frac hip_range_frac elbow_range_frac _compute_coordination_features Context stroke_rate_spm stroke_length_cm drive_ratio same features as Stage 0 metadata baseline x ∈ R^~41 (per stroke) Models Ridge regression Lasso regression Random Forest Gradient Boosted Trees PCA / fPCA coeffs ĉ_1, ĉ_2, … ĉ_k ↓ reconstruct F̂(s) ↓ evaluate vs Stage 0 Key output: feature importance ranking — which biomechanics explain force?

Stage A: stroke-level scalar features → classical ML → PCA force-curve coefficients

Stage B — Sequence Models ▶
Full progress-aligned kinematic sequences → TCN / Transformer encoder → full force curve

Temporal convolutional network or transformer encoder operating on the full progress-aligned feature sequences. Architecture selected via --stageB-arch {tcn, transformer}; captures nonlinear stroke dynamics and cross-joint interactions.

Conditioning: Coordination scalars (onsets, lags, range fractions, drive_ratio) are broadcast as constant extra input channels when --stageB-use-coordination is set — no separate scalar head.

Loss: Masked pointwise MSE using the per-stroke force_mask; optional derivative regularizer gated by --stageB-lambda-deriv.

Artifacts: Each run saves state.pt, arch_config.json, feature_norm.npz, and target_norm.npz from the best CV fold, plus architecture-suffixed copies for cross-arch comparison.

Inputs: X(s) [N_bins × N_features] (+ optional coord channels)
Models: TCN (_ForceCurveTCN), Transformer (_ForceCurveTransformer)
Targets: F(s) [N_bins], masked loss + optional d/ds regularizer
Stage B Full kinematic sequences to force curve Same diagram on model training page
INPUT: KINEMATIC SEQUENCES ON s-GRID s = 0.00 0.02 0.03 ... 0.98 1.00 —————————————————————————— knee(s) hip(s) elbow(s) trunk(s) spine(s) d_knee/ds d_hip/ds ... handle_vel(s) handle_acc(s) shape: (64, 12) 64 s-grid points × 12 kinematic channels SEQUENCE MODEL Temporal Conv Network causal dilated convolutions or Transformer Encoder self-attention over s positions + scalar conditioning stroke rate, length, ratios OUTPUT F̂(s) direct prediction shape: (64,) force at each s bin Loss function Masked pointwise MSE — bins beyond stroke length masked out. Shape regularizers / derivative losses only if validation shows pathological curves. Why sequences > scalars Captures when and how joints interact during the drive. Learns nonlinear dynamics that summary stats cannot encode. Stage B predicts the full force curve — no PCA bottleneck — evaluated against same metrics as Stages 0 and A

Stage B: full kinematic sequence (64×12) → temporal model → predicted force curve (64 bins)

Stage progression Each stage must beat the previous on the same metrics Same diagram on model training page
0 Sanity Baselines metadata → PCA coefficients = the floor to beat must beat A Kinematic Baselines scalar features → PCA coefficients interpretable + feature importance must beat B Sequence Models full sequences → full force curve captures nonlinear dynamics Best model Each stage strictly must beat all previous stages on curve error + rowing-relevant metrics

Stage progression: each stage must beat the previous on the same metrics

08 Prediction & Output
Force Curve Prediction ▶
Video-only CLI loads a self-describing model bundle → predicts per-stroke force curve + derived metrics

At inference, no RP3 input is used. inference/predict_force_cli.py --run-dir <run> --model-dir <bundle> loads a portable model_bundle/, runs drive detection, extracts pose features via segment_features.build_pose_drive_segments, runs Stage A or Stage B prediction depending on what the bundle contains, and writes normalized curves, an RP3-style cm-indexed force table, and derived metrics.

model_bundle/ (written by export_model_bundle.py)
 manifest.json — feature list, s_grid, active_side_default, include_head, target_representation, git_sha
 s_grid.npy, feature_names.json
 pca_model.joblib or fpca_model.joblib
 stageA/ — sklearn estimator (.joblib)
 stageB/ — state.pt, arch_config.json, feature_norm.npz, target_norm.npz
Outputs per stroke
 force_at_<distance>cm table (RP3-style, cm-indexed)
 normalized F̂(s) on the bundle s-grid
 predicted peak_force (N), peak_position (% drive), impulse (N·s)
 stroke_quality_score [0, 1]
09 Evaluation Protocol

Evaluation diagram on model training page

Evaluation protocol Splits and metrics (all stages, same held-out set) Same diagram on model training page
DATA SPLIT STRATEGY Current phase (limited athletes) train fold 1 train fold 2 held-out Session-held-out or time-block splits within athlete. Results labeled as provisional; no generalization claims. Future phase (multi-athlete) athlete A athlete B athlete C ath. D ath. E Leave-one-athlete-out or grouped folds. True generalization test: unseen biomechanics. EVALUATION METRICS (all stages, same held-out set) Curve-level RMSE(F, F̂) over all valid bins MAE(F, F̂) over all valid bins Pearson r per-stroke correlation Does the predicted curve match the overall shape and magnitude? Rowing-relevant peak force error (N) peak position error (%) impulse error (N·s) Downstream quantities that coaches use for technique feedback. Phase-specific early drive mid drive late drive RMSE per phase segment Reveals where the model struggles: catch, peak, or finish.

Evaluation: consistent splits and metrics across all stages for fair comparison

Curve-Level Metrics ▶
RMSE / MAE over force bins, per-stroke curve correlation

Standard regression metrics over all valid force bins in the hold-out set.

 RMSE(F, F̂), MAE(F, F̂)
 Pearson r per stroke
Rowing-Relevant Metrics ▶
Peak force error, peak position error, impulse error, phase-specific errors

Metrics that matter to coaches and athletes — the downstream quantities that inform technique feedback.

 peak_force_error (N)
 peak_position_error (% drive)
 impulse_error (N·s)
 early / mid / late drive phase errors
Cross-Representation / Cross-Architecture Report ▶
evaluation_report.json + text report group by target_representation and stage_b_arch

Every run annotates its overall metrics with target_representation (standard / fpca) and stage_b_arch (tcn / transformer). write_evaluation_report merges prior-run artifacts in the same output directory so PCA vs fPCA and TCN vs Transformer are compared head-to-head on the same held-out strokes, without clobbering either.

Artifacts
 stage_b_tcn.pt, stage_b_transformer.pt (arch-suffixed copies)
 evaluation_report.json — grouped by representation & arch
 evaluation_report.txt — human-readable comparison
Video pipeline
RP3 force data (training)
Shared processing
Inference output
Quality control gate

Click any card to toggle data contracts. Use the view switcher to focus on training or inference paths. For Stage 0–B training flows and evaluation metrics, see model training diagrams.