← projects3D-Aware Genomic Foundation Model (HopField-Mamba)
A genomic foundation model trained from scratch that conditions pretraining on 3D chromatin (Hi-C) structure rather than adding it post-hoc.
status in-progress
stack Python, PyTorch, Mamba/SSM, Hi-C genomic data, 2x NVIDIA L40S
An independent research project building a genomic foundation model from scratch on a 2x NVIDIA L40S cluster. The core idea is to condition self-supervised pretraining directly on 3D chromatin (Hi-C) structure, rather than bolting structure on after the fact the way the closest prior work does.
problem
Genomic foundation models read DNA as a linear sequence, but regulation is three-dimensional — enhancers act on promoters through chromatin folding. The closest prior art (Evo2HiC, Noble Lab 2025) adds structure post-hoc via contrastive distillation, and has no variant/ClinVar evaluation in that lineage.
solution
A structural-bias mechanism inside the SSM recurrence itself — a per-channel timescale bias plus a permeability penalty on the Caduceus-PH backbone — so 3D structure shapes pretraining directly, at only +0.43% parameter overhead and inside a 5% matched-compute constraint.
architecture
- Backbone: Caduceus-PH (Mamba/SSM), from-scratch self-supervised pretraining
- Structural bias: per-channel timescale bias + permeability penalty in the SSM recurrence
- Data pipeline: Hi-C contact maps, insulation score and compartment PC1 tracks
- Validation: independent 4DN reference tracks and CTCF ChIA-PET assay data
- Training: 2x NVIDIA L40S, 3 seeds, matched-compute constraint against baseline
- Evaluation: GUE benchmark suite
results
The 7.7M-parameter baseline trains to 1.52 bits/nucleotide validation loss across 3 seeds, establishing the floor a structural-vs-baseline comparison must clear. The data pipeline validates against independent reference data (insulation score r=0.997, compartment PC1 r=0.976 vs 4DN tracks), and a predicted regulatory loop was corroborated by independent CTCF ChIA-PET assay data. On the GUE suite it outperforms a pure MambaMAE baseline on 3 of 4 evaluated tasks. A separate finding — a memory-horizon collapse in Mamba default timestep initialization, whose fix raises median effective memory span ~30x with validation loss unchanged within seed noise — is being written up as a standalone transferable result.
questions
What makes this different from existing genomic foundation models?
Structure is part of pretraining rather than a post-hoc addition. The closest prior work distills structure contrastively after the fact; here the 3D signal biases the state-space recurrence during self-supervised training.
Is the memory-horizon fix specific to genomics?
No. The degradation comes from Mamba default timestep initialization, so the fix should transfer to any long-context SSM. That is why it is being written up separately.