ML2024

Multimodal GAN

A Generative Adversarial Network synthesising multimodal patient records (clinical text, medical imaging descriptors and time-series vitals) with roughly 40% improvement in training data diversity, deployed on GCP Vertex AI.

Training diversity improvement
~40%
Modalities generated jointly
3
Production deployment
Vertex AI

The problem

A patient is several kinds of data at once: a narrative, a set of images and a stream of numbers. The correlations between those modalities are what a clinical model needs to learn, and single-modality synthetic data throws them away. Generate the vitals and the notes independently and you get a dataset where nothing agrees with anything else, which is worse than no data at all.

The approach

Train a GAN across all three modalities jointly, so the generator has to produce records that hold together internally: vitals that match the diagnosis that matches the imaging descriptor. PaLM-E was integrated for multimodal reasoning over the combined representation, and the pipeline was deployed to GCP Vertex AI so inference could scale past a single machine.

Healthcare AI is bottlenecked by data scarcity and privacy constraints. I wanted to prove that synthetic, privacy-preserving patient data could unlock the next generation of medical ML models, without compromising a single real patient record.

Why I built it

Architecture

  1. 01

    Modality encoders

    Clinical text, imaging descriptors, and time-series vitals each encoded into a shared representation.

  2. 02

    Adversarial training

    Generator and discriminator trained against the joint distribution, so cross-modal consistency is what the discriminator penalises.

  3. 03

    Multimodal reasoning

    PaLM-E integrated to reason over the combined representation.

  4. 04

    Deployment

    Full pipeline deployed on GCP Vertex AI for scalable inference.

Stack

Architecture
GAN · PaLM-E
Analysis
NumPy · Pandas · Scikit-learn
Cloud
GCP Vertex AI
  • Python
  • GANs
  • PaLM-E
  • GCP Vertex AI
  • NumPy
  • Pandas
  • Scikit-learn
  • Matplotlib

Outcome

Roughly 40% improvement in training data diversity over the baseline, with generated records that hold together across all three modalities. It set the direction for the generative work I have done since.

NextML

Conditional diffusion model synthesising brain tumour MRIs.