ML2024

Multimodal GAN

A Generative Adversarial Network synthesising multimodal patient records — clinical text, medical imaging descriptors, and time-series vitals — with roughly 40% improvement in training data diversity, deployed on GCP Vertex AI.

Training diversity improvement
~40%
Modalities generated jointly
3
Production deployment
Vertex AI

The problem

A patient is not one kind of data. They are a narrative, a set of images, and a stream of numbers, and the correlations *between* those modalities are exactly what a clinical model needs to learn. Single-modality synthetic data throws that away: generate the vitals and the notes independently and you get a dataset where nothing agrees with anything else, which is worse than no data at all.

The approach

Train a GAN across all three modalities jointly so the generator has to produce records that are internally consistent — vitals that match the diagnosis that matches the imaging descriptor. PaLM-E was integrated for multimodal reasoning over the combined representation, and the whole pipeline was deployed to GCP Vertex AI so inference could scale past a single machine.

Healthcare AI is bottlenecked by data scarcity and privacy constraints. I wanted to prove that synthetic, privacy-preserving patient data could unlock the next generation of medical ML models, without compromising a single real patient record.

Why I built it

Architecture

  1. 01

    Modality encoders

    Clinical text, imaging descriptors, and time-series vitals each encoded into a shared representation.

  2. 02

    Adversarial training

    Generator and discriminator trained against the joint distribution, so cross-modal consistency is what the discriminator penalises.

  3. 03

    Multimodal reasoning

    PaLM-E integrated to reason over the combined representation.

  4. 04

    Deployment

    Full pipeline deployed on GCP Vertex AI for scalable inference.

Stack

Architecture
GAN · PaLM-E
Analysis
NumPy · Pandas · Scikit-learn
Cloud
GCP Vertex AI
  • Python
  • GANs
  • PaLM-E
  • GCP Vertex AI
  • NumPy
  • Pandas
  • Scikit-learn
  • Matplotlib

Outcome

Roughly 40% improvement in training data diversity over the baseline, with generated records that hold together across all three modalities. The work became the foundation for everything I have done since on generative models for healthcare.

NextML

Conditional diffusion model synthesising brain tumour MRIs.