Skip to content
All work

Case studyAcademic

LungVision: multimodal lung cancer detection

A final-year team project that combines chest X-rays, CT scans and clinical metadata for early lung cancer detection, with four-CNN ensembles per imaging modality and Grad-CAM and LIME explanations for every prediction.

  • 1st place, final-year project evaluation
  • Presented at Techtatwa 2K25
A chest X-ray uploaded to the LungVision application.The same chest X-ray with the Grad-CAM heatmap from model C of the LungVision ensemble, concentrated over both lung fields (colours corrected).
Real LungVision output: the same scan and model, colours corrected.
My role
Flask REST API and model serving for the ensemble, with Grad-CAM and LIME explanations
Team
Team of 4
Period
Mar 2025 – Jan 2026
Type
Academic · Team of 4
Reading depth

View repository (opens in a new tab)

The 30-second version

Problem
Lung cancer evidence is spread across chest X-rays, CT scans and a patient's symptoms; most student projects model only one of them, and few explain their decisions.
My contribution
Team of four. My part: the Flask REST API and model serving for the ensemble, returning predictions with Grad-CAM and LIME explanations.
Approach
Separate ensembles of four ImageNet-pretrained CNNs for X-ray and CT (softmax probabilities averaged), a small neural network for 15 clinical inputs, and two explanation methods.
Results
First place in the final-year project evaluation; presented at Techtatwa 2K25. Accuracy figures are not published because the training and evaluation code is not in the repository.
Stack
  • PyTorch
  • torchvision
  • Grad-CAM
  • LIME
  • Flask
  • SQLAlchemy
  • ReportLab
  • Docker
  • Gunicorn

Problem

Lung cancer is far more treatable when it is found early, but the evidence is spread across different sources: chest X-rays, CT scans and a patient’s symptoms and history. LungVision asks what a single system can do when it can look at all three, and how it can explain each decision to a clinician.

Team and my contribution

LungVision was our final-year project at Maharaja Institute of Technology Mysore, built by a team of four. It won first place in the final-year project evaluation and was presented at Techtatwa 2K25.

My part was the application layer: the Flask REST API that serves the ensemble models and returns predictions together with their Grad-CAM and LIME explanations. The models, datasets and evaluation were shared team work, so this page describes the system as a whole.

Images across modalities
20k+
Data modalities
3
X-ray · CT · metadata
CNNs per imaging modality
4
averaged ensemble
Clinical inputs
15
symptoms & history

Data

Modality Source Used for
Chest X-ray Custom subset of the NIH Chest X-ray dataset Four-CNN X-ray ensemble
CT scan Custom subset of the NSCLC collection (TCIA) Four-CNN CT ensemble
Clinical metadata Public Kaggle dataset Neural network on 15 inputs

Architecture

Inference pipeline · select a stage

Stage 1 of 6 · Input

Upload and validate

The user chooses a modality and uploads a chest X-ray or CT image, or fills in the clinical form. Uploads are size-limited and stored under random identifiers.

  • 16 MB upload limit
  • UUID-based file names
  • Separate form for clinical data

Stage 2 of 6 · Preprocess

Per-model transforms

The same image is prepared four ways, once per network, using each architecture's expected input size and normalisation.

  • 300 px · EfficientNet-B3
  • 299 px · Inception-v3
  • 224 px · DenseNet-121 and ResNet-50

Stage 3 of 6 · Models

Four CNNs per modality

Separate X-ray and CT ensembles of EfficientNet-B3, DenseNet-121, Inception-v3 and ResNet-50, fine-tuned from ImageNet weights. The metadata branch is a small fully connected network.

  • Classes: lung cancer, normal, and an 'unknown' reject class
  • Metadata: yes / no output

Stage 4 of 6 · Ensemble

Average the probabilities

Softmax probabilities from the four networks are averaged; the consensus label and confidence come from the averaged distribution.

  • Reduces the variance of any single model
  • Per-model results are kept for inspection

Stage 5 of 6 · Explain

Grad-CAM and LIME

Grad-CAM produces a heatmap for each network. LIME perturbs SLIC superpixels to show which regions support the prediction, and explains the clinical-data prediction feature by feature.

  • Grad-CAM on each model's last convolutional block
  • LIME with superpixels for images
  • LIME feature weights for metadata

Stage 6 of 6 · Output

Results and PDF report

The app shows the consensus prediction, confidence, per-model results and explanations, and generates a downloadable PDF report.

  • PDF reports generated with ReportLab
  • Optional Gemini-powered help assistant

Engineering decisions, with the code

Per-architecture preprocessing. Feeding every network the same 224 px image would quietly hurt the models pretrained at other resolutions, and Inception-v3 expects a different normalisation. One transform table keeps this correct and in one place:

model_transforms = {
    'EfficientNetB3': transforms.Compose([
        transforms.Resize((300, 300)),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
    ]),
    'InceptionV3': transforms.Compose([
        transforms.Resize((299, 299)),
        transforms.ToTensor(),
        transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5]),
    ]),
    # DenseNet121 and ResNet50: 224 × 224 with ImageNet normalisation
}

Averaging probabilities, not votes. Different architectures make different mistakes. Averaging the softmax outputs gives a steadier consensus and a meaningful confidence, while each model’s own result is kept so a reviewer can see when they disagree:

avg_probs = torch.stack(probabilities).mean(dim=0)
consensus_conf, consensus_pred = torch.max(avg_probs, 0)
consensus_label = class_names_list[consensus_pred.item()]

An explicit “unknown” class. Users can upload anything. The image models output a third, reject class, so the system can say “this does not look like the expected scan” instead of forcing a cancer/normal answer.

Serving eight large CNNs in one web process. Models are loaded once at start-up rather than per request, and Gunicorn runs with model preloading and long timeouts, so the first prediction is not penalised.

Evaluation

The team evaluated the models with accuracy, ROC curves, Cohen’s kappa and confusion matrices. I am not publishing figures here because the training and evaluation notebooks are not part of the public repository, so the numbers cannot be checked against code. They will be added once that evidence is published.

Explainability: what the outputs show

Every image prediction returns a Grad-CAM heatmap and a LIME explanation for each of the four networks. The figures below are real outputs from the app for one uploaded X-ray.

Chest X-ray uploaded to LungVision.Grad-CAM from ensemble model C, concentrated over both lung fields.
lowerGrad-CAM attention (relative, per image)higher
The original chest X-ray.
Original
Grad-CAM from ensemble model A, focused low and left of centre in the image, near the heart border.
Model A
Grad-CAM from ensemble model B, focused along the chest wall on the left side of the image.
Model B
Grad-CAM from ensemble model C, focused over both lung fields.
Model C
Grad-CAM from ensemble model D, focused on the left side of the image, extending below the diaphragm.
Model D

Read correctly, the four networks attend mostly to the chest, but each looks somewhere different: one low near the heart border, one along the chest wall on the left of the image, one across both lung fields, one extending below the diaphragm. (The app does not label which file came from which architecture, so they are shown as A–D.) The averaged prediction hides this disagreement, which is a good argument for showing per-model explanations to reviewers.

Ensemble lens · one scan, four saved outputs

Saved Grad-CAM output A for this chest X-ray (colours corrected).
Saved Grad-CAM output B for this chest X-ray (colours corrected).
Saved Grad-CAM output C for this chest X-ray (colours corrected).
Saved Grad-CAM output D for this chest X-ray (colours corrected).

Move the pointer over any panel, tap or drag on touch, or focus the grid and use the arrow keys: the lens moves across all four together. A–D are the order the app saved the files, not verified model identities. Nothing is computed from the overlays.Source: app-saved outputs, scan 1 of 4

LIME is far less consistent. The app saved four LIME outputs for this scan, one per model run, and they barely agree:

  • LIME run 1: supporting (green) regions at the top centre of the image and along the lower-left edge; opposing (red) regions over the lung on the left of the image.Run 1
  • LIME run 2: supporting regions inside both lung fields and outside the body near the lower corners.Run 2
  • LIME run 3: supporting regions at the top centre, inside both lung fields, along the lower-left edge and outside the body at the lower right.Run 3
  • LIME run 4: supporting regions on both shoulders, the image edge and below the diaphragm.Run 4
The four LIME outputs LungVision saved for this X-ray. The app runs LIME once per ensemble model but does not record which file belongs to which model, so the runs are numbered, not attributed. Green regions support the prediction, red regions oppose it. Across the 30 matched scans in the repository (180 pairs of runs), supporting regions had a median pairwise IoU of 0.00 (interquartile range 0.00–0.06, maximum 0.25).

Some runs place supporting superpixels inside the lung fields; others place them on the shoulders, the image edge, below the diaphragm or outside the body. Because the app does not record which LIME file came from which model, this variation cannot be split into “different models” versus “LIME’s own random sampling”, and no LIME run can be paired with a particular Grad-CAM above. What the evidence does support is narrower: these LIME explanations are too unstable to support conclusions on their own, and the off-lung regions that appear in some runs are a reason to test for shortcut learning, not proof of it.

Corrected Grad-CAM on an axial chest CT slice, shown at the scan's original proportions.
Grad-CAM on a CT slice (colours corrected; resampled from the app's square 512 × 512 output back to the scan's original 1024 : 689 proportions).Source: App output, repository
LIME bar chart of clinical features, with chronic disease, fatigue and chest pain pushing towards cancer and allergy pushing against it.
LIME explanation for the clinical-data model.Source: App output, repository

The full analysis, including how the colour bug nearly led to the wrong conclusion and how LIME stability was measured, is in What Grad-CAM revealed about shortcut learning.

Application and deployment

  • Flask application with registration and login (bcrypt-hashed passwords, SQLAlchemy), a diagnosis workflow per modality, and explanation views.
  • PDF reports that combine the prediction, confidence, explanations and plain-language clinical insights.
  • Help assistant using the Gemini API, with fallback responses when no API key is configured.
  • Containerised with a Dockerfile and Docker Compose: Gunicorn workers, models preloaded at start-up, and a container health check.

Challenges, trade-offs and lessons

  • Averaging buys stability but hides disagreement. The consensus is steadier than any single model, yet the per-model explanations show the networks attend to different regions. Both views belong in front of a reviewer.
  • Explanations need the same review as metrics. A one-line channel-order slip inverted every Grad-CAM. Without reading the overlay code, the heatmaps support the opposite conclusion.
  • Check explanation stability before reading meaning into it. The four LIME runs on a scan barely overlap; a single run would have told a confident but unreliable story.

What I would do next

  1. Fix the overlay colour order in the app.
  2. Segment or crop the lung fields before classification, then measure how much attribution falls outside them, using repeated LIME runs with a fixed seed per model so stability can be reported.
  3. Quantify explanations across the test set (for example, the share of activation inside a lung mask) instead of inspecting examples.
  4. Split data by patient, calibrate the ensemble’s probabilities, and publish the training and evaluation notebooks.

Repository

The application code is on GitHub.

100%