Case studyIndependent prototype
Network performance forecasting and early warning
An independent prototype that forecasts network throughput, latency and packet loss with statistical and deep-learning models, raises early warnings and drafts incident reports. Built as a tested Python package and developed on synthetic perfSONAR-style data.
- My role
- Sole developer: data generation, models, API, tests and deployment manifests
- Period
- 2026
- Type
- Independent prototype
The 30-second version
- Problem
- Degradation on research networks often goes unnoticed for hours; perfSONAR history could be used to predict it and warn operators early.
- My contribution
- Independent prototype; I wrote the whole repository: data generator, features, four forecasters, ensemble, early warning, reports, API, tests and deployment manifests.
- Approach
- ARIMA, Prophet, LSTM and Transformer forecasters on the same horizon, a weighted ensemble, threshold- and forecast-based alerts, and a FastAPI service with Docker and Kubernetes manifests.
- Results
- A working, tested end-to-end prototype on synthetic data. No accuracy figures are claimed. A code review for this page found an ensemble weight-alignment bug (below).
- Stack
- Python
- PyTorch
- statsmodels
- Prophet
- FastAPI
- pytest
- Docker
- Kubernetes
Problem
Research and education networks move very large scientific datasets, and a slow link or rising packet loss can stall work for hours before anyone notices. Monitoring tools such as perfSONAR record throughput, latency and loss continuously. The opportunity is to use that history to predict degradation and warn operators before users are affected.
Objective
Build an end-to-end forecasting system rather than a single model in a notebook: realistic data, several forecasting approaches compared fairly, an early-warning layer, readable incident reports, and a service that could be deployed and tested like production software.
My contribution
I wrote the whole repository: the data generator, feature engineering, the four forecasting models, the ensemble, the early-warning and reporting modules, the API, the tests, the Docker image, the Kubernetes manifests and the documentation.
- Forecasting models
- 4
- + weighted ensemble
- Metrics forecast
- 3
- throughput · latency · loss
- Test modules
- 7
- pytest
- Kubernetes manifests
- 4
- incl. GPU training job
Data and feature engineering
Real perfSONAR archives were not available for the prototype, so a generator produces hourly telemetry (throughput, latency, packet loss and retransmits) with daily and weekly seasonality, trend shifts and a configurable share of injected anomalies. Because the generator is seeded, every experiment is reproducible.
On top of the raw series, the preprocessing module adds:
- cyclical encodings of hour-of-day and day-of-week, so 23:00 and 00:00 are close together;
- rolling mean, standard deviation, minimum and maximum over several windows;
- lag features to capture autocorrelation;
- first differences to capture the rate of change.
Architecture
Forecasting system · select a stage
Stage 1 of 7 · Data
Telemetry and features
Hourly perfSONAR-style measurements are loaded (synthetic in this prototype) and enriched with time encodings, rolling statistics, lags and differences.
- Seeded, reproducible generator
- Anomaly injection for testing alerts
Stage 2 of 7 · Models
Four forecasters
Statistical and deep-learning models are trained side by side on the same data and horizon, so they can be compared fairly.
- ARIMA / SARIMA and Prophet
- LSTM and Transformer encoder (PyTorch)
Stage 3 of 7 · Ensemble
Weighted combination
Forecasts are combined with configurable weights and normalised to sum to one.
- Weights live in configuration, not code
- Known issue: weights misalign if Prophet is unavailable
Stage 4 of 7 · Adapt
Few-shot adaptation
A Reptile-style meta-learning routine trains across several network topologies, then adapts quickly to a new topology from a small number of examples.
- Meta-training across topology tasks
- Fast fine-tuning on a small support set
Stage 5 of 7 · Detect
Early warning
Current values and forecasts are checked against configurable thresholds, producing warnings with severity levels.
- Threshold-based and prediction-based checks
- Critical and warning severities
Stage 6 of 7 · Report
Incident reports
Warnings become structured incident reports with likely causes and remediation steps, from a deterministic template or an optional LLM endpoint.
- Template fallback when no LLM key is set
- Markdown and JSON output
Stage 7 of 7 · Serve
API and deployment
A FastAPI service exposes forecasts, warnings and reports. It ships with a Dockerfile and Kubernetes manifests.
- Deployment, service and config map
- Separate GPU training job
Models and engineering decisions
| Model | Why it is included | Implementation |
|---|---|---|
| ARIMA / SARIMA | Strong, interpretable baseline for linear and seasonal patterns | statsmodels |
| Prophet | Handles trend changes and missing data with little tuning | Prophet |
| LSTM | Learns non-linear temporal dependencies | PyTorch, 2 layers, 64 hidden units |
| Transformer | Captures longer-range dependencies with self-attention | PyTorch encoder, 2 layers, 4 heads |
| Ensemble | No single model is best on every metric | Configurable weighted average |
- Compare simple and complex models together. Classical baselines are fast and explainable. Deep models only earn their cost if they beat them under the same conditions.
- Ensemble rather than pick a winner. Different models tend to win on different metrics, so a weighted combination is more robust than betting on one.
- Few-shot adaptation for new topologies. Networks change. Meta-learning aims to make a forecaster useful on a new path after seeing only a little of its data.
Code review: a weight-alignment bug
The ensemble collects each model’s forecast in a dictionary and takes the matching slice of the configured weights. When Prophet is not fitted, its forecast is skipped and its weight set to zero:
if self.prophet.is_fitted:
prophet_result = self.prophet.predict(horizon)
model_preds["prophet"] = prophet_result.predictions
else:
weights[1] = 0.0
...
active_weights = np.array(weights[:len(model_preds)])
active_weights = active_weights / active_weights.sum()
all_preds = np.stack(list(model_preds.values()))
ensemble_pred = np.average(all_preds, axis=0, weights=active_weights)
With Prophet missing, model_preds holds three forecasts (ARIMA, LSTM, Transformer) but weights[:3] is ARIMA’s, Prophet’s zeroed weight and LSTM’s. The LSTM forecast is silently weighted zero and the Transformer receives the LSTM’s weight. With all four models available the slice lines up and the ensemble is correct. The robust fix is to key weights by model name rather than by position, plus a unit test for the missing-model path. This has not been fixed yet; I found it while preparing this page, and it is exactly the kind of bug a test for the fallback path would have caught.
Evaluation methodology
The evaluation module compares models on a held-out forecast horizon with MAE, RMSE, MAPE and directional accuracy (whether the forecast gets the direction of change right). Directional accuracy matters for alerting, because operators care about whether things are getting worse.
Results from synthetic data say little about real networks, so they are not reported here. The next step is to run the same evaluation on real perfSONAR archives with rolling-origin backtesting.
Software engineering
- Package layout under
src/with apyproject.toml, separate modules for data, models, evaluation, early warning, reporting and the API. - Seven pytest modules covering data generation, preprocessing, models, evaluation, early warning, reporting and the API.
- Repeatable setup: declared dependencies, a Makefile for common tasks, and a Docker image.
- Kubernetes manifests for the API deployment, service, configuration and a GPU training job.
- Documentation of the architecture and deployment in
docs/.
Challenges and solutions
- No access to real data. A seeded generator with realistic seasonality and injected anomalies made it possible to build and test the whole pipeline, including the alerting path, while being explicit that results are not real-world evidence.
- Very different model interfaces. Statistical and neural models fit, predict and fail differently. A common forecasting interface kept the rest of the system simple, but the fallback path for a missing model needed more care than it got (see the weight-alignment bug above).
- Making alerts actionable. A raw threshold breach is not enough for an operator, so warnings are turned into reports with severity, likely causes and suggested remediation.
Lessons learned
- Building the boring parts (tests, configuration, deployment) first made every later model change cheaper and safer.
- Synthetic data is excellent for engineering and dangerous for conclusions. Separating the two keeps a project honest.
- Directional accuracy is often more useful than raw error for operational forecasting.
Repository
The code, tests and documentation are on GitHub.