VulnerableCode Insights
Overview
VulnerableCode imports vulnerability advisories from dozens of data sources and grows its database continuously. However, it previously lacked a way to demonstrate its quality. This project enables VulnerableCode to visually showcase its data richness and coverage, while highlighting areas for improvement. This transparency gives users confidence in VulnerableCode and provides maintainers with actionable insights to improve it.
This GSoC project builds three key features:
Insights Dashboard: A multi-panel dashboard containing various interactive charts to showcase VulnerableCode’s strengths and identify areas for improvement.
History of Advisories: A diff view for tracking changes in an advisory and viewing its historical versions as snapshots.
EPSS History: Visual tracking of EPSS score and percentile trends over time.
Detailed Report
Insights Dashboard
The Insights Dashboard is a multi-panel interactive dashboard designed to visualize various features of VulnerableCode through infographic charts.
A dedicated insights Django app houses the dashboard’s core functionality.
To ensure fast and scalable chart rendering, an automated daily snapshot pipeline
(insights_snapshot_pipeline.py) pre-computes database-wide metrics once every night, avoiding
heavy on the fly queries across VulnerableCode’s million-row tables. These metrics
are stored in dedicated models linked to a DailySnapshot, keeping the telemetry
structured and instantly queryable.
Below is the tree view of insights for better understanding
insights
├── models.py
├── views.py
├── urls.py
├── utils.py
├── insights_snapshot_pipeline.py
├── charts/
│ ├── __init__.py
│ ├── overview_panel.py
│ ├── package_panel.py
│ ├── severity_panel.py
│ ├── importer_panel.py
│ └── data_quality_panel.py
├── templates/insights/
│ ├── dashboard.html
│ └── components/
└── static/insights/
├── css/insights.css
└── js/
The dashboard panels are implemented using a modular framework in the charts/
directory. Each chart is driven by a ChartDefinition dataclass, which encapsulates
all the metadata, collection, and formatting logic required to render it. These
definitions are centrally registered in the insights/charts/__init__.py chart registry.
At request time, the API simply fetches the latest pre-computed snapshot using these
definitions, ensuring instantaneous page loads.
The UI integrates Billboard.js to render interactive charts.
History of Advisories
Advisory data imported from upstream sources is constantly updated. Previously, VulnerableCode only exposed the latest version of an advisory, making it impossible for users to track what changed over time.
This feature introduces a chronological list of advisory versions, allowing users to click into any historical snapshot and view the advisory exactly as it existed at that moment. To provide a quick overview, the UI displays diffs between versions directly.
These diffs are generated at import time. For existing advisories,
they can also be backfilled using the HistoryDiffImproverPipeline. The diffs
are stored in the AdvisoryHistoryDiff model.
EPSS History
This feature adds an EPSS History Trend on the advisory page in the form of a line chart and a paginated history table showing the percentile and score of EPSS over time for a given advisory.
Linked Pull Requests
No. |
Name |
Link |
|---|---|---|
1 |
Insights Dashboard (Part 1) |
|
2 |
Insights Dashboard (Part 2) |
|
3 |
History of Advisories |
|
4 |
EPSS History |
Pre-GSoC Work
No. |
Name |
Link |
|---|---|---|
1 |
Add Mirror Pipeline for EUVD using GitHub Actions |
|
2 |
Document the readme.md for EUVD Mirror |
|
3 |
Add Importer Pipeline for Tuxcare Advisories |
|
4 |
Add Importer Pipeline for Vmware Photon Advisories |
|
5 |
Add Importer Pipeline for Openstack Advisories |
|
6 |
Fix Nix Flake error in GitHub Actions |
|
7 |
Add Importer Pipeline for EUVD |
|
8 |
Add alpine in RANGE_CLASS_BY_SCHEME |
|
9 |
Add support for Openstack Advisories |
|
10 |
Fix missing cwe2 dependency |
|
11 |
Identify and fix env load failures due to improper typecasts |
|
12 |
Researched and documented VulDB API for Vulntotal |
Post-GSoC
The Insights Dashboard can be extended further with additional charts and panels to provide even richer data analytics and visibility. These insights act as a direct feedback loop, helping us continuously refine VulnerableCode by guiding the development of new data pipelines, targeted improvers, and architectural enhancements.
Closing Thoughts
Working on VulnerableCode Insights was a truly rewarding experience that I thoroughly enjoyed. Designing the Insights Dashboard was a particularly thought-intensive process where I found myself drawing inspiration from various insights features like Spotify Capsule and GitHub Insights.
The most amazing part of this journey was the weekly meetings with my mentors. It was an incredibly smooth experience, and I owe a huge thanks to them for all their thoughtful inputs, guidance and feedback:
Mentors: Philippe Ombredanne, Hritik Vijay, Keshav Priyadarshi, Ziad Hany, Ayan Sinha Mahapatra, and Tushar Goel
It was truly amazing to have this opportunity to learn, contribute, and grow with AboutCode this summer, filled with tons of bugs and caffeine.
Until next time!