Measuring Impact#
Measuring impact is more than an exercise in reporting numbers—it is about telling the story of why VERSO matters. From the beginning, we understood that building an Open Source Program Office inside a university was an experiment, and like any experiment, it required evidence to validate its success.
However, the path to measurement was far from straightforward. What we assumed would be easy to track turned out to be irrelevant, incomplete, or misleading. In some cases, the data simply did not exist. Even our initial survey, which revealed that most respondents did not know what open source was, highlighted a fundamental challenge: we were trying to measure progress in a landscape where the baseline was unclear.
One of the earliest examples of this difficulty came from trying to gather statistics from the UVM‑hosted GitLab instance about the number of public repositories. A first‑pass query showed 43 repositories with open‑source licenses—yet one repository, upon inspection, consisted solely of a README stating: “I had no idea we had a GitLab.” The situation on GitHub was even more complicated: while we found nearly 14,000 public repositories referencing “UVM,” attributing them to actual UVM faculty, staff, or students was nearly impossible because most people used personal GitHub accounts. Attempts to catalogue those accounts raised important questions around privacy, autonomy, and the role of a university office in indexing individuals’ digital identities.
Beyond this, broader ecosystem metrics were opaque. UVM’s GitLab instance had low adoption, GitHub attribution was unreliable, and large open‑source projects involving UVM were often co‑owned by collaborators at many institutions. In these cases, what did “UVM ownership” even mean? We recognized that measuring the entire “open‑source ecosystem” was neither concrete nor functional. Instead, we pivoted toward creating our own projects—initiatives where we could define measurable goals, control data quality, and document clear outcomes.
Key Metrics#
From the earliest days of VERSO, we understood that enthusiasm alone could not sustain an academic Open Source Program Office. To justify long‑term investment—financial, organizational, and cultural—we needed metrics that demonstrated real, defensible impact. Over time, our approach evolved into a multidimensional framework reflecting not only program activity but also ecosystem growth, cultural transformation, and lasting institutional value.
Student Engagement & Workforce Development#
Students are the engine of VERSO and ORCA. Their participation and professional growth offer some of the most direct evidence of our success.
Core metrics include:
Total ORCA student participants per year - Unique count of students in the program each year, includes all semesters
2023 - 5
2024 - 25
2025 - 30
Hours students contribute to real open work in ORCA - Total hours worked for paid projects (this does not include volunteer work)
2023 - 338 hours
2024 - 4414 hours
2025 - 3166 hours
Retention - Students returning for multiple semesters by year, this includes if a student continued from a previous year or went into the next year
2023 - 60% (3/5 students)
2024 - 72% (18/25 students)
2025 - 73% (22/30 students)
Career outcomes - Including internships, research positions, or employment related to open‑source skills
This is a qualitative metric as student’s post-graduation are not required to send back any information to ORCA, but in once case a student actually works in UVM in IT, another from the zoning atlas team joined a Town Planning Office, another went to be a Public Works Intern, another to be a Field Geologist.
These metrics illuminate how VERSO prepares students for open‑source practice and supports a workforce trained in real production‑quality tools and methods.
ORCA Student Engagement & Growth Metrics#
Metric |
2023 |
2024 |
2025 |
|---|---|---|---|
Total Student Participants |
5 |
25 |
40 |
Total Hours Contributed |
338 |
4,414 |
7,850 |
Retention Rate |
60% (3/5) |
72% (18/25) |
73% (22/30) |
Project Portfolio & Research Output#
As VERSO’s work expanded, the number and diversity of open‑source projects became a tangible measure of impact. We began tracking:
Total active VERSO and ORCA projects - All repositories (public and private) by year, and approximate projects that had some form of contribution that year. This includes ORCA and Non-ORCA supported projects
2022 - 1 total repos (private and public) ~1 being actively worked on during that year
2023 - 4 total repos (private and public) ~4 being actively worked on during that year
2024 - 23 total repos (private and public) ~17 being actively worked on during that year
2025 - 42 total repos (private and public) ~14 being actively worked on during that year
Project categories - Approximate count of projects by category as of 2026.
Community Tool - 10 Projects
Research Tool - 4 Projects
Data Collection or Visualization - 10 projects
Documentation - 7 projects
Class Exercise - 3 projects
Project Portfolio & Research Output#
Repository & Project Growth
Year |
Total Repositories |
Active Projects |
|---|---|---|
2022 |
1 |
1 |
2023 |
4 |
4 |
2024 |
23 |
17 |
2025 |
42 |
14 |
3. Partnerships & Ecosystem Connectivity#
A healthy open ecosystem can only thrive through collaboration. Over time, VERSO built relationships across UVM and beyond.
Metrics include:
Number of UVM research collaborators - Count of institutes, labs, or a researcher if there is just one by year. This is approximate, this is not stored in an easy to find system. This can also range from consulting conversations to submitting grants to running an ORCA team on a project.
2023 - 25 (1 ORCA Research Project)
2024 - 45 (2 research ORCA project)
2025 - 60 (4 research ORCA projects)
External partners - Collaboration with nonprofits, state agencies, local governments, and other universities either from project exploration to building software. This is approximate, this is not stored in an easy to find system, especially year to year as once a partnership is created it may be in a semi-active state.
2024 - 14 (3 Community ORCA project)
2025 - 25 (10 Community ORCA project)
These metrics reflect how well VERSO strengthens UVM’s connections to regional, national, and international open‑source communities.
Partnership & Ecosystem Growth#
Year |
UVM Research Collaborators |
External Partners |
ORCA Research Projects |
ORCA Community Projects |
|---|---|---|---|---|
2023 |
25 |
0 |
1 |
0 |
2024 |
45 |
14 |
2 |
3 |
2025 |
60 |
25 |
4 |
10 |
Open Data Adoption & Dataverse Utilization#
With the launch of the UVM Dataverse, VERSO gained new visibility into open‑data practices across campus. The UVM Dataverse was only launched in 2025, metrics only will go back to that year
Key metrics include:
Datasets published - Count of datasets and dataverses published by faculty, students, and research units
2025 - 4 Dataverses and 57 datasets
Download and access counts - Count of Downloads, this is a rough estimate based on estimates given a sample size until API reporting is implemented
2025 - ~195 downloads
Number of Users - Count of users and superusers
2025 - 16 users
Citation and reuse metrics - Count of citations where available, showing downstream influence
2025 - Unknown
These measures help track the spread of open‑data culture and provide evidence of real‑world impact.
Institutional Presence & Cultural Change#
Building an open culture is long‑term work, and many of its indicators do not appear immediately. As awareness grew, however, VERSO began monitoring:
New GitHub/GitLab organizations - Count of created Gitlab or Github organizations by labs or research groups where VERSO was involved in the setup or maintenance. Each organization is only counted once
2022 - 1
2023 - 1
2024 - 3
2025 - 4
Growth of openly licensed repositories - Count of Gitlab or Github openly licensed repositories.This metric proved to be impossible to see all repos and while aspirational is not functional
Requests for consultations - Count of class guest lectures by VERSO
2022-2023 Academic Year - 8
2023-2024 Academic Year - 6
2024-2025 Academic Year - 7
Faculty and staff participation - Cumulative count of participants from UVM in open‑work initiatives, workshops, and conferences
2022 - ~10
2023 - ~30
2024 - ~20
2025 - ~30
These indicators provide insight into how open‑source thinking is taking root across the university.
Events, Training & Community Engagement#
As VERSO expanded its educational programs, event‑based metrics became valuable for understanding reach and community dynamics.
Events, Training & Community Engagement#
Year |
Workshops, Presentations & Conferences |
Event Attendance |
|---|---|---|
2022 |
2 |
~20 |
2023 |
7 |
~150 |
2024 |
20 |
~120 |
2025 |
11 |
~240 |
Beyond Numbers#
While metrics provide snapshots of progress, they cannot capture transformation on their own. The number of workshops held or repositories created illustrates activity but not meaning. To fully understand VERSO’s impact, we must also embrace stories, testimonials, unexpected outcomes, and culture shifts—forms of impact that numbers alone cannot quantify.