CHAOSS Metrics

Welcome to the CHAOSS Metrics repository. CHAOSS Metrics repository captures metrics for assessing open source community health and sustainability. Such metrics are aimed at understanding project diversity & inclusion, growth-maturity-decline, risk, and value. For more information about the CHAOSS project go to our website at: https://chaoss.community/

Goals of CHAOSS Metrics

(1) Capturing metrics based on the work of community members who have participated at CHAOSS events, worked in the repo, and discussed through on email. We capture the metrics that people find meaningful to their particular contexts when understanding project health and sustainability. In this, we work to represent metrics through concise definitions, known uses cases, sample visualizations, and sample implementations. The CHAOSS Metrics repository captures all potential metrics. It is the WGs that make these metrics meaningful, leading to our second goal.

(2) Participating with the CHAOSS workgroups. Through these workgroups, we consolidate metrics in meaningful ways and inform the metrics by questioning how to capture and deploy (or not) them:

CHAOSS Diversity & Inclusion Workgroup

CHAOSS Growth, Maturity, Decline Workgroup

CHAOSS Value Workgroup

CHAOSS Risk Workgroup

Some Reasons to Assess Community Health

• Risk Mitigation
• Track Corporate Engagement
• Identify Sustainable Projects
• Identify Single Points of Failure
• Avoid In-take of an Inactive Project
• Identify Open Source Projects that Need Support
• Assess Value Generated through Community and Engagement
• Show that Active Community Management Bears Desired Results

Some Issues To Look At

• Project Maturity
• Project Viability
• Growth of Community
• Momentum of Community
• Diversity of Community
• Timeliness of Maintainers
• Attentiveness of Maintainers
• Activity Level - Responsiveness
• Distribution of Code Contributions
• Vanity metrics (might have use in other cases, e.g. stars)
• Ecosystem Health (upstream, downstream, and related projects)
• Aggregate Project-tree Health (combined health metrics of all linked dependencies)

Some Contexts to Consider When Evaluating Health

• Value Derivation
• Style of Project
• Project Comparison
• Maturity of Project
• Programming Language
• Quality of Ecosystem
• Community Composition

Full List of Activity Metrics

The following is a full list of identified metrics. How the metrics live in practice is work that happens in the workgroups.

Please note that there is a folder called "metrics_scratchpad" underneath the activity-metrics folder. This directory is for metrics that are nascent ideas but have not been through our metric naming and classification process yet. We are trying to name all the metrics according to the types of data they characterize. For example, metrics related to "code" are prefixed with "code-", etc.

Name	Question/Description	Deployment in CHAOSS Workgroup
Accepted Code Contributions	Percentage of new contributor code versus total code over time.
Age of Community	Time since repository/organization was registered; or time since first release (Chengalur-Smith et al., 2010; Grewal, Lilien, & Mallapragada, 2006).
All Licenses	List of licenses.
Alternatives	Are there a variety of communication channels?	D&I
Apache Maturity Model	The Apache Project Maturity Model guidelines for assessing the maturity of a project.
Attendees Demographics	How diverse are the attendees?	D&I
Availability of Add-on Products	Availability of 3rd party plug-ins, modules, utilities, etc. that provide additional functionality for the project's software.
Average Issue Resolution Time	The average amount of time it takes for issues to be closed.
Average Time of First Maintainer Response to Code Merge Request	The average amount of time it takes for a maintainer to make the first response to a code merge request.
Average Time of First Response to Issue	The average amount of time it takes for the first response to an issue.
Average Time of Open Issues	The average amount of time open issues have remained open.
Board/Council Diversity	What is the diversity within our governing board or council?	D&I
Blog posts	Number of blog posts that mention the project.
Bug Age	Age of known bugs in issue tracker.
Bugs after Release	Number of bugs reported after a release.
Bus Factor	The number of developers/organizations it would need to lose to destroy its progress.
Captioning	Do we provide text captioning for spoken communication?	D&I
Change in Maintainer Number	Number of maintainers added/removed over time.
CII Best Practices Badge	The CII Best Practices Badge Program provide maturity self-certification: passing, silver, and gold.
Closed Issues	What is the number of closed issues?	GMD
Closed Issues New Contributors	What is the number of persons closing an issue for the first time?	GMD
Closed Issue Resolution Duration	What is the duration of time for issues to be resolved?	GMD
Code Commits	What is the number of code commits?
Code of Conduct at Event	How does the Code of Conduct for events support diversity and inclusion?	D&I
Code of Conduct Enforcement	Is enforcement process running at scale (volume, responsiveness, accuracy, fairness)?	D&I
Code Merge Duration	What is the duration of time between code merge request and code commit?
Code Modularity	Modular code allowing for parallel development (Baldwin & Clark, 2006).
Code Review Efficiency	What is the number of merged code changes/number of abandoned code change requests?	GMD
Code Review Iteration	What is the number of iterations that occur before a merge request is accepted or declined?	GMD
Code Reviews	What is the number of code reviews?	GMD
Collaboration Style	How inclusive is community collaboration?	D&I
Commercial Offerings	Availability of commercial products or services based on the project.
Commit Bias	Acceptance rate and time to acceptance differences per gender, ethnicity, and relevant diversity characteristics.
Communication Channels	How welcoming, responsive, respectful are interactions even on hot topics of debate? What is the diversity of voices speaking/being heard?	D&I
Community Activity	Contribution Frequency. Contribution = commit, issue, comment, etc).
Contributing Organizations	What is the number of contributing organizations?	GMD
Contribution Acceptance	Ratio of contributions accepted vs. closed without acceptance.	GMD
Contribution Age	Time since last contribution. Contribution = commit, issue, comment, etc.).
Contribution Diversity	Ratio of code committed by contributors other than original project initiator. Contributions going up beyond the core team.
Contribution Sentiment	What are the stars, thumbs up, sentiment in comments?	D&I
Contribution-Type	Coding, Quality Assurance/Testing, Localization/L10N, Diversity & Inclusion, Event Organization, Documenting, Community Building/Management, Teaching, Trouble-shooting/Support, Creative/Design, Social Media, Writing Articles, Bug Triaging, UI/UX, Security/Campaign Advocacy	D&I
Contribution Type	Does recognition skew to a particular kind of contribution?	D&I
Contribution Volume	Do we have a bias towards small contributions or multiple contributions?	D&I
Contributor Activity	Activity level of individual contributors.
Contributor Breadth	Ratio of non-core committers (drive-by and peripheral committers).
Contributor Demographics	Gender, age, location, education, and skills.
Contributor Diversity	Ratio of contributors from a single company over all contributors. Also described as: Maintainers from different companies. Diversity of contributor affiliation.
Contributor Importance	Percentage of commits by individual contributors from identified organizations.
Contributor Seniority	For each active contributor, time since first contribution. Experienced contributors providing value to the community, since they carry with them (in part) the history of the project.
Contributors	What is the number of contributors?	GMD
Copyright Declaration	The degree to which the project properly declares copyright ownership, including the copyright symbol or 'copyright' word, the year of the creation, the name of the author, and a rights statement.
Decision Distribution	Central vs. distributed decision making. Governance model, scalability of community.
Dependency Depth	Number of projects included in code base + number of projects relying on focal project. Indicator about centrality in open source dependency network.
Distribution of Work	How much is recent activity distributed?
Diversity Access Tickets	Are Diversity Access Tickets offered for an event?	D&I
Documentation	What is the thoroughness, and accessibility of documentation according to a set of criteria?	D&I
Downloads of Non-software Artifacts	Number of downloads of non-software artifacts (e.g. documentation, sample apps, test suites, etc).
Elephant Factor	The minimum number of companies whose employees perform 50% of the commits
Family Friendliness	Does the event empower those who care for children to attend?	D&I
File License Declarations	A list of license declarations on the software package files.
First Response to Issue Duration	Time between a new issue is opened and a maintainer responds Also called: bug response rate. The maintainer is believed to not “pile on” but try to solve an issue	GMD
Followers	Number of followers.
Forks	Number of forks.	GMD
Foundation Staff Diversity	What is the diversity of foundation staff?	D&I
Gatherings	Number of face-to-face/in-person meetings per year.
Installs	Number of software installations of the project.
Issue Comments	Number of comments per issue.
Issue Resolution Efficiency	What is the number of closed issues/number of abandoned issues?	GMD
Issues Submitted/Closed	Issues submitted vs. issues closed. Example.
Issue Tracker	How well is a project issue tracker setup to invite new contributors, skilled contributors, non-technical contributors?	D&I
Job Postings	Number of job postings that mention the project as a preferred or required skill.
Known Vulnerabilities	Number of reported vulnerabilities. Could be limited to issue-tracker or extended vulnerability databases (e.g. CVE).
Language Bias	Bias against gender and ethnicity in use of language.
Language Makeup	Makeup of a project in terms of whitespace, code, and comments.
Leadership Demographics	Demographics of project's leadership (e.g. Board, Technical Steering Committee, Maintainers, etc.) over time.
License Conflicts	Project containing incompatible licenses.
License Count	Number of licenses.
License Coverage	Number of files with a file notice (copyright notice + license notice).
License Declared	What license does the project declare?
License Identification Methods	A list of methods or tools used for identifying licenses in files.
Lines of Code Changed	What is the number of lines of code changed?
Listening	How well do our mechanisms for listening accommodate community?	D&I
Maintainer Promotion	Last time a maintainer was added.
Maintainer Response to Merge Request Duration	What is the duration of time for a maintainer to make a first response to a code merge request?	GMD
Mode Alternatives	What alternative communication modes do we offer?	D&I
New Contributing Organizations	What is the number of new contributing organizations?	GMD
New Contributions	Percentage of contributions (patches, pull requests, etc.) from new contributors vs all accepted contributions over time.
New Contributors	What is the number of new contributors?	GMD
New Contributors of Commits	What is the number of persons contributing with an accepted commit for the first time?	GMD
New Contributors of Initiated Code Reviews	What is the number of persons initiating a code review for the first time?	GMD
New Contributors of Code Reviews	What is the number of persons contributing with reviews of code for the first time?	GMD
New Contributors on the Email List	What is the number of persons posting messages in mailing lists for the first time?	GMD
New Contributors* vs Maintainers**	Ratio of new contributors to maintainers over time.
Non-source Contributions	Track contributions like running tests in test environment, writing blog posts, producing videos, giving talks, etc.
Number of Active Users	Number of active users of the project.
Number of Contributing Organizations	Number of organizations participating in the project over time.
Onion Layers	Distance between onion model layers (users, contributors, committers, and steering committee). Rule of thumb: factor of 10x between layers. (OSLS'17 Node.js keynote).
Open Issues	What is the number of open issues?	GMD
Open Issues New Contributors	What is the number of persons opening an issue for the first time?	GMD
Open Issue Age	What is the the age of open issues?	GMD
Package License Declaration	A list of license declarations on the software package.
Paid Developers	Number of paid developers in community over time.
Path to Influence	What opportunities are there to move into governance?	D&I
Path to Leadership	A communicated path from lurker to contributor to maintainer.
Path to Maintainership	Path to maintainership published.
People Opening Issues	How many people are opening issues?
Project Life Cycle	Community assigned label. For example: proposal, incubaton, active, deprecated, end of life (Source: Hyperledger).
Perceived Value	How are we valuing contributions and contribution types?	D&I
Percentile Distribution of First Maintainer Response to Code Merge Request	The proportional frequency of time it takes for a maintainer to make the first response to a code merge request.
Percentile Distribution of First Response Time	The proportional frequency of time it takes for the first response to an issue.
Percentile Distribution of Issue Resolution Time	Proportional frequency of closed issue time duration.
Percentile Distribution of Open Issue Time	Proportional frequency of open issue time duration.
Percentile Distribution of Time to Merge Code	Proportional frequency of code merge to upstream time duration.
Pony Factor	The minimum number of developers performing 50% of the commits. The Math
Pull Request Comments	Number of comments per pull request.	GMD
Pull Request Comment Duration	The difference between the timestamp of the pull request creation date and the most recent comment on the pull request.	GMD
Pull Request Discussion Diversity	Number of different people discussing each pull request.
Pull Request Made vs. Closed	Pull requests made vs. pull requests closed Example. Encompasses number of pull requests rejected.
[Pull Requests Merged]	Number of Code Commits
Pull Requests Open	Number of open pull requests.	GMD
Pull Requests Over Time	How many pull requests have been submitted over a specified time period?
Qualified Committers	Contributions over time and what components they commit to over time.
Quick Links	No Question/Definition	D&I
Recognition Type	No Question/Definition	D&I
Recognition Value	Do different demographics value different types of recognition?	D&I
Relative Activity	Ratio of GH issues+comments, GH pull requests+comments, and GH commits for the project members and for the non-project members. Compare the activity between committers-as-a-group and contributors-as-a-group. It shows when a project is not yet popular or when a project is not paying attention to its users.
Release Maturity	Ratio of major and minor releases.
Release Note Completeness	Number of functionality changes and bug fixes represented in release notes vs. release. Good for users, also shows diligence of community.
Release Velocity	Time between releases. Regular releases are a reliability indicator.
Reopened Issues	Rate of issues closed but discussion continues or issues that were closed and re-opened.
Repository Size	Overall size of the repository.
Response Times & Quality	How quickly and well do we respond to suggestions, PRs, questions?	D&I
Retrospectives	Existence of after release meetings. Collect lessons learned, improve processes, recognize contributors.
Review Efficiency	Number of merged patches / number of abandoned patches over time.
Rewards	Rewards, shout-outs, recognition, and mentions in pull-requests or change logs.
Roadmap	Existence and quality of roadmap. Best practice as community engagement and scalability (might not be automatically computable).
Role Definitions	Existence and quality of role definitions. Relates to "Path to Leadership".
Sentiment	What is the sentiment within external communication channels regarding our own press releases and within our internal communication channels, e.g., mail lists or IRC?	D&I
Size of Code Base	Lines of code.
Software Downloads	Number of project software downloads. Beware: downloads might be skewed by builders. Used as measure for success (Grewal, Lilien, & Mallapragada, 2006).
Speaker Demographics	How well does the speaker lineup for the event represent a diverse set of demographics?	D&I
Speaking	Are we speaking to our communities in an accessible range?	D&I
Stack Overflow	Number of questions asked, response rate, number of responding people that have verified solutions.
Stars	Number of stars.
Sub-Projects	What is the number of sub-projects?	GMD
Team/Module Ownership Diversity	What is the diversity of other bureaucratic and administrative foundation teams, e.g. working groups, committees, or ambassador groups?	D&I
Test Coverage	Percentage of codebase covered by developer tests.
Time to Contributor	Time to becoming a contributor.
Technical Jargon	Does the language skew to technical-confidence vs. technical ability?	D&I
Total Contributing Organizations	The total number of organizations contributing over time.
Total Contributors	The total number of contributors over time.
Total New Contributing Organizations	The total number of new organizations contributing over time.
Total New Contributors	The total number of new contributors over time.
Total Sub-projects	The total number of sub-projects over time.
Transparency	Number of comments per issue. Discussion is occuring openly.
Unity	Rivalry or unity of community.
Update Age	Time since last update.
Update Rate	Number of updates over time.
Update Regularity	How consistently and frequently are updates provided.
Use of Acronym	Frequency of acronyms used as a barrier for new contributors.
User Dependency	Number of users who are aware that they depend on the software over time.
User Groups	Number of user groups that perform a variety of crucial marketing, service support, and business-development functions at the grassroots level\ (Bagozzi & Dholakia, 2006)
V-index	An index of project's first-order and second-order downstream dependencies. Example
Watchers	Number of watchers.
YouTube Videos	Number of Youtube videos that mention or specifically deal with the project (e.g. tutorials).

How to Contribute and Participate to the CHAOSS Project

Contribute and Participate. If you would like to propose a new metric to this repository, open a new issue and provide a pull request to open the discussion about inclusion. If you would like to provide metric details, we have a metric-template to be used. Follow the same approach regarding an issue/pull request to have your detail changes discussed and potentially merged.

Repository Maintainers

• Dawn Foster
• Georg Link
• Matt Germonprez
• Ben Lloyd Pearson

How to become a maintainer

License

All contributions to implementation-agnostic metrics and standards, including associated scripts, SQL statements, and documentation, will be received and made available under the MIT License (https://opensource.org/licenses/MIT).