Metrics Index | Requirements & Use Cases

[j12y]

Using this thread for brainstorming / requirements gathering. Add new ideas and/or vote up and comment on ideas you like that were suggested.

If you had the perfect resource for discovery of metrics to measure the performance and impact of devrel programs what would you find?

• is it a database?
• is it a book?
• is it a playbook / series of articles?
• is it a tutorial?
• is it a framework or standard?
• is it a report template?
• is it a collection of scripts for ETL reusable with tools?
• is it a community?

fyi: @divya-mohan0209 @liz-acosta @HackyRoot

[j12y]

It wouldn't just define metrics, but would show practical examples of tools to collect the data and how to set them up, what a good benchmark value would be when things are going well, and how to report on it to stakeholders.

[anajsana]

Perhaps @divya-mohan0209 already mentioned this from CHAOSS, but one of the first things I learned when contributing there (and to GrimoireLab) was the Goal–Question–Metric (GQM) approach. Even though CHAOSS has evolved over the years, that underlying approach has remained. I still use it whenever I feel lost and don’t know what to measure. In fact, I based my entire thesis (which focused on measuring the value of DevRel in open source communities) on this approach.

That’s why I think a metrics index should start by defining an approach as a start. It could be GQM, which has already been validated in scientific research, or another one. Personally, I like using approaches grounded in research because they provide a more neutral and objective space.

[anajsana]

https://anajimenezsantamaria.gitlab.io/assets/img/Thesis.pdf

[j12y]

Super cool @anajsana, I didn't realize you had done research on this topic. Thanks for sharing and for the good read.

[arun-gupta]

I enjoyed reading the study as well @anajsana, it is very targeted at OpenDistro. I helped launch that project back in 2020 during my time at AWS and never knew this document existed :)

[palermo4]

I would like to provide the example of data-driven sentiment. In short, sentiment can be define as either positive, neutral, negative - or it could extend out to specific emotions such as angry, grateful, curious... AI analysis is typically used to analyze a given text and attribute it a sentiment. Text can come from feedback at an event, messaging platforms (like Slack), social media, forums, blog comments, emails. When aggregated and categorized by source, this can give a broad touch point of your developer community.

According to recent research, vanity metrics do not appeal to many executive stakeholders. Yet when presented an aggregated sentiment of all the devrel engagements, this gives a pulse of what moves sentiment in one direction or another. This also drives great conversations among stakeholders, such as "Why did sentiment go down on our blog comments this month?" The answer could be great feedback to the product team due to poor devex.

Tools like CommonRoom.io provide sentiment analysis (Positive/Negative). Here is a link to other tools that are out there.

[j12y]

I may have already listed this in another item, but to emphasize the key points like you provide in terms of identifying tools...

• how do I set up common room (or alternative) to gather this value?
• how do I track that value over time?
• what is a good benchmark value for me to see for my industry? or should I just evaluate it as a trend of up and to the right? what is a benchmark for a % increase quarter over a quarter?
• if I need to report on this, what should a report / deck / dashboard look like that has been used effectively for stakeholders (with perhaps fudged actual data)?
• if I don't like the value of my metric, what community engagement tactics, content metrics, etc. should I consider to move it in the direction i want it to go

These are hard questions, but I think what often is not described in other metrics presentations, blog posts, etc. that I've seen. I'd love to see a playbook that goes from metric identification through execution.

[j12y]

I'd use the Metrics Index to demonstrate all the metrics as a way of evaluating the Developer Relations Foundation itself, reporting on all the key indicators as a way for others to learn them that is transparent and open by natural extension of it being this same community.

[divya-mohan0209]

From my experience with the CHAOSS project where we have metrics and metric models to measure community and project health, it’d be a combination of the below

• A community of practitioners and enthusiasts coming up with + refining the metrics
• Central documentation that hosts the detailing of each of these metrics
• A tool that helps visualise these metrics

[j12y]

The work that the CHAOSS community has been doing is quite relevant and interesting for sure.

Is this the right place to start for somebody interested in learning more?
https://chaoss.community/kb-metrics-and-metrics-models/

[arun-gupta]

This work only provides metrics from the community perspective. Community is an essential and necessary part of DevRel, but there are many more aspects to DevRel.

[arun-gupta]

There are leading and lagging metrics. Leading predicts future success and provides early warning signals about program performance. Lagging metrics confirm past performance and show the ultimate results of your efforts. Quite often, leading metrics are confused with vanity metrics, but they are still very useful for showing progress in the activation and engagement parts of the funnel. Lagging metrics are beneficial for retention and are directly relevant to the business people. Remember, there is no retention without activation and engagement. So, both need to be measured.

I have categorized 150+ such metrics categorized under activation, engagement, and retention. Over my different DevRel companies, there is no one-tool-that-fits-them-all. We can possibly do that for leading metrics, but lagging metrics are very specific to the business.

Here is my wishlist:

• Define different stages of the developer funnel
• Define a base set of metrics for different stages of the developer funnel
• Have a clear description for each of the metrics
• Define which ones are leading vs lagging (how much lag)
• Provide a standard way that these metrics should be tracked and what a potential self-service dashboard could look like
• Timeseries dashboard that allows anybody to generate weekly, monthly, quarterly, annual, or whatever time period reports (think like https://devstats.cncf.io/)
• Define a community pulse that is a weighted sum of these metrics
• A tool that helps visualize these metrics

[arun-gupta]

Adding the complete set of metrics here, as the referenced repository is now archived.

metrics-framework.md

[j12y]

42

[j12y]

I'd like to see a Getting Started Guide for DevRel Program Metrics.

What are the TWO (2) metrics I can start measuring today, how do I get that data, how do I report on it and explain its impact, and what is a good benchmark for what I want that metric to be 90 days or 12 months later.

My ideal requirement is not something comprehensive, but something simple with a high probability of success for all programs.

[arun-gupta]

The Getting Started Guide is a great idea.

I was listening to Meta CMO Alex Shulz on 20VC this morning. He called out one metric that matters is retention. Full episode available here.

Created #139 to track this.

[j12y]

The Metrics Index would guide me in selecting a tool (spreadsheet perhaps) for a small organization to begin tracking something impactful over time. What should that metric be? What would that tool be?

Constraints:

• what would give me a 99% chance of success in getting to hello world for the measurement program?
• can I get to hello world measuring something in 5 days or less?

That would be a good proof point.

[j12y]

This may have just restated another comment I added before in a slightly different way:
#114 (comment)

[j12y]

I added an example to #139 using blog post publish rate for illustration