Developing GitLab Duo: AI Impact analytics dashboard measures the ROI of AI

Generative AI marks a monumental shift in the software development industry, making it easier to develop, secure, and operate software. Our new blog series, written by our product and engineering teams, gives you an inside look at how we create, test, and deploy the AI features you need integrated throughout the enterprise. Get to know new capabilities within GitLab Duo and how they will help DevSecOps teams deliver better results for customers.

As organizations adopt GitLab Duo, our suite of AI features to power DevSecOps workflows, business and engineering leaders need real-time visibility into the technology's ROI. Granular usage data, performance improvements, the trade-off between speed, security, and quality, and other productivity metrics are essential to evaluate the effectiveness of AI in software development. That's why we created the AI Impact analytics dashboard for GitLab Duo, available in GitLab 17.0, as a new way to measure the ROI of AI.

Take an interactive tour of the AI Impact analytics dashboard.

Understanding the ROI of GitLab Duo AI-powered capabilities

To properly evaluate AI's impact on the software development lifecycle, organizations have told us they want to:

• visualize which metrics improved as a result of investments in AI
• compare the performance of teams that are using AI against teams that are not using AI
• track the progress of AI adoption
• automate insights extraction from a large volume of performance data

AI Impact analytics dashboard features these capabilities and more with customizable visualization, which enables teams to:

• Monitor AI adoption: Observing AI adoption rates enables organizations to evaluate organizational strategies to maximize the ROI on their technology investments.
• Track performance improvements: By tracking performance metrics and observing changes after the adoption of AI, leaders can quickly assess the benefits and business value of AI features.

What is the AI Impact analytics dashboard?

In this first release of the AI Impact analytics dashboard, we focus on providing insights and metrics about GitLab Duo Code Suggestions adoption, including:

• Detailed usage metrics: Discover the ratio of monthly Code Suggestions usage compared to the total number of unique code contributors to know how deeply Code Suggestions is adopted within your teams.
• Correlation observations: Examine how trends in AI usage within a project or across a group influence other crucial productivity metrics, displayed for the current month and the trailing six months.
  • For this correlation analysis we added a new metric "Code Suggestions Usage Rate" as the Independent Variable (the cause). The monthly Code Suggestions Usage Rate is calculated as the number of monthly unique Code Suggestions users divided by total monthly unique contributors. GitLab considers the total monthly unique code contributors, which means only users with pushed events are included in the calculation.
  • As Dependent Variables (the effect), we added these performance metrics: Cycle Time, Lead Time and Deployment Frequency. And as Quality and Security Metrics, we added Change Failure Rate and Critical Vulnerabilities.
• Comparison view: Understand the difference in the performance of teams that are and are not using AI, and manage the trade-off between speed, quality, and security exposure.

What’s next for the AI Impact analytics dashboard?

Looking ahead, we have exciting plans to expand the capabilities of the AI Impact analytics dashboard. Here are some of the highlights:

1. New tile visualizations such as "GitLab Duo Seats: Assigned and Used," "Code Suggestions: Acceptance Rate %," and "GitLab Duo Chat: Unique Users" to gain a deeper insight into usage patterns for GitLab Duo.

2. New comparison bar chart to help users observe how changes in one metric correlate with changes in others:

3. AI statistics in the Contribution analytics report to understand how users interact with AI features. See which users are leveraging AI features and whether their performance has changed over time:

Get started today

We're excited about the potential of the AI Impact analytics dashboard to not only demonstrate the real-world business outcomes of AI but also to drive more informed decisions regarding future AI as optimization for the DevSecOps lifecycle. For more information about what is coming next and to share feedback or questions, please visit our AI Impact analytics dashboard epic.

Start your free trial of GitLab Duo and the AI Impact analytics dashboard today.

Read more of the "Developing GitLab Duo" series

• Developing GitLab Duo: How we validate and test AI models at scale
• Developing GitLab Duo: How we are dogfooding our AI features
• Developing GitLab Duo: Secure and thoroughly test AI-generated code
• Developing GitLab Duo: Blending AI and Root Cause Analysis to fix CI/CD pipelines

Disclaimer: This blog contains information related to upcoming products, features, and functionality. It is important to note that the information in this blog post is for informational purposes only. Please do not rely on this information for purchasing or planning purposes. As with all projects, the items mentioned in this blog and linked pages are subject to change or delay. The development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab.