Redpanda
Data pipelines
vision
Goals
Get leadership buy-in and projects prioritized to build a product experience that will increase user adoption by helping guide users in creating their data pipelines.
Roles
This was a self-initiated project and I was the only person working on the vision. I framed the problem, designed and prototyped the solution (including a visual refresh) and got buy-in from leadership.
Outcomes
Buy-in from leadership (CEO, CTO and VPs of engineering and product management).
Projects have been added to the roadmap and development has started.
Project details
What is Redpanda Data?
Redpanda is a system for moving and processing large volumes of data in real time, with high speed and reliability. It connects data sources such as payment systems, trading platforms, or customer apps to destinations like fraud detection models, risk dashboards, or analytics tools.
Example use case: Financial institutions could use Redpanda to stream credit card transactions as they happen, run them through fraud detection algorithms in milliseconds, and immediately alert customers or block suspicious activity.
Critical User Journey
01
Problem
76% of new users were able to do an initial reading and writing of data, however, this percentage drops to 15% at the next milestone of 1 MB of total data. The majority of new users are failing to send a meaningful amount of data.
02
Hypotheses
The majority of new users were able to successfully complete first write and read of data due to the hello world experience; on sign-up a sample cluster with data is created and the user is presented with clear and simple instructions to read and write a little data using the command line.
03
Factors contributing to drop-off
- There is little in-product guidance after the hello-world experience
- Users have to navigate to different sections of the application and understand how different functionality would be used to create their pipeline
- Connector experience gives little context and guidance to understand and create a connector
- Once a data pipeline is created there is no way to view and understand the overall status or connections. Users have to navigate to the different sections to get visibility into parts of the pipeline.
04
Solutions
Solution 01
Create an over-arching Data Pipelines construct
Facilitate end-to-end pipeline creation (user's use cases) by creating a high-level abstraction called a “Data Pipeline,” which will serve as the primary construct. This Data Pipeline will unify and visualize Redpanda's various functionalities, guiding users through the process instead of requiring them to navigate disparate components within a cluster.
Solution 02
Guide users in creating their pipeline
Guide users in creating their pipeline by focusing on what users are trying to accomplish instead of making users navigate and understand disparate functionality. This is accomplished by leveraging templates and using language that matches user goals, for example, "Reading from and writing data".
Solution 03
Connector catalog and guided flow
Data source and destination connectors are easily discoverable and have a guided flow that includes form creation.
Solution 04
Leverage AI
At Redpanda Data we have started building AI products for our customers, but had not thought deeply about how to leverage AI to help our users with their existing use cases.
The vision shows how we can use AI within our product to allow users to describe the pipeline they want to build and have it easily scaffolded. They can then use AI to iterate or directly make edits in the editor before deploying.
Solution 05
Role based navigation
One issue with the current navigation is that it encapsulates all functionality in clusters (infrastructure). Redpanda Data has many type of user roles / personas that are interested in different parts of the product, for example, Data engineers are interested in data (topics) and DevOps may be more interested in resources (clusters and networks).
The proposed navigation aligns more with user roles and allows bypassing of the infrastructure level.
05
Alternative directions considered
During the solutioning phase, I looked at multiple directions, for example:
Stepper wizard
This had a very guided experience for the user, however, due to the complexity and flexible nature of pipelines the steps involved would change dramatically based on the use case.
Tutorial
This would be a tutorial panel to guide user across the different parts of creating a data pipeline. It would help users navigate and understand the different parts, however, this jumping around would make users lose context, especially for complex pipelines.
06
Final solution
After creating the prototype I started sharing it with my manager and other key stakeholders, including the CTO. I then presented to the CEO, CTO, VP of engineering and VP of Product Management. I got buy-in from leadership and then added several projects to our roadmap and backlog. Development has started on the foundational pieces.
07
Outcomes
After creating the prototype I started sharing it with my manager and other key stakeholders, including the CTO. I then presented to the CEO, CTO, VP of engineering and VP of Product Management. I got buy-in from leadership and then added several projects to our roadmap and backlog. Development has started on the foundational pieces.