docs: a/b testing doc V2 #8501
docs: a/b testing doc V2 #8501
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.OpenSSF Scorecard
Scanned Files |
| - **an optional payload**: A variant can also have an associated payload. Use this to deliver more data or context. See the [payload section](/reference/strategy-variants#variant-payload) for more details. | ||
| - **a value** | ||
|
|
||
| While teams may have different goals for measuring performance, Unleash enables you to configure strategy variants within your application/service and the platform. |
There was a problem hiding this comment.
I don't understand what this sentence is trying to say. Can you rephrase?
There was a problem hiding this comment.
Removing since I make it clear strategy variants are useful for a/b testing and set up in unleash earlier.
|
|
||
| Once you have created a feature flag and configured your A/B test, you can use Unleash to collect insights about the ongoing results of the test. One way to collect this data is through enabling [impression data](/reference/impression-data#impression-event-data) per feature flag. Impression data contains information about a specific feature flag activation check. It’s important to review data from an A/B test, as this could inform you on how (and if) users interact with the feature you have released. | ||
| ### Track A/B Testing for your Key Performance Metrics |
There was a problem hiding this comment.
This section is missing key details, imo, about how this actually works. Let's say you are testing a new design of a signup form, with two versions of the form and tracking submissions of that form in Google Analytics. How would impression data help you determine which version is performing better. This doesn't need to be a tutorial on A/B testing in Google Analytics, but this doesn't give me enough information about how to tie impression data to a conversion event. We should add that.
|
|
||
| This approach minimizes the "regret" associated with allocating traffic to lower performing variants. Multi-arm bandit tests using Unleash can adapt to changing conditions, such as seasonal fluctuations or user behavior changes. In some cases, they can be used to ensure that users are not exposed to suboptimal experiences for extended periods. | ||
|
|
||
| A/B tests are performed safely and strategically with extra safeguards when you automate your flags based on user activity and other metrics of your choice. |
There was a problem hiding this comment.
This is the 2nd time this phrase was used in this section. Suggust changing it.
|
|
||
| ## How to Perform A/B Testing with Unleash | ||
|
|
||
| To follow along with this tutorial, you will need an Unleash instance. If you’d prefer to self-host Unleash, read our [Quick Start documentation](/quickstart). Alternatively, if you’d like your project to be hosted by Unleash, go to [www.getunleash.io](https://www.getunleash.io/pricing?_gl=1*1ytmg93*_gcl_au*MTY3MTQxNjM4OS4xNzIxOTEwNTY5*_ga*OTkzMjI0MDMwLjE3MDYxNDc3ODM.*_ga_492KEZQRT8*MTcyNzQzNTQwOS4yMzcuMS4xNzI3NDM1NDExLjU4LjAuMA). | ||
| To follow along with this tutorial, you will need access to an Unleash Instance. Head over to our [Quick Start documentation](/quickstart) for options, including running locally, or using an [Unleash SaaS instance](https://www.getunleash.io/pricing?). |
There was a problem hiding this comment.
Add a sentence before "To do A/B testing" outlining what the user will get out of this tutorial.
ex. In this tutorial, you will learn how to set up and run an A/B testing using feature flags. You will learn:
- how to use feature flags to define variants of your application for testing,
- target specific users for each test variant,
- manage cross-session visibility of test variants,
- connect feature flag impression data to conversion outcomes, and
- roll out the winning variant to all users.
You will also learn about how to automate advanced A/B testing strategies such as multi-arm bandit testing using feature flags.
There was a problem hiding this comment.
I would generally suggest writing in the present tense whenever possible. I think in this instance, it's good to keep the future tense, but I would suggest going for you'll instead of you will. It's friendlier and shorter
There was a problem hiding this comment.
I think we could try to reduce the number of UI screenshots, as a lot of them are of similar screens. I would suggest using up-to-date screenshots (added them for you in the comments) but only in cases where you think the UI itself + instructions are not intuitive enough.
Co-authored-by: Michael Ferranti <michael.ferranti+unleash@getunleash.io>
Co-authored-by: Michael Ferranti <michael.ferranti+unleash@getunleash.io>
Co-authored-by: Michael Ferranti <michael.ferranti+unleash@getunleash.io>
Co-authored-by: Melinda Fekete <melinda.fekete@getunleash.io>
Co-authored-by: Melinda Fekete <melinda.fekete@getunleash.io>
Co-authored-by: Melinda Fekete <melinda.fekete@getunleash.io>
Co-authored-by: Melinda Fekete <melinda.fekete@getunleash.io>
nnennandukwe
force-pushed
the
use-case-ab-testing-v2
branch
from
5c116b9 to
b311406
Compare
| slug: /feature-flag-tutorials/use-cases/a-b-testing | ||
| --- | ||
|
|
||
| ## What is A/B Testing? |
There was a problem hiding this comment.
@michaelferranti I've made additional updates here based off of Justin's suggestion that we remove "what is [x]?" intros because our UPP will know exactly what these topics are and we can jump right into how we approach the topic sooner.
| - a name: This must be unique among the strategy's variants. You typically use the name to identify the variant in your client. | ||
| - a weight: The [variant weight](/reference/strategy-variants#variant-weight) is the likelihood of any one user getting this specific variant. | ||
| - an optional payload: A variant can also have an associated [payload](/reference/strategy-variants#variant-payload) to deliver more data or context. | ||
| - a value: This describes the purpose of the variant. For example, "Most Read" or "Most Popular". |
There was a problem hiding this comment.
I don't think this is correct. Also, the value is not really the property of the variant, but it's a property of the payload. so the optional payload has a type and value
There was a problem hiding this comment.
Thanks, I'll ask in slack if there's a more clear way to describe this. It's not made very clear in other areas of our docs currently.
There was a problem hiding this comment.
the value is the actual payload
There was a problem hiding this comment.
So this would be an example:
payload type: json
payload value:
{
"color": "blue",
"position: 1
}
There was a problem hiding this comment.
I see! It's the entire payload. It seemed like a parameter of the payload not the whole thing, based on the Unleash admin UI
There was a problem hiding this comment.
Yes the payload 'entity' has a type, and a value, where the value is the actual payload you want to be sent back in the response. But at the moment the above docs say that value is a property of the variant
|
|
||
| Unleash gives you control over which environments you release your feature to, when you release the feature, and to whom. Every team's release strategy may vary, but the overarching goal of A/B testing is to select the most effective experience for users, whether it be a change in your app's UI, a web performance improvement, or backend optimizations. | ||
|
|
||
| When rolling out the winning variant, your flag may already be on in your production environment. Adjust the rollout strategy configurations to release to 100% of your user base in the Unleash Admin. |
There was a problem hiding this comment.
We should suggest archiving the flag one its been rolled out to 100% no? To clean up the code base?
| "items":[] | ||
| "event":"screen_view" | ||
| "app_name":"myAppName" | ||
| "feature":"myFeatureName" |
There was a problem hiding this comment.
I still have a question about how you would account for a variant having multiple flags at the same time. Like if I have a multivariate test that looks at a new version of the a. Nav, b. signup form and c. Marketing copy. Each of these flags could on/off, so there are 9 total variants. how is this accounted for?
There was a problem hiding this comment.
@kwasniew helped me with breaking down the way we'd do it in unleash for multivariate tests:
"Usually each flag can have multiple variants so the relationship is the opposite. If we are talking multivariate testing I think you’d rather have a flag for nav, a flag for sign up form and a flag for marketing copy. Each of those flags would have as many variants as required by each feature. And you’d always record what combination you got for a given user context and send it to the analytics platform and how it converted."
Co-authored-by: Melinda Fekete <melinda.fekete@getunleash.io>
Co-authored-by: Melinda Fekete <melinda.fekete@getunleash.io>
Co-authored-by: Melinda Fekete <melinda.fekete@getunleash.io>
Co-authored-by: Melinda Fekete <melinda.fekete@getunleash.io>
About the changes
Update A/B testing use cases tutorial. Preview link
Closes #
Important files
Discussion points