Nala automation framework - should we have cucumber into Nala?

[JackySun9]

https://cucumber.io/docs/guides/overview/

To be honest, I like cucumber, because it is clear to know what we are testing and also able to sync to JIRA and TestRail as test spec, no extra effort. I can also use scenario outline to test same logic with different pages, what do you think?

[amauch-adobe]

If we are doing cucumber for test documentation purposes I believe that can be handled anywhere and that it doesn't require gherkin to do that. Documentation can be in Word docs, Jira Tickets, or even in TestRail without having it be tightly coupled with Cucumber/Gherkin scenario and step files. That being said I know many testers like to use Cucumber in many organizations.

My concern with cucumber/gherkin is how one person's scenario/steps may have done the exact thing I needed in a method but the language for the scenario wasn't unique enough. I ran into this issue a couple of times when I put up a couple of PRs before in the Platform UI v1. I was told I needed to make my scenario and/or steps more unique to my test, but I was trying to reuse methods from others that had it in the common directory or in another team's repository. I was looking for the least path of resistance without having to make multiple scenarios/steps that did the exact same thing as other steps/scenarios.

My second concern with cucumber/gherkin is if someone ever slightly changed a scenario/step statement it breaks any other users' tests that were using it. (small edge case, but still possible).

The second issue wouldn't necessarily be a problem though as soon as the project is a consumable tool, tests could be housed in the team's site repository, therefore, eliminating teams from using each other's steps/scenario files except for common core files added in Nala (if any).

My last concern with cucumber/gherkin was how it has been in the past (with my experience anyways) a turn-off for Engineers that weren't QE/QA engineers. On the teams I have been on, my other team members that were engineers didn't want to or feel comfortable adding tests because of the cucumber/gherkin requirements. So when it came to getting PR comments on the tests made the other team members only provided feedback on the actual execution methods. Collaboration wasn't as productive in my experience.

One of the original goals for Nala was to provide a test runner that all could help out. Other engineers could help test and not just QEs.

That being said most of the contributors and users of Nala are QEs, so that goal may not be fully envisioned. If the majority of the Nala community wants it then I say it makes sense to add the support since most desire it.

As a side note Page Object Models are allowed in Nala/Playwright and can be used to reuse methods and templates for multiple pages. Playwright Fixtures are also another option for reuse, and lastly, we can create common files that can be used crossed many tests. Also if a page will need the same test scenario/steps, I may be thinking of this in a limited scope, but you can just add that URL to the feature.spec file that is imported in the test that you want to reuse since the logic is the same.

[JackySun9]

I just list out some benefits for cucumber automation framework:

• Cucumber bridges the understanding and communication gaps among developers, testers, business analysts, customers and product owners.
• Cucumber enables the participation of business stakeholders who do not have technical knowhow.
• Cucumber gives plain text representation which enables easy understanding for non-technical members in the team.
• Cucumber is easy to maintain and is scalable.
• Cucumber increases the reusability of important steps.
• Cucumber promotes teamwork since each individual in the team can contribute.
• Cucumber uses the Gherkin tool which is simple and easy to use.

[chrischrischris]

I'd say that all the points referring to non-devs sound good but ultimately aren't actually used.

My personal opinion is that Cucumber just adds an abstraction layer that isn't needed when only devs work on test scripts. Code re-use can be done with js just as easily as cucumber if organized correctly. I'm not a huge fan - debugging weirdness ends up being harder, but I'm also not the one who will be using / writing nala code so if it's what you guys decide is needed then that works.

[JackySun9]

This is the points from Cucumber product, my real experience, I can quickly pick up testing status to read other's cucumber test.
I can review and contribute that too, it is not about coding, but more on business perspective. I can also invite more people to review if they want

[amauch-adobe]

I'm going to just put out the question, so please don't be offended. How often have business stakeholders that don't have the technical know-how look at the scenario code, the TestRail scenarios/steps, or the Jenkins Zephyr tickets use case/steps? I'm really truly asking that since I have never been asked by a PM or business stakeholder to supply gherkin statements or test cases. Generally, in my experience thus far, they only have cared to know we have a test plan (may look at the test cases) and want to know the results for go/no go situations. Yes, documentation should be there, but how it gets there or what medium is used, I don't think IMO matters too much. If the additional implementation is to simplify things or to ensure the documentation happens, then that is a different issue/argument.

[JackySun9]

If we are open for discussion, we should listen to more voices, if we want to serve a community, we should listen, but not make a decision quickly.

[echampio-at-adobe]

I wouldn't get hung up on Jacky's first two bullet points that mention business folks. Rather, focus on the rest of the arguments. I think the best demonstration of the way the abstraction that cucumber provides has been useful in the past was in the shared test cases for the WDIO framework. From what I saw, the forced separation between the JavaScript step implementation and the tests in Cucumber resulted in the JavaScript steps being more atomic and reusable. Having the easy-to-read Cucumber abstraction, I felt, enabled the QE contractors on DC to understand the commerce tests that Jacky initially wrote more quickly than they would have been able to otherwise. It was also easy to take and modify those Cucumber tests back in Dexter. Without Cucumber, I'm not saying that pure Javascript code couldn't be well decomposed and readable, but even for a developer, I think Cucumber tests are easier to read when coming at something completely cold. We can also write external documentation, but I don't know that taking that approach would be easier than Cucumber. To make another analogy, as a developer, I hate Typescript. I would much rather just write simple Javascript as I've always done. But, whenever I have used Typescript, I think my code was better for it, unfortunately. Sometimes, good can come from having a layer of tooling that enforces certain habits.

[TsayAdobe]

@echampio-at-adobe You made a good analogy.

[TsayAdobe]

Readable and well-structured code is important not only for project managers and stakeholders, but also contractors and short-term employees who work on testing. Since these individuals may come and go, it's essential that they can quickly understand the product and test scenarios using a descriptive language like Cucumber. By ensuring clear and efficient communication through code, we can maintain an agile workflow and rapidly increase automation coverage.

For example, suppose someone is newly assigned to maintain the test case below; they will be able to understand it in a matter of seconds.

  Scenario: Submit the FAAS form
    Given I go to the holiday shopping report page
     When I submit the FAAS form:
       | First name                     | MiloTest                  |    
       | Last name                      | MiloTest                  |     
       | Business phone                 | 777-777-7777              |        
       | Job title or role              | Individual Contributor    |      
       | Functional area/department     | IT                        | 
       | Country/Region                 | United States             |
       | State/province                 | Utah                      | 
       | Organization name              | MiloTestOrg               |   
       | Company Type                   | Other                     | 
     Then I should see "Unboxing the 2022 Holiday Shopping Results" in the page content

[skumar09]

• I agree with Chris point that Cucumber & Gherkin adds additional abstraction layers, which just increases the size of code base and additional maintainability.
• On the points , test scenarios and test cases readability, understandability, test steps description etc..
  • If we add more discipline and guideline to test script creation template [ for example : using test / step title() and description() etc ]
    it is easy to understand the test scripts, steps and the flow.
  • Cucumber/Gherkin's: 'Given-When-Then' structure is quite verbose, and also it add constraint i.e. we have to fit test steps in 'Given-When-Then' structure only.
    • I think we are more used to cucumber and gherkin syntax because of past automation frameworks usage
    • Also from past experiences, I have observed cucumber / gherkin resulting in lot of duplicate and redundant step definitions,

[auniverseaway]

Readable and well-structured code is important not only for project managers and stakeholders

I have never seen a project manager or stakeholder look at a Gherkin file. I've been here 7 years. PMs want to know if the feature is live, they are not looking to see what tests were written.

From my understanding, our current playwright reporting shows what feature is or is not working. We also have the Nala Dashboard @JingleH built for high-level knowledge about a project.

[auniverseaway]

I wouldn't get hung up on Jacky's first two bullet points that mention business folks.

Ok. Fair.

Cucumber gives plain text representation which enables easy understanding for non-technical members in the team.

I don't know anyone non-techical that is looking at Gherkin files. Which is similar to your first two points @echampio-at-adobe said to not get hung up on.

Cucumber is easy to maintain and is scalable.

I don't know if I would agree with this. Based on the PRs in Platform UI I've seen, you have to write strings twice...

Are you saying writing things twice makes something easy to maintain and scalable?

Cucumber increases the reusability of important steps.

Is this unique to Cucumber? I was under the impression we can re-use steps in Playwright / Nala. Can you show me how re-usability cannot achieved with Playwright?

Cucumber promotes teamwork since each individual in the team can contribute.

I think being able to contribute and actually contributing are two different things. Having another language people have to learn is a barrier to entry. Learning a testing framework is already a big enough barrier. I would highly encourage you to look at the contributor list to Platform UI and see how many non-QEs made any contribution. That cannot continue in a Milo world.

When writing Gherkin is not your day job, there's a barrier to entry that I would rather avoid. I think we (you) would get much better adoption / contributions having one less language to learn. This is part of our pivot to Franklin as well. You need five languages to get an AEM component out the door (plus three testing frameworks). With Franklin you only need to know two languages and the two testing frameworks are eerily close at the moment.

I was also under the impression that Playwright also promotes teamwork and contributing. Can we not have teamwork if we use Playwright? Can we not have contributions?

Cucumber uses the Gherkin tool which is simple and easy to use.

I'm not saying Gherkin or Cucumber is not easy to use. What I am saying is that having another layer we have to maintain that is written in another language is probably not the best use of our talents and time and does not promote adoption from more engineers.

[TsayAdobe]

One is a step in the scenario, and the other is the step definition. They are similar to a function reference and the function definition.

[jenssingler]

I agree with @chrischrischris, @skumar09 and @auniverseaway.

Adding another layer adds complexity. If this layer is not used, then it's unnecessary and we should not use it. Especially if it holds back devs, who are not fluent in Cucumber/Gherkin.

And to reiterate Chris' point - I'm here almost 11 years and I also haven't seen a single ticket from the business side containing Gherkin specs.

[TsayAdobe]

The primary objective of having all teams use the same test automation framework is to encourage code sharing and reuse, which requires a certain level of standardization. This can be achieved through an abstraction layer, separation of concerns, and an organized structure. Without these, each team may develop their own approach without reusable code, rendering a unified framework useless. We have already proven that the old model works. All teams can significantly increase their test automation coverage and improve collaboration and efficiency.

[amauch-adobe]

TL;DR

• Cucumber as a base is a means to do BDD test methods, is that what we want for Nala, there are many tried and true test approaches.
• Cucumber isn't "the way", and "only way" to have collaboration, sharing, reuse, and standardization
• Nala is currently under development to be changed to a consumable tool for teams to use against tests housed in their own repositories, the concept of sharing and reuse will be a different type of model than the previous platform.
• There is a need to create, discuss, and decide on approaches, processes, methods, and even tools to build standards across Adobe, but is Cucumber the answer to that?
• We are creating the future not rebuilding a platform with a new test runner, i.e. Playwright, and slapping a new name on it and saying it is the next best thing.

To some degree code sharing amongst the entire project will be a different type of model than it is currently in the previous tools. In the previous automation platform, everyone had their tests in the same repository, so yes, sharing was entirely available and visible to all teams since there was a commons directory, and everyone else's team's scenarios/steps, methods, etc. were there as well. That being said it also created a problem of people using other teams' scenarios because driving down to the actual JavaScript method did exactly what they wanted for the test case so they chose that one (and why not since you should be able to reuse JavaScript code). Then coming to PR time they were pushed back saying the scenario/step definition wasn't unique enough for their test case and created confusion, (for documentation purposes) since the test case text is so tightly coupled to methods. Other times Cucumber itself would complain during execution about step definitions being too similar to one another so it didn't know which one it should grab, which causes a developer to have to write up a scenario so specific that the reality of anyone being able to use it again was a low probability. That is why there were so many step files and step definitions that trying to find something that would work as a shareable scenario, for at least me, took a long time to find and slowed down my work so writing a test took me 2-3x as long. You can easily say that I'm doing it wrong, and you may be right about that, all I'm saying is what my experience has been with Cucumber in the past. Sometimes the sharing works and sometimes it doesn't.

Nala is already under development to become a consumable tool that can be pulled in as a runner for tests that will be housed within the different Milo-based site repositories. All teams will have ownership of their repositories, their own utils/common methods, etc. to use that will not be available without specifically collaborating with another individual and sharing their stuff over into different repositories or pushing up a PR to contribute to the core Nala project so everyone gets it. Contributions to the Nala core project, just like Milo project, will go over various reviews and discussions before something is just added to the core project because someone wants it or because we can do it, we should add it. Which as you mentioned creates an interesting environment for creating standardization. That being said having standards and providing the ability for reuse and sharing is capable by many other means besides just Cucumber. Cucumber isn't "the way" and "the only way" of doing this. For years, engineers have been creating, testing, and releasing products that fit standards, code reuse, and sharing without the use of Cucumber. Cucumber at the base is a means to provide BDD testing approaches, but there are so many other testing approaches out there. I go to testing conferences often and on the Expo floors there are numerous test platforms/tools available that use Cucumber and don't use Cucumber, provide support for Cucumber but also provide other ways as well for developing tests in a standardized way, the reality is there are many ways to do this.

I agree there is a need to create, discuss, and decide on approaches, processes, methods, and even tools to build standards across Adobe. To say that we need Cucumber to do it is a limited foresight in my opinion. I know it has been mentioned it is familiar to those that have contributed to the Platform UI so that is a big reason we should use it...I think we are all smart and intelligent people that can learn different methods. We are to create the future, not rebuild a previous platform with a different type of UI test runner, i.e. Playwright and slap a name on it and say it is the next best thing.

[JackySun9]

if we can build something like Playwright, I should say We Rock! But not everyone or timeline or skills allows us to do that, we are not saying Cucumber is the only way to do this, but it is an option, just like Chat GPT, I can use it or not, up to me. I can take a plane to somewhere, or take a train to somewhere, it is about choice, but the choice is important sometimes.

[npeltier]

I wouldn't use history as an argument for this discussion, as it is not because it has been poorly used, we should not use it. The vision @JackySun9 & @TsayAdobe sell is still appealing to me, and will not be achieved by JS specs.
I still see a lot of value for that additional layer of abstraction, to build common vocabulary we agree on (and implement!) as developer, and use in our tickets, see in our tests. I'm not sure what hurts to try to get cucumber as an option for few tests and see how that works?