Add conformance tests into docs #1417
Conversation
✅ Deploy Preview for fdc3 ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
|
@openfin-johans this might be a good docs PR for you to take a quick look at Johan. |
docs/fdc3-compliance.md
Outdated
| @@ -70,6 +70,28 @@ FDC3 adopts the following experimental features policy: | |||
| 5. Experimental features are exempted from the normal versioning and deprecation policies that govern changes to FDC3. I.e. breaking changes may be made to experimental features between versions of the Standard without a major version release. | |||
| 6. The experimental designation may be removed from a feature in a minor version release (as this will be considered an additive change). | |||
|
|
|||
| ## Conformance testing | |||
|
|
|||
| The FDC3 Standards include a set of [definitions for conformance tests](api/conformance/Conformance-Overview) that may be used to determine if a Desktop Agent API implementation conforms to a particular standard version, to help disambiguate complex parts of the FDC3 Standard and to enable test-driven development of a Desktop Agent implementation. The current set of tests focus on the Desktop Agent API and the interface to it. There are no tests currently defined for the Intents, Context, AppD or Bridging Parts of the FDC3 Standard. Hence, conformance for those parts of the Standard must be determined manually. | |||
There was a problem hiding this comment.
@kriswest, question, am I missing something, in the code for the conformance framework there are tests for intents and context (like this one for intents: https://github.com/finos/FDC3-conformance-framework/blob/main/src/test/v2.0/advanced/fdc3.findIntent.ts). Or am I misunderstanding what you are referring to here?
There was a problem hiding this comment.
There are tests for raising an intent (fdc3.raiseIntent), which is not the same thing as testing that the definiton of an intent conforms to the rules in the standard. Hopefully, that clarifies what I mean. That said, glad you've asked the question as it indicates that I need to revise this copy to avoid confusion!
There was a problem hiding this comment.
Got it! Yes, it would be helpful to revise the wording, Thank you!
There was a problem hiding this comment.
I've updated the copy and just dropped intents and context from it - I couldn't find a great way to word the statement that they are not APIs and don't get implemented so conformance testing is not needed (other than when reviewing to see if they follow guidelines).
|
what's your thinking here, @kriswest? Why do we even want to do this? |
Do you mean why would we want to put the conformance test definitions into the docs? Over the last 6 months or so we've answered numerous clarifying questions to 3 teams implementing Desktop Agents. Of those at least 2, if not all 3 were referred to the conformance tests to clarify the intended behaviour in particular areas - they did not discover the tests on their own (because they are not in the documentation), but did find them useful. Two of those teams have since applied for and achieved conformance, by passing tests that implement the test definitions. Hence, the tests are useful additional detail; by adding them to the docs we make them easier to find and to scrutinize/comment on/suggest extensions and corrections. They are more formally part of the standard. We also want to start having PRs that alter the standard also address necessary changes or additions to the conformance test in the same PR - something we agreed to add to the checklist in the proposed PR template. Having the tests visible in the docs will make that easier. Finally, the tests should be better tied to specific standard versions than they are. This was a good opportunity to break from the current format which blends 1.2 and 2.0 tests (which gets preserved in the 2.0 and 2.1 versions) and to create a cleaner version for 2.2 onwards. I think that will help with your proposed rewrite of the implementation (and presumably definition of cases) in gherkin syntax for use with cucumber. |
resolves #1077
Adds the FDC3 conformance tests into the main documentation and docs for the 2.0 and 2.1 versions. Admonitions are included to indicate that 2.1 uses the 2.0 tests, while 2.2 tests are yet to be defined.
Pages updated:
This is a precursor to creating a version for 2.2 compliance (#1415). When creating that set, I believe we should withdraw the 1.2 tests from the docs pages (will exist as is in the version above) and focus on only the current test set. That has been applied to the 2.2 (/docs) versions already, ready for 2.2 tests to be added and labelled as new to that version.
I have not implemented the conformance test pages for 1.2 as they will require more editing to remove 2.0 tests. If that is needed in your opinion, let me know in a review.
N.B finos/FDC3-conformance-framework#264 should be merged after this to correct links in the confromance framework's readme to point to the new Overview page (replaces the README.md file)