Certificates XML to Frontend

Referenced ReadMes

prototype/README.md contains information on the markup and ontology of the survey xml files
prototype/HOWTO.md how the xml translation magic happens
surveys/README.md contains information for running the ODI extended version of surveyor

File setup

1: there are XML files in prototype/jurisdictions

** these XML files contain artefacts of earlier version of certificates, when the XML was parsed by javascript for the front end**

No, the questionnaire is divided into two sections. The general questionnaire (stored in prototype/translations/certificate.en.xml) containing the questions for General, Techincal and Social sections and the legal portion (stored in prototype/jurisdictions/certificate.GB.xml) for questions relating to copyright and IP ownership that differ by country or legal jurisdiction.

2: there is a surveyor.xsl in prototype/ : this is an XSL transform that translates these XML files into /surveys/odc_questionairre.$COUNTRY_CODE.rb and /surveys/translations/odc.$COUNTRY_CODE.en.yml

the steps involved in this translation are detailed in prototype/HOWTO.md
it does this by running a rake task employing example-translation.xlsx? The prototype/HOWTO.md makes reference to the following
bundle exec rake translations:update TRANSLATIONS=translations.xlsx COUNTRY={country code eg. CZ} LANG={lang code eg. cz}

Q: there is no translations.xlsx file in that directory, is this correct?

the translations.xlsx is saved from google docs when a translation has been made. It's 3 columns the first is an xpath to the node in the file that needs translating, the second column is the English text and the third column is the translation text.

The rake tasks to update translations in the newer version will be bundle exec rake translations:pull[$COUNTRY_CODE] to update translations for a country. bundle exec rake surveys/odc_questionnaire.$COUNTRY_CODE.rb to create the the survey files so the odc.$COUNTRY_CODE.en.xml can be created and uploaded to Transifex

3: the surveyor gem parses the odc_questionairre.$COUNTRY_CODE.rb file, and loads the survey into the following models

Survey, SurveySection, Question, QuestionGroup,Answer, Dependency, DependencyCondition.
which are stored in app/models

Q: the ....$COUNTRY_CODE.rb files are therefore static or likely to be amended if changes are made to any file in the prototype directory

they are generated by the surveyor.xsl so shouldn't be modified directly, they will be updated if the certificate.en.xml etc are changed and then regenerated.

Q: these files are then controlled by which controllers?

The SurveysController deals with starting a survey, and updating it. The ResponseSetsController also deals with a bit.

Q:these models are tested by which tests?

there is a large test suite in the surveyor gem, and the unit tests question_test.rb, response_set_test.rb etc. Things more specific to open data certificates such as the auto certifying is done by certificate_generator_test.rb etc.

Q: Within the surveys directory there is a transform subdirectory containing another surveyor.xsl file, which is 10 lines longer than prototype/surveyor.xsl

A: This is half way through a refactor for translations.

The surveyor.xsl that is in /prototype is the original which processes prototype/translations/certificate.$language_code.xml and prototype/jurisdictions/certificate.$COUNTRY_CODE.xml into the surveyor description file (surveys/odc_questionnaire.$COUNTRY_CODE.xml)

The surveyor.xsl that is in surveys is my new version that has a few changes and uses the questionnaire xml files in surveys/definition/questionnaire.general.xml and surveys/definition/questionnaire.jurisdiction.$COUNTRY_CODE.xml.

The reason it's different is the versions in prototype/ expect the language in the xml to be translated. For example for the Mexican questionnaire prototype/translations/certificate.es.xml and prototype/jurisdictions/certificate.MX.xml will be in Spanish but should otherwise be identical to certificate.en.xml, so when the survey is loaded it stores the text that is in Spanish in the database. This only gives one language option for filling in the questionnaire for Mexico.

In the new version that I'm working on surveys/definition/questionnaire.general.xml and surveys/definition/questionnaire.jurisdiction.MX.xml would both still be in English and when the xml files are translated to surveys/odc_questionnaire.MX.xml it also writes out surveys/translations/odc.MX.en.xml, we can then load that into Transifex (the translation tool we are using) so we can get someone to translate it to Spanish. We can then download and get a surveys/translations/odc.MX.es.xml and the Mexican questionnaire can be displayed in English and Spanish. Enabling us to understand/debug for example the legal section for Mexico without having to understand Spanish.

The reason it's only partly there is it's being done one country at a time, and it's quite a bit of work to go through the history and get to the point where the translations and xml definition is separated out.

Q: Within the surveys directory there is development subdirectory, @pezholio has indicated these are used for testing-should a refactor rename this folder to make this more apparent

could do, I'd move it out of the surveys folder entirely and put it in test/fixtures or test/surveys

Q: Within the surveys directory there is translations directory populated with yml, does this take precedence over prototype/translations

It will in the new setup, in the old setup it was just being put there for the future. There is a line at the start of the surveyor definition that says which .yml files to read translations from. The only working example at the moment is for Czech Republic which works in English and Czech.

Q: the main readme, underneath Surveyor Extensions lists all the models as extensions, with the exception of QuestionGroup, should this be updated?

Q: is the entire process detailed above in steps 1,2,3 run as a background task?

updates are run manually and then committed. When a file such as surveys/odc_questionnaire.GB.rb is updated there is a background task on deploy that checks and runs bundle exec rake surveyor:build_changed_survey FILE=... to reread the questionnaire definition into the database.

#Schema Testing

examples of where correctly formed tags is unclear

<yesno required="required"/>
<yesno required="required" yes="open data sources" />
<yesno required="pilot" yes="anonymised" />

requirement

The requirement attribute decides what level of certificate the answer is required for in order to achieve it. The system has been a few renames so it's unclear but required=required is required in order to achieve a Raw (basic internally) certificate. The other options are pilot, standard, exemplar (Expert level).

attributes

The yes attribute is used on the certificate display page. Take this question:

<question id="thirdPartyOpen" display="This data is created from">
  <label>Are <em>all</em> sources of this data already published as open data?</label>
  <yesno required="required" yes="open data sources" />
</question>

When @display is set on the question element then it will show up with a heading set to that text and if the answer is yes then it will put the text of @yes under the heading. eg

This data is created from: open data sources