Merge pull request #6 from manaakiwhenua/framework/procedure/validation

adds draft validation procedure

README.md

@@ -32,7 +32,7 @@ The NZSLUC Framework presents principles, procedures and definitions for designi
     - To be able to record commodities from a set of standard codes (e.g. DataLinker, Stats NZ)
     - To be able to record management practices from a set of standard codes (to be determined)
     <!-- - Tenure -->
-10. To aspire to ensure overall attribute accuracy of greater than 80 per cent.
+10. To aspire to ensure **overall attribute accuracy of greater than 80 per cent**.
 11. To carefully develop these principles, proceedures, definitions and classification systems themselves over time, with concern for backwards compatibility.
 12. To require allowing multiple land uses to be represented
 13. To require allowing land use classification systems to be multidimensional (ref. Rutledge) <!-- Semantic? -->
@@ -52,13 +52,27 @@ The NZSLUC Framework presents principles, procedures and definitions for designi
 
 <!-- Mutlidimensional? What dimensions? -->
 <!-- Proceedure for reporting land use change? -->
-<!-- Independent validation and quality assurance? -->
 - Recommended that published data is independently validated, and quality is described using a standard such as xyz
 - One or more established metadata standards shall be used when publishing data
 <!-- LUML? -->
 <!-- Compatibility with other classification systems in components, e.g. PAN-NZ, ANZLIC for industrial categories, Dairy NZ classes, etc.? -->
 <!-- What is the extent? Marine? What is "land"? EEZ, etc. -->
 
+### Validation
+
+A land use product should not be considered complete until an accuracy assessment has been conducted. Errors can be thematic or spatial, and these errors are not independent. Validation should be reported as a confusion matrix of classification classes, such that users can determine that a geographic unit mapped as class _i_ is class _j_ on the ground for any class _i, j_. This assessment requires that a number of known units are compared to their classified values. This assessment should be made on the basis of ground-truth, local knowledge, and aerial imagery (in descending preference order) and shall not be conducted by those involved in the classification proceedure itself.
+
+Recommended procedure (based on ALUM):
+1. Determine which classes can be validated. Not all classes in a classification system are assessable. Justifications for excluding a class from validation include:
+    1. If land use can only be determined based on tenure rather than physical attributes (typical examples: forms of conservation land; water supply areas)
+2. Conduct a stratified random sample strategy should be conducted, using the map classes as the strata; the sample size within each stratum is in proportion to the occurrence of that class in the land use product. The total sample size shall be limited to a fixed number of total assessments (e.g. 1,000). In practice, the sample may be larger; it should increase with the number of classes within the classification system, the number of available validators, and the size of the intended audience.
+3. The assessment unit is the same as the geographic unit, e.g. a parcel or cell.
+4. Construct an error matrix, comparing land use classes for sampled geographic units with the observed land use classes. Compute the total, user's and producer's accuracies for the map. Also compute 90% confidence intervals for each estimate. (See [[1]](#1).)
+5. Determine if the map meets the required level of accuracy. If the _lower bound_ of the confidence interval for total accuracy is greater than this level (**80%**), then the map meets the framework specification. If any of the upper bounds of the user's or producer's accuracies is less than 50%, then the map may also be considered to have failed the specification, however there may be some scope for judgement in certain classes due to unavoidable confusion between similar classes. If validation fails, perform a re-classification focussed on addressing the worst-performing classes, and repeat the validation proceedure for all clases.
+6. Report the accuracy, including the error matrix and a map of sample sites.
+
+
+
 ## Definitions / Key Concepts
 
 | Concept | Description | Reference(s) |
@@ -82,3 +96,9 @@ Lorem ipsum
 ## Current land use mapping in New Zealand
 
 <!-- TODO refer to a separate Markdown summary, or report section -->
+
+## References
+<a id="1">[1]</a>
+Denham, R. (2005).
+Accuracy assessment for land use mapping.
+Queensland Department of Natural Resources and Mines, Brisbane, and the Bureau of Rural Sciences, Canberra.