Merge pull request #5 from rolkar/feature/doc

Feature/doc

README

@@ -1,11 +1,21 @@
 This project contains tools for manipulating X3F files from Sigma cameras.
 See doc/readme.txt and doc/copyright.txt.
 
-To run the code:
-Download the code
+-------
+COMPILE
+-------
+
+Download the code.
 run 'make'
 <wait>
-The bin directory contains binaries that can be run directly from the command line.  Alternatively, you can create binaries for multiple platforms by calling:
+The bin directory contains binaries that can be run directly from the
+command line.
+
+-------------
+DISTRIBUTIONS
+-------------
+
+Alternatively, you can create binaries for multiple platforms by calling:
 
     make dist-all
 
@@ -15,16 +25,40 @@ as well as the platform specific calls:
     make dist-32
     make dist-64
 
-If you don't want to install all the dependencies, you can also install and run in a vagrant.  Doing so requires downloading Vagrant from http://www.vagrantup.com.  Once you've done so, the commands to run the code are:
+The distribution will be found in the file
+
+    dist/x3f_tools-<VER>-<PLATFORM>
+
+The <VER> part is either the git hash, or if there is a tag on current
+commit, that tag.
+
+-------
+VAGRANT
+-------
+
+If you don't want to install all the dependencies, you can also
+install and run in a vagrant.  Doing so requires downloading Vagrant
+from http://www.vagrantup.com.  Once you've done so, the commands to
+run the code are:
 
     vagrant up
     <wait>
     make
 
+-----
+TESTS
+-----
+
 If you want to run tests, type:
 
     make check
 
-The tests require that a python virtual environment be installable on the current system.  That precursor can be met either by installing `pip` and `virtualenv` on your local box, or by running through the vagrant.  Once that requirement has been met, the makefile will ensure that the other packages are installed properly.
+The tests require that a python virtual environment be installable on
+the current system.  That precursor can be met either by installing
+`pip` and `virtualenv` on your local box, or by running through the
+vagrant.  Once that requirement has been met, the makefile will ensure
+that the other packages are installed properly.
 
-The other requirement for the tests is to pull down sample images from the x3f_test_files repository.  This is a one-time download of about 90 mb of Sigma images that are used to run tests.
+The other requirement for the tests is to pull down sample images from
+the x3f_test_files repository.  This is a one-time download of about
+90 mb of Sigma images that are used to run tests.

doc/changes.txt

@@ -1,3 +1,6 @@
+x3f_053: Erik and Roland Karlsson and Mark Roden 2015-07-13
+  Lots of work, mainly adding conversion to color.
+
 x3f_052: Roland Karlsson 2014-11-30
   Running valgrind and removing some leaks and a buffer
   overrun bug. Now you can handle several files in the same

doc/copyright.txt

@@ -19,10 +19,10 @@ BSD-style License
 *       names of its contributors may be used to endorse or promote products
 *       derived from this software without specific prior written permission.
 *
-* THIS SOFTWARE IS PROVIDED BY ROLAND KARLSSON ''AS IS'' AND ANY
+* THIS SOFTWARE IS PROVIDED BY COPYRIGHT HOLDERS ''AS IS'' AND ANY
 * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
 * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-* DISCLAIMED. IN NO EVENT SHALL ROLAND KARLSSON BE LIABLE FOR ANY
+* DISCLAIMED. IN NO EVENT SHALL COPYRIGHT HOLDERS BE LIABLE FOR ANY
 * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
 * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
 * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND

doc/readme.txt

@@ -1,139 +1,98 @@
+----------------------------------------------------------------
+X3F Tools is a library for accessing Sigma X3F raw image files
+----------------------------------------------------------------
 
+Reading of the format:
 
+The code understands the old format for SD9, SD10 and SD14. It also
+understands the new format introduced with DP1. The latter format is
+associated with the TRUE engines, TRUE I and TRUE II. Currently it
+also supports the Merrill version and up to the slightly different
+Quattro version of the format.
 
-OUT OF DATE !!!!!!!
-OUT OF DATE !!!!!!!
-OUT OF DATE !!!!!!!
-OUT OF DATE !!!!!!!
-OUT OF DATE !!!!!!!
-OUT OF DATE !!!!!!!
-
-
-TODO : needs serious rewriting!
+Converting to color images:
 
+The code can convert all the above formats, except the Polaroid x530,
+to DNG files. Those DNG files contain color conversion data so that
+e.g. Lightroom can convert them to color-correct images.
 
+The code can also convert to color-correct TIFF images, but out of
+gamut colors are not handled correctly, in particular if the input
+channels are clipped. The latter leads to e.g. colorful skies.
 
 ----------------------------------------------------------------
-X3F tools is a library for accessing Sigma X3F raw image files
-----------------------------------------------------------------
+Included in the library are two tools:
 
-The code understands  the old format for SD9, SD10  and SD14. The code
-also understands the new format introduced with DP1. The latter format
-is associated with the TRUE engines  TRUE I and TRUE II. Currently the
-code supports  up to the Quattro  version of the cameras,  but do only
-understand meta data up to Merrill.
+  x3f_extract    A tool that extracts JPEG thumbnails and raw images.
+                 See below for usage. The RAW images may also be
+		 converted to DNG or TIFF color images. Metadata and
+		 histograms over the data might also be written.
 
-----------------------------------------------------------------
-Included in the library are two tools
-  x3f_extract    A tool that extracts JPEG thumbnail and raw images.
-                 See below for usage.
-  x3f_io_test    A tool that prints some meta data and tests that
+  x3f_io_test    A tool that prints some metadata and tests that
                  the code is working properly. This tool is not
-                 made to be user friendly. Its mainly a test
-                 tool for developing the code. This tool can
-                 also write x3f files. They should (with this
-                 tool) be identical to the origanal file.
-                 See below for usage.
+                 made to be user friendly. It is mainly a testing
+                 tool used for development.
 ----------------------------------------------------------------
 
 ----------------------------------------------------------------
 Executables
 ----------------------------------------------------------------
 
-It  is a  fair  chance that  you  can find  pre  built executables  on
-http://www.proxel.se/x3f.html.
+Prebuilt executables will be available at: http://www.proxel.se/x3f.html
 
 ----------------------------------------------------------------
 Building
 ----------------------------------------------------------------
 
-You are  supposed to have installed  gcc (and gmake)  on your machine.
-On Windows this currently (2010) means mingw.
+You are supposed to have gcc, gmake and cmake installed on your
+machine. Currently you need to build on Linux with cross-compilation
+for Windows and OSX.
 
 The command "make" builds the executables.
 
-The makefile  tries to find out  what platform you are  on. This might
-fail if your system has unexpected properties. In this case you are on
-your own and have to hack the makefile file.
+The makefile tries to find out which platform you are on. This might
+fail if your system has unexpected properties. In that case you are on
+your own and have to hack the makefile.
 
 ----------------------------------------------------------------
 Usage of the x3f_extract tool
 ----------------------------------------------------------------
 
-The tool can be used thus:
-
-(1) x3f_extract -raw file.x3f
-(2) x3f_extract -tiff file.x3f
-(3) x3f_extract -tiff -gamma 2.2 [-min 0] [-max 5000] file.x3f
-(4) x3f_extract -jpeg file.x3f
-(5) x3f_extract -ppm file.x3f
-(6) x3f_extract -ppm -gamma 2.2 [-min 0] [-max 5000] file.x3f
-(7) x3f_extract -meta file.x3f
-
-(1) dumps the  data block of the RAW image  verbatim. The original RAW
-    data is not parsed or  interpreted and is therefore for almost all
-    aspects useless. Mainly used for analyzing a new unknown format.
-
-(2) parses and interprets the RAW image data and then dumps the result
-    as  a 16  bit TIFF  file (called  file.x3f.tif) without  doing any
-    changes to the pixel values.
+You will get complete information on the switches by running
+x3f_extract without any switches, or e.g. with the switch -help.
 
-    NOTE - if  you load this TIFF file in e.g.  Photoshop - the result
-    is a very dark (almost  black) image. This image needs scaling and
-    gamma coding to look good.
+Here are some examples:
 
-    NOTE -  the result is  not a  regular RGB image  - its in  the X3F
-    "color space" - so it contains faulty colors.
+(1) x3f_extract -denoise file.x3f
+    This one creates the file file.dng for usage in e.g. Lightroom
+    or Photoshop (via ACR). The file contains denoised but unconverted
+    RAW data plus color conversion info.
 
-(3) is the same as (2) but it scales the image according to
-    a min, a max and a gamma value. If the min and max values are not
-    given then they are estimated.
+(2) x3f_extract -denoise -tiff -color sRGB -crop file.x3f
+    This one creates the file file.tif for usage in e.g. Photoshop.
+    The file is fully converted to sRGB, but without rendering intent
+    so you might get strange clipping.
 
-    NOTE  - the  current  code does  not  really know  how to  compute
-    the correct min and max values. Its recommended to set it yourself
-    - a guess is that max might be between 5000 and 10000. The program
-    writes the min and max value it finds. If you are outside the range,
-    the value is clipped.
+(3) x3f_extract -unprocessed file.x3f
+    This one creates the file file.tif with raw data. The data is
+    linear and unscaled. So, it will generally look black and needs to
+    be rescaled and gamma 2.2 has to be applied to it in your editor.
 
-(4) dumps the embedded JPEG thumbnail verbatim as file.x3f.jpg
+(4) x3f_extract -meta file.x3f
+    This one dumps metadata to file.meta
 
-    NOTE - this is not a JPEG of the RAW data.
-
-(5) Same as (2) but output is binary 16 bit PPM
-
-(6) Same as (3) but output is binary 16 bit PPM
-
-(7) dumps the meta data extracted from the file.
 
 ----------------------------------------------------------------
 Usage of the x3f_io_test tool
 ----------------------------------------------------------------
 
-The tool can be used thus
+This tool is really only a debugging tool for development. The tool
+can be used thus:
 
 (1) x3f_io_test file.x3f
-(2) x3f_io_test file.x3f -unpack
-(3) x3f_io_test file.x3f -write
-(4) x3f_io_test file.x3f -unpack -write
-
-(1) reads the file and parses  the main data structure of the file and
-    then prints some info about it.  NOTE - no parsing of data blocks,
-    e.g. image blocks, is made.
-
-(2) same  as (1) but  also parses data blocks, and then prints info again.
-
-(3) Same  as (1), but also writes from the unparsed version.
-
-(4) Same as (2), but also writes both from the umparsed version and
-    the parsed.
-
-    NOTE - detaled assembling of the blocks is not yet implemented, so
-    the  assembling  is partly  phony.  Therefore  -  that oufile2  is
-    identical does not say so much as it should.
-
-    NOTE  - the printouts  contain CAMF  parameter names.  The current
-    implementation stops  parsing CAMF when finding  something it does
-    not understand. That might  be wrong. Therefore - parameters might
-    be missing.
+    Reads the file, parses the main data structure, and then prints
+    some info about it. NOTE - no parsing of data blocks, e.g. image
+    blocks, is done.
 
-    NOTE - the actual contents of the CAMF parameters is not yet parsed.
+(2) x3f_io_test -unpack file.x3f
+    Same  as (1), but also prints info from the parsed data blocks.

doc/todo.txt

@@ -8,46 +8,23 @@ REFACTORING
   Simple naming fix. A layer should always be called a layer.
   The layers might be called colors after converting to RGB.
 
-* Better error handling.
-
 
 --------------------------------------------------------------------
 CAMF AND OTHER FILE FORMAT ISSUES
 --------------------------------------------------------------------
 
 * The CAMF matrix data has the dimensions in the description out of
-  order, at least for older cameras. What does that mean for the
-  stored data? Does that follow the order or the index n in the info?
+  order, at least for older cameras and also for Quattro. What does
+  that mean for the stored data? Does that follow the order or the
+  index n in the info?
 
 * Signed vs unsigned for CAMF matrix data ?
 
-* Different names for the same property list, e.g.
-  WhiteBalanceColorCorrection.
-
-* Real lookup of matrix names, via property lists.
-
-* Wrong file format for Quattro, e.g. white_balance is in other place.
-
-* We cannot get max raw values correctly.
-
-* Quattro probably also needs a min value.
-
-* Extended types ?
-
-* Double instead of float ?
 
 --------------------------------------------------------------------
 SMALL THINGS
 --------------------------------------------------------------------
 
-* Default write in curent directory.
-
-* Define output directory.
-
-* Define output file name.
-
-* Write to foo.tiff.parts and then rename to foo.tiff. 
-
 * Logic for getting thumbs images is wrong. There might be two
   and they might be of same type but different size.
 
@@ -58,53 +35,13 @@ SMALL THINGS
 COLOR AND CONVERSION
 --------------------------------------------------------------------
 
-* Support older (SD14 and older) file format for color conversion.
-  WBCorrection_<WB> and CamToXYZ_Flash.
-
-* What connection space is used in TRUE engine? Older is
-  CIE XYZ and TRUE uses some kind of RGB. We are using CIE
-  RGB now. We also convert D50 to D65. Are those right
-  choices?
-
-* Nonlinear coding of output.
-
-* How to handle the problem with clipped (overexposed) input.
-
-* Scaling. Why is the image dark?
-
-* Auto exposure correction?
-
-* ColorModeCompensations etc
-
-* Quattro ColorTemp is computed otherwise than other wb
-
-* There are also more matrices for color for Quattro
-
-* Contrast/(Sharpness)/Saturation
-
-* Crop to image rectangle
-
-* Noise!!
-
---------------------------------------------------------------------
-COLOR AND CONVERSION (ADVANCED)
---------------------------------------------------------------------
-
-* Lots of matrices with fault info? What to do?
-
-* Lookup tables ?
+* Green cast ?
 
 
 --------------------------------------------------------------------
 LONG TERM
 --------------------------------------------------------------------
 
-* Understanding important meta data
-
-* Making a usable color correct image
-
-* Making a usable B&W image
-
 
 --------------------------------------------------------------------
 DOCUMENTATION

makefile

@@ -74,7 +74,7 @@ $(VENV)/.setup.touch: $(REQUIREMENTS) | $(VENV)
 	$(VENV)/bin/pip install -r $< && touch $@
 
 check: check_deps dist
-	DIST_LOC=dist/x3f_tools-$(shell git describe --always --dirty)-$(TARGET)/bin/x3f_extract$(EXE) $(BEHAVE)
+	DIST_LOC=dist/x3f_tools-$(shell git describe --always --dirty --tags)-$(TARGET)/bin/x3f_extract$(EXE) $(BEHAVE)
 
 clean_deps:
 	rm -rf $(VENV)