From 36f2ed04beaf261d2e78961ebf425107b5b2788b Mon Sep 17 00:00:00 2001 From: Roland Karlsson Date: Mon, 13 Jul 2015 22:55:45 +0200 Subject: [PATCH 1/5] Updating the documentation --- doc/changes.txt | 3 ++ doc/readme.txt | 130 +++++++++++++++++------------------------------- doc/todo.txt | 71 ++------------------------ 3 files changed, 52 insertions(+), 152 deletions(-) diff --git a/doc/changes.txt b/doc/changes.txt index ab8425f..b699239 100644 --- a/doc/changes.txt +++ b/doc/changes.txt @@ -1,3 +1,6 @@ +x3f_053: Erik and Roland Karlsson and Mark Roden 2015-07-13 + Lots of work, mainly adding conversion to color. + x3f_052: Roland Karlsson 2014-11-30 Running valgrind and removing some leaks and a buffer overrun bug. Now you can handle several files in the same diff --git a/doc/readme.txt b/doc/readme.txt index 6b40e52..b4ade98 100644 --- a/doc/readme.txt +++ b/doc/readme.txt @@ -1,39 +1,37 @@ - - - -OUT OF DATE !!!!!!! -OUT OF DATE !!!!!!! -OUT OF DATE !!!!!!! -OUT OF DATE !!!!!!! -OUT OF DATE !!!!!!! -OUT OF DATE !!!!!!! - - -TODO : needs serious rewriting! - - - ---------------------------------------------------------------- X3F tools is a library for accessing Sigma X3F raw image files ---------------------------------------------------------------- +Reading of format: + The code understands the old format for SD9, SD10 and SD14. The code also understands the new format introduced with DP1. The latter format is associated with the TRUE engines TRUE I and TRUE II. Currently the -code supports up to the Quattro version of the cameras, but do only -understand meta data up to Merrill. +code supports also the Merrill version and up to the slightly +different Quattro version of the cameras. + +Converting to color images: + +The code can convert all above formats, except the Polaroid X370, to a +DNG file. This DNG file contains color conversion date so that +e.g. Lightroom can convert to correct color images. + +The code can also convert to color TIFF images, but out of gamut +colors are not handled correctly. In particular if the input channels +are clipped. The latter leads to e.g. colorful skies. ---------------------------------------------------------------- Included in the library are two tools + x3f_extract A tool that extracts JPEG thumbnail and raw images. - See below for usage. + See below for usage. The RAW images may also be + converted to DNG or TIFF color images. Meta data and + histograms over the data might also be written. + x3f_io_test A tool that prints some meta data and tests that the code is working properly. This tool is not made to be user friendly. Its mainly a test - tool for developing the code. This tool can - also write x3f files. They should (with this - tool) be identical to the origanal file. - See below for usage. + tool for developing the code. ---------------------------------------------------------------- ---------------------------------------------------------------- @@ -47,8 +45,9 @@ http://www.proxel.se/x3f.html. Building ---------------------------------------------------------------- -You are supposed to have installed gcc (and gmake) on your machine. -On Windows this currently (2010) means mingw. +You are supposed to have installed gcc (and gmake) on your machine. +Currently you need to build on Linux with cross compilation for +Windows and OSX. The command "make" builds the executables. @@ -60,80 +59,41 @@ your own and have to hack the makefile file. Usage of the x3f_extract tool ---------------------------------------------------------------- -The tool can be used thus: - -(1) x3f_extract -raw file.x3f -(2) x3f_extract -tiff file.x3f -(3) x3f_extract -tiff -gamma 2.2 [-min 0] [-max 5000] file.x3f -(4) x3f_extract -jpeg file.x3f -(5) x3f_extract -ppm file.x3f -(6) x3f_extract -ppm -gamma 2.2 [-min 0] [-max 5000] file.x3f -(7) x3f_extract -meta file.x3f - -(1) dumps the data block of the RAW image verbatim. The original RAW - data is not parsed or interpreted and is therefore for almost all - aspects useless. Mainly used for analyzing a new unknown format. - -(2) parses and interprets the RAW image data and then dumps the result - as a 16 bit TIFF file (called file.x3f.tif) without doing any - changes to the pixel values. - - NOTE - if you load this TIFF file in e.g. Photoshop - the result - is a very dark (almost black) image. This image needs scaling and - gamma coding to look good. - - NOTE - the result is not a regular RGB image - its in the X3F - "color space" - so it contains faulty colors. - -(3) is the same as (2) but it scales the image according to - a min, a max and a gamma value. If the min and max values are not - given then they are estimated. +You will get a total information on switches by running x3f_extract +without any switches, or e.g. with the switch -help. - NOTE - the current code does not really know how to compute - the correct min and max values. Its recommended to set it yourself - - a guess is that max might be between 5000 and 10000. The program - writes the min and max value it finds. If you are outside the range, - the value is clipped. +Here are some examples: -(4) dumps the embedded JPEG thumbnail verbatim as file.x3f.jpg +(1) x3f_extract -denoise file.x3f + This one creates the file file.dng for usage in e.g. Lightroom + or Photoshop (via ACR). The file contains denoised but unconverted + RAW data plus color conversion info. - NOTE - this is not a JPEG of the RAW data. +(2) x3f_extract -denoise -tiff -color sRGB file.x3f + This one creates the file file.tiff for usage in e.g. Photoshop. + The file is fully converted to sRGB, but without rendering intent + so you might get strange clipping. -(5) Same as (2) but output is binary 16 bit PPM +(3) x3f_extract -unprocessed file.x3f + This one creates the file file.tiff with RAW data. This data is linear + and unscaled. So, it is generally look black and needs to be scaled + and applied gamma 2.2 in your editor. -(6) Same as (3) but output is binary 16 bit PPM +(4) x3f_extract -meta file.x3f + This one dumnps meta data in file.meta -(7) dumps the meta data extracted from the file. ---------------------------------------------------------------- Usage of the x3f_io_test tool ---------------------------------------------------------------- -The tool can be used thus +This tool is really only a developer debug tool. +The tool can be used thus: (1) x3f_io_test file.x3f -(2) x3f_io_test file.x3f -unpack -(3) x3f_io_test file.x3f -write -(4) x3f_io_test file.x3f -unpack -write - -(1) reads the file and parses the main data structure of the file and + Reads the file and parses the main data structure of the file and then prints some info about it. NOTE - no parsing of data blocks, e.g. image blocks, is made. -(2) same as (1) but also parses data blocks, and then prints info again. - -(3) Same as (1), but also writes from the unparsed version. - -(4) Same as (2), but also writes both from the umparsed version and - the parsed. - - NOTE - detaled assembling of the blocks is not yet implemented, so - the assembling is partly phony. Therefore - that oufile2 is - identical does not say so much as it should. - - NOTE - the printouts contain CAMF parameter names. The current - implementation stops parsing CAMF when finding something it does - not understand. That might be wrong. Therefore - parameters might - be missing. - - NOTE - the actual contents of the CAMF parameters is not yet parsed. +(2) x3f_io_test -unpack file.x3f + Same as (1), but also writes from the parsed version. diff --git a/doc/todo.txt b/doc/todo.txt index 08f4752..e4e1bc3 100644 --- a/doc/todo.txt +++ b/doc/todo.txt @@ -8,46 +8,23 @@ REFACTORING Simple naming fix. A layer should always be called a layer. The layers might be called colors after converting to RGB. -* Better error handling. - -------------------------------------------------------------------- CAMF AND OTHER FILE FORMAT ISSUES -------------------------------------------------------------------- * The CAMF matrix data has the dimensions in the description out of - order, at least for older cameras. What does that mean for the - stored data? Does that follow the order or the index n in the info? + order, at least for older cameras and also for Quattro. What does + that mean for the stored data? Does that follow the order or the + index n in the info? * Signed vs unsigned for CAMF matrix data ? -* Different names for the same property list, e.g. - WhiteBalanceColorCorrection. - -* Real lookup of matrix names, via property lists. - -* Wrong file format for Quattro, e.g. white_balance is in other place. - -* We cannot get max raw values correctly. - -* Quattro probably also needs a min value. - -* Extended types ? - -* Double instead of float ? -------------------------------------------------------------------- SMALL THINGS -------------------------------------------------------------------- -* Default write in curent directory. - -* Define output directory. - -* Define output file name. - -* Write to foo.tiff.parts and then rename to foo.tiff. - * Logic for getting thumbs images is wrong. There might be two and they might be of same type but different size. @@ -58,53 +35,13 @@ SMALL THINGS COLOR AND CONVERSION -------------------------------------------------------------------- -* Support older (SD14 and older) file format for color conversion. - WBCorrection_ and CamToXYZ_Flash. - -* What connection space is used in TRUE engine? Older is - CIE XYZ and TRUE uses some kind of RGB. We are using CIE - RGB now. We also convert D50 to D65. Are those right - choices? - -* Nonlinear coding of output. - -* How to handle the problem with clipped (overexposed) input. - -* Scaling. Why is the image dark? - -* Auto exposure correction? - -* ColorModeCompensations etc - -* Quattro ColorTemp is computed otherwise than other wb - -* There are also more matrices for color for Quattro - -* Contrast/(Sharpness)/Saturation - -* Crop to image rectangle - -* Noise!! - --------------------------------------------------------------------- -COLOR AND CONVERSION (ADVANCED) --------------------------------------------------------------------- - -* Lots of matrices with fault info? What to do? - -* Lookup tables ? +* Green cast ? -------------------------------------------------------------------- LONG TERM -------------------------------------------------------------------- -* Understanding important meta data - -* Making a usable color correct image - -* Making a usable B&W image - -------------------------------------------------------------------- DOCUMENTATION From e5ad8cfc3de4395e213b841b7d3894693fa9591a Mon Sep 17 00:00:00 2001 From: Erik Karlsson Date: Tue, 14 Jul 2015 14:05:13 +0200 Subject: [PATCH 2/5] Corrected readme.txt --- doc/readme.txt | 81 +++++++++++++++++++++++++------------------------- 1 file changed, 40 insertions(+), 41 deletions(-) diff --git a/doc/readme.txt b/doc/readme.txt index b4ade98..97b0822 100644 --- a/doc/readme.txt +++ b/doc/readme.txt @@ -1,66 +1,65 @@ ---------------------------------------------------------------- -X3F tools is a library for accessing Sigma X3F raw image files +X3F Tools is a library for accessing Sigma X3F raw image files ---------------------------------------------------------------- -Reading of format: +Reading of the format: -The code understands the old format for SD9, SD10 and SD14. The code -also understands the new format introduced with DP1. The latter format -is associated with the TRUE engines TRUE I and TRUE II. Currently the -code supports also the Merrill version and up to the slightly -different Quattro version of the cameras. +The code understands the old format for SD9, SD10 and SD14. It also +understands the new format introduced with DP1. The latter format is +associated with the TRUE engines, TRUE I and TRUE II. Currently it +also supports the Merrill version and up to the slightly different +Quattro version of the format. Converting to color images: -The code can convert all above formats, except the Polaroid X370, to a -DNG file. This DNG file contains color conversion date so that -e.g. Lightroom can convert to correct color images. +The code can convert all the above formats, except the Polaroid x530, +to DNG files. Those DNG files contain color conversion data so that +e.g. Lightroom can convert them to color-correct images. -The code can also convert to color TIFF images, but out of gamut -colors are not handled correctly. In particular if the input channels -are clipped. The latter leads to e.g. colorful skies. +The code can also convert to color-correct TIFF images, but out of +gamut colors are not handled correctly, in particular if the input +channels are clipped. The latter leads to e.g. colorful skies. ---------------------------------------------------------------- -Included in the library are two tools +Included in the library are two tools: - x3f_extract A tool that extracts JPEG thumbnail and raw images. + x3f_extract A tool that extracts JPEG thumbnails and raw images. See below for usage. The RAW images may also be - converted to DNG or TIFF color images. Meta data and + converted to DNG or TIFF color images. Metadata and histograms over the data might also be written. - x3f_io_test A tool that prints some meta data and tests that + x3f_io_test A tool that prints some metadata and tests that the code is working properly. This tool is not - made to be user friendly. Its mainly a test - tool for developing the code. + made to be user friendly. It is mainly a testing + tool used for development. ---------------------------------------------------------------- ---------------------------------------------------------------- Executables ---------------------------------------------------------------- -It is a fair chance that you can find pre built executables on -http://www.proxel.se/x3f.html. +Prebuilt executables will be available at: http://www.proxel.se/x3f.html ---------------------------------------------------------------- Building ---------------------------------------------------------------- -You are supposed to have installed gcc (and gmake) on your machine. -Currently you need to build on Linux with cross compilation for -Windows and OSX. +You are supposed to have gcc, gmake and cmake installed on your +machine. Currently you need to build on Linux with cross-compilation +for Windows and OSX. The command "make" builds the executables. -The makefile tries to find out what platform you are on. This might -fail if your system has unexpected properties. In this case you are on -your own and have to hack the makefile file. +The makefile tries to find out which platform you are on. This might +fail if your system has unexpected properties. In that case you are on +your own and have to hack the makefile. ---------------------------------------------------------------- Usage of the x3f_extract tool ---------------------------------------------------------------- -You will get a total information on switches by running x3f_extract -without any switches, or e.g. with the switch -help. +You will get complete information on the switches by running +x3f_extract without any switches, or e.g. with the switch -help. Here are some examples: @@ -69,31 +68,31 @@ Here are some examples: or Photoshop (via ACR). The file contains denoised but unconverted RAW data plus color conversion info. -(2) x3f_extract -denoise -tiff -color sRGB file.x3f - This one creates the file file.tiff for usage in e.g. Photoshop. +(2) x3f_extract -denoise -tiff -color sRGB -crop file.x3f + This one creates the file file.tif for usage in e.g. Photoshop. The file is fully converted to sRGB, but without rendering intent so you might get strange clipping. (3) x3f_extract -unprocessed file.x3f - This one creates the file file.tiff with RAW data. This data is linear - and unscaled. So, it is generally look black and needs to be scaled - and applied gamma 2.2 in your editor. + This one creates the file file.tif with raw data. The data is + linear and unscaled. So, it will generally look black and needs to + be rescaled and gamma 2.2 has to be applied to it in your editor. (4) x3f_extract -meta file.x3f - This one dumnps meta data in file.meta + This one dumps metadata to file.meta ---------------------------------------------------------------- Usage of the x3f_io_test tool ---------------------------------------------------------------- -This tool is really only a developer debug tool. -The tool can be used thus: +This tool is really only a debugging tool for development. The tool +can be used thus: (1) x3f_io_test file.x3f - Reads the file and parses the main data structure of the file and - then prints some info about it. NOTE - no parsing of data blocks, - e.g. image blocks, is made. + Reads the file, parses the main data structure, and then prints + some info about it. NOTE - no parsing of data blocks, e.g. image + blocks, is done. (2) x3f_io_test -unpack file.x3f - Same as (1), but also writes from the parsed version. + Same as (1), but also prints info from the parsed data blocks. From b7bf8c847b5eaf0fcfc640599e36357dacea5673 Mon Sep 17 00:00:00 2001 From: Roland Karlsson Date: Wed, 15 Jul 2015 12:40:57 +0200 Subject: [PATCH 3/5] Make some more updates to documentation (and build) --- README | 27 +++++++++++++++++++++++++-- doc/copyright.txt | 4 ++-- makefile | 2 +- src/makefile | 4 ++-- 4 files changed, 30 insertions(+), 7 deletions(-) diff --git a/README b/README index 6efe7c3..2903111 100644 --- a/README +++ b/README @@ -1,11 +1,20 @@ This project contains tools for manipulating X3F files from Sigma cameras. See doc/readme.txt and doc/copyright.txt. -To run the code: +------- +COMPILE +------- + Download the code run 'make' -The bin directory contains binaries that can be run directly from the command line. Alternatively, you can create binaries for multiple platforms by calling: +The bin directory contains binaries that can be run directly from the command line. + +------------- +DISTRIBUTIONS +------------- + +Alternatively, you can create binaries for multiple platforms by calling: make dist-all @@ -15,12 +24,26 @@ as well as the platform specific calls: make dist-32 make dist-64 +The distribution will be found in the file + + dist/x3f_tools-- + +The part is either the git hash, or if there is a tag on current commit, that tag. + +------- +VAGRANT +------- + If you don't want to install all the dependencies, you can also install and run in a vagrant. Doing so requires downloading Vagrant from http://www.vagrantup.com. Once you've done so, the commands to run the code are: vagrant up make +----- +TESTS +----- + If you want to run tests, type: make check diff --git a/doc/copyright.txt b/doc/copyright.txt index 615c925..e85c42e 100644 --- a/doc/copyright.txt +++ b/doc/copyright.txt @@ -19,10 +19,10 @@ BSD-style License * names of its contributors may be used to endorse or promote products * derived from this software without specific prior written permission. * -* THIS SOFTWARE IS PROVIDED BY ROLAND KARLSSON ''AS IS'' AND ANY +* THIS SOFTWARE IS PROVIDED BY COPYRIGHT HOLDERS ''AS IS'' AND ANY * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE -* DISCLAIMED. IN NO EVENT SHALL ROLAND KARLSSON BE LIABLE FOR ANY +* DISCLAIMED. IN NO EVENT SHALL COPYRIGHT HOLDERS BE LIABLE FOR ANY * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND diff --git a/makefile b/makefile index fcde049..9be7375 100644 --- a/makefile +++ b/makefile @@ -74,7 +74,7 @@ $(VENV)/.setup.touch: $(REQUIREMENTS) | $(VENV) $(VENV)/bin/pip install -r $< && touch $@ check: check_deps dist - DIST_LOC=dist/x3f_tools-$(shell git describe --always --dirty)-$(TARGET)/bin/x3f_extract$(EXE) $(BEHAVE) + DIST_LOC=dist/x3f_tools-$(shell git describe --always --dirty --tag)-$(TARGET)/bin/x3f_extract$(EXE) $(BEHAVE) clean_deps: rm -rf $(VENV) diff --git a/src/makefile b/src/makefile index e28e702..b1e103d 100644 --- a/src/makefile +++ b/src/makefile @@ -118,12 +118,12 @@ $(BINDIR): # ----------------------------------------------------------- NAME = x3f_tools -VER := $(shell git describe --always --dirty) +VER := $(shell git describe --always --dirty --tag) DIST = $(NAME)-$(VER)-$(TARGET) DIST_DIR = ../dist/$(DIST) DIST_TGZ = ../dist/$(DIST).tar.gz DIST_ZIP = ../dist/$(DIST).zip -DIST_FILES = README doc/readme.txt doc/copyright.txt bin/x3f_extract$(EXE) +DIST_FILES = doc/readme.txt doc/copyright.txt bin/x3f_extract$(EXE) TARFLAGS = --owner=root --group=root ifeq ($(TARGET_SYS), windows) From dc7ce65255f1e7e488c95d75cb8bf77bcf8b816e Mon Sep 17 00:00:00 2001 From: Roland Karlsson Date: Wed, 15 Jul 2015 18:56:09 +0200 Subject: [PATCH 4/5] Make some more updates to build --- makefile | 2 +- src/makefile | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/makefile b/makefile index 9be7375..bce1510 100644 --- a/makefile +++ b/makefile @@ -74,7 +74,7 @@ $(VENV)/.setup.touch: $(REQUIREMENTS) | $(VENV) $(VENV)/bin/pip install -r $< && touch $@ check: check_deps dist - DIST_LOC=dist/x3f_tools-$(shell git describe --always --dirty --tag)-$(TARGET)/bin/x3f_extract$(EXE) $(BEHAVE) + DIST_LOC=dist/x3f_tools-$(shell git describe --always --dirty --tags)-$(TARGET)/bin/x3f_extract$(EXE) $(BEHAVE) clean_deps: rm -rf $(VENV) diff --git a/src/makefile b/src/makefile index b1e103d..620852b 100644 --- a/src/makefile +++ b/src/makefile @@ -118,7 +118,7 @@ $(BINDIR): # ----------------------------------------------------------- NAME = x3f_tools -VER := $(shell git describe --always --dirty --tag) +VER := $(shell git describe --always --dirty --tags) DIST = $(NAME)-$(VER)-$(TARGET) DIST_DIR = ../dist/$(DIST) DIST_TGZ = ../dist/$(DIST).tar.gz From 6470c6b06222a45dd03b90516d429433bdb837e5 Mon Sep 17 00:00:00 2001 From: Erik Karlsson Date: Thu, 16 Jul 2015 17:19:44 +0200 Subject: [PATCH 5/5] Line breaks in README --- README | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/README b/README index 2903111..fa3f544 100644 --- a/README +++ b/README @@ -5,10 +5,11 @@ See doc/readme.txt and doc/copyright.txt. COMPILE ------- -Download the code +Download the code. run 'make' -The bin directory contains binaries that can be run directly from the command line. +The bin directory contains binaries that can be run directly from the +command line. ------------- DISTRIBUTIONS @@ -28,13 +29,17 @@ The distribution will be found in the file dist/x3f_tools-- -The part is either the git hash, or if there is a tag on current commit, that tag. +The part is either the git hash, or if there is a tag on current +commit, that tag. ------- VAGRANT ------- -If you don't want to install all the dependencies, you can also install and run in a vagrant. Doing so requires downloading Vagrant from http://www.vagrantup.com. Once you've done so, the commands to run the code are: +If you don't want to install all the dependencies, you can also +install and run in a vagrant. Doing so requires downloading Vagrant +from http://www.vagrantup.com. Once you've done so, the commands to +run the code are: vagrant up @@ -48,6 +53,12 @@ If you want to run tests, type: make check -The tests require that a python virtual environment be installable on the current system. That precursor can be met either by installing `pip` and `virtualenv` on your local box, or by running through the vagrant. Once that requirement has been met, the makefile will ensure that the other packages are installed properly. +The tests require that a python virtual environment be installable on +the current system. That precursor can be met either by installing +`pip` and `virtualenv` on your local box, or by running through the +vagrant. Once that requirement has been met, the makefile will ensure +that the other packages are installed properly. -The other requirement for the tests is to pull down sample images from the x3f_test_files repository. This is a one-time download of about 90 mb of Sigma images that are used to run tests. +The other requirement for the tests is to pull down sample images from +the x3f_test_files repository. This is a one-time download of about +90 mb of Sigma images that are used to run tests.