diff --git a/README b/README index 6efe7c3..fa3f544 100644 --- a/README +++ b/README @@ -1,11 +1,21 @@ This project contains tools for manipulating X3F files from Sigma cameras. See doc/readme.txt and doc/copyright.txt. -To run the code: -Download the code +------- +COMPILE +------- + +Download the code. run 'make' -The bin directory contains binaries that can be run directly from the command line. Alternatively, you can create binaries for multiple platforms by calling: +The bin directory contains binaries that can be run directly from the +command line. + +------------- +DISTRIBUTIONS +------------- + +Alternatively, you can create binaries for multiple platforms by calling: make dist-all @@ -15,16 +25,40 @@ as well as the platform specific calls: make dist-32 make dist-64 -If you don't want to install all the dependencies, you can also install and run in a vagrant. Doing so requires downloading Vagrant from http://www.vagrantup.com. Once you've done so, the commands to run the code are: +The distribution will be found in the file + + dist/x3f_tools-- + +The part is either the git hash, or if there is a tag on current +commit, that tag. + +------- +VAGRANT +------- + +If you don't want to install all the dependencies, you can also +install and run in a vagrant. Doing so requires downloading Vagrant +from http://www.vagrantup.com. Once you've done so, the commands to +run the code are: vagrant up make +----- +TESTS +----- + If you want to run tests, type: make check -The tests require that a python virtual environment be installable on the current system. That precursor can be met either by installing `pip` and `virtualenv` on your local box, or by running through the vagrant. Once that requirement has been met, the makefile will ensure that the other packages are installed properly. +The tests require that a python virtual environment be installable on +the current system. That precursor can be met either by installing +`pip` and `virtualenv` on your local box, or by running through the +vagrant. Once that requirement has been met, the makefile will ensure +that the other packages are installed properly. -The other requirement for the tests is to pull down sample images from the x3f_test_files repository. This is a one-time download of about 90 mb of Sigma images that are used to run tests. +The other requirement for the tests is to pull down sample images from +the x3f_test_files repository. This is a one-time download of about +90 mb of Sigma images that are used to run tests. diff --git a/doc/changes.txt b/doc/changes.txt index ab8425f..b699239 100644 --- a/doc/changes.txt +++ b/doc/changes.txt @@ -1,3 +1,6 @@ +x3f_053: Erik and Roland Karlsson and Mark Roden 2015-07-13 + Lots of work, mainly adding conversion to color. + x3f_052: Roland Karlsson 2014-11-30 Running valgrind and removing some leaks and a buffer overrun bug. Now you can handle several files in the same diff --git a/doc/copyright.txt b/doc/copyright.txt index 615c925..e85c42e 100644 --- a/doc/copyright.txt +++ b/doc/copyright.txt @@ -19,10 +19,10 @@ BSD-style License * names of its contributors may be used to endorse or promote products * derived from this software without specific prior written permission. * -* THIS SOFTWARE IS PROVIDED BY ROLAND KARLSSON ''AS IS'' AND ANY +* THIS SOFTWARE IS PROVIDED BY COPYRIGHT HOLDERS ''AS IS'' AND ANY * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE -* DISCLAIMED. IN NO EVENT SHALL ROLAND KARLSSON BE LIABLE FOR ANY +* DISCLAIMED. IN NO EVENT SHALL COPYRIGHT HOLDERS BE LIABLE FOR ANY * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND diff --git a/doc/readme.txt b/doc/readme.txt index 6b40e52..97b0822 100644 --- a/doc/readme.txt +++ b/doc/readme.txt @@ -1,139 +1,98 @@ +---------------------------------------------------------------- +X3F Tools is a library for accessing Sigma X3F raw image files +---------------------------------------------------------------- +Reading of the format: +The code understands the old format for SD9, SD10 and SD14. It also +understands the new format introduced with DP1. The latter format is +associated with the TRUE engines, TRUE I and TRUE II. Currently it +also supports the Merrill version and up to the slightly different +Quattro version of the format. -OUT OF DATE !!!!!!! -OUT OF DATE !!!!!!! -OUT OF DATE !!!!!!! -OUT OF DATE !!!!!!! -OUT OF DATE !!!!!!! -OUT OF DATE !!!!!!! - - -TODO : needs serious rewriting! +Converting to color images: +The code can convert all the above formats, except the Polaroid x530, +to DNG files. Those DNG files contain color conversion data so that +e.g. Lightroom can convert them to color-correct images. +The code can also convert to color-correct TIFF images, but out of +gamut colors are not handled correctly, in particular if the input +channels are clipped. The latter leads to e.g. colorful skies. ---------------------------------------------------------------- -X3F tools is a library for accessing Sigma X3F raw image files ----------------------------------------------------------------- +Included in the library are two tools: -The code understands the old format for SD9, SD10 and SD14. The code -also understands the new format introduced with DP1. The latter format -is associated with the TRUE engines TRUE I and TRUE II. Currently the -code supports up to the Quattro version of the cameras, but do only -understand meta data up to Merrill. + x3f_extract A tool that extracts JPEG thumbnails and raw images. + See below for usage. The RAW images may also be + converted to DNG or TIFF color images. Metadata and + histograms over the data might also be written. ----------------------------------------------------------------- -Included in the library are two tools - x3f_extract A tool that extracts JPEG thumbnail and raw images. - See below for usage. - x3f_io_test A tool that prints some meta data and tests that + x3f_io_test A tool that prints some metadata and tests that the code is working properly. This tool is not - made to be user friendly. Its mainly a test - tool for developing the code. This tool can - also write x3f files. They should (with this - tool) be identical to the origanal file. - See below for usage. + made to be user friendly. It is mainly a testing + tool used for development. ---------------------------------------------------------------- ---------------------------------------------------------------- Executables ---------------------------------------------------------------- -It is a fair chance that you can find pre built executables on -http://www.proxel.se/x3f.html. +Prebuilt executables will be available at: http://www.proxel.se/x3f.html ---------------------------------------------------------------- Building ---------------------------------------------------------------- -You are supposed to have installed gcc (and gmake) on your machine. -On Windows this currently (2010) means mingw. +You are supposed to have gcc, gmake and cmake installed on your +machine. Currently you need to build on Linux with cross-compilation +for Windows and OSX. The command "make" builds the executables. -The makefile tries to find out what platform you are on. This might -fail if your system has unexpected properties. In this case you are on -your own and have to hack the makefile file. +The makefile tries to find out which platform you are on. This might +fail if your system has unexpected properties. In that case you are on +your own and have to hack the makefile. ---------------------------------------------------------------- Usage of the x3f_extract tool ---------------------------------------------------------------- -The tool can be used thus: - -(1) x3f_extract -raw file.x3f -(2) x3f_extract -tiff file.x3f -(3) x3f_extract -tiff -gamma 2.2 [-min 0] [-max 5000] file.x3f -(4) x3f_extract -jpeg file.x3f -(5) x3f_extract -ppm file.x3f -(6) x3f_extract -ppm -gamma 2.2 [-min 0] [-max 5000] file.x3f -(7) x3f_extract -meta file.x3f - -(1) dumps the data block of the RAW image verbatim. The original RAW - data is not parsed or interpreted and is therefore for almost all - aspects useless. Mainly used for analyzing a new unknown format. - -(2) parses and interprets the RAW image data and then dumps the result - as a 16 bit TIFF file (called file.x3f.tif) without doing any - changes to the pixel values. +You will get complete information on the switches by running +x3f_extract without any switches, or e.g. with the switch -help. - NOTE - if you load this TIFF file in e.g. Photoshop - the result - is a very dark (almost black) image. This image needs scaling and - gamma coding to look good. +Here are some examples: - NOTE - the result is not a regular RGB image - its in the X3F - "color space" - so it contains faulty colors. +(1) x3f_extract -denoise file.x3f + This one creates the file file.dng for usage in e.g. Lightroom + or Photoshop (via ACR). The file contains denoised but unconverted + RAW data plus color conversion info. -(3) is the same as (2) but it scales the image according to - a min, a max and a gamma value. If the min and max values are not - given then they are estimated. +(2) x3f_extract -denoise -tiff -color sRGB -crop file.x3f + This one creates the file file.tif for usage in e.g. Photoshop. + The file is fully converted to sRGB, but without rendering intent + so you might get strange clipping. - NOTE - the current code does not really know how to compute - the correct min and max values. Its recommended to set it yourself - - a guess is that max might be between 5000 and 10000. The program - writes the min and max value it finds. If you are outside the range, - the value is clipped. +(3) x3f_extract -unprocessed file.x3f + This one creates the file file.tif with raw data. The data is + linear and unscaled. So, it will generally look black and needs to + be rescaled and gamma 2.2 has to be applied to it in your editor. -(4) dumps the embedded JPEG thumbnail verbatim as file.x3f.jpg +(4) x3f_extract -meta file.x3f + This one dumps metadata to file.meta - NOTE - this is not a JPEG of the RAW data. - -(5) Same as (2) but output is binary 16 bit PPM - -(6) Same as (3) but output is binary 16 bit PPM - -(7) dumps the meta data extracted from the file. ---------------------------------------------------------------- Usage of the x3f_io_test tool ---------------------------------------------------------------- -The tool can be used thus +This tool is really only a debugging tool for development. The tool +can be used thus: (1) x3f_io_test file.x3f -(2) x3f_io_test file.x3f -unpack -(3) x3f_io_test file.x3f -write -(4) x3f_io_test file.x3f -unpack -write - -(1) reads the file and parses the main data structure of the file and - then prints some info about it. NOTE - no parsing of data blocks, - e.g. image blocks, is made. - -(2) same as (1) but also parses data blocks, and then prints info again. - -(3) Same as (1), but also writes from the unparsed version. - -(4) Same as (2), but also writes both from the umparsed version and - the parsed. - - NOTE - detaled assembling of the blocks is not yet implemented, so - the assembling is partly phony. Therefore - that oufile2 is - identical does not say so much as it should. - - NOTE - the printouts contain CAMF parameter names. The current - implementation stops parsing CAMF when finding something it does - not understand. That might be wrong. Therefore - parameters might - be missing. + Reads the file, parses the main data structure, and then prints + some info about it. NOTE - no parsing of data blocks, e.g. image + blocks, is done. - NOTE - the actual contents of the CAMF parameters is not yet parsed. +(2) x3f_io_test -unpack file.x3f + Same as (1), but also prints info from the parsed data blocks. diff --git a/doc/todo.txt b/doc/todo.txt index 08f4752..e4e1bc3 100644 --- a/doc/todo.txt +++ b/doc/todo.txt @@ -8,46 +8,23 @@ REFACTORING Simple naming fix. A layer should always be called a layer. The layers might be called colors after converting to RGB. -* Better error handling. - -------------------------------------------------------------------- CAMF AND OTHER FILE FORMAT ISSUES -------------------------------------------------------------------- * The CAMF matrix data has the dimensions in the description out of - order, at least for older cameras. What does that mean for the - stored data? Does that follow the order or the index n in the info? + order, at least for older cameras and also for Quattro. What does + that mean for the stored data? Does that follow the order or the + index n in the info? * Signed vs unsigned for CAMF matrix data ? -* Different names for the same property list, e.g. - WhiteBalanceColorCorrection. - -* Real lookup of matrix names, via property lists. - -* Wrong file format for Quattro, e.g. white_balance is in other place. - -* We cannot get max raw values correctly. - -* Quattro probably also needs a min value. - -* Extended types ? - -* Double instead of float ? -------------------------------------------------------------------- SMALL THINGS -------------------------------------------------------------------- -* Default write in curent directory. - -* Define output directory. - -* Define output file name. - -* Write to foo.tiff.parts and then rename to foo.tiff. - * Logic for getting thumbs images is wrong. There might be two and they might be of same type but different size. @@ -58,53 +35,13 @@ SMALL THINGS COLOR AND CONVERSION -------------------------------------------------------------------- -* Support older (SD14 and older) file format for color conversion. - WBCorrection_ and CamToXYZ_Flash. - -* What connection space is used in TRUE engine? Older is - CIE XYZ and TRUE uses some kind of RGB. We are using CIE - RGB now. We also convert D50 to D65. Are those right - choices? - -* Nonlinear coding of output. - -* How to handle the problem with clipped (overexposed) input. - -* Scaling. Why is the image dark? - -* Auto exposure correction? - -* ColorModeCompensations etc - -* Quattro ColorTemp is computed otherwise than other wb - -* There are also more matrices for color for Quattro - -* Contrast/(Sharpness)/Saturation - -* Crop to image rectangle - -* Noise!! - --------------------------------------------------------------------- -COLOR AND CONVERSION (ADVANCED) --------------------------------------------------------------------- - -* Lots of matrices with fault info? What to do? - -* Lookup tables ? +* Green cast ? -------------------------------------------------------------------- LONG TERM -------------------------------------------------------------------- -* Understanding important meta data - -* Making a usable color correct image - -* Making a usable B&W image - -------------------------------------------------------------------- DOCUMENTATION diff --git a/makefile b/makefile index fcde049..bce1510 100644 --- a/makefile +++ b/makefile @@ -74,7 +74,7 @@ $(VENV)/.setup.touch: $(REQUIREMENTS) | $(VENV) $(VENV)/bin/pip install -r $< && touch $@ check: check_deps dist - DIST_LOC=dist/x3f_tools-$(shell git describe --always --dirty)-$(TARGET)/bin/x3f_extract$(EXE) $(BEHAVE) + DIST_LOC=dist/x3f_tools-$(shell git describe --always --dirty --tags)-$(TARGET)/bin/x3f_extract$(EXE) $(BEHAVE) clean_deps: rm -rf $(VENV) diff --git a/src/makefile b/src/makefile index e28e702..620852b 100644 --- a/src/makefile +++ b/src/makefile @@ -118,12 +118,12 @@ $(BINDIR): # ----------------------------------------------------------- NAME = x3f_tools -VER := $(shell git describe --always --dirty) +VER := $(shell git describe --always --dirty --tags) DIST = $(NAME)-$(VER)-$(TARGET) DIST_DIR = ../dist/$(DIST) DIST_TGZ = ../dist/$(DIST).tar.gz DIST_ZIP = ../dist/$(DIST).zip -DIST_FILES = README doc/readme.txt doc/copyright.txt bin/x3f_extract$(EXE) +DIST_FILES = doc/readme.txt doc/copyright.txt bin/x3f_extract$(EXE) TARFLAGS = --owner=root --group=root ifeq ($(TARGET_SYS), windows)