Skip to content
/ if2ktool Public

A gui-based tool used to migrate some stats (play count, rating, date added and last played) from iTunes to foobar2000's foo_playcount component (Playback Statistics).

6 stars 1 fork Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

macklinb/if2ktool

BranchesTags

Repository files navigation

if2ktool

Overview

This tool is aimed at people migrating a music library from iTunes to foobar2000, and allows you to keep the various statistics and dates that are saved by iTunes. It reads these fields from your iTunes Library XML, and writes them to the file tags that are expected by the import function of foo_playcount. You can transfer playlists too.

Note that it only works in one direction! You cannot import a collection of files and have them be matched with the XML.

NOTICE: This tool is very much experimental, and is still in development. By using it, you accept this fact, and you take full responsibility for whatever may happen to your library, including corruption of files, loss of data and general unpleasantness. Please make sure to back up your files before writing any data!

Requirements

  • .NET Framework Version 4.7.2 runtime - download
  • The above requires Windows 7 SP1 or higher.

Third party libraries used

Thanks

Quick start

  1. Import an iTunes Library XML (File -> Open library XML).
  2. Map the XML to files (Tools -> Map entries to files).
  3. Write the statistics to files either:
    • By directly writing the tags in if2ktool (Tools -> Write statistics tags to files).
    • Or in foobar2000 with a Masstagger script (File -> Export masstagger mts). This requires you to map files in "Lookup" mode, instead of Direct or Mapped mode.
  4. Import statistics, by right clicking the tracks in foobar -> Playback Statistics -> Import statistics from file tags.
  5. Remove the tags, either by using Masstagger or with Tools -> Remove statistics tags from files.

Mapping to files

The first step in the process (after importing an iTunes Library XML) is to match every entry in the XML to a file on disk. This process is called "mapping". When a track/entry is mapped, it means the tool has found a file path for that particular track. Before any playback statistics data is written, an entry needs to be mapped to a file first.

There are a couple of different modes for this. For most, it simply involves looking at the Location key in the XML, and writing the tags to the file that exists there. However, if you need to match an XML pulled from another machine, you can use a number of other methods to accomplish the same thing.

Mapping is done using the menu item "Tools -> Map entries to files". Depending on the mode selected, and the number of entries to process, this process may take a few seconds or several minutes.

Modes: Direct

For tagging an iTunes library in-place. Generally most people should use this. It assumes that the xml refers to an existing library that is accessible on the machine (or on the network), and that the files referenced by iTunes are the same exact ones to be loaded by foobar2000. Note that matching is only based on the file path, so if the path varies slightly, the file will not be matched (there are a few options that are used to mitigate this possibility).

Modes: Library Mapping

Used to map the paths in the XML to a different library directory. This substitutes the "Music Folder" portion of the track path in the XML with a path of your choosing. This assumes that the structure of the original library folder is the same as the one provided. If you have copied your library from another computer, or to another directory, but have otherwise not changed the file structure - use this mode.

If you haven't checked "Keep iTunes Media folder organized" in iTunes, this process is very likely to have issues. The closer your file structure is to the original, the better off you'll fare.

Modes: Lookup

Instead of mapping entries based on file path alone, if you have already imported your library into foobar2000, you can use that as a reference for mapping the tracks in the XML to their associated files. In lookup mode, it is required that you generate a JSON file containing tags for all tracks that if2ktool can use to reliably match the data from the XML to. The tags used for matching are; track title, artist, album, track number, and file name - or a subset of those that are present in both the XML and JSON entry. See below for how to generate the data required.

Note that this requires tags to remain consistent between iTunes and foobar2000.

On every iTunes Library XML entry, we collect a list of candidate foobar2000/JSON entry matches, scoring each based on the percentage of tags (present in both entries) are equal, then choosing the candidate with the highest score. Candidates are entries with scores >= 50% (meaning 50% of the present tags were matched). If there are multiple candidate matches with the same score, you are prompted to resolve it yourself.

This mode is very thorough, so it takes much longer than the other modes. It also ignores the anyExtension and fuzzy options for the file path - since we can assume that the JSON points to valid files. We do not perform fuzzy comparison on tags either, as this can introduce too many options with larger libraries.

Loading a lookup JSON will also let if2ktool know the sort order of the tracks in foobar, allowing you to export data that takes advantage of this. This opens up the possibility of writing tags using a Masstagger script (see method 2), which may be more reliable under certain circumstances.

Generating a lookup JSON

Automatically:

if2ktool provides a method of automating the lookup creation process using an AutoHotkey script. It can be launched using "Tools -> Generate JSON lookup", however may not work for everyone. Once created, continue with step 5 and 6 below.

Manually:

Use the following steps to generate track data in the form of a JSON file. You can then use it to map the XML to the associated files.

  1. Make sure you have foo_texttools installed.
  2. Sort the playlist view in foobar2000 in a somewhat uniform order. It doesn't matter how they're sorted, just as long as it's sorted the same way later, when you import the mts file (assuming you take that route). Consider using the Album List (Library -> Album List) to ensure uniformity.
  3. Select all tracks (if in the playlist view). From the context menu, click "Utilities" -> "Text Tools" -> "Advanced...". This will bring up a window which will generate a text output of metadata for the selected files, depending on the pattern used.
  4. Input the patterns below to generate a JSON-formatted list of tracks, then copy the text that is generated into a new plain text file.
    NOTE: Ensure that you aren't using foo_unpack, and that there are no archived tracks in your library before doing this.
  5. In if2ktool (after opening an iTunes Library XML), click "Map XML to files..." under the tools menu. Set the mode to "Lookup", and browse to the location of the JSON file you exported in the previous step.
  6. Click "Start". This wil take a while to match, since it's a fairly rigerous process.

Writing the data

After mapping the iTunes Library XML to the files, the next step is to actually write the tags so that they can be read by foo_playcount.

The following table represents the tags that are written to the files and read by foo_playcount:

Tag Plist key Description
added_timestamp Date Added Integer - LDAP timestamp (also called Windows NT time format, FILETIME, SYSTEMTIME, NTFS file time)
last_played_timestamp Play Date Integer - Same as above
play_count Play Count Integer - Number of plays
rating Rating Popularimeter (POPM) tag in a range of 0-255 (byte):
1     = ★
64   = ★★
128 = ★★★
196 = ★★★★
255 = ★★★★★

Method 1 - Writing tags directly

This writes tags directly to the file using the excellent TagLib-Sharp library. Writing tags with this method is much faster than doing so with Masstagger.

Note that the time that it takes to write tags depends on the size, tag complexity and file format of your library. Larger files take longer to process, and AAC files take significantly longer to write than MP3s. On average this method takes about a minute to process 1000 files on a standard HDD, and likely faster on an SSD.

You can write tags with "Tools -> Write statistics tags to file". Removing tags after they have been imported is done with the "Remove statistics tags from file" in the same menu. Note that you may need to "Reload info" in foobar, using the Tools button in the properties window.

Which formats are supported?

It supports most of the same formats that iTunes does, and a few more:

  • Writing ID3v2 TXXX frames/tags to MP3/AAC/ALAC/AIFF/WAVE files
  • Writing MP4 "Apple annotation box"s (MPEG-4 Part 14) to ALAC and AAC files
  • Writing Vorbis/Xiphcomments for FLAC and OGG files.
  • Writing APE tags for MP3 files (ID3v2 is preferred).

Unsupported file formats:

  • RIFF INFO in WAV files aren't supported as they do not allow custom data. An IDv3 tag will be created in these instances.
  • AIFF files that do not use an ID3v2 chunk are not supported. For these files you will have to manually create an ID3v2 chunk with a tool like kid3.

Regarding WAV files:

WAV files are supported (via ID3v2 tags), however since iTunes does not save tags to WAV files, these files will not contain any informational tags besides the ones written by this tool (unless they had tags before being added to iTunes). This presents an issue in foobar2000, as a playback statistics entry for these files will not be created until there are identifying tags. When importing the playback statistics data written by this tool, these files will be ignored, which will leave gaps in your library.

To avoid this issue, you can check "Write info tags to WAV files" in Preferences -> Tagging. This will cause the tool to check the file tags against the iTunes data, and write the missing data to the file. The data written will include the following tags: Name, Artist, Album Artist, Album, Year, Genre, Track Number + Track Count (as x/y), Disc Number + Disc Count (as x/y), and Comments. For the purpose of simplifying this process, these tags are only written to and read from the ID3v2 chunk, if it exists.

Despite this attempt at a fix, due to TagLib's inability to write foobar-readable tags to WAV files correctly (be it ID3 or RIFF INFO) they will still not be detected in foobar2000. While I don't yet have a solution that isn't annoying, you can try the following:

  • Write tags using the "Write info tags to WAV files" option, then re-write with kid3 (which seems to read the tags fine), or another similar tag-writing program. In kid3 this can be done by converting the ID3 version, or by adding/removing/editing any tag.
  • Use the "export info" option of "File -> Export masstagger mts".

Method 2 - Exporting tags for use with Masstagger

For unsupported formats, and stubborn files (or just as another option) you have the option to export the data to a masstagger script so that playback statistics tags may be written in foobar2000, instead of externally. This assumes that you have already moved your library from iTunes to foobar2000

This method can in some cases be more reliable with writing tags, as the fields, encoding, format and container of the tags written will be the same as what is expected by foobar, and therefore is guaranteed to be read. However, writing a large amount of tags will take much, much longer in masstagger than it does with if2ktool.

This uses Masstagger's "Input data" action, to write tags to tracks, where it is one line of data per track in the current context of foobar2000. Because of this, if2ktool needs to know the sort order of tracks in foobar so that it matches the exported script. This can be accomplished with two different methods.

Method 2.1 - Sort order matching (not recommended)

This method is no longer recommended for matching to many files at once, please use method 2.2 instead!

This method does not require that you map the XML to files, and is only recommended if you need to write tags for a small amount of tracks. It is NOT recommended for large or inconsistently tagged libraries.

  1. In if2ktool_gui (after opening an iTunes Library XML), go "Tools" -> "Generate mts..." and select the files that you want to include. You can choose how to order the export, but by default the file generated will be sorted alphabetically by track name.
  2. In foobar2000, select the target tracks in the playlist view. Make sure that the view is currently sorted in descending order, and by the same value that the exported script is sorted by.
  3. From the playlist view context menu, click "Tagging" -> "Manage scripts" -> "Import from file...", browsing to the file generated earlier. After writing tags, you can import the statistics as normal using the foo_playcount context menu option.

While you no longer have to worry about mapping XML entries to file paths with this method, masstagger's input data doesn't seem to support conditional operators, so matching the data to the track is entirely based on the assumption that order of the export and the order of the playlist view are the same. This is a major concern, since if even one track is out of line or misplaced, it could cause every track after it to be written with incorrect data. I've tested this a lot, and have confirmed that the sorting method used by foobar2000 is the exact same as what is generated in if2ktool (unless you use a custom sorting method for the track title) - so it is up to you to ensure that there are no stray or misnamed files.

Additionally, there can be issues with WAVE files when using the Track title sort order. iTunes doesn't write the tags to file and instead stores tags in the XML). Since foobar2000 doesn't have this data, it will show the file name as the title field, which will muddle up the sort order.

To combat this issue, you can export the lines of data in the masstagger script in different orders. When applying the script, ensure the playlist is sorted in the same order, via a column using the associated f2k field below.

Sort order f2k Field Notes
Track title %title% f2k's title is determined by the tag, or the file name if the track name tag is not present.
Path %path% Will only work if the iTunes Library XML references files at the exact same path as in foobar. The path in the XML will be formatted to match the standard Windows-form path. Note that if the XML is from a Mac, the path will begin with '/' and will not contain a drive portion - making it unsuitable for sorting.
File name %filename_ext% Probably the most reliable, assuming the file names are the same as the ones in the XML, and that you don't have duplicates which may mess up the sort order.
File size (bytes) %filesize% The file size will only be valid prior to additional tagging - thankfully it doesn't change the sort order while writing tags.

Method 2.2 - Cross reference with foobar2000 data (recommended)

By using foo_texttools to get a list of track data from foobar2000, if2ktool is able to sort the exported mts in a way that conforms to a track listing in foobar2000. This will eliminate the possibility of having offset rows and mismatched data (as with the above method). As a side effect, we also retrieve the file paths directly from foobar2000.

In the case that an XML entry can't be matched to a track in foobar2000, we use an empty line to represents its position in the exported Masstagger script.

Steps:

  1. Firstly, create a lookup JSON as per the steps above, and map your XML to it.
  2. Click "Export masstagger mts..." under the File menu. Set the sort order to "Lookup sort order", and then click Export.
  3. In foobar2000, firstly ensure that the sort order is the same as when you exported the data using foo_texttools - otherwise you'll end up mapping to incorrect files. Select all tracks and then from the playlist view context menu, click "Tagging" -> "Manage scripts" -> "Import from file...", browsing to the mts file that was exported. After writing tags, you can import the statistics as normal using the foo_playcount context menu option.

Troubleshooting sort order

In the export window, you can check "Debug sort order". This will write the track title (instead of any iTunes data) to the mts file, and can be used to test that the sort order of the exported file matches up with foobar2000. In the masstagger window, scroll to the bottom and check that the titles match up. If they do, the data has not been offset and should work fine after exporting the full script.

Transferring playlists from iTunes

If2ktool provides a simple way of copying playlists from iTunes, by creating an .m3u or .m3u8 from the XML for import in foobar2000.

If your library is in the exact same place as is listed in the XML, you should just be able to export an .m3u directly from iTunes, and import it straight into foobar2000 with no hassles. Otherwise if your folder structure has changed, or you just want to import them all in one go, you can use this tool.

Steps

  1. Firstly, import an iTunes Library XML and map it to the files using any one of the above methods. This is required to ensure existence of the file paths in foobar2000.
  2. Click File -> Export playlist(s). This will bring up a window showing all the playlists that were imported from the XML.
  3. Select the playlists you want to export (using Control / Shift to select multiple) and then click the "Export" button at the bottom, saving them to wherever. You can optionally check "Extended M3U" to add track information, though this is ignored in foobar, and will have no effect.
  4. In foobar2000, either drag the created file(s) onto the window, or click File -> Load playlist... to import the playlist.

Notes

  • You currently cannot export an iTunes Smart playlist to a foobar2000 Autoplaylist (though this is a feature I'd like to add at some point). You can however still choose to export a smart playlists' results as a regular playlist.

  • Some playlists are automatically skipped, including "Music", "Podcasts", and "Audiobooks".

  • During the playlist creation process, if the path has any non-ASCII, special characters - the file needs to be saved as an .m3u8 and NOT an .m3u so that foobar2000 knows how to properly interpret these characters. Otherwise these paths will not be found, and the playlist will have gaps. Check either the foobar console for errors, or the number of tracks in the playlist view against the number that is listed in the export window of if2ktool.

foo_texttool JSON patterns

These patterns should be used in the "Text Tools" window, of foobar2000, and will create a JSON file with some tags pertaining to each track that if2ktool use to match entries to. You can also find them under "Tools -> Generate JSON lookup".

These patterns will only escape double-quote, tab and backslash characters. I doubt many people have other control characters in their tags. If this is the case, you can add yet another $replace surrounding each value.

Formatted / pretty

  • Track pattern:
$pad( ,4){$char(13)$char(10)$pad( ,8)"index": $sub(%list_index%,1),$char(13)$char(10)$pad( ,8)"title": "$replace($replace($replace([$meta(Title)],\,\\),",\"),$tab(),\t)",$char(13)$char(10)$pad( ,8)"artist": "$replace($replace($replace([%artist%],\,\\),",\"),$tab(),\t)",$char(13)$char(10)$pad( ,8)"album": "$replace($replace($replace([%album%],\,\\),",\"),$tab(),\t)",$char(13)$char(10)$pad( ,8)"trackNumber": $replace("[%tracknumber%]","",null),$char(13)$char(10)$pad( ,8)"path": "$replace($replace($replace([%path%],\,\\),",\"),$tab(),\t)",$char(13)$char(10)$pad( ,4)},
  • Group header pattern:
{$char(13)$char(10)$pad( ,4)"tracks": '['
  • Group footer pattern:
$pad( ,4)']'$char(13)$char(10)}

Unformatted / not pretty

  • Track pattern:
{"index":$sub(%list_index%,1),"title":"$replace($replace($replace([$meta(Title)],\,\\),",\"),$tab(),\t)","artist":"$replace($replace($replace([%artist%],\,\\),",\"),$tab(),\t)","album":"$replace($replace($replace([%album%],\,\\),",\"),$tab(),\t)","trackNumber":$replace("[%tracknumber%]","",null),"path":"$replace($replace($replace([%path%],\,\\),",\"),$tab(),\t)"},
  • Group header pattern:
{"tracks":'[' * Group footer pattern:
']'}

This generates a file like the following:

{
    "tracks": [ 
    {
        "title": "High Fidelity",
        "artist": "Daft Punk",
        "album": "Homework",
        "trackNumber": 01
        "path": "D:\\some\\path\\to\\Music\\Daft Punk\\Homework\\High Fidelity\\10 High Fidelity.mp3"
    },
    ...
    ]
}

Preferences

General

Logging

  • Show console:
    Show or hide the console.
  • Full logging:
    Logs much more information (including some debug information) to the console. Enabling this will slow down processes, and will consume much more memory.
  • Log to file:
    Logs all console output to a file, in the same folder as the executable. This may have a significant performance impact when logging in the same thread, especially when "Full logging" is enabled. Consider enabling ""Log in separate thread"" if this has an impact on performance. This file is closed when the application is closed, and ideally shouldn't be open while the application is open.
  • Log in separate thread:
    Performs all logging in a separate thread. This greatly speeds up processing time while logging, but means that the logging is no longer in sync with the process, which has a side-effect of not being able to pause the application by selecting the log.

Tagging

  • Dry run: (bool) [default: false]
    Doesn't write anything to files but will simulate output. This is useful for previewing the output of if2ktool without making changes. This will still ask you to write tags, but won't follow through (promise!). Dry run isn't able to catch file-saving related errors.

  • Write info tags to wave files: (bool) [default: true]
    iTunes does not write tags to WAV files. This presents an issue with foo_playcount, as it relies on present tags in order to create a playback statistics entry. If any identifying tags are not present, foobar will skip over them, leaving gaps in your library.

    If this option is enabled, (and for WAV files only) the tool will check the file tags against the iTunes data, and will write the missing data to the file. This includes the following tags: Name, Artist, Album Artist, Album, Genre, Comments, Year, Disc Number, Disc Count, Track Number, Track Count. See Method 1 - Writing tags directly for more info.

  • ID3

    • Force ID3v2 version: (bool + dropdown) [default: false, ID3v2.3]
      Force TagLib to write ID3v2 tags in a specific version for every file, rather than in the version of the source file.
    • Force ID3v2 encoding: (bool + dropdown) [default: false, UTF-8]
      This sets the encoding used for new ID3v2 text information frames. Note that UTF-8 isn't officially supported in ID3v2.3 and below, though many applications support it anyway. This setting is only really used when ""Write info tags to WAV files"" is used, as all playback statistics tags use ASCII encoding.
      • False: UTF-8 will be used as the default encoding for new text-based frames, and TagLib will automatically fall back to using UTF-16 if the version is < ID3v2.4.
      • True: The encoding for newly-written frames will be forced to this encoding, regardless of ID3v2 version. Existing frames will continue to use whatever encoding they were previously using.
    • Force remove ID3v1: (bool) [default: false]
      Forcibly remove all ID3v1 tags from files processed.
    • Use numeric genres in ID3: (bool) [default: true]
      Disable to prevent genres from being written as indices in ID3v2 tags.

Mapping

  • Any Extension:
    If you wish to map to files that may have different extensions. It will try to match files with the original XML entry extension, then looks in the same directory and choose the first file with the same file name (minus extension) that is supported by TagLib.

  • Fuzzy distance:
    The maximum number of characters a file name can be off in order to be mapped. This value depends on how messed up your target file names are, so if perhaps they are missing a padded zero for the track number, or have had a special character substituted with '-' instead of '_', you can set the fuzzy distance to 1 to correct this.

    Internally this calculates the Levenshtein distance on every other file name (NOT file path, and minus extension) in the target folder, and then selects the one with the least changes necessary to make a match.

    Do NOT use a large value for the fuzzy distance, because then every name will be mapped to every other name!

  • Normalize strings:
    "Normalizes" the file path string if a file at the mapped/original path was not found. This converts the string to a "unicode-normalized form C", helping circumvent issues that may arise through the use of combining diacritic characters, and accents being able to be represented in different ways on the file system.

    For example the visual character ō can be represented in the URL-encoded path as either %C5%8D (decoded to ō), or o%CC%84 (decoded to the separate o character, and a combining maron diacritic ̄  ). In this example, both characters are perfectly valid for file names in Windows, and therefore you can have two files that appear to have the same name, but in fact are at different paths.

    If your XML uses one form, and the actual file name uses another, the path in the XML will not be found. This can occur when files are moved over a network, and some applications will automatically normalize the file name - while iTunes usually keeps whatever was used originally. This process simply calls string.Normalize() where applicable, converting combining characters to their single character representations, among other things.

  • Check for duplicates:
    Having this option checked will force the mapper to ignore a matching path if an entry has already been mapped to it, preventing duplicates. This occurs inherantly in Lookup mode (even without this option enabled)

  • Lookup minimum matching percent:
    This is the minimum percentage of present and matching tags that an entry needs to have in order to be matched to a track in the lookup JSON file, during Lookup matching. This is 50% by default.

    Low percentages can yield incorrect matches, while higher percentages will limit the margin for error and can yield less matches. Note that a 100% match where both entries only have one present tag is still a perfect match!

Troubleshooting + other stuff

Missing iTunes Library XML

In recent versions of iTunes, Apple have hidden the XML from the iTunes directory. It can be shown again with the setting "Share iTunes Library XML wth other applications" under Edit -> Preferences -> Advanced.

Resetting playback statistics in foobar2000

If at any point you wish to remove the saved playback statistics in foobar, you can delete the index-data folder in C:\Users\user\AppData\Roaming\foobar2000\ or %APPDATA%\foobar2000\. Make sure that it isn't running while doing so. Note that this doesn't delete tagged playback statistics.

Removing playback statistics tags after they have been written

This can be done using "Tools -> Remove playback statistics tags from files". It's a good idea to keep the window open after writing tags, as currently there is no way to save mappings. You might need to "Reload info" in foobar2000 after doing this, via the Tools button in the properties window, accessed from the context menu of any track.

Warning regarding ID3v2.4-tagged tracks not being recognized by foobar2000

TabLib has an issue with a small amount of ID3v2.4.0 files. After writing tags in if2ktool, the tags and ID3 version will be unable to be read by foobar2000 when importing the files for the first time. As a precaution, you should import your library into foobar2000 first. If this has occured, you can try editing the tags in another application and/or converting the tags from ID3v2.4 to v2.3 using a tool like kid3.

Help! My target files are one character off, and aren't being found by the mapper!

You can try enabling the "Normalize strings" or "Fuzzy distance" options as above.

Help! I'm getting an IOException while writing tags!

I've encountered this issue with about 30 or so of the files in my library or 20,000 or so. The exact error is The process cannot access the file 'C:\path\to\the\file.mp3' because it is being used by another process.". While sometime this can be the case, if you've accidentally left a program open which is locking the file (in which case, the tool prints the program in question), other times it occurs without explanation. I have a set of files that seem to always do this, no matter what. After looking into it, it seems TagLib is having trouble creating a WriteStream for these specific files. If this is the case, the tool will retry with modified load settings until the file is loaded correctly.

About

A gui-based tool used to migrate some stats (play count, rating, date added and last played) from iTunes to foobar2000's foo_playcount component (Playback Statistics).

Topics

windows itunes tagging foobar2000 windows-forms idv3

Resources

Readme
Activity

Stars

6 stars

Watchers

1 watching

Forks

1 fork
Report repository

Releases 1

v1.0.7091.6435 Latest
May 31, 2019

Packages

No packages published

Contributors 2

  •  
  •  

Languages