Merge pull request #13 from mynttt/experimental

Experimental

CHANGELOG.md

@@ -1,3 +1,9 @@
+## 1.3:
+  - Series can now also be updated by utilizing TVDB to resolve IMDB ids
+  - TVDB <=> IMDB resolvement only has to be done once
+  - Items that fail to be resolved will be blacklisted for 14 days to prevent spamming TVDB on every iteration of the tool
+  - TVDB authorization is a bit more complex than simply providing an API key, more about the topic is in the README.md
+
 ## 1.2.5:
   - Removed deprecated legacy CLI interface from project
   - Deprecated and removed OMDB interfaces and implementations, this tool will no longer use the OMDB API as IMDB provides a rating dataset that is refreshed daily and thus more up to date

README.md

@@ -1,6 +1,6 @@
 # Rating update tool for IMDB ratings in Plex libraries
 
-A tool to update the IMDB ratings for Plex libraries that contain movies.
+A tool to update the IMDB ratings for Plex libraries that contain movies and series.
 
 ## What does this do?
 
@@ -12,7 +12,7 @@ This tool allows you to update the database that stores this data with the corre
 
 An advantage is that it works outside Plex by manipulating the local Plex database. Thus, no metadata refresh operations have to be done within Plex. It is faster and will not lead into the unforeseen consequences that one sometimes experiences with a Plex metadata refresh (missing or changed posters if not using a custom poster).
 
-This tool currently only works on movies and will only allow you to select libraries that use the Plex IMDB agent (because it depends on the IMDB ids). In my library with 1800 movies it transformed entries for 698 items. In case that even tho you use the IMDB agent you still have items that are TMDB matched you can run it with an TMDB API key and it will match an IMDB rating to the TMDB item (if TMDB provides an IMDB id).
+This tool currently works on movies/series that use the Plex IMDB agent as source of ratings. For the movies it will match items that use the imdb/tmdb agent. For series, it will use the tvdb to resolve the tvdb <=> imdb relationship (which can fail if the tvdb has no imdb id matched to the item). In my library with 1800 movies it transformed entries for 698 items and 1000+ entries for series. In case that even tho you use the IMDB agent you still have items that are TMDB matched you can run it with an TMDB API key and it will match an IMDB rating to the TMDB item (if TMDB provides an IMDB id).
 
 Before (Not IMDB matched)            |  After Match
 :-------------------------:|:-------------------------:
@@ -43,6 +43,15 @@ docker run -dit -e RUN_EVERY_N_HOURS=12 \
     -v "/mnt/data/Plex Media Server":/plexdata \
     -v "/mnt/data/imdpupdaterconfig":/config \
     mynttt/updatetool
+
+# With TMDB fallback and TVDB resolvement for series
+
+docker run -dit -e RUN_EVERY_N_HOURS=12 \
+    -e TMDB_API_KEY=yourkey \
+    -e TVDB_AUTH_STRING="tvdbusername;tvdbuserid;tvdbapikey" \
+    -v "/mnt/data/Plex Media Server":/plexdata \
+    -v "/mnt/data/imdpupdaterconfig":/config \
+    mynttt/updatetool
 ```
 
 Explained:
@@ -53,6 +62,9 @@ docker run -dit
     -e RUN_EVERY_N_HOURS=12 \
      # Optional parameter: will try to get an IMDB ID from TMDB matched items
     -e TMDB_API_KEY=yourkey \
+     # Three items are required to auth with TVDB username, userkey, apikey
+     # Supply these as semicolon seperated values. Example: username;DAWIDK9CJKWFJAWKF;e33914feabd52e8192011b0ce6c8
+    -e TVDB_AUTH_STRING="tvdbusername;tvdbuserkey;tvdbapikey" \
      # The plex data root (that contains Plug-ins, Metadata, ...
      # https://support.plex.tv/articles/202915258-where-is-the-plex-media-server-data-directory-located/
     -v "/mnt/data/Plex Media Server":/plexdata \
@@ -61,6 +73,8 @@ docker run -dit
     mynttt/updatetool
 ```
 
+[TVDB User Key](https://thetvdb.com/dashboard/account/editinfo) - [TVDB API Key](https://thetvdb.com/dashboard/account/apikey)
+
 *"/mnt/data/Plex Media Server" and "/mnt/data/imdpupdaterconfig" are just sample paths! Set your own paths there or it will probably not work!*
 
 *On windows the \ syntax to make the command multiline will not work. You have to remove those and make the command a single line command!*
@@ -81,7 +95,9 @@ docker run -dit
 
 **If the /config folder does not exist yet in appdata unraid will create it! It is important to access logs easily!**
 
-![](img/unraidv2.PNG)
+![](img/unraidv3.PNG)
+
+*TMDB and TVDB are optional settings that are not required for base movie imdb operations! TMDB unlocks matching for movies that have a TMBD match for whatever reason and TVDB allows to update series as well!*
 
 6.) You can now start the container. If it has errors it will stop. The log in the config folder shows you what it does or why it crashed if that happens.
 
@@ -100,9 +116,11 @@ Provides a watchdog that once started will run every N hours over all IMDB suppo
 # Created files in PWD
 
 - cache-tmdb2imdb.json - If TMDB fallback is enabled this file will contain the resolved TMDB <=> IMDB mappings.
+- cache-tvdb2imdb.json - TVDB to IMDB mapping.
+- cache-tvdbBlacklist.json - Items that TVDB provides no IMDB id for or that fail being looked up. The blacklist is reset every 14 days.
 - state-imdb.json - Set of jobs that have not finished
 - xml-error-{uuid}-{library}.log - List of files that could not be updated by the XML transform step (not important tbh, plex reads from the DB)
-- updatetool.log - Log file
+- updatetool.{increment}.log - Log file
 - rating_set.tsv - latest IMDB rating set
 - ratingSetLastUpdate - UNIX timestamp of last rating set update
 

VERSION

@@ -1 +1 @@
-1.2.5
+1.3

build.gradle

@@ -4,7 +4,7 @@ plugins {
     id 'eclipse'
 }
 
-version = '1.2.5'
+version = '1.3'
 sourceCompatibility = '11'
 
 new File(projectDir, "VERSION").text = version;

src/main/java/updatetool/Main.java

@@ -2,6 +2,7 @@
 
 import java.io.IOException;
 import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
 import java.nio.file.Path;
 import java.nio.file.Paths;
 import java.util.Arrays;
@@ -11,6 +12,7 @@
 import updatetool.api.Implementation;
 import updatetool.common.AbstractApi;
 import updatetool.common.TmdbApi;
+import updatetool.common.TvdbApi;
 import updatetool.imdb.ImdbDockerImplementation;
 
 public class Main {
@@ -41,6 +43,9 @@ public enum Implementations {
                 new String[] { "The following environment variables must be set and exported before launching this tool successfully!",
                              "PLEX_DATA_DIR: Used for the data directory of plex",
                              "(Optional) TMDB_API_KEY: Used to convert TMDB matched items to IMDB items. The fallback will only be available if this is set.",
+                             "(Optional) TVDB_AUTH_STRING: Used to auth with the TVDB API. Must be entered as a ';' seperated string of username, userid, apikey",
+                             "Example: username;DAWIDK9CJKWFJAWKF;e33914feabd52e8192011b0ce6c8",
+                             "",
                              "No parameters starts with the default of {every_n_hour} = 12, {cache_pruge_in_days} = 14 and {new_movie_cache_purge_threshold} = 12",
                              "{every_n_hour} : Invoke this every n hour on all IMDB supported libraries"});
 
@@ -66,6 +71,8 @@ public static Implementations of(String string) {
     }
 
     public static void main(String[] args) throws Exception {
+        preLogPurge();
+
         Thread.setDefaultUncaughtExceptionHandler((t, e) -> {
             Logger.error("Uncaught " + e.getClass().getSimpleName() + " exception encountered...");
             Logger.error("Please contact the maintainer of the application with the stacktrace below if you think this is unwanted behavior.");
@@ -100,6 +107,30 @@ public static void main(String[] args) throws Exception {
         constructor.newInstance().invoke(args);
     }
 
+    public static void rollingLogPurge() throws IOException {
+        var files = Files.list(Main.PWD)
+                .filter(p -> p.getFileName().toString().startsWith("updatetool.")).collect(Collectors.toList());
+        var keep = files.stream().max((p1, p2) -> {
+            try {
+                return Files.getLastModifiedTime(p2).compareTo(Files.getLastModifiedTime(p1));
+            } catch (IOException e) {}
+            return 0;
+        });
+        if(files.size() > 1) {
+            keep.ifPresent(f -> files.remove(f));
+            for(var f : files)
+                Files.delete(f);
+        }
+    }
+
+    private static void preLogPurge() throws IOException {
+        var files = Files.list(Main.PWD)
+            .filter(p -> p.getFileName().toString().startsWith("updatetool."))
+            .collect(Collectors.toList());
+        for(var p : files)
+            Files.delete(p);
+    }
+
     public static void printHelp(Implementations i, boolean datahint) {
         if(datahint)
             System.out.println("Data folder: https://support.plex.tv/articles/202915258-where-is-the-plex-media-server-data-directory-located");
@@ -128,6 +159,17 @@ public static void testApiTmdb(String apikeyTmdb) throws Exception {
         var api = new TmdbApi(apikeyTmdb);
         genericApiTest(api);
     }
+
+    public static void testApiTvdb(String[] credentials) {
+        Logger.info("Testing TVDB API authorization: username={} | userkey={} | apikey={}", credentials[0], credentials[1], credentials[2]);
+        try {
+            new TvdbApi(credentials);
+        } catch(IllegalArgumentException e) {
+            Logger.error("API Test failed: " + e.getMessage());
+            Logger.error("Keys available under: https://thetvdb.com/");
+            System.exit(-1);
+        }
+    }
 
     private static void genericApiTest(AbstractApi api) throws Exception {
         var response = api.testApi();

src/main/java/updatetool/api/Job.java

@@ -2,15 +2,19 @@
 
 import java.util.Objects;
 import updatetool.api.Pipeline.PipelineStage;
+import updatetool.common.DatabaseSupport.Library;
+import updatetool.common.DatabaseSupport.LibraryType;
 
 public abstract class Job {
     public String library;
     public String uuid;
+    public LibraryType libraryType;
     public PipelineStage stage = PipelineStage.CREATED;
 
-    public Job(String library, String uuid) {
-        this.library = library;
-        this.uuid = uuid;
+    public Job(Library library) {
+        this.library = library.name;
+        this.uuid = library.uuid;
+        this.libraryType = library.type;
     }
 
     public abstract String whatKindOfJob();

src/main/java/updatetool/common/AbstractApi.java

@@ -6,18 +6,15 @@
 import java.net.http.HttpClient;
 import java.net.http.HttpClient.Version;
 import java.net.http.HttpRequest;
+import java.net.http.HttpRequest.BodyPublishers;
 import java.net.http.HttpResponse;
 import java.net.http.HttpResponse.BodyHandlers;
 import java.time.Duration;
-import java.util.Objects;
 
 public abstract class AbstractApi {
-    private final String apiKey;
     private final HttpClient client;
 
-    public AbstractApi(String apiKey) {
-        Objects.requireNonNull(apiKey);
-        this.apiKey = apiKey;
+    public AbstractApi() {
         this.client = HttpClient.newBuilder()
                 .version(Version.HTTP_2)
                 .connectTimeout(Duration.ofMillis(2000))
@@ -36,12 +33,20 @@ protected final HttpRequest get(String url) {
             throw Utility.rethrow(e);
         }
     }
+
+    protected final HttpRequest postJson(String url, String jsonBody) {
+        try {
+            return HttpRequest.newBuilder(new URI(url))
+                    .header("Content-Type", "application/json")
+                    .POST(BodyPublishers.ofString(jsonBody))
+                    .build();
+        } catch(URISyntaxException e) {
+            throw Utility.rethrow(e);
+        }
+    }
 
     protected HttpResponse<String> send(HttpRequest request) throws IOException, InterruptedException {
         return client.send(request, BodyHandlers.ofString());
     }
 
-    protected final String apikey() {
-        return apiKey;
-    }
 }

src/main/java/updatetool/common/DatabaseSupport.java

@@ -12,18 +12,37 @@ public DatabaseSupport(SqliteDatabaseProvider provider) {
         this.provider = provider;
     }
 
-    public class LibraryItem {
+    public enum LibraryType {
+        MOVIE(1),
+        SERIES(2);
+
+        private int n;
+
+        LibraryType(int n) {
+            this.n = n;
+        }
+
+        public static LibraryType of (int n) {
+            for(var s : values())
+                if(s.n == n) return s;
+            throw new IllegalArgumentException("number not present");
+        }
+    }
+
+    public class Library {
+        public final LibraryType type;
         public final long id;
         public final int items;
         public final String name;
         public final String uuid;
 
-        private LibraryItem(ResultSet r, SqliteDatabaseProvider p) throws SQLException {
+        private Library(ResultSet r, SqliteDatabaseProvider p) throws SQLException {
             try(var handle = p.queryFor("SELECT count(*) FROM media_items WHERE library_section_id = " + r.getLong(1))) {
                 items = handle.result().getInt(1);
                 id = r.getLong(1);
                 name = r.getString(2);
                 uuid = r.getString(3);
+                type = LibraryType.of(r.getInt(4));
             } catch(SQLException e) {
                 throw e;
             }
@@ -35,15 +54,23 @@ public String toString() {
         }
     }
 
-    public List<LibraryItem> requestLibraries() {
-        try(var handle = provider.queryFor("SELECT id, name, uuid FROM library_sections WHERE section_type = 1 AND agent = 'com.plexapp.agents.imdb'")) {
-            var list = new ArrayList<LibraryItem>();
+    public List<Library> requestMovieLibraries() {
+        return requestLibrary("SELECT id, name, uuid, section_type FROM library_sections WHERE section_type = 1 AND agent = 'com.plexapp.agents.imdb'");
+    }
+
+    public List<Library> requestSeriesLibraries() {
+        return requestLibrary("SELECT id, name, uuid, section_type FROM library_sections WHERE section_type = 2 AND agent = 'com.plexapp.agents.thetvdb'");
+    }
+
+    private List<Library> requestLibrary(String sql) {
+        try(var handle = provider.queryFor(sql)) {
+            var list = new ArrayList<Library>();
             while(handle.result().next())
-                list.add(new LibraryItem(handle.result(), provider));
+                list.add(new Library(handle.result(), provider));
             return list;
         } catch (SQLException e) {
             throw Utility.rethrow(e);
-        }
+        } 
     }
 
 }

src/main/java/updatetool/common/TmdbApi.java

@@ -4,6 +4,7 @@
 import java.net.http.HttpResponse;
 
 public class TmdbApi extends AbstractApi {
+    private final String apiKey;
 
     public static class TMDBResponse {
         public final String imdb_id, title;
@@ -15,7 +16,8 @@ public TMDBResponse(String imdb_id, String title) {
     }
 
     public TmdbApi(String apiKey) {
-        super(apiKey);
+        super();
+        this.apiKey = apiKey;
     }
 
     public HttpResponse<String> tmdbId2imdbId(String tmdbId) throws IOException, InterruptedException {
@@ -31,7 +33,7 @@ public HttpResponse<String> testApi() throws IOException, InterruptedException {
     }
 
     private String of(String tmdbId) {
-        return String.format("https://api.themoviedb.org/3/movie/%s?api_key=%s", tmdbId, apikey());
+        return String.format("https://api.themoviedb.org/3/movie/%s?api_key=%s", tmdbId, apiKey);
     }
 
     @Override