Merge branch 'IQSS:develop' into archival-metadata-block

doc/release-notes/10472-review-modify-jsonprinter-for-datasetfieldtype.md

@@ -0,0 +1,8 @@
+### Json Printer Bug fix
+
+DatasetFieldTypes in MetadataBlock response that are also a child of another DatasetFieldType were being returned twice. The child DatasetFieldType was included in the "fields" object as well as in the "childFields" of it's parent DatasetFieldType. This fix suppresses the standalone object so only one instance of the DatasetFieldType is returned (in the "childFields" of its parent).
+This fix changes the Json output of the API `/api/dataverses/{dataverseAlias}/metadatablocks`
+
+## Backward Incompatible Changes
+
+The Json response of API call `/api/dataverses/{dataverseAlias}/metadatablocks` will no longer include the DatasetFieldTypes in "fields" if they are children of another DatasetFieldType. The child DatasetFieldType will only be included in the "childFields" of it's parent DatasetFieldType.

doc/release-notes/10516_legacy_permalink_config_fix.md

@@ -0,0 +1 @@
+Support for legacy configuration of a PermaLink PID provider, e.g. using the :Protocol,:Authority, and :Shoulder settings, is fixed. 

doc/release-notes/11038-unconsidered-harvesting-granularity.md

@@ -0,0 +1,2 @@
+Bug Fix:
+OAI Client harvesting now uses the correct granularity while re-run a partial harvest (using the `from` parameter). The correct granularity comes from the `Identify` verb request.

doc/release-notes/11083-mydata-npe-with-harvested-dataverses.md

@@ -0,0 +1 @@
+Fix a bug with My Data where listing dataverses for a user with only rights on harvested dataverses would result in a server error response.

doc/release-notes/11107-fake-to-perma-demo.md

@@ -0,0 +1,3 @@
+### Demo/Eval Container Tutorial
+
+The demo/eval container tutorial has been updated to use the Permalink PID provider instead of the FAKE DOI Provider. See also #11107.

doc/release-notes/11113-avoid-orphan-perm-docs.md

@@ -0,0 +1,5 @@
+This release fixes a bug that caused Dataverse to generate unnecessary solr documents for files when a file is added/deleted from a draft dataset. These documents could accumulate and potentially impact performance.
+
+Assuming the upgrade to solr 9.7.0 also occurs in this release, there's nothing else needed for this PR. (Starting with a new solr insures the solr db is empty and that a reindex is already required.)
+
+

doc/release-notes/11130-update-dataverse-api-remove-metadatablocks.md

@@ -0,0 +1,8 @@
+### Fixes consequences for not adding some optional fields in update dataverse API
+
+Omitting optional fields inputLevels, facetIds, or metadataBlockNames caused data to be deleted.
+This fix no longer deletes data for these fields. Two new flags have been added to the ``metadataBlocks`` Json object to signal the deletion of the data.
+- ``inheritMetadataBlocksFromParent: true`` will remove ``metadataBlockNames`` and ``inputLevels`` if the Json objects are omitted.
+- ``inheritFacetsFromParent: true`` will remove ``facetIds`` if the Json object is omitted.
+
+For more information, see issue [#11130](https://github.com/IQSS/dataverse/issues/11130)

doc/release-notes/11133-search-fix.md

@@ -0,0 +1,3 @@
+### Search fix when using AVOID_EXPENSIVE_SOLR_JOIN=true
+
+Dataverse v6.5 introduced a bug which causes search to fail for non-superusers in multiple groups when the AVOID_EXPENSIVE_SOLR_JOIN feature flag is set to true. This releases fixes the bug.

doc/release-notes/8808-10575-update-global-role.md

@@ -0,0 +1,11 @@
+## Release Highlights
+
+### Update a Global Role
+
+A new API endpoint has been added that allows a global role to be updated. See [Native API Guide > Update Global Role](https://guides.dataverse.org/en/6.3/api/native-api.html#update-global-role) (#10612)
+
+## Bug fixes
+
+### Edition of custom role fixed
+
+It is now possible to edit a custom role with the same alias (reported in #8808)

doc/sphinx-guides/source/api/native-api.rst

@@ -128,12 +128,23 @@ Note that setting any of these fields overwrites the previous configuration.
 
 When it comes to omitting these fields in the JSON:
 
-- Omitting ``facetIds`` or ``metadataBlockNames`` causes the Dataverse collection to inherit the corresponding configuration from its parent.
-- Omitting ``inputLevels`` removes any existing custom input levels in the Dataverse collection.
-- Omitting the entire ``metadataBlocks`` object in the request JSON would exclude the three sub-objects, resulting in the application of the two changes described above.
+- Omitting ``facetIds`` or ``metadataBlockNames`` causes no change to the Dataverse collection. To delete the current configuration and inherit the corresponding configuration from its parent include the flag ``inheritFacetsFromParent`` and/or ``inheritMetadataBlocksFromParent`` respectively.
+- Omitting ``inputLevels`` causes no change to the Dataverse collection. Including the flag ``inheritMetadataBlocksFromParent`` will cause the custom ``inputLevels`` to be deleted and inherited from the parent.
+- Omitting the entire ``metadataBlocks`` object in the request JSON would cause no change to the ``inputLevels``, ``facetIds`` or ``metadataBlockNames`` of the Dataverse collection.
 
 To obtain an example of how these objects are included in the JSON file, download :download:`dataverse-complete-optional-params.json <../_static/api/dataverse-complete-optional-params.json>` file and modify it to suit your needs.
 
+To force the configurations to be deleted and inherited from the parent's configuration include the following ``metadataBlocks`` object in your JSON
+
+.. code-block:: json
+
+  "metadataBlocks": {
+    "inheritMetadataBlocksFromParent": true,
+    "inheritFacetsFromParent": true
+  }
+
+.. note:: Including both the list ``metadataBlockNames`` and the flag ``"inheritMetadataBlocksFromParent": true`` will result in an error being returned {"status": "ERROR", "message": "Metadata block can not contain both metadataBlockNames and inheritMetadataBlocksFromParent: true"}. The same is true for ``facetIds`` and ``inheritFacetsFromParent``.
+
 See also :ref:`collection-attributes-api`.
 
 .. _view-dataverse:
@@ -4558,12 +4569,12 @@ The JSON representation of a role (``roles.json``) looks like this::
 
   {
     "alias": "sys1",
-    "name": “Restricted System Role”,
-    "description": “A person who may only add datasets.”,
+    "name": "Restricted System Role",
+    "description": "A person who may only add datasets.",
     "permissions": [
       "AddDataset"
     ]
-  } 
+  }
 
 .. note:: alias is constrained to a length of 16 characters
 
@@ -5721,22 +5732,43 @@ Creates a global role in the Dataverse installation. The data POSTed are assumed
 .. code-block:: bash
 
   export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
-  export SERVER_URL=https://demo.dataverse.org
-  export ID=root
+  export SERVER_URL=http://localhost:8080
 
-  curl -H "X-Dataverse-key:$API_TOKEN" -X POST "$SERVER_URL/api/admin/roles" --upload-file roles.json
+  curl -H "Content-Type: application/json" -H "X-Dataverse-key:$API_TOKEN" -X POST "$SERVER_URL/api/admin/roles" --upload-file roles.json
+
+``roles.json`` see :ref:`json-representation-of-a-role`
+
+Update Global Role
+~~~~~~~~~~~~~~~~~~
+
+Update a global role in the Dataverse installation. The PUTed data is assumed to be a complete JSON role as it will overwrite the existing role. ::
+
+    PUT http://$SERVER/api/admin/roles/$ID
+
+A curl example using an ``ID``
+
+.. code-block:: bash
+
+  export SERVER_URL=http://localhost:8080
+  export ID=24
+
+  curl -H "Content-Type: application/json" -X PUT "$SERVER_URL/api/admin/roles/$ID" --upload-file roles.json
 
 ``roles.json`` see :ref:`json-representation-of-a-role`
 
 Delete Global Role
 ~~~~~~~~~~~~~~~~~~
 
+Deletes an ``DataverseRole`` whose ``id``  is passed. ::
+
+    DELETE http://$SERVER/api/admin/roles/$ID
+
 A curl example using an ``ID``
 
 .. code-block:: bash
 
   export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
-  export SERVER_URL=https://demo.dataverse.org
+  export SERVER_URL=http://localhost:8080
   export ID=24
 
   curl -H "X-Dataverse-key:$API_TOKEN" -X DELETE "$SERVER_URL/api/admin/roles/$ID"
@@ -6625,6 +6657,8 @@ MyData
 
 The MyData API is used to get a list of just the datasets, dataverses or datafiles an authenticated user can edit.
 
+The API excludes dataverses linked to an harvesting client. This results in `a known issue <https://github.com/IQSS/dataverse/issues/11083>`_ where regular datasets in harvesting dataverses are missing from the results.
+
 A curl example listing objects
 
 .. code-block:: bash

doc/sphinx-guides/source/container/running/demo.rst

@@ -160,6 +160,11 @@ Next, set up the UI toggle between English and French, again using the unblock k
 
 Stop and start the Dataverse container in order for the language toggle to work.
 
+PID Providers
++++++++++++++
+
+Dataverse supports multiple Persistent ID (PID) providers. The ``compose.yml`` file uses the Permalink PID provider. Follow :ref:`pids-configuration` to reconfigure as needed.
+
 Next Steps
 ----------
 

docker/compose/demo/compose.yml

@@ -20,12 +20,12 @@ services:
         -Ddataverse.files.file1.type=file
         -Ddataverse.files.file1.label=Filesystem
         -Ddataverse.files.file1.directory=${STORAGE_DIR}/store
-        -Ddataverse.pid.providers=fake
-        -Ddataverse.pid.default-provider=fake
-        -Ddataverse.pid.fake.type=FAKE
-        -Ddataverse.pid.fake.label=FakeDOIProvider
-        -Ddataverse.pid.fake.authority=10.5072
-        -Ddataverse.pid.fake.shoulder=FK2/
+        -Ddataverse.pid.providers=perma1
+        -Ddataverse.pid.default-provider=perma1
+        -Ddataverse.pid.perma1.type=perma
+        -Ddataverse.pid.perma1.label=Perma1
+        -Ddataverse.pid.perma1.authority=DV
+        -Ddataverse.pid.perma1.permalink.separator=/
         #-Ddataverse.lang.directory=/dv/lang
     ports:
       - "8080:8080" # HTTP (Dataverse Application)

src/main/java/edu/harvard/iq/dataverse/DataFile.java

@@ -1142,4 +1142,12 @@ public boolean isDeaccessioned() {
         }
         return inDeaccessionedVersions; // since any published version would have already returned
     }
+    public boolean isInDatasetVersion(DatasetVersion version) {
+        for (FileMetadata fmd : getFileMetadatas()) {
+            if (fmd.getDatasetVersion().equals(version)) {
+                return true;
+            }
+        }
+        return false;
+    }
 } // end of class

src/main/java/edu/harvard/iq/dataverse/DataverseRoleServiceBean.java

@@ -23,7 +23,6 @@
 import jakarta.persistence.EntityManager;
 import jakarta.persistence.PersistenceContext;
 import jakarta.persistence.TypedQuery;
-//import jakarta.validation.constraints.NotNull;
 
 /**
  *
@@ -40,6 +39,9 @@ public class DataverseRoleServiceBean implements java.io.Serializable {
 
     @EJB
     RoleAssigneeServiceBean roleAssigneeService;
+
+    @EJB
+    DataverseServiceBean dataverseService;
     @EJB
     IndexServiceBean indexService;
     @EJB
@@ -48,22 +50,23 @@ public class DataverseRoleServiceBean implements java.io.Serializable {
     IndexAsync indexAsync;
 
     public DataverseRole save(DataverseRole aRole) {
-        if (aRole.getId() == null) {
+        if (aRole.getId() == null) { // persist a new Role
             em.persist(aRole);
-            /**
-             * @todo Why would getId be null? Should we call
-             * indexDefinitionPoint here too? A: it's null for new roles.
-             */
-            return aRole;
-        } else {
-            DataverseRole merged = em.merge(aRole);
-            /**
-             * @todo update permissionModificationTime here.
-             */
-            IndexResponse indexDefinitionPountResult = indexDefinitionPoint(merged.getOwner());
-            logger.info("aRole getId was not null. Indexing result: " + indexDefinitionPountResult);
-            return merged;
+        } else { // update an existing Role
+            aRole = em.merge(aRole);
+        }
+
+        DvObject owner = aRole.getOwner();
+        if(owner == null) { // Builtin Role
+            owner = dataverseService.findByAlias("root");
+        }
+
+        if(owner != null) { // owner may be null if a role is created before the root collection as in setup-all.sh
+            IndexResponse indexDefinitionPointResult = indexDefinitionPoint(owner);
+            logger.info("Indexing result: " + indexDefinitionPointResult);
         }
+
+        return aRole;
     }
 
     public RoleAssignment save(RoleAssignment assignment) {

src/main/java/edu/harvard/iq/dataverse/api/Admin.java

@@ -1,11 +1,30 @@
 package edu.harvard.iq.dataverse.api;
 
-import edu.harvard.iq.dataverse.*;
+import edu.harvard.iq.dataverse.BannerMessage;
+import edu.harvard.iq.dataverse.BannerMessageServiceBean;
+import edu.harvard.iq.dataverse.BannerMessageText;
+import edu.harvard.iq.dataverse.DataFile;
+import edu.harvard.iq.dataverse.DataFileServiceBean;
+import edu.harvard.iq.dataverse.Dataset;
+import edu.harvard.iq.dataverse.DatasetServiceBean;
+import edu.harvard.iq.dataverse.DatasetVersion;
+import edu.harvard.iq.dataverse.DatasetVersionServiceBean;
+import edu.harvard.iq.dataverse.Dataverse;
+import edu.harvard.iq.dataverse.DataverseRequestServiceBean;
+import edu.harvard.iq.dataverse.DataverseServiceBean;
+import edu.harvard.iq.dataverse.DataverseSession;
+import edu.harvard.iq.dataverse.DvObject;
+import edu.harvard.iq.dataverse.DvObjectServiceBean;
+import edu.harvard.iq.dataverse.FileMetadata;
 import edu.harvard.iq.dataverse.api.auth.AuthRequired;
 import edu.harvard.iq.dataverse.settings.JvmSettings;
 import edu.harvard.iq.dataverse.util.StringUtil;
 import edu.harvard.iq.dataverse.util.json.NullSafeJsonBuilder;
 import edu.harvard.iq.dataverse.validation.EMailValidator;
+import edu.harvard.iq.dataverse.EjbDataverseEngine;
+import edu.harvard.iq.dataverse.Template;
+import edu.harvard.iq.dataverse.TemplateServiceBean;
+import edu.harvard.iq.dataverse.UserServiceBean;
 import edu.harvard.iq.dataverse.actionlogging.ActionLogRecord;
 import edu.harvard.iq.dataverse.api.dto.RoleDTO;
 import edu.harvard.iq.dataverse.authorization.AuthenticatedUserDisplayInfo;
@@ -49,7 +68,8 @@
 import java.io.InputStream;
 import java.io.StringReader;
 import java.nio.charset.StandardCharsets;
-import java.util.*;
+import java.util.Collections;
+import java.util.Map;
 import java.util.Map.Entry;
 import java.util.function.Predicate;
 import java.util.logging.Level;
@@ -65,6 +85,7 @@
 
 import org.apache.commons.io.IOUtils;
 
+import java.util.List;
 import edu.harvard.iq.dataverse.authorization.AuthTestDataServiceBean;
 import edu.harvard.iq.dataverse.authorization.AuthenticationProvidersRegistrationServiceBean;
 import edu.harvard.iq.dataverse.authorization.DataverseRole;
@@ -101,14 +122,17 @@
 import static edu.harvard.iq.dataverse.util.json.JsonPrinter.json;
 import static edu.harvard.iq.dataverse.util.json.JsonPrinter.rolesToJson;
 import static edu.harvard.iq.dataverse.util.json.JsonPrinter.toJsonArray;
-
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Date;
 import jakarta.inject.Inject;
 import jakarta.json.JsonArray;
 import jakarta.persistence.Query;
 import jakarta.ws.rs.QueryParam;
 import jakarta.ws.rs.WebApplicationException;
 import jakarta.ws.rs.core.StreamingOutput;
 import java.nio.file.Paths;
+import java.util.TreeMap;
 
 /**
  * Where the secure, setup API calls live.
@@ -990,6 +1014,22 @@ public Response createNewBuiltinRole(RoleDTO roleDto) {
             actionLogSvc.log(alr);
         }
     }
+    @Path("roles/{id}")
+    @PUT
+    public Response updateBuiltinRole(RoleDTO roleDto, @PathParam("id") long roleId) {
+        ActionLogRecord alr = new ActionLogRecord(ActionLogRecord.ActionType.Admin, "updateBuiltInRole")
+                .setInfo(roleDto.getAlias() + ":" + roleDto.getDescription());
+        try {
+            DataverseRole role = roleDto.updateRoleFromDTO(rolesSvc.find(roleId));
+            return ok(json(rolesSvc.save(role)));
+        } catch (Exception e) {
+            alr.setActionResult(ActionLogRecord.Result.InternalError);
+            alr.setInfo(alr.getInfo() + "// " + e.getMessage());
+            return error(Response.Status.INTERNAL_SERVER_ERROR, e.getMessage());
+        } finally {
+            actionLogSvc.log(alr);
+        }
+    }
 
     @Path("roles")
     @GET