This repository demonstrates how the Relativity Server Import SDK can be used to import the following:
- Native documents
- Images
- Objects
- Productions
All of the samples are based on NUnit, the AAA test structure pattern is followed, and app.config settings define test parameters including Relativity Urls and login credentials. Once these settings have been made, the tests are 100% responsible for all required setup and tear down procedures.
Note: the test project relies on Relativity NuGet packages to ensure Import API and all dependencies are deployed to the proper target directory.
This page contains the following information:
- Visual Studio 2022 and above
- A Server 2023 or above developer Dev VM
- Visual C++ 2015 x64 Runtime (Outside In and FreeImage)
Note: Visual Studio 2022 and above is required due to NuGet 6 and C# language feature usage. Relativity strongly recommends using a developer Dev VM for the test environment.
The steps below should only be required when the repository is being cloned for the first time.
View instructions
With each new Relativity Server annual release, a new branch is created to not only test and verify the Import SDK package but to ensure all package references are updated properly. As of this writing, the following three releases are available for consideration:
Use the command-line or Visual Studio to clone the repository and target a branch from the previous step.
git clone -b server-release-2023 https://github.com/relativitydev/server-import-sdk-samples.gitLaunch Visual Studio 2022 and open the Relativity.DataExchange.Samples.NUnit.sln solution file.
Open a PowerShell terminal and run the following script to autogenerate the .\src\FunctionalTest.runsettings file.
.\New-TestSettings.ps1 -TestVMName <SPECIFY-DEVVM-NAME>Once the file is created, click Test → Configure Run Settings → Select Solution Wide runsettings File, and select the .\src\FunctionalTest.runsettings file.
Use Visual Studio to build the solution.
At this point, the setup is complete and should now be able to run all of the tests. If the test explorer isn't visible, go to Test->Windows->Test Explorer and click the Run All hyper-link at the top. Regardless of which method is used to execute tests, the following actions occur:
- A new test workspace is created (normally 15-30 seconds) to ensure tests are isolated
- The tests are executed
- The test workspace is deleted
- The Test Explorer displays the results
If everything is working properly, the Test Explorer should look something like this:
Note: If the Test Explorer lists the tests but none of the tests run, ensure the value is set to X64. Although this setting is cached, Visual Studio is known to reset the value to X86.
The ImportAPI class is a top-level class that includes functionality for importing documents, images, production sets, and Relativity Dynamic Objects (RDOs). It includes methods for performing these import jobs, as well as other methods for retrieving workspace, field, and other objects.
The Import SDK design uses the Relativity Kepler REST Services and proper URLs must be supplied when constructing the object.
- https://hostname.mycompany.corp (Relativity instance)
- https://hostname.mycompany.corp/relativitywebapi (Relativity web services)
When constructing the ImportAPI object, the API caller must first decide which authentication model is used in order to determine how the object is constructed.
Note: Authentication is immediately performed within the ImportAPI constructor and will throw an exception if a failure occurs. API callers should wrap this call with a try/catch block.
The user must be a member of the System Administrators group in Relativity. These permissions are similar to those required to import a load file through the Relativity Desktop Client.
ImportAPI importApi = new ImportAPI(relativityUserName, relativityPassword, relativityWebServiceUrl);Uses the current claims principal token to authenticate and should only be used by Relativity Service Account hosted processes.
ImportAPI importApi = ImportAPI.CreateByRsaBearerToken(relativityWebServiceUrl);Note: This is the preferred method for using Import API within an agent or custom page.
The user is validated against the Relativity instance located at WebServiceURL.
ImportAPI importApi = new ImportAPI(relativityWebServiceUrl);Given the ImportAPI object, the API caller calls a method to create the appropriate job object. This is later used to configure and execute the import job.
const int SomeObjectArtifactTypeId = 1111111;
const int ProductionSetArtifactId = 2222222;
kCura.Relativity.DataReaderClient.ImportBulkArtifactJob job = importApi.NewNativeDocumentImportJob();
kCura.Relativity.DataReaderClient.ImportBulkArtifactJob job = importApi.NewObjectImportJob(SomeObjectArtifactTypeId);
kCura.Relativity.DataReaderClient.ImageImportBulkArtifactJob job = importApi.NewImageImportJob();
kCura.Relativity.DataReaderClient.ImageImportBulkArtifactJob job = importApi.NewProductionImportJob(ProductionSetArtifactId);When configuring either native document or object import jobs, the Settings property exposes a number of options to control import behavior.
View settings
| Setting | Description |
|---|---|
| ArtifactTypeId | The target object artifact type identifier. |
| Billable | Indicates whether imported files are billable. |
| BulkLoadFileFieldDelimiter | The field delimiter used when writing out the bulk load file. |
| CaseArtifactId | The target workspace artifact identifier. |
| CopyFilesToDocumentRepository | Indicates whether to enable or disable copying files to the document repository.
|
| DisableControlNumberCompatibilityMode | Indicates whether to enable or disable the use of Control Number to override SelectedIdentifierField.
|
| DisableExtractedTextEncodingCheck | Indicates whether to enable or disable encoding checks for each file.
|
| DisableExtractedTextFileLocationValidation | Indicates whether to enable or disable validation of the extracted text file location.
|
| DisableNativeLocationValidation | Indicates whether to enable or disable validation of the native file path.
|
| DisableNativeValidation | Indicates whether to enable or disable validation of the native file type for the current job.
|
| DisableUserSecurityCheck | Indicates whether to enable or disable user permission checks per document or object.
|
| ExtractedTextEncoding | The extracted text file encoding. |
| ExtractedTextFieldContainsFilePath | Indicates whether the extracted text field contains a path to the extracted text file or contains the actual extracted text.
|
| FileSizeColumn | The column that contains the FileSize on the SourceData property. |
| FileSizeMapped | Indicates whether to enable or disable skipping file size checks.
|
| FolderPathSourceFieldName | The metadata field used to build the folder structure. Note: All folders are built under the Import Destination folder, indicated by the DestinationFolderArtifactID value. If a folder matching the entered string already exists, the documents will be added to it; otherwise, the folder(s) (including nested folders) will be created and the documents will be imported into the new folder(s). |
| IdentityFieldId | The key field that's set only on Overwrite mode. |
| LoadImportedFullTextFromServer | Indicates whether to enable or disable loading extracted text directly from its file path.
|
| MaximumErrorCount | The maximum number of errors displayed. This property is optional. |
| MoveDocumentsInAppendOverlayMode | Indicates whether to enable or disable moving documents to a new folder for Append/Overlay mode.
|
| NativeFileCopyMode | The native file copy behavior.
|
| NativeFilePathSourceFieldName | The name of the field that contains the full path and filename for the native files. |
| OIFileIdColumnName | The name of the field that contains the OutsideInFileId on the data source object.Note: If OIFileIdMapped or FileSizeMapped is True, set this property to the value that indicates the field that. contains the OutsideInFileId value. |
| OIFileIdMapped | Indicates whether to enable or disable file identification.
|
| OIFileTypeColumnName | The name of the field that contains the Outside In file type on the data source. Note: If OIFileIdMapped is True, this field must be set. |
| OverwriteMode | The import overwrite behavior.
|
| SelectedIdentifierFieldName | The name of the field used as an identifier. Note: If this identifier cannot be resolved, the control number will be used in its place. |
| StartRecordNumber | The record number from which to start the import. |
When configuring either image or production import jobs, the Settings property exposes a number of options to control import behavior.
View settings
| Setting | Description |
|---|---|
| ArtifactTypeId | The target object artifact type identifier. |
| AutoNumberImages | Indicates whether to enable or disable appending a page number to a page-level identifier.
|
| BatesNumberField | The name of the field that defines the unique identifier. Note: This unique identifier may be called Bates Number or Control Number in a database. |
| Billable | Indicates whether imported files are billable. |
| CaseArtifactId | The target workspace artifact identifier. |
| CopyFilesToDocumentRepository | Indicates whether to enable or disable copying files to the document repository.
|
| DisableExtractedTextEncodingCheck | Indicates whether to enable or disable encoding checks for each file.
|
| DisableImageLocationValidation | Indicates whether to enable or disable image location validation.
|
| DisableImageTypeValidation | Indicates whether to enable or disable image type validation.
|
| DisableUserSecurityCheck | Indicates whether to enable or disable user permission checks per document or object.
|
| DocumentIdentifierField | The name of the field that corresponds to the DocumentIdentifier field. |
| ExtractedTextEncoding | The extracted text file encoding. |
| ExtractedTextFieldContainsFilePath | Indicates whether the extracted text field contains a path to the extracted text file or contains the actual extracted text.
|
| FileLocationField | The name of the field that corresponds with the FileLocation field. |
| FolderPathSourceFieldName | The metadata field used to build the folder structure. Note: All folders are built under the Import Destination folder, indicated by the DestinationFolderArtifactID value. If a folder matching the entered string already exists, the documents will be added to it; otherwise, the folder(s) (including nested folders) will be created and the documents will be imported into the new folder(s). |
| IdentityFieldId | The key field that's set only on Overwrite mode. |
| ImageFilePathSourceFieldName | The name of the field that contains the full path to the image file. |
| LoadImportedFullTextFromServer | Indicates whether to enable or disable loading extracted text directly from its file path.
|
| MaximumErrorCount | The maximum number of errors displayed. This property is optional. |
| MoveDocumentsInAppendOverlayMode | Indicates whether to enable or disable moving documents to a new folder for Append/Overlay mode.
|
| NativeFileCopyMode | The native file copy behavior.
|
| OverlayBehavior | The method for overlay imports with multiple choice and multi-object fields.
|
| OverwriteMode | The import overwrite behavior.
|
| SelectedIdentifierFieldName | The name of the field used as an identifier. Note: If this identifier cannot be resolved, the control number will be used in its place. |
| StartRecordNumber | The record number from which to start the import. |
Regardless of which job is created, the API caller uses standard ADO.NET constructs like System.Data.DataTable to define or System.Data.IDataReader to read data sources.
// Once the data source is defined, the DataColumn names are mapped via the job Settings object.
System.Data.DataTable dataSource = new System.Data.DataTable();
dataSource.Columns.AddRange(new[]
{
new DataColumn("control number", typeof(string)),
new DataColumn("file path", typeof(string))
});
dataSource.Rows.Add("REL-4444444", @"C:\temp\sample.pdf");
job.Settings.SelectedIdentifierFieldName = "control number";
job.SourceData.SourceData = dataSource.CreateDataReader();The ImportBulkArtifactJob and ImageImportBulkArtifactJob objects expose a number of useful events to obtain document-level errors, completion, fatal exception, and progress details.
This object is exposed by several events and provides useful job summary details.
| Property | Description |
|---|---|
| EndTime | The import end time. |
| ErrorRowCount | The total number of non-fatal document-level errors that occurred. |
| ErrorRows | The collection of non-fatal document-level error objects that occurred. |
| FatalException | The exception that resulted in a fatal job error. |
| FieldMap | The collection of field map entries that map source fields to destination fields in the workspace. |
| FileBytes | The total number of transferred native file bytes. |
| MetadataBytes | The total number of transferred metadata bytes. |
| StartTime | The import start time. |
| TotalRows | The total number of processed rows. This value doesn't indicate the number of successful rows. |
This event is raised when an import job is finished. A JobReport object is passed with detailed information about the job. A completed job may have errors if the data wasn't imported properly.
// This event provides the JobReport object.
job.OnComplete += report =>
{
Console.WriteLine("The job has completed.");
};This event is raised when an error occurs while importing a row of data.
// This event provides an IDictionary object with well-known parameters.
job.OnError += row =>
{
Console.WriteLine(row["Line Number"]);
Console.WriteLine(row["Identifier"]);
Console.WriteLine(row["Message"]);
};Note: The MaximumErrorCount is a configurable setting available on all import jobs that determines the number of errors to return.
This event is raised when an import job encounters a fatal exception caused by invalid import settings or other issues. The fatal exception can be retrieved by the passed JobReport object.
// This event provides the JobReport object.
job.OnFatalException += report =>
{
Console.WriteLine("The job experienced a fatal exception: " + report.FatalException);
};This event is raised throughout the import job life cycle and is similar to the messages displayed in the Relativity Desktop Client.
// This event provides the Status object.
job.OnMessage += status =>
{
Console.WriteLine("Job message: " + status.Message);
};This event is raised at the same rate as the OnMessage event and provides detailed progress information about the import job.
// This event provides the FullStatus object.
job.OnProcessProgress += status =>
{
Console.WriteLine("Job start time: " + status.StartTime);
Console.WriteLine("Job end time: " + status.EndTime);
Console.WriteLine("Job process ID: " + status.ProcessID);
Console.WriteLine("Job total records: " + status.TotalRecords);
Console.WriteLine("Job total records processed: " + status.TotalRecordsProcessed);
Console.WriteLine("Job total records processed with warnings: " + status.TotalRecordsProcessedWithWarnings);
Console.WriteLine("Job total records processed with errors: " + status.TotalRecordsProcessedWithErrors);
Console.WriteLine("Job total records: " + status.TotalRecordsDisplay);
Console.WriteLine("Job total records processed: " + status.TotalRecordsProcessedDisplay);
Console.WriteLine("Job status suffix: " + status.StatusSuffixEntries);
};This event is raised for each row found in the data set.
// This event provides the row number.
job.OnProgress += row =>
{
Console.WriteLine("Job progress line number: " + row);
};Once the job has been fully configured, the API caller invokes the Execute method. Soon thereafter, events are raised and the method returns once the job either completes or a fatal exception occurs.
// Wait for the job to complete.
job.Execute();Note: Although fatal exceptions are normally caught and raised by the OnFatalException event, API callers should wrap this method call with a try/catch block.
The section below outlines each of the import job test samples.
- The DocImportTests imports sample documents.
- The DocImportFolderTests imports sample documents and also specifies folder paths.
- The DocNegativeImportTests imports documents that expect job-level import failures.
- The ObjectSimpleImportTests imports a custom object with a single-object field.
- The ObjectAdvancedImportTests imports a custom object with a multi-object field.
- The ObjectNegativeImportTests imports single-object and multi-object fields that expect document-level and job-level import failures.
Note: The tests create custom RDO types during the test setup.
- The ImageImportTests imports sample images.
- The ProductionImportTests imports sample documents, creates a production, and then validates the bates numbers.
Note: This test relies upon the Productions NuGet package to perform all required functionality.