Skip to content

Commit

Permalink
Merge pull request #331 from Bhashinee/main
Browse files Browse the repository at this point in the history
Integrate bal persist code generation to the `bal build` command
  • Loading branch information
Bhashinee authored Mar 5, 2024
2 parents 2d55dd9 + a723b9a commit ec3d528
Show file tree
Hide file tree
Showing 654 changed files with 3,227 additions and 1,535 deletions.
1 change: 1 addition & 0 deletions changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added
- [Added support for PostgreSQL as a datasource](https://github.com/ballerina-platform/ballerina-standard-library/issues/5829)
- [Integrate the persist code generation to the bal build command](https://github.com/ballerina-platform/ballerina-library/issues/5784)

## [1.2.1] - 2021-11-21

Expand Down
52 changes: 39 additions & 13 deletions docs/spec/spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,13 +29,14 @@ This specification elaborates on the `Persist CLI Tool` commands.
## 2. Initializing the Bal Project with Persistence Layer

```bash
bal persist init --datastore="datastore" --module="module_name"
bal persist add --datastore="datastore" --module="module_name"
```

| Command Parameter | Description | Mandatory | Default Value |
|:-----------------:|:----------------------------------------------------------------------------------------:|:---------:|:--------------:|
| --datastore | used to indicate the preferred database client. Currently, only 'mysql' is supported. | No | inmemory |
| --datastore | used to indicate the preferred database client. Currently, 'mysql', 'mssql', 'google sheets' and 'postgresql' are supported. | No | inmemory |
| --module | used to indicate the persist enabled module in which the files are generated. | No | <package_name> |
| --id | Used as an identifier | No | generate-db-client |


The command initializes the bal project with the persistence layer. This command includes the following steps,
Expand All @@ -47,32 +48,47 @@ The command initializes the bal project with the persistence layer. This command
3. Update Ballerina.toml with persist module configurations.
It will update the Ballerina.toml file with persist configurations.
```ballerina
[persist]
datastore = "<datastore>"
module = "<package_name>.<module_name>"
[[tool.persist]]
id = "generate-db-client"
targetModule = "<package_name>.<module_name>"
options.datastore = "<datastore>"
filePath = "persist/model.bal"
```

The directory structure will be,
```
medical-center
├── persist
└── medical-center.bal
└── model.bal
├── Ballerina.toml
└── main.bal
```

Behaviour of the `init` command,
- User should invoke the command within a Ballerina project
- User can use the optional arguments to indicate the preferred module name and data store, otherwise default values will be used.
- User cannot execute the command multiple times within the same project. User needs to remove the Ballerina.toml configurations, if the user wants to reinitialize the project.
Behaviour of the `add` command,
- Users should invoke the command within a Ballerina project.
- Users can use optional arguments to indicate the preferred module name and data store; otherwise, default values will be used.
- Users cannot execute the command multiple times within the same project. They need to remove the persist configurations from the Ballerina.toml if they want to reinitialize the project.

Apart from the `bal persist add` command, if you want to use the `bal persist generate` command you can initialize the project with the following `init` command,

```bash
bal persist init
```

This command includes the following steps,

1. Create persist directory:
Within this directory, a data model definition file should be created. This file will outline the necessary entities according to the [`persist` specification](https://github.com/ballerina-platform/module-ballerina-persist/blob/main/docs/spec/spec.md#2-data-model-definition)
2. Generate a model definition file within the persist directory:
This action will create a file named model.bal with the requisite imports (import ballerina/persist as _;) if no files currently exist in the persist directory.

## 3. Generating Persistence Derived Types, Clients, and Database Schema

```bash
bal persist generate
bal build
```

The command will generate [Derived Entity Types and Persist Clients](https://github.com/ballerina-platform/module-ballerina-persist/blob/main/docs/spec/spec.md#3-derived-entity-types-and-persist-clients)
The `bal build` command will generate [Derived Entity Types and Persist Clients](https://github.com/ballerina-platform/module-ballerina-persist/blob/main/docs/spec/spec.md#3-derived-entity-types-and-persist-clients)
as per the `persist` specification and the database schema associated with the data model definition.
Additionally, this command will create(update) `Config.toml` file with configurables used to initialize variables in `persist_db_config.bal`.
```ballerina
Expand Down Expand Up @@ -130,9 +146,19 @@ The database schema will contain the code to create,
1. Tables for each entity with defined primary keys
2. Create foreign key associations between tables if the model has defined associations between entities

Apart from the `bal build` users can also use the following command to generate the same as above mentioned,

```bash
bal persist generate --datastore mysql --module db
```

| Command Parameter | Description | Mandatory | Default Value |
|:-----------------:|:----------------------------------------------------------------------------------------:|:---------:|:--------------:|
| --datastore | used to indicate the preferred database client. Currently, 'mysql', 'mssql', 'google sheets' and 'postgresql' are supported. | Yes | |
| --module | used to indicate the persist enabled module in which the files are generated. | No | <package_name> |

Behaviour of the `generate` command,
- User should invoke the command within a Ballerina project
- The user should have initiated the persistence layer in the project and update the model definition file.
- The model definition file should contain the `persist` module import (`import ballerina/persist as _;`)
- The Model definition file should contain at least one entity
- If the user invokes the command twice, it will not fail. It will generate the files once again.
Expand Down
12 changes: 6 additions & 6 deletions examples/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ task initDbExamples {
delete "${project.projectDir}/build/generated-examples/${example}/generated"
workingDir "${project.projectDir}/build/generated-examples/${example}"
if (!Os.isFamily(Os.FAMILY_WINDOWS)) {
commandLine 'sh', '-c', "${ballerinaDist}/bin/bal persist init --module=entities --datastore=mysql"
commandLine 'sh', '-c', "${ballerinaDist}/bin/bal persist generate --module=entities --datastore=mysql"
} else {
commandLine 'cmd', 'slmgr /dlv'
println('\n' + "Windows Instance detected")
Expand All @@ -164,9 +164,9 @@ task initInMemoryExamples {
exec {
workingDir "${project.projectDir}/build/generated-examples/${example}"
if (!Os.isFamily(Os.FAMILY_WINDOWS)) {
commandLine 'sh', '-c', "${ballerinaDist}/bin/bal persist init --module=entities"
commandLine 'sh', '-c', "${ballerinaDist}/bin/bal persist generate --module=entities --datastore=inmemory"
} else {
commandLine 'cmd', '/c', "${ballerinaDist}/bin/bal.bat persist init --module=entities"
commandLine 'cmd', '/c', "${ballerinaDist}/bin/bal.bat persist generate --module=entities --datastore=inmemory"
}
}
} catch (Exception e) {
Expand Down Expand Up @@ -211,7 +211,7 @@ task generateDbExamples {
exec {
workingDir "${project.projectDir}/build/generated-examples/${example}"
if (!Os.isFamily(Os.FAMILY_WINDOWS)) {
commandLine 'sh', '-c', "${ballerinaDist}/bin/bal persist generate"
commandLine 'sh', '-c', "${ballerinaDist}/bin/bal persist generate --module=entities --datastore=inmemory"
} else {
commandLine 'cmd', 'slmgr /dlv'
println('\n' + "Windows Instance detected")
Expand All @@ -234,9 +234,9 @@ task generateInMemoryExamples {
exec {
workingDir "${project.projectDir}/build/generated-examples/${example}"
if (!Os.isFamily(Os.FAMILY_WINDOWS)) {
commandLine 'sh', '-c', "${ballerinaDist}/bin/bal persist generate"
commandLine 'sh', '-c', "${ballerinaDist}/bin/bal persist generate --datastore=inmemory --module=entities"
} else {
commandLine 'cmd', '/c', "${ballerinaDist}/bin/bal.bat persist generate"
commandLine 'cmd', '/c', "${ballerinaDist}/bin/bal.bat persist generate --datastore=inmemory --module=entities"
}
}
} catch (Exception e) {
Expand Down
6 changes: 3 additions & 3 deletions gradle.properties
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
org.gradle.caching=true
group=io.ballerina
version=1.2.2-SNAPSHOT
version=1.3.0-SNAPSHOT

#dependency versions
checkstylePluginVersion=10.12.1
Expand All @@ -16,7 +16,7 @@ mySqlDriverVersion=8.0.29
mssqlDriverVersion=11.2.3.jre17
postgresqlDriverVersion=42.6.0

ballerinaLangVersion=2201.9.0-20240208-103300-0823dc95
ballerinaLangVersion=2201.9.0-20240229-103900-a949e6d4

# Level 01
stdlibIoVersion=1.6.0
Expand Down Expand Up @@ -48,7 +48,7 @@ stdlibHttpVersion=2.10.4
stdlibSqlVersion=1.11.1

# Level 09
stdlibPersistVersion=1.2.1-20231130-100000-cca5690
stdlibPersistVersion=1.3.0-20240301-141700-02872be

# Level 10
stdlibPersistSqlVersion=1.2.2-20231206-130200-881839f
Expand Down
8 changes: 8 additions & 0 deletions persist-cli-tests/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,7 @@ task copyStdlibs() {
doLast {
configurations.ballerinaStdLibs.resolvedConfiguration.resolvedArtifacts.each { artifact ->
def artifactExtractedPath = "${project.buildDir}/extracted-stdlibs/" + artifact.name + "-zip"
def persistCliJar = "${project.rootDir}/persist-cli/build/libs/persist-cli-${project.version}.jar"
copy {
into ballerinaDist
into("repo/bala") {
Expand All @@ -142,6 +143,13 @@ task copyStdlibs() {
from "${artifactExtractedPath}/cache"
}
}
copy {
def testDistribution = "${project.rootDir}/persist-cli-tests/build/ballerina-distribution"
into testDistribution
into("bre/lib") {
from "${persistCliJar}"
}
}
}
}
}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
/*
* Copyright (c) 2024, WSO2 LLC. (http://www.wso2.org) All Rights Reserved.
*
* WSO2 LLC. licenses this file to you under the Apache License,
* Version 2.0 (the "License"); you may not use this file except
* in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

package io.ballerina.persist.tools;

import org.testng.Assert;
import org.testng.annotations.Test;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.nio.charset.StandardCharsets;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.LinkedList;
import java.util.List;
import java.util.stream.Collectors;
import java.util.stream.Stream;

/**
* persist tool in `bal build` command tests.
*/
public class BuildCodeGeneratorTest {
public static final Path TARGET_DIR = Paths.get(System.getProperty("user.dir"), "build");
public static final Path TEST_DISTRIBUTION_PATH = TARGET_DIR.resolve("ballerina-distribution");

@Test(enabled = true)
public void testBuildWithMysql() throws IOException, InterruptedException {
String log = "Persist client and entity types generated successfully in " +
"the persist_build_1 directory.";
Path project = TARGET_DIR.resolve("generated-sources/tool_test_build_1");
assertLogs(log, project);
}

@Test(enabled = true)
public void testBuildWithInvalidTargetModule() throws IOException, InterruptedException {
String log = "ERROR: invalid module name : 'persist_add_1' :" + System.lineSeparator() +
"module name should follow the template <package_name>.<module_name>";
Path project = TARGET_DIR.resolve("generated-sources/tool_test_build_2");
assertLogs(log, project);
}

@Test(enabled = true)
public void testBuildWithInvalidCharachtersInTargetModule() throws IOException, InterruptedException {
String log = "ERROR: invalid module name : '*****' :" + System.lineSeparator() +
"module name can only contain alphanumerics, underscores and periods";
Path project = TARGET_DIR.resolve("generated-sources/tool_test_build_3");
assertLogs(log, project);
}

@Test(enabled = true)
public void testBuildWithInvalidLengthOfTargetModule() throws IOException, InterruptedException {
String log = "ERROR: invalid module name : 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" +
"aaaaaaaaaaaaaaaaaaaaaaaaddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd" +
"ddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd" +
"dddddddddddddddddddddd' :" + System.lineSeparator() +
"maximum length of module name is 256 characters";
Path project = TARGET_DIR.resolve("generated-sources/tool_test_build_4");
assertLogs(log, project);
}

@Test(enabled = true)
public void testBuildWithInvalidDataSource() throws IOException, InterruptedException {
String log = "ERROR: the persist layer supports one of data stores:";
Path project = TARGET_DIR.resolve("generated-sources/tool_test_build_5");
assertContainLogs(log, project);
}

@Test(enabled = true)
public void testBuildWithoutEntities() throws IOException, InterruptedException {
String log = "ERROR: the model definition file(model.bal) does not contain any entity definition.";
Path project = TARGET_DIR.resolve("generated-sources/tool_test_build_6");
assertLogs(log, project);
}

private String collectLogOutput(Path project) throws IOException, InterruptedException {
List<String> buildArgs = new LinkedList<>();
Process process = executeBuild(TEST_DISTRIBUTION_PATH.toString(), project, buildArgs);
try (BufferedReader br = new BufferedReader(new InputStreamReader(process.getErrorStream(),
StandardCharsets.UTF_8))) {
Stream<String> logLines = br.lines();
String generatedLog = logLines.collect(Collectors.joining(System.lineSeparator()));
logLines.close();
return generatedLog;
}
}

private void assertLogs(String log, Path project) throws IOException, InterruptedException {
String generatedLog = collectLogOutput(project);
Assert.assertEquals(generatedLog, log);
}

private void assertContainLogs(String log, Path project) throws IOException, InterruptedException {
String generatedLog = collectLogOutput(project);
Assert.assertTrue(generatedLog.contains(log));
}

public static Process executeBuild(String distributionName, Path sourceDirectory,
List<String> args) throws IOException, InterruptedException {
args.add(0, "build");
Process process = getProcessBuilderResults(distributionName, sourceDirectory, args);
process.waitFor();
return process;
}

public static Process getProcessBuilderResults(String distributionName, Path sourceDirectory, List<String> args)
throws IOException {
String balFile = "bal";

if (System.getProperty("os.name").startsWith("Windows")) {
balFile = "bal.bat";
}
args.add(0, TEST_DISTRIBUTION_PATH.resolve(distributionName).resolve("bin").resolve(balFile).toString());
ProcessBuilder pb = new ProcessBuilder(args);
pb.directory(sourceDirectory.toFile());
return pb.start();
}
}
Loading

0 comments on commit ec3d528

Please sign in to comment.