Draft for core:build.

[DragosPopse]

This is an idea of how core:build API could look and function, for @gingerBill to review. This is still in drafting/experimenting stage, so there are some compiler flags that are not yet implemented (-show-timings for example)

A (somewhat) minimal example was written here.

Example

package src_build

import "core:build"
import "core:os"
import "core:strings"

Mode :: enum {
    Debug,
    Release,
}

// Each target in our project will generate a Config, which contains the flags for the compiler and post/pre-build commands.
Target :: struct {
    using target: build.Target,
    mode: Mode,
}

project: build.Project
target_debug: Target = { // Debug target
    target = {
        name = "deb",
        platform = {ODIN_OS, ODIN_ARCH},
    },
    mode = .Debug,
}
target_release: Target = { // Release target
    target = {
        name = "rel",
        platform = {ODIN_OS, ODIN_ARCH},
    },
    mode = .Release,
}

// This is the bulk of the build system. Every build system requires a proc with this signature to generate Config objects based on Targets
config_target :: proc(project: ^build.Project, target: ^build.Target) -> (config: build.Config) {
    target := cast(^Target)target
    config.name = target.name // config names need to be unique. This is the name that will be used when calling `build <config-name>
    config.platform = target.platform
    config.out_file = "demo.exe" if target.platform.os == .Windows else "demo.out"
    config.out_dir = strings.concatenate({"out/", target.name})
    config.build_mode = .EXE
    config.src_path = "src" // the path to the package we want to build

    config.defines["DEFINED_INT"] = 99
    config.defines["DEFINED_STRING"] = "Hellope #config"
    config.defines["DEFINED_BOOL"] = true
    
    switch target.mode {
    case .Debug:
        config.flags += {.Debug}
        config.opt = .None

    case .Release:
        config.flags += {.Disable_Assert}
        config.opt = .Speed
    }

    

    return config
}

// Doing this setup in an @init proc will be important for adding multiple build systems into eachother. This feature is still a WIP
@init
_ :: proc() {
    project.name = "Build System Demo"
    build.add_target(&project, &target_debug)
    build.add_target(&project, &target_release)
    build.add_project(&project)
    project.configure_target_proc = config_target
}

main :: proc() {
    opts := build.parse_args(os.args)
    opts.default_config_name = "deb" // this will be used when no config name is specified, effectively making `odin run build` possible without saying `odin run build -- deb`
    opts.display_external_configs = true // Display the configs of other imported build systems. This is a WIP and currently this statement doesn't do anything
    build.run(&project, opts)
}

Running the example

1. Open a command line in the examples/package_build directory
2. Run odin run build (this is the simplest usage of the build system)
3. The executables will be available in subfolders of out, as configured by the build script.
4. The build script using core:build is available in the build folder, while the actual app is in src

Syntax

Running odin run build -- -help will give you an overview of all the available flags and configurations that you are able to call via the build system.

C:\dev\Odin\examples\package_build>odin build build

C:\dev\Odin\examples\package_build>build -help
Build System Demo build system
        Syntax: build <flags> <configuration name>
        Available Configurations:
                deb
                rel

        Flags
                -help <optional config name>
                        Displays build system help. Cannot be used with other flags.
                        [WIP] Specifying a config name will give you information about the config.

                -ols
                        Generates an ols.json for the configuration.

                -vscode
                        [WIP] Generates .vscode/launch.json configuration for debugging. Must be used for other VSCode flags to function.

                -build-pre-launch
                        [WIP] VSCode: Generates a pre launch command to build the project before debugging.
                        Effectively runs `build <config name>` before launching the debugger.

                -include-build-system:"<args>"
                        [WIP] VSCode: Includes the build system as a debugging target.

                -cwd-workspace
                        [WIP] VSCode: Use the workspace directory as the CWD when debugging.

                -cwd-out
                        [WIP] VSCode: Use the output directory as the CWD when debugging.

                -cwd:"<directory>"
                        [WIP] VSCode: Use the specified directory as the CWD when debugging.

                -launch-args:"<args>"
                        [WIP] VScode: Specify the args sent to the executable when debugging.

                -use-cppvsdbg
                        [WIP] VSCode: Use the VSCode debugger. Used by default with -vscode.

                -use-cppdbg
                        [WIP] VSCode: Use the GDB/LLDB debugger.

Features

• Multiple configurations, 1 executable: configure different compiler parameters, defines, post/pre-build commands for different configurations and call them via odin run build -- <config_name>
• ols.json generation via odin run build -- <config_name> -ols. The ols.json usually contains absolute paths, and I believe it's a good feature to be able to generate it and not have it as part of third party repositories as it is usually the case.
• vscode integration for debugging via odin run build -- <config_name> -vscode (With some [WIP] flags that will be available in the future). Call the build system with the -vscode flag, open vscode and hit F5. Different debuggers and editors will be added as requested.
• Planned: include 3rd party build systems into your own via import and use their configurations and setup dependencies.

Known Issues

Since this is a draft and I don't know if it's gonna be accepted, the code has some memory leaks and the error handling is not ideal (relying on eprintf and not good error handling flow). The memory leaks are not a big issue in 99% of use cases, because core:build is meant to be used as a command line one-time utility.

The dependency handling is still a big feature i want to play with, but i'm still experimenting. This feature is the main reason why the example API has an add_project and the setup is done in @init, and i hope it will make more sense once that feature is in.

[DragosPopse]

A small extra idea. Having a build system written into Odin allows you to use things like ODIN_ROOT, which is very useful for copying DLLs without relying on system variables and whatnot.

[flysand7]

In addition to the comments in review I'd like to ask you to maybe document this package in odin doc style comments. It would be helpful to start documenting packages beyond mere function prototypes..

[flysand7]

syscall associates with x86 instruction to call into kernel functions... maybe call this shell_exec?

[DragosPopse]

Will change the name, thanks. I was planning to document the package of course, but I wanted to get it going first and get some feedback on the general API

[flysand7]

You seem to be using tabs throughout your project, this line contains two spaces, which looks misaligned to me. Don't thank me :)

Actually looking closely some parts of your code use tabs while other use spaces... you gotta figure it out

[flysand7]

Hint: Try selecting line 17 and line 19 in github, the selection increments should help you tell spaces and tabs apart, in case you set tab width to 4

[DragosPopse]

Could u have a quick look to see if the tab problem was fixed?

[flysand7]

Looks correct. By the way, which text editor are you using? If it's supported maybe there are options to display invisible characters. I know sublime displays invisible characters when you select text, that sort of functionality may be useful if you're working with projects that have different tab/space rules and you're not sure if your editor messes it up.

[DragosPopse]

im using vscode. I haven't been paying much attention to tabs/spaces but it seemed that i was able to just convert the entire file to tabs

[flysand7]

As far as I'm aware this is a required option. Maybe print error when this isn't specified?

[DragosPopse]

Error handling is still dumb. Will start making the code more friendly in that regard

[gingerBill]

General comments:

• I wouldn't rely too much on libc for a build system, if possible. I understand it is mostly for libc.system, but we will need to replace that with something native.
• The parse_args needs to be a little more generic so that it can handle errors and such. We may need a flag parsing library as part of core at any rate, but I cannot decide on the style because there is too much choice and compromises between each style.
• I think flags are going should be extremely generic and user specified, especially since this is a user-created build system tool, not one for general Odin code. The user may only want to specify custom flags such as "build", "install", "tooling", and then they handle what that does.
• There appears to be a random mixture of tabs and spaces for indentation throughout the codebase. Please use tabs for indentation, and spaces for alignment.

[DragosPopse]

Thanks for the reviews. I'll get to fixing some of the issues mentioned.
In response to @gingerBill

• libc.system is the only thing used and i agree that it should be purged from there. I wasn't sure if there is any implementation for that in core:os or if i should provide some native implementations
• parse_args is still a bit temporary. I'm not entirely sure how to approach this. But I'll do better error handling for creating the options.
• Relating to flags. The current flags specified by the build system are for setting up development environment. Things like -help -ols. The way the user "handles" custom things is via the config name. For example you could create tooling.app1 tooling.app2 configs, and build them all via odin run build -- tooling.* (wildcards work). We could experiment with custom command line flags too, but it needs a system in place to make them available in build -help, just like the build system provided flags
• It's about time I setup tabs/spaces in my editor. My bad.

[flysand7]

libc.system is the only thing used and i agree that it should be purged from there. I wasn't sure if there is any implementation for that in core:os or if i should provide some native implementations

From what I've seen I don't see that there's a cross-platform implementation. I suggest you hold that improvement in your head until the other packages catch up. Since there is work being done in those areas, maybe it's best not to spend time reimplementing things already being worked on.

Relating to configs, maybe it's best to avoid directly feeding the build with the parsed configs but rather provide user an API to use the configs? I'm not sure whether it's a good idea to provide the interface to build system on behalf of the users since if they're using a build system they want something custom anyway. Can you explain the meaning behind these configs and why they're being added?

[DragosPopse]

@flysand7 the config is what's used by build.build_package(config) to build your project. You could skip the arg parsing stuff and do everything by hand if you want something more "custom", but i think most use cases are handled, and I plan to handle more as they appear.

To explain configs more, a Config is just a big struct that contains everything required to call odin build (flags, defines, collections) + post/pre-build commands + (planned) dependencies. You create those configs based on Targets via configure_target user provided procedure. The idea of Targets is that you usually want the same set of common configuration, with small changes depending on what platform you are on, what type of build you are making (you would enable -debug for debug builds, or -o:speed for release builds). That's the problem that a Target solves. The configure_target_proc is there to aid generating Config objects based on the Target. It's similar to how Sharpmake does things, but it's simplified. The targets are related to a Project for the experimental feature of adding external build systems, so that might or might not be part of the final API, but it's just a simple call anyway.

Let me know if this clarifies things.

[flysand7]

Yes I understand now. I might have misunderstood what you meant by the config

[DragosPopse]

I'll get back to working on this today. Are there any significant issues that would need to be resolved for this PR? My plan right now is to add the dependency feature and handle errors better.

@gingerBill what do you propose for libc currently? Implement a native version for windows/linux and wait for someone to do the darwin OR wait for core:os2 and modify it in a later PR?

[DragosPopse]

I have made some modifications in response to the reviews. I added better error handling with return vals for most procs (some might still be missing, I'll keep working on it). I'm experimenting with using the context.logger for displaying warnings and errors, although this would require the user to call context = build.default_context() in main and their @init proc, so I'm not yet sure if that's the best approach, especially for a core lib.

I also added the option for custom flags as Bill suggested. build.settings_init_from_args(s: ^Settings, args: []string, allow_custom_flags := false). When allow_custom_flags == true all flags that are "unknown" will be appended to s.custom_args. I also made Configure_Target_Proc to accept Settings as the last parameter so that you can use that.

-install is also a handled flag now. All it does is set settings.command_type to .Install rather than .Build. The user can add different post-build commands in the configuration if .Install is present.

The API is mostly stable now. I'm still unsure about passing the settings to the configure_target. I'll have to check how that behaves when the dependencies feature is implemented.

I'd also want to know if the error handling+ error message handling is ok now or I should try something different? I'm really unsure about requiring the user the make the context in order for the logs to show up. It might be the best to just return an ok and do the error messaging with os.stderr

[flysand7]

I like the switch to log. Allows the developer to configure the logging level and potentially handle the errors themselves, maybe avoid printing extra lines and focus on success state.

[flysand7]

Called archive (as in, object archive), it just contains a collection of obj files. Though I doubt people would recognize it by that name. I don't think it's worth it: Odin isn't supposed to be built from multiple modules and then archived/linked together.

If you want to ship a static library from Odin, compile .obj and if it has external C dependencies, ship with those .lib dependencies (foreign imports should take care of linking). Also I don't think if you create an archive, that, foreign import would be able to find them.

I'm not at home so I can't check whether Odin supports creating archives. If not, then maybe it's not even worth it.

[DragosPopse]

All I did was take the build modes from here (not the duplicate ones). It doesn't seem that it supports static libs.

[flysand7]

I'm not sure what's the roundabout with creating a global context with a logger, to then have the user retrieve that context. Can't default_context create the context and return newly-created instead of returning a global context? I'm not seeing the purpose here...

[DragosPopse]

i can move the global into a @static in default_context, otherwise create_console_logger would keep creating a new one on each call.

[DragosPopse]

this don't work

[DragosPopse]

I have finalized the API. Things might still need extra testing, but so far it seems to work.

example build script:

package src_build

import "core:build"
import "core:os"
import "core:strings"
import "core:fmt"

Mode :: enum {
	Debug,
	Release,
}

Target :: struct {
	using target: build.Target,
	mode: Mode,
}

project: build.Project

target_debug: Target = {
	target = {
		name = "dbg",
		platform = {ODIN_OS, ODIN_ARCH},
	},
	mode = .Debug,
}
target_release: Target = {
	target = {
		name = "rel",
		platform = {ODIN_OS, ODIN_ARCH},
	},
	mode = .Release,
}

run_target :: proc(target: ^build.Target, mode: build.Run_Mode, args: []build.Arg, loc := #caller_location) -> bool {
	target := cast(^Target)target
	config: build.Odin_Config
	config.platform = target.platform
	config.out_file = "demo.exe" if target.platform.os == .Windows else "demo"
	
	// paths must be set with build.relpath / build.abspath in order for them to be relative to the build system root directory (build/../). build.t*path alternatives use context.temp_allocator
	config.src_path = build.trelpath(target, "src")
	config.out_dir = build.trelpath(target, fmt.tprintf("out/%s", target.name))

	switch target.mode {
	case .Debug:
		config.opt = .None
		config.flags += {.Debug}
	case .Release:
		config.opt = .Speed
	}

	switch mode {
	case .Build:
		// Pre-build stuff here
		build.odin(target, .Build, config) or_return 
		// Post-build stuff here
	case .Dev: 
		return build.generate_odin_devenv(target, config, args)
	
	case .Help:
		return false // mode not implemented
	}
	return true
}

@init
_ :: proc() {
	project.name = "Build System Demo"
	build.add_target(&project, &target_debug, run_target)
	build.add_target(&project, &target_release, run_target)
}

main :: proc() {
	info: build.Cli_Info
	info.project = &project
	info.default_target = &target_debug
	build.run_cli(info, os.args)
}

output of odin run build:

odin build "src"
        -out:"out\dbg/demo.exe"
        -subsystem:console
        -build-mode:exe
        -o:none
        -reloc-mode:default
        -debug
        -target:windows_amd64

output of odin run build -- -help:

Syntax: build.exe <flags> <target>
Available Targets:
        dbg
        rel

Builtin Flags - Only 1 [Type] group per call. Groups are incompatible
        -help
                [Help] Displays information about the build system or the target specified.

        -ols
                [Dev] Generates an ols.json for the configuration.

        -vscode
                [Dev] Generates .vscode folder for debugging.

        -build-pre-launch
                [Dev] Runs the build system before debugging. (WIP)

        -include-build-system:"<args>"
                [Dev] Include the build system as a debugging target. (WIP)

        -cwd:"<dir>"
                [Dev] Sets the CWD to the specified directory.

        -cwd-workspace
                [Dev] Sets the CWD to the root of the build system executable.

        -cwd-out
                [Dev] Sets the CWD to the output directory specified in the -out odin flag

        -launch-args:"<args>"
                [Dev] The arguments to be sent to the output executable when debugging.

        -dbg:"<debugger name>"
                [Dev] Debugger type used. Works with -vscode. Sets the ./vscode/launch.json "type" argument

output of odin run build -- -ols -vscode
ols.json:

{
	"collections": [

	],
	"enable_document_symbols": true,
	"enable_semantic_tokens": true,
	"enable_hover": true,
	"enable_snippets": true,
	"checker_args": "-out:\"out/dbg/demo.exe\" -subsystem:console -build-mode:exe -o:none -reloc-mode:default -debug -target:windows_amd64"
}

.vscode/launch.json:

{
	"version": "0.2.0",
	"configurations": [
		{
			"type": "cppvsdbg",
			"request": "launch",
			"preLaunchTask": "",
			"name": "dbg",
			"program": "C:\\dev\\Odin\\examples\\package_build\\out\\dbg\\demo.exe",
			"args": [
				""
			],
			"cwd": "${workspaceFolder}\\out\\dbg"
		}
	]
}

.vscode/tasks.json:

{
	"version": "2.0.0",
	"tasks": [

	]
}

Concept explanation

The idea of core:build would be to provide an api for running complex odin builds + a CLI to help you decide what you want to run. A Target is what it's being run by the cli or via build.run_target. This can be anything from debug/release targets to install or clean. These are setup by the user. In order to run a target, you need to create it, add it with build.add_target to a Project (a collection of targets), and either call build.run_target in code or odin run build -- <target name> via the CLI

Run_Mode is the mode the CLI runs in. It can be one of {.Build, .Dev, .Help}. The mode defaults to Build unless certain flags make it a different mode (running odin run build -- -ols sets the mode to .Dev).
The Run_Mode allows to user to determine what they want to do when the target is being ran. .Build would be the general build mode, where they can call build.odin, copy dlls, etc.
.Dev is used for configuring development enviornment (generating ols.json, information for the debugger they use. As a builtin debugger example, VSCode is implemented, but more can be added in future PRs, and the user can configure their own development environment.
.Help is for displaying information about your target to the user of the CLI. The current PR doesn't have a build.default_help proc. It's planned for later.

In order to facilitate importing build systems into eachother, the build.*path(target, path) functions make sure that your paths are relative to the build system. I have experimenting with ensuring that automatically on the library-side, but it seemed less "magical" this way (even if it still is a tiny bit). Internally, this system uses #caller_location when calling build.add_target in order to set Target.root_dir to dir(#caller_location)/../. This effectively means that the build system will always need to be built from the parent folder that it's in (odin build build as opposed to odin build .). This is the strangest part of this system, but it's going to make a lot of sense for dependency handling (being able to run imported targets like this odin run build -- project2/debug from the "main" build system.

Additional Features

• Wildcard selection for targets: odin run build -- *
• No allocations (or at least aims to be). The system is defined in a way that everything can be temp allocated. There might still be some artifacts of maps and [dynamic] arrays, but those will be removed in later PRs.

Testing for yourself

Open a command line in ODIN_ROOT/examples/package_build
run odin run build to build the project, or odin run build -- -help to display available options

Final notes

The (WIP) flags are still left to be implemented. The system of setting up custom flag is nearly in place, but i'm still thinking if it's supposed to be "general flags" or "per target" flags. There still needs to be an advanced example written where dependencies are being handled and including build systems into eachother.
If the API is good, I'd like to have this PR merged before doing other updates, as I'd like to open a clean PR for further improvements. I want to start clean on the git side, with a custom branch on my own fork where i can keep working on other things.

Future PRs (if this is accepted)

• Implement (WIP) flags
• Better user-code arg handling
• Dependency handling example

Future dream

core:build can be expanded to handle more than odin code. With the dependency handling, users could do something like import vendor:glfw/build and include the build system for glfw into their own, calling it's targets. This can be helpful for a lot of vendor libraries. vendor:glfw/build could be as simple as calling the build.bat on windows, but core:build can be expanded to be able to configure flags for C and CMake projects too.

[DragosPopse]

@gingerBill have a look too when you got some free time. I'm having a weird time figuring out how to start other pull requests while this one is open since I pushed to main in the fork. If this one is merged, I'll start a new PR draft with documentation and all the other missing things, along with core:odin/frontend work.

[flysand7]

I'm having a weird time figuring out how to start other pull requests while this one is open

Try making a new branch and pushing that.

$ git checkout -b other-pull-request
$ # Commit some stuff
$ git push -u my-remote other-pull-request

I strongly recommend that you have two remotes, one for your fork and another for upstream and use the master branch to pull changes from upstream and merge onto pull-requests in-progress

[flysand7]

Spaces

[DragosPopse]

fixed

[flysand7]

Creates the issue when one of the arguments contains spaces. If one of the elements in the args array contains a string with spaces, the user expects this string to be treated as a single argument (same as with execve() call).

However after this concatenation it's gonna be multiple arguments e.g.

{"program", "a b c"} ---> {"program a b c"}

not

{"program", "a b c"} ---> {"program \"a b c\""}

[DragosPopse]

i'll add better support for this once os2 is in place. For now there might not be a point in getting this further than the main use case. I'll see if i can fix it quickly though

[DragosPopse]

PR moved to #2958