wip

the function doesn't return anything (see function signature)

.github/workflows/checks.yml

@@ -13,12 +13,12 @@ jobs:
 
     steps:
     - name: Checkout repository
-      uses: actions/checkout@v2
+      uses: actions/checkout@v3
 
     - name: Set up Python
       uses: actions/setup-python@v3
       with:
-        python-version: '3.x'
+        python-version: '3.11'
 
     - name: Install dependencies
       run: |

.github/workflows/publish.yml

@@ -20,11 +20,14 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-    - uses: actions/checkout@v3
+    - name: Checkout repository
+      uses: actions/checkout@v3
+
     - name: Set up Python
       uses: actions/setup-python@v3
       with:
-        python-version: '3.x'
+        python-version: '3.11'
+
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip

.gitignore

@@ -127,3 +127,4 @@ dmypy.json
 
 # Pyre type checker
 .pyre/
+.vscode/

README.md

@@ -12,23 +12,20 @@
 <br/>
 <br/>
 
-## Table of contents
-
-- [Table of contents](#table-of-contents)
-- [Introduction](#introduction)
-- [Quick-start](#quick-start)
-- [Example](#example)
-
-
 ## Introduction
 
-**CuteKit** is a simple - yet - powerful build system and package manager for C and C++. It:
+**CuteKit** is a suite of tools and utilities for compiling, cross-compiling, linking, and packaging project written in low-level languages such as C, C++ or, Rust. Anything from a simple library to an operating system can be built using CuteKit.
 
-- ✨ It uses **JSON**: Cutekit uses JSON instead of introducing a whole new programming language for describing the project. And also has macros to help the user experience (see [Jexpr](doc/spec/jexpr.md)).
+- ✨ It uses **JSON**: Cutekit uses JSON instead of introducing a whole new programming language for describing the project. And also has macros for more advanced use cases (see [Jexpr](doc/spec/jexpr.md)).
 - ✨ It's a **package manager**: Cutekit package manager is based on **Git**, nothing is centralized.
 - ✨ It's **extendible**: Cutekit can be [extended](./doc/extends.md) by writing custom Python plugins.
 - ✨ It's **easy**: the [**templates**](./doc/templates.md) help the user quick-start a project.
-- ✨ It's **portable**: Cutekit can run on MacOS Gnu/Linux and Windows.
+- ✨ It's **portable**: Cutekit works on Linux, Windows, and MacOS.
+
+## CuteKit in the wild
+
+- [SkiftOS](https://github.com/skift-org/skift) : A hobbyist operating system written in C++.
+- [WKHtmlToPdf](https://github.com/odoo/wkhtmltopdf) : [Odoo](https://github.com/odoo/odoo)'s fork of wkhtmltopdf which is a command line tools to render HTML into PDF and various image formats using the Qt WebKit rendering engine.
 
 ## Installation
 
@@ -47,15 +44,15 @@ $ pip install --user -e .
 
 ## Quick-start
 
--> If you directly want to start using Cutekit for a new project, you can just run `$ ck I host` and it will create a new project in the host directory (you can rename it later).
+- If you directly want to start using Cutekit for a new project, you can just run `$ ck I host` and it will create a new project in the host directory (you can rename it later).
 
--> If you want to use Cutekit for writing operating systems, you can create a new [limine](https://github.com/limine-bootloader/limine/)-based project by running `$ ck I limine-barebone`.
+- If you want to use Cutekit for writing operating systems, you can create a new [limine](https://github.com/limine-bootloader/limine/)-based project by running `$ ck I limine-barebone`.
 
 ## Example
 
 If you want to see how it works you can read the [doc/cutekit.md](doc/cutekit.md) file.
 
-# License
+## License
 
 <a href="https://opensource.org/licenses/MIT">
   <img align="right" height="96" alt="MIT License" src="doc/mit.svg" />

cutekit/__init__.py

@@ -9,6 +9,8 @@ from . import (
     graph,  # noqa: F401 this is imported for side effects
     model,
     plugins,
+    pods,  # noqa: F401 this is imported for side effects
+    shell,
     vt100,
 )
 
@@ -26,47 +28,59 @@ def ensure(version: tuple[int, int, int]):
     )
 
 
-def setupLogger(verbose: bool):
-    if verbose:
-        logging.basicConfig(
-            level=logging.DEBUG,
-            format=f"{vt100.CYAN}%(asctime)s{vt100.RESET} {vt100.YELLOW}%(levelname)s{vt100.RESET} %(name)s: %(message)s",
-            datefmt="%Y-%m-%d %H:%M:%S",
-        )
-    else:
-        projectRoot = model.Project.topmost()
-        logFile = const.GLOBAL_LOG_FILE
-        if projectRoot is not None:
-            logFile = os.path.join(projectRoot.dirname(), const.PROJECT_LOG_FILE)
+class LoggerArgs:
+    verbose = cli.Arg[bool]("v", "verbose", "Enable verbose logging", default=False)
 
-        # create the directory if it doesn't exist
-        logDir = os.path.dirname(logFile)
-        if not os.path.isdir(logDir):
-            os.makedirs(logDir)
 
-        logging.basicConfig(
-            level=logging.INFO,
-            filename=logFile,
-            filemode="w",
-            format="%(asctime)s %(levelname)s %(name)s: %(message)s",
-            datefmt="%Y-%m-%d %H:%M:%S",
-        )
+class logger:
+    @staticmethod
+    def setup(args: LoggerArgs):
+        if args.verbose:
+            logging.basicConfig(
+                level=logging.DEBUG,
+                format=f"{vt100.CYAN}%(asctime)s{vt100.RESET} {vt100.YELLOW}%(levelname)s{vt100.RESET} %(name)s: %(message)s",
+                datefmt="%Y-%m-%d %H:%M:%S",
+            )
+        else:
+            projectRoot = model.Project.topmost()
+            logFile = const.GLOBAL_LOG_FILE
+            if projectRoot is not None:
+                logFile = os.path.join(projectRoot.dirname(), const.PROJECT_LOG_FILE)
+
+            shell.mkdir(os.path.dirname(logFile))
+
+            logging.basicConfig(
+                level=logging.INFO,
+                filename=logFile,
+                filemode="w",
+                format="%(asctime)s %(levelname)s %(name)s: %(message)s",
+                datefmt="%Y-%m-%d %H:%M:%S",
+            )
+
+
+class MainArgs(pods.PodArgs, plugins.PluginArgs, LoggerArgs):
+    pass
+
+
+@cli.command(None, "/", const.DESCRIPTION)
+def _(args: MainArgs):
+    shell.mkdir(const.GLOBAL_CK_DIR)
+    logger.setup(args)
+    const.setup()
+    plugins.setup(args)
+    pods.setup(args, sys.argv[1:])
 
 
 def main() -> int:
     try:
-        a = cli.parse(sys.argv[1:])
-        setupLogger(a.consumeOpt("verbose", False) is True)
-        plugins.loadAll()
-        cli.exec(a)
-        print()
+        cli.exec(sys.argv)
+
         return 0
     except RuntimeError as e:
         logging.exception(e)
         cli.error(str(e))
         cli.usage()
-        print()
-        return 1
     except KeyboardInterrupt:
         print()
-        return 1
+
+    return 1

cutekit/bootstrap.sh

@@ -0,0 +1,86 @@
+#!/usr/bin/env bash
+
+# This script is meant to be place at the root of any cutekit project.
+# It will make sure that the virtual environment is set up and that the
+# plugins requirements are installed.
+
+set -e
+
+if [ -z "$CUTEKIT_PYTHON" ]; then
+    export CUTEKIT_PYTHON="python3.11"
+fi
+
+if [ -z "$CUTEKIT_VERSION" ]; then
+    export CUTEKIT_VERSION="0.7-dev"
+fi
+
+if [ -n "$CUTEKIT_NOVENV" ]; then
+    echo "CUTEKIT_NOVENV is set, skipping virtual environment setup."
+    exec cutekit $@
+    exit $?
+fi
+
+if [ "$1" == "tools" -a "$2" == "nuke" ]; then
+    rm -rf .cutekit/tools .cutekit/venv
+    exit 0
+fi
+
+if [ ! -f .cutekit/tools/ready ]; then
+    if [ ! \( "$1" == "tools" -a "$2" == "setup" \) ]; then
+        echo "CuteKit is not installed."
+        echo "This script will install cutekit into $PWD/.cutekit"
+
+        read -p "Do you want to continue? [Y/n] " -n 1 -r
+        echo
+        if [[ ! $REPLY =~ ^[Yy]$ ]]; then
+            echo "Aborting."
+            exit 1
+        fi
+    else
+        echo "Installing CuteKit..."
+    fi
+
+    mkdir -p .cutekit
+    if [ ! -d .cutekit/venv ]; then
+        echo "Setting up Python virtual environment..."
+        $CUTEKIT_PYTHON -m venv .cutekit/venv
+    fi
+    source .cutekit/venv/bin/activate
+
+    echo "Downloading CuteKit..."
+    if [ ! -d .cutekit/tools/cutekit ]; then
+        git clone --depth 1 https://github.com/cute-engineering/cutekit .cutekit/tools/cutekit --branch "$CUTEKIT_VERSION"
+    else
+        echo "CuteKit already downloaded."
+    fi
+
+    echo "Installing Tools..."
+    $CUTEKIT_PYTHON -m pip install -e .cutekit/tools/cutekit
+
+    echo "Installing plugins requirements..."
+    if [ -f "meta/plugins/requirements.txt" ]; then
+        echo "Root plugin requirements found."
+        $CUTEKIT_PYTHON -m pip install -r meta/plugins/requirements.txt
+    fi
+
+    for extern in meta/externs/*; do
+        if [ -f "$extern/meta/plugins/requirements.txt" ]; then
+            echo "Plugin requirements found in $extern."
+            $CUTEKIT_PYTHON -m pip install -r "$extern/meta/plugins/requirements.txt"
+        fi
+    done
+
+    touch .cutekit/tools/ready
+    echo "Done!"
+fi
+
+if [ "$1" == "tools" -a "$2" == "setup" ]; then
+    echo "Tools already installed."
+    exit 0
+fi
+
+source .cutekit/venv/bin/activate
+export PATH="$PATH:.cutekit/venv/bin"
+
+$CUTEKIT_PYTHON -m cutekit $@
+

cutekit/builder.py

@@ -3,32 +3,117 @@ import logging
 import dataclasses as dt
 
 from pathlib import Path
-from typing import TextIO, Union
+from typing import Callable, Literal, TextIO, Union, Any
 
 from . import shell, rules, model, ninja, const, cli
 
 _logger = logging.getLogger(__name__)
 
 
-def aggregateCincs(target: model.Target, registry: model.Registry) -> set[str]:
+@dt.dataclass
+class Scope:
+    registry: model.Registry
+
+    @staticmethod
+    def use(args: Any, props: model.Props = {}) -> "Scope":
+        registry = model.Registry.use(args, props)
+        return Scope(registry)
+
+    def key(self) -> str:
+        return self.registry.project.id
+
+    def openTargetScope(self, t: model.Target):
+        return TargetScope(self.registry, t)
+
+
+@dt.dataclass
+class TargetScope(Scope):
+    registry: model.Registry
+    target: model.Target
+
+    @staticmethod
+    def use(args: Any, props: model.Props = {}) -> "TargetScope":
+        registry = model.Registry.use(args, props)
+        target = model.Target.use(args, props)
+        return TargetScope(registry, target)
+
+    def key(self) -> str:
+        return super().key() + "/" + self.target.id + "/" + self.target.hashid
+
+    def openComponentScope(self, c: model.Component):
+        return ComponentScope(self.registry, self.target, c)
+
+
+@dt.dataclass
+class ComponentScope(TargetScope):
+    component: model.Component
+
+    def key(self) -> str:
+        return super().key() + "/" + self.component.id
+
+    def openComponentScope(self, c: model.Component):
+        return ComponentScope(self.registry, self.target, c)
+
+    def openProductScope(self, path: Path):
+        return ProductScope(self.registry, self.target, self.component, path)
+
+
+@dt.dataclass
+class ProductScope(ComponentScope):
+    path: Path
+
+
+# --- Variables -------------------------------------------------------------- #
+
+Compute = Callable[[TargetScope], str]
+_vars: dict[str, Compute] = {}
+
+
+def var(name: str) -> Callable[[Compute], Compute]:
+    def decorator(func: Compute):
+        _vars[name] = func
+        return func
+
+    return decorator
+
+
+@var("builddir")
+def _computeBuilddir(scope: TargetScope) -> str:
+    """
+    This variable is needed by ninja to know where to put
+    the .ninja_log file.
+    """
+    return scope.target.builddir
+
+
+@var("hashid")
+def _computeHashid(scope: TargetScope) -> str:
+    return scope.target.hashid
+
+
+@var("cincs")
+def _computeCinc(scope: TargetScope) -> str:
     res = set()
 
-    for c in registry.iterEnabled(target):
+    for c in scope.registry.iterEnabled(scope.target):
         if "cpp-root-include" in c.props:
             res.add(c.dirname())
+        elif "cpp-excluded" in c.props:
+            pass
         elif c.type == model.Kind.LIB:
             res.add(str(Path(c.dirname()).parent))
 
-    return set(map(lambda i: f"-I{i}", res))
+    return " ".join(sorted(map(lambda i: f"-I{i}", res)))
 
 
-def aggregateCdefs(target: model.Target) -> set[str]:
+@var("cdefs")
+def _computeCdef(scope: TargetScope) -> str:
     res = set()
 
     def sanatize(s: str) -> str:
         return s.lower().replace(" ", "_").replace("-", "_").replace(".", "_")
 
-    for k, v in target.props.items():
+    for k, v in scope.target.props.items():
         if isinstance(v, bool):
             if v:
                 res.add(f"-D__ck_{sanatize(k)}__")
@@ -36,40 +121,51 @@ def aggregateCdefs(target: model.Target) -> set[str]:
             res.add(f"-D__ck_{sanatize(k)}_{sanatize(str(v))}__")
             res.add(f"-D__ck_{sanatize(k)}_value={str(v)}")
 
-    return res
+    return " ".join(sorted(res))
 
 
-def buildpath(target: model.Target, component: model.Component, path) -> Path:
-    return Path(target.builddir) / component.id / path
+def buildpath(scope: ComponentScope, path) -> Path:
+    return Path(scope.target.builddir) / scope.component.id / path
 
 
 # --- Compilation ------------------------------------------------------------ #
 
 
-def wilcard(component: model.Component, wildcards: list[str]) -> list[str]:
-    dirs = [component.dirname()] + list(
-        map(lambda d: os.path.join(component.dirname(), d), component.subdirs)
-    )
-    return shell.find(dirs, list(wildcards), recusive=False)
+def subdirs(scope: ComponentScope) -> list[str]:
+    component = scope.component
+    result = [component.dirname()]
+
+    for subs in component.subdirs:
+        result.append(os.path.join(component.dirname(), subs))
+
+    return result
+
+
+def wilcard(scope: ComponentScope, wildcards: list[str]) -> list[str]:
+    return shell.find(subdirs(scope), list(wildcards), recusive=False)
 
 
 def compile(
-    w: ninja.Writer,
-    target: model.Target,
-    component: model.Component,
-    rule: str,
-    srcs: list[str],
+    w: ninja.Writer, scope: ComponentScope, rule: str, srcs: list[str]
 ) -> list[str]:
     res: list[str] = []
     for src in srcs:
-        rel = Path(src).relative_to(component.dirname())
-        dest = buildpath(target, component, "obj") / rel.with_suffix(".o")
-        t = target.tools[rule]
+        rel = Path(src).relative_to(scope.component.dirname())
+        dest = buildpath(scope, path="__obj__") / rel.with_suffix(rel.suffix + ".o")
+        t = scope.target.tools[rule]
         w.build(str(dest), rule, inputs=src, order_only=t.files)
         res.append(str(dest))
     return res
 
 
+def compileObjs(w: ninja.Writer, scope: ComponentScope) -> list[str]:
+    objs = []
+    for rule in rules.rules.values():
+        if rule.id not in ["cp", "ld", "ar"]:
+            objs += compile(w, scope, rule.id, wilcard(scope, rule.fileIn))
+    return objs
+
+
 # --- Ressources ------------------------------------------------------------- #
 
 
@@ -79,13 +175,12 @@ def listRes(component: model.Component) -> list[str]:
 
 def compileRes(
     w: ninja.Writer,
-    target: model.Target,
-    component: model.Component,
+    scope: ComponentScope,
 ) -> list[str]:
     res: list[str] = []
-    for r in listRes(component):
-        rel = Path(r).relative_to(component.subpath("res"))
-        dest = buildpath(target, component, "res") / rel
+    for r in listRes(scope.component):
+        rel = Path(r).relative_to(scope.component.subpath("res"))
+        dest = buildpath(scope, "__res__") / rel
         w.build(str(dest), "cp", r)
         res.append(str(dest))
     return res
@@ -94,204 +189,228 @@ def compileRes(
 # --- Linking ---------------------------------------------------------------- #
 
 
-def outfile(target: model.Target, component: model.Component) -> str:
-    if component.type == model.Kind.LIB:
-        return str(buildpath(target, component, f"lib/{component.id}.a"))
+def outfile(scope: ComponentScope) -> str:
+    if scope.component.type == model.Kind.LIB:
+        return str(buildpath(scope, f"__lib__/{scope.component.id}.a"))
     else:
-        return str(buildpath(target, component, f"bin/{component.id}.out"))
+        return str(buildpath(scope, f"__bin__/{scope.component.id}.out"))
 
 
 def collectLibs(
-    registry: model.Registry, target: model.Target, component: model.Component
+    scope: ComponentScope,
 ) -> list[str]:
     res: list[str] = []
-    for r in component.resolved[target.id].resolved:
-        req = registry.lookup(r, model.Component)
+    for r in scope.component.resolved[scope.target.id].required:
+        req = scope.registry.lookup(r, model.Component)
         assert req is not None  # model.Resolver has already checked this
 
-        if r == component.id:
+        if r == scope.component.id:
             continue
         if not req.type == model.Kind.LIB:
             raise RuntimeError(f"Component {r} is not a library")
-        res.append(outfile(target, req))
+        res.append(outfile(scope.openComponentScope(req)))
+
+    return res
+
+
+def collectInjectedLibs(scope: ComponentScope) -> list[str]:
+    res: list[str] = []
+    for r in scope.component.resolved[scope.target.id].injected:
+        req = scope.registry.lookup(r, model.Component)
+        assert req is not None  # model.Resolver has already checked this
+
+        if r == scope.component.id:
+            continue
+        if not req.type == model.Kind.LIB:
+            raise RuntimeError(f"Component {r} is not a library")
+        res.append(outfile(scope.openComponentScope(req)))
+
     return res
 
 
 def link(
     w: ninja.Writer,
-    registry: model.Registry,
-    target: model.Target,
-    component: model.Component,
+    scope: ComponentScope,
 ) -> str:
     w.newline()
-    out = outfile(target, component)
+    out = outfile(scope)
 
-    objs = []
-    objs += compile(w, target, component, "cc", wilcard(component, ["*.c"]))
-    objs += compile(
-        w, target, component, "cxx", wilcard(component, ["*.cpp", "*.cc", "*.cxx"])
-    )
-    objs += compile(
-        w, target, component, "as", wilcard(component, ["*.s", "*.asm", "*.S"])
-    )
-
-    res = compileRes(w, target, component)
-    libs = collectLibs(registry, target, component)
-    if component.type == model.Kind.LIB:
+    res = compileRes(w, scope)
+    objs = compileObjs(w, scope)
+    if scope.component.type == model.Kind.LIB:
         w.build(out, "ar", objs, implicit=res)
     else:
-        w.build(out, "ld", objs + libs, implicit=res)
+        wholeLibs = collectInjectedLibs(scope)
+        libs = collectLibs(scope)
+        w.build(
+            out,
+            "ld",
+            objs + wholeLibs + libs,
+            variables={
+                "objs": " ".join(objs),
+                "wholeLibs": " ".join(wholeLibs),
+                "libs": " ".join(libs),
+            },
+            implicit=res,
+        )
     return out
 
 
 # --- Phony ------------------------------------------------------------------ #
 
 
-def all(w: ninja.Writer, registry: model.Registry, target: model.Target) -> list[str]:
+def all(w: ninja.Writer, scope: TargetScope) -> list[str]:
     all: list[str] = []
-    for c in registry.iterEnabled(target):
-        all.append(link(w, registry, target, c))
+    for c in scope.registry.iterEnabled(scope.target):
+        all.append(link(w, scope.openComponentScope(c)))
     w.build("all", "phony", all)
     w.default("all")
     return all
 
 
-def gen(out: TextIO, target: model.Target, registry: model.Registry):
+def gen(out: TextIO, scope: TargetScope):
     w = ninja.Writer(out)
 
     w.comment("File generated by the build system, do not edit")
     w.newline()
 
-    w.variable("builddir", target.builddir)
-    w.variable("hashid", target.hashid)
+    w.separator("Variables")
+    for name, compute in _vars.items():
+        w.variable(name, compute(scope))
+    w.newline()
 
     w.separator("Tools")
 
-    w.variable("cincs", " ".join(aggregateCincs(target, registry)))
-    w.variable("cdefs", " ".join(aggregateCdefs(target)))
-    w.newline()
-
-    for i in target.tools:
-        tool = target.tools[i]
+    for i in scope.target.tools:
+        tool = scope.target.tools[i]
         rule = rules.rules[i]
         w.variable(i, tool.cmd)
         w.variable(i + "flags", " ".join(rule.args + tool.args))
         w.rule(
             i,
-            f"{tool.cmd} {rule.rule.replace('$flags',f'${i}flags')}",
+            f"{tool.cmd} {(tool.rule or rule.rule).replace('$flags',f'${i}flags')}",
             depfile=rule.deps,
         )
         w.newline()
 
     w.separator("Build")
 
-    all(w, registry, target)
-
-
-@dt.dataclass
-class Product:
-    path: Path
-    target: model.Target
-    component: model.Component
+    all(w, scope)
 
 
 def build(
-    target: model.Target,
-    registry: model.Registry,
-    components: Union[list[model.Component], model.Component, None] = None,
-) -> list[Product]:
+    scope: TargetScope,
+    components: Union[list[model.Component], model.Component, Literal["all"]] = "all",
+) -> list[ProductScope]:
     all = False
-    shell.mkdir(target.builddir)
-    ninjaPath = os.path.join(target.builddir, "build.ninja")
-    with open(ninjaPath, "w") as f:
-        gen(f, target, registry)
+    shell.mkdir(scope.target.builddir)
+    ninjaPath = os.path.join(scope.target.builddir, "build.ninja")
 
-    if components is None:
+    with open(ninjaPath, "w") as f:
+        gen(f, scope)
+
+    if components == "all":
         all = True
-        components = list(registry.iterEnabled(target))
+        components = list(scope.registry.iterEnabled(scope.target))
 
     if isinstance(components, model.Component):
         components = [components]
 
-    products: list[Product] = []
+    products: list[ProductScope] = []
     for c in components:
-        r = c.resolved[target.id]
+        s = scope.openComponentScope(c)
+        r = c.resolved[scope.target.id]
         if not r.enabled:
             raise RuntimeError(f"Component {c.id} is disabled: {r.reason}")
 
-        products.append(
-            Product(
-                path=Path(outfile(target, c)),
-                target=target,
-                component=c,
-            )
-        )
+        products.append(s.openProductScope(Path(outfile(scope.openComponentScope(c)))))
 
     outs = list(map(lambda p: str(p.path), products))
-    if all:
-        shell.exec("ninja", "-f", ninjaPath)
-    else:
-        shell.exec("ninja", "-f", ninjaPath, *outs)
+
+    shell.exec("ninja", "-f", ninjaPath, *(outs if not all else []))
+
     return products
 
 
 # --- Commands --------------------------------------------------------------- #
 
 
-@cli.command("b", "build", "Build a component or all components")
-def buildCmd(args: cli.Args):
-    registry = model.Registry.use(args)
-    target = model.Target.use(args)
+@cli.command("b", "builder", "Build/Run/Clean a component or all components")
+def _():
+    pass
+
+
+@cli.command("b", "builder/build", "Build a component or all components")
+def _(args: Any):
+    scope = TargetScope.use(args)
     componentSpec = args.consumeArg()
-    if componentSpec is None:
-        raise RuntimeError("No component specified")
-    component = registry.lookup(componentSpec, model.Component)
-    build(target, registry, component)[0]
+    component = None
+    if componentSpec is not None:
+        component = scope.registry.lookup(componentSpec, model.Component)
+    build(scope, component if component is not None else "all")[0]
 
 
-@cli.command("r", "run", "Run a component")
-def runCmd(args: cli.Args):
-    registry = model.Registry.use(args)
-    target = model.Target.use(args)
-    debug = args.consumeOpt("debug", False) is True
+class RunArgs(model.RegistryArgs, shell.DebuggerArgs):
+    debug = cli.Arg[bool]("d", "debug", "Enable debug mode", default=False)
+    profile = cli.Arg[bool]("p", "profile", "Enable profiling", default=False)
+    component = cli.FreeFormArg(
+        "component", "Component to run", default="__main__"
+    )
+    extra: cli.RawArg
 
-    componentSpec = args.consumeArg() or "__main__"
-    component = registry.lookup(componentSpec, model.Component, includeProvides=True)
+
+@cli.command("r", "builder/run", "Run a component")
+def runCmd(args: RunArgs):
+    scope = TargetScope.use(args, {"debug": args.debug})
+
+    component = scope.registry.lookup(
+        args.component, model.Component, includeProvides=True
+    )
     if component is None:
-        raise RuntimeError(f"Component {componentSpec} not found")
+        raise RuntimeError(f"Component {args.component} not found")
 
-    product = build(target, registry, component)[0]
+    product = build(scope, component)[0]
 
-    os.environ["CK_TARGET"] = target.id
+    os.environ["CK_TARGET"] = product.target.id
+    os.environ["CK_BUILDDIR"] = product.target.builddir
     os.environ["CK_COMPONENT"] = product.component.id
-    os.environ["CK_BUILDDIR"] = target.builddir
 
-    shell.exec(*(["lldb", "-o", "run"] if debug else []), str(product.path), *args.args)
+    try:
+        command = [str(product.path), *args.extra]
+
+        if args.debug:
+            shell.debug(command, debugger=args.debugger, wait=args.wait)
+        elif args.profile:
+            shell.profile(command)
+        else:
+            shell.exec(*command)
+    except Exception as e:
+        cli.error(str(e))
 
 
-@cli.command("t", "test", "Run all test targets")
-def testCmd(args: cli.Args):
+@cli.command("t", "builder/test", "Run all test targets")
+def _(args: RunArgs):
     # This is just a wrapper around the `run` command that try
     # to run a special hook component named __tests__.
-    args.args.insert(0, "__tests__")
+    args.component = "__tests__"
     runCmd(args)
 
 
-@cli.command("d", "debug", "Debug a component")
-def debugCmd(args: cli.Args):
+@cli.command("d", "builder/debug", "Debug a component")
+def _(args: RunArgs):
     # This is just a wrapper around the `run` command that
     # always enable debug mode.
-    args.opts["debug"] = True
+    args.debug = True
     runCmd(args)
 
 
-@cli.command("c", "clean", "Clean build files")
-def cleanCmd(args: cli.Args):
-    model.Project.use(args)
+@cli.command("c", "builder/clean", "Clean build files")
+def _():
+    model.Project.use()
     shell.rmrf(const.BUILD_DIR)
 
 
-@cli.command("n", "nuke", "Clean all build files and caches")
-def nukeCmd(args: cli.Args):
-    model.Project.use(args)
+@cli.command("n", "builder/nuke", "Clean all build files and caches")
+def _():
+    model.Project.use()
     shell.rmrf(const.PROJECT_CK_DIR)

cutekit/cli.py

@@ -1,75 +1,214 @@
+import enum
 import inspect
 import logging
 import sys
 import dataclasses as dt
 
+from functools import partial
 from pathlib import Path
-from typing import Optional, Union, Callable
+from typing import (
+    Any,
+    NewType,
+    Optional,
+    Union,
+    Callable,
+    Generic,
+    cast,
+    get_origin,
+    get_args,
+)
 
-from . import const, vt100
+from . import const, vt100, utils
 
 Value = Union[str, bool, int]
 
 _logger = logging.getLogger(__name__)
 
 
-class Args:
-    opts: dict[str, Value]
-    args: list[str]
-
-    def __init__(self):
-        self.opts = {}
-        self.args = []
-
-    def consumePrefix(self, prefix: str) -> dict[str, Value]:
-        result: dict[str, Value] = {}
-        copy = self.opts.copy()
-        for key, value in copy.items():
-            if key.startswith(prefix):
-                result[key[len(prefix) :]] = value
-                del self.opts[key]
-        return result
-
-    def consumeOpt(self, key: str, default: Value = False) -> Value:
-        if key in self.opts:
-            result = self.opts[key]
-            del self.opts[key]
-            return result
-        return default
-
-    def tryConsumeOpt(self, key: str) -> Optional[Value]:
-        if key in self.opts:
-            result = self.opts[key]
-            del self.opts[key]
-            return result
-        return None
-
-    def consumeArg(self, default: Optional[str] = None) -> Optional[str]:
-        if len(self.args) == 0:
-            return default
-
-        first = self.args[0]
-        del self.args[0]
-        return first
+# --- Arg parsing -------------------------------------------------------------
 
 
-def parse(args: list[str]) -> Args:
-    result = Args()
+@dt.dataclass
+class Arg(Generic[utils.T]):
+    shortName: str
+    longName: str
+    description: str
+    default: Optional[utils.T] = None
 
-    for arg in args:
-        if arg.startswith("--"):
-            if "=" in arg:
-                key, value = arg[2:].split("=", 1)
-                result.opts[key] = value
-            else:
-                result.opts[arg[2:]] = True
+    def __get__(self, instance, owner) -> utils.T:
+        if instance is None:
+            return self  # type: ignore
+
+        return instance.__dict__.get(self.longName, self.default)
+
+    def __set__(self, instance, value: utils.T):
+        instance.__dict__[self.longName] = value
+
+
+@dt.dataclass
+class FreeFormArg(Generic[utils.T]):
+    longName: str
+    description: str
+    default: Optional[utils.T] = None
+
+    def __get__(self, instance, owner) -> utils.T:
+        if instance is None:
+            return self  # type: ignore
+
+        return instance.__dict__.get(self.longName, self.default)
+
+    def __set__(self, instance, value: utils.T):
+        instance.__dict__[self.longName] = value
+
+
+@dt.dataclass
+class RawArg:
+    longName: str
+    description: str
+
+    def __get__(self, instance, owner) -> list[str]:
+        if instance is None:
+            return self  # type: ignore
+
+        return instance.__dict__.get(self.longName, [])
+
+    def __set__(self, instance, value):
+        instance.__dict__[self.longName] = value
+
+
+class ParserState(enum.Enum):
+    FreeForm = enum.auto()
+    ShortArg = enum.auto()
+
+
+def parse(argv: list[str], argType: type[utils.T]) -> utils.T:
+    def set_value(options: dict[str, Any], name: str, value: Any):
+        if name is not options:
+            options[name] = value
         else:
-            result.args.append(arg)
+            raise RuntimeError(f"{name} is already set")
+
+    def is_optional(t: type) -> bool:
+        return get_origin(t) is Union and type(None) in get_args(t)
+
+    def freeforms_get(argType: type) -> tuple[list[str], list[str]]:
+        freeforms = []
+        required_freeforms = []
+
+        found_optional = False
+        for arg, anno in [
+            arg
+            for arg in argType.__annotations__.items()
+            if get_origin(arg[1]) is FreeFormArg
+        ]:
+            freeforms.append(arg)
+            if is_optional(get_args(anno)[0]):
+                found_optional = True
+            elif found_optional:
+                raise RuntimeError(
+                    "Required arguments must come before optional arguments"
+                )
+            else:
+                required_freeforms.append(arg)
+
+        return (freeforms, required_freeforms)
+
+    result = argType()
+    options: dict[str, Any] = {}
+    args: dict[str, partial] = {}
+    freeforms: list[Any] = []
+
+    state = ParserState.FreeForm
+    current_arg: Optional[str] = None
+
+    for arg in dir(argType):
+        if isinstance(getattr(argType, arg), Arg):
+            args[getattr(argType, arg).shortName] = partial(set_value, options, arg)
+            args[getattr(argType, arg).longName] = partial(set_value, options, arg)
+
+    i = 0
+    while i < len(argv):
+        match state:
+            case ParserState.FreeForm:
+                if argv[i] == "--":
+                    freeargs = argv[i + 1 :]
+                    i += 1
+                    break
+                if argv[i].startswith("--"):
+                    if "=" in argv[i]:
+                        # --name=value
+                        name, value = argv[i][2:].split("=", 1)
+                        if name in args:
+                            args[name](value)
+                    else:
+                        # --name -> the value will be True
+                        if argv[i][2:] in args:
+                            args[argv[i][2:]](True)
+                elif argv[i].startswith("-"):
+                    if len(argv[i][1:]) > 1:
+                        for c in argv[i][1:]:
+                            # -abc -> a, b, c are all True
+                            if c in args:
+                                args[c](True)
+                    else:
+                        state = ParserState.ShortArg
+                        current_arg = argv[i][1:]
+                else:
+                    freeforms.append(argv[i])
+
+                i += 1
+            case ParserState.ShortArg:
+                if argv[i].startswith("-"):
+                    # -a -b 4 -> a is True
+                    if current_arg in args:
+                        args[current_arg](True)
+                else:
+                    # -a 4 -> a is 4
+                    if current_arg in args:
+                        args[current_arg](argv[i])
+
+                i += 1
+                current_arg = None
+                state = ParserState.FreeForm
+
+    freeforms_all, required_freeforms = freeforms_get(argType)
+    if len(freeforms) < len(required_freeforms):
+        raise RuntimeError(
+            f"Missing arguments: {', '.join(required_freeforms[len(freeforms):])}"
+        )
+    if len(freeforms) > len(freeforms_all):
+        raise RuntimeError("Too many arguments")
+
+    for i, freeform in enumerate(freeforms):
+        setattr(result, freeforms_all[i], freeform)
+
+    # missing arguments
+    missing = set(
+        [
+            arg[0]
+            for arg in argType.__annotations__
+            if get_origin(arg[1]) is Arg and getattr(argType, arg[0]).default is None
+        ]
+    ) - set(options.keys())
+    if missing:
+        raise RuntimeError(f"Missing arguments: {', '.join(missing)}")
+
+    for key, value in options.items():
+        # print(getattr(argType, key).type())
+        field_type = get_args(argType.__annotations__[key])[0]
+        setattr(result, key, field_type(value))
+
+    raw_args = [arg[0] for arg in argType.__annotations__.items() if arg[1] is RawArg]
+
+    if len(raw_args) > 1:
+        raise RuntimeError("Only one RawArg is allowed")
+    elif len(raw_args) == 1:
+        setattr(result, raw_args[0], freeargs)
 
     return result
 
 
-Callback = Callable[[Args], None]
+Callback = Callable[[Any], None] | Callable[[], None]
 
 
 @dt.dataclass
@@ -79,26 +218,60 @@ class Command:
     helpText: str
     isPlugin: bool
     callback: Callback
+    argType: Optional[type]
+
+    subcommands: dict[str, "Command"] = dt.field(default_factory=dict)
+
+    def help(self):
+        print(f"{self.longName} - {self.helpText}")
+        print()
+        self.usage()
+
+    def usage(self):
+        usage = f"Usage: {self.longName}"
+
+        if self.argType is not None:
+            usage += " [args...]"
+
+        if len(self.subcommands) > 0:
+            usage += " <command> [args...]"
+
+        print(usage)
 
 
-commands: list[Command] = []
+commands: dict[str, Command] = {}
+rootCommand: Optional[Command] = None
 
 
 def command(shortName: Optional[str], longName: str, helpText: str):
     curframe = inspect.currentframe()
     calframe = inspect.getouterframes(curframe, 2)
 
-    def wrap(fn: Callable[[Args], None]):
+    def wrap(fn: utils.T) -> utils.T:
         _logger.debug(f"Registering command {longName}")
-        commands.append(
-            Command(
-                shortName,
-                longName,
-                helpText,
-                Path(calframe[1].filename).parent != Path(__file__).parent,
-                fn,
-            )
+        if len(fn.__annotations__) == 0:
+            argType = None
+        else:
+            argType = list(fn.__annotations__.values())[0]
+        path = longName.split("/")
+        command = Command(
+            shortName,
+            path[-1] if longName != "/" else "/",
+            helpText,
+            Path(calframe[1].filename).parent != Path(__file__).parent,
+            cast(Callback, fn),
+            argType,
         )
+
+        if longName == "/":
+            global rootCommand
+            rootCommand = command
+        else:
+            parent = commands
+            for p in path[:-1]:
+                parent = parent[p].subcommands
+            parent[path[-1]] = command
+
         return fn
 
     return wrap
@@ -108,7 +281,7 @@ def command(shortName: Optional[str], longName: str, helpText: str):
 
 
 @command("u", "usage", "Show usage information")
-def usage(args: Optional[Args] = None):
+def usage():
     print(f"Usage: {const.ARGV0} <command> [args...]")
 
 
@@ -116,8 +289,30 @@ def error(msg: str) -> None:
     print(f"{vt100.RED}Error:{vt100.RESET} {msg}\n", file=sys.stderr)
 
 
+def warning(msg: str) -> None:
+    print(f"{vt100.YELLOW}Warning:{vt100.RESET} {msg}\n", file=sys.stderr)
+
+
+def ask(msg: str, default: Optional[bool] = None) -> bool:
+    if default is None:
+        msg = f"{msg} [y/n] "
+    elif default:
+        msg = f"{msg} [Y/n] "
+    else:
+        msg = f"{msg} [y/N] "
+
+    while True:
+        result = input(msg).lower()
+        if result in ("y", "yes"):
+            return True
+        elif result in ("n", "no"):
+            return False
+        elif result == "" and default is not None:
+            return default
+
+
 @command("h", "help", "Show this help message")
-def helpCmd(args: Args):
+def helpCmd():
     usage()
 
     print()
@@ -127,8 +322,8 @@ def helpCmd(args: Args):
 
     print()
     vt100.title("Commands")
-    for cmd in sorted(commands, key=lambda c: c.longName):
-        if cmd.longName.startswith("_"):
+    for cmd in sorted(commands.values(), key=lambda c: c.longName):
+        if cmd.longName.startswith("_") or len(cmd.subcommands) > 0:
             continue
 
         pluginText = ""
@@ -139,6 +334,21 @@ def helpCmd(args: Args):
             f" {vt100.GREEN}{cmd.shortName or ' '}{vt100.RESET}  {cmd.longName} - {cmd.helpText} {pluginText}"
         )
 
+    for cmd in sorted(commands.values(), key=lambda c: c.longName):
+        if cmd.longName.startswith("_") or len(cmd.subcommands) == 0:
+            continue
+
+        print()
+        vt100.title(f"{cmd.longName.capitalize()} - {cmd.helpText}")
+        for subcmd in sorted(cmd.subcommands.values(), key=lambda c: c.longName):
+            pluginText = ""
+            if subcmd.isPlugin:
+                pluginText = f"{vt100.CYAN}(plugin){vt100.RESET}"
+
+            print(
+                f"     {vt100.GREEN}{subcmd.shortName or ' '}{vt100.RESET}  {subcmd.longName} - {subcmd.helpText} {pluginText}"
+            )
+
     print()
     vt100.title("Logging")
     print("    Logs are stored in:")
@@ -147,19 +357,53 @@ def helpCmd(args: Args):
 
 
 @command("v", "version", "Show current version")
-def versionCmd(args: Args):
+def versionCmd():
     print(f"CuteKit v{const.VERSION_STR}")
 
 
-def exec(args: Args):
-    cmd = args.consumeArg()
+def _exec(args: list[str], cmd: Command):
+    # let's slice the arguments for this command and the sub command
+    # [-a -b] [sub-cmd -c -d]
 
-    if cmd is None:
-        raise RuntimeError("No command specified")
+    selfArgs = []
+    if len(cmd.subcommands) > 0:
+        while len(args) > 0 and args[0].startswith("-"):
+            selfArgs.append(args.pop(0))
+    else:
+        selfArgs = args
 
-    for c in commands:
-        if c.shortName == cmd or c.longName == cmd:
-            c.callback(args)
+    if "-h" in selfArgs or "--help" in selfArgs:
+        cmd.help()
+        return
+
+    if "-u" in selfArgs or "--usage" in selfArgs:
+        cmd.usage()
+        return
+
+    if cmd.argType is not None:
+        parsedArgs = parse(selfArgs, cmd.argType)
+        cmd.callback(parsedArgs)  # type: ignore
+    else:
+        cmd.callback()  # type: ignore
+
+    if cmd.subcommands:
+        if len(args) == 0:
+            error("Missing subcommand")
+            cmd.usage()
             return
 
-    raise RuntimeError(f"Unknown command {cmd}")
+        for c in cmd.subcommands.values():
+            if c.shortName == args[0] or c.longName == args[0]:
+                _exec(args, c)
+                return
+
+        raise RuntimeError(f"Unknown command {args[0]}")
+
+
+def exec(args: list[str]):
+    if not rootCommand:
+        raise RuntimeError("No root command")
+
+    rootCommand.longName = Path(args[0]).name
+    rootCommand.subcommands = commands
+    _exec(args[1:], rootCommand)

cutekit/const.py

@@ -1,7 +1,21 @@
 import os
 import sys
 
-VERSION = (0, 6, 0, "dev")
+from . import utils
+
+
+class Uninitialized:
+    def __repr__(self):
+        raise Exception("Uninitialized constant")
+
+    def __str__(self):
+        raise Exception("Uninitialized constant")
+
+    def __bool__(self):
+        raise Exception("Uninitialized constant")
+
+
+VERSION = (0, 7, 0, "dev")
 VERSION_STR = f"{VERSION[0]}.{VERSION[1]}.{VERSION[2]}{'-' + VERSION[3] if len(VERSION) >= 4 else ''}"
 MODULE_DIR = os.path.dirname(os.path.realpath(__file__))
 ARGV0 = os.path.basename(sys.argv[0])
@@ -10,10 +24,24 @@ GLOBAL_CK_DIR = os.path.join(os.path.expanduser("~"), ".cutekit")
 BUILD_DIR = os.path.join(PROJECT_CK_DIR, "build")
 CACHE_DIR = os.path.join(PROJECT_CK_DIR, "cache")
 EXTERN_DIR = os.path.join(PROJECT_CK_DIR, "extern")
+TMP_DIR = os.path.join(PROJECT_CK_DIR, "tmp")
 SRC_DIR = "src"
 META_DIR = "meta"
 TARGETS_DIR = os.path.join(META_DIR, "targets")
 DEFAULT_REPO_TEMPLATES = "cute-engineering/cutekit-templates"
 DESCRIPTION = "A build system and package manager for low-level software development"
 PROJECT_LOG_FILE = os.path.join(PROJECT_CK_DIR, "cutekit.log")
-GLOBAL_LOG_FILE = os.path.join(os.path.expanduser("~"), ".cutekit", "cutekit.log")
+GLOBAL_LOG_FILE: str = os.path.join(os.path.expanduser("~"), ".cutekit", "cutekit.log")
+HOSTID: str | Uninitialized = Uninitialized()
+
+
+def setup():
+    global HOSTID
+    hostIdPath = GLOBAL_CK_DIR + "/hostid"
+    if os.path.exists(hostIdPath):
+        with open(hostIdPath, "r") as f:
+            HOSTID = f.read().strip()
+    else:
+        HOSTID = utils.randomHash()
+        with open(hostIdPath, "w") as f:
+            f.write(HOSTID)

cutekit/entrypoint.sh

@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+
+# This script makes sure that the virtual environment is
+# set up and that the plugins requirements are installed.
+
+set -e
+
+if [ -z "$CUTEKIT_PYTHON" ]; then
+    export CUTEKIT_PYTHON="python3.11"
+fi
+
+if [ ! -d "/tools/venv" ]; then
+    echo "Creating virtual environment..."
+
+    $CUTEKIT_PYTHON -m venv /tools/venv
+    source /tools/venv/bin/activate
+    $CUTEKIT_PYTHON -m ensurepip
+    $CUTEKIT_PYTHON -m pip install -r /tools/cutekit/requirements.txt
+
+    echo "Installing plugins requirements..."
+    if [ -f "/project/meta/plugins/requirements.txt" ]; then
+        echo "Root plugin requirements found."
+        $CUTEKIT_PYTHON -m pip install -r /project/meta/plugins/requirements.txt
+    fi
+
+    for extern in /project/meta/externs/*; do
+        if [ -f "$extern/meta/plugins/requirements.txt" ]; then
+            echo "Plugin requirements found in $extern."
+            $CUTEKIT_PYTHON -m pip install -r "$extern/meta/plugins/requirements.txt"
+        fi
+    done
+
+    echo "Virtual environment created."
+else
+    source /tools/venv/bin/activate
+fi
+
+cd /project
+export PYTHONPATH=/tools
+$CUTEKIT_PYTHON -m cutekit $@

cutekit/graph.py

@@ -36,7 +36,7 @@ def view(
         if (
             scopeInstance is not None
             and component.id != scope
-            and component.id not in scopeInstance.resolved[target.id].resolved
+            and component.id not in scopeInstance.resolved[target.id].required
         ):
             continue
 
@@ -46,7 +46,7 @@ def view(
 
             g.node(
                 component.id,
-                f"<<B>{component.id}</B><BR/>{vt100.wordwrap(component.decription, 40,newline='<BR/>')}>",
+                f"<<B>{component.id}</B><BR/>{vt100.wordwrap(component.description, 40,newline='<BR/>')}>",
                 shape=shape,
                 style="filled",
                 fillcolor=fillcolor,
@@ -67,7 +67,7 @@ def view(
         elif showDisabled:
             g.node(
                 component.id,
-                f"<<B>{component.id}</B><BR/>{vt100.wordwrap(component.decription, 40,newline='<BR/>')}<BR/><BR/><I>{vt100.wordwrap(str(component.resolved[target.id].reason), 40,newline='<BR/>')}</I>>",
+                f"<<B>{component.id}</B><BR/>{vt100.wordwrap(component.description, 40,newline='<BR/>')}<BR/><BR/><I>{vt100.wordwrap(str(component.resolved[target.id].reason), 40,newline='<BR/>')}</I>>",
                 shape="plaintext",
                 style="filled",
                 fontcolor="#999999",
@@ -83,13 +83,20 @@ def view(
     g.view(filename=os.path.join(target.builddir, "graph.gv"))
 
 
+class GraphCmd(model.TargetArgs):
+    scope = cli.Arg("s", "scope", "Scope to show", default="")
+    onlyLibs = cli.Arg("l", "only-libs", "Only show libraries", default=False)
+    showDisabled = cli.Arg(
+        "d", "show-disabled", "Show disabled components", default=False
+    )
+
+
 @cli.command("g", "graph", "Show the dependency graph")
-def graphCmd(args: cli.Args):
-    registry = model.Registry.use(args)
-    target = model.Target.use(args)
-
-    scope = cast(Optional[str], args.tryConsumeOpt("scope"))
-    onlyLibs = args.consumeOpt("only-libs", False) is True
-    showDisabled = args.consumeOpt("show-disabled", False) is True
-
-    view(registry, target, scope=scope, showExe=not onlyLibs, showDisabled=showDisabled)
+def _(args: GraphCmd):
+    view(
+        model.Registry.use(args),
+        model.Target.use(args),
+        scope=args.scope,
+        showExe=not args.onlyLibs,
+        showDisabled=args.showDisabled,
+    )

cutekit/mixins.py

@@ -57,15 +57,33 @@ def makeMixinTune(tune: str) -> Mixin:
     return mixinTune
 
 
+def combineMixins(*mixins: Mixin) -> Mixin:
+    def combined(target: model.Target, tools: model.Tools) -> model.Tools:
+        for mixin in mixins:
+            tools = mixin(target, tools)
+        return tools
+
+    return combined
+
+
 mixins: dict[str, Mixin] = {
     "cache": mixinCache,
     "debug": mixinDebug,
-    "asan": makeMixinSan("address"),
+    "asan": combineMixins(makeMixinSan("address"), makeMixinSan("leak")),
     "msan": makeMixinSan("memory"),
     "tsan": makeMixinSan("thread"),
     "ubsan": makeMixinSan("undefined"),
+    "lsan": makeMixinSan("leak"),
+    "san": combineMixins(
+        makeMixinSan("address"),
+        makeMixinSan("undefined"),
+        makeMixinSan("leak"),
+    ),
     "tune": makeMixinTune("native"),
-    "fast": makeMixinOptimize("fast"),
+    "release": combineMixins(
+        makeMixinOptimize("3"),
+        makeMixinTune("native"),
+    ),
     "o3": makeMixinOptimize("3"),
     "o2": makeMixinOptimize("2"),
     "o1": makeMixinOptimize("1"),

cutekit/model.py

@@ -8,6 +8,7 @@ from typing import Any, Generator, Optional, Type, cast
 from pathlib import Path
 from dataclasses_json import DataClassJsonMixin
 
+
 from cutekit import const, shell
 
 from . import jexpr, compat, utils, cli, vt100
@@ -71,7 +72,7 @@ class Manifest(DataClassJsonMixin):
         """
         Return the directory of the manifest
         """
-        return os.path.dirname(self.path)
+        return os.path.relpath(os.path.dirname(self.path), Path.cwd())
 
     def subpath(self, path) -> Path:
         return Path(self.dirname()) / path
@@ -96,6 +97,7 @@ _project: Optional["Project"] = None
 class Extern(DataClassJsonMixin):
     git: str
     tag: str
+    deep: bool = dt.field(default=False)
 
 
 @dt.dataclass
@@ -121,19 +123,18 @@ class Project(Manifest):
 
     @staticmethod
     def ensure() -> "Project":
+        """
+        Ensure that a project exists in the current directory or any parent directory
+        and chdir to the root of the project.
+        """
         project = Project.topmost()
         if project is None:
             raise RuntimeError(
                 "No project found in this directory or any parent directory"
             )
+        os.chdir(project.dirname())
         return project
 
-    def chdir(self):
-        """
-        Change the current working directory to the root of the project
-        """
-        os.chdir(self.dirname())
-
     @staticmethod
     def at(path: Path) -> Optional["Project"]:
         projectManifest = Manifest.tryLoad(path / "project")
@@ -151,48 +152,64 @@ class Project(Manifest):
                 continue
 
             print(f"Installing {extSpec}-{ext.tag} from {ext.git}...")
-            shell.popen(
+            cmd = [
                 "git",
                 "clone",
                 "--quiet",
-                "--depth",
-                "1",
                 "--branch",
                 ext.tag,
                 ext.git,
                 extPath,
-            )
+            ]
+
+            if not ext.deep:
+                cmd += ["--depth", "1"]
+
+            shell.exec(*cmd, quiet=True)
             project = Project.at(Path(extPath))
             if project is not None:
                 Project.fetchs(project.extern)
 
     @staticmethod
-    def use(args: cli.Args) -> "Project":
+    def use() -> "Project":
         global _project
         if _project is None:
             _project = Project.ensure()
         return _project
 
 
-@cli.command("i", "install", "Install required external packages")
-def installCmd(args: cli.Args):
-    project = Project.use(args)
+@cli.command("m", "model", "Manage the model")
+def _():
+    pass
+
+
+@cli.command("i", "model/install", "Install required external packages")
+def _():
+    project = Project.use()
     Project.fetchs(project.extern)
 
 
-@cli.command("I", "init", "Initialize a new project")
-def initCmd(args: cli.Args):
+class InitArgs:
+    repo = cli.Arg(
+        "r",
+        "repo",
+        "Repository to use for templates",
+        default=const.DEFAULT_REPO_TEMPLATES,
+    )
+    list = cli.Arg("l", "list", "List available templates", default=False)
+    template = cli.FreeFormArg("template", "Template to use")
+    name = cli.FreeFormArg("name", "Name of the project")
+
+
+@cli.command("I", "model/init", "Initialize a new project")
+def _(args: InitArgs):
     import requests
 
-    repo = args.consumeOpt("repo", const.DEFAULT_REPO_TEMPLATES)
-    list = args.consumeOpt("list")
-
-    template = args.consumeArg()
-    name = args.consumeArg()
-
     _logger.info("Fetching registry...")
 
-    r = requests.get(f"https://raw.githubusercontent.com/{repo}/main/registry.json")
+    r = requests.get(
+        f"https://raw.githubusercontent.com/{args.repo}/main/registry.json"
+    )
 
     if r.status_code != 200:
         _logger.error("Failed to fetch registry")
@@ -200,30 +217,29 @@ def initCmd(args: cli.Args):
 
     registry = r.json()
 
-    if list:
+    if args.list:
         print(
             "\n".join(f"* {entry['id']} - {entry['description']}" for entry in registry)
         )
         return
 
-    if not template:
-        raise RuntimeError("Template not specified")
-
     def template_match(t: jexpr.Json) -> str:
-        return t["id"] == template
+        return t["id"] == args.template
 
     if not any(filter(template_match, registry)):
-        raise LookupError(f"Couldn't find a template named {template}")
+        raise LookupError(f"Couldn't find a template named {args.template}")
 
-    if not name:
-        _logger.info(f"No name was provided, defaulting to {template}")
-        name = template
+    if args.name is None:
+        _logger.info(f"No name was provided, defaulting to {args.template}")
+        name = args.template
+    else:
+        name = args.name
 
     if os.path.exists(name):
         raise RuntimeError(f"Directory {name} already exists")
 
-    print(f"Creating project {name} from template {template}...")
-    shell.cloneDir(f"https://github.com/{repo}", template, name)
+    print(f"Creating project {name} from template {args.template}...")
+    shell.cloneDir(f"https://github.com/{args.repo}", args.template, name)
     print(f"Project {name} created\n")
 
     print("We suggest that you begin by typing:")
@@ -244,6 +260,7 @@ class Tool(DataClassJsonMixin):
     cmd: str = dt.field(default="")
     args: list[str] = dt.field(default_factory=list)
     files: list[str] = dt.field(default_factory=list)
+    rule: Optional[str] = None
 
 
 Tools = dict[str, Tool]
@@ -253,24 +270,44 @@ DEFAULT_TOOLS: Tools = {
 }
 
 
+class RegistryArgs:
+    mixins = cli.Arg[list[str]]("m", "mixins", "Mixins to apply", default=[])
+    # props = cli.Arg[dict[str]]("p", "props", "Properties to set", default="")
+
+
+class TargetArgs(RegistryArgs):
+    target = cli.Arg(
+        "t", "target", "The target to use", default="host-" + shell.uname().machine
+    )
+
+
 @dt.dataclass
 class Target(Manifest):
     props: Props = dt.field(default_factory=dict)
     tools: Tools = dt.field(default_factory=dict)
     routing: dict[str, str] = dt.field(default_factory=dict)
 
+    _hashid: Optional[str] = None
+
     @property
     def hashid(self) -> str:
-        return utils.hash((self.props, [v.to_dict() for k, v in self.tools.items()]))
+        if self._hashid is None:
+            self._hashid = utils.hash(
+                (self.props, [v.to_dict() for k, v in self.tools.items()])
+            )
+        return self._hashid
 
     @property
     def builddir(self) -> str:
-        return os.path.join(const.BUILD_DIR, f"{self.id}-{self.hashid[:8]}")
+        postfix = f"-{self.hashid[:8]}"
+        if self.props.get("host"):
+            postfix += f"-{str(const.HOSTID)[:8]}"
+        return os.path.join(const.BUILD_DIR, f"{self.id}{postfix}")
 
     @staticmethod
-    def use(args: cli.Args) -> "Target":
-        registry = Registry.use(args)
-        targetSpec = str(args.consumeOpt("target", "host-" + shell.uname().machine))
+    def use(args: TargetArgs, props: Props = {}) -> "Target":
+        registry = Registry.use(args, props)
+        targetSpec = args.target
         return registry.ensure(targetSpec, Target)
 
     def route(self, componentSpec: str):
@@ -290,7 +327,8 @@ class Target(Manifest):
 @dt.dataclass
 class Resolved:
     reason: Optional[str] = None
-    resolved: list[str] = dt.field(default_factory=list)
+    required: list[str] = dt.field(default_factory=list)
+    injected: list[str] = dt.field(default_factory=list)
 
     @property
     def enabled(self) -> bool:
@@ -299,7 +337,7 @@ class Resolved:
 
 @dt.dataclass
 class Component(Manifest):
-    decription: str = dt.field(default="(No description)")
+    description: str = dt.field(default="(No description)")
     props: Props = dt.field(default_factory=dict)
     tools: Tools = dt.field(default_factory=dict)
     enableIf: dict[str, list[Any]] = dt.field(default_factory=dict)
@@ -437,11 +475,11 @@ class Resolver:
                 self._cache[keep] = Resolved(reason=reqResolved.reason)
                 return self._cache[keep]
 
-            result.extend(reqResolved.resolved)
+            result.extend(reqResolved.required)
 
         stack.pop()
         result.insert(0, keep)
-        self._cache[keep] = Resolved(resolved=utils.uniq(result))
+        self._cache[keep] = Resolved(required=utils.uniqPreserveOrder(result))
         return self._cache[keep]
 
 
@@ -516,18 +554,25 @@ class Registry(DataClassJsonMixin):
         return m
 
     @staticmethod
-    def use(args: cli.Args) -> "Registry":
+    def use(args: RegistryArgs, props: Props = {}) -> "Registry":
         global _registry
 
         if _registry is not None:
             return _registry
 
-        project = Project.use(args)
-        mixins = str(args.consumeOpt("mixins", "")).split(",")
-        if mixins == [""]:
-            mixins = []
-        props = cast(dict[str, str], args.consumePrefix("prop:"))
+        project = Project.use()
 
+        if not hasattr(args, "mixins"):
+            mixins = []
+        else:
+            if not isinstance(args.mixins, str):
+                raise RuntimeError("Mixins attribute on provided args is not a string")
+            else:
+                mixins = args.mixins.split(",")
+                if mixins == [""]:
+                    mixins = []
+
+        # props |= args.props
         _registry = Registry.load(project, mixins, props)
         return _registry
 
@@ -569,24 +614,33 @@ class Registry(DataClassJsonMixin):
         # Resolve all dependencies for all targets
         for target in r.iter(Target):
             target.props |= props
-            resolver = Resolver(r, target)
-
-            # Apply injects
-            for c in r.iter(Component):
-                if c.isEnabled(target)[0]:
-                    for inject in c.injects:
-                        victim = r.lookup(inject, Component)
-                        if not victim:
-                            raise RuntimeError(f"Cannot find component '{inject}'")
-                        victim.requires += [c.id]
 
             # Resolve all components
+            resolver = Resolver(r, target)
             for c in r.iter(Component):
                 resolved = resolver.resolve(c.id)
                 if resolved.reason:
                     _logger.info(f"Component '{c.id}' disabled: {resolved.reason}")
                 c.resolved[target.id] = resolved
 
+            # Apply injects
+            for c in r.iter(Component):
+                if c.resolved[target.id].enabled:
+                    for inject in c.injects:
+                        victim = r.lookup(inject, Component, includeProvides=True)
+                        if not victim:
+                            _logger.info(
+                                f"Could not find component to inject '{inject}' with '{c.id}'"
+                            )
+                        else:
+                            victim.resolved[target.id].injected.append(c.id)
+                            victim.resolved[
+                                target.id
+                            ].required = utils.uniqPreserveOrder(
+                                c.resolved[target.id].required
+                                + victim.resolved[target.id].required
+                            )
+
             # Resolve tooling
             tools: Tools = target.tools
 
@@ -610,8 +664,8 @@ class Registry(DataClassJsonMixin):
         return r
 
 
-@cli.command("l", "list", "List all components and targets")
-def listCmd(args: cli.Args):
+@cli.command("l", "model/list", "List all components and targets")
+def _(args: RegistryArgs):
     registry = Registry.use(args)
 
     components = list(registry.iter(Component))

cutekit/plugins.py

@@ -1,7 +1,8 @@
-import os
 import logging
+import os
+import sys
 
-from . import shell, model, const
+from . import shell, model, const, cli
 
 import importlib.util as importlib
 
@@ -14,10 +15,16 @@ def load(path: str):
 
     if not spec or not spec.loader:
         _logger.error(f"Failed to load plugin {path}")
-        return None
+        return
 
     module = importlib.module_from_spec(spec)
-    spec.loader.exec_module(module)
+    sys.modules["plugin"] = module
+
+    try:
+        spec.loader.exec_module(module)
+    except Exception as e:
+        _logger.error(f"Failed to load plugin {path}: {e}")
+        cli.warning(f"Plugin {path} loading skipped due to error")
 
 
 def loadAll():
@@ -33,11 +40,21 @@ def loadAll():
 
     for dirname in paths:
         pluginDir = os.path.join(project.dirname(), dirname, const.META_DIR, "plugins")
+        pluginDir = os.path.normpath(pluginDir)
+        initFile = os.path.join(pluginDir, "__init__.py")
 
-        for files in shell.readdir(pluginDir):
-            if files.endswith(".py"):
-                plugin = load(os.path.join(pluginDir, files))
+        if os.path.isfile(initFile):
+            load(initFile)
+        else:
+            for files in shell.readdir(pluginDir):
+                if files.endswith(".py"):
+                    load(os.path.join(pluginDir, files))
 
-                if plugin:
-                    _logger.info(f"Loaded plugin {plugin.name}")
-                    plugin.init()
+
+class PluginArgs:
+    safemode = cli.Arg[bool]("s", "safemode", "Skip loading plugins", default=False)
+
+
+def setup(args: PluginArgs):
+    if not args.safemode:
+        loadAll()

cutekit/pods.py

@@ -0,0 +1,246 @@
+import sys
+from typing import Optional
+import docker  # type: ignore
+import os
+import dataclasses as dt
+
+from . import cli, model, shell, vt100, const
+
+
+podPrefix = "CK__"
+projectRoot = "/project"
+toolingRoot = "/tools"
+defaultPodName = f"{podPrefix}default"
+defaultPodImage = "ubuntu"
+
+
+@dt.dataclass
+class Image:
+    id: str
+    like: str
+    image: str
+    setup: list[str]
+
+
+@dt.dataclass
+class Pod:
+    name: str
+    image: Image
+
+
+IMAGES: dict[str, Image] = {
+    "ubuntu": Image(
+        "ubuntu",
+        "ubuntu",
+        "ubuntu:jammy",
+        [
+            "apt-get update",
+            "apt-get install -y python3.11 python3.11-venv ninja-build build-essential",
+        ],
+    ),
+    "debian": Image(
+        "debian",
+        "debian",
+        "debian:bookworm",
+        [
+            "apt-get update",
+            "apt-get install -y python3 python3-pip python3-venv ninja-build build-essential",
+        ],
+    ),
+    "alpine": Image(
+        "alpine",
+        "alpine",
+        "alpine:3.18",
+        [
+            "apk update",
+            "apk add python3 python3-dev py3-pip py3-virtualenv build-base linux-headers ninja make automake gcc g++ bash",
+        ],
+    ),
+    "arch": Image(
+        "arch",
+        "arch",
+        "archlinux:latest",
+        [
+            "pacman -Syu --noconfirm",
+            "pacman -S --noconfirm python python-pip python-virtualenv ninja base-devel",
+        ],
+    ),
+    "fedora": Image(
+        "fedora",
+        "fedora",
+        "fedora:39",
+        [
+            "dnf update -y",
+            "dnf install -y python3 python3-pip python3-venv ninja-build make automake gcc gcc-c++ kernel-devel",
+        ],
+    ),
+}
+
+
+class PodArgs:
+    pod = cli.Arg[bool]("p", "enable-pod", "Enable pod", default=False)
+    podName = cli.Arg[str]("n", "pod-name", "The name of the pod", default="")
+
+
+def setup(args: PodArgs, argv: list[str]):
+    """
+    Reincarnate cutekit within a docker container, this is
+    useful for cross-compiling
+    """
+    if not args.pod:
+        return
+
+    pod = args.podName.strip() or defaultPodName
+    pod = podPrefix + args.podName
+
+    model.Project.ensure()
+    print(f"Reincarnating into pod '{pod[len(podPrefix) :]}'...")
+    try:
+        shell.exec(
+            "docker",
+            "exec",
+            "-w",
+            projectRoot,
+            "-it",
+            pod,
+            "/tools/cutekit/entrypoint.sh",
+            "--reincarnated",
+            *argv,
+        )
+        sys.exit(0)
+    except Exception:
+        sys.exit(1)
+
+
+@cli.command("p", "pod", "Manage pods")
+def _():
+    pass
+
+
+def tryDecode(data: Optional[bytes], default: str = "") -> str:
+    if data is None:
+        return default
+    return data.decode()
+
+
+class PodCreateArgs:
+    name: cli.Arg[str] = cli.Arg(
+        "n", "name", "The name of the pod to use", default="default"
+    )
+    image = cli.Arg[str]("i", "image", "The image to use", default=defaultPodImage)
+
+
+@cli.command("c", "pod/create", "Create a new pod")
+def _(args: PodCreateArgs):
+    """
+    Create a new development pod with cutekit installed and the current
+    project mounted at /project
+    """
+    project = model.Project.ensure()
+
+    if not args.name.startswith(podPrefix):
+        name: str = f"{podPrefix}{args.name}"
+    else:
+        name = args.name
+
+    image = IMAGES[args.image]
+
+    client = docker.from_env()
+    try:
+        existing = client.containers.get(name)
+        if cli.ask(f"Pod '{name[len(podPrefix):]}' already exists, kill it?", False):
+            existing.stop()
+            existing.remove()
+        else:
+            raise RuntimeError(f"Pod '{name[len(podPrefix):]}' already exists")
+    except docker.errors.NotFound:
+        pass
+
+    print(f"Staring pod '{name[len(podPrefix) :]}'...")
+
+    container = client.containers.run(
+        image.image,
+        "sleep infinity",
+        name=name,
+        volumes={
+            const.MODULE_DIR: {
+                "bind": toolingRoot + "/cutekit",
+                "mode": "ro",
+            },
+            os.path.abspath(project.dirname()): {"bind": projectRoot, "mode": "rw"},
+        },
+        detach=True,
+    )
+
+    print(f"Initializing pod '{name[len(podPrefix) :]}'...")
+    for cmd in image.setup:
+        print(vt100.p(cmd))
+        exitCode, output = container.exec_run(f"/bin/sh -c '{cmd}'", demux=True)
+        if exitCode != 0:
+            raise RuntimeError(
+                f"Failed to initialize pod with command '{cmd}':\n\nSTDOUT:\n{vt100.indent(vt100.wordwrap(tryDecode(output[0], '<empty>')))}\nSTDERR:\n{vt100.indent(vt100.wordwrap(tryDecode(output[1], '<empty>')))}"
+            )
+
+    print(f"Created pod '{name[len(podPrefix) :]}' from image '{image.image}'")
+
+
+class KillPodArgs:
+    name: cli.Arg[str] = cli.Arg(
+        "n", "name", "The name of the pod to kill", default=defaultPodName
+    )
+
+
+@cli.command("k", "pod/kill", "Stop and remove a pod")
+def _(args: KillPodArgs):
+    client = docker.from_env()
+    if not args.name.startswith(podPrefix):
+        name: str = f"{podPrefix}{args.name}"
+    else:
+        name = args.name
+
+    try:
+        container = client.containers.get(name)
+        container.stop()
+        container.remove()
+        print(f"Pod '{name[len(podPrefix) :]}' killed")
+    except docker.errors.NotFound:
+        raise RuntimeError(f"Pod '{name[len(podPrefix):]}' does not exist")
+
+
+@cli.command("l", "pod/list", "List all pods")
+def _():
+    client = docker.from_env()
+    hasPods = False
+    for container in client.containers.list(all=True):
+        if not container.name.startswith(podPrefix):
+            continue
+        print(container.name[len(podPrefix) :], container.status)
+        hasPods = True
+
+    if not hasPods:
+        print(vt100.p("(No pod found)"))
+
+
+class PodExecArgs:
+    name = cli.Arg("n", "name", "The name of the pod to use", default=defaultPodName)
+    cmd = cli.FreeFormArg("cmd", "The command to execute", default="/bin/bash")
+    args = cli.FreeFormArg("args", "The arguments to pass to the command")
+
+
+@cli.command("e", "pod/exec", "Execute a command in a pod")
+def podExecCmd(args: PodExecArgs):
+    if not args.name.startswith(podPrefix):
+        name: str = f"{podPrefix}{args.name}"
+    else:
+        name = args.name
+
+    try:
+        shell.exec("docker", "exec", "-it", name, args.cmd, *args.args)
+    except Exception:
+        raise RuntimeError(f"Pod '{name[len(podPrefix):]}' does not exist")
+
+
+@cli.command("s", "pod/shell", "Open a shell in a pod")
+def _(args: PodExecArgs):
+    args.cmd = "/bin/bash"
+    podExecCmd(args)

cutekit/requirements.txt

@@ -1,3 +1,4 @@
 requests ~= 2.31.0
 graphviz ~= 0.20.1
 dataclasses-json ~= 0.6.2
+docker ~= 6.1.3

cutekit/rules.py

@@ -1,52 +1,58 @@
+import dataclasses as dt
+
 from typing import Optional
 
 
+@dt.dataclass
 class Rule:
     id: str
     fileIn: list[str]
-    fileOut: list[str]
+    fileOut: str
     rule: str
-    args: list[str]
-    deps: Optional[str] = None
-
-    def __init__(
-        self,
-        id: str,
-        fileIn: list[str],
-        fileOut: list[str],
-        rule: str,
-        args: list[str] = [],
-        deps: Optional[str] = None,
-    ):
-        self.id = id
-        self.fileIn = fileIn
-        self.fileOut = fileOut
-        self.rule = rule
-        self.args = args
-        self.deps = deps
+    args: list[str] = dt.field(default_factory=list)
+    deps: list[str] = dt.field(default_factory=list)
 
 
 rules: dict[str, Rule] = {
-    "cp": Rule("cp", ["*"], ["*"], "$in $out"),
+    "cp": Rule("cp", ["*"], "*", "$in $out"),
     "cc": Rule(
         "cc",
         ["*.c"],
-        ["*.o"],
+        "*.o",
         "-c -o $out $in -MD -MF $out.d $flags $cincs $cdefs",
-        ["-std=gnu2x", "-Wall", "-Wextra", "-Werror"],
-        "$out.d",
+        [
+            "-std=gnu2x",
+            "-Wall",
+            "-Wextra",
+            "-Werror",
+            "-fcolor-diagnostics",
+        ],
+        ["$out.d"],
     ),
     "cxx": Rule(
         "cxx",
         ["*.cpp", "*.cc", "*.cxx"],
-        ["*.o"],
+        "*.o",
         "-c -o $out $in -MD -MF $out.d $flags $cincs $cdefs",
-        ["-std=gnu++2b", "-Wall", "-Wextra", "-Werror", "-fno-exceptions", "-fno-rtti"],
-        "$out.d",
+        [
+            "-std=gnu++2b",
+            "-Wall",
+            "-Wextra",
+            "-Werror",
+            "-fcolor-diagnostics",
+            "-fno-exceptions",
+            "-fno-rtti",
+        ],
+        ["$out.d"],
+    ),
+    "as": Rule("as", ["*.s", "*.asm", "*.S"], "*.o", "-o $out $in $flags"),
+    "ar": Rule("ar", ["*.o"], "*.a", "$flags $out $in"),
+    "ld": Rule(
+        "ld",
+        ["*.o", "*.a"],
+        "*.out",
+        "-o $out $objs --whole-archive $wholeLibs --no-whole-archive $libs $flags",
     ),
-    "as": Rule("as", ["*.s", "*.asm", "*.S"], ["*.o"], "-o $out $in $flags"),
-    "ar": Rule("ar", ["*.o"], ["*.a"], "$flags $out $in"),
-    "ld": Rule("ld", ["*.o", "*.a"], ["*.out"], "-o $out $in $flags"),
 }
 
 

cutekit/shell.py

@@ -10,23 +10,22 @@ import fnmatch
 import platform
 import logging
 import tempfile
+import dataclasses as dt
 
-
+from pathlib import Path
 from typing import Optional
-from . import const
+from . import const, cli
 
 _logger = logging.getLogger(__name__)
 
 
+@dt.dataclass
 class Uname:
-    def __init__(
-        self, sysname: str, nodename: str, release: str, version: str, machine: str
-    ):
-        self.sysname = sysname
-        self.nodename = nodename
-        self.release = release
-        self.version = version
-        self.machine = machine
+    sysname: str
+    nodename: str
+    release: str
+    version: str
+    machine: str
 
 
 def uname() -> Uname:
@@ -52,7 +51,7 @@ def sha256sum(path: str) -> str:
 def find(
     path: str | list[str], wildcards: list[str] = [], recusive: bool = True
 ) -> list[str]:
-    _logger.info(f"Looking for files in {path} matching {wildcards}")
+    _logger.debug(f"Looking for files in {path} matching {wildcards}")
 
     result: list[str] = []
 
@@ -88,7 +87,7 @@ def find(
 
 
 def mkdir(path: str) -> str:
-    _logger.info(f"Creating directory {path}")
+    _logger.debug(f"Creating directory {path}")
 
     try:
         os.makedirs(path)
@@ -99,11 +98,14 @@ def mkdir(path: str) -> str:
 
 
 def rmrf(path: str) -> bool:
-    _logger.info(f"Removing directory {path}")
+    _logger.debug(f"Removing directory {path}")
 
     if not os.path.exists(path):
         return False
-    shutil.rmtree(path, ignore_errors=True)
+    if os.path.isfile(path):
+        os.remove(path)
+    else:
+        shutil.rmtree(path, ignore_errors=True)
     return True
 
 
@@ -118,7 +120,7 @@ def wget(url: str, path: Optional[str] = None) -> str:
     if os.path.exists(path):
         return path
 
-    _logger.info(f"Downloading {url} to {path}")
+    _logger.debug(f"Downloading {url} to {path}")
 
     r = requests.get(url, stream=True)
     r.raise_for_status()
@@ -131,56 +133,113 @@ def wget(url: str, path: Optional[str] = None) -> str:
     return path
 
 
-def exec(*args: str, quiet: bool = False) -> bool:
-    _logger.info(f"Executing {args}")
+def exec(*args: str, quiet: bool = False, cwd: Optional[str] = None) -> bool:
+    _logger.debug(f"Executing {args}")
+    cmdName = Path(args[0]).name
 
     try:
         proc = subprocess.run(
             args,
+            cwd=cwd,
             stdout=sys.stdout if not quiet else subprocess.PIPE,
             stderr=sys.stderr if not quiet else subprocess.PIPE,
         )
 
         if proc.stdout:
-            _logger.info(proc.stdout.decode("utf-8"))
+            _logger.debug(proc.stdout.decode("utf-8"))
 
         if proc.stderr:
-            _logger.error(proc.stderr.decode("utf-8"))
+            _logger.debug(proc.stderr.decode("utf-8"))
 
     except FileNotFoundError:
-        raise RuntimeError(f"{args[0]}: Command not found")
+        if cwd and not os.path.exists(cwd):
+            raise RuntimeError(f"{cwd}: No such file or directory")
+        else:
+            raise RuntimeError(f"{args[0]}: Command not found")
 
     except KeyboardInterrupt:
-        raise RuntimeError(f"{args[0]}: Interrupted")
+        raise RuntimeError(f"{cmdName}: Interrupted")
 
     if proc.returncode == -signal.SIGSEGV:
-        raise RuntimeError(f"{args[0]}: Segmentation fault")
+        raise RuntimeError(f"{cmdName}: Segmentation fault")
 
     if proc.returncode != 0:
-        raise RuntimeError(f"{args[0]}: Process exited with code {proc.returncode}")
+        raise RuntimeError(f"{cmdName}: Process exited with code {proc.returncode}")
 
     return True
 
 
 def popen(*args: str) -> str:
-    _logger.info(f"Executing {args}")
+    _logger.debug(f"Executing {args}...")
+
+    cmdName = Path(args[0]).name
 
     try:
         proc = subprocess.run(args, stdout=subprocess.PIPE, stderr=sys.stderr)
     except FileNotFoundError:
-        raise RuntimeError(f"{args[0]}: Command not found")
+        raise RuntimeError(f"{cmdName}: Command not found")
 
     if proc.returncode == -signal.SIGSEGV:
-        raise RuntimeError(f"{args[0]}: Segmentation fault")
+        raise RuntimeError(f"{cmdName}: Segmentation fault")
 
     if proc.returncode != 0:
-        raise RuntimeError(f"{args[0]}: Process exited with code {proc.returncode}")
+        raise RuntimeError(f"{cmdName}: Process exited with code {proc.returncode}")
 
-    return proc.stdout.decode("utf-8")
+    return proc.stdout.decode("utf-8").strip()
+
+
+def debug(cmd: list[str], debugger: str = "lldb", wait: bool = False):
+    if debugger == "lldb":
+        exec(
+            "lldb",
+            *(("-o", "b main") if wait else ()),
+            *("-o", "run"),
+            *cmd,
+        )
+    elif debugger == "gdb":
+        exec(
+            "gdb",
+            *(("-ex", "b main") if wait else ()),
+            *("-ex", "run"),
+            *cmd,
+        )
+    else:
+        raise RuntimeError(f"Unknown debugger {debugger}")
+
+
+def profile(cmd: list[str]):
+    mkdir(const.TMP_DIR)
+    perfFile = f"{const.TMP_DIR}/perf.data"
+    try:
+        exec(
+            "perf",
+            "record",
+            "-g",
+            "-o",
+            perfFile,
+            "--call-graph",
+            "dwarf",
+            *cmd,
+        )
+    except Exception as e:
+        if not os.path.exists(perfFile):
+            raise e
+
+    try:
+        proc = subprocess.Popen(
+            ["perf", "script", "-i", perfFile], stdout=subprocess.PIPE
+        )
+        subprocess.run(["speedscope", "-"], stdin=proc.stdout)
+        proc.wait()
+    except Exception as e:
+        rmrf(perfFile)
+        raise e
+
+    rmrf(perfFile)
 
 
 def readdir(path: str) -> list[str]:
-    _logger.info(f"Reading directory {path}")
+    _logger.debug(f"Reading directory {path}")
 
     try:
         return os.listdir(path)
@@ -189,19 +248,19 @@ def readdir(path: str) -> list[str]:
 
 
 def cp(src: str, dst: str):
-    _logger.info(f"Copying {src} to {dst}")
+    _logger.debug(f"Copying {src} to {dst}")
 
     shutil.copy(src, dst)
 
 
 def mv(src: str, dst: str):
-    _logger.info(f"Moving {src} to {dst}")
+    _logger.debug(f"Moving {src} to {dst}")
 
     shutil.move(src, dst)
 
 
 def cpTree(src: str, dst: str):
-    _logger.info(f"Copying {src} to {dst}")
+    _logger.debug(f"Copying {src} to {dst}")
 
     shutil.copytree(src, dst, dirs_exist_ok=True)
 
@@ -241,10 +300,9 @@ def latest(cmd: str) -> str:
     if cmd in LATEST_CACHE:
         return LATEST_CACHE[cmd]
 
-    _logger.info(f"Finding latest version of {cmd}")
+    _logger.debug(f"Finding latest version of {cmd}")
 
     regex: re.Pattern[str]
-
     if platform.system() == "Windows":
         regex = re.compile(r"^" + re.escape(cmd) + r"(-.[0-9]+)?(\.exe)?$")
     else:
@@ -263,7 +321,7 @@ def latest(cmd: str) -> str:
     versions.sort()
     chosen = versions[-1]
 
-    _logger.info(f"Chosen {chosen} as latest version of {cmd}")
+    _logger.debug(f"Chosen {chosen} as latest version of {cmd}")
 
     LATEST_CACHE[cmd] = chosen
 
@@ -275,3 +333,113 @@ def which(cmd: str) -> Optional[str]:
     Find the path of a command
     """
     return shutil.which(cmd)
+
+
+def nproc() -> int:
+    """
+    Return the number of processors
+    """
+    return os.cpu_count() or 1
+
+
+def gzip(path: str, dest: Optional[str] = None) -> str:
+    """
+    Compress a file or directory
+    """
+
+    if dest is None:
+        dest = path + ".gz"
+
+    with open(dest, "wb") as f:
+        proc = subprocess.run(
+            ["gzip", "-c", path],
+            stdout=f,
+            stderr=sys.stderr,
+        )
+
+    if proc.returncode != 0:
+        raise RuntimeError(f"gzip: Process exited with code {proc.returncode}")
+
+    return dest
+
+
+def compress(path: str, dest: Optional[str] = None, format: str = "zstd") -> str:
+    """
+    Compress a file or directory
+    """
+
+    EXTS = {
+        "zip": "zip",
+        "zstd": "zst",
+        "gzip": "gz",
+    }
+
+    if dest is None:
+        dest = path + "." + EXTS[format]
+
+    _logger.debug(f"Compressing {path} to {dest}")
+
+    if format == "zip":
+        exec("zip", "-r", dest, path)
+    elif format == "zstd":
+        exec("zstd", "-q", "-o", dest, path)
+    elif format == "gzip":
+        gzip(path, dest)
+    else:
+        raise RuntimeError(f"Unknown compression format {format}")
+
+    return dest
+
+
+# --- Commands --------------------------------------------------------------- #
+
+
+@cli.command("s", "scripts", "Manage scripts")
+def _():
+    pass
+
+
+class DebuggerArgs:
+    debugger = cli.Arg[str](
+        "d", "debugger", "Debugger to use (lldb, gdb)", default="lldb"
+    )
+    wait = cli.Arg[bool]("w", "wait", "Wait for debugger to attach")
+
+
+class DebugArgs(DebuggerArgs):
+    cmd = cli.FreeFormArg[str]("cmd", "Command to debug", default="a.out")
+    args = cli.RawArg("args", "Arguments to pass to the command")
+
+
+@cli.command("d", "debug", "Debug a program")
+def _(args: DebugArgs):
+    command = [args.cmd, *args.args]
+    debug(command, args.debugger, args.wait)
+
+
+class ProfilerArgs:
+    pass
+
+
+class ProfileArgs:
+    cmd = cli.FreeFormArg[str]("cmd", "Command to profile", default="a.out")
+    extra: cli.RawArg
+
+
+@cli.command("p", "profile", "Profile a program")
+def _(args: ProfileArgs):
+    command = [args.cmd, *args.extra]
+    profile(command)
+
+
+class CompressArgs:
+    dest = cli.Arg[str]("d", "dest", "Destination file")
+    format = cli.Arg[str](
+        "f", "format", "Compression format (zip, zstd, gzip)", default="zstd"
+    )
+    path = cli.FreeFormArg[str]("path", "Path to compress")
+
+
+@cli.command("c", "compress", "Compress a file or directory")
+def _(args: CompressArgs):
+    compress(args.path, args.dest, args.format)

cutekit/utils.py

@@ -6,7 +6,7 @@ import hashlib
 T = TypeVar("T")
 
 
-def uniq(lst: list[T]) -> list[T]:
+def uniqPreserveOrder(lst: list[T]) -> list[T]:
     result: list[T] = []
     for i in lst:
         if i in result:
@@ -29,6 +29,10 @@ def hash(
     return hashlib.sha256(data.encode("utf-8")).hexdigest()
 
 
+def randomHash() -> str:
+    return hashlib.sha256(os.urandom(32)).hexdigest()
+
+
 def camelCase(s: str) -> str:
     s = "".join(x for x in s.title() if x != "_" and x != "-")
     s = s[0].lower() + s[1:]

doc/cutekit.md

@@ -1,15 +1,15 @@
-# Cutekit 
+# Cutekit
 
 Cutekit is a build system that aims to be simple, fast and easy to use.
 A project is described using json files.
 
-## Project file 
+## Project file
 
 The project file is used to describe the project and its dependencies.
 
 See: [doc/spec/project.md](doc/spec/project.md) for the full specification.
 
-Example: 
+Example:
 
 > project.json
 ```json
@@ -35,7 +35,7 @@ Here we describe a project with the id `skift-org/skift` and a dependency to `cu
 When you want to create an executable package, you need to create a `manifest.json` file in any directory under `src/`.
 This is the file that describe an executable with its dependencies.
 
-> src/nyan-cat-app/manifest.json   
+> src/nyan-cat-app/manifest.json
 ```json
 {
     "$schema": "https://schemas.cute.engineering/stable/cutekit.manifest.component.v1",
@@ -56,7 +56,7 @@ You can run the executable by running `$ ck run nyan-cat-app`.
 
 When you want to create a library package, you need to create a `manifest.json` file in any directory under `src/`, like an executable package.
 
-> src/easy-lib/manifest.json   
+> src/easy-lib/manifest.json
 ```json
 {
     "$schema": "https://schemas.cute.engineering/stable/cutekit.manifest.component.v1",

doc/extends.md

@@ -11,8 +11,6 @@ For example you can add a new command to the CLI:
 from cutekit import cli
 
 @cli.command("h", "hello", "Print hello world")
-def bootCmd(args: cli.Args) -> None:
+def _(args: cli.Args) -> None:
     print("Hello world!")
 ```
-
-This feature is used - for example - by [SkiftOS](https://github.com/skift-org/skift/blob/main/meta/plugins/start-cmd.py) to add the `start` command, that build packages and run a virtual machine.

pyproject.toml

@@ -12,7 +12,7 @@ authors = [
     { name = "Cute Engineering", email = "contact@cute.engineering" },
 ]
 readme = "README.md"
-requires-python = ">=3.10"
+requires-python = ">=3.11"
 license = { text = "MIT" }
 dynamic = ["version", "dependencies"]
 
@@ -26,7 +26,7 @@ packages = ["cutekit"]
 
 [tool.setuptools.dynamic]
 version = { attr = "cutekit.const.VERSION" }
-dependencies = { file = ["requirements.txt"] }
+dependencies = { file = ["cutekit/requirements.txt"] }
 
 [tool.setuptools.package-data]
-"cutekit" = ["py.typed"]
+"cutekit" = ["py.typed", "requirements.txt", "entrypoint.sh", "bootstrap.sh"]

tests/test_resolver.py

@@ -10,7 +10,7 @@ def test_direct_deps():
 
     resolved = res.resolve("myapp")
     assert resolved.reason is None
-    assert resolved.resolved == ["myapp", "mylib"]
+    assert resolved.required == ["myapp", "mylib"]
 
 
 def test_indirect_deps():
@@ -20,7 +20,7 @@ def test_indirect_deps():
     r._append(model.Component("myimpl", provides=["myembed"]))
     t = model.Target("host")
     res = model.Resolver(r, t)
-    assert res.resolve("myapp").resolved == ["myapp", "mylib", "myimpl"]
+    assert res.resolve("myapp").required == ["myapp", "mylib", "myimpl"]
 
 
 def test_deps_routing():
@@ -31,11 +31,11 @@ def test_deps_routing():
     r._append(model.Component("myimplB", provides=["myembed"]))
     t = model.Target("host", routing={"myembed": "myimplB"})
     res = model.Resolver(r, t)
-    assert res.resolve("myapp").resolved == ["myapp", "mylib", "myimplB"]
+    assert res.resolve("myapp").required == ["myapp", "mylib", "myimplB"]
 
     t = model.Target("host", routing={"myembed": "myimplA"})
     res = model.Resolver(r, t)
-    assert res.resolve("myapp").resolved == ["myapp", "mylib", "myimplA"]
+    assert res.resolve("myapp").required == ["myapp", "mylib", "myimplA"]
 
     t = model.Target("host", routing={"myembed": "myimplC"})
     res = model.Resolver(r, t)
@@ -54,11 +54,11 @@ def test_deps_routing_with_props():
     )
     t = model.Target("host", routing={"myembed": "myimplB"}, props={"myprop": "b"})
     res = model.Resolver(r, t)
-    assert res.resolve("myapp").resolved == ["myapp", "mylib", "myimplB"]
+    assert res.resolve("myapp").required == ["myapp", "mylib", "myimplB"]
 
     t = model.Target("host", routing={"myembed": "myimplA"}, props={"myprop": "a"})
     res = model.Resolver(r, t)
-    assert res.resolve("myapp").resolved == ["myapp", "mylib", "myimplA"]
+    assert res.resolve("myapp").required == ["myapp", "mylib", "myimplA"]
 
     t = model.Target("host", routing={"myembed": "myimplC"}, props={"myprop": "c"})
     res = model.Resolver(r, t)
@@ -79,11 +79,11 @@ def test_deps_routing_with_props_and_requires():
     )
     t = model.Target("host", routing={"myembed": "myimplB"}, props={"myprop": "b"})
     res = model.Resolver(r, t)
-    assert res.resolve("myapp").resolved == ["myapp", "mylib", "myimplB"]
+    assert res.resolve("myapp").required == ["myapp", "mylib", "myimplB"]
 
     t = model.Target("host", routing={"myembed": "myimplA"}, props={"myprop": "a"})
     res = model.Resolver(r, t)
-    assert res.resolve("myapp").resolved == ["myapp", "mylib", "myimplA"]
+    assert res.resolve("myapp").required == ["myapp", "mylib", "myimplA"]
 
     t = model.Target("host", routing={"myembed": "myimplC"}, props={"myprop": "c"})
     res = model.Resolver(r, t)