MetaCall Runtime Environment Profile (MREP)

[k5602]

Status

RFC 0001

Drafted "v0.0.2"

Related PRs:
#734

Abstract

This RFC proposes a standardized mechanism for defining and propagating runtime environment configuration across MetaCall loaders and ports. The mechanism introduces a declarative environment profile format that centralizes runtime home paths, search paths, and initialization requirements, replacing the current pattern of duplicative platform-specific code in each loader and port implementation.

Motivation

The Problem

The current MetaCall architecture requires each loader and port to implement its own platform-specific initialization logic. Consider the changes introduced in PR #734 to fix Windows Python initialization and mainly it's a repeated problem in CIs and Infrastructure:

1. py_loader adds py_loader_impl_initialize_python_home() to set PYTHONHOME before Py_InitializeEx()
2. rs_port adds ensure_python_home() as a fallback for backward compatibility

This pattern creates several issues:

• Code duplication: Each loader must reimplement the same initialization patterns for its target runtime
• Platform-specific pollution: #if defined(WIN32) blocks scattered across the codebase
• Maintenance burden: When runtime initialization logic changes, it must be updated in multiple places
• Inconsistent behavior: Different loaders may handle the same initialization scenarios differently

The Opportunity

Other polyglot runtimes and version managers have solved similar problems through declarative profile systems:

• SDKMAN uses ~/.sdkman/etc/config and ~/.sdkman/candidates/ layout
• Conda uses environment.yml with explicit channel and dependency declarations
• GraalVM uses META-INF/native-image/ with structured metadata
• asdf uses .tool-versions for project-level runtime specification

MetaCall already has a configuration system (loader.json.in, global.json). The opportunity is to extend this with a dedicated runtime environment profile that:

1. Declares runtime requirements per language
2. Provides platform-aware path resolution
3. Enables automatic environment variable propagation
4. Supports both system-wide and per-project profiles

Delimma

A developer installs MetaCall on a fresh Linux system. They want to run Python and Node scripts that call each other. With the current system, they must:

1. Manually set LOADER_LIBRARY_PATH
2. Manually set SERIAL_LIBRARY_PATH
3. Manually configure PYTHONHOME and NODE_PATH
4. Ensure these variables are set consistently across all shell sessions

With the proposed system, they run:

source /etc/profile.d/metacall.sh

And the profile script automatically sets all necessary variables based on the installed runtime configuration.

Specification

1. Terminology

Term	Definition
MREP	MetaCall Runtime Environment Profile, the declarative configuration format
Runtime Profile	A JSON file containing MREP declarations
Runtime Home	The base directory containing a language runtime installation
Profile Directory	The directory containing runtime profiles and associated metadata
Port	A language binding that loads the MetaCall runtime (Rust, Go, Java, C#, etc.)
Loader	A plugin that embeds a language runtime (Python, Node, Ruby, etc.)

2. Profile Format

The MREP format uses JSON with the following schema:

{
  "$schema": "https://metacall.io/schemas/mrep-v1.json",
  "meta": {
    "version": "1.0.0",
    "schema": "https://metacall.io/schemas/mrep-v1.json",
    "generated": "2024-01-15T10:30:00Z"
  },
  "runtimes": {
    "<language>": {
      "home": "<absolute-path-or-variable-reference>",
      "search_paths": ["<path>", "<path>"],
      "environment": {
        "<VAR_NAME>": "<value>"
      },
      "options": {
        "<option>": "<value>"
      }
    }
  },
  "defaults": {
    "loaders_path": "<path>",
    "scripts_path": "<path>",
    "config_path": "<path>"
  }
}

2.1 Field Definitions

Field	Type	Description
meta.version	string	MREP format version (semver)
meta.schema	string	JSON Schema URI for validation
meta.generated	string	ISO 8601 timestamp of profile generation
runtimes.<language>.home	string	Runtime home directory or variable reference
runtimes.<language>.search_paths	array	Platform-specific library search paths
runtimes.<language>.environment	object	Environment variables to set before initialization
runtimes.<language>.options	object	Loader-specific configuration options
defaults.loaders_path	string	Default directory for loader plugins
defaults.scripts_path	string	Default directory for scripts
defaults.config_path	string	Default path to global configuration

2.2 Variable References

The home field supports variable references for flexible path resolution:

Reference	Expansion	Example
$METACALL_PREFIX	Installation prefix	/usr/local
$METACALL_HOME	Profile directory parent	/etc/metacall
$ORIGIN	Loader binary directory	Platform-specific
<os-family>	Platform-specific path	Windows: %LOCALAPPDATA%\MetaCall

2.3 Example Profiles

Linux System Installation:

{
  "meta": {
    "version": "1.0.0",
    "schema": "https://metacall.io/schemas/mrep-v1.json",
    "generated": "2024-01-15T10:30:00Z"
  },
  "runtimes": {
    "python": {
      "home": "$METACALL_PREFIX/lib/metacall/runtimes/python",
      "search_paths": [
        "$METACALL_PREFIX/lib",
        "$METACALL_PREFIX/lib/metacall/runtimes/python/lib"
      ],
      "environment": {
        "PYTHONIOENCODING": "utf-8",
        "PYTHONDONTWRITEBYTECODE": "1"
      }
    },
    "ruby": {
      "home": "$METACALL_PREFIX/lib/metacall/runtimes/ruby",
      "search_paths": [
        "$METACALL_PREFIX/lib",
        "$METACALL_PREFIX/lib/metacall/runtimes/ruby/lib"
      ]
    },
    "node": {
      "home": "$METACALL_PREFIX/lib/metacall/runtimes/node",
      "search_paths": [
        "$METACALL_PREFIX/lib"
      ]
    }
  },
  "defaults": {
    "loaders_path": "$METACALL_PREFIX/lib/metacall/loaders",
    "scripts_path": "$METACALL_PREFIX/share/metacall/scripts",
    "config_path": "$METACALL_PREFIX/etc/metacall/global.json"
  }
}

Windows Installation:

{
  "meta": {
    "version": "1.0.0",
    "schema": "https://metacall.io/schemas/mrep-v1.json",
    "generated": "2024-01-15T10:30:00Z"
  },
  "runtimes": {
    "python": {
      "home": "%LOCALAPPDATA%\\MetaCall\\runtimes\\python",
      "search_paths": [
        "%LOCALAPPDATA%\\MetaCall\\lib",
        "%LOCALAPPDATA%\\MetaCall\\runtimes\\python"
      ],
      "environment": {
        "PYTHONHOME": "%LOCALAPPDATA%\\MetaCall\\runtimes\\python",
        "PYTHONIOENCODING": "utf-8"
      }
    }
  },
  "defaults": {
    "loaders_path": "%LOCALAPPDATA%\\MetaCall\\loaders",
    "scripts_path": "%LOCALAPPDATA%\\MetaCall\\scripts",
    "config_path": "%LOCALAPPDATA%\\MetaCall\\config\\global.json"
  }
}

2.4 Python Path Matrix

The table below captures the current Python-related paths used by MetaCall as requested.

Install method	MetaCall library discovery (Python port)	Python runtime root	Python module path / PYTHONPATH	MetaCall configuration path	Loader paths / env profile
Linux distributable tarball (Guix-based)	/gnu/lib/libmetacall(.so) (also searched by Python/Node/Ruby ports via /gnu/lib)	Guix Python runtime in /gnu/store/..., exposed through /gnu profile	PYTHONPATH=/gnu/lib/python<major.minor>/site-packages (from /gnu/etc/profile); installer wrapper also appends lib-dynload	CONFIGURATION_PATH=/gnu/configurations/global.json	LOADER_LIBRARY_PATH=/gnu/lib, SERIAL_LIBRARY_PATH=/gnu/lib, DETOUR_LIBRARY_PATH=/gnu/lib, PORT_LIBRARY_PATH=/gnu/lib
macOS Homebrew formula (metacall/homebrew)	/opt/homebrew/lib/libmetacall(.dylib) (arm64) or /usr/local/lib/libmetacall(.dylib) (intel)	Homebrew Python formula (python@3.x) via python.opt_* (no fixed PYTHONHOME)	Wrapper exports PYTHONPATH=${PREFIX}/lib/python:${PYTHONPATH:-}; formula tests also use ${HOMEBREW_PREFIX}/lib/python	Wrapper exports CONFIGURATION_PATH=${PREFIX}/configurations/global.json	Wrapper exports SERIAL_LIBRARY_PATH, DETOUR_LIBRARY_PATH, PORT_LIBRARY_PATH to ${PREFIX}/lib (note: current formula uses LOADER_LIBRARY, not LOADER_LIBRARY_PATH)
Linux DEB/RPM package from core releases (apt/dpkg)	Default package build currently targets /usr/local/lib/libmetacall(.so)	System Python from package dependencies (not a bundled runtime folder)	No package-managed PYTHONPATH export by default; relies on system Python layout	Build/install default resolves to /usr/local/share/metacall/configurations/global.json (or configured prefix equivalent)	Compile-time defaults point loader/serial/detour/port install paths to <prefix>/lib
Windows distributable installer script (install.ps1 + zip asset)	Port default search root is %LOCALAPPDATA%\MetaCall\metacall and recursively finds metacall(.dll) (currently stored under ...\lib)	%LOCALAPPDATA%\MetaCall\metacall\runtimes\python (PYTHONHOME)	PIP_TARGET=%LOCALAPPDATA%\MetaCall\metacall\runtimes\python\Lib\site-packages; runtime PATH includes Python/Scripts/site-packages bin	CONFIGURATION_PATH=%LOCALAPPDATA%\MetaCall\metacall\configurations\global.json	LOADER_LIBRARY_PATH, SERIAL_LIBRARY_PATH, DETOUR_LIBRARY_PATH, PORT_LIBRARY_PATH all exported to %LOCALAPPDATA%\MetaCall\metacall\lib

Windows installer note: current core packaging generators are NSIS;ZIP (not MSI). The Windows one-line installer consumes metacall-tarball-win-<arch>.zip assets from metacall/distributable-windows.

Implication for MREP abstraction: Python path resolution needs at least three explicit layers:

1. User override layer (METACALL_INSTALL_PATH, PYTHONHOME, etc.)
2. Installer/profile layer (distributable wrappers or generated shell/profile scripts)
3. Platform fallback layer (compiled install prefix + known platform discovery roots)

This preserves compatibility with existing installs while removing per-port/per-loader hardcoded path duplication.

3. Profile Discovery

3.1 Discovery Order

The system MUST search for profiles in the following order, with earlier entries taking precedence:

1. $METACALL_ENVIRONMENT - Explicit profile path via environment variable
2. $METACALL_HOME/environment.json - User-level profile
3. $METACALL_PREFIX/etc/metacall/environment.json - System installation profile
4. Embedded default profile in loader library

3.2 Path Resolution

Variable references in profiles MUST be resolved according to the following rules:

Variable	Source	Example
$METACALL_PREFIX	CMake CMAKE_INSTALL_PREFIX or equivalent	/usr/local
$METACALL_HOME	User-configurable, defaults to $HOME/.metacall	~/.metacall
$ORIGIN	Resolved at runtime to loader binary location	Platform-specific

3.3 Profile Validation

The system SHOULD validate profiles against the MREP JSON Schema before use. Invalid profiles MUST be logged and MUST NOT prevent operation, but the invalid sections MUST be ignored.

4. Shell Profile Integration

4.1 Linux and Unix-like Systems

The installation MUST provide a shell profile script at:

• System-wide: /etc/profile.d/metacall.sh
• User-installable: $METACALL_HOME/etc/metacall.sh

The profile script MUST:

1. Determine the installation prefix
2. Source the runtime environment profile
3. Export all necessary environment variables
4. Provide a metacall_env helper function for debugging

Example /etc/profile.d/metacall.sh:

# MetaCall Runtime Environment Profile
# Generated by MetaCall installation

METACALL_PREFIX="/usr/local"
METACALL_HOME="${METACALL_HOME:-${HOME}/.metacall}"

if [ -f "${METACALL_PREFIX}/etc/metacall/environment.json" ]; then
    # Source generated environment from profile
    . "${METACALL_PREFIX}/etc/metacall/metacall-env.sh"
elif [ -f "${METACALL_HOME}/environment.json" ]; then
    . "${METACALL_HOME}/etc/metacall-env.sh"
fi

# Helper function to display current MetaCall environment
metacall_env() {
    echo "MetaCall Environment:"
    echo "  METACALL_PREFIX: ${METACALL_PREFIX}"
    echo "  METACALL_HOME: ${METACALL_HOME}"
    echo "  LOADER_LIBRARY_PATH: ${LOADER_LIBRARY_PATH}"
    echo "  LOADER_SCRIPT_PATH: ${LOADER_SCRIPT_PATH}"
}

4.2 Windows

On Windows, the installation SHOULD:

1. Create a PowerShell profile snippet at $HOME\Documents\PowerShell\Microsoft.PowerShell_profile.ps1
2. Or provide a batch file at %USERPROFILE%\metacall_env.bat for cmd.exe users
3. Register METACALL_ENVIRONMENT in the user environment variables

4.3 macOS

On macOS with Homebrew, the formula MUST:

1. Provide a bash/zshrc snippet via etc/profile.d/metacall.sh
2. Respect HOMEBREW_PREFIX as the installation prefix
3. Follow Homebrew conventions for profile.d integration

5. C API Extensions

5.1 New Functions

The following functions SHOULD be added to metacall.h:

/**
 * Initialize the runtime environment from the active profile.
 * This function reads the MREP, resolves paths, and sets
 * necessary environment variables before loader initialization.
 *
 * Returns 0 on success, non-zero on failure.
 */
int metacall_initialize_environment(void);

/**
 * Get the runtime configuration for a specific language.
 * The returned pointer MUST NOT be freed by the caller.
 *
 * Returns NULL if the language is not configured.
 */
const metacall_runtime_config_t metacall_get_runtime_config(const char *language);

/**
 * Iterate over all configured runtimes.
 * The callback is invoked for each runtime with its name and config.
 * Return non-zero from callback to stop iteration.
 */
int metacall_foreach_runtime(
    int (*callback)(const char *language, const metacall_runtime_config_t config, void *user),
    void *user
);

5.2 Runtime Configuration Structure

typedef struct metacall_runtime_config_type {
    const char *language;      // Language identifier (e.g., "python", "ruby")
    const char *home;          // Resolved runtime home path
    const char **search_paths; // Array of search paths (NULL-terminated)
    const char **environment;  // Array of "KEY=VALUE" strings (NULL-terminated)
} * metacall_runtime_config_t;

6. Loader Integration

6.1 Profile-Driven Initialization

Loaders SHOULD use the MREP system for runtime initialization instead of implementing platform-specific logic. The loader initialization flow becomes:

1. metacall_initialize_environment() reads and parses MREP
2. For each runtime in profile:
   a. Resolve variable references
   b. Set environment variables from profile
   c. Store resolved paths for loader use
3. Loader initialization reads resolved paths from profile
4. Platform-specific env var code is replaced by profile lookup

6.2 Fallback Behavior

The system MUST maintain backward compatibility:

• If no profile is found, fall back to existing environment variable behavior
• If a specific runtime is not in the profile, use loader-specific defaults
• Log a warning when falling back to deprecated behavior

7. Port Integration

7.1 MetaCall-sys (Rust)

The Rust port build script MUST generate METACALL_RUNTIME_CONFIG from the installed profile:

// In build.rs
fn main() {
    // Read installed profile if present
    if let Some(profile) = find_metacall_profile() {
        // Emit runtime config to generated code
        println!("cargo:rustc-env=METACALL_RUNTIME_CONFIG={}", profile);
    }
}

The ensure_python_home() function in rs_port SHOULD be replaced with:

fn ensure_runtime_env() {
    // Call metacall_initialize_environment() from C API
    // This handles all languages, not just Python
}

7.2 Other Ports

All ports (Go, Java, C#, etc.) SHOULD follow the same pattern:

1. Load the C library with metacall_initialize_environment() call
2. Or read METACALL_RUNTIME_CONFIG environment variable if set
3. Do not implement per-language fallback logic

8. Profile Generation

8.1 CMake Integration

The CMake installation process SHOULD generate environment.json during installation:

# In loaders/CMakeLists.txt
configure_file(
    "${CMAKE_CURRENT_SOURCE_DIR}/environment.json.in"
    "${CMAKE_CURRENT_BINARY_DIR}/environment.json"
    @ONLY
)

The template file MUST use CMake variables for path substitution:

{
  "meta": {
    "version": "1.0.0",
    "generated": "@METACALL_GENERATED_TIMESTAMP@"
  },
  "runtimes": {
    "python": {
      "home": "@PYTHON_HOME@",
      "search_paths": [@LOADER_SEARCH_PATHS@]
    }
  },
  "defaults": {
    "loaders_path": "@CMAKE_INSTALL_PREFIX@/@INSTALL_LIB@/metacall/loaders"
  }
}

8.2 Shell Script Generation

During installation, a shell script MUST be generated from the profile:

#!/bin/bash
# Generated from MetaCall environment.json

export METACALL_PREFIX="@CMAKE_INSTALL_PREFIX@"
export LOADER_LIBRARY_PATH="@CMAKE_INSTALL_PREFIX@/@INSTALL_LIB@/metacall/loaders"
export LOADER_SCRIPT_PATH="@CMAKE_INSTALL_PREFIX@/@INSTALL_DATA@/metacall/scripts"
export CONFIGURATION_PATH="@CMAKE_INSTALL_PREFIX@/@INSTALL_DATA@/metacall/config/global.json"

# Python runtime environment
if [ -z "$PYTHONHOME" ]; then
    export PYTHONHOME="@PYTHON_HOME@"
fi

9. Backward Compatibility

9.1 Environment Variables

Existing environment variables MUST continue to work:

Variable	Behavior	Deprecated
LOADER_LIBRARY_PATH	Still honored, profile takes precedence	No
LOADER_SCRIPT_PATH	Still honored, profile takes precedence	No
PYTHONHOME	Still honored, profile takes precedence	Yes
METACALL_CONFIG	Still honored, profile takes precedence	No

9.2 Configuration Files

Existing loader.json files MUST continue to be loaded. The MREP system provides additional configuration that supplements but does not replace existing mechanisms.

Requirements

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

Implementation Requirements

Implementations of this RFC:

1. MUST provide a mechanism for profile discovery as specified in Section 3.1
2. MUST resolve variable references according to the rules in Section 3.2
3. MUST maintain backward compatibility with existing environment variables
4. MUST provide shell profile integration for Linux, Windows, and macOS
5. MUST generate shell scripts from profiles during installation
6. MUST NOT break existing installations that do not use profiles

Design Recommendations

Implementations:

1. SHOULD validate profiles against the MREP JSON Schema
2. SHOULD log warnings when falling back to deprecated behavior
3. SHOULD provide tooling for profile generation and validation
4. SHOULD document the profile format in user-facing documentation
5. MAY provide additional variable reference formats beyond those specified
6. MAY extend the MREP schema with loader-specific fields

References

Standards

1. RFC 2119 - Key words for use in RFCs to Indicate Requirement Levels
2. ISO 8601 - Date and time format
3. JSON Schema - JSON Schema specification

Related Systems

1. SDKMAN - Software Development Kit Manager
2. Conda Environment Files - Conda environment specification
3. GraalVM Native - GraalVM metadata approach
4. asdf Version Manager - Multi-language version manager

MetaCall Resources

1. MetaCall GitHub Repository - Main codebase
2. PR #734 - Windows Python initialization fix
3. MetaCall Configuration - Current configuration system
4. PR #738

Appendix A: Migration Guide "which all of them are one person"

For Loader Developers

To migrate an existing loader to use MREP:

1. Remove platform-specific environment variable code
2. Replace with calls to metacall_get_runtime_config()
3. Use resolved paths from the configuration structure
4. Remove #if defined(WIN32) blocks for environment setup

For Port Developers

To update a port to use MREP:

1. Remove per-language fallback functions (e.g., ensure_python_home())
2. Call metacall_initialize_environment() early in initialization
3. Or read METACALL_RUNTIME_CONFIG from environment

For Package Maintainers

To integrate MREP into distribution packages:

1. Ensure environment.json is generated at build time
2. Install shell profile script to /etc/profile.d/
3. Set METACALL_PREFIX to installation prefix
4. Document the profile requirement in package metadata

Appendix B: Schema

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "https://metacall.io/schemas/mrep-v1.json",
  "title": "MetaCall Runtime Environment Profile",
  "description": "Declarative configuration for MetaCall runtime environments",
  "type": "object",
  "required": ["meta", "runtimes"],
  "properties": {
    "meta": {
      "type": "object",
      "required": ["version", "schema"],
      "properties": {
        "version": {
          "type": "string",
          "pattern": "^\\d+\\.\\d+\\.\\d+$"
        },
        "schema": {
          "type": "string",
          "format": "uri"
        },
        "generated": {
          "type": "string",
          "format": "date-time"
        }
      }
    },
    "runtimes": {
      "type": "object",
      "additionalProperties": {
        "type": "object",
        "properties": {
          "home": { "type": "string" },
          "search_paths": {
            "type": "array",
            "items": { "type": "string" }
          },
          "environment": {
            "type": "object",
            "additionalProperties": { "type": "string" }
          },
          "options": {
            "type": "object",
            "additionalProperties": true
          }
        }
      }
    },
    "defaults": {
      "type": "object",
      "properties": {
        "loaders_path": { "type": "string" },
        "scripts_path": { "type": "string" },
        "config_path": { "type": "string" }
      }
    }
  }
}

[VanshikaSabharwal]

@k5602 @viferga is anyone already working on this?

[viferga]

@k5602 @viferga is anyone already working on this?

No, but I still want to think about it, the proposal is too complex for our needs. There is a prior work here:
#734

[VanshikaSabharwal]

@k5602 @viferga is anyone already working on this?

No, but I still want to think about it, the proposal is too complex for our needs. There is a prior work here: #734

Yeah got it

[k5602]

@viferga any updates or a start with Python only