mlp

Simple multilayer perceptron (MLP) from scratch

Build Status	License

Quickstart

Installation

Download the binary compatible with your hardware:

This list is severely limited at the moment, but please feel free to build it from source. The instructions are found below.

Quick demo

Simulate data, fit, and extract marginal effects (-v for verbose):

./mlp -v

Using default hyperparameters

Using the default hyperparamereters should suffice for trial analyses where the main objective is the estimation of the ranks of the entries, treatments, sites, etc.

Simulate a field trial dataset:

INPUT="input.tsv"
time \
./mlp \
    --simulate-data-only \
    --simulation-fname-output=$INPUT \
    --simulation-n-observations $(echo "2*3*5*6*10" | bc) \
    --simulation-n-features-continuous 0 \
    --simulation-n-features-categorical 2,3,5,6,10 \
    --simulation-n-output-columns 1 \
    --verbose

Fit without extracting marginal effects:

INPUT="input.tsv"
OUTPUT="output.json"
./mlp -f $INPUT -o $OUTPUT --skip-marginals

Extract marginal effects:

INPUT="input.tsv"
OUTPUT="output.json"
./mlp --marginals-only --model $OUTPUT -f $INPUT
cat ${OUTPUT%.json*}-marginal_effects.tsv

Using fixed hyperparameters

Here we demonstrate fitting a dataset with more predictors than observations using fixed hyperparameters where our objective is prediction rather than description, i.e. we do not intend to estimate marginal effects.

Simulate a genomic prediction dataset:

INPUT="input_n1k.tsv"
time \
./mlp \
    --simulate-data-only \
    --simulation-fname-output=$INPUT \
    --simulation-n-observations 1000 \
    --simulation-n-features-continuous 23000 \
    --simulation-n-features-categorical 0 \
    --simulation-n-output-columns 1 \
    --simulation-n-hidden-layers 2 \
    --simulation-weights-distribution normal \
    --simulation-weights-distribution-param-1 0.0 \
    --simulation-weights-distribution-param-2 0.01 \
    --verbose
head  $INPUT | cut -f1-5
head -n901 $INPUT > ${INPUT%.tsv*}-TRAINING.tsv
head -n1 $INPUT > ${INPUT%.tsv*}-VALIDATION.tsv
tail -n100 $INPUT >> ${INPUT%.tsv*}-VALIDATION.tsv

Fit the training set without extracting marginal effects: (Note: make sure to use the --skip-marginals flag because with large number of predictors, extracting marginal effects of each will take a long time)

INPUT="input_n1k-TRAINING.tsv"
OUTPUT="output_n1k.json"
time \
./mlp \
    -f $INPUT \
    -o $OUTPUT \
    --n-batches=1 \
    --n-hidden-layers=2 \
    --n-hidden-nodes=1024,128 \
    --n-epochs=1000 \
    --n-burnin-epochs=10 \
    --f-patient-epochs=0.1 \
    --f-validation=0.1 \
    --skip-marginals \
    --verbose

Predict the validation set

INPUT_TO_PREDICT="input_n1k-VALIDATION.tsv"
OUTPUT="output_n1k.json"
time \
./mlp \
    --predict-only \
    -f $INPUT_TO_PREDICT \
    --model $OUTPUT \
    --verbose
# Extract true and predicted values for assessment
PREDICTIONS=${OUTPUT%.json*}-predictions.tsv
cut -f1 $INPUT_TO_PREDICT > TRUE.tmp
cut -f1 $PREDICTIONS > PREDICTED.tmp
paste -d'\t' TRUE.tmp PREDICTED.tmp > TRUE_VS_PREDICTED.tsv
rm TRUE.tmp PREDICTED.tmp

Assess prediction:

df = read.delim("TRUE_VS_PREDICTED.tsv", header=TRUE)
colnames(df) <- c("true", "predicted")
cor(df$true, df$predicted)
txtplot::txtplot(df$true, df$predicted)

Using hyperparameter optimisation

We also demonstrate fitting a dataset with more predictors than observations but now using hyperparameter optimisation.

Simulate a genomic prediction dataset:

INPUT="input_n1k.tsv"
time \
./mlp \
    --simulate-data-only \
    --simulation-fname-output=$INPUT \
    --simulation-n-observations 1000 \
    --simulation-n-features-continuous 23000 \
    --simulation-n-features-categorical 0 \
    --simulation-n-output-columns 1 \
    --simulation-n-hidden-layers 2 \
    --simulation-weights-distribution normal \
    --simulation-weights-distribution-param-1 0.0 \
    --simulation-weights-distribution-param-2 0.01 \
    --verbose
head  $INPUT | cut -f1-5
head -n901 $INPUT > ${INPUT%.tsv*}-TRAINING.tsv
head -n1 $INPUT > ${INPUT%.tsv*}-VALIDATION.tsv
tail -n100 $INPUT >> ${INPUT%.tsv*}-VALIDATION.tsv

Fit the training set without extracting marginal effects: (Note: again make sure to use the --skip-marginals flag because with large number of predictors, extracting marginal effects of each will take a long time)

INPUT="input_n1k-TRAINING.tsv"
OUTPUT="output_n1k.json"
time \
./mlp \
    -f $INPUT \
    -o $OUTPUT \
    --hyperparameter-optimisation \
    --range-hidden-layers=1,2,1 \
    --range-hidden-layer-nodes=100,1000,900 \
    --range-dropout-rates=0.0,0.0,0.01 \
    --range-learning-rates=1e-3,1e-3,1e-3 \
    --range-n-epochs=1000,1000,1000 \
    --range-n-burnin-epochs=10,10,10 \
    --range-f-patient-epochs=0.1,0.1,0.1 \
    --range-f-validation=0.0,0.2,0.1 \
    --range-n-batches=1,1,1 \
    --selection-activations=ReLU \
    --selection-costs=MSE \
    --selection-optimisers=Adam \
    --selection-weights-initialisations=He,Cauchy \
    --skip-marginals \
    --verbose

Predict the validation set

INPUT_TO_PREDICT="input_n1k-VALIDATION.tsv"
OUTPUT="output_n1k.json"
time \
./mlp \
    --predict-only \
    -f $INPUT_TO_PREDICT \
    --model $OUTPUT \
    --verbose
# Extract true and predicted values for assessment
PREDICTIONS=${OUTPUT%.json*}-predictions.tsv
cut -f1 $INPUT_TO_PREDICT > TRUE.tmp
cut -f1 $PREDICTIONS > PREDICTED.tmp
paste -d'\t' TRUE.tmp PREDICTED.tmp > TRUE_VS_PREDICTED.tsv
rm TRUE.tmp PREDICTED.tmp

Assess prediction:

df = read.delim("TRUE_VS_PREDICTED.tsv", header=TRUE)
colnames(df) <- c("true", "predicted")
cor(df$true, df$predicted)
txtplot::txtplot(df$true, df$predicted)

Build from source code

Install pixi:

wget -qO- https://pixi.sh/install.sh | sh

Setup the workspace:

git clone https://github.com/jeffersonfparil/mlp.git
cd mlp
pixi init

Add cargo, cuda-nvrtc:

cd mlp
pixi shell
pixi add rust
pixi add cuda-nvrtc==12.8.93
which cargo
ls -lhtr ${PIXI_PROJECT_ROOT}/.pixi/envs/default/lib/libnvrtc*

Build:

cd mlp
cargo build --release
./target/release/mlp -h

Unit testing

cd mlp
pixi shell
# export LD_LIBRARY_PATH=${PIXI_PROJECT_ROOT}/.pixi/envs/default/lib # in case the dynamic linker library is not in the path
time cargo test -- --show-output

Data formats

Input data:
- Default format is TSV (tab-delimited); other delimiters are supported via -d or --delim
- The first column is assumed to contain numeric response values, but you may use one or more target columns anywhere in the file
- Remaining columns are explanatory variables:
  - numeric → continuous or binary
  - non-numeric → categorical factor levels, converted to binary via one-hot encoding
- See example: ./misc/input_simulated-T20260527230243-R43101535.tsv
Model:
- JSON: network model exported from the Network struct
- n_observations (usize): number of observations
- n_features (usize): number of input features
- n_targets (usize): number of output dimensions
- n_hidden_layers (usize): number of hidden layers
- n_hidden_nodes (Vec): nodes per hidden layer
- dropout_rates (Vec): dropout rate for each hidden layer
- targets (Vec): observed values, standardised
- targets_mean_sd (f32,f32): mean and standard deviation of targets
- predictions (Vec): predicted values
- weights_per_layer (Vec<Vec>): weight matrices by layer
- biases_per_layer (Vec<Vec>): bias vectors by layer
- weights_x_biases_per_layer (Vec<Vec>): pre-activation sums by layer
- activations_per_layer (Vec<Vec>): layer outputs, including input layer
- weights_gradients_per_layer (Vec<Vec>): weight gradients by layer
- biases_gradients_per_layer (Vec<Vec>): bias gradients by layer
- activation (String): activation function
- cost (String): cost function
- weights_initialisation (String): weights initialisation method (He, Cauchy, Uniform, StandardNormal)
- n_epochs (usize): number of training epochs
- seed (usize): random seed used for dropouts
- loss (f32): mean loss (not part of the actual Network struct)
- See example: ./misc/output_network-T20260527230245-R616739134.json
Predictions: same as the input format, but response columns hold predicted values
Marginal effects:
- TSV: tab-delimited
- Estimates come from perturbation or SHAP methods
- See example: ./misc/output_network-T20260527230245-R616739134-marginal_effects.tsv
Figures/plots:
- SVG: loss curve and observed vs predicted scatterplot
- PNG: marginal effects barplot
- See examples:
Special characters
- Used in progress bars: █
- Used as delimiters between non-numeric or categorical variable names and their levels: ➵
- Used as delimiters in marginals' combinations: ▓

Benchmarking

Install Nextflow (if not on an HPC or just wants to use pixi)

pixi global install -c conda-forge -c bioconda nextflow

Run the workflow:

On a single machine or one HPC node for testing:

cd mlp/tests
pixi run nextflow run nextflow_pipeline/main.nf \
    -c nextflow_pipeline/nextflow_test.config \
    -resume
pixi run dot -Tsvg logs/flowchart.dot > logs/flowchart.svg

On an HPC system with Slurm

cd mlp/tests
sbatch run_nextflow.sh

Miscellaneous

MLPInterrogator.jl

Details

Planning on making an mlp model interrogator, i.e. an interactive Julia library to interrogate the model output (*.json*). I anticipate users to ask: What if I want to see how some of the same categorical factor levels (e.g. a subset of entries in a yield trial) perform under some other categorical factor levels or continuous explanatory variable values they did not have empirical observations on (e.g. on a different set of environments)? --> Now, that I've written this down, the --predict-only flag does this. However, the new input feature data needs to be generated by-hand and so this interactive Julia tool is better poised to just generate these new sets of input data sets (with dummy target values set as NAN) for use with mlp!

Simulate an HPC

Under construction...

Details

Install slurm:

sudo apt install slurmd slurmctld -y
sudo chmod 777 /etc/slurm
sudo cat << EOF > /etc/slurm/slurm.conf
# slurm.conf file generated by configurator.html.
# Put this file on all nodes of your cluster.
# See the slurm.conf man page for more information.
#
ClusterName=localcluster
SlurmctldHost=localhost
MpiDefault=none
ProctrackType=proctrack/linuxproc
ReturnToService=2
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmctldPort=6817
SlurmdPidFile=/var/run/slurmd.pid
SlurmdPort=6818
SlurmdSpoolDir=/var/lib/slurm/slurmd
SlurmUser=slurm
StateSaveLocation=/var/lib/slurm/slurmctld
SwitchType=switch/none
TaskPlugin=task/none
#
# TIMERS
InactiveLimit=0
KillWait=30
MinJobAge=300
SlurmctldTimeout=120
SlurmdTimeout=300
Waittime=0
# SCHEDULING
SchedulerType=sched/backfill
SelectType=select/cons_tres
SelectTypeParameters=CR_Core
#
#AccountingStoragePort=
AccountingStorageType=accounting_storage/filetxt
JobCompType=jobcomp/none
JobAcctGatherFrequency=30
JobAcctGatherType=jobacct_gather/linux
SlurmctldDebug=info
SlurmctldLogFile=/var/log/slurm/slurmctld.log
SlurmdDebug=info
SlurmdLogFile=/var/log/slurm/slurmd.log
AccountingStorageLoc=/var/log/slurm/accounting.txt
#
# COMPUTE NODES
NodeName=localhost CPUs=4 Boards=1 SocketsPerBoard=1 CoresPerSocket=2 ThreadsPerCore=2 RealMemory=15868 Gres=gpu:h100:1 State=UNKNOWN
PartitionName=gpu Nodes=ALL Default=YES MaxTime=INFINITE State=UP
GresTypes=gpu
EOF
sudo chmod 755 /etc/slurm/
sudo systemctl start slurmctld
sudo systemctl start slurmd
sudo scontrol update nodename=localhost state=idle
sinfo
sudo cat /var/log/slurm/slurmd.log
sudo cat /var/log/slurm/slurmctld.log

Install Lmod:

sudo apt install lua5.4 liblua5.4-dev lmod -y
sudo apt install tcl-dev -y
wget https://sourceforge.net/projects/lmod/files/Lmod-8.7.tar.bz2
tar xfvj Lmod-8.7.tar.bz2
rm Lmod-8.7.tar.bz2
cd Lmod-8.7/
./configure --prefix=$HOME --with-fastTCLInterp=no
sudo make install
echo 'export PATH=$HOME/lmod/8.7/libexec:$PATH' >> ~/.bashrc
echo 'source $HOME/lmod/8.7/init/bash' >> ~/.bashrc
echo 'export LMOD_CMD=$HOME/lmod/8.7/libexec/lmod' >> ~/.bashrc
echo 'export MODULEPATH="/etc/lmod/modules/"' >> ~/.bashrc

Sample module file (/etc/lmod/modules/R.lua):

sudo chmod -R 777 /etc/lmod/modules/
sudo cat << EOF > /etc/lmod/modules/R.lua
help([[
...
]])
whatis("Version: 4.1.2")
whatis("R statistical computing environment")
prepend_path("LD_LIBRARY_PATH","/usr/local/lib/R/site-library/")
prepend_path("LIBRARY_PATH","\$HOME/R/x86_64-pc-linux-gnu-library/4.3")
prepend_path("PATH","/usr/bin")
EOF
sudo chmod -R 755 /etc/lmod/modules/

sshare setup for snakemake-executor-plugin-slurm

cat << 'EOF' | sudo tee /usr/local/bin/sshare > /dev/null
#!/bin/bash
# Fake sshare data to satisfy Snakemake SLURM plugin
echo "Account|User|RawShares|NormShares|RawUsage|NormUsage|FairShare"
echo "localuser"
EOF
sudo chmod +x /usr/local/bin/sshare

GPU setup

ls -lh /etc/slurm/
sudo touch /etc/slurm/fake_gpu
cat << 'EOF' | sudo tee /etc/slurm/gres.conf > /dev/null
NodeName=localhost Name=gpu Type=h100 Count=1 Flags=CountOnly
EOF
sudo chmod 644 /etc/slurm/gres.conf

sudo sed -i 's/localhost/paril-ThinkPad-T470/g' /etc/slurm/slurm.conf
sudo sed -i 's/localhost/paril-ThinkPad-T470/g' /etc/slurm/gres.conf

sudo systemctl restart slurmd
sudo systemctl restart slurmctld
sudo scontrol update NodeName=paril-ThinkPad-T470 State=DOWN Reason="name change reset"
sudo scontrol update NodeName=paril-ThinkPad-T470 State=RESUME

sudo tail -n 15 /var/log/slurm/slurmctld.log

sudo mknod -m 666 /dev/fake_nvidia c 195 255

cat << 'EOF' | sudo tee /etc/slurm/gres.conf > /dev/null
NodeName=paril-ThinkPad-T470 Name=gpu Type=h100 File=/dev/fake_nvidia
EOF

sudo systemctl restart slurmd
sudo systemctl restart slurmctld

sudo scontrol update NodeName=paril-ThinkPad-T470 State=DOWN Reason="hardware fix"
sudo scontrol update NodeName=paril-ThinkPad-T470 State=RESUME

sudo touch /var/log/slurm/accounting.txt
sudo chmod 777 /var/log/slurm/accounting.txt

sinfo

Test

conda config --set channel_priority strict
pixi run snakemake --executor slurm --jobs 1 --use-conda --default-resources slurm_account="localuser"

module avail R
module add R

Run comparisons on an on-going/incomplete run

DIR=/home/jp3h/Documents/mlp/tests/output/gp
RSCRIPT=/home/jp3h/Documents/mlp/tests/scripts/comparison.R
cd $DIR

for f_mlp in $(ls output-*-*-MLP.tsv)
do
    f_linear=$(echo $f_mlp | sed 's/-MLP/-LINEAR/g')
    f_trees=$(echo $f_mlp | sed 's/-MLP/-TREES/g')
    # echo $f_mlp
    # echo $f_linear
    # echo $f_trees
    if [ -f  $f_linear ]
    then
        echo $f_linear
        pixi run Rscript $RSCRIPT gp $f_mlp $f_linear .
    fi
    if [ -f  $f_trees ]
    then
        echo $f_trees
        pixi run Rscript $RSCRIPT gp $f_mlp $f_trees .
    fi
done

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
misc		misc
src		src
tests		tests
.gitattributes		.gitattributes
.gitignore		.gitignore
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
LICENSE		LICENSE
README.md		README.md
pixi.lock		pixi.lock
pixi.toml		pixi.toml

Navigation Menu

Repository files navigation

mlp

Quickstart

Installation

Quick demo

Using default hyperparameters

Using fixed hyperparameters

Using hyperparameter optimisation

Build from source code

Unit testing

Data formats

Benchmarking

Install Nextflow (if not on an HPC or just wants to use pixi)

Run the workflow:

On a single machine or one HPC node for testing:

On an HPC system with Slurm

Miscellaneous

MLPInterrogator.jl

Simulate an HPC

Install slurm:

Install Lmod:

Sample module file (/etc/lmod/modules/R.lua):

sshare setup for snakemake-executor-plugin-slurm

GPU setup

Test

Run comparisons on an on-going/incomplete run

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

mlp

Quickstart

Installation

Quick demo

Using default hyperparameters

Using fixed hyperparameters

Using hyperparameter optimisation

Build from source code

Unit testing

Data formats

Benchmarking

Install Nextflow (if not on an HPC or just wants to use pixi)

Run the workflow:

On a single machine or one HPC node for testing:

On an HPC system with Slurm

Miscellaneous

MLPInterrogator.jl

Simulate an HPC

Install slurm:

Install Lmod:

Sample module file (/etc/lmod/modules/R.lua):

sshare setup for snakemake-executor-plugin-slurm

GPU setup

Test

Run comparisons on an on-going/incomplete run

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages