isciml tdem

Data generation module for Time Domain Electromagnetic Inverse Imaging

Overview

ISCIML is a command line tool designed to generate training data for building artificial intelligence inversion models for time domain electromagnetic (TDEM) data acquisition methods. The tool simulates electromagnetic field propagation through different materials and records the resulting signals at specified receiver locations, which can then be used to train machine learning models for geophysical inversion.

The tool supports both forward and adjoint modeling approaches, allowing users to generate comprehensive datasets that capture the complex relationship between subsurface properties and electromagnetic responses. By adjusting various parameters such as mesh configuration, material properties, transmitter characteristics, and temporal settings, users can create customized training datasets tailored to specific geological environments and survey configurations.

Installation

Please email [e-mail] for isciml singularity image for TDEM training data generation.

Usage

singularity exec ./container/isciml_tdem.sif isciml --mesh_path ./data/em61_cart_mesh_0p25.xdmf --material_path ./data/em61_flat_n_pipes_3.npz --receiver "Rx_0" --loop_z 0.25 --solver_type adjoint --receiver_locati
on 0.001 0.001 0.25 --r_values_path ./data/em61_cart_r_values.npz

Option	Type	Default	Description
--mesh_path	TEXT	mesh/mesh.xdmf	Path to the mesh file.
--material_path	TEXT	input/material_properties.npz	Path to the material properties file.
--output_dir	TEXT	./output	Directory to save the output files.
--loop_dimensions	FLOAT FLOAT	1.0 1.0	Transmitter loop parameters as a tuple (Lx, Ly).
--loop_z	FLOAT	0.001	Height of the transmitter loop.
--current_amplitude	FLOAT	1.0	Current amplitude in the transmitter loop.
--receiver_location	FLOAT FLOAT FLOAT	0.001 0.001 0.001	Receiver location coordinates.
--dt_list	FLOAT FLOAT FLOAT	1e-08 1e-07 1e-06	Time step sizes to use in the simulation.
--time_steps	INTEGER INTEGER INTEGER	500 500 1260	Number of time steps to use in the simulation.
--dbz_dbt_key	TEXT	dBz_dt	Key for the dBz/dt data in the output solution.
--t_step_key	TEXT	t_step	Key for the time step data in the output solution.
--r_values_path	TEXT	input/r_values.npz	Path to the file containing r values.
--adjoint_npz_path	TEXT	output/results_adjoint_all.npz	Path to the output adjoint results file.
--mu_0	FLOAT	1.2566370614359173e-06	Magnetic permeability in free space.
--time_gates	FLOAT FLOAT FLOAT FLOAT	0.000256 0.000406 0.000706 0.001306	Time gates for the simulation.
--solver_type	[forward|adjoint]	forward	Type of solver to use: 'forward' or 'adjoint'.
--receiver	TEXT	all	Receiver to use in the simulation.
--help	FLAG	N/A	Show this message and exit.

Other utilities

generate_r_values.py

generate_r_values.py is a utility script for calculating distance values (r_values) between mesh nodes and a specified receiver location in electromagnetic simulations. These distance values are essential for accurate modeling of electromagnetic field propagation and are used by the ISCIML simulation tool for time domain electromagnetic (TDEM) modeling.

Command Line Options

Option	Type	Required	Default	Description
--mesh_path	PATH	Yes	N/A	Path to the mesh file containing the geometric information.
--receiver_location	FLOAT FLOAT FLOAT	Yes	N/A	Coordinates of the receiver location (x, y, z).
--r_values_key	TEXT	No	"r_values"	Key used to reference the distance values in the output file.
--output_path	PATH	Yes	N/A	Path where the output NPZ file will be saved.
--help	FLAG	No	N/A	Display help information and exit.

Usage Examples

Basic Usage

python generate_r_values.py --mesh_path mesh/simulation_mesh.xdmf --receiver_location 0.0 0.0 0.001 --output_path input/r_values.npz

This command will:

• Load the mesh from mesh/simulation_mesh.xdmf

• Calculate distances from each mesh node to the receiver at coordinates (0.0, 0.0, 0.001)

• Save the results to input/r_values.npz using the default key "r_values"

• Load the mesh from mesh/fine_mesh.xdmf

• Calculate distances from each mesh node to the receiver at coordinates (1.0, 2.0, 0.05)

• Save the results to input/custom_r_values.npz using the key "distances_receiver1"

The output file generated by this script is typically used as input to the ISCIML simulation tool via the --r_values_path option:

generate_npz_mappings.py

Overview

generate_npz_mappings.py is a utility script that generates a list of NPZ filenames and their corresponding receiver indices. It scans through a specified folder containing NPZ files and creates a mapping file that associates each file with its receiver information.

Purpose

This script creates an index that maps simulation output files to their respective receiver locations, making it easier to organize and process multiple simulation results. This mapping is especially useful when working with large batches of TDEM simulations that involve multiple receiver positions.

Command Line Options

Option	Type	Default	Description
--folder	TEXT	Current directory	Directory containing NPZ files to index.
--output_file	TEXT	mappings.txt	Path where the output text file (containing filenames and receiver indices) will be saved.
--help	FLAG	N/A	Show help message and exit.

Usage

Basic Usage

singularity exec isciml_tdem.sif python3 /isciml/generate_npz_mappings.py --folder ./simulation_outputs --output_file mappings.txt

This command will:

• Scan all NPZ files in the ./simulation_outputs directory

• Generate a mapping that associates each filename with its receiver index

• Save the mapping information to a text file named mappings.txt

SLURM Array Job Script for ISCIML TDEM Simulations

Overview

This SLURM batch script is designed to run multiple electromagnetic simulations in parallel using ISCIML with the Singularity container system. It processes a list of material paths and receiver indices stored in a mapping file, executing one simulation per array task.

Script Details

#!/bin/bash
#SBATCH --ntasks=1
#SBATCH --time=2:00:00
#SBATCH --mem=4G
#SBATCH --array=0-2
#SBATCH --output=array_job_slurm_%A_%a.out
SIF_FILE=/path/to/isciml_tdem.sif
NPZ_MAPPINGS_FILE=/path/to/npz_mappings.txt
MESH_PATH=/path/to/mesh.xdmf
R_VALUES_PATH=/path/to/r_values.npz
OUTPUT_PATH=/path/to/output
START_INDEX=0
LINE=$((SLURM_ARRAY_TASK_ID+1 + START_INDEX)) 
MATERIAL_PATH=$(sed -n "${LINE}p" ${NPZ_MAPPINGS_FILE} | cut -f1)
RECEIVER_KEY=$(sed -n "${LINE}p" ${NPZ_MAPPINGS_FILE} | cut -f2 | tr -d ' ')
echo "Job started for ${MATERIAL_PATH} ${RECEIVER_KEY}"
singularity exec isciml_tdem.sif isciml --loop_dimensions 1.0 0.5 --mesh_path ${MESH_PATH} --material_path ${MATERIAL_PATH} --receiver "${RECEIVER_KEY}" --loop_z 0.25 --solver_type adjoint --receiver_location 0.001 0.001 0.251 --r_values_path ${R_VALUES_PATH} --output_dir ${OUTPUT_PATH}
echo "Job completed ... "

Purpose

This script facilitates running multiple adjoint solver simulations across different material properties and receiver configurations using SLURM's array job functionality. Each array task processes a specific simulation defined by a line in the mapping file.

SLURM Resource Allocation

Directive	Value	Description
--ntasks	1	Allocates 1 CPU core per job
--time	2:00:00	Sets a 2-hour time limit per job
--mem	4G	Allocates 4GB of memory per job
--array	0-2	Runs 3 array jobs (indices 0, 1, and 2)
--output	array_job_slurm_%A_%a.out	Sets the output file naming pattern

Parameters

Variable	Description
SIF_FILE	Path to the Singularity container image for ISCIML TDEM.
NPZ_MAPPINGS_FILE	Path to the text file containing material paths and receiver indices (tab-separated).
MESH_PATH	Path to the mesh file (.xdmf) for the simulation domain.
R_VALUES_PATH	Path to the pre-computed r-values NPZ file.
OUTPUT_PATH	Directory where simulation results will be saved.
START_INDEX	Offset value for array task indexing (default: 0).

How It Works

1. The script uses the SLURM array task ID to determine which line of the mapping file to process.

2. It extracts the material path and receiver key from the specified line in the mapping file.

3. It runs the ISCIML simulation with these parameters using the Singularity container.

4. Output logs are saved with the naming pattern array_job_slurm_%A_%a.out, where:

   • %A is the master job ID

   • %a is the array task ID

Usage

Submission Command

To submit this job with the predefined settings (3 array tasks):

sbatch path/to/this_script.sh

To modify the array range (for example, to run jobs for lines 5-15 in the mapping file):

sbatch --array=5-15 path/to/this_script.sh

Mapping File Format

The mapping file (NPZ_MAPPINGS_FILE) should be a tab-separated text file with each line containing:

• Material properties file path (first column)

• Receiver index/key (second column)

Example:

/path/to/material1.npz    receiver_0
/path/to/material2.npz    receiver_1
/path/to/material3.npz    receiver_2

Simulation Configuration

The script runs ISCIML with the following fixed parameters:

• Loop dimensions: 1.0 x 0.5 units

• Loop height: 0.25 units

• Solver type: adjoint

• Receiver location: (0.001, 0.001, 0.251)

These parameters can be modified directly in the script as needed.