Command Line Tool¶

The command line tool is called sim_db, but can also be called with sdb. It has a syntax simular to git, where commands are passed to sim_db followed by the arguments to the command: $ sim_db <command> [<args>].

All the available commands can be listed with the list_commands (run as $ sim_db list_commands). What they do and which arguments they take is found by passing the --help or -h option to any of the commands. The same information is found below.

Commands ending in _sim can also be used without this ending, so add is the same commend as add_sim.

All the commands can be called from anywhere is your project after the init command is called in your projects root directory. The only exception is inside the sim_db/ directory if that is included.

Example of Commands to Run the Other Examples¶

The minimal and extensive examples of the Python, C and C++ versions of sim_db can all be run with the bash script run_all_examples.sh found inside the sim_db/examples/ directory of the source code. The part of the script that run the minimal python example is show here:

######################### Run minimal python example ########################### 

# Add example parameters to database for minimal python example.
sim_db add --filename root/examples/params_minimal_python_example.txt

# Get hold of the ID of the exampel parameters for minimal python example.
id_for_minimal_python_example=`sim_db print -n 1 --columns id --no_headers`

# Run minimal_example.py.
sim_db run --id ${id_for_minimal_python_example}

# Delete example simulation from database.
sim_db delete --id ${id_for_minimal_python_example} --no_checks

Note that sim_db/ is here the root directory of the project, and sim_db init have already been called in that directory.

sim_db¶

For running simulations and keeping track of its parameters, results and metadata.

usage: sim_db [--help] [--version] <command> [<args>]

Some common sim_db commands:

init           Initialise 'sim_db' for use in project.
add            Add set of simulation parameters to database.
print          Print parameters in database.
run            Run simulation with parameters from database.
list_commands  List all available commands.

Positional Arguments¶

command: The command, ‘list_commands’, will print all available commands.

Named Arguments¶

--version

Print version of sim_db. Must be passed as the only parameter.

Default: False

Commands¶

add_and_run¶

Add simulation and submit it.

usage: sim_db add_and_run [-h] [--filename FILENAME] [-n N]
                          [--add_unique_results_dir]

Named Arguments¶

--filename, -f

Name of parameter file to add and run.

-n

Number of threads/core to run the simulation on.

--add_unique_results_dir, -u

Add a unique subdirectory for the simulation in the ‘superdir_for_results’ directory in the settings and write it to ‘results_dir’ in the database.

Default: False

add_and_submit¶

Add simulation and submit it.

usage: sim_db add_and_submit [-h] [--filename FILENAME]
                             [--max_walltime MAX_WALLTIME] [--n_tasks N_TASKS]
                             [--n_nodes N_NODES]
                             [--additional_lines ADDITIONAL_LINES]
                             [--notify_all] [--notify_fail] [--notify_end]
                             [--no_confirmation] [--do_not_submit_job_script]
                             [--add_unique_results_dir]

Named Arguments¶

--filename, -f

Name of parameter file added and submitted.

--max_walltime

Maximum walltime the simulation can use, given in ‘hh:mm:ss’ format.

--n_tasks

Number of tasks to run the simulation with. A warning is given if it is not a multiple of the number of logical cores on a node.

--n_nodes

Number of nodes to run the simulation on.

--additional_lines

Additional lines added to the job script.

--notify_all

Set notification for when simulation begins and ends or if it fails.

Default: False

--notify_fail

Set notification for if simulation fails.

Default: False

--notify_end

Set notification for when simulation ends or if it fails.

Default: False

--no_confirmation

Does not ask for confirmation about submitting all simulations with status ‘new’

Default: False

--do_not_submit_job_script

Makes the job script, but does not submit it.

Default: False

--add_unique_results_dir, -u

Add a unique subdirectory for the simulation in the ‘superdir_for_results’ directory in the settings and write it to ‘results_dir’ in the database.

Default: False

add_column¶

Add column to database.

usage: sim_db add_column [-h] --column COLUMN --type TYPE

Named Arguments¶

--column, -c: <Required> Name of the new column.
--type, -t: <Required> Type of the column. ‘INTEGER’, ‘REAL’, ‘TEXT’, ‘int’, ‘float’, ‘string’, ‘bool’ and ‘int/float/string/bool array’ are the valid choices.

add_comment¶

Add comment to simulation in database.

usage: sim_db add_comment [-h] --id ID [--comment COMMENT]
                          [--filename FILENAME] [--append]

Named Arguments¶

--id, -i

<Required> ID of the simulation to add the comment.

--comment, -c

Comment to add.

--filename, -f

Filename of a file which content are to be added as a comment. Only the last 3000 characters will be added.

--append, -a

Append comment or file to the current comment.

Default: False

add_range¶

Add a range of simulations to the database.

usage: sim_db add_range_sim  [-h] [--filename FILENAME] --columns COLUMNS
                             [COLUMNS ...]
                             [--lin_steps LIN_STEPS [LIN_STEPS ...]]
                             [--exp_steps EXP_STEPS [EXP_STEPS ...]]
                             [--end_steps END_STEPS [END_STEPS ...]]
                             [--n_steps N_STEPS [N_STEPS ...]]

Named Arguments¶

--filename, -f

Name of parameter file added as the first in the range.

--columns, -c

<Required> Names of the column for which the range varies. The cartisian products of the varing columns are added to the database. The column type MUST be a integer or a float.

Default: []

--lin_steps

Linear step distance. NEXT_STEP = PREV_STEP + LIN_STEP. If columns have both linear and exponential steps, both will be used. NEXT_STEP = LIN_STEP + PREV_STEP * EXP_STEP

Default: []

--exp_steps

Exponential step distance. NEXT_STEP = PREV_STEP * EXP_STEP. If columns have both linear and exponential steps, both will be used. NEXT_STEP = LIN_STEP + PREV_STEP * EXP_STEP

Default: []

--end_steps

End step of range. The range includes the end, but not anything past it. If both ‘end_steps’ and ‘n_steps’ are used, both endpoint need to be reached.

Default: []

--n_steps

Number of steps in the range. That means one step gives to simulations added. If both ‘end_steps’ and ‘n_steps’ are used, both endpoint need to be reached.

Default: []

add_sim¶

Add simulation to database.

usage: sim_db add_sim [-h] [--filename FILENAME]

Named Arguments¶

--filename, -f: Name of parameter file added and submitted.

cd_results_dir¶

Change the current working directory to the ‘results_dir’ of the specified simulation or the last entry if not specified. (This is done by creating a new subshell, so ‘$ exit’ can be used to return to the original directory and shell instance.)

usage: sim_db cd_results_dir [-h] [--id ID] [-n N]

Named Arguments¶

--id, -i: ‘ID’ of the simulation in the ‘sim.db’ database.
-n: n’th last entry in the ‘sim.db’ database.

combine_dbs¶

Combine two databases into a new one.

usage: sim_db combine_dbs [-h] path_db_1 path_db_2 name_new_db

Positional Arguments¶

path_db_1: <Required> Path to ‘ database 1.
path_db_2: <Required> Path to ‘ database 2.
name_new_db: <Required> Name of the new database.

delete_empty_columns¶

Delete all empty columns in the sim.db, except the default ones.

usage: sim_db delete_empty_columns [-h]

delete_results_dir¶

Delete results in ‘results_dir’ of specified simulations.

usage: sim_db delete_results_dir [-h] [--id ID [ID ...]] [--where WHERE]
                                 [--no_checks]
                                 [--not_in_db_but_in_dir NOT_IN_DB_BUT_IN_DIR]

Named Arguments¶

--id, -i

ID’s of simulation which ‘results_dir’ to deleted.

Default: []

--where, -w

Condition for which simulation’s ‘results_dir’ to deleted. Must be a valid SQL (sqlite3) command when added after WHERE in a SELECT command.

--no_checks

No questions are asked about wheather you really want to delete the ‘results_dir’ of specified simulation.

Default: False

--not_in_db_but_in_dir

Delete every folder in the specified directory that is not a ‘results_dir’ in the ‘, so use with care. Both relative and absolute paths can be used.

delete_sim¶

Delete simulations from sim.db.

usage: sim_db delete_sim [-h] [--id ID [ID ...]] [--where WHERE] [--all]
                         [--no_checks]

Named Arguments¶

--id, -i

ID’s of runs to delete.

Default: []

--where, -w

Condition for which entries should be deleted. Must be a valid SQL (sqlite3) command when added after WHERE in a DELETE command.

--all

Delete all simulation from database.

Default: False

--no_checks

No questions are asked about wheter you really want to delete simulation or the ‘results_dir’ of the simulation.

Default: False

duplicate_and_run¶

Duplicate simulation in database and run it. All parameters (including possible results) of specified simulation is duplicated with the exception of ‘id’ and ‘status’, which is kept unique and set to ‘new’ respectfully.

usage: sim_db duplicate_and_run [-h] --id ID [-n N]

Named Arguments¶

--id, -i: <Required> ‘ID’ of the simulation parameters in the ‘sim.db’ database that should be duplicated.
-n: Number of threads/core to run the simulation on.

duplicate_sim¶

Duplicate simulation in database. All parameters (including possible results) of specified simulation is duplicated with the exception of ‘id’ and ‘status’, which is kept unique and set to ‘new’ respectfully.

usage: sim_db duplicate_sim  [-h] --id ID

Named Arguments¶

--id, -i: <Required> ‘ID’ of the simulation parameters in the ‘sim.db’ database that should be duplicated.

extract_params¶

Extract parameter file from sim.db.

usage: sim_db extract_params [-h] --id ID [--filename FILENAME]
                             [--default_file] [--also_results] [--all]

Named Arguments¶

--id, -i

<Required> ID of the simulation which parameter one wish to extract.

--filename, -f

Name of parameter file generated.

--default_file, -d

Write parameters to the first of the ‘Parameter filenames’ in settings.txt. Ask for confirmation if file exists already.

Default: False

--also_results

Also extract results - parameters added during the simulation excluding metadata. Default is to just extract the parameters added before the simulation was run and found in ‘initial_parameters’ column.

Default: False

--all

Extract all non empty parameters, including metadata.

Default: False

get¶

Get value from ‘column’ of simulation specified or last entry if not specified.

usage: sim_db get [-h] [--id ID] [-n N] column

Positional Arguments¶

column: Column in database from where to get the value.

Named Arguments¶

--id, -i: ‘ID’ of the simulation in the ‘sim.db’ database.
-n: n’th last entry in the ‘sim.db’ database.

init¶

Initialises ‘sim_db’ and must be called before using ‘sim_db’. Will create a ‘.sim_db/’ directory.

usage: sim_db init [-h] [--path PATH]

Named Arguments¶

--path: Path to the top directory of project. If not passed as an argument, the current working directory is assumed to be the top directory.

list_commands¶

Print a list of all the commands.

usage: sim_db list_commands [-h]

list_print_configs¶

Print a list of all the personalized print configurations.

usage: sim_db list_print_configs [-h]

print_sim¶

Print content in sim.db. The default configuration corresponding to the ‘-p default’ option is applied first, as long as the ‘–columns’/’-c’ option is not passed. It can can however be overwritten, as only the last occcurence of any flag is used.

usage: sim_db print_sim [-h] [--id ID [ID ...]] [--not_id NOT_ID [NOT_ID ...]]
                        [-n N] [--columns COLUMNS [COLUMNS ...]]
                        [--not_columns NOT_COLUMNS [NOT_COLUMNS ...]]
                        [--col_by_num COL_BY_NUM [COL_BY_NUM ...]]
                        [--where WHERE] [--sort_by SORT_BY] [--column_names]
                        [--all_columns] [--empty_columns] [--params]
                        [--results] [--metadata] [--no_headers]
                        [--max_width MAX_WIDTH] [--first_line] [--vertically]
                        [-p P] [--diff]

Named Arguments¶

--id, -i

List of ID’s.

--not_id

List of ID’s not to print. Takes president over ‘–id’.

-n

Number of row printed from the bottom up.

--columns, -c

Name of the columns to print.

--not_columns

Name of the columns not to print. Takes presidents over ‘–columns’.

--col_by_num

Number of the columns to print. All non empty columns are printed by default.

--where, -w

Add constraints to which columns to print. Must be a valid SQL (sqlite3) command when added after WHERE in a SELECT command.

Default: “id > -1”

--sort_by

What to sort the output by. Must be a valid SQL (sqlite3) command when added after ORDER BY in a SELECT search. Defalut is id.

Default: “id”

--column_names

Print name and type of all columns.

Default: False

--all_columns

Print all columns. Takes president over ‘–not_columns’.

Default: False

--empty_columns

Print empty columns. Otherwise only non empty columns are printed.

Default: False

--params

Print the parameters added before the simulation run.

Default: False

--results

Print results - the parameters added during the simulation, excluding metadata.

Default: False

--metadata

Print metadata. ‘–params’, ‘–results’ and ‘–metadata’ will together print all non empty columns.

Default: False

--no_headers

Print without any headers.

Default: False

--max_width

Upper limit for the width of each column. Default is no limit.

--first_line

Print only the first line of any entry.

Default: False

--vertically, -v

Print columns vertically.

Default: False

-p

Personal print configuration. Substituted with the print configuration in ‘settings.txt’ corresponding to the provided key string.

--diff, -d

Remove columns with the same value for all the simulations. This leaves only the parameters that are different between the simulations.

Default: False

run_seriel_sims¶

Run multiple simulations in series. If no ID’s or conditions are given all the new simulations are run.

usage: sim_db run_serial_sims [-h] [--id ID [ID ...]]
                              [--where WHERE [WHERE ...]]

Named Arguments¶

--id, -i

‘IDs’ of the simulation parameters in the ‘sim.db’ database that should be used in the simulation.

Default: []

--where, -w

Conditions of the simulation parameters in the ‘sim.db’ database that should be used in the simulation.

Default: []

run_sim¶

Run simulation with ID in database.

usage: sim_db run_sim [-h] [--id ID] [-n N] [--allow_reruns]
                      [--add_unique_results_dir]

Named Arguments¶

--id, -i

‘ID’ of the simulation parameters in the ‘sim.db’ database that should be used in the simulation.

-n

Number of threads/core to run the simulation on.

--allow_reruns

Allow simulations with non ‘new’ status to run.

Default: False

--add_unique_results_dir, -u

Add a unique subdirectory for the simulation in the ‘superdir_for_results’ directory in the settings and write it to ‘results_dir’ in the database.

Default: False

settings¶

Print and change settings. The settings can also be changed be editing the ‘.settings.txt’ file.

usage: sim_db settings [-h] command

Positional Arguments¶

command: ‘print’, ‘add’, ‘remove’ or ‘reset_to_default’

submit_sim¶

Submit job

usage: sim_db submit_sim [-h] [--id ID [ID ...]] [--allow_reruns]
                         [--max_walltime MAX_WALLTIME [MAX_WALLTIME ...]]
                         [--n_tasks N_TASKS [N_TASKS ...]]
                         [--n_nodes N_NODES [N_NODES ...]]
                         [--additional_lines ADDITIONAL_LINES [ADDITIONAL_LINES ...]]
                         [--notify_all] [--notify_fail] [--notify_end]
                         [--no_confirmation] [--do_not_submit_job_script]
                         [--add_unique_results_dir]

Named Arguments¶

--id, -i

ID of simulations to submit.

--allow_reruns

Allow simulations with non ‘new’ status to be submitted.

Default: False

--max_walltime

Maximum walltime the simulation can use, given in ‘hh:mm:ss’ format.

--n_tasks

Number of tasks to run the simulation with. A warning is given if it is not a multiple of the number of logical cores on a node.

--n_nodes

Number of nodes to run the simulation on.

--additional_lines

Additional lines added to the job script.

Default: []

--notify_all

Set notification for when simulation begins and ends or if it fails.

Default: False

--notify_fail

Set notification for if simulation fails.

Default: False

--notify_end

Set notification for when simulation ends or if it fails.

Default: False

--no_confirmation

Does not ask for confirmation about submitting all simulations with status ‘new’

Default: False

--do_not_submit_job_script

Makes the job script, but does not submit it.

Default: False

--add_unique_results_dir, -u

Add a unique subdirectory for the simulation in the ‘superdir_for_results’ directory in the settings and write it to ‘results_dir’ in the database.

Default: False

update_sim¶

Update content in sim.db.

usage: sim_db update_sim [-h] [--id ID] [--where WHERE] --columns COLUMNS
                         [COLUMNS ...] --values VALUES [VALUES ...]
                         [--db_path DB_PATH]

Named Arguments¶

--id, -i

ID of run to update.

--where, -w

Condition for which entries should be updated. Must be a valid SQL (sqlite3) command when added after WHERE in a UPDATE command.

Default: “id > -1”

--columns, -c

<Required> Name of column to update in runs.

--values, -v

<Required> New value updated at run with id and column as specifed.

--db_path

Full path to the database used.