v1.12 of MCwrapper

v1.12 of MCwrapper

--Error Checking--
To begin with error checking is in place at all steps of the MC process. This should allow users to monitor their batch jobs now as well as provide mode detailed logging for diagnosis

--OSG Support--
The osg is now supported in MCwrapper! Simply set the batch system to “osg” in your MC.config file and run in batch mode on a submit node (make sure you have your certificate and a valid proxy). It is that “easy”. Currently, MCwrapper takes a local copy of MakeMC.(c)sh to run which can cause some version confusion between the software stack and the job itself. This difference is not known at present to cause issues. In the near term the osg software stack will be updated at which point MCwrapper will be updated to better integrate with the software stack (and unburden the job submission process). After a final update of the software stack there should be parity between the local version and the osg version. In principle this should lead to a state where “if it works interactively it should work on the osg”

--Bug fixes--
As always there have been a handful of bug fixes affecting specific use-cases as well as touch-ups

MCwrapper v1.11

This is the version of MCwrapper used in the group MC for 2017recon_v2. In the process of generating samples a number of bugs were discovered in the handling of BGRate for runs that do not have beam_on_current, certain AMO runs, and a typo that did not effect the groupMC but has also been fixed.

Using this version will perform the same processes as the group MC runs

v 1.10 MCwrapper

Besides the many bug fixes comes a fairly large update in how batch jobs are handled (now with less chance to DDOS the rcdb servers). It is still recommended that you run batch with sqlite. To aid in this, SQLITEPATH has been split and relabled to ccdbSQLITEPATH and rcdbSQLITEPATH to handle both sqlite objects. The ordering of operations within MakeMC has been changed to better utilize SQLite as well as to utilize the newest feature

BGRate_calc: Now when using either the BeamPhotons or TagOnly backgrounds MCwrapper will invoke BGRate_calc, a new executable that will calculate BGRate based on the parameters of the Simulation. This means that when running over a run range with one of these background types each run will get the "proper" BGRate without all the hassle of dealing with each run explicitly. This can still be overwritten via the colon option in MC.config.

additionally bggen_jpsi and gen_ee have been added with the disclaimer that gen_ee appears to be broken. When this is patched generator side it should begin functioning nominally in MCwrapper without intervention on the wrapper's behalf

MCwrapper v1.9

Several small bug fixes. Improvements to generating with background. Improvements to AMO runs.

New Feature!

Now you can call "gluex_MC.py MC.config 11366-11555 1000000"! In this case all production and approved runs in the given run range will be queried for the number of events. Each eligible run will then contribute the proportional number of events to the requested number of events. This allows users to easily create MC data sets that are more comparable to an entire run period (thus averaging out run dependent effects). Note: that because one cannot generate a partial event MCwrapper currently rounds. If the number of events requested is too small it may not produce any events from a specific run number. Additionally, due to this rounding, the total set may not be equal to the requested size. It is recommended to only use this feature with larger sample size requests to ensure a good mix of events from all applicable runs.

reRelease to fix bug in W&M

Introducing MCwrapper v1.8

The biggest change comes in the form of support for qsub systems natively. A big thanks to the guinea pigs who helped test/develop MCwrapper for use outside of JLAB. It has been tested on IU's, CMU's, and William and Mary's clusters.

Gcontrol.in now has more hooks to use values pulled from the rcdb unless other values are specified. These include coherent peak position, electron beam energy, and radiator thickness. If AMO runs are chosen it should handle these cases properly too.

gen_2k has been added! See the examples folder for example generator config files

MakeMC will now cd into the running_directory first. This is defaulted to “./” so should only effect people who need to specify such things to batch systems eg qsub

SQLITE can now be used. Simply specify the path in your wrapper configuration file...or don't...it's not forcing you to use SQLITE...

A new command line option “logdir” has been added. This currently only applies to qsub; if you successfully run but do not get anything in your output log directory then use this command.

The command line argument “variation=….” can now handle multiple “=” signs

Some better error catching before too much has time been wasted.

A few minor bug fixes

As always please see the wiki https://github.com/JeffersonLab/hd_utilities/wiki/MCwrapper for more detailed documentation

v1.7 MCwrapper

A couple of minor bugs and added support for condor. If you have a local condor cluster simply set BATCH_SYSTEM=condor in your MCwrapper configuration file.

Note: This does submit and should run, however without access to a condor system bugs may be present. If you are running into trouble please reach out so I can correct them quickly.

version 1.6 with qsub compatibility

There have been several large changes and many smaller changes/additions:

qsub support is in! simply set BATCH_SYSTEM=qsub. The limited tests show it functioning but users should be aware that there may be bugs with individual usage. Bugs should be reported and hopefully dealt with swiftly.

--Because of the needs of qsub several configuration parameters in MC.config have been added/modified. Additions include "BATCH_SYSTEM" and "RUNNING_DIRECTORY". BATCH_SYSTEM tells the wrapper what submission style you are targeting. Currently this can be set to "swif" and "qsub"; hopefully with "condor" on the way

--NCORES and TIME_LIMIT have been modified. In the case of NCORES you may, besides providing the number of cores to request provide NODES:NODE_ID:PPN. Do NOT put in the equal signs xxxx:xxx:ppn=4; the wrapper is smart enough to put those in for you. For institutions not using the above style NODES:PPN is also supported. Additionally, TIME_LIMIT can be set in the qsub style xx:xx:xx.

--The command line option for the wrapper "swif=1" has been superseded by "batch=1". This should provide a better user experience for those not using swif.

--additionally batch=2 can be provided. In the case of swif batch=2 will not only create the workflow if needed and add jobs to it but it will also run that workflow so you don't have to remember. If batch=1 is provided to the wrapper in a BATCH_SYSTEM=swif job then the workflow will be created (if needed) and jobs added to it. Instead of running the workflow it will, instead, print out a reminder for the user which can be copied.

--updated default radiator thickness.

--random trigger skims are now portable; use: BKG=loc:/path/to/folder outside JLAB

--error check to prevent a specific kind of auger submit error for swif

v 1.4 with better pregenerated file managament

v1.4 many small bug fixes/changes as well as:

--fixed a bug which broke random trigger merging for c shell users

--incorporated David L's suggestion; now if you have a pre-generated file of events (say 1M events) each of the 100 default sub-job will run over a different chunk of the data.

--fixed an issue in which files already present at run time would be treated as job output and be moved. Now, any files that exist at the start of the run should not be picked up and moved

--more targeted moving/cleaning of files generated by the wrapper

--new output directory structure. Now, the configurations folder contains two sub-folders: "generation" which will contain each subjob's configuration file used in the generation step and "geant" which contains each sub-job's control.in file.

--inclusion of gen_omega_3pi generator. See the Generator/.../examples/ folder for example configuration files

patch of v1.3.1

small patch to improve the robustness of the use of jana config files which also makes it conform to the philosophy of the wrapper (especially with use with swif)

premade files

v1.3.1 comes with it a couple of extensions to features already present as well as some minor bug fixes

If you have your own "generator level" hddm file you wish to pass into geant/smearing/reconstruction you can now by modifying the GENERATOR variable in the configuration file. Simply put GENERATOR=file:/path/to/file/myfile.hddm to bypass generation

Also MCwrapper now supports the use of a configuration file for the hd_root step. Simply tell the MCwrapper's config file CUSTOM_PLUGINS=file:/path/to/file/myconfiguration.cfg and it will be used. Note: this removes all of the default plugins, thus if you don't tell it to run danarest you won't get danarest output.