Restart

[DanRRRR]

• Is this cone.deck demo taken from example_decks (from large SDF at least) restartable from any SDF or only from the last one defined by "force_final_to_be_restartable = T" ? But seems the "force_final_to_be_restartable = T" which i suppose automatically includes everything needed rto restart and continue is generated only ones at the end of simulation making it useless in case of crash.
• Please provide at least a single example of restart from intermediate time (suppose we had an outage). Take any examples of input.decks, for example code.deck below
• Are cone.deck restartable from the large outputs ? Are large outputs designed specifically for restarts? Then what minimum requirement of output has to be included there? It is almost impossible to be sure that for continuation you included into large SDF output everything needed for the code to continue.
• Are input.deck, restart.visit, normal.visit etc necessary to restart from any intermediate times ?
• Can we change something in the middle of simulation and restart from the intermediate output ? For example, can we change number of cores when continue run from restart ?

/***- Noticed that the first line (begin:control) in this example is missing ? This is GitHub problem, it actually is there when i try to edit the text. Or this was my problem, though i followed an advice i got here. To add this input.deck i used 3 grave accents, since one grave accent (suggested by GUI above when you prepare the question) does not work, it distorts the text. If you agree with me that this is a bug and not a feature I will stop complaining here and will ask the developers of GitHub, this is really laughable that they can not fix elementary things.

On a side note, may be good idea would be to add the "!" sign as a comment for the EPOCH input.deck in addition to "#" which is common in Fortran ?

  nx = 250
  ny = 250
  nz = 250
  nparticles = nx * ny * nz * 1.123

  # Final time of simulation
  t_end = 50 * femto

  # Size of domain
  x_min = -10 * micron
  x_max = -x_min
  y_min = x_min
  y_max = x_max
  z_min = x_min
  z_max = x_max
end:control


begin:boundaries
  bc_x_min = simple_laser
  bc_x_max = simple_outflow
  bc_y_min = periodic
  bc_y_max = periodic
  bc_z_min = periodic
  bc_z_max = periodic
end:boundaries


begin:constant
  lambda = 1.06 * micron
  omega = 2 * pi * c / lambda
  den_cone = 4.0 * critical(omega)
  th = 1 * micron / 2.0
  ri = abs(x - 5*micron) - sqrt(2.0) * th
  ro = abs(x - 5*micron) + sqrt(2.0) * th
  xi = 3*micron - th
  xo = 3*micron + th
  r = sqrt(y^2 + z^2)
end:constant


begin:species
  name = proton
  charge = 1.0
  mass = 1836.2
  fraction = 0.5
  number_density = if((r gt ri) and (r lt ro), den_cone, 0.0)
  number_density = if((x gt xi) and (x lt xo) and (r lt ri), \
                      den_cone, number_density(proton))
  number_density = if(x gt xo, 0.0, number_density(proton))
end:species


begin:species
  name = electron
  charge = -1.0
  mass = 1.0
  fraction = 0.5
  number_density = number_density(proton)
end:species


begin:laser
  boundary = x_min
  amp = 1e13
  lambda = lambda
  profile = gauss(r, 0.0, 2.5*micron)
end:laser


begin:output_global
  force_final_to_be_restartable = T
end:output_global


begin:output
  name = normal

  # Simulated time between output dumps
  dt_snapshot = 1 * femto

  # Properties on grid
  grid = always
  ey = always
  average_particle_energy = always
  number_density = always + species
  temperature = always + species

  # Extended io
  distribution_functions = always
end:output


begin:output
  name = large

  # Simulated time between output dumps
  dt_snapshot = 10 * femto

  # Properties at particle positions
  particles = always
  particle_weight = always

  charge_density = always
end:output


begin:dist_fn
  name = x_px
  ndims = 2

  direction1 = dir_x
  direction2 = dir_px

  # Range is ignored for spatial coordinates
  range1 = (1, 1)
  range2 = (-3.0e-20, 3.0e-20)

  # Resolution is ignored for spatial coordinates
  resolution1 = 1
  resolution2 = 100

  include_species:electron
  include_species:proton
end:dist_fn


begin:dist_fn
  name = x_px_py
  ndims = 3
  dumpmask = never

  direction1 = dir_x
  direction2 = dir_px
  direction3 = dir_py

  # Range is ignored for spatial coordinates
  range1 = (1, 1)
  range2 = (-3.0e-20, 3.0e-20)
  range3 = (-3.0e-20, 3.0e-20)

  # Resolution is ignored for spatial coordinates
  resolution1 = 1
  resolution2 = 100
  resolution3 = 100

  include_species:electron
  include_species:proton
end:dist_fn```

 

[Status-Mirror]

• Yes, in this input deck, only the final SDF file would be restartable. This would only be useful in continuing the simulation where you left off, and would not be useful if the simulation crashed while running. For more regular checkpoints to protect against crashes, see the "Multiple output blocks" section of the output block documentation.
• Restarting from an intermediate restart dump is the same as any other code restart. Simply add the restart_snapshot key to the control block, as discussed in the control block documentation.
• You can't restart from any large dump, only a restart dump. These output the time, all the fields, all the current densities, and all the particle variables. Everything the code would need to initialise a simulation to the point where the restart dump was written.
• input.deck is needed for a restart (as you need to specify the restart snapshot), but the .visit files are not.
• The code may be able to use a different number of cores on restart, but if it can, this isn't something which has been tested (as far as I'm aware).
• I think the 3 grave accents need to be on their own line, like this:

```
begin:control
...
end:output
```

to write this:

  begin:control
  ...
  end:output

[DanRRRR]

Thanks Stuart.

In the Output block was many many many options about restart and no single hint how to start the code using these options and only you for the first time mentioned that this is described in other place, in control block . I tried before to find this myself and opened the search in the EPOCH documentation and got reference on 45 topics essentially on all numerous EPOCH blocks. Search suggested to read the entire doc :))))

Curiosity for i counted the number of words in the online documentation: 59,138 words
For comparison "Hamlet", Shakespeare's longest play, has 29,551 words :)))
So before you read and memorize the entire documentation you will not be able to succeed with the restart. As well as anything else i suspect.

My major suggestions always were : please provide more example on everything. Much more than you already have. On restart good to provide at least 3 of them using different options. And of course provide the reference on restart_snapshot in output block.

[DanRRRR]

Examples are needed because with so many conflicting options user will stop trying anything very quickly. I spent half of the day even after your help. Had a lot of questions. For example

• is this allowed to put "restart_dump_every = 10" into normal output block above?
• ... or into large output block above?
• ...or only into additional block called full ?
• And is it allowed to put it into output_global block, the description of this block does not mention that this is allowed or disallowed ?
• Also is restart_dump_every = 10 is enough and additional keys have to be set ?
• Or none the above and an additional output block has to be made and the name set to *name = restart ? That conclusion also can be made from reading documentation.

begin:output
  name = restart
  restart_dump_every = 10
end:output 

But after making such block i got Element "restart_dump_every" not allowed in a named output block. It has been ignored." .
Then i tried to place restart_dump_every = 10 line in other output blocks. Same result...
So i gave up...nothing works. Restart in the documentation and possibly in the implementation too is total disaster. Because nobody ever prepared any demo and hence did not check restart the the bugs.

• One more comment. Somebody may smile, but with rolling_restart will the code write all the time into the same two spots of the media and make a "hole boring" there? Some media quickly deteriorate like the NVMe flash drives memory T or specifically Q cells of which do not survive few hundreds of rewrite cycles. Being busy with the run and massive writes not all NVMes will take care the wear off timely. I had problems ones when the whole system was off for half a day and only then automatically recovered. You do not care about this if your run on multi-user supercomputers but with EPOCH being very fast you can run some tasks on your own HPC and then you start to care (run on your PC/server is very very convenient by the way) .

[Status-Mirror]

Hey Dan,

I have included some basic examples, detailing a few different ways EPOCH can run with restart dumps. The new documentation can be found here.

I'm happy to extend this documentation if there are parts you find aren't clear enough. Remember that this is only intended to be an overview to get people started, the main documentation remains in the output block sections.

Cheers,
Stuart