http://ab-initio.mit.edu/wiki/index.php?title=Parallel_Meep&action=history&feed=atomParallel Meep - Revision history2024-03-29T11:37:03ZRevision history for this page on the wikiMediaWiki 1.7.3http://ab-initio.mit.edu/wiki/index.php?title=Parallel_Meep&diff=4870&oldid=prevArdavan: /* Technical Details */2016-09-06T04:30:22Z<p><span class="autocomment">Technical Details</span></p>
<table border='0' width='98%' cellpadding='0' cellspacing='4' style="background-color: white;">
<tr>
<td colspan='2' width='50%' align='center' style="background-color: white;">←Older revision</td>
<td colspan='2' width='50%' align='center' style="background-color: white;">Revision as of 04:30, 6 September 2016</td>
</tr>
<tr><td colspan="2" align="left"><strong>Line 50:</strong></td>
<td colspan="2" align="left"><strong>Line 50:</strong></td></tr>
<tr><td> </td><td style="background: #eee; font-size: smaller;"></td><td> </td><td style="background: #eee; font-size: smaller;"></td></tr>
<tr><td> </td><td style="background: #eee; font-size: smaller;">'''Warning''': do not attempt to perform different Meep commands in different processes by using the <code>(meep-my-rank)</code>. All processes must (for the most part) execute the same Meep commands in the same sequence or they will deadlock, waiting forever for one another.</td><td> </td><td style="background: #eee; font-size: smaller;">'''Warning''': do not attempt to perform different Meep commands in different processes by using the <code>(meep-my-rank)</code>. All processes must (for the most part) execute the same Meep commands in the same sequence or they will deadlock, waiting forever for one another.</td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;"></td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;">For large multicore jobs with I/O, it may be necessary to have <code>(meep-all-wait)</code> as the last line in the <code>.ctl</code> file to ensure that all processors terminate at the same point in the execution. Otherwise, one processor may finish and abruptly terminate the other processors.</td></tr>
</table>
Ardavanhttp://ab-initio.mit.edu/wiki/index.php?title=Parallel_Meep&diff=2073&oldid=prevStevenj: /* Using Parallel Meep */2006-04-03T02:12:32Z<p><span class="autocomment">Using Parallel Meep</span></p>
<table border='0' width='98%' cellpadding='0' cellspacing='4' style="background-color: white;">
<tr>
<td colspan='2' width='50%' align='center' style="background-color: white;">←Older revision</td>
<td colspan='2' width='50%' align='center' style="background-color: white;">Revision as of 02:12, 3 April 2006</td>
</tr>
<tr><td colspan="2" align="left"><strong>Line 21:</strong></td>
<td colspan="2" align="left"><strong>Line 21:</strong></td></tr>
<tr><td> </td><td style="background: #eee; font-size: smaller;"></td><td> </td><td style="background: #eee; font-size: smaller;"></td></tr>
<tr><td> </td><td style="background: #eee; font-size: smaller;">There is one important requirement: every MPI process must be able to read the <code>foo.ctl</code> input file (or whatever your control file is called). On most systems, this is no problem, but if for some reason your MPI processes don't all have access to the local filesystem then you may need to make copies of your input file or something.</td><td> </td><td style="background: #eee; font-size: smaller;">There is one important requirement: every MPI process must be able to read the <code>foo.ctl</code> input file (or whatever your control file is called). On most systems, this is no problem, but if for some reason your MPI processes don't all have access to the local filesystem then you may need to make copies of your input file or something.</td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;"></td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;">(You cannot run Meep interactively on multiple processors.)</td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;"></td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;">===Different forms of parallelization===</td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;"></td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;">Parallel Meep works by taking your simulation and dividing the computational cell among the MPI processes. This is the only way of parallelizing a single simulation, and allows you to attack very large problems.</td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;"></td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;">However, there is an alternative strategy for parallelization. If you have many smaller simulations that you want to run, say for many different values of some parameter, then you can just run these as separate jobs. Such parallelization is known as "embarrassingly parallel" because no communication is required. Meep provides no explicit support for this mode of operation, but of course it is quite easy to do yourself: just launch as many Meep jobs as you want, perhaps [[libctl User Reference#Command-line parameters|changing the parameters via the command-line]] using a shell script.</td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;"></td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;">==Technical Details==</td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;"></td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;">When you run Meep under MPI, the following is a brief description of what is happening behind the scenes. For the most part, you shouldn't ''need'' to know this stuff...just use the same <code>.ctl</code> exactly as you would for a uniprocessor simulation.</td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;"></td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;">First, every MPI process executes the <code>.ctl</code> file in parallel. The processes communicate however, to only perform one simulation in sync with one another. In particular, the computational cell is divided into "chunks", one per process, to roughly equally divide the work and the memory.</td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;"></td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;">When you time-step (via <code>run-until</code> or whatever), the chunks are time-stepped in parallel, communicating the values of the pixels on their boundaries with one another. In general, any Meep function that performs some collective operation over the whole computational cell or a large portion thereof is parallelized, including: time-stepping, HDF5 I/O, accumulation of flux spectra, and field integration (via <code>integrate-field-function</code>, although the ''results'' are communicated to all processes).</td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;"></td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;">Computations that only involve isolated points, such as <code>get-field-point</code> or <code>harminv</code> analyses, are performed by all processes redundantly. (In the case of <code>get-field-point</code>, Meep figures out which process "knows" the field at the given field, and then sends the field value from that process to all other processes.) This is harmless because such computations are rarely performance bottlenecks.</td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;"></td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;">Although all processes execute the <code>.ctl</code> in parallel, <code>print</code> statements are ignored for all process but one (process #0). In this way, you only get one copy of the output.</td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;"></td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;">If for some reason you need to distinguish different MPI processes in your <code>.ctl</code> file, you can use the following two functions:</td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;"></td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;">; (meep-count-processors)</td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;">: Returns the number of processes that Meep is using in parallel.</td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;">; (meep-my-rank)</td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;">: Returns the index of the process running the current <code>.ctl</code> file, from zero to <code>(meep-count-processors)</code>&nbsp;&ndash;&nbsp;1.</td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;"></td></tr>
<tr><td colspan="2"> </td><td>+</td><td style="background: #cfc; font-size: smaller;">'''Warning''': do not attempt to perform different Meep commands in different processes by using the <code>(meep-my-rank)</code>. All processes must (for the most part) execute the same Meep commands in the same sequence or they will deadlock, waiting forever for one another.</td></tr>
</table>
Stevenjhttp://ab-initio.mit.edu/wiki/index.php?title=Parallel_Meep&diff=793&oldid=prevStevenj at 19:13, 2 April 20062006-04-02T19:13:14Z<p></p>
<table border='0' width='98%' cellpadding='0' cellspacing='4' style="background-color: white;">
<tr>
<td colspan='2' width='50%' align='center' style="background-color: white;">←Older revision</td>
<td colspan='2' width='50%' align='center' style="background-color: white;">Revision as of 19:13, 2 April 2006</td>
</tr>
<tr><td colspan="2" align="left"><strong>Line 1:</strong></td>
<td colspan="2" align="left"><strong>Line 1:</strong></td></tr>
<tr><td> </td><td style="background: #eee; font-size: smaller;">{{Meep}}</td><td> </td><td style="background: #eee; font-size: smaller;">{{Meep}}</td></tr>
<tr><td>-</td><td style="background: #ffa; font-size: smaller;">Meep supports distributed-memory parallelism using [[w:MPI|MPI]]. This allows it to scale up from small dual-processor machines to large parallel supercomputers, and to work on very large problems that may not even fit into the memory of one machine. We've run it using hundreds of processors<span style="color: red; font-weight: bold;">; of </span>course, your problem must be sufficiently large in order to benefit from many processors.</td><td>+</td><td style="background: #cfc; font-size: smaller;">Meep supports distributed-memory parallelism using [[w:MPI|MPI]]. This allows it to scale up from small dual-processor machines to large parallel supercomputers, and to work on very large problems that may not even fit into the memory of one machine. We've run it using <span style="color: red; font-weight: bold;">at least </span>hundreds of processors<span style="color: red; font-weight: bold;">. Of </span>course, your problem must be sufficiently large in order to benefit from many processors.</td></tr>
<tr><td> </td><td style="background: #eee; font-size: smaller;"></td><td> </td><td style="background: #eee; font-size: smaller;"></td></tr>
<tr><td> </td><td style="background: #eee; font-size: smaller;">==Installing Parallel Meep==</td><td> </td><td style="background: #eee; font-size: smaller;">==Installing Parallel Meep==</td></tr>
</table>
Stevenjhttp://ab-initio.mit.edu/wiki/index.php?title=Parallel_Meep&diff=791&oldid=prevStevenj at 19:12, 2 April 20062006-04-02T19:12:48Z<p></p>
<table border='0' width='98%' cellpadding='0' cellspacing='4' style="background-color: white;">
<tr>
<td colspan='2' width='50%' align='center' style="background-color: white;">←Older revision</td>
<td colspan='2' width='50%' align='center' style="background-color: white;">Revision as of 19:12, 2 April 2006</td>
</tr>
<tr><td colspan="2" align="left"><strong>Line 1:</strong></td>
<td colspan="2" align="left"><strong>Line 1:</strong></td></tr>
<tr><td> </td><td style="background: #eee; font-size: smaller;">{{Meep}}</td><td> </td><td style="background: #eee; font-size: smaller;">{{Meep}}</td></tr>
<tr><td>-</td><td style="background: #ffa; font-size: smaller;">Meep supports distributed-memory parallelism using [[w:MPI|MPI]]. This allows it to scale up from small dual-processor machines to large parallel supercomputers, and to work on very large problems that may not even fit into the memory of <span style="color: red; font-weight: bold;">system</span>. We've run it using hundreds of processors; of course, your problem must be sufficiently large in order to benefit from many processors.</td><td>+</td><td style="background: #cfc; font-size: smaller;">Meep supports distributed-memory parallelism using [[w:MPI|MPI]]. This allows it to scale up from small dual-processor machines to large parallel supercomputers, and to work on very large problems that may not even fit into the memory of <span style="color: red; font-weight: bold;">one machine</span>. We've run it using hundreds of processors; of course, your problem must be sufficiently large in order to benefit from many processors.</td></tr>
<tr><td> </td><td style="background: #eee; font-size: smaller;"></td><td> </td><td style="background: #eee; font-size: smaller;"></td></tr>
<tr><td> </td><td style="background: #eee; font-size: smaller;">==Installing Parallel Meep==</td><td> </td><td style="background: #eee; font-size: smaller;">==Installing Parallel Meep==</td></tr>
</table>
Stevenjhttp://ab-initio.mit.edu/wiki/index.php?title=Parallel_Meep&diff=790&oldid=prevStevenj: added2006-04-02T19:08:51Z<p>added</p>
<p><b>New page</b></p><div>{{Meep}}<br />
Meep supports distributed-memory parallelism using [[w:MPI|MPI]]. This allows it to scale up from small dual-processor machines to large parallel supercomputers, and to work on very large problems that may not even fit into the memory of system. We've run it using hundreds of processors; of course, your problem must be sufficiently large in order to benefit from many processors.<br />
<br />
==Installing Parallel Meep==<br />
<br />
To install the parallel version of Meep, you must have a version of MPI installed on your system: see the [[Meep Installation#MPI (parallel machines)|installation manual]].<br />
<br />
We also ''strongly'' recommend installing the HDF5 with parallel I/O support if you are going to run with more than a few processors. (configure HDF5 with <code>--enable-parallel</code>; you may also have to set the <code>CC</code> environment variable to <code>mpicc</code>.) Unfortunately, the parallel HDF5 library then does not work with serial code, so you have may have to choose to install either the serial or the parallel Meep, but not both.<br />
<br />
If you don't install HDF5 with parallel I/O support, you can still do I/O from MPI &mdash; Meep has some hacks to let it write HDF5 files using serial I/O from multiple processes, one at a time. However, this does not scale very well to many processors. (We've also observed some MPI implementations to freeze under the strain of trying to write from many processes at once. YMMV).<br />
<br />
Then you just <code>configure</code> Meep <code>--with-mpi</code>. The <code>meep</code> executable is installed as <code>meep-mpi</code>, so that you can have both the serial and parallel versions installed at the same time.<br />
<br />
==Using Parallel Meep==<br />
<br />
The parallel version of Meep is designed to operate completely transparently: you use the same <code>.ctl</code> file as for the serial version, and the output is the same, but it is just faster (hopefully).<br />
<br />
In order to run MPI programs, you typically have to use a command like <code>mpirun</code> with an argument to say how many processes you want to use. (Consult your MPI documentation.) For example, with many popular MPI implementations, to run with 4 processes you would use something like:<br />
<br />
mpirun -np 4 meep-mpi foo.ctl > foo.out<br />
<br />
There is one important requirement: every MPI process must be able to read the <code>foo.ctl</code> input file (or whatever your control file is called). On most systems, this is no problem, but if for some reason your MPI processes don't all have access to the local filesystem then you may need to make copies of your input file or something.</div>Stevenj