Skip to content

PyCOMPSs AutoParallel v1.2

Latest
Latest
Compare
Choose a tag to compare
@cristianrcv cristianrcv released this 27 Aug 08:20
v1.2
281c706
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

PyCOMPSs AutoParallel

The PyCOMPSs AutoParallel module.

The implementation includes:

  • The @parallel() decorator for Python to run applications with PyCOMPSs
  • Different Translator modules to automatically generate parallel Python code
    • Py2Scop Translator: Translation of Python code into OpenScop format
    • Scop2PScop2Py Translator: Integration with the PLUTO tool to generate
      possible parallelizations of the given code. The output is written in Python
      (by means of a CLooG extension) with parallel annotations following an
      OMP fashion.
    • Py2PyCOMPSs Translator: Translation of parallel annotated Python code into
      PyCOMPSs task definitions.
  • A Code Replacer module to replace the user code by autogenerated code.
  • Several example applications (examples/ folder).

Table of Contents

Dependencies

Software Dependencies

COMPSs: The COMP Superscalar (COMPSs) framework is mainly compose of a
programming model which aims to ease the development of applications for distributed
infrastructures, such as Clusters, Grids and Clouds and a runtime system that exploits
the inherent parallelism of applications at execution time. The framework is
complemented by a set of tools for facilitating the development, execution monitoring
and post-mortem performance analysis. It natively supports Java but has bindings for
Python, C and C++.

PLUTO: PLUTO is an automatic parallelization tool based on the polyhedral
model. The polyhedral model for compiler optimization provides an abstraction to
perform high-level transformations such as loop-nest optimization and parallelization
on affine loop nests. Pluto transforms C programs from source to source for
coarse-grained parallelism and data locality simultaneously. The core transformation
framework mainly works by finding affine transformations for efficient tiling. OpenMP
parallel code for multicores can be automatically generated from sequential C program
sections. The outer (communication-free), inner, or pipelined parallelization is achieved
purely with OpenMP parallel for pragmas.

Included Dependencies inside PyCOMPSs

DILL: It is recommended to install the DILL Python package for better
serialization

Included Dependencies inside PLUTO

OpenSCOP: OpenScop is an open specification defining a file format and a set
of data structures to represent a static control part (SCoP for short), i.e., a program
part that can be represented in the polyhedral model. The goal of OpenScop is to
provide a common interface to various polyhedral compilation tools in order to
simplify their interaction. The OpenScop aim is to provide a stable, unified format
that offers a durable guarantee that a tool can use an output or provide an input
to another tool without breaking a toolchain because of some internal changes in one
element of the chain. The other promise of OpenScop is the ability to assemble or
replace the basic blocks of a polyhedral compilation framework at no, or at least
low engineering cost.

CLooG: CLooG is a free software and library to generate code for scanning
Z-polyhedra. That is, it finds a code (e.g. in C, FORTRAN...) that reaches each
integral point of one or more parameterized polyhedra. CLooG has been originally
written to solve the code generation problem for optimizing compilers based on the
polytope model. Nevertheless, it is used now in different areas e.g. to build control
automata for high-level synthesis or to find the best polynomial approximation of
a function. CLooG may help in any situation where scanning polyhedra matters. While
the user has full control of generated code quality, CLooG is designed to avoid
control overhead and to produce a very effective code.

Python Module Dependencies

Extra Dependencies

Commands

Examples

The examples/ folder contains a folder per example application.

Each example application contains:

  • README.md: Description of the application and the generated
    task graphs.
  • run.sh: Script to launch all the versions of the application (internally calls
    the run.sh script of each version of the application).
  • A folder per version of the application (for instance, userparallel
    or autoparallel).

Each version of an example application contains:

  • results: Folder containing the execution results
  • run.sh: Script to launch the application locally using 4 cores
  • *.py: Python source files of the application

Some important notes:

  • All the run.sh scripts work without parameters. However, advanced users can call
    the scripts with extra parameters that are appended directly to the runcompss command.
  • The autoparallel versions contain a special *_autogen.py file containing
    the automatically generated code by the COMPSs AutoParallel module. This
    file is only saved for clarity purposes but is re-generated every time
    the application is launched.
  • The examples/ folder contains a run.sh script to run all the available
    applications locally using 4 cores. It displays a result table at the end of the
    execution.

Test

With debug mode enabled:

export PYTHONPATH=${git_base_dir}
cd pycompss
python nose_tests.py -s

With debug mode disabled:

export PYTHONPATH=${git_base_dir}
cd pycompss
python -O nose_tests.py

Coverage

Run coverage:

./coverage_run.sh

Upload coverage:

echo "YOUR_TOKEN" > .CODACY_PROJECT_TOKEN
echo "YOUR_TOKEN" > .CODECOV_PROJECT_TOKEN

./coverage_upload.sh

Style

This project follows the PyCodeStyle guide (formerly called pep8).

This project tolerates the following relaxations:

  • E501 line too long : Code lines can be up to 120 characters

You can verify the code style by running:

pycodestyle . --max-line-length=120

Clean

find . -name "*.pyc" -delete
find . -name "*.pyo" -delete

Contributing

All kinds of contributions are welcome. Please do not hesitate to open a new issue,
submit a pull request or contact the author if necessary.

Author

Cristián Ramón-Cortés Vilarrodona <cristian.ramoncortes(at)bsc.es> (Personal Website)

This work is supervised by:

  • Rosa M. Badia (BSC)
  • Philippe Clauss (INRIA)
  • Jorge Ejarque (BSC)

Disclaimer

This is part of a collaboration between the CAMUS Team at INRIA and
the Workflows and Distributed Computing Team at BSC and is still
under development.

License

Licensed under the Apache 2.0 License

Assets 2
Loading