This repository has been archived by the owner on Oct 25, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 1
The Gold Allocation manager
License
UCL-RITS/Gold
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
NAME/VERSION @PACKAGE_NAME@ @PACKAGE_VERSION@ IN BRIEF Gold is an open source accounting system that tracks and manages resource usage on High Performance Computers. It acts much like a bank in which resource credits are deposited into accounts with access controls designating which users, projects and machines may access the account. As jobs complete or as resources are utilized, accounts are charged and resource usage recorded. Gold supports familiar operations such as deposits, withdrawals, transfers and refunds. It provides balance and usage feedback to users, managers, and system administrators. Since accounting needs vary widely from organization to organization, Gold has been designed to be extremely flexible, featuring customizable accounting and supporting a variety of accounting models. Attention has been given to scalability, security, and fault tolerance. Gold facilitates resource sharing between organizations or within a Grid by providing distributed accounting while preserving local site autonomy. Gold was originally developed at Pacific Northwest National Laboratory (PNNL) under the Scalable Systems Software (SSS) SciDAC project. It is now being actively maintained and enhanced by Adaptive Computing. STATUS This is a general release of Gold; the second release from Adaptive Computing. DESCRIPTION In order for an organization to efficiently use its high performance computers, it must be able to allocate resources to the users and projects that need them most in a manner that is fair and according to mission objectives. Tracking the historical resource usage allows for insightful capacity planning and in making decisions on how to best mete out these resources. It allows the funding sources that have invested heavily in a supercomputing resource a means to show that it is being utilized efficiently. Additionally, accounting and allocation management are critical to being able to take advantage of the tremendous utilization gains afforded by meta-scheduling. The Gold Accounting and Allocation Manager tracks and manages job and resource usage. Much like a bank, an allocation manager associates a cost to computing resources and allows resource credits to be allocated to users and projects and meted out in a fair and judicious manner. As jobs complete or as resources are utilized, projects are dynamically charged and resource usage recorded. Gold was originally developed at PNNL as open source software under the Scalable Systems Software (SSS) SciDAC project. A standard communication protocol has been designed to facilitate communication between resource management and accounting components and to allow component substitution. A flexible GUI exists to simplify use and the management of project and accounting data. FEATURES · Rather than post-processing resource usage records on a periodic basis to rectify project balances, acounts are updated immediately at job completion. · A reservation is placed against the account for the estimated number of resource credits before the job runs, followed by an appropriate charge at the moment the job completes, thereby preventing projects from using more resources than were allocated to them. · A uniquely flexible account design allows resource credits to be allocated to specific projects, users and machines. · Resource credits may be restricted for use within a designated time period allowing sites to implement a use-it-or-lose-it policy to prevent year-end resource exhaustion and establishing a project cycle. · The system can track and charge for composite resource usage (memory, disk, CPU, etc) and custom charge multipliers can be applied (Quality of Service, Node Type, Time of Day, etc). · Users and resource brokers can determine ahead of time the cost of using resources via a guaranteed quote mechanism. · Accounts feature an optional credit limit allowing support for both debit and credit models. This feature can also be used to enable overdraft protection for specific accounts. · Accounts may be nested creating an account hierarchy. This allows for the delegation of management responsibilities, the establishment of automatic rules for the distribution of downstream resource credits, and the option of making higher level credits available to lower level accounts. · Gold supports a powerful querying and update mechanism that facilitates flexible reporting and streamlines administrative tasks. · Gold allows the establishment of default projects, machines and users. Additionally Gold can allow user, machines and projects to be automatically created the first time they are seen by the resource management system. These features allow job submitters to use the system without even knowing it. · Gold provides fine-grained (instance-level) Role Based Access Control for all operations. · Sites can create or modify record types on the fly enabling them to meet their custom accounting needs. Dynamic object creation allows sites to customize the types of accounting data they collect without modifying the code. This capability turns this system into a generalized information service. This capability is extremely powerful and can be used to manage all varieties of custom data. · Gold implements a journaling mechanism that preserves the indefinite historical state of all objects and records. This powerful mechanism allows historical bank statements to be generated, provides an undo/redo capability and allows commands to be run as if it were any arbitrary time in the past. · A traceback mechanism can allow all parties of a transaction (resource requestor and provider) to have a first-hand record of the resource utilization and to have a say as to whether or not the job should be permitted to run, based on their independent policies and priorities. A job will only run if all parties are agreeable to the idea that the target resources can be used in the manner and amount requested. Support for traceback debits will facilitate the establishment of trust and exchange relationships between administrative domains. · The system was designed from the outset with goals of scalability, fault tolerance, security, portability and ease of use in mind. · A web-based graphical user interface can be used to simplify use of Gold's powerful features. · Being open source allows for site self-sufficiency, customizability and promotes community development and interoperability. LICENSE Gold has an open source license under copyright of PNNL as expressed in LICENSE. DISTRIBUTION TESTED CONFIGURATIONS To date Gold has been successfully tested in the following environments: OS Version Architecture Database Version Batch Interface --------- -------- ------------- -------- --------- --------------- RedHat 7.3 i386 Postgres 7.1 Maui RedHat 9.0 i386 SMP Postgres 7.3 Maui RedHat AS2.1 ia64 Postgres 7.4,8.0 LSF Fedora Core 2 Athlon Postgres 7.4 Suse 9.0 Athlon i586 Postgres 7.3 AIX 5.1 RS/6K Power4 Postgres 7.3 Tru64 4.0d Alpha Postgres 7.4 OS X 7.5.0 Mac Powerpc Postgres 8.0 HP-UX 11.0 HP J5600 Postgres 7.3 IRIX 6.5 SGI O2 Postgres 7.3 SunOS 5.8 SUN Ultra10 Postgres 7.3 RedHat 7.3 i386 SQLite 3.0 RedHat 7.3 i386 MySQL 3.23,4.1 REQUIREMENTS You will first need to build, test and install the following prerequisites: Software Name and Version Where to Obtain Necessity ----------------------------------- -------------------------------- --------- Perl 5.8 or higher w/ suidperl http://www.perl.com/ Required a tested Database (Postgres 7.2+) ftp://ftp.postgresql.org/pub/ Required LibXML2 (2.4.25 or higher) http://www.xmlsoft.org/ Required Apache Httpd Server (2.0 or higher) http://httpd.apache.org/ Optional OpenSSL (0.9.5a or higher) http://www.openssl.org/ Optional mod_ssl (2.26 or higher) http://www.modssl.org/ Optional Gnu ReadLine (2.0 or higher) ftp://ftp.gnu.org/gnu/ Optional Please reference the hints in INSTALL. INSTALLATION Installation instructions are contained in INSTALL. The high level steps are: 1) Prepare to build and install Gold by unpacking the archive and creating an appropriate user to manage Gold. 2) Install the prerequisites. These include Perl 5.6 (with suidperl), libxml2 and PostgreSQL 7. Apache Httpd server, OpenSSL and mod_ssl are needed if you want to use the web GUI. Gnu readline is optional. 3) Perform pre-install configurations by running the "configure" script. 4) Compile the program with "make". Also use "make gui" if you intend to install the web GUI. 4) Install the prerequisite Perl modules with "make deps". 5) Install Gold with "make install". Generate a secret key with "make auth_key". Install the gui with "make install-gui" if necessary. 6) Perform general setup by editing Gold config files and setting environment variables. 7) Database setup includes insuring postgres allows internet domain sockets, authorizing postgres to connect to your host, defining users, creating the gold database and initially populating the database. 8) Web-Server setup (optional) involves modifying the config file to allow SSL connections, creating a certificate keystore, and getting httpd started up. 9) Startup the gold server daemon (goldd). 10) Initialize Gold by defining users, projects, accounts, etc as necessary. TESTING There are no built-in tests for Gold at this time. Automated and manual testing is conducted at Adaptive Computing Enterprises, Inc. PROJECT WEB PAGE The current project web page is at Adaptive Computing: <http://www.clusterresources.com/products/gold>. DOCUMENTATION A Gold User's Guide is available in the docs directory both in pdf and html formats. Each of the perl clients have built-in usage and man-page documentation accessible via the --help and --man flags. Some documents, such as a Requirements document, communication protocol docs, presentations, etc. may be found at the Adaptive Computing Gold Project Documentation page at <http://www.clusterresources.com/products/gold/docs/>. COMPATIBILITY For semantic purposes let's assume VRMF nomenclature (Version, Release, Maintenance level, Fix level), such that Gold 2.7.3.2 represents Version 2, Release 7, Maintenance level 3, Fix level 2. Gold Release numbers are changed when a change to the database format is made, and hence different Gold Releases are not compatible. Different Gold Maintenance Levels should however be compatible and may coexist and operate on a single database. CHANGES Changes are enumerated in the CHANGES file. PROBLEM REPORTING You may send comments and bug reports to <gold-support@adaptivecomputing.com>. Please include as much detail as possible in order to expedite a resolution. Be advised that this queue will be handled on a best-effort basis unless a support contract is established with Adaptive Computing. DISTRIBUTION LIST A majordomo list has been setup for Gold contributors, administrators and users in <gold-users@supercomputing.org>. Anyone may subscribe and then post to the list. AUTHORS Gold has been developed by Scott Jackson <scottmo@adaptivecomputing.com> while at Pacific Northwest National Laboratory. It continues to be enhanced and maintained by Scott Jackson and other staff at Adaptive Computing.
About
The Gold Allocation manager
Resources
License
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published