-
Notifications
You must be signed in to change notification settings - Fork 2
/
Copy pathREADME.txt
275 lines (203 loc) · 11.4 KB
/
README.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
---------------------------------------------------------
Stonith Plugin Agent for VMWare VM vCenter SOAP Fencing
---------------------------------------------------------
Author:
Olaf Reitmaier <olafrv@gmail.com>
Version:
16/Jan/2015
License:
GNU General Public License (GPL) v3
http://www.gnu.org/licenses/gpl.html
------------------------------------------
Tested on the following virtual platform
------------------------------------------
- Ubuntu Linux Server Edition 14.04 64 bits (Pacemaker/Heartbeat).
- SUSE Enterprise Linux 11 SP3 64 bits (Pacemaker/Heartbeat).
- SUSE Enterprise Linux 11 SP4 64 bits (Pacemaker/Corosync).
- VMWare ESXi/vCenter 5.1 U1 (Used as Fencing Device)
---------
History
---------
- 14/Jan/2015 - Testing on Ubuntu Linux 14.04 64 bits.
- 15/Jan/2015 - Implementation and testing on SLES Linux 11 SP3 64 bits.
- 25/Oct/2015 - Testing on SLES Linux 11 SP4 64 bits.
-----------------
Plugin Workflow
-----------------
1. stonithd (Cluster Fencing Daemon)
2. /usr/lib/stonith/plugins/external/fence_vmware_soap (Stonith Plugin Agent)
3. /usr/sbin/fence_vmware_soap (SOAP Fence Request, provided by fence-agents)
4. VMWare vCenter (SOAP Web Service, Authentication, Search, Triggering)
5. VMWare ESXi Hypervisor (Virtual Machine On/Off)
--------
Notice
--------
Right now the plugin not permits declaring two different VMWare vCenter
devices for fencing the same list of cluster nodes. But could be tested and
implemented changing the attribute "unique" to "false" for the "hostlist"
parameter in the plugin XML definition schema.
--------------
Requirements
--------------
- Fully functional Linux operating system (VMWare Virtual Machines).
- Fully functional Pacemaker/Heartbeat cluster (Know How About It).
------------------------------
Packages needed Ubuntu 14.04
------------------------------
fence-agents in its default version 3.1.5 and the python suds library:
apt-get install python-suds fence-agents
-----------------------------
Packages needed SUSE 11 SP3
-----------------------------
fence-agents, version 3.1.6 is provided by default but a newer version is needed 4.0 included in OpenSUSE 13.2:
http://software.opensuse.org/package/fence-agents?search_term=fence-agents
In order to download the package execute the following command:
wget http://download.opensuse.org/repositories/openSUSE:/13.2/standard/x86_64/fence-agents-4.0.10-2.4.1.x86_64.rpm
Many fence agents depend on Python libraries, some could be installed directly:
zypper install python-curl python-openssl python-pexpect python-request
And other must be installed using python setup tools (https://pypi.python.org/pypi/setuptools) as follows:
wget https://bootstrap.pypa.io/ez_setup.py
python ez_setup.py
Now you can install the required libraries "request" and "suds" from the official repositories:
easy_install requests suds
Now cleanly you can install the "fence-agents" package downloaded previously:
rpm -i --nodeps fence-agents-4.0.10-2.4.1.x86_64.rpm
-----------------------------
Packages needed SUSE 11 SP4
-----------------------------
fence-agents, version 4.0.12 is provided by SUSE Linux Enterprise High Availability Extension 11 SP4:
https://www.suse.com/products/highavailability/download/
In order to just install the package via the YaST2 repositories (if the HA extension is enabled):
zypper install fence-agents
Many fence agents depend on Python libraries, available with SLES 11 SP4 (and/or the HA extension):
zypper install python-curl python-openssl python-pexpect python-requests python-suds
Manual installation based on SLES-11-SP4-DVD-x86_64-GM-DVD1.iso and SLE-HA-11-SP4-x86_64-GM-CD1.iso:
fence-agents-4.0.12-1.45.x86_64.rpm -> SLE-HA-11-SP4-x86_64-GM-CD1.iso
python-curl-7.19.0-5.2.1.2.x86_64.rpm -> SLES-11-SP4-DVD-x86_64-GM-DVD1.iso
python-openssl-0.7.0-1.17.2.x86_64.rpm -> SLES-11-SP4-DVD-x86_64-GM-DVD1.iso
python-pexpect-3.1-0.7.1.x86_64.rpm -> SLE-HA-11-SP4-x86_64-GM-CD1.iso
python-requests-2.0.1-0.9.37.x86_64.rpm -> SLES-11-SP4-DVD-x86_64-GM-DVD1.iso
python-suds-0.4-0.20.1.x86_64.rpm -> SLE-HA-11-SP4-x86_64-GM-CD1.iso
Unfortunately fence-agents-4.0.12-1.45 contains a bug resulting in the following behaviour/output:
$ fence_vmware_soap -a <VCenterIP> -l "<VCenterUser>" -p "<VCenterPassword>"
Traceback (most recent call last):
File "/usr/sbin/fence_vmware_soap", line 257, in <module>
main()
File "/usr/sbin/fence_vmware_soap", line 223, in main
options_global = check_input(device_opt, process_input(device_opt))
File "/usr/share/fence/fencing.py", line 721, in check_input
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stderr))
TypeError: __init__() got an unexpected keyword argument 'stream'
$
A non-update safe fix for this bug can be easily applied using the command
sed -e 's/stream=sys.stderr/sys.stderr/' -i /usr/share/fence/fencing.py
while the unified diff of this fix can be seen at upstream at:
https://github.com/ClusterLabs/fence-agents/commit/b914f75f5c467f7faf1184e786b44f74ba8dddb4
---------------------
Plugin Installation
---------------------
*** CAUTION: Use the version 15/01/2015 of fence_vmware_soap stonith plugin agent,
which includes compatibility for the fence_vmware_soap script provided
by both fence-agents versions (3.X and 4.X). ***
Copy the fence_vmware_soap.sh stonith plugin agent to ONE of the following EXISTENT
directories with the following NAME (Strip the .sh extension):
- /usr/lib/stonith/plugins/external/fence_vmware_soap
- /usr/lib64/stonith/plugins/external/fence_vmware_soap
For more information, about the stonith plugin agents visit the following links:
- 8.1. STONITH Agents: https://www.suse.com/documentation/sle_ha/book_sleha/data/sec_ha_stonithagents.html
- External STONITH Plugins: http://www.linux-ha.org/ExternalStonithPlugins
Give the following permissions:
chmod 755 /usr/lib/stonith/plugins/external/fence_vmware_soap
Check correct plugin installation and detection by stonith:
stonith -L | grep fence_vmware_soap
The output should be:
external/fence_vmware_soap
Check correct parameter listing for the plugin agent:
stonith -t "external/fence_vmware_soap" -n
The output should be:
hostlist vcenterip username password
----------------------
Plugin Configuration
----------------------
Extract the cluster node VMWare VM UUID BIOS (Case Sensitive Output) on each virtual machine:
dmidecode | grep -i uuid | awk '{print $2}' | tr '[:upper:]' '[:lower:]'
The output should be something like this (and different for each cluster node):
4233cc22-770f-3027-c090-889054979c45
Extract the cluster node names (the same defined in the cluster configuration, example: node1, node2):
uname -n
Determine the VMWare vCenter IPv4 address controlling the VMWare ESX Hypervisor Hosts
serving the Virtual Machines:
<VCenterIP> (Ask the virtualization platform administrator)
Obtain the VMWare VCenter credentials (VCenterUser and VCenterPassword) with fencing permissions,
for more information look at:
"What user permissions/roles are required for the VMware vCenter
user account to perform fence action using fence_vmware_soap?"
https://access.redhat.com/solutions/82333
-------------
Plugin Test
-------------
*** NOTICE: The following test DO NOT require cluster configuration modifications!. ***
First, test the correct function of the command "fence_vmware_soap" provided
by package "fence-agents" issuing the following command (CAUTION: Cluster nodes will be restarted!):
On fence-agents 3.X issuing the command:
fence_vmware_soap -o reboot -a <VCenterIP> -l "<VCenterUser>" -p "<VCenterPassword>" -z -U "<UUID>"
On fence-agents 4.X issuing the command:
fence_vmware_soap -o reboot -a <VCenterIP> -l "<VCenterUser>" -p "<VCenterPassword>" -z --ssl-insecure -n "<UUID>"
Second, test fencing plugin agent (CAUTION: Cluster nodes will be restarted):
stonith -t "external/fence_vmware_soap" hostlist="node1,uuid1;node2,uuid2" \
vcenterip="<VCenterIP>" username="<VCenterUser>" password="<VCenterPassword>" <node1|node2>
The ouput of this command should be something like this (the last line is the must important):
info: external_run_cmd: '/usr/lib/stonith/plugins/external/fence_vmware_soap reset vsaporat1' output: Success: Rebooted
The following messages can be safely ignored as they only warns about the "--ssl-insecure"
parameter included by plugin in order to allow the use of self-signed certificates for the SSL tunnel:
/usr/local/lib64/python2.6/site-packages/requests-2.5.1-py2.6.egg/requests/packages/urllib3/connectionpool.py:734: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
The plugin agent output to syslog (or messages) entries tagged as "fence_vmware_soap" for debugging purposes:
Jan 13 17:04:59 node1 stonith-ng[1284]: notice: stonith_device_register: Added 'fence_vmware_soap1' to the device list (1 active devices)
Jan 13 17:05:02 node1 stonith-ng[1284]: notice: stonith_device_register: Device 'fence_vmware_soap1' already existed in device list (1 active devices)
Jan 13 17:05:03 node1 crmd[1288]: notice: process_lrm_event: LRM operation fence_vmware_soap1_start_0 (call=23, rc=0, cib-update=14, confirmed=true) ok
Jan 13 17:09:06 node1 fence_vmware_soap: hostlist: node1#012node2
Jan 13 17:09:06 node1 stonith-ng[1284]: notice: can_fence_host_with_device: fence_vmware_soap1 can fence node2: dynamic-list
Jan 13 17:09:06 node1 fence_vmware_soap: hostlist: node1,uuid1;node2,uuid2
Jan 13 17:09:06 node1 fence_vmware_soap: fencing: VCenterIP VCenterUser reboot uuid2
Jan 13 17:09:25 node1 stonith: [1996]: info: external_run_cmd: '/usr/lib/stonith/plugins/external/fence_vmware_soap reset node2' output: Success: Rebooted
If all the previous test are passed so the nodes are correctly restarted, everything is correct.
---------------------------------------------
Plugin Activation (Fencing Device Creation)
---------------------------------------------
*** NOTICE: The following test DO REQUIRE cluster configuration modifications!. ***
It is important to DELETE current stonith devices and DISABLE stonith components
to avoid fencing device conflicts:
crm configure
property stonith-enabled=false
commit
exit
After the previous configuration, both nodes should be restarted.
Declare the VMWare Fencing Device in the cluster:
crm configure
primitive fence_vmware_soap1 stonith:external/fence_vmware_soap \
params hostlist="node1,uuid1;node2,uuid2" vcenterip="<VCenterIP>" username="<VCenterUser>" password="<VCenterPassword>"
commit
exit
Clone the VMWare Fencing Device to set it up and available on all cluster nodes:
crm configure
clone clone_fence_vmware fence_vmware_soap1
commit
exit
For more infomation this cloning approach visit:
- "Fencing and Stonith" visit: http://clusterlabs.org/doc/crm_fencing.html):
Check if the fencing device (fence_vmware_soap1) is available and working in the cluster:
stonith_admin -L
Check if the fencing device (fence_vmware_soap1) is available and can fence all nodes:
stonith_admin --list=<node1|node2>
Now activate fencing mechanism:
crm configure
property stonith-enabled=true
property stonith-action="reboot"
commit
exit
After the previous configuration, both nodes should be restarted, in order to do
the final certification tests.
First, disable heartbeat NIC/Ethernet (ifdown eth0) in one node, then you should
see in the syslog the same entries about fencing actions shown at the previous fencing tests.
Finally, do the same in the other node.
Done!.