UPS management with NUT
Any server must be always connected to an UPS unit
You really need to protect your server from electrical surges, power cuts or outages. A common way to achieve that protection is with a Uninterruptible Power Supply (UPS) unit, like the one part of this guide’s hardware setup.
This chapter explains how to make your Proxmox VE host monitor the UPS unit with a generic package called NUT (Network UPS Tool). However, be aware that your UPS unit’s brand may already have its own particular software for UPS management.
Important
NUT may not support your concrete UPS unit
Still, it may have compatible drivers for your UPS unit’s brand. Check in the NUT hardware compatibility list to verify if your brand or model line has compatible drivers there.
Connecting your UPS with your PVE node using NUT
This chapter uses the Eaton 3S700D UPS unit listed in the hardware setup. This model (and similar ones) comes with a USB 2.0 cable to connect the UPS unit with any computer.
Warning
This procedure is for UPS units connected through USB
If your UPS unit does not come with a USB cable, you cannot proceed with the instructions explained below.
Assuming your UPS unit has the required USB cable, proceed to configure it in your Proxmox VE server:
First, check the following:
- Your server is plugged in one of your UPS’s battery-protected sockets.
- You have your UPS connected with its USB cable to a USB plug in your server.
- The UPS unit is on. If it were not, you would not be able to switch your server on.
With your Proxmox VE server running, get into the
pvenode through arootshell and execute thelsusbcommand:$ lsusb Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 05e3:0608 Genesys Logic, Inc. Hub Bus 001 Device 004: ID 0bda:0129 Realtek Semiconductor Corp. RTS5129 Card Reader Controller Bus 001 Device 005: ID 046d:c52b Logitech, Inc. Unifying Receiver Bus 001 Device 006: ID 0463:ffff MGE UPS Systems UPS Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 002 Device 002: ID 152d:0578 JMicron Technology Corp. / JMicron USA Technology Corp. JMS578 SATA 6Gb/sThe
lsusbreturns a list of the USB devices currently connected to your physical server. Please notice how the Eaton 3S700D UPS unit is listed in theBus 001 Device 006line with the rather misleading nameMGE UPS Systems UPS, confirming the correct connection of the unit to thepvenode.Install the NUT software package:
$ apt install -y nutYou need to configure NUT properly to make it manage your UPS unit. There are several files to change, start by editing
/etc/nut/nut.conf. But before you change it, first make a backup of it:$ cd /etc/nut $ cp nut.conf nut.conf.origNow you can edit the
nut.conffile. In it, there is only one line you have to modify:MODE=noneThis
MODEline must be changed into the following:MODE=standaloneBy setting the
standalonemode you’re enabling NUT to work in local mode or, in other words, it will “worry” only about yourpvenode.Note
The
nut.conffile has comments explaining configuration details
Check the comments in thenut.confto have a better idea of which are the modes that NUT supports.You need to tell NUT about the UPS unit connected to your system. To do that, you must edit the file
/etc/nut/ups.conf, so make a backup of it:$ cp ups.conf ups.conf.origThen, append the following parameters to the
ups.conffile, although with values fitting for your UPS unit:[eaton] driver = usbhid-ups port = auto desc = "Eaton 3S700D"The parameters above mean this:
[eaton]
This is the name that will identify your UPS unit in NUT. This can be any string of characters, but without spaces. Type something there that truly identifies your UPS in a clear manner.driver
Here you indicate NUT how to connect to your UPS unit. For UPS units with their data port plugged in a USB, the correct value isusbhid-ups.port
For an USB connection this has to be left toauto.desc
This is just a descriptive string where you can type the full name and model of your UPS unit, or anything else you might prefer.
At this point, you can check if NUT can start its driver and detect your UPS unit properly:
$ upsdrvctl start Network UPS Tools - UPS driver controller 2.8.1 Network UPS Tools - Generic HID driver 0.52 (2.8.1) USB communication driver (libusb 1.0) 0.46 Using subdriver: MGE HID 1.46The command tells you what components it is using and, in the last line, also indicates you what driver NUT used to connect to your particular UPS. In this guide’s scenario, you can see it is using the correct
MGEsubdriver.On the other hand, if you get the following warnings:
libusb1: Could not open any HID devices: insufficient permissions on everything No matching HID UPS found upsnotify: notify about state 4 with libsystemd: was requested, but not running as a service unit now, will not spam more about it upsnotify: failed to notify about state 4: no notification tech defined, will not spam more about it Driver failed to start (exit status=1)Just reboot your Proxmox VE server and try the
upsdrvctl startcommand again, it should run fine this time.Note
Do not worry if you get a “Duplicate driver” warning
In the output of theupsdrvctl startcommand you may get a warning like this one:Duplicate driver instance detected (PID file /run/nut/usbhid-ups-eaton.pid exists)! Terminating other driver!It seems that NUT already creates automatically a driver for the UPS unit in the moment it is configured in the
ups.conffile. This is not an issue at this point, and NUT itself takes care of this duplicity.Configure the NUT daemon by editing the file
/etc/nut/upsd.conf. Again, make a backup first:$ cp upsd.conf upsd.conf.origEdit the
upsd.conffile to uncomment only theLISTENline referred to the IPv4 connection. It should end looking as follows:# LISTEN <IP address or name> [<port>] LISTEN 127.0.0.1 3493 # LISTEN ::1 3493The
LISTENlines declares on which ports theupsddaemon will listen, and provides a basic access control mechanism. Uncomment the IPv6 line when you use this protocol in your network setup.In NUT there are also users, which are NOT the same ones as in the
passwdfile of yourpvenode. At this point, you require two user:- One for the NUT monitor agent (
upsmon). - Other for acting as a NUT ups administrator(
upsadmin).
To add them, you must edit the file
/etc/nut/upsd.users. First, back it up:$ cp upsd.users upsd.users.origThen, append to
upsd.usersthe following configuration block:[upsmon] password = s3c4R3_p4sSw0rD! upsmon primary [upsadm] password = D1Ff3rEnT_s3c4R3_p4sSw0rD! actions = SET instcmds = ALLThe parameters shown above mean the following:
[upsmon]/[upsadm]
This is the NUT user’s name. It can be any string, but with no spaces.password
The NUT user’s password. Bear in mind that this is an unencrypted configuration file. Be careful with who can access it.upsmon primary
Roughly speaking, this line means that the NUT user is from a machine directly connected to the UPS unit, and that NUT should be run with high privileges there.Note
Learn more about UPS primary and secondary types in the NUT official documentation
The UPS primary and secondary types are further explained in the NUTupsmonman documentation, in the UPS types section.actions = SET
Allows the NUT user to set values on the managed UPS unit.instcmds = ALL
Allows the NUT user to execute all available instant commands supported by the managed UPS unit.
- One for the NUT monitor agent (
Finally, configure the NUT monitoring service that watches over the UPS unit. The configuration file you have to edit is
/etc/nut/upsmon.conf, so back it up:$ cp upsmon.conf upsmon.conf.origThen, do the following in the
upsmon.conffile:Search for and comment out the lines of the
RBWARNTIMEandSHUTDOWNCMDparameters:# -------------------------------------------------------------------------- # RBWARNTIME - replace battery warning time in seconds # # upsmon will normally warn you about a battery that needs to be replaced # every 43200 seconds, which is 12 hours. It does this by triggering a # NOTIFY_REPLBATT which is then handled by the usual notify structure # you've defined above. # # If this number is not to your liking, override it here. #RBWARNTIME 43200# -------------------------------------------------------------------------- # SHUTDOWNCMD "<command>" # # upsmon runs this command when the system needs to be brought down. # # This should work just about everywhere ... if it doesn't, well, change it, # perhaps to a more complicated custom script. # # Note that while you experiment with the initial setup and want to test how # your configuration reacts to power state changes and ultimately when power # is reported to go critical, but do not want your system to actually turn # off, consider setting the SHUTDOWNCMD temporarily to do something benign - # such as posting a message with 'logger' or 'wall' or 'mailx'. Do be careful # to plug the UPS back into the wall in a timely fashion. # # For Windows setup use something like: # SHUTDOWNCMD "C:\\WINDOWS\\system32\\shutdown.exe -s -t 0" # If you have command line using space character you have to add double quote to them, like this: # SHUTDOWNCMD "\"C:\\Program Files\\some command.bat\" -first_arg -second_arg" # Or use the old DOS 8.3 file name, like this: # SHUTDOWNCMD "C:\\PROGRA~1\\SOMECO~1.bat -first_arg -second_arg" #SHUTDOWNCMD "/sbin/shutdown -h +0"Append the following lines at the end of the
upsmon.conffile:# -------------------------------------------------------------------------- # Customized settings MONITOR eaton@localhost 1 upsmon s3c4R3_p4sSw0rD! primary SHUTDOWNCMD "logger -t upsmon.conf \"SHUTDOWNCMD calling /sbin/shutdown to shut down system\" ; /sbin/shutdown -h +0" NOTIFYMSG ONLINE "UPS %s: On line power." NOTIFYMSG ONBATT "UPS %s: On battery." NOTIFYMSG LOWBATT "UPS %s: Battery is low." NOTIFYMSG REPLBATT "UPS %s: Battery needs to be replaced." NOTIFYMSG FSD "UPS %s: Forced shutdown in progress." NOTIFYMSG SHUTDOWN "Auto logout and shutdown proceeding." NOTIFYMSG COMMOK "UPS %s: Communications (re-)established." NOTIFYMSG COMMBAD "UPS %s: Communications lost." NOTIFYMSG NOCOMM "UPS %s: Not available." NOTIFYMSG NOPARENT "upsmon parent dead, shutdown impossible." NOTIFYFLAG ONLINE SYSLOG NOTIFYFLAG ONBATT SYSLOG NOTIFYFLAG LOWBATT SYSLOG NOTIFYFLAG REPLBATT SYSLOG NOTIFYFLAG FSD SYSLOG NOTIFYFLAG SHUTDOWN SYSLOG NOTIFYFLAG COMMOK SYSLOG NOTIFYFLAG COMMBAD SYSLOG NOTIFYFLAG NOCOMM SYSLOG NOTIFYFLAG NOPARENT SYSLOG RBWARNTIME 7200 # 2 hours
Here is an explanation for all the added parameters in the configuration snippet:
MONITOR
This is the line where you must specify your UPS unit (eaton@localhost), the username (upsmon) and your NUT user’s password (s3c4R3_p4sSw0rD!). The number1indicates the number of power supplies feeding your system through the connected UPS.SHUTDOWNCMD
Declares the command to be used to shut down the host. In the line shown in the snippet, the command is prepared to write a log before it executes the system’s shut down.NOTIFYMSG
Each of these lines are assigning a text message to each NOTIFY event. Within each message, the marker%sis replaced by the name of the UPS unit producing the event.upsmonpasses this message to the program wall to notify the event to the system administrator.NOTIFYFLAG
Each line declare what is to be done at eachNOTIFYevent. Up to three flags can be specified for each event. In the example, all events will be registered in the host’s/var/log/sysloglog file.RBWARNTIME
When an UPS unit says it needs to have its battery replaced,upsmonwill generate aNOTIFY_REPLBATTevent. This parameter means that this warning will be repeated every this many seconds.
Before you can try the configuration, restart the NUT-related services to be sure that they load with the right parameters:
$ systemctl restart nut-client.service nut-server.serviceThe service’s restart will take a couple of seconds, and should not return any line if everything goes fine.
Check if the NUT services are working properly with your UPS unit:
upscmd -l upsunitname
Lists the instant commands supported by the UPS unit namedupsunitname(replace the placeholder string with the name you gave to your UPS back in point 5). Below you can see its output with this guide’s UPS unit:$ upscmd -l eaton Instant commands supported on UPS [eaton]: beeper.disable - Disable the UPS beeper beeper.enable - Enable the UPS beeper beeper.mute - Temporarily mute the UPS beeper beeper.off - Obsolete (use beeper.disable or beeper.mute) beeper.on - Obsolete (use beeper.enable) driver.killpower - Tell the driver daemon to initiate UPS shutdown; should be unlocked with driver.flag.allow_killpower option or variable setting driver.reload - Reload running driver configuration from the file system (only works for changes in some options) driver.reload-or-error - Reload running driver configuration from the file system (only works for changes in some options); return an error if something changed and could not be applied live (so the caller can restart it with new options) driver.reload-or-exit - Reload running driver configuration from the file system (only works for changes in some options); exit the running driver if something changed and could not be applied live (so service management framework can restart it with new options) load.off - Turn off the load immediately load.off.delay - Turn off the load with a delay (seconds) load.on - Turn on the load immediately load.on.delay - Turn on the load with a delay (seconds) shutdown.return - Turn off the load and return when power is back shutdown.stayoff - Turn off the load and remain off shutdown.stop - Stop a shutdown in progressupsc upsunitname
Returns the statistics from the UPS unit specified asupsunitname(again, replace this string with your own UPS unit’s name). See below the output this commands returns for this guide’s reference hardware:$ upsc eaton Init SSL without certificate database battery.charge: 100 battery.charge.low: 20 battery.runtime: 3840 battery.type: PbAc device.mfr: EATON device.model: Eaton 3S 700 device.serial: Blank device.type: ups driver.debug: 0 driver.flag.allow_killpower: 0 driver.name: usbhid-ups driver.parameter.pollfreq: 30 driver.parameter.pollinterval: 2 driver.parameter.port: auto driver.parameter.synchronous: auto driver.state: quiet driver.version: 2.8.1 driver.version.data: MGE HID 1.46 driver.version.internal: 0.52 driver.version.usb: libusb-1.0.28 (API: 0x100010a) input.transfer.high: 264 input.transfer.low: 184 outlet.1.desc: PowerShare Outlet 1 outlet.1.id: 1 outlet.1.status: on outlet.1.switchable: no outlet.desc: Main Outlet outlet.id: 0 outlet.switchable: yes output.frequency.nominal: 50 output.voltage: 230.0 output.voltage.nominal: 230 ups.beeper.status: enabled ups.delay.shutdown: 20 ups.delay.start: 30 ups.firmware: 02.08.0010 ups.load: 4 ups.mfr: EATON ups.model: Eaton 3S 700 ups.power.nominal: 700 ups.productid: ffff ups.realpower: 22 ups.serial: Blank ups.status: OL ups.timer.shutdown: -1 ups.timer.start: -1 ups.type: offline / line interactive ups.vendorid: 0463
Checking the NUT logs
NUT’s services write logs in the journal of the Proxmox VE system. Look out for them with the journalctl command:
nut-serverlog entries from the NUT server.nut-monitorlog entries from the NUT monitor.upsdlog entries from the UPS daemon.
Remember to check them out, specially when you have created or modified your NUT configuration. For instance, if NUT cannot connect with your UPS unit, you can see in your PVE system’s journal warning logs from the NUT monitor:
Aug 26 10:53:52 pve nut-monitor[6684]: UPS eaton@localhost: Not available.
Aug 26 10:53:57 pve nut-monitor[6684]: Poll UPS [eaton@localhost] failed - Poll UPS [eaton@localhost] failed - [eaton] does not exist on server localhostExecuting instant commands on your UPS unit
To execute the so called instant commands on your UPS unit (wich you can list with upscmd -l upsunitname), you have to use the command upscmd with the NUT administrator user defined in the previous section. A safe way to test this is by disabling and enabling the beeper usually embedded in any UPS:
$ upscmd eaton beeper.disable
Username (root): upsadm
Password:
OK
$ upscmd eaton beeper.enable
Username (root): upsadm
Password:
OKYou can also execute the whole upscmd command in just one line:
$ upscmd -u upsadm -p D1Ff3rEnT_s3c4R3_p4sSw0rD! eaton beeper.disableWarning
Use this one-line format only for task automation with shell scripts
Do not execute the upscmd with the password in your normal shell, to avoid exposing your password in your user’s shell history (the .bash_history text file in bash shells).
Also, remember that:
You have to check out first with
upscmd -l upsunitnamewhich are the instant commands supported by your UPS unit.The NUT software cannot really know if the instant command has been executed on the UPS unit. It only sees what the UPS unit is answering back in response to the command’s request.
Other possibilities with NUT
If you feel curious about what else you can do with NUT, there is a pdf document providing a good number of configuration examples. Get it in this GitHub page.
Relevant system paths
Directories
/etc/nut/var/log
Files
/etc/nut/nut.conf/etc/nut/nut.conf.orig/etc/nut/ups.conf/etc/nut/ups.conf.orig/etc/nut/upsd.conf/etc/nut/upsd.conf.orig/etc/nut/upsd.users/etc/nut/upsd.users.orig/etc/nut/upsmon.conf/etc/nut/upsmon.conf.orig/var/log/syslog
References
Network UPS Tools (NUT)
Related NUT contents
- rogerprice.org. Network UPS Tools (NUT). Configuration Examples
- ichasco. Instalar y configurar NUT por SNMP (in Spanish)
- Random Bits. Monitoring a UPS with nut on Debian or Ubuntu Linux
- ServerFault. Can’t get upsmon service started to monitor (and respond to) remote UPS
- mn3m. Configure UPS monitor using NUT on Debian
- DIYblindGuy.com. HOWTO: Configure A UPS on Proxmox 5.x
- Reddit. Homelab. UPS Server on Raspberry Pi
- Stuart’s Notes. Configuring NUT for the Eaton 3S UPS on Ubuntu Linux