Kaltura-monit.service not enabled after Kaltura 12.16 installed on CentOS7

stanrate · June 6, 2017, 2:30am

CentOS 7.3 was installed and then Kaltura 12.16 followed a run of kaltura-config-all.sh and then kaltura-sanity.sh which logged a failure because kaltura-monit.service was not enabled. See below:

[Check kaltura-batch daemon is started by Monit] [PASSED, RC: 0] - [10.360302065]
[check daemon kaltura-batch init status] [PASSED, RC: 0] - [.004970531]
[Check kaltura-monit daemon status] [PASSED, RC: 0] - [.046024846]
Note: Forwarding request to ‘systemctl is-enabled kaltura-monit.service’.
disabled
[check_daemon_init_status] [FAILED, RC: 1] - [.012589252]
Redirecting to /bin/systemctl status memcached.service
[Check memcached daemon status] [PASSED, RC: 0] - [.028697571]

Please change Kaltura so that kaltura-monit.service is enabled and CentOS7 users do not have to run the following command:

systemctl enable kaltura-monit.service

Thanks.

jess · June 6, 2017, 11:34am

Hi @stanrate,

This is curious because the kaltura-monit %post has:

if [ "$1" -eq 1 ];then
%if %{use_systemd}
	/usr/bin/systemctl preset %{name}.service >/dev/null 2>&1 ||:

See:

github.com

kaltura/platform-install-packages/blob/Lynx-12.18.0/RPM/SPECS/kaltura-monit.spec#L87




%pre
if ! /usr/bin/id monit &>/dev/null; then
	/usr/sbin/useradd -M -r -d %{prefix}/var/lib/monit -s /bin/sh -c "monit daemon" monit || \
		%logmsg "Unexpected error adding user \"monit\". Aborting installation."
fi


%post
if [ "$1" -eq 1 ];then
%if %{use_systemd}
	#/usr/bin/systemctl preset %{name}.service >/dev/null 2>&1 ||:
	/usr/bin/systemctl enable %{name}.service >/dev/null 2>&1 ||:
%else
	/sbin/chkconfig --add %{name}
	/sbin/chkconfig %{name} on
%endif
fi
/sbin/service monit restart &>/dev/null || :




%preun

The if [ "$1" -eq 1 ];then is of course very important as, when its a first time install, the %post hook gets 1 as its first arg [$1], during an upgrade, the value is 2, so it’s a way to test whether we’re upgrading or installation for the first time.

from the systemctl man:

       preset NAME...
           Reset the enable/disable status one or more unit files, as specified on the command line, to the defaults configured in
           the preset policy files. This has the same effect as disable or enable, depending how the unit is listed in the preset
           files.

This is also what other official packages do, for example, consider:

# rpm -q --scripts httpd
postinstall scriptlet (using /bin/sh):
if [ $1 -eq 1 ] ; then 
        # Initial installation 
        systemctl preset httpd.service htcacheclean.service >/dev/null 2>&1 || : 
fi

And is also what the kaltura-nginx package does.
We have a nightly CI that installs a full instance on all supported distros [CentOS 7 included] and this issue was not reproduced during its run. I’ll try to manually reproduce, however. Does this reproduce for you?

stanrate · June 7, 2017, 1:28am

I installed kaltura 12.16 twice under CentOS7.3 and both times kaltura-monit service was not enabled. I have reproduced the problem showing that preset disables the service - see below:

systemctl enable kaltura-monit.service

Created symlink from /etc/systemd/system/multi-user.target.wants/kaltura-monit.service to /usr/lib/systemd/system/kaltura-monit.service.

/usr/bin/systemctl preset kaltura-monit.service

Removed symlink /etc/systemd/system/multi-user.target.wants/kaltura-monit.service.

systemctl is-enabled kaltura-monit.service

disabled

I found the CentOS preset files:

/usr/lib/systemd/system-preset/85-display-manager.preset
/usr/lib/systemd/system-preset/90-default.preset
/usr/lib/systemd/system-preset/90-systemd.preset
/usr/lib/systemd/system-preset/99-default-disable.preset

The last file contains “disable *”, which is probably the reason why kaltura-monit.service became disabled when the RPM post script ran.

I confirmed the post script does use “preset”. Perhaps it should use enable?

rpm -qp --scripts kaltura-monit-5.21.0-4.x86_64.rpm

warning: kaltura-monit-5.21.0-4.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID ba443ef2: NOKEY
preinstall scriptlet (using /bin/sh):
if ! /usr/bin/id monit &>/dev/null; then
/usr/sbin/useradd -M -r -d /opt/kaltura//var/lib/monit -s /bin/sh -c “monit daemon” monit ||
logger -t kaltura-monit/rpm "Unexpected error adding user “monit”. Aborting installation."
fi
postinstall scriptlet (using /bin/sh):
if [ “$1” -eq 1 ];then
/usr/bin/systemctl preset kaltura-monit.service >/dev/null 2>&1 ||:
else
/usr/bin/systemctl daemon-reload >/dev/null 2>&1 ||:
fi
/sbin/service monit restart &>/dev/null || :
preuninstall scriptlet (using /bin/sh):
if [ $1 -eq 0 ]; then
/usr/bin/systemctl --no-reload disable kaltura-monit.service >/dev/null 2>&1 ||:
/usr/bin/systemctl stop kaltura-monit.service >/dev/null 2>&1 ||:
fi
postuninstall scriptlet (using /bin/sh):
/usr/bin/systemctl daemon-reload >/dev/null 2>&1 ||:
/sbin/service monit condrestart &>/dev/null || :
if [ $1 -eq 0 ]; then
/usr/sbin/userdel monit || logger -t kaltura-monit/rpm "User “monit” could not be deleted."
fi

I installed httpd on a test RHEL7 host and found the service was disabled:

systemctl is-enabled httpd.service

disabled

Thus, if monit uses the same post script logic as httpd, we should expect the service to be disabled after the RPM has been installed.

Thanks.

jess · June 7, 2017, 9:49am

Hi @stanrate,

It works fine as is on my own CentOS 7 machine [I even rebooted it] but I’ll change it to read:

/usr/bin/systemctl enable %{name}.service >/dev/null 2>&1 ||:

Just so that we’re on the safe side.
Also, does kaltura-nginx have the same issue on your machine? The %postinst code for it does the same.

Thanks for reporting,

stanrate · June 8, 2017, 12:52am

After further investigation I now understand why kaltura-monit.service was not enabled after Kaltura was installed and kaltura-config-all.sh was run.

When Kaltura RPMs are installed, kaltura services are disabled as per CentOS7 preset defaults because kaltura RPM post scripts contain “/usr/bin/systemctl preset xxxx.service” commands.

When I ran /opt/kaltura/bin/kaltura-config-all.sh with an answer file, I saw the following in the output:

Starting kaltura-monit (via systemctl): [ OK ]
Running Batch config…
kaltura-batch-12.16.0-1.noarch
base-config completed successfully, if you ever want to re-configure your system (e.g. change DB hostname) run the following script:
rm /opt/kaltura/app/base-config.lock
/opt/kaltura/bin/kaltura-base-config.sh
Note: Forwarding request to ‘systemctl enable httpd.service’.
Redirecting to /bin/systemctl reload httpd.service
Note: Forwarding request to ‘systemctl enable memcached.service’.
Redirecting to /bin/systemctl restart memcached.service
Starting kaltura-monit (via systemctl): [ OK ]
kaltura-nginx-1.12.0-4.x86_64
Note: Forwarding request to ‘systemctl enable kaltura-nginx.service’.
Redirecting to /bin/systemctl reload kaltura-nginx.service
Running DWH config…

The output shows that kaltura-config-all.sh enabled several kaltura services including kaltura-nginx.service but kaltura-monit.service was NOT enabled. My conclusion is that someone at Kaltura made a design decision to have kaltura-config-all.sh enable and start services but unfortunately the code to enable monit was omitted. Thus, if we agree that the designer probably intended this (and the evidence suggests it), we should not change the RPM post script but instead add code to the config script to enable kaltura-monit.service

Thanks.

jess · June 8, 2017, 9:13am

Hi @stanrate,

Actually, there’s quite a simple explanation for that, in the industry it’s fondly known as “for historical reasons”
Up until quite recently, kaltura-monit did not ship with a systemd config, only an init script, which was used for both CentOS 6 and 7.
At any rate, since Monit is needed for every node role [and for an all in one server, of course], it’s best to enable it during the RPM’s %postinst rather than during the kaltura-.*config.sh scripts.
I’ve already committed, built and pushed a new kaltura-monit package to the repo.
If you upgrade kaltura-monit, you should get kaltura-monit-5.21.0-5.x86_64 which runs:

/usr/bin/systemctl enable kaltura-monit.service >/dev/null 2>&1 ||:

in its %postinst hook.

Thanks,

stanrate · June 9, 2017, 1:40am

I tried to install the latest kaltura RPMs but there was a syntax error in a post script - see below:

==========================================================
Updating : kaltura-ffmpeg-3.2-1.x86_64 10/37
Updating : kaltura-nginx-1.12.0-4.x86_64 11/37
Updating : kaltura-monit-5.21.0-4.x86_64 12/37
/var/tmp/rpm-tmp.KxWA4Z: line 5: syntax error near unexpected token `fi' /var/tmp/rpm-tmp.KxWA4Z: line 5:`fi’
warning: %post(kaltura-monit-5.21.0-4.x86_64) scriptlet failed, exit status 2
Non-fatal POSTIN scriptlet failure in rpm package kaltura-monit-5.21.0-4.x86_64
Shutting down monit: [ OK ]
httpd (pid 2387) is running…
Stopping httpd: [ OK ]
Updating : kaltura-base-12.17.0-20.noarch

The following lines of code seemed to have caused the syntax error:

if [ “$1” -eq 1 ];then
/sbin/chkconfig --add kaltura-monit
/sbin/chkconfig kaltura-monit on
else
fi

Perhaps an else statement cannot be immediately followed by a “fi” statement.

Thanks

jess · June 9, 2017, 9:20am

Hi @stanrate,

I’m guessing this ran on CentOS/RHEL version where systemd is not supported? if so, then I know the cause [which I will explain] if not, please let me know which distro and version this happened on cause on CentOS/RHEL 7 it shouldn’t.

Assuming I’m right, here is what happened.
The %post code for kaltura-monit read:

if [ "$1" -eq 1 ];then
%if %{use_systemd}
	#/usr/bin/systemctl preset %{name}.service >/dev/null 2>&1 ||:
	/usr/bin/systemctl enable %{name}.service >/dev/null 2>&1 ||:
%else
	/sbin/chkconfig --add %{name}
	/sbin/chkconfig %{name} on
%endif
else
%if %{use_systemd}
	/usr/bin/systemctl daemon-reload >/dev/null 2>&1 ||:
%endif
fi
/sbin/service monit restart &>/dev/null || :

%{use_systemd} is defined like so:

%define use_systemd (0%{?fedora} && 0%{?fedora} >= 18) || (0%{?rhel} && 0%{?rhel} >= 7)

And so, on CentOS/RHEL 7 [and onward], the above code would result in:

        /usr/bin/systemctl enable kaltura-monit.service >/dev/null 2>&1 ||:
else
        /usr/bin/systemctl daemon-reload >/dev/null 2>&1 ||:
fi

which is fine, but on CentOS/RHEL 6, it would be:

postinstall scriptlet (using /bin/sh):
if [ "$1" -eq 1 ];then
        /sbin/chkconfig --add kaltura-monit
        /sbin/chkconfig kaltura-monit on
else
fi

In BASH, an empty else block is not supported and will trigger:

syntax error near unexpected token fi'

Because /usr/bin/systemctl daemon-reload >/dev/null 2>&1 ||: is not really needed during an upgrade on a systemd enabled system at the moment [this is only important if the new version of the package changes /usr/lib/systemd/system/kaltura-monit.service], I simply removed that else completely.
While the warning is not fatal, I still released kaltura-monit-5.21.0-6.x86_64.rpm so it won’t happen to others.

As usual, thanks for reporting!