HTTPD startup Error :: Admin Page Endless loop

hiphopservers · December 31, 2015, 6:27pm

When I following the upgrade instructions:

kaltura/platform-install-packages/blob/master/doc/install-kaltura-redhat-based.md#upgrade-kaltura

# Installing Kaltura on a Single Server (RPM)
This guide describes RPM installation of an all-in-one Kaltura server and applies to all major RH based Linux distros including Fedora Core, RHEL, CentOS, etc.
([Note the supported distros and versions](http://kaltura.github.io/platform-install-packages/#supported-distros)).

[Kaltura Inc.](http://corp.kaltura.com) also provides commercial solutions and services including pro-active platform monitoring, applications, SLA, 24/7 support and professional services. If you're looking for a commercially supported video platform  with integrations to commercial encoders, streaming servers, eCDN, DRM and more - Start a [Free Trial of the Kaltura.com Hosted Platform](http://corp.kaltura.com/free-trial) or learn more about [Kaltura' Commercial OnPrem Edition™](http://corp.kaltura.com/Deployment-Options/Kaltura-On-Prem-Edition). For existing RPM based users, Kaltura offers commercial upgrade options.

#### Table of Contents
[Prerequites](pre-requisites.md)

[Pre-Install Steps](install-kaltura-redhat-based.md#pre-install-steps)

[Non-SSL Step-by-step Installation](install-kaltura-redhat-based.md#non-ssl-step-by-step-installation)

[Apache SSL Step-by-step Installation](install-kaltura-redhat-based.md#apache-ssl-step-by-step-installation)

[Nginx SSL Configuration](nginx-ssl-config.md)

[Unattended Installation](install-kaltura-redhat-based.md#unattended-installation)

[Live Streaming with Nginx and the RTMP module](install-kaltura-redhat-based.md#live-streaming-with-nginx-and-the-rtmp-module)

This file has been truncated. show original

I still get the caption error message when running the update database functions.

/opt/kaltura/bin/kaltura-db-update.sh
Error occurred during DB update:

I can confirm that the script is now properly writing the ‘/etc/httpd/conf.d/zzzkaltura.ssl.conf’ file and does not require me to manually edit the file to get it working properly.

However, HTTPD still returns an error when it tries to restart as shown here. My answers have not changed. Then tells me to rerun the frontend script once I fix the issue.

========================================================================================================================
Kaltura install answer file written to /tmp/kaltura_31_12_13_19.ans - Please save it!
This answers file can be used to silently-install re-install this machine or deploy other hosts in your cluster.

Stopping httpd: [FAILED]
Starting httpd: [FAILED]

kaltura-front-config.sh FAILED with: 9714 on line 305

Archving logs to /opt/kaltura/log/log_31_12_15_13_20.tar.gz…
ERROR: /opt/kaltura/bin/kaltura-front-config.sh failed:( You can re-run it when the issue is fixed.

I am not sure what to fix to resolve this issue. Nor why I am still getting the error when running the database update script.

Thanks,
Richard

jess · December 31, 2015, 7:36pm

Hi Richard,

In this particular case, there is no need to run /opt/kaltura/bin/kaltura-db-update.sh so we can ignore that part.
Now our issue is with why restarting Apache fails.
Please:
# service httpd restart
from the shell and then look at the error logs under /opt/kaltura/log/kaltura_apache_errlog
and /var/log/httpd/errorlog

lets see what it says…

hiphopservers · December 31, 2015, 11:34pm

okay… I check the error logs and there are no entries in the logs for kaltura but I did find the following errors in the '/var/log/httpd/error*log

[Thu Dec 31 18:21:26 2015] [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:httpd_t:s0
[Thu Dec 31 18:21:26 2015] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Dec 31 18:21:26 2015] [error] (28)No space left on device: Cannot create SSLMutex
Configuration Failed

First issue is I know that my SELinux is disabled.

SELINUX=disabled

I don’t know much about the suEXEC error and I will have to take a look into why the SSLMutex is happening because this system is on a 10TB system. I will contact my ISP support as they are the ones that setup the box from the start and I always allocate more than enough space to tmp folder and other dynamic resources to avoid this type of issue.

If you have any input on the other warning let me know.

Thanks,
Richard

hiphopservers · January 1, 2016, 12:17am

Jess:

I Google the error previously mention and found one of three solution. First, was to reboot the server. and the alternate was run one of the following commands depending on your system.

on sh system

ipcs -s | grep apache | awk ’ { print $2 } ’ | xargs ipcrm sem

Or

ipcs -s | grep apache | sudo perl -alne ‘qx(ipcrm -s $F[1])’

So after I ran the command I found HTTPD was able to start again. However, even after fixing this error I still got the following errors.

Reloading httpd: not reloading due to configuration syntax error
[FAILED]

kaltura-batch-config.sh FAILED with: 14187 on line 98

Archving logs to /opt/kaltura/log/log_31_12_15_19_22.tar.gz…
ERROR: /opt/kaltura/bin/kaltura-batch-config.sh failed:( You can re-run it when the issue is fixed.

I remember from before this was a result of the install script not properly creating one of the HTTPD configuration files for Kaltura. Specifically, the required tokens are not being updated like I mention in the SSL configuration file. Here is the error you get after attempting a restart of HTTPD.

Starting httpd: httpd: Syntax error on line 226 of /etc/httpd/conf/httpd.conf: Syntax error on line 7 of /etc/httpd/conf.d/zzzkaltura.conf: Include directory ‘/etc/httpd/@APP_DIR@/configurations/apache/conf.d’ not found

Unlike the SSL configuration file this cannot be fixed manually prior to the script attempting to restart Apache so it breaks the install. You would need to find the error in the configuration script to allow it to properly update the ‘/etc/httpd/conf.d/zzzkaltura.conf’ configuration file prior to restart.

However, after editing the file manually and replacing the tokens with valid information then restarting HTTPD (Apache), I was able to get the host running again.

Thanks,
Richard

hiphopservers · January 1, 2016, 4:27am

Hey @jess

After getting what seem to be everything working, I getting errors that I am missing files required for the Kaltura All In One plugin to work properly. I open a new thread to address this issue.

I am not sure if this is a plugin issue or an installation issue. It would appear to be an installation issue since files are missing from the Kaltura complete installation package required for this plugin to work. Your assistance is appreciated.

Thanks,
@hiphopservers

hiphopservers · January 4, 2016, 5:48pm

Hello @jess

I updated to the latest release of Kaltura CE you released six hours ago. However, I am still getting the following error during the configuration of the script.

kaltura-batch-11.5.0-1.noarch
base-config completed successfully, if you ever want to re-configure your system (e.g. change DB hostname) run the following script:
# rm /opt/kaltura/app/base-config.lock
# /opt/kaltura/bin/kaltura-base-config.sh


Reloading httpd: not reloading due to configuration syntax error
                                                           [FAILED]


kaltura-batch-config.sh FAILED with: 1731 on line 98

Archving logs to /opt/kaltura/log/log_04_01_16_11_41.tar.gz...
ERROR: /opt/kaltura/bin/kaltura-batch-config.sh failed:( You can re-run it when the issue is fixed.

To get this install running again I still had to edit the:

after updating the tokens in this file with the correct path information and making other manual configuration changes previously described, I was able to restart HTTPD and access Kaltura KMC and Admin Console, Red5 and run secure Kaltura Monit without errors.

However, the issues described in the following thread still exist.

mark · January 4, 2016, 5:56pm

Hi @hiphopservers,

the only way I am able to get httpd to work is by deleting file zzzkaltura.conf

mark · January 4, 2016, 5:57pm

However, when I login to Admin Console I get:

An error occurred
(error code: API:-1)

hiphopservers · January 4, 2016, 6:22pm

Hello @mark

I had that happen and I had to rerun the configuration script. Can you post a copy of your answer file and delete or hide all passwords IPs and URLs? It is related to your SSL configuraiton.

BTW, post the content of your ‘/etc/yum.repos.d/kaltura.repo’ file so I can see were you are pulling the RPMs from. Sometimes you get old files or other don’t update cause the main server has not updated with the latest RPMs. Some of this was mention by @jess in this thread.

mark · January 4, 2016, 6:37pm

@hiphopservers, Going for install #4 on clean CentOS 6.7. Will keep you posted. Thank you.

mark · January 4, 2016, 6:43pm

New error on install:

[root@player ~]# yum install mysql mysql-server

http://installrepo.kaltura.org/releases/latest/RPMS/x86_64/repodata/4962aed3c4ef2555d29a0210153e9775d74da7bf297b33ba029517b833b2cf51-primary.sqlite.bz2: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"
Trying other mirror.
Error: failure: repodata/4962aed3c4ef2555d29a0210153e9775d74da7bf297b33ba029517b833b2cf51-primary.sqlite.bz2 from Kaltura: [Errno 256] No more mirrors to try.

jess · January 4, 2016, 6:57pm

My guess is that your CDN edge has not fully updated from the origin server.
If so, you can change installrepo.kaltura.org to installrepo.origin.kaltura.org in /etc/yum.repos.d/kaltura.repo and retry.
Remember to change back later though since using CDN is naturally much faster.

mark · January 4, 2016, 6:58pm

That worked. Thank you @jess.

mark · January 4, 2016, 8:11pm

Install failed with the following error:
Running Batch config…

kaltura-batch-11.5.0-1.noarch
base-config completed successfully, if you ever want to re-configure your system (e.g. change DB hostname) run the following script:

rm /opt/kaltura/app/base-config.lock

/opt/kaltura/bin/kaltura-base-config.sh

Reloading httpd: not reloading due to configuration syntax error
[FAILED]

kaltura-batch-config.sh FAILED with: 9235 on line 98

Archving logs to /opt/kaltura/log/log_04_01_16_22_58.tar.gz…
ERROR: /opt/kaltura/bin/kaltura-batch-config.sh failed:( You can re-run it when the issue is fixed.
[root@player conf.d]# rpm -q kaltura-postinst
kaltura-postinst-1.0.30-10.noarch

SSL certs have been added to /etc/httpd/conf.d/zzzkaltura.ssl.conf.

mark · January 5, 2016, 3:51am

Hi @jess, I was able to install with nonSSL. Now that the nonSSL instance is working…is there any safe way to change to SSL? Please advise. Thank you.

hiphopservers · January 5, 2016, 8:56am

Hi @mark

Did you edit the tokens in this file after that reload error? There is a SSL file and regular file both have to have the tokens updated if they were not set during installation. I had a similar issue. Re-read my comments and errors for more information.

mark · January 5, 2016, 4:26pm

Hello @hiphopservers, as per your suggestion I have added the tokens to file /etc/httpd/conf.d/zzzkaltura.conf.

I am now getting the following error:

[root@player conf.d]# service httpd reload
Reloading httpd: not reloading due to configuration syntax error
[FAILED]
[root@player conf.d]# service httpd restart
Stopping httpd: [ OK ]
Starting httpd: httpd: Syntax error on line 221 of /etc/httpd/conf/httpd.conf: Syntax error on line 7 of /etc/httpd/conf.d/zzzkaltura.conf: Include directory ‘/etc/httpd/@APP_DIR@/configurations/apache/conf.d’ not found
[FAILED]

Here are the contents of file: /etc/httpd/conf.d/zzzkaltura.conf

<VirtualHost *:@KALTURA_VIRTUAL_HOST_PORT@>
# for SSL offloading support, if LB has X_FORWARDED_PROTO set to ‘https’, set HTTPS to 'on’
SetEnvIf X-Forwarded-Proto https HTTPS=on
ErrorLog "@LOG_DIR@/kaltura_apache_errors.log"
CustomLog @LOG_DIR@/kaltura_apache_access.log vhost_kalt

    Include "@APP_DIR@/configurations/apache/conf.d/enabled.*.conf"

SSLCertificateFile /etc/pki/tls/certs/player_HIDDEN.crt
SSLCertificateKeyFile /etc/pki/tls/private/server.key
SSLCACertificateFile /etc/pki/tls/certs/player_HIDDEN.ca-bundle

hiphopservers · January 5, 2016, 4:35pm

Hello @mark

I do not see where you have replaced:

@KALTURA_VIRTUAL_HOST_PORT@

with your port number for the virtual host you set in configuration. This is usually port [80].

Nor, have you replaced the path to the Kaltura log and app folder tokens:

@LOG_DIR@ this is usually /opt/kaultura/logo/
@APP_DIR@ this is usually /opt/kautura/app/

If you did not properly update those tokens found in the file with valid information then try that and restart HTTPD.

mark · January 5, 2016, 4:49pm

Hello @hiphopservers, I have updated the file /etc/httpd/conf.d/zzzkaltura.conf with:

<VirtualHost *:80>
# for SSL offloading support, if LB has X_FORWARDED_PROTO set to ‘https’, set HTTPS to 'on’
SetEnvIf X-Forwarded-Proto https HTTPS=on
ErrorLog "/opt/kaltura/log/kaltura_apache_errors.log"
CustomLog /opt/kaltura/log/kaltura_apache_access.log vhost_kalt

    Include "/opt/kaltura/app/configurations/apache/conf.d/enabled.*.conf"

SSLCertificateFile /etc/pki/tls/certs/player_HIDDEN.crt
SSLCertificateKeyFile /etc/pki/tls/private/server.key
SSLCACertificateFile /etc/pki/tls/certs/player_HIDDEN.ca-bundle

I rebooted and when I restart httpd I get the following error:

[root@player conf.d]# service httpd reload
Reloading httpd: [FAILED]

mark · January 5, 2016, 4:50pm

I also manually added the VirtualHost *:80