Inherited broken system... (video uploads/converting not working)

Community Forums Backend Server Support
1

Hi,

I just inherited a previously built Single Server CentOS (RPM based) Kaltura CE Server. I only found out about the server when videos stopped working on our website. Once I found this Kaltura server the problem was very easy to find, since the /opt filesystem was full. Once I added another disk and extended the LVM filesystem, I rebooted server. Now videos play but you can not upload or trans-code videos?

Although I am a veteran UNIX/Linux admin I no next to nothing about the Kaltura environment. I spent time running the “/opt/kaltura/bin/kaltura-sanity.sh” script checking logs, even re-ran “/opt/kaltura/bin/kaltura-config-all.sh” a few times after reading some similar issues from this forum. Unfortunately, I just don’t know the Kaltura processes and architecture enough to get this resolved in a timely manner (on a production environment). Any and all help, hints and guidance is very welcome. Thanks you in advance for your support and help!

Here are some info on system:

allkaltlog | tail -50:
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘Peer certificate cannot be authenticated with known CA certificates’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:Stack trace:
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘Peer certificate cannot be authenticated with known CA certificates’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:Stack trace:
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘Peer certificate cannot be authenticated with known CA certificates’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:Stack trace:
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘Peer certificate cannot be authenticated with known CA certificates’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:Stack trace:
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘Peer certificate cannot be authenticated with known CA certificates’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:Stack trace:
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘Peer certificate cannot be authenticated with known CA certificates’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:Stack trace:
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘Peer certificate cannot be authenticated with known CA certificates’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:Stack trace:
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘Peer certificate cannot be authenticated with known CA certificates’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:Stack trace:
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘Peer certificate cannot be authenticated with known CA certificates’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:Stack trace:
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘Peer certificate cannot be authenticated with known CA certificates’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:Stack trace:
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘Peer certificate cannot be authenticated with known CA certificates’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:Stack trace:
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘Peer certificate cannot be authenticated with known CA certificates’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:Stack trace:
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘Peer certificate cannot be authenticated with known CA certificates’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:Stack trace:
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘Peer certificate cannot be authenticated with known CA certificates’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:Stack trace:
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘Peer certificate cannot be authenticated with known CA certificates’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:Stack trace:
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘Peer certificate cannot be authenticated with known CA certificates’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:Stack trace:
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘Peer certificate cannot be authenticated with known CA certificates’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:Stack trace:
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘Peer certificate cannot be authenticated with known CA certificates’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:Stack trace:
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘Peer certificate cannot be authenticated with known CA certificates’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:Stack trace:
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘Peer certificate cannot be authenticated with known CA certificates’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:Stack trace:
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘Peer certificate cannot be authenticated with known CA certificates’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:Stack trace:
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘Peer certificate cannot be authenticated with known CA certificates’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:Stack trace:
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘Peer certificate cannot be authenticated with known CA certificates’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:Stack trace:
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘Peer certificate cannot be authenticated with known CA certificates’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:Stack trace:
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘Peer certificate cannot be authenticated with known CA certificates’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
/opt/kaltura/log/batch/validatelivemediaservers-0-2017-05-07.err.log:Stack trace:

rpm -qa “kalture*”:
kaltura-kupload-v1.2.16-2.noarch
kaltura-html5-studio-v2.0.9-1.noarch
kaltura-monit-5.21.0-1.x86_64
kaltura-postinst-1.0.33-9.noarch
kaltura-x264-0.140-2.20140104.x86_64
kaltura-libmemcached-1.0.16-2.x86_64
kaltura-ksr-v1.0.44-1.noarch
kaltura-krecord-1.0.0-1.noarch
kaltura-front-12.14.0-1.noarch
kaltura-libopencore-amr-0.1.2-2.x86_64
kaltura-kvpm-v1.0.6-1.noarch
kaltura-batch-12.14.0-2.noarch
kaltura-base-12.14.0-31.noarch
kaltura-a52dec-0.7.4-11.x86_64
kaltura-lame-3.99.5-3.x86_64
kaltura-librtmp-2.3-3.x86_64
kaltura-fdk-acc-0.1.3-1.x86_64
kaltura-flexwrapper-v1.2-1.noarch
kaltura-kclip-v1.1.2.1-2.noarch
kaltura-kcw-1.0.0-6.noarch
kaltura-server-12.14.0-1.noarch
kaltura-html5lib-v2.55.2-1.noarch
kaltura-red5-1.0.6-1.x86_64
kaltura-sphinx-2.2.1-20.x86_64
kaltura-segmenter-1.0-2.x86_64
kaltura-widgets-1.0.0-8.noarch
kaltura-dwh-12.14.0-1.noarch
kaltura-libfaac-1.26-1.x86_64
kaltura-sshpass-1.05-1.x86_64
kaltura-ffmpeg-3.2-1.x86_64
kaltura-kdp3-v3.9.9-2.noarch
kaltura-nginx-1.12.0-3.x86_64
kaltura-libass-0.9.11-2.x86_64
kaltura-clipapp-1.3-2.noarch
kaltura-kdp3wrapper-v37.0-1.noarch
kaltura-mencoder-3.4.6-9.x86_64
kaltura-mediainfo-0.7.61-6.x86_64
kaltura-release-12.14.0-1.noarch
kaltura-kmc-v5.41.2-8.noarch
kaltura-kdp-v2.7.0-1.noarch
kaltura-kdpwrapper-v11.0-1.noarch
kaltura-pentaho-4.2.1-2.x86_64
kaltura-libmediainfo-0.7.61-6.x86_64
kaltura-ffmpeg-aux-2.1.3-1.x86_64

rpm -qa “mysql”:
mysql-5.1.73-8.el6_8.x86_64
php-mysql-5.3.3-49.el6.x86_64
mysql-server-5.1.73-8.el6_8.x86_64
mysql-libs-5.1.73-8.el6_8.x86_64
mysql-connector-java-5.1.17-6.el6.noarch

/opt/kaltura/bin/kaltura-sanity.sh output:

[Space on /] [PASSED, RC: 0] - [.009115971]
[Space on /opt/kaltura/web] [PASSED, RC: 0] - [.005201627]

INFO: Downloading ready-made schema from: /opt/kaltura/web/content/clientlibs/KalturaClient.xml
INFO: Generating from api version: 3.3.0, generated at: 07-05-2017
INFO: Now generating: php5full using Php5ClientGenerator
INFO: Delete old files [/opt/kaltura/web/content/clientlibs/php5full, /opt/kaltura/app/clients/php5]
INFO: Generate client library [php5full]
INFO: Copy sources from [/opt/kaltura/clients-generator/sources/php5]
INFO: Copy tests from [/opt/kaltura/clients-generator/sources/php5]
INFO: Saving client library to [/opt/kaltura/web/content/clientlibs/php5full]
INFO: Trying to package
DEBUG: Changing dir to [/opt/kaltura/web/content/clientlibs/php5full]
INFO: Executing: tar -czf “…/php5full_07-05-2017.tar.gz” …/php5full
tar: Removing leading `…/’ from member names
INFO: Package created successfully: …/php5full_07-05-2017.tar.gz
DEBUG: Restoring dir to [/opt/kaltura/clients-generator]
INFO: php5full generated successfully
[Check httpd daemon status] [PASSED, RC: 0] - [.075662721]
Napping 1 seconds to allow the daemon httpd to finish its init after monit restarted it…
Napping 2 seconds to allow the daemon httpd to finish its init after monit restarted it…
Napping 3 seconds to allow the daemon httpd to finish its init after monit restarted it…
Napping 4 seconds to allow the daemon httpd to finish its init after monit restarted it…
Napping 5 seconds to allow the daemon httpd to finish its init after monit restarted it…
Napping 6 seconds to allow the daemon httpd to finish its init after monit restarted it…
Napping 7 seconds to allow the daemon httpd to finish its init after monit restarted it…
Napping 8 seconds to allow the daemon httpd to finish its init after monit restarted it…
Napping 9 seconds to allow the daemon httpd to finish its init after monit restarted it…
[Check httpd daemon is started by Monit] [PASSED, RC: 0] - [10.701785964]
[check daemon httpd init status] [PASSED, RC: 0] - [.014112502]
[Check kaltura-sphinx daemon status] [PASSED, RC: 0] - [.187730524]
[Check kaltura-sphinx daemon is started by Monit] [PASSED, RC: 0] - [10.751940247]
[check daemon kaltura-sphinx init status] [PASSED, RC: 0] - [.008016185]
[Check kaltura-batch daemon status] [PASSED, RC: 0] - [.098646092]
Napping 1 seconds to allow the daemon kaltura-batch to finish its init after monit restarted it…
[Check kaltura-batch daemon is started by Monit] [PASSED, RC: 0] - [1.230869321]
[check daemon kaltura-batch init status] [PASSED, RC: 0] - [.008188884]
[Check kaltura-monit daemon status] [PASSED, RC: 0] - [.045807244]
[check daemon kaltura-monit init status] [PASSED, RC: 0] - [.006946899]
[Check memcached daemon status] [PASSED, RC: 0] - [.057254582]
Napping 1 seconds to allow the daemon memcached to finish its init after monit restarted it…
Napping 2 seconds to allow the daemon memcached to finish its init after monit restarted it…
Napping 3 seconds to allow the daemon memcached to finish its init after monit restarted it…
Napping 4 seconds to allow the daemon memcached to finish its init after monit restarted it…
Napping 5 seconds to allow the daemon memcached to finish its init after monit restarted it…
Napping 6 seconds to allow the daemon memcached to finish its init after monit restarted it…
Napping 7 seconds to allow the daemon memcached to finish its init after monit restarted it…
Napping 8 seconds to allow the daemon memcached to finish its init after monit restarted it…
Napping 9 seconds to allow the daemon memcached to finish its init after monit restarted it…
Napping 10 seconds to allow the daemon memcached to finish its init after monit restarted it…
[Check memcached daemon is started by Monit] [PASSED, RC: 0] - [10.785374175]
[check daemon memcached init status] [PASSED, RC: 0] - [.008258579]
[kaltura-html5lib ver in KMC config.ini] [PASSED, RC: 0] - [.184350168]
[kaltura-kdp3 ver in KDP3 config.ini] [PASSED, RC: 0] - [.102599234]
[kaltura-kmc ver in KMC config.ini] [PASSED, RC: 0] - [.122084556]
PHP Warning: file_get_contents(https://eliademyvideos.cloudberrytec.com:443/kmc): failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found
in /opt/kaltura/bin/get_kmc_swfs.php on line 9
[Get KMC SWFs] [FAILED, RC: 255] - [.313537322]
curl: no URL specified!
curl: try ‘curl --help’ or ‘curl --manual’ for more information
[check_start_page] [FAILED, RC: 1] - [.166639121]
curl: no URL specified!
curl: try ‘curl --help’ or ‘curl --manual’ for more information
[check_testme_page] [FAILED, RC: 1] - [.158836232]
[check_kmc_index_page] [FAILED, RC: 1] - [.152396426]
curl: no URL specified!
curl: try ‘curl --help’ or ‘curl --manual’ for more information
[check_admin_console_index_page] [FAILED, RC: 1] - [.165476737]
[check_studio_index_page] [FAILED, RC: 1] - [.173651818]
[check_clipapp_index_page] [FAILED, RC: 1] - [1.982496781]
[Create Partner] [FAILED, RC: 255] - [.504222900]
Partner creation failed. I will skip all tests that require it.
[Red5 file upload] [SKIPPED as OflaDemo isn’t configured]
see: https://github.com/kaltura/platform-install-packages/blob/master/doc/install-kaltura-redhat-based.md#configure-red5-server

Thank you for running Kaltura! To keep Kaltura viable, stable and well tested, please join the community and help by contributing sanity tests that verify overall platform stability: http://bit.ly/kaltura-ci , and by contributing to the project roadmap by solving simple tasks and challenges: http://bit.ly/kaltura-tasks.

2

Hi @dannybe,

First of all, seems you have an issue with your SSL certificate.
What does:
# apachectl -t -DDUMP_VHOSTS
output?
what does:
# curl -I -v $SERVICE_URL
output?

Once the SSL issues are resolved, if there are additional issues, easiest way to debug is to open your browser’s dev tools, make the request that fails [on KMC and Admin Console] and look at the “Network” and “Console” tabs for errors. In parallel, from a super user shell, run:
# kaltlog
and look at the errors outputted to STDOUT and STDERR.
kaltlog is an alias for

tail -f /opt/kaltura/log/*.log /opt/kaltura/log/batch/*.log | grep -A 1 -B 1 --color "ERR:\|PHP\|trace\|CRIT\|\[error\]"

So it should output all erroneous log lines, giving you an indication as to the final failure, then, you need to view the relevant log [in your case, it will probably be /opt/kaltura/log/kaltura_api_v3.log, /opt/kaltura/log/kaltura_apache_errors*.log and /opt/kaltura/log/kaltura_prod.log] to trace the root cause.

Feel free to paste all that info here so I can help you troubleshoot further.

3

Thanks Jess for responding so quickly…

I cleaned up the certs as best I could, here are the current output:

VirtualHost configuration:
52.178.190.241:* eliademyvideos.cloudberrytec.com (/etc/httpd/conf.d/zzzkaltura.ssl.conf:22)
wildcard NameVirtualHosts and default servers:
default:443 eliademyvideos.cloudberrytec.com (/etc/httpd/conf.d/ssl.conf:74)
Syntax OK

HEAD / HTTP/1.1
User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Host: eliademyvideos.cloudberrytec.com
Accept: /

< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Tue, 09 May 2017 03:25:01 GMT
Date: Tue, 09 May 2017 03:25:01 GMT
< Server: Apache/2.2.15 (CentOS)
Server: Apache/2.2.15 (CentOS)
< Connection: close
Connection: close
< Content-Type: text/html;charset=UTF-8
Content-Type: text/html;charset=UTF-8

<

  • Closing connection #0

I do have a slightly different error now:

I tried to follow (https://github.com/kaltura/platform-install-packages/blob/Lynx-12.14.0/doc/install-kaltura-redhat-based.md#ssl-step-by-step-installation) but not sure I set-up the SSL correctly for Kaltura. I am starting to go through logs but nothing yet points anywhere…

Thanks for the support, I’ll keep looking and get back to you when I find something.

4

Hi,

I work with Danny. The SSL errors were there because the certificate chain was not configured. Now everything works except the video conversion.
I can see the uploaded files correctly in the folder /opt/kaltura/web/content/uploads/
I tried the ‘entry-investigation’ in admin console and found that ‘Entry Processing History’ is empty for the uploaded entries. There is no In-progress tasks and Failed Tasks under ‘Batch Process Control’.
Can some one give me few pointers to debug?

  • Thanks,
    Rajeev
5

Issue is fixed. It was problem with directory permissions.