Nginx issues, presumably after upgrade

Hey all,

I recently updated my RPM kaltura cluster to 13.9. Everything seemed to be working fine after the upgrade, but now i have embedded players that won’t play videos. The browser consoles logs that nginx is returning 404 for url’s like this:

http://kaltura-prod-stream.intranet.hdr:88/hls/p/103/sp/10300/serveFlavor/entryId/0_djsuyld6/v/2/ev/4/flavorId/0_uuc59qwm/name/a.mp4/index.m3u8

After the upgrade, I ran all the required config scripts. Base, frontend, sphinx and nginx on the required machines. I’m kind of at a loss as to what to look at. The nginx errors log does contain lines like this…

2018/01/03 21:10:16 [error] 17439#17439: *7733 media_set_parse_json: failed to parse json -1: expected digit got 0x0 while sending to client, client: 10.4.16.32, server: kaltura-prod-stream.intranet.hdr, request: "GET /hls/p/103/sp/10300/serveFlavor/entryId/0_djsuyld6/v/2/ev/4/flavorId/0_uuc59qwm/name/a.mp4/index.m3u8 HTTP/1.1", host: "kaltura-prod-stream.intranet.hdr:88", referrer: "http://hdrtv.intranet.hdr/watch.aspx?id=0_djsuyld6"

I’ve checked the nginx.conf and it seems to be correct. Everything else seems to be working as well.

Could someone help point me in the right direction? I know I’ve fixed this before but I can’t recall what i did nor do my notes have anything in them about this.

Thanks!

OK, so I guess this has something to do with the internal IP range I had defined.

I changed this on the front end servers, restarted httpd and plays are happening again. I’m not sure why, though. The range should work, though I am going through a load balancer that wouldn’t be in the range I set…

What model of the nginx cookbook are you the usage of? In v2.2.zero, i am getting a barely different hassle in which it downloads the new version and tries to collect – but it fails within the assemble step presumably because of wrong path records being used from the pre-present node facts.

  • bash[unarchive_source] action run[2014-09-15T21:21:48+00:00] data: Processing bash[unarchive_source] action run (nginx::source line 82)
    [2014-09-15T21:21:48+00:00] info: bash[unarchive_source] ran efficaciously

       - execute "bash"  "/tmp/chef-script20140915-20734-1lacceo"
     * bash[compile_nginx_source] movement run[2014-09-15T21:21:48+00:00] data: Processing bash[compile_nginx_source] motion run (nginx::supply line 97)
    

    ================================================================================
    blunders executing action run on resource ‘bash[compile_nginx_source]’

    Mixlib::ShellOut::ShellCommandFailed

       predicted process to exit with [0], however acquired '1'
       ---- start output of "bash"  "/tmp/chef-script20140915-20734-16zr4ti" ----
       STDOUT: 
    
       STDERR: /tmp/chef-script20140915-20734-16zr4ti: line 1: cd: nginx-1.6.1: No such file or directory
       ---- give up output of "bash"  "/tmp/chef-script20140915-20734-16zr4ti" ----
       Ran "bash"  "/tmp/chef-script20140915-20734-16zr4ti" again 1
    
       aid announcement:
    
       ---------------------
    

In /tmp/kitchen/cache/cookbooks/nginx/recipes/supply.rb

        ninety seven: bash 'compile_nginx_source' do
        98:   cwd  ::file.dirname(src_filepath)
        ninety nine:   code <<-EOH
       a hundred:     cd nginx-#{node['nginx']['source']['version']} &&
       one hundred and one:     ./configure #{node.run_state['nginx_configure_flags'].be part of(" ")} &&
       102:     make && make deploy

       103:   EOH
       104: 
       one hundred and five:   not_if do
       106:     nginx_force_recompile == fake &&
       107:       node.automatic_attrs['nginx'] &&
       108:       node.automatic_attrs['nginx']['version'] == node['nginx']['source']['version'] &&
       109:       node.automatic_attrs['nginx']['configure_arguments'].type == configure_flags.sort
       110:   give up
       111: 
       112:   notifies :restart, 'service[nginx]'
       113:   notifies :reload,  'ohai[reload_nginx]', :right away
       114: give up

Hi @batmanandjoker,

Kaltura CE requires a special version of Nginx which includes, among other Nginx modules, the Nginx VOD module. It will not work with the default Nginx Chef recipe.

A long time ago, I uploaded a Chef recipe for deploying Kaltura CE which you can find here:
https://supermarket.chef.io/cookbooks/kaltura

I’m afraid it hasn’t been maintained in a while now and will not work as is but it can be a good starting point if you’re interested in updating it and submitting a pull request here:

I can tell you right off the bat that you will need to modify kaltura/templates/default/kaltura.repo.erb since the repo structure changed [see details here Kaltura CE 13.4.0 released] and also kaltura/templates/default/kaltura.ans.erb seeing how additional configuration options were added since the last time this recipe was updated.
You can use this template as reference for the changes you will need to make in kaltura/templates/default/kaltura.ans.erb.

Also, the existing recipe will not work on Debian/Ubuntu since when it was last updated, only RPMs for CentOS/RHEL were available whereas now, Kaltura CE also has deb packages for Debian and Ubuntu.

If you do decide to take this project on, feel free to post any questions you may have here:)

Hey @jess

I’m afraid this is happening again on our kaltura ce cluster after upgrading to 14.10. I would appreciate it if you or anyone else could point me in the right direction. This happens every time we upgrade and I haven’t been able to figure out exactly why yet.

I have two front end servers, both running apache and nginx. Both apache and nginx are behind a load balancer (F5). with unique IP’s and hostnames. kaltura-prod and kaltura-prod-stream, respectively

I’ve checked the URL’s in the delivery profiles, they are right.

The internal_ip_range is 0.0.0.0-255.255.255.255. These machines are inside a private network only.

The messages in the kaltura_nginx_errors.log is the same as in the original post.

I’m going to keep digging here. Any help is greatly appreciated.

Just an update here. It seems kaltura-config-all with a good answer file cures all. Things are playing again, though, I’m occasionally getting timeouts for playback in the nginx log. I experimented with using the uivar disableHLSOnJs just in case, to fall back to progressive download. That seems to work OK as well.