Load Balancer and Kaltura

Hello,
Is it possible to deploy a Load Balancer after the cluster was already installed, and if that is possible what would be the necessary required steps Thank you.

Regards.
Jacob R.

Hi Jacob,

Yes, it is certainly possible and for a cluster, it is even needed.
Essentially, whatever load balancer you choose, you just need to make sure all front nodes use it as SERVICE_URL and that it is also set in the kaltura.delivery_profile if applicable [i.e, when not serving from a CDN].

There is a guide here explaining how to configure either Apache with mod_proxy or HAProxy as load balancers:

But really, Kaltura can work with any other so long as it supports sticky sessions, which most if not all, do.

Thanks Jess,
I changed the hostname of the front node with the name of load balancer and put a new name on the front node, after disabling iptables and everything seems kaltura is responding from behind the load balancer with no problems, i’m adding the ssl certificate now. Last dash before the cluster is fully functional with all the services working as they should…fingers crossed ! A case of beer for Jess if everything goes right :slight_smile:

Regards,
Jacob R

G’luck Jacob.
Let me know if you need further assistance.

Dear Jess,
I’ve deployed the SSL certificated on my load balancer with HA proxy, but with the SSL on I cannot access the API.

curl -I -v $SERVICE_URL/api_v3/

HEAD /api_v3/ HTTP/1.1
User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Host: portal.servcast.net
Accept: /

< HTTP/1.1 302 Found
HTTP/1.1 302 Found
< Cache-Control: no-cache
Cache-Control: no-cache
< Content-length: 0
Content-length: 0
< Location: https://portal.servcast.net/api_v3/
Location: https://portal.servcast.net/api_v3/
< Connection: close
Connection: close

This is my HAproxy configuration:

global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
#log loghost local0 info
maxconn 4096
#chroot /usr/share/haproxy
user haproxy
group haproxy
daemon
#debug
#quiet
tune.ssl.default-dh-param 2048

defaults
log global
mode http
option forwardfor
option http-server-close
option dontlognull
retries 3
option redispatch
option forceclose
stats enable
maxconn 2000
contimeout 5000
clitimeout 50000
srvtimeout 50000

frontend www-http
bind 162.244.80.123:80
reqadd X-Forwarded-Proto:\ http
default_backend www-backend

frontend www-https
bind 162.244.80.123:443 ssl crt /tmp/portal.servcast.pem
reqadd X-Forwarded-Proto:\ https
default_backend www-backend

backend www-backend
redirect scheme https if !{ ssl_fc }
server www-1 162.244.80.215:80 check

listen stats *:1936
mode http
stats enable
stats uri /
stats hide-version
stats auth admin:K4ltclust3r!

Managed to solve this issue, i was missing a little thing.
Thanks,

Actually i can access the KMC but i cannot access the admin console behind the SSL … Please advise.

Basically i am getting

An error occurred

(error code: API:-1)

After entering the credentials on the admin console login page, which gets load correctly with the ssl.

But the KMC works.

SERVICE_URL= is the front node. ( node1.servcast.net)

Now I get
curl -I -v $SERVICE_URL/api_v3/index.php

HEAD /api_v3/index.php HTTP/1.1
User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Host: node1.servcast.net
Accept: /

< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Wed, 06 Jul 2016 15:20:04 GMT
Date: Wed, 06 Jul 2016 15:20:04 GMT
< Server: Apache/2.2.15 (CentOS)
Server: Apache/2.2.15 (CentOS)
< X-Powered-By: PHP/5.3.3
X-Powered-By: PHP/5.3.3
< Access-Control-Allow-Origin: *
Access-Control-Allow-Origin: *
< Expires: Sun, 19 Nov 2000 08:52:00 GMT
Expires: Sun, 19 Nov 2000 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< Pragma: no-cache
Pragma: no-cache
< X-Kaltura-Session: 172602483
X-Kaltura-Session: 172602483
< Vary: Accept-Encoding
Vary: Accept-Encoding
< X-Me: portal.servcast.net
X-Me: portal.servcast.net
< Connection: close
Connection: close
< Content-Type: text/xml
Content-Type: text/xml

But still get the same error in admin console.
What i did was change the service_url on the front node and batch to point to the front node itself and not the balancer.

Hello,

First of all, please make sure the LB is configured to support sticky sessions.
Secondly, I see your X-Me header says portal.servcast.net, which I imagine is the LB’s addr. Change it on each node here:
/opt/kaltura/app/configurations/apache/conf.d/enabled.kaltura.conf
so that is has:
Header set X-Me "YOUR_NODE"
and reload Apache. This is not the reason why it does not work but looking at the X-Me header will indicate which node you hit. Then, you can go to that node and run:
# kaltlog
from the shell to see the errors the request generates, then, go to each relevant log, usually the errors will be stored in either /opt/kaltura/log/kaltura_api_v3.log or /opt/kaltura/log/kaltura_apache_errors.log and look at the lines leading to the final error. If you cannot understand what the root cause is, please paste the relevant lines here so I can further help you.

Thanks,

Hi Jess,
This is from kaltlog:

2016-07-07 07:31:38 [0.000180] [1194904296] [7] [%context%] [Kaltura_Client_ClientBase->doCurl] DEBUG: post: {“format”:“2”,“ignoreNull”:true,“clientTag”:“Kaltura-admin”,“apiVersion”:“3.3.0”,“ks”:“YzhiM2EwYmM4NjEyMGY2MzIyMDc4MDEyODYwNjY4MTY5Y2UwMzgwY3wtMjstMjsxNDY3OTc3Mzg1OzI7MTQ2Nzg5MDk4NS45NjA2O2FkbWluQHNlcnZjYXN0Lm5ldDtkaXNhYmxlZW50aXRsZW1lbnQ7Ow==”,“kalsig”:“b3b81005d667d36a131af34cf8065a7d”}
2016-07-07 07:31:38 [0.002163] [1194904296] [8] [%context%] [ErrorController->errorAction] ERR: exception ‘Kaltura_Client_ClientException’ with message ‘. RC : 302’ in /opt/kaltura/app/admin_console/lib/Kaltura/Client/ClientBase.php:919
Stack trace:
#0 /opt/kaltura/app/admin_console/lib/Kaltura/Client/ClientBase.php(249): Kaltura_Client_ClientBase->getKalturaClientException(’. RC : 302’, -1)

This is from apache:

[Thu Jul 07 07:47:48 2016] [error] [client 162.244.80.123] File does not exist: /opt/kaltura/app/alpha/web/opt, referer: https://portal.servcast.net/admin_console/index.php/partner/list

In the api_v3 log i get nothing when i do the request via https://

The kmc works on https, i only get this on the admin console.

This is basically the error i’m getting when accessing via https, the wierd thing is as soon as i inform the load balancer to go only via http the error goes away.

Can you try logging into admin_console from a diff browser? I suspect it might be a local browser side cache issue.

Hi,
I get the same thing from different browsers, different machines. different external ip’s, it’s the first thing I tried.
If needed I can provide access to the front node or to any of the machines.

Also this is the curl response:

curl ‘https://portal.servcast.net/admin_console/index.php/user/login’ --data ‘email=xxxxxxx&password=xxxxxxxx’ --compressed -v -k

  • About to connect() to portal.servcast.net port 443 (#0)
  • Trying 162.244.80.123… connected
  • Connected to portal.servcast.net (162.244.80.123) port 443 (#0)
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • warning: ignoring value of ssl.verifyhost
  • skipping SSL peer certificate verification
  • SSL connection using TLS_DHE_RSA_WITH_AES_256_CBC_SHA
  • Server certificate:
  •   subject: CN=portal.servcast.net,OU=Domain Control Validated
    
  •   start date: Jul 04 15:16:38 2016 GMT
    
  •   expire date: Jul 04 15:16:38 2017 GMT
    
  •   common name: portal.servcast.net
    
  •   issuer: CN=Go Daddy Secure Certificate Authority - G2,OU=http://certs.godaddy.com/repository/,O="GoDaddy.com, Inc.",L=Scottsdale,ST=Arizona,C=US
    

POST /admin_console/index.php/user/login HTTP/1.1
User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Host: portal.servcast.net
Accept: /
Accept-Encoding: deflate, gzip
Content-Length: 46
Content-Type: application/x-www-form-urlencoded

< HTTP/1.1 500 Internal Server Error
< Date: Fri, 08 Jul 2016 13:42:27 GMT
< Server: Apache/2.2.15 (CentOS)
< X-Powered-By: PHP/5.3.3
< Set-Cookie: PHPSESSID=t9fdkp846hr07ch3i1vj4dvjj1; path=/admin_console; secure; HttpOnly
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< Pragma: no-cache
< X-Kaltura-Session: 43360132
< Set-Cookie: PHPSESSID=g6a1tc54v2f0ivc1rjiseqknp1; path=/admin_console; secure; HttpOnly
< X-Kaltura-Errorcode: API:-1
< X-Frame-Options: SAMEORIGIN
< X-Me: node1.servcast.net
< Content-Length: 522
< Connection: close
< Content-Type: text/html; charset=UTF-8
<

Kaltura Admin Console
            <h1>An error occurred</h1>
            (error code: API:-1)

    </div>
    <div id='loader'></div>
* Closing connection #0

< Set-Cookie: PHPSESSID=t9fdkp846hr07ch3i1vj4dvjj1; path=/admin_console; secure; HttpOnly—
So basically this means that the cookies are accepted only via http that’s why it works on http and not on https ?
And if so what are the necessary steps to remedy this ?

Regards,
Jacob R.

Hi Jacob,

You’re getting < HTTP/1.1 500 Internal Server Error when making this request.
This means somewhere in /opt/kaltura/log/kaltura_api_v3.log or /opt/kaltura/log/kaltura_apache_errors.log there is a fatal error to find and fix. The request you posted reached node1.servcast.net, look at the logs there to find the error, then I can help you fix it.

Dear Jess,
On https mode, i am not getting any type of request in the API log when accessing the admin_console, if i changed it back to http mode i’m getting a lot of requests…
Also in the apache logs the only entry i get when accessing the page is

[error] [client 162.244.80.123] File does not exist: /opt/kaltura/app/alpha/web/opt, referer: https://portal.servcast.net/admin_console/index.php/partner/list , entry which when I change to http mode i do not get in logs.

API LOG

Httpd Logs

Hi,
I’ve managed to get it working on https by removing the redirect check to https from the load balancer, but still very strange that when I enable the redirect i get that error only in the admin console.
The only inconvenience is that one has to manually input https:// in the link otherwise he will go by default on the 80 port.

redirect scheme https if !{ ssl_fc } – this was the line for the redirect.

So I guess even if it’s not the ideal way to go, i can say it’s done, the thread can be closed.