Add Proxy for Bulk Upload

Community Forums Backend Server Support
#1

Hey,

I am currently working on a clustered Kaltura 12.2.0 Community Edition.
The entire Cluster is behind a Firewall and the only possibility to reach the web is via a proxy.

Sadly this means that I can not execute a bulk upload, since with every request the Batch-server tries to download the video without using the proxy.

Is it possible to tell the Batch-server to use the proxy for bulk uploads?

best regards

#2

Hi @Raumen837,

Yes it is. In /opt/kaltura/app/configurations/batch/batch.ini under the ‘Worker’ section, add:
clientConfig.proxyHost
clientConfig.proxyPort
clientConfig.proxyUser
clientConfig.proxyPassword

The code that handles that is in /opt/kaltura/app/batch/client/KalturaClientBase.php, starting at line 545:

                if (isset($this->config->proxyHost)) {
                        curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, true);
                        curl_setopt($ch, CURLOPT_PROXY, $this->config->proxyHost);
                        if (isset($this->config->proxyPort)) {
                                curl_setopt($ch, CURLOPT_PROXYPORT, $this->config->proxyPort);
                        }    
                        if (isset($this->config->proxyUser)) {
                                curl_setopt($ch, CURLOPT_PROXYUSERPWD, $this->config->proxyUser.':'.$this->config->proxyPassword);
                        }    
                        if (isset($this->config->proxyType) && $this->config->proxyType === 'SOCKS5') {
                                curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_SOCKS5);
                        }    
                }

in case you’ll need to debug it.

#3

@jess

I added the code but nothing changed :confused:

I Added (with other url):

[Worker]
maximumExecutionTime                    = 300
params.tempDirectoryPath                = /opt/kaltura/tmp
engineOptions.asperaTempFolder  = /opt/kaltura/tmp/aspera_upload
clientConfig.verifySSL          = 0
clientConfig.proxyHost          = http://proxy.proxy.zsa.com
clientConfig.proxyPort          = 8080

[message] => Error: Failed connect to media.cdn.edge-cnd.net:80; Connection refused

This is the same message if i dont set the Proxy.

best regards

#4

Can you add a debug print in /opt/kaltura/app/batch/client/KalturaClientBase.php, right after:

if (isset($this->config->proxyHost)) {

?
something like:

error_log($this->config->proxyHost.':'.$this->config->proxyPort."\n",3,'/tmp/proxy');

and then see if /tmp/proxy was created on disk? Also, you mentioned you have a cluster, perhaps you did not update all batch servers?

#5

So,
I added the code to the KalturaClientBase.php.

The Log Contains (changed Domain):
http://proxy.proxy.zsa.com:8080
http://proxy.proxy.zsa.com:8080
…

I use the same proxy in my bash environment.

http_proxy=http://proxy.proxy.zsa.com:8080
FTP_PROXY=http://proxy.proxy.zsa.com:2121
HTTPS_PROXY=http://proxy.proxy.zsa.com:8080
https_proxy=http://proxy.proxy.zsa.com:8080
HTTP_PROXY=http://proxy.proxy.zsa.com:8080

hmmm

#6

Let’s start by running this simple PHP snippet from the batch server:

<?php
$ch = curl_init();  // Initialise a cURL handle
$proxy='http://proxy.proxy.zsa.com:8080' ;
$url='media.cdn.edge-cnd.net';
// Setting proxy option for cURL
curl_setopt($ch, CURLOPT_PROXY, $proxy);    // Set CURLOPT_PROXY with proxy in $proxy variable
 
// Set any other cURL options that are required
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_COOKIESESSION, TRUE);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_URL, $url);
 
$results = curl_exec($ch);  // Execute a cURL request
curl_close($ch);    // Closing the cURL handle
var_dump($results);

You can put this into a file anywhere, even under /tmp/test.php and use PHP CLI to run it, like so:
$ php /tmp/test.php

Let’s see what the result is…

#7

Hey,

I tried this script and with a success ! :slight_smile:

[custadm@cluster_trans_01 ~]$ php test.php > video.mp4
* About to connect() to proxy proxy.proxy.zsa.com port 8080 (#0)
*   Trying 10.160.4.90...
* Connected to proxy.proxy.zsa.com (10.160.4.90) port 8080 (#0)
> GET http://media.cdn.edge-cdn.net/video.mp4 HTTP/1.1
Host: media.cdn.edge-cdn.net
Accept: */*
Proxy-Connection: Keep-Alive

< HTTP/1.1 200 OK
< Date: Thu, 27 Oct 2016 08:39:00 GMT
< ETag: "3626881712"
< Server: edge
< Content-Type: video/mp4
< Accept-Ranges: bytes
< Last-Modified: Fri, 07 Oct 2016 09:12:24 GMT
< Content-Length: 10637933
< Proxy-Connection: Keep-Alive
< Zsa-Upstreamproxy-Hostname: proxy.proxy.zsa.com
< access-control-allow-origin: *
< 
* Connection #0 to host proxy.proxy.zsa.com left intact
#8

Hm…

Can you try commenting out the below block in /opt/kaltura/web/content/clientlibs/php5/KalturaClientBase.php:

                if (isset($this->config->proxyHost)) {
                        curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, true);
                        curl_setopt($ch, CURLOPT_PROXY, $this->config->proxyHost);
                        if (isset($this->config->proxyPort)) {
                                curl_setopt($ch, CURLOPT_PROXYPORT, $this->config->proxyPort);
                        }
                        if (isset($this->config->proxyUser)) {
                                curl_setopt($ch, CURLOPT_PROXYUSERPWD, $this->config->proxyUser.':'.$this->config->proxyPassword);
                        }
                        if (isset($this->config->proxyType) && $this->config->proxyType === 'SOCKS5') {
                                curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_SOCKS5);
                        }
                }

Replacing it with just:
curl_setopt($ch, CURLOPT_PROXY, ‘http://proxy.proxy.zsa.com:8080’);
and see if that works for you?

#10

Due to issues with our Deadline we were able to pressure our client enough to give us a firewall exception.

Thank you very much for helping me with this issue @jess!

#11

Sure, happy to help.