KalturaSshUrlResource

Hi all,

I’m trying to use KalturaSshUrlResource to ingest a video (media->addcontent) after media->add, but it keeps failing. Maybe I’m not using the right url structure ?

$resource = new \KalturaSshUrlResource();
$resource->url = $url;
$resource->privateKey = "privateKeyAsString";
$resource->publicKey = "publicKeyAsString";

The $url looks like the following

user@host-dns:/home/MVI_7757.MOV

Couldn't resolve host 'host-dns:'
Couldn't read file. Error: Couldn't resolve host 'host-dns:'

It’s working fine when I just use the scp command outside Kaltura’s API. I cannot find any proper example for KalturaSshUrlResource, can someone help me to figure this out or point me to a working example ?

Many thanks.

Hi @luca.guindani,

You can see an example for the usage here:
/opt/kaltura/app/plugins/bulk_upload/csv/batch/BulkUploadEntryEngineCsv.php

protected function addBulkUploadResult(KalturaBulkUploadResult $bulkUploadResult) 

URL should look like this:

'scp://user@host-dns:/home/MVI_7757.MOV';

An example for parsing it and extracing the data can be found here:
/opt/kaltura/app/batch/batches/Import/KAsyncImport.class.php

private function fetchFileSsh(KalturaBatchJob $job, KalturaSshImportJobData $data)
...
                        $parsedUrl = parse_url($sourceUrl);

                        $host = isset($parsedUrl['host']) ? $parsedUrl['host'] : null;
                        $remotePath = isset($parsedUrl['path']) ? $parsedUrl['path'] : null;
                        $username = isset($parsedUrl['user']) ? $parsedUrl['user'] : null;
                        $password = isset($parsedUrl['pass']) ? $parsedUrl['pass'] : null;
                        $port = isset($parsedUrl['port']) ? $parsedUrl['port'] : null;

                        $privateKey = isset($data->privateKey) ? $data->privateKey : null;
                        $publicKey  = isset($data->publicKey) ? $data->publicKey : null;
                        $passPhrase = isset($data->passPhrase) ? $data->passPhrase : null;

So, you can see PHP’s parse_url() function is used.

Long story short, you’re missing the scheme, just use:

'scp://user@host-dns:/home/MVI_7757.MOV'

as URL.

Hello @jess, thanks for the help.

Looks like the url is now working. But I still have a strange behavior.

The entry is stuck in the “Importing” state for a looooong time with some informations on the file size (Downloading file, size: 1162039088). So it looks like the file was found by Kaltura’s import job but is stuck and is not doing anything. But after some time I get the following error from kaltlog.

==> /opt/kaltura/log/batch/import-3-2017-06-27.log <==
2017-06-27 09:25:33 [886.858762] [2085999653] [40] [BATCH] [KJobHandlerWorker->closeJob] ERR: exception 'Exception' with message 'Error: Can't get file [/MVI_7757.MOV] - ' in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/batch/batches/KJobHandlerWorker.class.php(364): KalturaLog::err('Error: Can't ge...')

Am I missing something ?

Many thanks.

Hi @luca.guindani,

But from previous messages, I gather your file is under ‘/home’, not ‘/’, correct?

Hello @jess

Yes but if I do /home/MVI_7757.MOV I get a very clear error message.

Error: remote file [/home/MVI_7757.MOV] does not exist

When I try with /MVI_7757.MOV it seems like the file is found (bc the import task shows the file size: 1162039088). I just checked “/opt/kaltura/tmp/imports/” and I see the imported file but as soon as the transfer is finished I get the “Error: Can’t get file” in kaltlog.

I’m on a dev env with a dockerized version of Kaltura server (12.17) if it’s relevant.

Hi @luca.guindani,

Just so I’m sure, this is a bulk upload job you’re trying to run, correct?
First off, let’s increase log verbosity so that all directives with ‘.*priority.priority=4’ become ‘.*priority.priority=7’ in /opt/kaltura/app/configurations/logger.ini and reload Apache. Next, when a bulk upload job fails, you can download the log from KMC->Content->Bulk Upload Log.
In addition to that [and looking at the batch logs under /opt/kaltura/log/batch for errors], you should also go to Admin Console->Batch Process Control->Failed Tasks, where you should be able to get a drill down view of the given job and see what happened exactly.

Feel free to share your findings here for additional analysis and troubleshooting.

Hi @jess

Thanks for your answer.

It’s not a bulk upload, it’s a simple API call $client->media->addcontent($mediaEntry->id, $resource) with a KalturaSshUrlResource. I don’t want to upload multiple contents, I just want to ingest a video file from an other server on the network instead of ingesting it from a public url (KalturaUrlResource) which is working.

In the Batch Process Control, the only information I get is “Error: Can’t get file” & “import Failed”.

I’ll try to increase verbosity.

Hi @jess

After changing the verbosity I got the following in /opt/kaltura/log/batch/import-0-2017-06-27.log.

2017-06-27 16:14:28 [0.000212] [103760477] [56] [BATCH] [KalturaClientBase->doQueue] NOTICE: result (object dump): KalturaFreeJobResponse Object
(
    [job] => KalturaBatchJob Object
        (
            [id] => 202
            [partnerId] => 101
            [createdAt] => 1498572782
            [updatedAt] => 1498572868
            [deletedAt] => 
            [lockExpiration] => 
            [executionAttempts] => 
            [lockVersion] => 6
            [entryId] => 0_eb7yqv9e
            [entryName] => 
            [jobType] => 1
            [jobSubType] => 2
            [data] => KalturaSshImportJobData Object
                (
                    [privateKey] => private-key
                    [publicKey] => public-key
                    [passPhrase] => 
                    [srcFileUrl] => scp://user@host-dns:/test.m4v
                    [destFileLocalPath] => /opt/kaltura/tmp/imports/import_595267fb347cf.m4v
                    [flavorAssetId] => 
                    [fileSize] => 87921337
                    [relatedObjects] => 
                )

            [status] => 9
            [abort] => 
            [checkAgainTimeout] => 
            [message] => Error: Can't get file [/test.m4v] - 
            [description] => 
Error: Can't get file [/test.m4v] - 
            [priority] => 0
            [history] => Array
                (
                    [0] => KalturaBatchHistoryData Object
                        (
                            [schedulerId] => 87608
                            [workerId] => 20
                            [batchIndex] => 0
                            [timeStamp] => 1498572794
                            [message] => 
                            [errType] => 
                            [errNumber] => 
                            [hostName] => e3dc2d69a06a
                            [sessionId] => 1618889607
                            [relatedObjects] => 
                        )

                    [1] => KalturaBatchHistoryData Object
                        (
                            [schedulerId] => 
                            [workerId] => 
                            [batchIndex] => 
                            [timeStamp] => 1498572795
                            [message] => Downloading file, size: 87921337
                            [errType] => 0
                            [errNumber] => 0
                            [hostName] => 
                            [sessionId] => 
                            [relatedObjects] => 
                        )

                    [2] => KalturaBatchHistoryData Object
                        (
                            [schedulerId] => 
                            [workerId] => 
                            [batchIndex] => 
                            [timeStamp] => 1498572868
                            [message] => Error: Can't get file [/test.m4v] - 
                            [errType] => 1
                            [errNumber] => 99
                            [hostName] => 
                            [sessionId] => 
                            [relatedObjects] => 
                        )

                )

            [bulkJobId] => 
            [batchVersion] => 
            [parentJobId] => 
            [rootJobId] => 202
            [queueTime] => 
            [finishTime] => 
            [errType] => 1
            [errNumber] => 99
            [estimatedEffort] => 
            [urgency] => 
            [schedulerId] => 
            [workerId] => 
            [batchIndex] => 
            [lastSchedulerId] => 87608
            [lastWorkerId] => 20
            [dc] => 0
            [jobObjectId] => 
            [jobObjectType] => 4
            [relatedObjects] => 
        )

    [jobType] => 1
    [queueSize] => 1
    [relatedObjects] => 
)

2017-06-27 16:14:28 [0.000114] [103760477] [57] [BATCH] [KalturaClientBase->doQueue] NOTICE: execution time for [http://e3dc2d69a06a/api_v3/service/batch/action/freeExclusiveJob]: [0.036285161972046]
2017-06-27 16:14:28 [0.000097] [103760477] [58] [BATCH] [KJobHandlerWorker->freeExclusiveJob] INFO: Queue size: 1 sent to scheduler
2017-06-27 16:14:28 [0.000473] [103760477] [59] [BATCH] [KJobHandlerWorker::unsetCurrentJob] DEBUG: End job[202]
2017-06-27 16:14:28 [0.000101] [103760477] [60] [BATCH] [KBatchBase->done] INFO: Done after [74.039715766907] seconds

The file /opt/kaltura/tmp/imports/import_595267fb347cf.m4v exists and is a valid video file.

Hi @luca.guindani,

Maybe I am missing something here but if the file is under /home/test.m4v on the FS, I can’t see how:

Will work…
Is /opt/kaltura/tmp/imports/import_595267fb347cf.m4v really the file you tried to upload? you can easily verify by checking its SHA1.

Hi @jess

Yes it is

To be precise, the real path is /home/user/test.m4v

And I think when kaltura’s api is doing the scp://user@host-dns the current directory is already /home/user/. Anyway /opt/kaltura/tmp/imports/import_595267fb347cf.m4v is indeed the same as test.m4v. So the file seems to be found and downloaded in /opt/kaltura/tmp/imports/. But then I get the “Can’t get file” error, which I find odd since the file was found and downloaded previously.

@luca.guindani,

What are the permissions on /opt/kaltura/tmp/imports/import_595267fb347cf.m4v? and also:

ls -ald /opt/kaltura/tmp/imports/ 

@jess

[root@e3dc2d69a06a imports]# ls -la
-rw-r–r-- 1 kaltura kaltura 87921337 Jun 27 15:17 import_595267fb347cf.m4v

[root@e3dc2d69a06a imports]# ls -ald /opt/kaltura/tmp/imports/
drwxr-xr-x 2 kaltura kaltura 4096 Jun 27 15:16 /opt/kaltura/tmp/imports/

@jess

I also tried a KalturaServerFileResource ingest with files on a NFS share. But I got a permission issue and found the following post stating only -1 partner can use this type of resource.

So I’m stuck trying to make KalturaSshUrlResource work.

Hi @luca.guindani,

Please sure your full script and I’ll try to debug it locally.

Hi @luca.guindani,

After giving the code another look, I realised you should use this for the KalturaSshUrlResource::url:

sftp://user@host:/full/path/to/source/file

So, in your case, that would be:

sftp://user@host-dns:/home/user/test.m4v

I’m pretty sure if you’ll do the same from your code it will just work but just in case [and since I had to write this anyhow to test] here is full PHP CLI script [using the php5 client] you can run directly from the Kaltura Server for testing:

<?php

if (count($argv)<5){
    echo 'Usage:' .__FILE__ .' <service_url> <partnerid> <secret> <user@host:/path/to/remote/file>'."\n";
    exit (1);
}
function upload($client,$uri,$title,$conv_profile=null,$type=null)
{
        try{
                $entry = new KalturaBaseEntry();
                $entry->name = $title;
                $entry->tags = 'Example';
                $entry->description = 'Example Entry Description';
                if (isset($conv_profile)){
                        $entry->conversionProfileId = $conv_profile;
                }
                if (!isset($type)){
                        $type = KalturaEntryType::AUTOMATIC;
                }
                $resource = new KalturaSshUrlResource();
                $resource->url = 'sftp://'.$uri;
                $resource->publicKey = file_get_contents('/root/.ssh/id_rsa.pub');
                $resource->privateKey = file_get_contents('/root/.ssh/id_rsa');
                $result = $client->baseEntry->add($entry);
                $result = $client->baseEntry->addContent($result->id, $resource);
                $id=$result->id;
                return($id);
        }catch(exception $e){
                throw $e;
        }
}
$service_url = $argv[1];
$partnerId=$argv[2];
$secret=$argv[3];
$remote_uri = $argv[4];
$basedir=dirname(__FILE__);
require_once($basedir.'/create_session.php');
$client=generate_ks($service_url,$partnerId,$secret,$type=KalturaSessionType::USER,$userId=null);
$ext=explode(".",$remote_uri);
$id=upload($client,$remote_uri,date ("U",time()).'.'.$ext[1],null,null);
echo "Entry ID is $id\n";

Note that this makes use of /opt/kaltura/bin/create_session.php to generate a client, so just put the above script under /opt/kaltura/bin/ on your Kaltura Server and invoke it with:

# php /opt/kaltura/bin/upload_test.php <service_url> <partnerid> <secret> user@host-dns:/home/user/test.m4v

and it should work.
Note the script already concats ‘sftp://’ to the remote URI so do not pass that in the last arg.
Also, you’ll want to adjust the $resource->publicKey and $resource->privateKey to match your paths, these can be passed as args to the script of course but this was just a POC.

Hello @jess

Awesome ! It’s working perfectly with sftp://.

Thanks a lot for your help :+1: