How to setup Remote Storage and a Delivery Profile

rpelletier · March 30, 2015, 6:15pm

I have an iscsi disk mounted to a local directory on my kaltura box at /opt/kaltura/web/content/mediastore
I tried to setup a remote storage profile using base path /opt/kaltura/web/content/mediastore and automatic but none of my uploaded media goes to this folder. I have read this tutorial from blackyboy but it is far more complicated that I need.

github.com

blackyboy/RedHat-Centos-Common-Stuffs/blob/master/Setting-up-Kaltura-Drop-folder,-Amazon-S3-Remote-storage,CloudFront-CDN-RTMP,RTMPE-Video-Streaming.md

### Setting up a Drop Folder

After Creating a Publisher in Kaltura 

1. Creat a Transcoding Profile ( I have created in my name as blackyboy transcoding 

2. Then Configure DropBox for Publisher by choosing configure in Drop menu

3. And Tick the check box, Content Ingestion - Drop Folder/s (config)

4. Then Click on configure and change the settings.

Note your Publisher ID from user's list ( My Publisher ID 102) and Create using Type : Local

Drop Folder Name: our Wish ( Here i have used blackyboy)

Description: As our Wish (This is blackyboy's Drop)

5. Conversion Profile ID: Choose your created name Here from Drop list

This file has been truncated. show original

If there is already a write up for how to configure remote storage and a delivery profile, I have been unable to find it.

I would appreciate any assistance to get this setup and if there is indeed no write available, I would be happy to write one myself once I figure out what I am doing. Thank you in advance.

jess · March 31, 2015, 10:21am

Hello there,

What exactly do you wish to accomplish? do you want to upload entries to this drive and have them auto processed? or do you want entries to be uploaded to this drive instead of the current location?

if the first, you just need to setup a dropfolder which you can do by going to the admin console, and then, on your partner’s row select ‘drop folders’ under profile, then select ‘local’ and input relevant data.
There is also a test that sets up a local drop folder using API:
/opt/kaltura/bin/dropbox_test.php
you can run it to configure your dropbox as well.
For the later, you can just copy the contents of /opt/kaltura/web to it and then mount it over NFS so that the entries will be saved there instead of on the local file system. If you wish to have them both on local and on this drive, then you will need to create a remote storage of type ‘local’ and also a delivery profile but I don’t think this is what you need…

Thanks,

rpelletier · March 31, 2015, 4:45pm

I suppose you are correct in that setting up remote storage might be overkill.

Regarding the drop folder, I do not believe this will work for me, because I want the files to be imported with different users/owners, and my experience when I tried it before is it imported all media with the owner of the publisher name. Maybe I was missing something.

Just mounting the iscsi volume to /opt/kaltura/web might be the way to go, but I was avoiding this because I liked the idea of configuring it as remote storage instead. However, this might indeed be overkill. Do I really need to copy all of the contents from /opt/kaltura/web to the mounted volume, or would just the content directory suffice? Your thoughts.

Thank You

jess · April 1, 2015, 1:58pm

Essentially, entries are all kept under /opt/kaltura/web/content so yes, you can do that.

rpelletier · April 29, 2015, 1:37pm

It seems that since I did this my bandwidth and storage reports are no longer working. They just say N/A for the months of March and April. Are there are reasons this may be happening?

jess · April 29, 2015, 2:10pm

Hello,

I suggest you start by clearing all the logs from /opt/kaltura/dwh/logs, then run:
# ps fax|grep pentaho
Make sure no proc is running, if it does, kill it.
run this query:
mysql> select * from kalturadw_ds.locks;
Make sure the lock_state is set to 0.

Then run:
logrotate -vvv -f /etc/logrotate.d/kaltura_apache
/opt/kaltura/dwh/etlsource/execute/etl_hourly.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh
/opt/kaltura/dwh/etlsource/execute/etl_update_dims.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh
/opt/kaltura/dwh/etlsource/execute/etl_daily.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh
/opt/kaltura/dwh/etlsource/execute/etl_perform_retention_policy.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh
/opt/kaltura/app/alpha/scripts/dwh/dwh_plays_views_sync.sh >> /opt/kaltura/log/cron.log

and check the log for lines with the string ERR in them. Then we can advance further…

rpelletier · April 29, 2015, 2:20pm

I will do, but just to give you more information. It appears as though none of my analytics are working since 2/19, which is the day I performed an upgrade.

rpelletier · April 29, 2015, 2:26pm

mysql> select * from kalturadw_ds.locks;
+---------+------------------------------+---------------------+------------+
| lock_id | lock_name                    | lock_time           | lock_state |
+---------+------------------------------+---------------------+------------+
|       1 | daily_lock                   | 2015-02-19 08:59:09 |          0 |
|       2 | retention_lock               | 2015-03-20 12:30:08 |          1 |
|       3 | update_dims_lock             | 2015-02-19 10:00:13 |          1 |
|       4 | hourly_emlnx-media1.emcc.edu | 2015-04-29 10:00:10 |          0 |
+---------+------------------------------+---------------------+------------+
4 rows in set (0.00 sec)

rpelletier · April 29, 2015, 2:27pm

A couple lock_states are listed as 1. I am waiting to perform the next steps until you advise me how to proceed.

I apologize for the multiple replies.

jess · April 29, 2015, 2:38pm

no need to apologize:) just update the lock_state and set to 0. Then continue as detailed above.

rpelletier · April 29, 2015, 3:02pm

# cat /opt/kaltura/log/cron.log | grep ERR
2015-04-29 10:00:01 [global] ERR: exception 'Exception' with message 'Couldn't find entry [0_xzl7jb6o]' in /opt/kaltura/app/infra/log/KalturaLog.php:82
2015-04-29 10:00:01 [global] ERR: exception 'Exception' with message 'Couldn't find entry [0_bob3ey3n]' in /opt/kaltura/app/infra/log/KalturaLog.php:82
2015-04-29 10:00:01 [global] ERR: exception 'Exception' with message 'Couldn't find entry [0_5triojc4]' in /opt/kaltura/app/infra/log/KalturaLog.php:82
2015-04-29 10:00:01 [global] ERR: exception 'Exception' with message 'Couldn't find entry [0_hi8krfeh]' in /opt/kaltura/app/infra/log/KalturaLog.php:82
2015-04-29 10:00:01 [global] ERR: exception 'Exception' with message 'Couldn't find entry [0_bscrnhtc]' in /opt/kaltura/app/infra/log/KalturaLog.php:82
2015-04-29 11:03:00 [KalturaStatement->execute] DEBUG: /* emlnx-media1.emcc.edu[1023309429][propel] */ SELECT entry_distribution.ID, entry_distribution.CREATED_AT, entry_distribution.UPDATED_AT, entry_distribution.SUBMITTED_AT, entry_distribution.ENTRY_ID, entry_distribution.PARTNER_ID, entry_distribution.DISTRIBUTION_PROFILE_ID, entry_distribution.STATUS, entry_distribution.DIRTY_STATUS, entry_distribution.THUMB_ASSET_IDS, entry_distribution.FLAVOR_ASSET_IDS, entry_distribution.ASSET_IDS, entry_distribution.SUNRISE, entry_distribution.SUNSET, entry_distribution.REMOTE_ID, entry_distribution.PLAYS, entry_distribution.VIEWS, entry_distribution.VALIDATION_ERRORS, entry_distribution.ERROR_TYPE, entry_distribution.ERROR_NUMBER, entry_distribution.ERROR_DESCRIPTION, entry_distribution.LAST_REPORT, entry_distribution.CUSTOM_DATA FROM `entry_distribution` WHERE entry_distribution.ENTRY_ID='0_4d6gmocf' AND entry_distribution.STATUS<>'3'
2015-04-29 11:03:00 [global] ERR: exception 'Exception' with message 'Couldn't find entry [0_xzl7jb6o]' in /opt/kaltura/app/infra/log/KalturaLog.php:82
2015-04-29 11:03:00 [global] ERR: exception 'Exception' with message 'Couldn't find entry [0_bob3ey3n]' in /opt/kaltura/app/infra/log/KalturaLog.php:82
2015-04-29 11:03:00 [global] ERR: exception 'Exception' with message 'Couldn't find entry [0_5triojc4]' in /opt/kaltura/app/infra/log/KalturaLog.php:82
2015-04-29 11:03:00 [global] ERR: exception 'Exception' with message 'Couldn't find entry [0_hi8krfeh]' in /opt/kaltura/app/infra/log/KalturaLog.php:82
2015-04-29 11:03:00 [KalturaStatement->execute] DEBUG: /* emlnx-media1.emcc.edu[1023309429][propel] */ SELECT entry_distribution.ID, entry_distribution.CREATED_AT, entry_distribution.UPDATED_AT, entry_distribution.SUBMITTED_AT, entry_distribution.ENTRY_ID, entry_distribution.PARTNER_ID, entry_distribution.DISTRIBUTION_PROFILE_ID, entry_distribution.STATUS, entry_distribution.DIRTY_STATUS, entry_distribution.THUMB_ASSET_IDS, entry_distribution.FLAVOR_ASSET_IDS, entry_distribution.ASSET_IDS, entry_distribution.SUNRISE, entry_distribution.SUNSET, entry_distribution.REMOTE_ID, entry_distribution.PLAYS, entry_distribution.VIEWS, entry_distribution.VALIDATION_ERRORS, entry_distribution.ERROR_TYPE, entry_distribution.ERROR_NUMBER, entry_distribution.ERROR_DESCRIPTION, entry_distribution.LAST_REPORT, entry_distribution.CUSTOM_DATA FROM `entry_distribution` WHERE entry_distribution.ENTRY_ID='0_qnfmz0os' AND entry_distribution.STATUS<>'3'
2015-04-29 11:03:00 [global] ERR: exception 'Exception' with message 'Couldn't find entry [0_bscrnhtc]' in /opt/kaltura/app/infra/log/KalturaLog.php:82
2015-04-29 11:03:00 [global] ERR: exception 'Exception' with message 'Couldn't find entry [0_029lzwwu]' in /opt/kaltura/app/infra/log/KalturaLog.php:82
2015-04-29 11:03:00 [global] ERR: exception 'Exception' with message 'Couldn't find entry [0_h6vaq2mp]' in /opt/kaltura/app/infra/log/KalturaLog.php:82
2015-04-29 11:03:00 [global] ERR: exception 'Exception' with message 'Couldn't find entry [0_2i1a2qhg]' in /opt/kaltura/app/infra/log/KalturaLog.php:82
2015-04-29 11:03:01 [KalturaStatement->execute] DEBUG: /* emlnx-media1.emcc.edu[1023309429][propel] */ SELECT entry_distribution.ID, entry_distribution.CREATED_AT, entry_distribution.UPDATED_AT, entry_distribution.SUBMITTED_AT, entry_distribution.ENTRY_ID, entry_distribution.PARTNER_ID, entry_distribution.DISTRIBUTION_PROFILE_ID, entry_distribution.STATUS, entry_distribution.DIRTY_STATUS, entry_distribution.THUMB_ASSET_IDS, entry_distribution.FLAVOR_ASSET_IDS, entry_distribution.ASSET_IDS, entry_distribution.SUNRISE, entry_distribution.SUNSET, entry_distribution.REMOTE_ID, entry_distribution.PLAYS, entry_distribution.VIEWS, entry_distribution.VALIDATION_ERRORS, entry_distribution.ERROR_TYPE, entry_distribution.ERROR_NUMBER, entry_distribution.ERROR_DESCRIPTION, entry_distribution.LAST_REPORT, entry_distribution.CUSTOM_DATA FROM `entry_distribution` WHERE entry_distribution.ENTRY_ID='0_g9wc82co' AND entry_distribution.STATUS<>'3'
2015-04-29 11:03:01 [KalturaStatement->execute] DEBUG: /* emlnx-media1.emcc.edu[1023309429][propel] */ SELECT entry_distribution.ID, entry_distribution.CREATED_AT, entry_distribution.UPDATED_AT, entry_distribution.SUBMITTED_AT, entry_distribution.ENTRY_ID, entry_distribution.PARTNER_ID, entry_distribution.DISTRIBUTION_PROFILE_ID, entry_distribution.STATUS, entry_distribution.DIRTY_STATUS, entry_distribution.THUMB_ASSET_IDS, entry_distribution.FLAVOR_ASSET_IDS, entry_distribution.ASSET_IDS, entry_distribution.SUNRISE, entry_distribution.SUNSET, entry_distribution.REMOTE_ID, entry_distribution.PLAYS, entry_distribution.VIEWS, entry_distribution.VALIDATION_ERRORS, entry_distribution.ERROR_TYPE, entry_distribution.ERROR_NUMBER, entry_distribution.ERROR_DESCRIPTION, entry_distribution.LAST_REPORT, entry_distribution.CUSTOM_DATA FROM `entry_distribution` WHERE entry_distribution.ENTRY_ID='0_96gasyju' AND entry_distribution.STATUS<>'3'
2015-04-29 11:03:01 [KalturaStatement->execute] DEBUG: /* emlnx-media1.emcc.edu[1023309429][propel] */ SELECT entry_distribution.ID, entry_distribution.CREATED_AT, entry_distribution.UPDATED_AT, entry_distribution.SUBMITTED_AT, entry_distribution.ENTRY_ID, entry_distribution.PARTNER_ID, entry_distribution.DISTRIBUTION_PROFILE_ID, entry_distribution.STATUS, entry_distribution.DIRTY_STATUS, entry_distribution.THUMB_ASSET_IDS, entry_distribution.FLAVOR_ASSET_IDS, entry_distribution.ASSET_IDS, entry_distribution.SUNRISE, entry_distribution.SUNSET, entry_distribution.REMOTE_ID, entry_distribution.PLAYS, entry_distribution.VIEWS, entry_distribution.VALIDATION_ERRORS, entry_distribution.ERROR_TYPE, entry_distribution.ERROR_NUMBER, entry_distribution.ERROR_DESCRIPTION, entry_distribution.LAST_REPORT, entry_distribution.CUSTOM_DATA FROM `entry_distribution` WHERE entry_distribution.ENTRY_ID='0_f2gzuosx' AND entry_distribution.STATUS<>'3'
2015-04-29 11:03:01 [KalturaStatement->execute] DEBUG: /* emlnx-media1.emcc.edu[1023309429][propel] */ SELECT entry_distribution.ID, entry_distribution.CREATED_AT, entry_distribution.UPDATED_AT, entry_distribution.SUBMITTED_AT, entry_distribution.ENTRY_ID, entry_distribution.PARTNER_ID, entry_distribution.DISTRIBUTION_PROFILE_ID, entry_distribution.STATUS, entry_distribution.DIRTY_STATUS, entry_distribution.THUMB_ASSET_IDS, entry_distribution.FLAVOR_ASSET_IDS, entry_distribution.ASSET_IDS, entry_distribution.SUNRISE, entry_distribution.SUNSET, entry_distribution.REMOTE_ID, entry_distribution.PLAYS, entry_distribution.VIEWS, entry_distribution.VALIDATION_ERRORS, entry_distribution.ERROR_TYPE, entry_distribution.ERROR_NUMBER, entry_distribution.ERROR_DESCRIPTION, entry_distribution.LAST_REPORT, entry_distribution.CUSTOM_DATA FROM `entry_distribution` WHERE entry_distribution.ENTRY_ID='0_en0zz1hc' AND entry_distribution.STATUS<>'3'
2015-04-29 11:03:01 [KalturaStatement->execute] DEBUG: /* emlnx-media1.emcc.edu[1023309429][propel] */ SELECT entry_distribution.ID, entry_distribution.CREATED_AT, entry_distribution.UPDATED_AT, entry_distribution.SUBMITTED_AT, entry_distribution.ENTRY_ID, entry_distribution.PARTNER_ID, entry_distribution.DISTRIBUTION_PROFILE_ID, entry_distribution.STATUS, entry_distribution.DIRTY_STATUS, entry_distribution.THUMB_ASSET_IDS, entry_distribution.FLAVOR_ASSET_IDS, entry_distribution.ASSET_IDS, entry_distribution.SUNRISE, entry_distribution.SUNSET, entry_distribution.REMOTE_ID, entry_distribution.PLAYS, entry_distribution.VIEWS, entry_distribution.VALIDATION_ERRORS, entry_distribution.ERROR_TYPE, entry_distribution.ERROR_NUMBER, entry_distribution.ERROR_DESCRIPTION, entry_distribution.LAST_REPORT, entry_distribution.CUSTOM_DATA FROM `entry_distribution` WHERE entry_distribution.ENTRY_ID='0_uohp5iyx' AND entry_distribution.STATUS<>'3'