Remote Storage Only Not working

jlotito · January 3, 2018, 4:40pm

Hi experts,
We have deployed a kaltura cluster in different ec2 machines in aws. We have also configured a partner ID, with a delivery profile and storage profile. In the Partner ID we have checked the “REMOTE STORAGE ONLY” option… Our Remote profile is configured to use a S3 aws bucket. When we create a new entry, the flavors and its content is well stored in the bucket. And the Delivery Profile works ok because the video is delivered under the cloudfront aws connected to the bucket.

So aparently everything is working correct. But the problem is that THE NFS. The shared disk created under aws too, prepared to share all the html files in all the fronts (I have 3 fronts) is increasing and increasing without stop. That makes me to pay a lot of money to keep the NFS. Why kaltura stored the videos in the NFS if i have Remote Storage ONLy checked??

Navigating acroos the forum and the files in the deploy, I discovered the next things… Aparently in the batch proccess when you create a new entry… all flavors (with extension *.mp4) are stored in the content directory. (/opt/kaltura/web/content/entry/data/0)… in this directory i have found not only the *.mp4 files moreover the *.jpg files related to the thumbanils of the entry… The first question would be… Why there is so many jpg for each entry?? then Why these *.jpg are not stored in the s3??

These files are increasing my NFS to limits that nobody can achieve… (last checked 450 GB) I have around 1500 entries in 3 different partners… The next question will be… after the batch process and the flavors be upload to the s3… why are not erased from the NFS? Which are the files that I must check in order to solve this problem?? maybe is a permission issue??

After that I would like to know too where I have to stop (which lines do i have to comment and in which files) the automatic thumbnail creator because my system use personal images to create the thumbanil and this proccess which makes a lot of images is competely inefficient.

And finally a simple question why the mp4 files are not stored in the NFS directory following the PartnerID location?? example for the partner 104 the path should be “/opt/kaltura/web/content/entry/data/0/104” the system creates directory with names (0,1,2) and i would like to know what are the logic to create the names by this way. And why the entry are stored in those folders without any criteria… some videos of one partners in one folder some in others…

Thanks for all

jess · January 10, 2018, 8:33pm

Hi @jlotito,

The “REMOTE STORAGE ONLY” option simply means that there will be no attempt to serve the content directly from the NFS in the event it was not found on the remote storage. The content will ALWAYS have to first be saved on /opt/kaltura/web/content because the source files need to be manipulated in various ways, mainly, different flavours should be created based on your transcoding profiles. Only after this is done, can the resulted files be exported to the remote storage.
Go to Admin Console->Batch Process Control and input an entry ID into the search box.
This will show the entire ingestion flow and all the jobs performed on a given entry.

There is an option in the Admin Console->partner->configure view called “Delete exported storage from Kaltura” but it does not seem to be working. I’ll try to allocate some time to debug this. Either way, you’d want to leave at least the source media file on the NFS because that’s needed in case you’ll want to reconvert, create thumbnails or perform other manipulations that require the source.

In regards to exporting the images [thumbnails] to the remote storage, there is such an option in the Admin Console->remote storage view, under the “Advanced” section. It is called “Should export thumbnail assets” but it does not seem to work. Again, I’ll try to allocate some time to debug this and understand why.

As to your question about creating thumbnails, one thumbnail will always be created when ingesting a new entry.
I would really advise against commenting pieces of code to disable the creation of a thumbnail, especially since you’ll have to manually patch it after every upgrade which is not a good idea.

That said, you can disable the creation of additional thumb assets by calling conversionprofile->update() on your default conversion profile and setting flavorParamsIds to ‘0,2,3,4,5,6,7’ instead of the default ‘0,2,3,4,5,6,7,19’.
19 is a flavour param called “Default Thumbnail”, the ID is a static one so it’s the same for all ENVs. When removing this ID from flavorParamsIds on the default conversion profile, the additional thumb asset you see under “Thumbnails” in KMC’s entry drill down view will not be auto created.

You can perform the update with a simple API script but you can also use the Kaltura CLI client for that.
To generate the CLI client, run this command on one of your Kaltura nodes:

# php /opt/kaltura/app/generator/generate.php cli

then:

# cd  /opt/kaltura/web/content/clientlibs/cli
# chmod +x setup.sh
# ./setup.sh </path/to/target/dir> <api_hostname> <partner_id> <partner_admin_secret>

Once that’s done, go to /path/to/target/dir and run:

$ . kalcliAutoComplete
$ . kalcliAliases.sh 
$ kalcli -x conversionprofile list filter:objectType=KalturaConversionProfileFilter filter:systemNameEqual=Default ks=`genks -b $PARTNER_ID`

This will return something similar to:

KalturaConversionProfileListResponse
        objects array
                0       KalturaConversionProfile
                        id      9
                        partnerId       101
                        status  2
                        type    1
                        name    Default
                        systemName      Default
                        tags
                        description     The default set of flavors. If not specified otherwise all media uploaded will be converted based on the definition in this profile
                        createdAt       1515498358      (2018-01-09 06:45:58)
                        flavorParamsIds 0,2,3,4,5,6,7,19
                        isDefault       1
                        isPartnerDefault        1
                        cropDimensions  KalturaCropDimensions
                                left    -1
                                top     -1
                                width   -1
                                height  -1
                        clipStart       -1
                        clipDuration    -1
                        mediaParserType 0
                        calculateComplexity     1
                        collectionTags  mbr,ism
                        detectGOP       0
        totalCount      1

So, in the above example, the default conversion profile ID is 9.
Then, call:

$ kalcli -x conversionprofile update id=9 conversionProfile:objectType=KalturaConversionProfile conversionProfile:flavorParamsIds=0,2,3,4,5,6,7 ks=`genks -b 101`

Replacing 9 with your profile ID and 101 with your partner ID.

Regardless of your current question, the CLI client is a rather powerful tool when you need to perform simple API operations without writing code. It features tab completion for the services, actions and enums. You can learn more about how it works from the README under /opt/kaltura/web/content/clientlibs/cli/README.md

That brings us to your final question about a different FS structure for your media.

Before reading on, please understand that this operation should be carefully tested on a test machine before making any changes to your live production ENV.

In the Remote Storage profile configuration view, there is a selectbox called “Path Manager”.
Set that to “XSL Path” and go to the “Advanced” section.
There, in the “Path Manager Params (JSON)” textarea, you can input a JSON that contains a XSL describing an alternative structure.
For example, inputting this

[{"key":"path_format","value":"<xsl:value-of select=\"$partnerId\" \/><xsl:text>\/<\/xsl:text><xsl:value-of select=\"php:function('date', 'Y', $currentTime)\"\/><xsl:text>-<\/xsl:text><xsl:value-of select=\"php:function('date', 'm', $currentTime)\"\/><xsl:text>\/<\/xsl:text><xsl:value-of select=\"php:function('date', 'd', $currentTime)\"\/><xsl:text>\/<\/xsl:text><xsl:value-of select=\"$entryId\" \/><xsl:text>_<\/xsl:text><xsl:value-of select=\"$objectId\"\/><xsl:text>.<\/xsl:text><xsl:value-of select=\".\/content\/@extension\"\/>","relatedObjects":null}]

Would create the following structure under your remote volume’s root:

$REMOTE_STORAGE_ROOT/$PARTNER_ID/%Y-%m/%d/$ENTRY_ID_$FLAVOR_ID.$FILE_EXT

For example:

$REMOTE_STORAGE_ROOT/101/2018-01/10/0_mka2lp86_0_lfo55tdw.mp4

This will only take affect when uploading new entries, of course, the old will still need to be served from the original location. You can move them and then adjust the file_sync table but it’s a risky operation that should be carefully tested outside of production and I’d advise against it unless it is absolutely necessary.

Also, under “Advanced” you’ll see that you can choose which flavours to export to the remote storage [like I said the “Should export thumbnail assets” does not seem to be working at the moment but exporting only selected flavours does work, I tested]. Since you intend to serve all flavours from the remote storage, you should export them all but you can skip exporting the source flavour which would save some space on the remote volume. Depending on your needs, you may want to remove some flavours from the default transcoding profile or create several profiles you can choose from when ingesting an entry, naturally, the less flavours you have and the more compact they are, the less storage will be used.

In relation to conversion profiles and flavour params, you may find this thread instructive as well: