Batch in cluster

Hello Kalturians!

I’m trying to configure more batch servers/nodes in order to speed up the conversion process in my Kaltura installation.
By the way, in my installation I use a remote storage configuration.

Reading the documentation on github and on http://knowledge.kaltura.com/introduction-kaltura-batch-processes I found manu useful information , but I can’t understand if, in Kaltura cluster installtion, any batch process “push” the converted file/flavor, from local filesystem directly to remote storage or before to another Kaltura server and, then, from this server to remote storage.

This information would be useful to find a correct ha configuration for batch server/node.

Any suggestion would be really appreciated.

Thanks

AML

No one is using Kaltura clustered with more batch nodes? :no_mouth:

Hello,

Batch machines operation by pulling jobs from the operational kaltura DB.
The files they operate on are placed on the NFS volume which is common to all front and batch nodes and is mounted on /opt/kaltura/web.

Each such batch node has a unique ID in /opt/kaltura/app/configurations/batch/batch.ini
for example:
[jessex : mainTemplate]
id = 91674

See also:


or:

If you have more specific questions, let me know.

Hi jess, the files are placed on the NFS volume although I have configured the remote storage?

Yes. They will always be placed on NFS as well. No point in having all the nodes go all the way to the CDN when processing the jobs. CDN is most important when serving the content to end users. For batch job processing, it will take longer for no good reason.

As far as deciding how to serve the content, you can set that up in admin console->partner row->actions->configure
Where you’ll see:

Remote Storage Policy
Delivery Policy
Which has the following options:
Kaltura Only - serve directly from NFS and only from it
Kaltura First - first try from NFS
Remote storage first - try remote first
Remote storage only - only serve from remote

Hi jess, yes I understand the reason to don’t push directly the video on CDN from batch nodes, but I wrote about remote storage, no reference to the cdn. I’m a bit 'confused for your reply.

Thanks also for pointing the remote storage policies, I have already set this policy of “Remote Storage first - try first remote” and I also enabled both “Delete exported” and “Purge on deletion” files.

I continue to explain just hoping it will be useful for other users who might want the same configuration. My configuration creates the local flavors (videos and thumbnails), then copy them to remote storage (only the videos), but it keeps the thumbnails locally on NFS. Then delete the local file system the exported video.

Now it is also clear that the NFS has the burden to manage all the files from the various batches. I suppose both video thumbnails.

Thanks

Remote storage would typically be a CDN where you intend to serve the files from.
Although yes, you can push them to any remote location provided there’s a class that knows how to talk to it. So indeed, it does not HAVE to be a CDN. Hope things are clearer now.