Delete files from remote storage when a Media Entry is deleted

Community Forums Backend Server Support
1

Hi,

I use Kaltura configured with external storage and I configured it to remove from local all exported files to remote storage.

I see that when I delete a media entry from Kaltura (via KMC or via API), Kaltura’s batch process don’t deletes the source and flavors from remote storage.

anyone can help me to understand if exists a workaround to do it?

thanks in advance for your suggestions

AML

1 Like
2

anyone knows if Kaltura can remove (automatically of course) from remote storage the flavors of deleted media entries ?

3

we had investigated this about a year ago and at the time this functionality was not working. To my Knowledge it is STILL not working, at least for S3 remote storage.

4

maybe I have to change something in post-convert processing.

anyway, thanks for your reply :wink:

5

Hi AML,

As it stands now, when deleting an entry from the API, it should also delete it from the remote storage, assuming the remote storage is set to “automatic” state.
An entry deletion via the API [which happens when using KMC or any other Kaltura management I/F as well], is suppose to trigger a deletion job from the remote storage.

These jobs are kept in the scheduler_worker table which you can look at with say:
mysql> select * from scheduler_worker where name like ‘%KAsyncStorageDelete%’\G

you can also check this specific job log under /opt/kaltura/log/batch/storagedelete*log

This is provided you have enabledWorkers.KAsyncStorageDelete set to 1 in /opt/kaltura/app/configurations/batch/batch.ini

6

Hi @jess,

as usual your reply is always useful.

I prefer start defining my hw architecture, I have a cluster installation with, in particular:
#2 FrontEnd + Sphinx + Batch
#1 Batch

By the way, I tried unsucessfly to turn off the batch on frontends (they seems restart automatically also if server is not restarted).

Then, following your suggestion I checked the database and I have (as expected) 3 KAsyncStorageDelete worker active and 2 old worker disabled around 1 month ago.
You can find the result of the query here in csv format:
https://dl.dropboxusercontent.com/u/72357116/batch_scheduled_worker_KAsyncStorageDelete.xml.csv

Also I checked the logs on the frontend and on dedicated batch server, but they seems old logs (around 1 month ago, but I deleted videos also during the last month):
https://dl.dropboxusercontent.com/u/72357116/storagedelete_logs.zip

Finally in my admin console I can see around 3000 jobs in queue, in particular they are “delete_file” jobs, but they seem frozen (see this screenshot: https://www.dropbox.com/s/u9w1nkq1qjnleoa/Screenshot%202016-04-05%2014.42.17.png?dl=0)

thanks in advance for you support

7

anyone have reported the same problem deleting entries from remote storage?