Polling for asset creation completion: Thumbs generated < 10 seconds last week, this week they take several minutes

Hi -

My solution uploads a video to Kaltura API, gets the new media’s entryId, and then sits and polls for completed creation of media assets (media duration, thumbnail, etc) by calling $kaltura_client->getThumbAssetService()->getByEntryId($entry_id); until the response is neither NULL nor throws THUMB_ASSET_IS_NOT_READY exception.

This was working just fine for the past week, but this week it fell apart: The thumb asset is usually eventually processed, but sometimes only after several minutes after initial upload. Sometimes, for the very same video assets, it seems it never gets processed at all (note: I’ve not made such a study that I have tried coming back to the same entryId hours later). I am working with a very small test video ~1MB in size, ~15 secongs in duration.

  • Is there an implied assumption that thumb/assets will be generated within some timeframe?
  • Is there maybe some throttle per-account if I am rapidly sending many videos up?

I am just looking for consistency and am not seeing what in my local codebase might possibly have changed to bring about this observations :frowning:

Context: My solution requires a thumbnail asset to already be attached to a newly uploaded Kaltura video before it can continue. It’s in Drupal 8 using Kaltura for Media in core which automatically processes the media thumbnail computed property.

Thanks
-Bronius

I ran another test just now and found I could process two videos and retrieve thumb assets just fine, reran, and they both failed (timed out) on me again. I re-reran the same videos, and one processed, and the other did not. When I went back to look at a much older entryId which previously had responded with THUMB_ASSET_IS_NOT_READY, it still responds with the same “thumb asset not ready” exception.

Are there times that Kaltura API gets really, really bogged down?

You are referring to their Saas service? If so, I can confirm that in the past year turnaround on media has slowed greatly. Conversion of edits used to start immediately and we now find that items are stuck in pending state for upwards of an hour at times. We have used it for 5 years and this slowness started about a year ago. I would have to assume they took on a large client and haven’t scaled up or downsized their infrastructure to save on cost.

1 Like

Ah. Maybe it was a larger influx of videos to process, maybe some key personnel who used to always manually monitor status and click a button left, but if videos aren’t getting processed in a timely way then some systems are getting crippled. Ours is more flexible.

Ok thanks for the personal testimony: it helps to validate my observations and plan next steps. We ended up building a queue of our own that periodically checks back until, eventually, it retrieves the needed metadata. It’s gotten complicated :slight_smile:

What I haven’t seen yet is a definitive or best-practices way of confirming an asset is complete other than polling for and handling THUMB_ASSET_IS_NOT_READY.

Cheers
-Bronius

I believe Kaltura can set notifications back to you via a http call when asset events are completed. I know you need to engage support to get that setup. You’d just need a publicly addressable http endpoint.