Now that Cantemo’s iconik system is live and available, I thought it would be time to show of some examples of automation in a cloud based system like this. It is a little different than working with pure on-prem systems and there are a few things that we need to go over first.
First and foremost, iconik is a hosted system that is always on and running in the cloud. This means that you don’t necessarily have the access to it as you would a local Portal system (or any other premise based software). While it does have a full API stack for sending it information and querying information, you can’t install a script “on iconik” that sits and listens for work to do. That said, the devs at Cantemo were clever enough to give us tools to enable pro-active scripting. This comes in the form of WebHooks. For the uninitiated, WebHooks are simply notification options that send a payload of data (in this case JSON) to a URL. Simple as that. So if we want to pro-actively monitor what is going on in iconik and then do stuff when our event happens, we’ll have to build a listener.
For this example, I’ve used Flask and it’s restful component to build a simple standalone application that simply listens to notifications from iconik and then, well, does stuff.
The next thing we have to worry about here is the the URL we put into iconik has to be visible to the iconik service. This means you’ll either have to make your on-prem dev server public facing OR you can spin up a small instance in your favorite cloud provider. For the sake of this example, I used Amazon’s EC2 service and spun up a nano instance of CentOS 7. There are plenty of good tutorials on the web to do this yourself if you’d like.
Once you have a working system and have shell access, I needed to set up my Flask environment. I like to use Python’s virtualenv system so my little app can be self contained. So first we need to install it. Which means we need pip.
On a clean shell, installing pip is pretty easy:
sudo curl https://bootstrap.pypa.io/get-pip.py | python |
This will install pip on our system. Once pip is installed, we can then install virtualenv:
pip install virtualenv |
Now lets make a little container for our app.
mkdir iconik-listener |
Then we go into our directory and set up virtualenv
cd iconik-listener virtualenv . |
This will create a self contained python environment in our new folder. Lastly, we need activate our environment and install a few dependencies.
source bin/activate . pip install requests Flask flask-restful |
This will install the Flask framework, flask-restful extension, and requests (so we can talk back to the iconik API)
Now that we have our environment fired up, time to do a few other things. First, we need to get our App-ID and App-Secret for interacting with the iconik API. Log into your iconik system as an admin, click the Admin pane at the top, then Settings on the left side. Two options will spin down, one for WebHooks and one for Application Tokens. The Webhooks panel is where we are going to tell iconik where to send information. We’ll come back to that in a bit. Application Tokens will allow us to create authorization for our application via a unique ID and shared key. Go ahead and create a new Application Token, name it whatever you choose, and generate a new secret for it. In my example here I’ve just called my key “test” and allowed it admin rights to iconik. Depending on your use case, you may want to limit rights and name it more appropriately.
Copy both the App ID and Token, we’ll need them in our script.
Last thing we need to do is quickly initialize our sqlite database that will store our object IDs and their status. Go to your virtualenv folder and run the following:
sqlite3 items.db SQLite version 3.7.17 2013-05-20 00:56:22 Enter ".help" for instructions Enter SQL statements terminated with a ";" sqlite> create table items (object_id,finished); sqlite> .quit |
Now its time to write some code. Lets create a new text file called “app.py” in our virtualenv we created earlier. Go ahead and open up that file. We are going to build a simple app that looks for all new items added to iconik and then submits them for analytics once a proxy is successfully associated with them (necessary to generate analytics)
First we need to import or libraries:
from flask import Flask, jsonify, request from flask_restful import Api, Resource, reqparse import json import sqlite3 import requests |
This allows us to use the flask framework for quickly building a REST API (which is what iconik’s web hooks will talk to), sqlite lite to store what we are working on locally, and requests to make URL calls to iconik’s REST API. Next we put in our auth tokens from above:
app_key = 'your app ID from iconik' app_secret = 'your app secret from iconik' app_url = 'your iconik domain url (default is https://www.iconik.io, but if you are in an enterprise/single tenant install you'll need to put your URL in)' |
Next we initialize our Flask environment
app = Flask(__name__) api = Api(app) |
After this, I’m going to create a simple function that takes an object ID and submits it for video/image analytics. More info on this can be found in the iconik API docs.
def submit_analytics(object_id): headers = {'Content-Type': 'application/json','Accept':'application/json','App-ID':app_key,'Auth-Token':app_secret} r = requests.post(app_url + '/API/transcode/v1/analyze/assets/' + object_id + '/',headers=headers,data='{}') r.raise_for_status() |
Next I’m going to build a class/REST endpoint using flask for every new object that gets created. It will store that object ID in a sqlite database with a flag whether or not the proxy has finished being generated/uploaded for this item. When JSON is sent to the web hook, it will look something like this:
{
"data": {
"analyze_status": "N/A",
"created_by_user": "f1b89efa-6cb5-11e7-b50b-0a580a0000f8",
"date_created": "2018-01-03T18:30:26.885375+00:00",
"date_deleted": null,
"date_modified": "2018-01-03T18:30:26.885390+00:00",
"deleted_by_user": null,
"external_id": null,
"id": "2b45a8ca-f0b4-11e7-a36b-0a580a00054a",
"status": "ACTIVE",
"title": "1_zps7uj5xqff.jpg",
"type": "ASSET"
},
"event_type": "assets",
"object_id": "2b45a8ca-f0b4-11e7-a36b-0a580a00054a",
"operation": "create",
"realm": "entity",
"system_domain_id": "bf0b8c12-7ccc-11e7-8d93-0a580a00022b"
} |
So I parse that data in the post, then I pass the object ID into my sql database.
class Create(Resource): def post(self): json_data = request.get_json(force=True) print "Creating new queue item for object_id " + json_data['object_id'] conn = sqlite3.connect('items.db') c = conn.cursor() c.execute('insert into items values (\'' + json_data['object_id'] + '\',\'False\')') conn.commit() conn.close() |
Next I’m going to create a simple function that listens for any updates to proxies in the system. I’m waiting on proxy update web hook responses. It is going to look something like this:
{
"data": {
"asset_id": "c521e01a-f0a2-11e7-b528-0a580a000257",
"bit_rate": 8,
"codec": "JPEG",
"filename": "c87231ac-f0a2-11e7-9777-0a580a000756.jpg",
"format": "JPEG",
"frame_rate": "0",
"id": "c87231ac-f0a2-11e7-9777-0a580a000756",
"is_drop_frame": false,
"name": "2015-Mille-lowres.jpg",
"resolution": {
"height": 720,
"width": 1280
},
"start_time_code": null,
"status": "CLOSED",
"storage_id": "f7d92dd8-7ccc-11e7-8801-0a580a000229"
},
"event_type": "assets",
"object_id": "c521e01a-f0a2-11e7-b528-0a580a000257",
"operation": "update",
"realm": "proxies",
"system_domain_id": "bf0b8c12-7ccc-11e7-8d93-0a580a00022b"
} |
So my class is going to look and see if my proxy is completed – I’m specifically watching the assets event type and proxies realm (we will get into this at the end), so the only thing I really need here is to watch for status in the data block and make sure it is CLOSED. Once I’m confident it is, I look to see if that object ID is in my list of items needing to be processed (the sqlite database), mark it as processed, and then submit it back to the iconik API for analytics.
class Proxy(Resource): def post(self): json_data = request.get_json(force=True) #print json.dumps(json_data,indent=4, sort_keys=True) current_items = [] conn = sqlite3.connect('items.db') c = conn.cursor() for row in c.execute('select object_id from items where finished=\'False\''): current_items.append(row[0]) if json_data['data']['status'] == 'CLOSED': if json_data['object_id'] in current_items: print "Found object " + str(json_data['object_id']) + " in finished proxy queue" c.execute('update items set finished = \'True\' where object_id = \'' + json_data['object_id'] + '\'') conn.commit() conn.close() print "Submitting Analyze Job" submit_analytics(json_data['object_id']) |
Now that we have the functions for our endpoints done, we just have to finish up the script. We have our classes, we have to tell them the endpoints on our http server:
api.add_resource(Proxy,'/proxy') api.add_resource(Create,'/create') |
Lastly, we need to tell our app to start up.
if __name__ == '__main__': app.run(host='0.0.0.0',port=8000,debug=True) |
In that last block, I told the flask server to fire up on port 8000 and run on any interface, and enable debug (just in case there are errors, I can see them in the response. You wouldn’t do this in production code). So now I’ve built a short REST API that runs on port 8000 of our server with /proxy and /create endpoints. These are what we are going to put into iconik’s web hook interface.
http://ip.of.my.server:8000/create http://ip.of.my.server:8000/proxy
Let’s go ahead and fire up our app. Make sure we are in our virtual env directory and it is activated, and simply run:
python app.py |
If everything is working, you should be presented with info about your flask app, a debugger pin, and status output of any commands that come into the system.
* Running on http://0.0.0.0:8000/ (Press CTRL+C to quit) * Restarting with stat * Debugger is active! * Debugger PIN: 286-658-059 |
Since we haven’t actually told iconik to talk to our API yet, we still have one more step. Let’s go back to iconik, log in as an admin, click on the Admin menu, click settings, and then web hooks. Now we need to tell iconik where to send its notifications – to our new app. We have to create messages for our create endpoint and our proxy endpoint. For create, we’ll use the “assets” Type, “create” Operation, and “entity” Realm. Then put in our URL to our new app (http://ip.of.my.server:8000/create) into the last box. For proxy, we’ll use the “assets” Type, “update” Operation, and “proxies” Realm. Put in our URL for our proxy class here (http://ip.of.my.server:8000/proxy).
Go ahead and upload a new item to iconik and watch your console. If all is working well, you should see some feedback to the information being sent to your new API as well as some output when it decides to do things.
Creating new queue item for object_id cea2a082-f0b0-11e7-bd64-0a580a00054a 35.187.117.108 - - [03/Jan/2018 18:06:23] "POST /create HTTP/1.1" 200 - 35.187.117.108 - - [03/Jan/2018 18:06:40] "POST /proxy HTTP/1.1" 200 - Found object cea2a082-f0b0-11e7-bd64-0a580a00054a in finished proxy queue Submitting Analyze Job 35.189.206.72 - - [03/Jan/2018 18:06:41] "POST /proxy HTTP/1.1" 200 - |
I can see that a new queue item got created with an object ID, then a few other posts happened to my proxy endpoint, and one of them triggered completion. I went to the item I uploaded, and my analytics were there!
As you can see, this is only scratching the tip of what is possible with web hooks and automation. Any sort of action that occurs in iconik can generate a web hook to notify external applications to do whatever you may want them to do, from moving files, updating metadata, triggering external tools, etc. Hopefully this rough primer gets you up and running on your path to writing some web hook integration with iconik.