Note: I originally posted this on Xsanity but it seemed to make sense to move it to my personal blog as well.
So I figured since this is an often talked about product that I would share a little bit of configuration information for everyone out there. This is by no means a definitive way of accomplishing the task at hand, but a start for those interested in using Archiware PresSTORE for advanced archive management. This article assumes that you have a basic knowledge of PresSTORE’s Archive module already. This article is also written assuming MacOS X is the server running PresSTORE. While all the concepts are true across all platforms, the qt_tools command line package is only available on Mac OS X. Other platforms may have to use utilities like FFMPEG or FlipFactory. Remember, if using the PresSTORE server to encode your proxy clips before archival, your PresSTORE server has to be able to read and open those files! If you are in a very heavy quicktime environment, Mac/Windows may be your only option unless you want to write a script that would hand off preview generation to an existing Episode Engine/Flip Factory/Qmaster cluster.
It should be noted that this is only possible in version of PresSTORE newer than 3.2.4 as offloading of proxy generation was not available in previous versions of the software, only the standard quicktime clipping is used. By default, if a media file is quicktime viewable and no proxy generator is set, PresSTORE can trim a segment of that clip in full raster/full res for later use. If all of your content is already low res, this may be a better option, but for the majority of us the archive tool is being used to take high res content off of expensive storage so in practice this isn’t that practical.
Configuration of preview generation is done in the “Preview Generation” menu in any Archive plan that you would like to set up.
The “User Defined Preview Generators” is where we are going to set up our scripts and our wildcards to support various media encodes. For example, if we have a better tool for encoding .MXF files or we want to pre-combine audio and video from OP-Atom MXF files, we may want to use a different script for these than for our .mov and .dv files. The most important thing to note here is that if a filetype is not called out, the system falls back onto the stock audio/video preview generator. In the example case above, if any media file was selected for archive that was not a .mov or .dv file, a 3 second section of that clip, starting 3 seconds into the clip, would be saved at full res into our preview index. If we are dealing with uncompressed 10-bit HD, this could be almost a 600MB file, so think about your preview generators and your media workflow before enabling any of this.
1) Install qt_tools
The first thing we need to do is grab qt_tools. This package gives MacOS X systems command line access to the quicktime export capabilities. This is a freeware package supported by David Van Brink. If you like the software and use it heavily, I would recommend donating to him :D.
The installer package can be downloaded here: http://omino.com/sw/qt_tools/
The disk image has an install.sh file on it. You will need to be locally in the directory to run the installer.
shell# cd /Volumes/qt_tools/2_8_574/ shell# sudo ./install.sh
2) Creating Settings
To use qt_tools we need to come up with the settings that we would like to use for qt_tools to encode with. Since quicktime uses a binary settings file, we actually have to do some test encodes and find a setting that meets our size/quality standards. I have built some basic encode settings that can be downloaded here, but lets walk through creating your own if mine do not fit your needs.
I have created a directory at “/usr/local/scripts” and a subdirectory “/usr/local/scripts/encode_settings” to hold our scripts and settings.
To generate a settings file, we have to trigger qt_export (newly installed with qt_tools) to encode a file and present us with a dialog to use our quicktime gui. If you are creating these scripts in directories not accessible to your current user, it would make sense to run the following commands as “sudo” or simply entering a root shell with “sudo -s”. To create our settings, we use the following command:
shell# qt_export --dodialog --savesettings="/usr/local/scripts/encode_settings/test.setting"
This will create a setting file with the options you have chosen for your proxy. I’ve found H.264 to work well and create visible yet extremely small files.
I generally set dimensions to 320×180, which is roughly 1/4 SD resolution (16:9 aspect ratio). I also set preserve to letterbox and deinterlace all incoming video.
For audio, MPEG4 @ 22.05kHz, 16 bit mono generates extremely small files and is still usable.
Overall, our settings dialog for this example would look like so:
Disable “Prepare for Internet Streaming” as we are not streaming this file, it is a progressive download.
Verify that your settings file got created and audit it quickly with the following command:
shell# qt_atom /usr/local/scripts/encode_settings/test.setting
If the settings file was created successfully, you will get a hexadecimal dump of the settings file. If creation failed, qt_atom will exit in an error.
3) Testing our encode settings
Now we are ready to do our first test encode to make sure our settings look good. For this article, I will use “test.mov” that is located in “/Users/Shared/Video”. Replace as necessary.
To test our encode, we will use the following command:
shell# qt_export --loadsettings="/usr/local/scripts/encode_settings/test.setting" /Users/Shared/Video/test.mov /tmp/test.mov
We are loading the settings file we just created in the first input, specifying our input file in the second input, and our destination output in the third input. For this example, I’ve put our test file in /tmp/test.mov. If all went well, you should have had a progress output in your CLI and qt_export should exit cleanly back to the CLI. We can now look at our movie.
shell# open /tmp/test.mov
Run a few movies through this process, return back to step two to tweak settings until you have a settings file that you want to use. Once you are ready, on to the next step.
4) Creating our script for PresSTORE
I put my scripts into /usr/local/scripts, but any location will work.
By default, PresSTORE passes one variable to the script – the full filesystem path of the media file being encoded. It expects only one return to STDOUT, and that is the full path to the proxy file that was generated. It is important to note that you need to make sure all calls of your script either go to a log file or that you echo the entire script to a log file and explicitly return just the full path of your encoded clip to STDOUT. The first bit of text returned to STDOUT will be interpreted as the path, so if you are using a 3rd party codec or have codecs installed that return info to STDOUT, you will want to make sure you use the second method.
Here is an example of a script that will work for this kind of setup. This script is saved to “/usr/local/scripts/proxy.sh” on my system.
#!/bin/bash
MYDIR=/tmp
MYFILE=${1##*/}
PROXYPATH="$MYDIR/$MYFILE"
echo "`date` -- encoding proxy for $1" >> /Library/Logs/awproxy.log
/usr/local/bin/qt_export --loadsettings=/usr/local/scripts/encode_settings/test.setting "$1" "$PROXYPATH" >> /Library/Logs/awproxy.log
echo "`date` -- encoding proxy complete, proxy at $PROXYPATH" >> /Library/Logs/awproxy.log
echo "$PROXYPATH"
Lets break this script down a bit.
MYDIR=/tmp
MYFILE=${1##*/}
PROXYPATH="$MYDIR/$MYFILE"
These three lines are simply setting up our environment and creating some variables for us. Line 1 is where I am going to encode to, line 2 is taking just the file name from our full path that was passed by PresSTORE to the script, and line 3 is just creating a single argument full path of our destination.
echo "`date` -- encoding proxy for $1" >> /Library/Logs/awproxy.log
This line simply opens a log file at “/Library/Logs/awproxy.log” and lets us know encoding started.
/usr/local/bin/qt_export --loadsettings=/usr/local/scripts/encode_settings/test.setting "$1" "$PROXYPATH" >> /Library/Logs/awproxy.log
This is the meat of our script. It is simply running our settings file against our original file and encoding it to our destination. I am also logging the entire verbose output of this encode to our log file. We could send this to /dev/null if we wanted, but then we wouldn’t know if there were any errors during encode
echo "`date` -- encoding proxy complete, proxy at $PROXYPATH" >> /Library/Logs/awproxy.log
This simply logs that the encode completed. Note that it doesn’t mean it completed successfully, but that it simply exited the encode process. Looking above this line in the log will tell you if the encode was successful
echo "$PROXYPATH"
This is the single return line that PresSTORE is expecting. PresSTORE takes this output and runs a process that involves moving the file into its archive database. PresSTORE MOVES the proxy file into its own tree, so do not write a script that points this to an existing proxy file that is being used by another system (FCSvr/CatDV/etc) as it will remove it from that file tree (assuming it has rights to).
5) Enabling the script in PresSTORE
Coming back to our Preview Generation menu in our Archive Plan, we now have to add the appropriate file extensions and script location that we would like PresSTORE to use. This is fairly simple. In the bottom of this window, click “new” under the User Defined Preview Generators. Add your extensions, script path, and timeout. The timeout should be set to something reasonable given the types of clips you are converting. Test a few clips of average length on your system with a stopwatch and then set the timeout appropriately. For my environment, everything is short form, so a 10 minute timeout was acceptable. If you are using this is a sports/long form environment, you may need to set this much much higher.
6) Did it work?
Now that we have our preview generator set up, time to go archive something in PresSTORE. When archival is complete, we can see if our script is working by looking at our job monitor and double clicking the job. At the bottom of the window, it will tell us how many files were archived and how many previews were generated. These should be the same number assuming all the files fit inside your filter:
You can now log into your archive index, browse to the archived clip, and verify that you can see your proxy.
7) Advanced index setup (optional)
By default, PresSTORE stores all of its indexes in out location, “/usr/local/aw/conf/index/archive/indexname”. If you have a large amount of media, you probably don’t want this running off of the system drive as your proxy media will start eating up space. Inside this directory, the proxies are stored in their own directory tree that starts simply with “1”, so all proxies for “Default-Archive” would be stored in the following location:
/usr/local/aw/conf/index/archive/Default-Archive/1
PresSTORE fully supports symlinking the proxy location to a different storage area. Because this is a non-human-readable data tree, I like to symlink this into a .bundle file on Apple systems to make it easy to move an relink in the future if necessary. To do this, simply make a new bundle wherever you want to store your proxies and symlink back into the conf directory.
shell# mkdir -p /path/to/my/proxies/aw_proxy.bundle shell# mv /usr/local/aw/conf/index/archive/Default-Archive/1 /path/to/my/proxies/aw_proxy.bundle/ shell# ln -s /path/to/my/proxies/aw_proxy.bundle/1 /usr/local/aw/conf/index/archive/Default-Archive/1
Hopefully this is a helpful tutorial. I’m open to ideas on expansion as this is by no means all inclusive and only explores one place where PresSTORE Archive can be used. Questions/comments are welcome.