20

Jul

DVD Metabase

Posted by JeffC  Published in JBCobb.net

Update 14 Jan 2009:Of course after a year of no activity, the day I drop source on this HandBrake makes a new version which breaks the setup tool. I am fixing and testing against version 0.9.3 as I write this. Files on SourceForge will be updated as soon as I confirm things work. Update 30 minutes later: Patched and corrected files uploaded to subversion on SourceForge. Fixing tarball now. Thank you for your patience. JJC.


Description: DVD Metabase (DVD-MB) is a new public service and toolkit for extracting and cross-referencing DVD metadata on OSS platforms. DVD-MB is comprised of two important elements:

1. The back-end: DVD Metabase is a Python-based XML-RPC server. Metadata is stored in an SQLite3 database. Currently our sample database is running at jbcobb.net:8000. Here a hash of the physical DVD media is mapped to metadata ranging from track type to cast and review information.

2. The frond-end: EZRip is a Python-based wrapper around the excellent HandBrake ripper tool. EZRip is very configurable and has a number of features not found on other platforms or similar tools such as automatic audio track selection, automatic track naming, automated advanced use of HandBrake options and more. EZRip is highly configurable and can use HandBrake to generate anything from PSP and iPod-ready video to video ready for a home-theater. It can automatically find the correct language track and subtitles if requested (handy for Anime!),  apply track-type specfic rip options to HandBrake (rip features at a higher rate than supplemental or “extra” features for example). With proper shared drive mapping, EZRip can serve as an easy conduit for putting new content onto your video-on-demand system.

EZRip quick demo

A client can insert a DVD into an Internet-connected PC and have a wealth of information available:

  • Title
  • Disc ID
  • Links to other metadata sites such as IMDB
  • Links to vendors
  • Track information including title, track type and number, useful for rippers.
  • Disc and track descriptions
  • Technical data such as language and video settings
  • Links to reviews, synopsis and trivia

Usecase:  User purchases a new DVD and wishes to add it to his or her personal media center. The DVD is inserted, a unique hash is calculated and the DVD Metabase server is queried. If found, some or all of the metadata pertaining to that DVD is returned to the user. Finally the system imports all of the proper tracks (including special features, trailers, etc) with proper names and stored them locally with no intervention on the part of the user.

Usecase: DVD’s are no longer static. A user can insert a disc into an Internet-connected player and as part of the normal menu options current trailers or other related titles can be displayed

Usercase: A user rents a movie and watches it on his or her computer. As part of the retrieved metadata, a link to purchase the DVD at (for example) Amazon.com is returned and the user has an option to buy.

Now, to support some coming features:

  • Limited character, actor and director information will be associated with each item (movie or show). Combine this with the proper DB (remote or local) and it will enable:
  • At the touch of a button the user can freeze the picture, see who is in the scene, look up related titles.
  • This will answer the question we all ask of “what did I see him in…?
  • This will also allow a suggestion of other titles this performer/director was in.
  • This will allow you to see which episodes this character was in with a long-running series as well. For example with links to local files or a free remote stream, the user could automatically see all of the episodes of Stargate SG1 where Apophis died (4 or 5 at least).
  • This will also allow the back-end to help suggest related content titles that they don’t already have. For example if the user has a lot more scfi than anything else but there is no Aliens movie, it is suggested with maybe a link to Amazon or something to fulfill the order.

Status:

Currently DVD Metabase consists of a Python XMLRPC server working with a database abstraction class (DVDDB) which interacts with an SQLite3 database. The server listens on port 8000 and exports methods for querying, updating and deleting entries. However with the rather simplistic security in place on the server, non-admin people can only query and add to the database. Users can be banned or have normal or administration level access.

If however you had the inclination to see how it all works you can easily drop the DVD Metabase package on a networked machine (or local if not) and enjoy full admin functionality.

The database: As mentioned, the database on the back-end is SQLite3. It consists of a single file (dvddb.sqlite) which contains the following tables:

  1. main: disc id, disc title // main dvd record
  2. tracks: track id, parent id (same as main), track number, track title, track type. // represents a single track.
  3. links: parent (disc)id, description, url. // 1-n links to associate with that disc. Can be metadata sites such as IMDB, link to a review, to a vendor for purchase, anything.
  4. description: id, description.// textual description of the DVD, story synopsis, etc.
  5. hbprefs: id, desc, prefs // presets for optimal transcoding
  6. role: id, banned, allowed, admin // for security filtering
  7. reviews: id, review.

Demo applet:

While a client can be written in any language that supports XMLRPC, examples here will be presented in Python. To run the sample client you will need Python 2.4 and the xmlrpc libraries installed (they came installed with my Python). Included are the following files:

  1. dbquery.py: main query script. To search call with -s <searchword> to search the back-end for all DVDs with <searchword> in the title. Calling with -g will generate an .export file which is a simplified schema for the main and track records that go with a particular DVD; the full command would be ./dbquery.py -g <disc_id> -o <outputfile.export>
  2. dvdmainrec.py: main record type (wraps disc name and ID)
  3. dvdtrackdata.py: track record type (wraps track name, number, id, parent and type)
  4. ezrconfigy.py: main configuration file

A simple query:

I will use a simple query to illustrate how simple DVD Metabase is to access. Assume we want to search for Braveheart; you can search by disc ID or by searching the main title list for a partial name. We will do the latter here. The command to perform the search would be:

./dbquery.py -s brave

An XMLRPC connection is instantiated to jbcobb.net and a local DVD Metabase client is created:

server = 'http://%(rpcserver)s:%(port)s' % {'rpcserver':self.Cfg.dvddbHost, 'port':self.Cfg.dvddbPort}

self.dvdMetabase = xmlrpclib.Server(server, allow_none = True)

The command line switches are parsed and the DVDDB::grepMain() is called:

results = self.dvdMetabase.grepMain(searchKey)

The results come back as a list of stringified Python objects of type DVDMainRec which can then be iterated through and processed:

for recstr in results:
tmpMain = DVDMainRec()
rec = tmpMain.str2MainRec(recstr)
print rec.getTitle(),' ID=',rec.getID()

A complete session looks like:

jeff@jeff-laptop:~/ezrip$ ./dbquery.py -s alien
connecting to ... http://dvddb:8000
alien_vs_predator  ID= 0de8c15da9c938f4a23432d81ceec377
jeff@jeff-laptop:~/ezrip$

From here, the ID can be used to query all other tables for more information and the usecase requires.

The other primary demonstration of DBQuery is the generation of export files based on DVD data. For example:

./dbquery -g 0de8c15da9c938f4a23432d81ceec377 -o avp_metadata.export

This will query the back-end for all track information and store it in the form of an export file named avp_metadata.export:

jeff@jeff-laptop:~/ezrip$ ./dbquery.py -g 0de8c15da9c938f4a23432d81ceec377 -o avp_metadata.export
connecting to ... http://dvddb:8000
doing record search...
converting from a string...
Title found:  alien_vs_predator
Disc ID:  0de8c15da9c938f4a23432d81ceec377
1 tracks found:
[01]-m:alien_vs_predator
avp_metadata.export written.
jeff@jeff-laptop:~/ezrip$ cat ./avp_metadata.export
[main_rec]
id = 0de8c15da9c938f4a23432d81ceec377
title = alien_vs_predator

[track_01]
track = 1
type = m
id = 0de8c15da9c938f4a23432d81ceec377-1
parent = 0de8c15da9c938f4a23432d81ceec377
title = alien_vs_predator

jeff@jeff-laptop:~/ezrip$

The generated file can be edited and submitted (with proper access) to make corrections in the database.  Moreover, this also demonstrates how to programmatically access secondary tables. The code to access the main record and then the track records by ID is simple:

mainStr = self.dvdDB.findMainRecAsStr(disc_id)

The main record is returned in stringified form which is then converted to an actual DVDMainRec object:

dvdRec = tmpMain.str2MainRec(mainStr)
if dvdRec.isNull() <> 1:
print 'Title found: ',dvdRec.getTitle()
print 'Disc ID: ', dvdRec.getID()

BTW you will have to forgive the formatting; Wordpress is having a field day with it. In any event, the disc ID is then used to query the tracks table:

foundTracksStr = self.dvdDB.getTrackListAsStr(disc_id)
foundTracks = []
for str in foundTracksStr:
foundTracks.append(tmpTrack.str2Track(str))

At the end of this you will have a list of DVDTrackData objects, one for each track on the disc.

Now if you are running a Linux system (specifically with libdvdread3 installed along with EZRip) you have some added functionality. By inserting a DVD and running ./chkdisc.py it will calculate the disc_id and query the back-end. If found the information is displayed.

Functionality for accessing the additional secondary tables is currently not enabled. I expect to publish this  sometime this week.

Documentation is coming soon. The source drop as of this writing and what is currently running the back-end is being made available via SourceForge. This page will be updated when its ready.

Well, here is the SourceForge page! Let’s see what is possible…

Peace,

JeffC