In any event we finally got around to getting some backup for our systems but up until now we had not found good method of backing up 6 terabytes of data. Well as luck would have it Seagate just put out a new FreeAgent external 3 TB HD for 250 dollars US. We looked a little more, almost bought a pair of them figuring 500 was a cheap date for backing up all of our stuff when Beth found Seagate 2TB models for $139 plus shipping. This meant we were able to get 6TB of backup for less than 420 bones! Thus while repopulating the NAS we have been able to back up the other two and once the movie NAS is done (again) we can back that up too.

This means however that while the process is a painful one, the database will have probably 20x the content that it has now. I look at this as a good thing. To show it off I mean to put up some kind of website demonstrating the metadata and the many uses to which it can be put.

So while this is going on, no work has been done on libfoundation for a while. I finished a prototype for the virtual memory core, XML configuration read/write libs, threading tools (thread class wrapper, read-write locks, scoped mutex classes), shared memory tools and a really good start on the IPC mechanism. Once this is done I will add in the component loading system and call that libfoundation 0.5 and publish that. Each subsystem will come with a demo app showing how to use it.

Until then, I am working for Bally from ~06:45 until 16:30 then coming home and working on this for usually another 4 hours a day, 12 plus hours on the weekends. The thing is, running that many machines doing transcoding generates a lot of heat and being in the desert, it is even now (21:00) 106 degrees F outside so you can imagine how hot it gets inside, even with AC.

I will post more when done….

Me to you..

JeffC

As some or many of you know I have a lot of concurrently-running projects between DVD Metabase, Bitmimic, the parallel processing initiative and more. All of these projects use forms of IPC, multithreading, logging, configuration and so on. All of this stuff is already available at the low-level but when re-implementing these things errors creep in but plus when coding it I get the case of Deja-Moo (I have coded this BS before). However when I am doing it I tend to be coding for a given employer and they tend to put their stamp of ownership on it.

Now however my employer has tasked me with no such endeavor plus as I am at the beginning of several larger projects, I find myself saying “Oh look: I will be needing an IPC system again or a template-less container or a mutex controller or….”.

Hence, the creation of libfoundation.  I am going to recode all of this stuff one last time and publish it under a business/hacker friendly license so no one can take it away. More, it will be loosely-coupled to the point that I or anyone can use a subset of the functionality without recoding or sacrificing functionality.  In a strange twist though, if you need an IPC system that talks to a plugin loader which uses an XML configuration scheme (for example), this will work too.

This time however I intend to be a bit more proactive on things. My intention is to take what I have learned over the years in terms of best-practices and implement all of the tools that I seem to need over and over again…one…last…time. They will be published under the bsd or some business-friendly license so I can use them when and where I need to and not constantly feel like I am reinventing the wheel. More to the point though; while this is a single overall library, it is really a loosely-couple collection of sublibraries that I have found useful over the years. Another aspect of this code is that every single API call will be exercised in a demo app that will serve multiple purposes:

1. Aside from any printed documentation, the demo code will show how to use the API to accomplish a variety of purposes.

2. Because each demo app tests (or will) every API call, it is a simple matter to just run Valgrind against that in order to confirm that any given component isn’t causing memory-leaks, corruption, etc. Since this is a foundational layer, I expect stuff to be built upon it (well that is what *I* am going to do anyways) and since most of the stuff I have in mind is non-trivial to begin with, having a justified sense of confidence in your foundation is critical.

For much of what I intend to cover, Boost and other libs cover this as well but at least in the case of Boost not all of the low-level libraries are of the same and consistent maturity level.  Moreover, while each sublibrary can be used independently, they are as a rule designed to work together, hence the API for the XML configuration lib has the same feel as the data library or the socket library.

It is my intention to grow this over time as needs change. How best to describe this would be to say that if you are needing something upon which to build a multithreaded (but not necessarily), dynamic (but not necessarily), embedded (but not necessarily) distributed (but not necessarily) system, this will work. It uses as much or as little C++ as you want/need. For example, I have a variant (an “any” class) and a return code class (enforces the checking of return codes) that either uses templates and RTTI (thinner classes)  or I have the same thing as almost C-like versions, no templates/RTTI, resulting in heavier classes but fewer system constraints. Most systems always support all this but some embedded systems don’t.

Most of the classes/tools are like this. In version 1.0 (probably due soon the basic library will contain:

* Thread class

* Scoped mutex, scoped try mutex classes

* Wait condition classes

* XML configuration classes (translates an XML file to a C++ class)

* IPC class framework (easy server and client messaging)

* Template-less data containers

* Variant class (same class can be anything from a string to an int to a data block)

* Return code class (if you return this and the caller doesn’t check the return code it asserts)

* Reader/writer locks

* Dynamic component loading system.

* File or memory-based error logging system.

* Quick and reliable TCP/UDP socket server and client system

* Virtual memory system

* MD5 hashing system.

This is not all inclusive but will give you an idea of the kinds of tools that will be available.  The first cut of this will be Linux-only, built with autotools.  I expect to get to an alpha shortly which will probably not have everything in the first release but I gotta start somewhere…..

More to come.  Me to you…

JeffC

CMakeWizard 2.3

To install:

Untar this file in your home directory. It will create a folder called .cmwiz and put all of the template files and installation code into it. Look at the readme.txt file for final installation details (setting up symlinks, adding GTK support, etc).

When done creating a new project for hacking should be as simple as:

1. Create a folder in your development directory.

2. Change to that folder.

3. Run cmwiz

4. Answer the questions; if you are in a hurry the drunken-chicken path can be followed, resulting in a basic C++ project with getopt support, a main executable and a single shared library. The drunken-chicken path can be best described as simply hitting Enter over and over again to all queries (except for naming the project; this is kind of critical as many of the file names are derived from this) will result in a working project.

5. After the interview stage the CMake project is generated along with a C++ framework which is then compiled and linked, resulting in a working project in just a few seconds.

Enjoy and as usual, if you have any questions you can write to me at JeffC@JBCobb.net. This is still released under the GPL 3.0.

Me to you.

JeffC

1. Microlog 1.1a: It was pointed out to me that for all the tricks that it did, it was missing the basics of proritizing log message a la Info, Warning, Error. This has been fixed and is in the new code dump here:
Microlog 1.1a

Plus some forward-looking statements…

When you start a sabbatical like I have you think you have the eternity of time to get done what you want. Well now it seems that it is time for the sabbatical to end and for me to rejoin the ranks of the employed.. What this means is that my time will no longer be mine to use as I please. While I cannot put an exact time-frame on things, I am expecting things to be occupied within the next 45 days. This means I need to wrap up several projects PDQ. As for the main projects, here is where they are, were I think I can get in the time I have left and the abilities is each:

1 The BIG project. I have the micrologger done and 45% of the IPC system done. In some ways I wanted the IPC to be stand-alone for use is in other pojects but ran into the basic problem of: An IPC system needs someone to talk to and the module loaders need some way to communicate or they are not going to work well. So. While I was going to split the distributed project into IPC and module loading as separate projects, now I find that I must roll then into on. Consequently the next release of this will be the module loader along with the IPC system so the modules can intercommunicate. In other words the next release of this will contain both. An initial set of modules will be released to do the analysis outlined here , The Problem. Then as time goes forward I can just swap out the modules to make the string search smarter and I already have some ideas for that.  Once this is done. it will be a minor task enlisting armies of machines to do the analysis work and come up with better solutions faster, the point of the game. I have the IPC stuff coded and the module loading going in a separate project; need to meld the two and this part should be useful…

2. DVD Metabase: I am going to push out a new version to cover version 0.94 of Handbrake though I will be supporting version 9.3 (AVI + Xvid) for now. This will happen RSN. To be honest, the reliance on x264 and MP4 has has positive and some very negative impacts on the project.. Therefore the use of 94 will be there for those who want it and I will maintain a code-drop of 93 on my webserver for those who don’t.

3. The C++ version of EZRip and the plugins….it is in progress, has some ups and downs and if I don’t get some help on this could take a while b/c I am getting some rather dodgy results from using libdvdread3. PM me offline for specifics.

4. I will probably be making some upgrades to CMakeWizard shortly since GCC 4.4 has produced some irregularities…I know what the fix is but not sure how to gracefully make it work with all versions….meh.

All of this will be done in the next 30 days. No pressure….

Me to you

JeffC

EDIT: I must have been seriously tired when I first posted this entry due to the mistakes involved…sorry folks, all should be well now…

J.

Often the application of #1 results in #2 meaning the “bug” can go away or change in nature to the point where the original bug is no longer obvious. You can assume that you have the bug fixed yet as soon as you remove/disable the debug code and the bug can return or worse, uncover a completely new one.

Part of the reason for this is that writing to a log, socket or other place to store the log will cause the kernel to execute a context switch. This means that different threads will execute in a different order under debug conditions…this is what I call “the observer effect”: a watched program executes differently than one running in release mode. This is even more noticeable when running it under GDB or Valgrind for example.  There are two ways to deal with this:

1. Force the context switches to happen the same way under both conditions.
2. Reduce/eliminate the context switches incurred by the logging process itself.

While #1 is interesting and the subject of another project I have on the back-burner, #2 is the subject of this article. One way to achieve what we want is to defer all file IO until such a time that it no longer matters. This means storing the log-contents in memory which brings up another problem: if you have multiple threads of execution all attempting to write to the same object/data store, you have to gate access via synchronization tools such as the mutex or semaphore and that means blocking some threads while others work and that still brings about the dreaded artificially-induced context switches. One way around this is to give each thread its own area of memory to write to. However since 90% of logging messages are text this could quickly become a resource-hungry monster. This would be bad enough in a normal system but on an embedded system this would make such a method all but unusable.

One way around this is by tokenizing the log messages. If you look at most of your logs you will find most if not all of the messages follow a similar pattern, often with only a few bits of data being different.  A few examples of typical log messages reveal:

[11:50:34] Test Process 1: Starting test begin
[11:50:34] Test Process 1: Locking resource mainQueue
[11:50:34] Test Process 1: Unlocking resource mainQueue
[11:50:34] Test Process 1: Sleeping for 1000
[11:50:34] Test Process 1: HWEvent Mouse event
[11:50:34] Test Process 1: Info Finishing task
[11:50:34] Test Process 1: Warning low resources

If we look more closely at this we find four basic kinds of information:

1. Time stamp

2. Originator of the message

3. General text of the message

4. The message “payload” or specific data being conveyed.

In the entry: “[11:50:34] Test Process 1: Locking resource mainQueue”

1. The time stamp is obvious, in brackets

2. “Test Process 1″ is the process originating the message

3. “Locking resource X” is the generalized text of the message and

4. “mainQueue” is the message payload

Total message consumption: ~56 bytes.

With the goals of reducing the chance of triggering a context switch and reducing the memory footprint of the message in mind I created a message token like so:

typedef struct TBasicMsg
{
        time_t msgTime;// time of message
        mType msgType;// enumeration of message type
        payloadType plType;// enumeration of payload type
        basePayload basePL;// 16-byte payload
}basicMsg;

The time_t time stamp is obvious. The message type is an enumeration of basic core messages:

typedef enum {
    msgNone = 0,
    msgStart,
    msgStop,
    msgLockResource,
    msgUnlockResource,
    msgSleep,
    msgWake,
    msgWait,
    msgResume,

… etc. Each of these corresponds to a full textual message or more accurately a message format string suitable for sprintf(), stored in an STL map in the server. For example, msgSleep may correspond to the string “Sleeping for %d seconds”.

The payload is simply up to 16 bytes of data. It could be four integers, 16 characters, 15 characters + a NULL, two doubles, etc. Exactly how that data block is interpreted is dictated by the payload type field, enumerated as so:

typedef enum {plEmbeddedChar = 0, plLUChar, plDouble1, plDouble2, plInt1, plInt2, plInt3, plInt4, plLong1, plLong2, plNone} payloadType;

Each of these corresponds to a union to easily translate the 16 byte block defined as:

const short payloadSize = 16;
typedef struct TPayload
{
    char base[payloadSize];
}basePayload;

The basePayload is then translated into a specific set of data through the use of a union. For example, plInt reflects:

union intPayload
{
    basePayload core;
    int n1, n2, n3, n4;
};

There are unions to translate this into various combinations of values such as doubles, integers, longs, short character arrays and so on.

Using this combination of elements, all messages can be contained in a single 28-byte block. The call to store such a message as “Sleeping for 10 seconds” would look like this at the client level:

Log.logMsg(msgSleep, plInt1, Log.makePayload(1000));

This call sends the message token about sleeping along with a payload type meaning a single integer.  The makePayload() overloaded function converts the single integer into a basePayload type and returns it. The whole thing is then added to a BasicMsg type along with the current timestamp and pushed into a memory slot provided by the server.

The Bigger Picture:

Overview of client/server relationship

The idea is that there is a single microlog “server” created and 1-n clients.  The server is never instantiated directly but rather by the first instance of the first client in the constructor:

CMicroLog::CMicroLog(string strProcName, long lThreadID, string &strLogName)
{
    // if server is not there, create
    if(ptrServer == NULL)
    {
        ptrServer = new CMLogServer(strLogName);
        nClients = 1;
    }
    else
    {
        nClients++;
    }

ptrServer and nClients are both static members of the CMicroLog class. The first client instance created will build the server instance and the last one destroyed will delete the server:

CMicroLog::~CMicroLog()
{
    // this should be wrapped in a lock
    nClients--;
    if( nClients == 0 )
    {
         cout << "Deleting server..." << endl;
         delete ptrServer;
    }
}

When the destructor for the server is called, the stored byte-ranges for all client-generated messages are translated into textual equivalents and written to the filename strLogName.  Thus it makes a certain kind of sense to have a “control” client created so the log is not written before you are truly ready to exit.

Upon instantiation, after the server is created or the client count is updated, the client registers itself with the server through the registerLog() method:

nBufSize = ptrServer->registerLog(strProcName, lThreadID, &ptrMainBufStart);

strProcName is the textual name of this process (ie “Database manager” or “Socket service”), lThreadID is a long that can store the actual ID or handle for that thread/process or an arbitrary number more meaningful to the debug process. Finally &ptrMainBufStart is simply a reference to a pointer to a memory block managed by the server. On making this call the server sets aside a block of memory for direct use by this client that is the size of the basic message times MSG_BUF_SLOTS, currently 1024. This essentially sets up a circular queue managed by the server associated with this client. Each call by the client to logMsg() pushes the 28-byte block in the slot pointed to by ptrMainBuf (initialized to ptrMainBufStart, established by the server) and updates the value of ptrMainBuf. When ptrMainBuf reaches the end of the block it is reset to the beginning held by ptrMainBufStart. All pointer manipulation and memory writes are performed by the client. Since the client “owns” this zone, no synchronization is needed.

Finally when the last client is destroyed the server calls flushLog() which iterates through all allocated memory blocks, all valid (non-zero) messages are entered into an STL multimap, keyed on the time-stamp. This has the net effect of aggregating all messages from all registered clients in time-stamp order into a single list. For each message in the list, the payload is converted to a string and inserted into the message.  That list is then translated into textual messages and written to the log file.

The Code:

The link at the bottom of the article has the source code for the library (CMicroLog (client), CMlogServer (server) and mlog_main.cpp (test/demo application). Please note that in its current version it is very much alpha code and will probably see a lot of refinement and extension in the area of message variety and construction.  Consider this to be a proof of concept as much as anything. You will need the cross-platform make system CMake installed to build everything (apt-get install cmake).

Two notes on extending the system (and it is *meant* to be extended!):

1. If you want to extend the core message library, add the new message to the msgType enumeration in mlogmsgs.hpp and then add a string insertion into the msfDigest in mlog.cpp:CMlogserver::initMsgDigest().

2. There is an experimental extension type called auxMsgTypes in auxmsgs.hpp.  It is triggered by specifying msgAuxMsg in the logMsg() call, plLUChar in the payload type and then the desired auxMsgType in the overloaded CMicroLog::makePayload() call.

The test app can be run in two different ways:

1. mlog_test -t basic will run a single-threaded test to demonstrate core message building. All messages are written to singletest.log.

2. mlog_test -t multiple -n <number of threads> to test a multithreaded model. This will spin up <number of threads> threads, each of which will create a CMicroLog client and go through a series of sleep and wake periods at random intervals which in turn emits a number of log messages to the server. Once all of the threads are done the server will collate the messages into a single log and write it to multitest.log.

There are a lot of logging systems that are more complete and easier to use. The value this brings to the table is if you need to constrain the number of artificially triggered context switches and/or constrain the amount of memory consumed by the logging process.  If you want smaller circular queues, adjust the value of MSG_BUF_SLOTS in mlog.hpp. If you need to add more/different messages to the system or add custom payloads, edit mlogmsgs.hpp and mlog.cpp.

The main logging library is contained in libmlog.so.  The main logging code is in mlog.cpp/hpp. The message enums and unions are stored in mlogmshs.hpp and auxmsgs.hpp. The entire codebase is released under the GPL 3.0. Sorce Code is here:

microlog-1.0.a.tar.gz

Experiment and share your findings…

This story is about how easy it is to do all of this using free tools and then to make it work under both Linux and Windows….only this is not about Linux vs Windows. It is more of a case of Microsoft Internet Explorer vs the rest of the world….

At first the general project sounded simple; dress in a suit, iterate the highlights of my resume on video, edit it to fit into a nice bite-sized format, convert it flash to save on my bandwidth and host it on my web site. Doing it all with free tools made things even nicer.

The Video:

In anticipation of needing to do some editing I broke the general ’script’ apart into four or five sections: An introduction, skills summary, early background, later background, current projects and closing. Then I went into my office with a shovel and cleaned out an area to film in. I used the so-handy RCA Small Wonder video camera to film with. These things are great; even without additional RAM, it can record up to 30 minutes of high-quality video or an hour at low-quality. When you get done filming you press a button on the side of the device and a USB port pops out; plug it into your Linux laptop and it appears as a camera or external drive and the files are already in .AVI containers. I went through all of the section four times so that I could cherry-pick the best ones. As you will see, I may be an excellent coder but I am nobodies movie star.

Processing:

The next step was to break apart the various video files and assemble the best parts into one continuous film. That task was relegated to my favourite video editor for idiots (me being an idiot at such things), AVIDemux (apt-get install avidemux). With this tool it was childs play to break them apart, preview the various elements and then “stack” them together into one video:

Once I chose the parts I wanted it was a simple matter of stacking them with successive calls to Append File (File -> Append). When done, I simply saved the result to a file called interview1.avi. This was fine except for two things that should be apparent: 1. I am unemployed so funds are limited and 2. This file was 89 megs in length! I could see my bandwidth bills going through the roof, defeating the purpose of the exercise. That answer? The oft-despised but comparatively anorexic Flash medium but I had no experience at creating Flash videos! Oh what is an OSS dude to do? FFMpeg to the rescue but I was on the clock, having put it off all weekend and needed it for the following morning. A nice front-end for FFMpeg (and my new best friend) is WinFF (apt-get install winff):

WinFF is great; you point at the file you are starting with, choose from dozens of types of outputs and press Convert. It’s just that simple. I love it. Out pops interview2.flv at a slender 15 megs.

The Fun: Flash

At this point I somewhat stupidly assumed I was in the home stretch; the only thing that was to stretch was my ability to curse at inanimate objects and insanely rich billionaires. All I had to do was to embed the Flash video into my website which contained a page with my experience on it anyways and just sticking the file out there with an HREF anchor did the trick but it just started up and played with no way to stop or to pause the video…and made it look even more lame than it was. A little research revealed that you had to embed a Flash player on your website (or somewhere) to provide the VCR-like controls we are all accustomed to…and most were for “non-commercial” use only. Well job-hunting left me in a strange spot in that it would be non-commercial … until and unless it worked. Consquently I went out in a search for an open source Flash player and sure enough, the OSS world provides OSFLV.

Creating the page:

First to set up the web page; I run WordPress on a spare box for my main site. Great product, BTW. In any event, OS FLV needed me to muck with elements of the HTML normally out of the reach of the common user so I just created a subdirectory under the main site called ‘interview’ and from the OS FLV tarball I extracted the flash folder along with AC_RunActiveContent.js along with player.swf and put them all in the interview folder. To that I added the interview2.flv file, a snapshot from it called sample.png (AVIDemux provided this) which gives the Flash player an image to display when the video is not running. All that was left was the HTML to play it. Since I had it in a directory of it’s own, I fired up Bluefish HTML editor and created a default index.html file using the Quick Start wizard (I was definitely into the “why work when you don’t have to” mode at this point) and this is where the fun began. To get Flash to embed nicely on everything (regardless of platform) except Microsoft Internet Explorer, you need the embed tag:

<EMBED href="http://www/jbcobb.net/interview/interview2.flv"
quality=high bgcolor=#FFFFFF WIDTH="550" HEIGHT="400"  NAME="interview2" ALIGN=""
TYPE="application/x-shockwave-flash" PLUGINSPAGE="http://www.macromedia.com/gogetflashplayer">
</EMBED>

And to get it to work in IE you have to wrap all of *that* in an object tag from hell:

<object> classid="clsid:D27CDB6E-AE6D-11cf-96B8-444553540000" type="application/x-shockwave-flash"
codebase="http://download.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=9,0,16,0"
width='480' height='360' id='flvPlayer'>
  <param name='allowFullScreen' value='true'>
  <param name='movie' value="player.swf?movie=interview2.flv&bgcolor=0x051615&fgcolor=0x13ABEC&
volume=90&autoload=on&autorewind=on&clickurl=interview2.flv&clicktarget=_self&
postimage=sample.png">
 </object>

You can view the semi-finished product here:

http://www/jbcobb.net/interview

At the end of the day this was pretty simple to do once you deal with the browser insanity. I did a lot of browser work years ago; I would have thought that Microsoft would have learned by now….the thing is about any form of HTML/javascript/etc worked in Linux and only one in IE….there was a time when it was the other way around…

Update 11 November 2009:

I added another similar section here: jbcobb.net/14_things/. Basically I thought the interview itself was pretty dry and I wanted an easy way to quickly outline some of the more interesting projects I have worked on or invented. Again using free tools I used Open Office.org Presentation tool to create a slideshow of the material, ran the slideshow under recordmydesktop (apt-get install grecordmydesktop), converted the file to .AVI with WinFF (see above), edited the cruft out of it, overlaid some background unobtrusive background music and saved. Then I used WinFF again to convert the video to Flash, I cloned the ‘interview’ directory on my webserver, replaced the video payload with the new ‘14_things.flv’ and updated the index.html file there to reflect that. Badda-bing we have a new video…

Me to you

JeffC

* A new DVDMB C++ API

* To plug-in or not?

* To Mono or not?

* To Java or not?

Greetings all, JeffC here. While I am here in Las Vegas looking for work I have started to take EZRip/DVD Metabase into a few new directions. First, while there will *always* be a Python version of EZRip (and probably DVD Metabase), I have never made a secret of the fact that I have used the Python as a way of quick prototyping new project ideas.  Now after 5 years of development I have observed that for the most part the feature-set of the system has settled down. Also, while I see it as a useful rip tool in and of itself, I can also see uses as a component of a larger/different system, both on the rip side and the playback side, not to mention that holy grail of Netflix-like services, suggestions.

A C++ API…

One way to make this useful in other places is to wrap the core functionality in a plug-in-esque environment. As it is EZRip uses the following key services:

* Volname (collects the disc label, comes with Linux)

* XML-RPC (Web services architecture for performing transactions with the DVD Metabase server: Python)

* DVD scanning (collecting disc track, language and subtitle information, provided by lsdvd)

* DVD Disc hashing (calculating an MD5 hash of the disc contents for disc lookups, provided by disc_id from the libdvdread source)

To this end, I have started rewriting all of EZRip as a C++ project. With the exception of the rip engine itself (Handbrake in this case but more on that in a moment) all services will either use stock Linux libraries (libdvdread, libxmlrpc) or other open source libraries (SimpleINI). I am creating an interesting API that should allow other programmers to use these services however they wish.

So far I have readers/writers for the configuration file (which is now an INI-style affair), profile and filter files. On the XML-RPC side I have readers for doing a DVD lookup (supply a hash ID and get a main record plus a deque of 1-n track objects). On the DVD scanning side, I am still wrapping volname calls but intend to use libdvdread for that, a simple call that returns an MD5 hash based on a device path and am in the middle of sorting out the direct calls needed by libdvdread to perform the scanning job. So far I have that last bit collecting track, time duration and available audio track information; after fixing a few bugs and gathering the subtitle information, you should be able to pass in a device path and get back all of the information you need to do track processing.

All of this will be exposed via a well-documented API, allowing other developers to do disc lookups, retrieve DVD metadata, read and use filters and profiles and more, all by simply including a single shared library and a header file; no more screen scraping (woowoo)! As a proof of completeness of this, I will  be rewriting EZRip to use this API and make for a faster/more extensible rip tool. As a part of this I will be implementing the concept of a modular rip engine system so that you can easily apply the metadata lookup abilities to not only HandBrake but OGMRip and Mencoder as well. Currently the make system for it will be the ever-trusty cross-platform maker, CMake. I have a lot of years of experience with this and feel it to be a safe choice for this project.

The first version of this will be pretty much a work-alike of the existing system (with the exception of a different format for the configuration files, natch) but then things are going to get interesting. First, for the sake of my long-suffering wife I am going to create a GTK GUI-based version of this tool. While it won’t be anything terribly fancy, it should remove most of the rough-edges that the command line tool presented (like the interrogation in interactive mode).

The next step however may cause some consternation and indeed it has in my own mind as well. One of the larger debates (flame-wars) happening in the open-source community right now is over the use of Mono which is an open-source impelentation of the .NET framework and tools invented by…..Microsoft. One thing I have never made a secret of is the fact that I do in my professional work a lot of cross-platform development and so whenever I approach any non-trivial project, I do with with an eye towards that end. On the system side the only remaining non-neutral component is libdvdread and that will by then be sufficiently abstracted. Handbrake is already cross-platform. That pretty much leaves the presentation side of things which is one of the places where solid cross-platform development strategies totally live or totally die. Which brings us back to Mono.

To Mono or not to Mono?

As someone who has lived through Microsoft’s early and shamelessly anti-competitive years, I cannot say I would ever trust them.  I can see any number of ways they can screw over the open source community with .NET/Mono and not for a second do I think they would hesitate if given the chance. However I also have to say I am a pragmatist to an extent and since I started doing Windows coding back in the early 90’s and had to wrestle with the SDK of the time, I really appreciated the power and architecture of Delphi (and later Borland Builder). It just seems like GUI and component-based development done right. And guess who is behind the design of .NET at Microsoft? You guessed it: Anders Hejlsberg, creator of Delphi. This suddenly puts things in a whole new light. Frankly if all it presented was another way of doing cross-platform development, I would say the risk of working with Microsoft technologies was not worth the gain since there are several other (safer) ways of doing the same thing (wxWidgets, GTK, Java/SWING, etc). However if you have ever had to develop with Delphi/Borland Builder you would know this is absolutely NOT just another cross-platform solution.

It is with this mindset that I am even remotely consideging using Mono for the next evolution of this tool.  Yes I KNOW the Mono side will always be years behind the .NET side (regardless of what Miguel De Icaza would have you believe but then no developer with his or her salt believes anything he says anymore. Once he was a great developer but now he is a salesman, pure and simple. I don’t trust car salesmen either) and will never be feature-comparable. That said I am going to take a serious, non-biased look at the platform and if it seems safe enough and feature complete so I know it won’t be a total waste of my time, I will probably make a Mono version of the toolset and a GUI front-end for it. This however brings me to one last point on this….

To plugin or not?

While I do have a few patents to my name, I still don’t like what it does to otherwise sane people. Lately the company Eolas has brought suit against the world + dog (as seen here ) for creating things with plugins or using a plug-in like architecture…people, *please*! This model of application design is not only the most flexible and sane way of applicaition development, it is OBVIOUS! This application (EZRip) will be no different but if Eolas gain any traction with their ridiculous claims, this may have to be revisited.

Another direction I might take this (mostly to scratch an intellectual itch) is to write a Java-based version of this. I see it as a way of taking a design I know well and using a variety of technologies to bring it to life. I figure in this way I can honestly determine if one technology or the other is worth spending my time on. You as a C++, .NET/Mono or Java fan what is the best tool for the job and their answer is predictable. This will be my way of walking a mile in someone else’s shoes…

Status:

Currently I have proof of concept code working with configuration, profile and filter files.

I have volname wrapped (popen) but intend to use libdvdread to get the volume label directly.

I have the DVD hashing code done as a library.

I have most of the DVD scanning code done but this needs lots of testing.

I have XML-RPC code done to interact with the back-end server @ jbcobb.net:8000. I still need to flesh out the code to do a disc submission and update existing records to complete this section.

Unless something comes up, I am expecting to have the C++ command-line version of this done within two weeks. I will publish all code at that time and under the GPL.

Until later then, me to you…

JeffC

Howdy. It has been a while since I have updated this handy development tool. In case you don’t know, CMakeWizard is a simple set of Python scripts and C++ templates for jump-starting development projects. In short, you can start with an empty folder, run the CMakeWizard script and in seconds have a complete if skeletal C++ project up and running. It can create a shared or static library and an optional executable to test it all. If you want POSIX thread support you can have that too, along with optional sample code so that you can be up and running in no time. Finally, after generating the proper CMake-based project, it creates the C++ source and builds everything to prove that it works.

Now it is even better; it has built-in GTK support…

Recently I had a need to get into GTK+ development and since I did not wish to spend all of my time fooling with the makefiles and such to support it, I decided to see if my CMakeWizard was extensible enough to support GTK. As it turned out, it was. When I initially wrote this wizard some time ago I intended for it to have this kind of extensibility and I am pleased to say that things worked out as I hoped.

New Functionality:

One thing that made GTK a little more difficult to work with was the dependency on pkg-config to find the proper libgtk2.0 headers and libraries. In order to build a typical GTK project you would need to do something like this:

gcc myfile.c -o mybin `pkg-config –cflags gtk+-2.0 –libs gtk+-2.0 `

As it turned out, in the development branch of CMake there was an extension that did the same thing called FindGTK2.cmake. In the default distribution there is a FindGTK.cmake but that is only geared towards finding and dealing with GTK 1.2, an outdated version of the library. You can use this new extension by simply copying the provided FindGTK2.cmake to your CMake Modules folder. I have included the necessary file in the CMakeWizard tarball. It allows one to create CMake project files like so:

FIND_PACKAGE(GTK2 2.6 REQUIRED gtk)
IF(GTK2_FOUND)
ADD_DEFINITIONS( -Wall -ggdb )
INCLUDE_DIRECTORIES(${GTK2_INCLUDE_DIRS})
ADD_LIBRARY( foo SHARED foo.cpp )
ADD_EXECUTABLE(foo_test foo_main.cpp)
TARGET_LINK_LIBRARIES(foo_test foo ${GTK2_LIBRARIES} )
ENDIF(GTK2_FOUND)

This will lookup the right locations of the GTK libraries and headers and provide them via the variables GTK2_INCLUDE_DIRS and GTK2_LIBRARIES.

I have also extended the templating system to optionally generate a working minimal GTK GUI app, the venerable Hello World.

To use this just unpack the contents of the tarball to your home folder; it should create/overwrite a hidden folder called .cmwiz. If this is your initial installation, please read the readme file for finishing the installation.

Once installed the steps to go from absolute nothing to a working GTK GUI app are:

1. Create a new folder and cd into it:

mkdir gtktest && cd gtktest

2. Run the CMakeWizard:

cmwiz

3. When asked if you want to generate a GTK project, say yes and accept all defaults. That is it. When it finishes running you will have a working GTK app that you can now extend to meet your needs. The entire session looks like this:

jeff@jeff-gate:~/dev/gtktest$ cmwiz
CMakeWizard 2.2
A CMake Project jumpstart script
This script is published under the GPL.
CMakeWizard will first ask you a few simple questions
and generate a CMake-based project with source in
the current directory. Finally the project will be built.

Answer the following questions or Ctrl-C to abort:
Project name: gtktest
Use GTK 2.0 (n): y
NOTE: This depends on FindGTK2.cmake being in your modules folder to work
Gen sample GTK code? (y):
Should this build a library (y):
Generate a shared library (y):
Library source name:  (gtktest.cpp):
Additional include directories:
Use pthreads?
Should this create a test harness (y):
Test harness source (gtktest_main.cpp):
CMakeLists.txt written
– The C compiler identification is GNU
– The CXX compiler identification is GNU
– Check for working C compiler: /usr/bin/gcc
– Check for working C compiler: /usr/bin/gcc — works
– Detecting C compiler ABI info
– Detecting C compiler ABI info - done
– Check for working CXX compiler: /usr/bin/c++
– Check for working CXX compiler: /usr/bin/c++ — works
– Detecting CXX compiler ABI info
– Detecting CXX compiler ABI info - done
– Found GTK2_GTK: /usr/lib/libgtk-x11-2.0.so
– Configuring done
– Generating done
– Build files have been written to: /home/jeff/dev/gtktest
Scanning dependencies of target gtktest
[ 50%] Building CXX object CMakeFiles/gtktest.dir/gtktest.cpp.o
Linking CXX shared library libgtktest.so
[ 50%] Built target gtktest
Scanning dependencies of target gtktest_test
[100%] Building CXX object CMakeFiles/gtktest_test.dir/gtktest_main.cpp.o
Linking CXX executable gtktest_test
[100%] Built target gtktest_test
jeff@jeff-gate:~/dev/gtktest$

Running the generated binary (gtktest_test) yields the following:

GTK Applet generated by CMakeWizard

Future directions:

As I learn more about Glade and how to automate that process, I will extend/modify this to use a more generic resource model. As-is, the GUI is pretty hard-coded, more a proof that it works than useful in an of itself. Obviously for any of this new functionality to work you need the proper GTK development libraries installed to begin with….

Enjoy and as usual, any feedback is appreciated.

1. There is nothing totally original in nature or created by man.

2. Anything that *is* considered unique could be decomposed until the uniqueness was gone and essentially all elements of the unique creation could be found in another ‘unique’ creation.

3. In short, any supposedly unique creation was simply the re-assembling of other non-unique elements.

To prove this, I would take two supposedly unique items, lets call them target and source. The source would be some bit of digital work that is unique yet not copyrighted, hence in the public domain. In this case, I am using a JPG of my house:

Not much to look at but will serve as our source. The target is a file that is supposedly completely different than the source. In this case, I am using a picture of my Epi Riviera guitar:

The idea then is that even thought the two look nothing like each other, decompose the target picture into smaller and smaller byte strings and find those strings in the source. Once found, the offset and length of the byte string would be recorded to what I am calling a recipe file. The proof of success is then to take the source and the recipe file and completely (and faithfully) reconstruct the target file from scratch.

The application provided here, what I am calling Bitmimic does exactly that. It does a brute-force decomposition of the target file and finds the byte strings that match in the source file, recording the results in the recipe file. Note that it is intentional that no attempts at optimization of either the recipe file or the searching. The project linked at the end of the file will contain all of the source along with the two JPGs for testing. Feel free to test this with your own digital media. The JPG test will execute relatively quickly (10m for the deconstruction/recipe generation and seconds for the reconstruction); this was done to give you a feel that the system works. However to give the larger concurrent project something real to chew on, the Epi picture will take the place of the source and the target will be a 30 minute video, something like 190M in size. On a relatively recent machine it took 2 days to deconstruct an episode of a 30 minute show and something like 10m to reconstruct it.

About the software:

Bitmimic is written in basic C++ and constructed with CMake which should allow it to build under Windows, Mac and Linux. I do use getopt to grab the command line args but that should be the only platform-specific bit to the system.  The source and the target and memmapped and so are treated like memory blocks. As mentioned, there has intentionally been no effort put into optmimization of any kind. Since this problem was designed to serve as a test subject for a larger system for problem-solving I needed a worst-case baseline to use as a foundation. The idea is to use the larger problem-solving mechanism to split the work up between processes and even machines on a network to resolve the recipe file in record time.

To make the system just:

cmake .

and then

make

To use the program to break up the Epi picture using the house picture you would:

jeff@jeffhpdev:~/dev/bitmimic$ ./bitmimic -a -r epirecipe.dat -d ./house.JPG -s ./epismall1.jpg

Parameters were:

-a = analyze (as opposed to build)

-r = name of recipe file to generate

-d = data file file to use

-s = source file to analyze

Two other command line args I could have used were:

-l = Open this text file and read each file listed therein. The system would spot check each data file listed and select the one with the longest byte-strings from the source. This is good if you want to check a list of JPGs (in this case) to the find ones that have the longest naturally occurring matches. The result is a smaller recipe file.

-D = root directory of files listed in the file with the -l option. For example, all my jpg candidates were in a directory called home/jeff/data; the library file referenced with the -l option was simply the output of ls *.jpg ../library.dat. The whole line would then be:

./bitmimic -a -d <path to library file> -l -D /home/jeff/data/ -r recipe.dat -s epismall1.jpg.

This would iterate through all files listed in the file library.dat, pick 5 spots to check in the source file and see how long the discovered byte-strings were in each file. The file with the largest score would be used as the actual data file.

To then reconstruct the original file you would:

jeff@jeffhpdev:~/dev/bitmimic$ ./bitmimic -b -d ./house.JPG -t epinew.jpg -r epirecipe.dat

-b means to build -t from recipe in -r using -d as a data source.

output of program

And there you have it. A few notes about this:

1. In the example the target file (epismall1.jpg) is 54K. The data file of the house is 1.5M.

2. In this exact scenario the analysis bit took just over 8 minutes to process.

3. The generated recipe file is 212K.

Challenges to the system:

There are two primary things that the larger system shall attempt to resolve:

1. Recipe file size. Right now it is just a text of offsets and byte ranges. LOTS could be done to mitigate this, many having nothing to do with parallel processing such as making it a binary file, smarter recording of repeating byte-patterns and so on. However one of the things that could seriously help all of this and still be a result of massive CPU power is finding longer byte strings. Most are in the 2-3 byte range which means more entries and a longer recipe file. By having multiple processes searching on different data files for the longer byte strings could help with this. This may however be at odds with the second thing the larger system will try to resolve:

2. Faster execution. If smaller recipe files were not a concern, having process/machine A break up the search and spread it amongst machines/processes B-n it would obviously accomplish the overall search many times faster.

All of this will require some specialized IPC along with some serious thought which I am applying to the problem right now and will be the subject of future articles. As a side note however, I have just moved to a serious HOT climate and am finding it hard to do more than a few hours per day since our AC is borked. It will happen, just on my own schedule. No one is paying me for this so it gets done when it gets done.

All of that said, it is interesting being able to recreate any piece of data from any other piece of data when you have none of the original data available (in it’s original form that is). I have used that same JPG of my guitar to generate everything from other JPGs to hour-long videos and everything inbetween.  However as a computationally intensive problem this is perfect for the larger system which I hope to present soon as it develops/the weather cools/I get the air fixed in our home. It’s going to be a great ride though.

The code here is simple but I encourage people to experiment with it; it will be much much more when it “grows up”.

The code for the sample is here.

Me to you,

JeffC

First, the project: DVD Metabase.

DVD Metabase is a collection of tools that does some things for myself and at least 300+ other like-minded individuals with Home Theater PC (HTPC) environments that you probably cannot find anywhere else. While the feature-list of DVD Metabase is lengthy, one of the keystone features is the idea of hashing or creating a unique checksum of your commercial DVD, looking up that checksum in a database and if found, returning a host of metadata about the disc to the calling application. What is this good for or actually mean to you? Well, if you integrate this with your DVD media extractor what you get is something that can rip titles with proper names to your hard disc for later viewing.

Here is a real-world example:

Take the DVD version of the scifi epic Total Recall (great Philip K. Dick story, iffy Schwarzenegger vehicle), insert the disc into your PC to prepare it for your HTPC and even with some bits of intelligence your ripped files will look like this:

./total_recall
|-- features
| |-- total_recall_features_03.avi
| |-- total_recall_features_05.avi
| `-- total_recall_features_07.avi
|-- total_recall01.avi
`-- total_recall_02.avi

1 directory, 5 files

Now the same disc when run with the DVD Metabase-enhanced tools. First the disc hashed with tools from libdvdread and we get:

7c4673a5a32cb10588dd635218857542

Then that hash is submitted to the database and we get back:

jeff@jeffhpdev:~/dvdmetabase$ ./chkdisc.py
connecting to ... http://jbcobb.net:8000
doing record search...
converting to a string...
total_recall|7c4673a5a32cb10588dd635218857542|unkid
Title found:  total_recall
Disc ID:  7c4673a5a32cb10588dd635218857542
5 tracks found:
[tracknumber]-type:title
[01]-m:total_recall
[02]-x:featurette
[03]-x:visions_of_mars
[05]-x:storyboards
[07]-x:trailer

We (and by extension our rip tools) know that track one is the main feature, named “total recall”  and that tracks 2,3,5,7 are all extra features (Featurette, Visions of Mars, Storyboards and the trailer). This allows our rip tools to rip the files to a disc tree that looks like the following:


jeff@jeffhpdev:~/dvdmetabase/rips$ tree ./total_recall
./total_recall
|-- features
| |-- featurette.avi
| |-- storyboards.avi
| |-- trailer.avi
| `-- visions_of_mars.avi
`-- total_recall.avi

Quite the difference, no? It gets even better with episodic television DVDs:

Disc 3 of the seventh season of Crime Scene Investigation:

jeff@jeffhpdev:~/dvdmetabase/rips$ tree ./dvd_video/
./dvd_video/
|-- dvd_video_01.avi
|-- dvd_video_02.avi
|-- dvd_video_03.avi
`-- dvd_video_04.avi

becomes:

jeff@jeffhpdev:~/dvdmetabase/rips$ tree ./csi_season7_disc3/
./csi_season7_disc3/
|-- 09_living_legend.avi
|-- 10_loco_motives.avi
|-- 11_leaving_las_vegas.avi
`-- 12_sweet_jane.avi

0 directories, 4 files
jeff@jeffhpdev:~/dvdmetabase/rips$

Even better. If this were the only feature of such a system it would be cool enough but the metadata can also include things like links to trailers, web-based content, advanced rip settings and more. The ripper itself can automatically select specific audio and/or subtitle tracks so if the disc is in a language you don’t understand, your preferred audio track is selected and if that is not present, the correct subtitle track is selected. Different track types (movie, show, cartoon, etc) can have specialized rip quality settings applied. A great number of features can be configured.

Since a system like this is only as useful as the data sitting behind it, I created a public XML-RPC service that I host on a spare Linux box here at Casa Cobb (http://jbcobb.net:8000). Authorized users can scan unknown discs, fill out a simple submission form and update the central server so that others can benefit from this work.

This entire system was built with free software:

Front-end:

Ripper: Handbrake (http://handbrake.fr/)

Hashing code (http://freshmeat.net/projects/libdvdread/)

Python (http://python.org)

Back-end:

Sqlite (http://sqlite.org)

Python (http://python.org)

XML-RPC (http://xml-rpc.org)

Project host (https://sourceforge.net/projects/dvdmetabase/)

The point of the article:

The project itself is useful and powerful when used properly. However the point is that much of this would not have been as easy and in some cases possible with only proprietary software. Creating this costed nothing; the only on-going costs are my bandwidth fees. With proprietary tools I would need to have bought a server version of Windows, a client version of Windows, a Python development environment (to get the same IDE-abilities I get with Dr. Python), I would have to have developed the hashing algorithm from scratch and more.

This was brought to mind lately when I had to fulfil a contract for a client that needed to support 32 and 64-bit Windows with C++ and AutoCAD. First, to fully support the needed environments I would need two versions of Windows (Linux: 0$), two specific versions of the C++ compiler (32/64, 2005 and 2008 at $200 each) (Linux C++ compiler for all platforms: 0$) and of course AutoCAD ($$$; if you have to ask, you cannot afford it). Granted there are no versions of AutoCAD for Linux but there ARE DWF-compliant CAD programs for Linux (QCAD for one). As you can see you are paying a toll at each step of the toolchain.

It is enough to make you go running for GCC and embrace it with open arms. If I had to develop my projects using Microsoft tools, the project would probably never have gotten off the ground. With proprietary tools and environments you keep hitting what I call intellectual speed bumps…with FOSS if you want to see how something works, you just *look*. If you need a tool to get a job done, you just install it and not give it another thought. With proprietary tools you have to find it, make sure you can afford it, wait for it to arrive if electronic delivery is not an option, get it installed and dance around with DRM just to get into your project. By the time you get your tools installed, your competitor that uses FOSS tools is probably delivering milestone 1 at the very least…

It is times like these that FOSS feels like a comfortable old sweater. The pragmatist in me says I will continue to have to wrestle with proprietary tools as time goes on but every single experience like this ensures that every project where I need to get something done and done quickly and right the first time will be done with FOSS tools, no question about it.

Me to you.
JeffC