Concurrent Development, part 3: The Problem

Welcome back. I have been noodling things over and tried to come up with an interesting yet useful problem to solve. The problem needed to be one that was challenging from a computational standpoint yet simple to state the parameters of. It also has to be something that at least I have never solved before nor should there be any known applications of it. We needed to be cruising in some uncharted waters; otherwise I feared the same old problems would suggest the same old solutions and this column is all about thinking outside the box

There is a saying the more things change, the more they stay the same. I was watching a supposedly “new” television show the other night and had “plot deja vu” where I swore I had seen it before. Of course I had not but rather the same basic tired plot being applied to a new show. Hollywood.

It got me thinking however that there really isn’t anything new under the sun; and that lead to the thought that most things in the digital realm were indeed not original but rather a composition of naturally occurring sequences found in either nature or other public domain work. It is only in the totality of all of the sequences does anything resembling originality begin to appear. If this is true, it is highly ironic that I could have bits of public domain data A, B and C but if I combine them into D, I have something that you could get sued over.

But how to prove it? Well, this is where a good concurrently executing application comes in: We will start with what would normally be considered an original work, decompose it to a series of byte patterns and find them in naturally occurring data. So that we don’t get the copyright cops after us, we will start with a picture of my 1965 Epiphone Riviera:

My 65 Epi Riviera

We will take that supposedly-unique picture, break it down into smaller elements, then attempt to find those elements in nature or the public domain.

The proof for success of this will be ability to create a set of links and byte ranges and offsets and recreate the “original” work completely using external sources. This will prove the process is possible and achievable. The next step will be to retool the solution such that it takes the full advantage of the processing power of a dual-core Intel box. Step three will be to generalize the solution so that it could be applied to similar type of pattern search. This could have applications in the medical research field. Fourth, we will create an application framework to allow others to use this to solve other problems. The final goal will be to make it run as fast as possible on as many platforms as I can get access to.

Some key design goals are:

• Process management should happen naturally as a part of the problem-solving process. If there is anything specific in the solution to the problem that involves direct thread or mutex management, its time to rethink. The problem solution should be all about the problem, not a solution to the platform.
• Likewise, resources should be naturally managed by the framework such that when viewed from the perspective of the problem solution, it would appear as if it were the only process on the machine.
• InterProcess Communication should be seamless and work as well across threads, processors or networked machines.
• Finally, data needs to be shared in the same way; it should be transparent to the problem domain if the data to be processed is local, shared or remote.

First we need a new way to express the problem, then build a framework for accepting that expression and executing it. If we abstract it correctly, it should be an easier job moving it between platforms. However if at any point we find ourselves breaking any of the rules above, starting over is always an option. It has to be or we will wind up with the same tired solutions. The true test of this system when done will be to put it into the hands of someone *not* skilled in this and have them solve a somewhat similar problem quickly and efficiently.

Until next time. I am on a bit of a deathmarch at work right now so it may be a few weeks before much more happens on this but this will happen. I have a larger goal in mind that I think is worthy but will only be possible with a system such as this. It will require all the computing power I have which means every box in the house gets in on the act, from the iPod to the PS3 to the laptops and even the webserver. This is going to be great.

Jeffc