Greetings all. This will just a be short post to provide CMake Wizard 2.3. This is just a maintenance release to cover some changes in GCC that have come out recently that prevented the generated projects from compiling. Essentially it is just the inclusion of stdio.h to the file to provide the macro EOF for the getopt stuff. I do have some other directions I want to take this (and some bug-fixes to cover edge-cases) but this will at least keep the project working for now. The link to the project is here:

CMakeWizard 2.3

continue reading "CMake Wizard 2.3 (Maint. release)"

Greetings. I have a couple of things to cover here so will get right to it.

1. Microlog 1.1a: It was pointed out to me that for all the tricks that it did, it was missing the basics of proritizing log message a la Info, Warning, Error. This has been fixed and is in the new code dump here:
Microlog 1.1a

Plus some forward-looking statements…

continue reading "Microlog 1.1a"

There are two things every programmer of multithreaded applications knows:
1. Logging is the simplest form of debugging for such systems.
2. The act of logging triggers context switches which changes the execution which in turn means the application behaves differently under debugging conditions than it does under normal conditions.

Often the application of #1 results in #2 meaning the “bug” can go away or change in nature to the point where the original bug is no longer obvious. You can assume that you have the bug fixed yet as soon as you remove/disable the debug code and the bug can return or worse, uncover a completely new one.

Part of the reason for this is that writing to a log, socket or other place to store the log will cause the kernel to execute a context switch. This means that different threads will execute in a different order under debug conditions…this is what I call “the observer effect”: a watched program executes differently than one running in release mode. This is even more noticeable when running it under GDB or Valgrind for example.  There are two ways to deal with this:

continue reading "MicroLog 1.0a"

Man oh man have I had an experience. It stems largely from living in the Windows world when it was king (and therefore the only standard) and then moving to larger world of free software and real standards.  Like many in this down economy I am looking for work and in an effort to make myself stand out and thought that a video resume might be a good way to do it.

This story is about how easy it is to do all of this using free tools and then to make it work under both Linux and Windows….only this is not about Linux vs Windows. It is more of a case of Microsoft Internet Explorer vs the rest of the world….

continue reading "A Video Interview Using Free Tools"

In this article:

* A new DVDMB C++ API

* To plug-in or not?

* To Mono or not?

* To Java or not?

continue reading "New directions for DVD Metabase: development odds and ends…"

CMakeWizard 2.0 is here.

Howdy. It has been a while since I have updated this handy development tool. In case you don’t know, CMakeWizard is a simple set of Python scripts and C++ templates for jump-starting development projects. In short, you can start with an empty folder, run the CMakeWizard script and in seconds have a complete if skeletal C++ project up and running. It can create a shared or static library and an optional executable to test it all. If you want POSIX thread support you can have that too, along with optional sample code so that you can be up and running in no time. Finally, after generating the proper CMake-based project, it creates the C++ source and builds everything to prove that it works.

Now it is even better; it has built-in GTK support…

continue reading "CMakeWizard 2.2 is released"

Hi Folks;
Waaaay back in 2007 I wrote about coming up with a new development model for dealing with todays multicore systems and larger than life problems. The stories are here, here, here and here. As of here I outlined a problem that would fit our needs in developing such a system. It was a problem that should tax a single-processor system to the point where it would be relatively inefficient to resolve yet had a tangible result. The first step was to create an non-efficient yet runnable program to illustrate the problem and act as a baseline. That is what this article represents.  You can read the original article or just take in the outline below:

continue reading "Concurrent development challenge: The Problem (part 2)"

This article is a profile on experiences I have had while being an administrator of an open source project. While some aspects of this have been published before, recent work in the world of proprietary software has highlighted in my mind a few of the more powerful aspects of living in a FOSS world that I am sure the proprietary software makers would prefer you not to embrace.

continue reading "FOSS: Freedom to explore, create and avoid intellectual speed-bumps"

Authors note: This was initially started some time ago but was discontinued due to schedule and other constraints. It is my intention to bring the rest of this theory to fruition with code and example. For now, theory:

Consistent Thread Execution Manager.

Theory:
A context switch is when the scheduler halts one thread of execution and starts another. These context switches are often done based on what the kernel thinks is best for any given situation.

However when developing a non-trivial multithreaded application the engineer will have a perspective on how things should work from a task-accomplishment point of view and code his or her logic accordingly. The problem is that context switches usually result in threads executing different than the engineer envisioned, resulting in unexpected behavior. This is usually the case when sleep-statements are used to force context switches when things are behaving in an erratic manner. One of the many reasons this is bad that is that the engineer is generally patching a situational problem (different load, different OS, machine, etc).

For any given non-trivial application, there are many permutations of context switches. Of these, X context switches will allow the application to complete, Y permutations will not resulting in deadlock or worse. Moreover, within X there are some permutations of context switches that will work better or more efficiently than others.

So step one is to identify the X group from the Y group.

This will be done using the thread context switch recording technique embodied in Modern Multithreaded Programming book.

Two key support routines for this will be the scoring module which will report how well any given thread does its job and completes, and the switch learning module which could use a GA or time-based neural net to determine success. Context switch patterns that result in deadlock or race conditions, even temporarily are discarded and successful patterns are retained.

It is unrealistic to think that a single context switch pattern can be determined as “best” for any non-trivial application and load situation.  Therefore it seems that a series of context switch subpatterns can be determined and applied to different problem domains based on load or other system stress indicator.

Further, these subpatterns must be repeatable else storing and enforcing a playback pattern could use an unacceptable amount of memory. Thus the subpatterns must be generalized and tagged. With a proper thread class not only can the context switches be stored and played back but also thread status (”interruptible” vs “let me process” vs “critical section”) giving further guidance to the scheduler.

Each thread object (or struct in C) also has the concept of a default score. Lets say the range of the score can be 0-100; if a thread runs to completion a minimum score of 50 is achieved. The caller or creator of the thread can then further augment the score with up to 50 additional points based logically on how well that thread accomplished its task. When in “score” mode, the scheduler then collects scores from threads as they exit (C++ would make this easier but may not be available everywhere), it can feed these values into a GA or neural network set up to determine best of breed sets of switches. Logical scoring can include how quickly a routine accomplished its task, how well it used the time-slice it has been allotted, etc. For example, if there are two threads and the second thread spends an inordinate amount of time waiting for the first thread to do something, the excessive context switches will count against the score of the second thread which takes away from the overall score of the subroutine or app.

Once an optimal set of context switches has been determined (all scores must be at least 50 (an arbitrary number meaning the task/thread ran to completion), these switches are archived and when played back, assures predictable application performance and stability.

At first these patterns will be application-wide until the basic bugs can be worked out; from there, submanagers will handle subsystem patterns. Note: there may need to be come intersubsystem communications to coordinate activities.

To demonstrate this idea, the following elements will require development:

The task scheduler:
* Issues new threads based on ID (0-256)
* In training mode: records context switches, stores patterns, collects scores from threads, stores final score.
* In playback mode: Manages all context switches.

The thread class:
* Basic threading functionality (pthread)
* Has API for application-level score
* Reports to scheduler at a minimum a complete/non-complete score, augmented by the logical application score.

The API for application-level scoring should include two components:
1. A score for “thrash”
2. A logical score for fitness to purpose (whatever the purpose the thread has)

(To be continued…)

It seems strange being in Vegas and doing all the things that most people are doing involuntarily. Where other folks have quit their jobs involuntarily I did mine on purpose to take a sabbatical. I look forward to working on all of the crazy things that I never had the time for before plus being out from under Sony’s umbrella I can work on what I want.  So the things that will be coming out of my skunk works are:

1. A lockless self-adapting Producer-Consumer engine. I have long suspected the context switches were the killer of productivity so measuring those context switches and and finding alternative things for a thread or process to so instead of wasting time worker threads can do other work, keeping the CPU(s) at a predictable level of busy. I have seen this work in the lab and it’s cool.

2. Writing a Blues-based test adventure.

3. Finishing DVD Metabase.

4. Getting better at Wind Chill, Hideaway and other blues greats.

5. Spending time with my sweetie for the first time. I have done nothing but work hard since we got married. I owe her so much.

6. Get caught up on all of the anime I have collected.

7. Same for my B-movies.

8. Making a home for us.

9. Learning. I miss that.

Since Sony no longer looks like they will be sending contract work my way I will use this time well.

JeffC