March « 2008 « Brainrack Aborning – Developing a Search Engine

Qt: Quality Control, Documentation, Pre-Release Testing

March 31, 2008

In a response to a previous post, Henrik Hartz of Trolltech asked for some specific comments on the documentation and testing issues I find troubling in Trolltech’s Qt. After some consideration, I decided to publish my response to Henrik instead of taking it off-line. Although what I have to say is critical, I believe Qt is a good product and would not want anyone to construe what I have to say as advice not to use Qt in their development work. Do use Qt.

Since we’ve talked a bit about plugins, perhaps I can illustrate my concerns with a few recent examples regarding plugins. The first is quite simple: The Qt 4.3.4 distribution contains a ‘trivial’ example meant to illustrate the use of Qt’s plugin system — the echowindow program. The program will compile, but does not work (reported as Qt Issue N202995). To me, that fact is simply astounding. Trolltech should have a long series of tests that need to be run before any new version of Qt is released; the simplest tests should make sure that all of the demonstration code works. How could such a thing be overlooked? If Trolltech hasn’t completed the simplest tests properly, how can I be confident they have uncovered any problem that is complicated? When I add that fact together with the fact that QPluginLoader actually *crashes* your application when you try to use it in a static build of Qt (reported as Issue N202989), it makes me think QPluginLoader was tested very superficially indeed.

The second problem that really made me worry about Qt’s commitment to quality was the problem with Qt internal plugins (style, imageformat, sqldriver, …) being loaded in static builds of Qt. Because of this problem I had been unable to use the debugger or to run any Qt application from within Visual Studio for several months; my applications would crash unpredictably if I ran them within the Visual Studio environment. I brought the problem to Trolltech’s attention in August of 2007 (Issue N173669). It was finally resolved with the 4.3.4 release of Qt this year. Since the problem was painful for me, I can only assume it was painful for anyone else who tried to use a static build of Qt with Visual Studio. The big question is: why wasn’t it painful for Trolltech? Was nobody at Trolltech trying to use a static build of Qt under Visual Studio? If they were, their applications must have been crashing just like mine. But, obviously, they weren’t. If Trolltech claims to support Visual Studio, and if Trolltech claims to support static builds of Qt, Trolltech must use and test static builds under Visual Studio. But do they?.

I could tell similar stories about several different areas of Qt, but perhaps you get the idea. The amount of testing that is required for a product like Qt is *enormous*. And the bigger the application library gets, the more testing that is required. The pre-release testing requirements scale with the number of possible interactions in the code – the code-base squared, not linearly!

As much as anything, this is a question of attitude and management philosophy. I have attached a letter I sent to Trolltech late last year that tries to explain some of the attitude problems I see. It is meant to be constructive. The thing to remember is that it takes time for a user to report problems to Trolltech – I limit myself to an average of a couple bug reports a month, and for every one I report there are at least three or four I don’t. It is a thankless process for the users that results in a better product for Trolltech. Trolltech should realize this, use *every* report as an opportunity to improve either the documentation or the product, and be grateful.

I will close with the worst example I have seen from Trolltech’s support. Late last year, in regard to a problem where the documentation of one of Trolltech’s classes did not match the behavior of the class, and the example application from Trolltech confused the issue even further (reported as Issue N179283), I received an email from one of Trolltech’s support people that nicely stated the attitude problem,

“This is just an example application,” he said, “It is not supposed to be perfect.”

Nonsense, I say! If Trolltech isn’t supposed to get their sample applications perfect, how can we expect to get our real applications perfect? Trolltech’s goal *must* be to try to get their samples perfect, otherwise, how does Trolltech find problems? Sometimes I wonder whether Trolltech tries very hard. At the very least, I wonder whether Trolltech’s support people have the correct dog-on-a-bone attitude when it comes to finding and fixing problems.

Please understand that I find Qt to be a very good product; it is what Microsoft should have written for MFC, but didn’t. I have taken the time to write this post because I want Qt to succeed even more.

Letter to Trolltech

Before I give Trolltech a renewal order, I’d like to express my displeasure with your support group. The attitude I’d like to see from support is a dogged persistence in hunting down and removing bugs. Indeed, a bug report from a user like me should be welcomed as an opportunity to improve either the documentation (when the user misunderstands how to apply a Qt functionality) or the code (when some part of the Qt functionality is broken). I’m not sure that Trolltech understands this.
You should understand that it takes hours from my time to prepare a bug report for Trolltech. Trolltech won’t even look at a bug unless it is distilled down to a simple case and wrapped up in a nice, neat package — even when this is possible IT TAKES TIME! Since each bug report represents an opportunity to improve the Qt product, Trolltech’s first response upon receiving a bug-report should be gratitude: Gratitude that someone would take the time. Second, Trolltech should understand that since it is a lot of trouble to report a bug, most people don’t bother. We find work-arounds.
For every bug I report, I probably ignore or work around another five. So don’t blow them off by pretending something is an isolated problem. If someone brings it to your attention, it isn’t an isolated problem. Third, I often get the feeling that Trolltech’s attitude is that we, the users, are uniformly foolish, wrongheaded, and we never read the f’g manual. We, the users, are smart enough to use your product, so drop the attitude.
Trolltech support tends to disbelieve or excuse bugs until it is illustrated in glaringly obvious ways that a bug exists. Trolltech’s attitude should be quite the contrary: assume everything reported to you is serious until proved otherwise. At the very least, try to understand the source of your user’s confusion and improve your documentation accordingly.
Trolltech is playing a dangerous game here. Sure, you want to spend as little as possible on support and bug-fixing; I understand that. But the more problems get pushed under the rug, and the less extensive testing Trolltech does of their product, the greater the likelihood that your product eventually reaches a byzantine point of no return with complexity piled on complexity and scary no-go zones in the code where everybody is afraid to touch anything for fear of breaking something.
…

Posted by brainrack

Filed in Development Environment ·Tags: Debugging, Documentation, Qt, Quality Control, Testing, Trolltech

Deriving the Expectation-Variance Model

March 29, 2008

Last week I did some work trying to make mathematical sense of the Precautionary Principle. For better or worse, except for a couple weeks in a Stanford econ class that I had to drop, I haven’t got any formal econ. (The prof told me that I was so obstreperous that most of his colleagues would deal with me by tossing me from their offices — I got the hint.) So, every time I wrote something down I had to go check to see if it was a known result, or simply false, or maybe a little bit new. Thus, I got a quickee education in some modern economics.

Several times along the way I ran across some hand-waving arguments about the functional form of the `total marginal utility curve’ and its relationship to peoples’ tendencies towards risk-aversion. When I tried to prove what I thought various authors were suggesting about this relationship I found I couldn’t. One hint about doing proofs: when you are stuck, look for a counterexample. It often helps understand things well enough to make the proof go through. Of course, sometimes, you actually find a counterexample; as in this case!

I had been wasting time trying to prove something that wasn’t true. It happens; but I was curious as to what was going on and started looking in the literature. It turned out that I had run across one of those things that is just too beautiful to be false. In spite of the fact that everyone knows the relationship only holds in very narrow circumstances, a lot of people continue to pretend it is more true than it is.

In the hope that it saves someone else some confusion, I have written up my notes on the subject. There’s nothing particularly new here, but it was a pretty good test of my LaTex to WordPress conversion macros in emacs, which I described here. I will publish them after they have ripened a little bit.

Along these lines I will also write up my notes on the mathematics of the Precautionary Principle in a few days, and then return to describing the latest developments on the Brainrack Search Engine.

[Update: 04 April 2008 - Fixed a few typesetting errors and other minor errors in the math.]

Pitfalls in Deriving
the Expectation-Variance Model
for Investment Value

G. Neil Haven
Liberty Reach, Inc.
Clearwater, Idaho

28 March 2008

ABSTRACT

The Expectation-Variance Model for Investment Value posits that the value of an investment to an investor is a function of the investor’s sensitivity to risk, the expectation value of the return from the investment, and the volatility in that expectation value. Gossen’s First Law, or the law of diminishing marginal utility, is often used to derive this relationship between the investment value of a set of assets, on the one hand, and the expected return and volatility of the set of assets, expressed in terms of a variance, on the other hand. Misleading evidence for and derivations of this relationship are rife in the formal and informal literature.¹The primary confusions arise from two assertions: that the utility function of wealth is quadratic;² and that the Taylor expansion of the utility function may be reasonably truncated at low order.³ This note reviews the fallacies in turn, provides a relevant counterexample, and exhibits the missing constraints needed for the argument establishing the Expectation-Variance Model to go through.

Statement of the Hypothesis

The statement of the false hypothesis is the following:

Hypothesis: Given $U(x)$ , a utility function,

$U(x) \ni: \forall x\, U'(x) > 0, U''(x) < 0$

and given two distributions

$f_1, f_2: \mathfrak{R}\Rightarrow[0,1] \ni: 1=\int_{\forall x}f_1(x)\,dx = \int_{\forall x}f_2(x)\,dx$

with the same mean

$\overline{x} = \int_{\forall x}f_1(x)x\,dx = \int_{\forall x}f_2(x)x\,dx,$

the inequality

$\int_{\forall x}f_1(x)(x-\overline{x})^2\,dx - \int_{\forall x}f_2(x)(x-\overline{x})^2\,dx$

$= \sigma^2(f_1) - \sigma^2(f_2) = \Delta_{\sigma^2} > 0$

implies

$\int_{\forall x}f_1(x)U(x)\,dx < \int_{\forall x}f_2(x)U(x)\,dx.$

The economic interpretation of this hypothesis is that, given the law of diminishing marginal utility, and assuming the expected payouts from two different investments is the same, the investment with less variance will be preferred.

Plausibility of the Hypothesis — Taylor Series Expansion

Expand $U(x)$ around $\overline{x}$ as a Taylor Series:

$U(x) = U(\overline{x}) + (x-\overline{x})U^{(1)}(\overline{x}) + \frac{(x-\overline{x})^2}{2!}U^{(2)}(\overline{x}) + \sum_{j>2}\frac{(x-\overline{x})^j}{j!}U^{(j)}(\overline{x}).$

Then form the integral $\int_{\forall x}f(x)U(x)\,dx$ and evaluate

$\int_{\forall x}f(x)\left(\sum_{j}\frac{(x-\overline{x})^j}{j!}U^{(j)}(\overline{x}) \right) dx$

$= U(\overline{x}) + \frac{\sigma^2(f)}{2}U^{(2)}(\overline{x})\,dx + \int_{\forall x}f(x)\sum_{j>2}\frac{(x-\overline{x})^j}{j!}U^{(j)}(\overline{x}) \,dx.$

Calculate the increase in investment value by forming the difference:

$\int_{\forall x}(f_2-f_1)(x)U(x)\,dx$

$= \frac{\sigma^2(f_2)-\sigma^2(f_1)}{x}U^{(2)}(\overline{x}) - \int_{\forall x}(f_2(x) - f_1(x))\sum_{j>2}\frac{(x-\overline{x})^j}{j!}U^{(j)}(\overline{x}) \,dx$

$= \frac{-\Delta_{\sigma^2}}{2}U^{(2)}(\overline{x}) - \int_{\forall x}(f_2(x) - f_1(x))\sum_{j>2}\frac{(x-\overline{x})^j}{j!}U^{(j)}(\overline{x}) \,dx.$

This last expression is positive, so that the investment value increases with decreasing variance at constant expected value, when

$-U^{(2)}\Delta_{\sigma^2} > 2\int_{\forall x}(f_2(x) - f_1(x))\sum_{j>2}\frac{(x-\overline{x})^j}{j!}U^{(j)}(\overline{x}) \,dx.$

Confusion arises because, assuming the Taylor Series expansion is convergent, the right hand side of this expression is close to zero. (Since $U^{(2)} < 0$ , the left hand side is positive, by hypothesis.)

Illegitimate Derivation — Truncation of the Taylor Series Expansion

The illegitimate step from here is to assert that, being a difference of two tails from Taylor Series expansions, we may treat the right hand side as zero compared with $\Delta_{\sigma^2}$ . That is, since

$\int_{\forall x}(f_2(x) - f_1(x))\sum_{j>2}\frac{(x-\overline{x})^j}{j!}U^{(j)}(\overline{x}) \,dx \approx 0$

then (falsely)

$-U^{(2)}\Delta_{\sigma^2} - 2\int_{\forall x}(f_2(x) - f_1(x))\sum_{j>2}\frac{(x-\overline{x})^j}{j!}U^{(j)}(\overline{x}) \,dx > 0.$

Since $\Delta_{\sigma^2}$ itself is the difference of two terms, and may be arbitrarily close to zero, the simplification is illegitimate.

An extra condition is needed for the valid applicability of the Expectation-Variance Model, to wit:

$-U^{(2)}\Delta_{\sigma^2} > 2\int_{\forall x}(f_2(x) - f_1(x))\sum_{j>2}\frac{(x-\overline{x})^j}{j!}U^{(j)}(\overline{x}) \,dx.$

Illegitimate Derivation — Quadratic Utility Function

Rearranging the extra condition we can obtain

$-U^{(2)}\Delta_{\sigma^2} > 2\sum_{j>2}\frac{U^{(j)}(\overline{x})}{j!} \int_{\forall x}(f_2(x) - f_1(x))(x-\overline{x})^j \,dx.$

A simple way to satisfy this condition is to require $U^{(j)}(\overline{x})$ to be identically zero for $j>2$ . That is, $U(x)$ must be quadratic. So $U(x) = ax^2 + bx + c$ . But a quadratic $U(x)$ cannot be a utility function: diminishing marginal utility implies $U^{(2)}(x) = a \prec 0$ , but then, for $x \succ -b/(2a)$ we have $U'(x) \prec 0$ , a contradiction. $\square$

I surmise that what happens here is that commentators confuse $U(x) = ax^2$ , which has a zero second derivative but the wrong structure for a utility function, with the inverse function $U(x) = kx^{1/2}$ , which has the correct structure for a utility function but a non-zero second derivative. The needed mental hybrid cannot exist.

Legitimate Derivation — Gaussian Distribution

Look at the extra condition again

$-U^{(2)}\Delta_{\sigma^2} > 2\sum_{j>2}\frac{U^{(j)}(\overline{x})}{j!} \int_{\forall x}(f_2(x) - f_1(x))(x-\overline{x})^j \,dx.$

A little rearranging shows that the right hand side is a sum of third-order and higher moments from the distributions yielding $f_1(x)$ and $f_2(x)$ . If the distributions are gaussian, this sum is identically zero. Thus, a sufficient condition for the Expectation-Variance Model is that $f_1(x)$ and $f_2(x)$ are pdf’s from gaussian distributions.

While it is true that a gaussian distribution arises naturally as the distribution of the sum of i.i.d. random variables, and so the hypothesis that $f_1$ and $f_2$ represent normal distributions may seem initially plausible, careful consideration, a priori, indicates that the default distribution of investment returns ought to be the result of a multiplicative, not an additive, process.⁴ In that case $f_1$ and $f_2$ would be most naturally taken to be pdf’s from log-normal distributions, not gaussian ones. The higher-order moments from a log-normal distribution are not zero.

Counterexample to the Hypothesis

Take $U(x) = \lg (x+1)$ . It is trivial to verify $U(x)$ is a utility function.⁵ To contradict the hypothesis above, it is enough to exhibit two distributions with identical means and identical variances, but non-identical expectation values.

Consider the distributions $D_1 = [x=1,p=2/3; x=4,p=1/3]$ and $D_2 = [x=0,p=1/3; x=3,p=2/3]$ . For both distributions, mean = variance = 2. The expectation value of $U(x)$ in distribution $D_1$ is $1+\lg 5$ . The expectation value of $U(x)$ in distribution $D_2$ is $2$ . $\square$

Restatement

From the above discussion we may collect a true restatement of the original hypothesis.

Given $U(x)$ , a utility function,

$U(x) \ni: \forall x\, U'(x) > 0, U''(x) < 0$

and given two distributions,

$f_1(x), f_2(x) \ni: 1=\int_{\forall x}f_1(x)\,dx = \int_{\forall x}f_2(x)\,dx$

with the same mean,

$\overline{x} = \int_{\forall x}f_1(x)x\,dx = \int_{\forall x}f_2(x)x\,dx,$

define $\Delta_{\sigma^2} =$

$\int_{\forall x}f_1(x)(x-\overline{x})^2\,dx - \int_{\forall x}f_2(x)(x-\overline{x})^2\,dx$ .

The inequality

$-U^{(2)}(\overline{x})\Delta_{\sigma^2} > 2\sum_{j>2}\frac{U^{(j)}(\overline{x})}{j!} \int_{\forall x}(f_2(x) - f_1(x))(x-\overline{x})^j \,dx$

implies

$\int_{\forall x}f_1(x)U(x)\,dx < \int_{\forall x}f_2(x)U(x)\,dx.$

In particular, if $f_1$ and $f_2$ are the pdf’s from gaussian distributions, $\Delta_{\sigma^2} > 0$ is a sufficient condition for the conclusion.

Comments

It seems a more natural statement of investment value at constant expected return would be in terms of the entropy of a distribution. Entropy is a single number describing a distribution, whereas most distributions require a series of moments to describe.

¹ – This is not a new observation.

² – Although the authors do not treat the origin of the confusion, for a review containing nearly a dozen examples within 10 years from the formal agricultural economics literature alone (!) see “Quadratic Utility and Linear Mean-Variance: A Pedagogic Note” Robert A. Collins, Edward E. Gbur Review of Agricultural Economics, 13(2) (Jul., 1991), pp. 289-291

³ – For a numerical discussion of the problem see Loistl, Otto “The Erroneous Approximation of Expected Utility by Means of a Taylor’s Series Expansion” The American Economic Review, 66(5) (Dec., 1976), pp. 904-910.

⁴ – See, for example, Conover, John “The Quantitative Analysis of Non-Linear Entropic Economics” online at http://www.johncon.com/ndustrix/.

⁵ – This is a reasonable utility function. If we assume that marginal utility decreases directly with increasing wealth, so that $U(x+dx) - U(x) = k/x dx,$ then $U(x) = lg(kx+c)$ .

Posted by brainrack

Filed in Off-Topic ·Tags: expectation variance, investment value, marginal utility

3 Comments »

News Feed Scraper Application

March 27, 2008

Intro to the Feed Manager Application

There are plenty of feed readers, but there don’t seem to be any feed scrapers readily available. By a `feed scraper’ I mean a program that will download and store the actual text of feed items gathered from rss (or atom) feeds.

News Feed Scraper

I am releasing my news feed scraper. The development environment for the release is C++ under Microsoft Visual Studio 2005 using Qt Commercial Edition from Trolltech. If you want to just run the executables you do not need Visual Studio or Qt. The executables have been tested on Vista and XP.

The release consists of three downloads:

A set of development libraries here;
A generic template for creating plugins here;
A feed server and feed client here.

In order to run the software you will need to:

Download the feed server and client;
Unzip the download. This will create a FeedManager and a FeedServer directory;
Run FeedClient/Executable/FeedClient.exe;
Run FeedManager/Executable/FeedManager.exe;
Drag the url from a feed into the FeedManager…

In order to compile the software you will need to:

Download the three downloads;
Follow the directions here to install the development libraries (in the course of which you will define an environment variable $(DIR4APPS), which contains the name of your preferred installation directory);
Install the plugin solution in the $(DIR4APPS) directory. This should create a $(DIR4APPS)/ModifyUrlPlugins/ directory;
Install the FeedManager solution in the $(DIR4APPS) directory;
You should be able load the solution file $(DIR4APPS)/FeedManager/FeedManager.sln.

The ModifyUrlPlugins directory illustrates how to write .dll plugins under Qt that can be explicitly linked at run-time with a statically-linked executable. See here for a write-up on this.

Function of the Feed Manager Application

The news feed scraper consists of two separate applications: a Feed Client, and a Feed Server.

The Feed Server maintains a database containing the URLs of a series of rss feeds. Periodically, the Feed Server checks to see if there are any new articles mentioned in the rss feeds, by downloading a new version of the stalest feed in its database. When the server encounters a new article mentioned in a new version of a feed, it puts the URL of the article into a table in its database.

The Feed Client periodically checks with the Feed Server to see if it knows of any new articles. If it does, the Feed Client downloads the article and stores it, uncompressed, in a directory called Articles. If the article is unavailable, for any reason, it is skipped and not retried.

The Feed Server and Feed Client communicate with one another via the UDP protocol. They can be run on the same machine or on different machines visible to one another on the same network. A maximum of 3 Feed Clients can communicate with one Feed Server. The Feed Server downloads feeds; the Feed Client downloads items (articles) mentioned in those feeds.

Screen Shots

The Feed Server

Here’s a screen shot of the Feed Server. The application has two modes: server mode, and database Mode. In server mode the application displays the articles being served and their status. In database mode the application allows you to edit the database of feeds the application is managing. The application continues to download feeds and manage communication with the Feed Client in the background while in database mode.

The Feed Client

Here’s a screen shot of the Feed Client. The application has only one mode: client mode. In client mode the application displays the articles being downloaded and their status.

Status of the Application

I have used the application to download about a gigabyte (so far) of text from news feeds. The application is a reasonably well-behaved net citizen: it has a substantial delay between repeat visits to the same web-site, for instance. However, it does not use the http `if-modified-since’ feature. It probably should.

The application runs pretty smoothly for an application that hasn’t been banged on too much, but I’m sure there are still a few bugs in there somewhere. Never forget the General Rule of Debugging:

Bugs occur with all frequencies;
The time it takes to identify and fix a bug is proportional to the inverse of its frequency.

Therefore, if you’ve only banged on an application for a few weeks, you are going to miss the bugs that only occur on a monthly or yearly schedule. And just because you haven’t seen such infrequent bugs doesn’t mean they aren’t there.

Posted by brainrack

Filed in Corpora ·Tags: atom, feed scraper, freeware, Qt, rss

7 Comments »

Qt Plugins and Static Builds

March 19, 2008

Trolltech’s Qt

The Trolltech development library ‘Qt’ is a useful set of lovely software libraries that has two purposes: First, it is a platform-independent way to code in C++. Second, it offers a set of libraries with logical, documented interfaces apparently written by adults. Even if you don’t need platform-independence in your code, you should use Qt.

That said, I have my problems with Qt. It is insufficiently tested, insufficiently documented, and parts of the libraries suffer from poor design (beware the QTableView and QRelationalTableModel classes!). These problems generally arise from an overly-ambitious feature list and a weak mindset for debugging. But I suffer from these tendencies as well, so I can be a little forgiving… even if I give technical support merry hell from time to time.

Qt offers a nice plugin architecture for modular application extension. The interface necessary to use it is a little bloated, consisting of six (!) different macros, classes, and templated functions you need to strategically sprinkle through your code with some dubious and apparently useless arguments and options, but, so long as you use the plugin architecture for dynamically linked .dll’s attached to a Qt configured for dynamic linking, it seems to work perfectly well.

Let me take a moment to define some terms that haven’t got standard usage:

Static linking: functions from a library (a .lib in Microsoft-world) are compiled inside your executable;
Explicit linking: functions from a library (a load-time library, or .dll in Microsoft-world) are resolved by loading their definitions from the library (not the executable), as needed, after your program starts. In this case, after resolving a function, all you get is a function pointer — you need to know how to use the function pointer yourself;
Implicit linking: a hybrid between static and explicit linking where your executable knows where to find the functions inside a load-time library (a .dll) and how to use them (from definitions in a .lib), but the functions themselves are not compiled inside your executable. They are loaded from the (dll) library at load-time;
Dynamic linking: a program uses dynamic linking if it uses load-time libraries either explicitly or implicitly.

Implicit linking has all of the disadvantages of static linking (you need have a .lib defined when you write your executable) and all the disadvantages of using dynamic linking (you must maintain compatibility between two different loosely-coupled code bases in two different files: the executable and the dll) without gaining the advantage of using a .dll (you can decide if you need the .dll after your program starts).

Plugins with Qt Static Builds

Here’s the rub: I generally hate .dll’s. In my opinion they are overused and tend to create more problems than they solve. I have no problem with modular architecture but I am an inveterate static linker. There are times when you absolutely must use a .dll; sometimes it makes sense, but it should never be the default technique. However, Qt does not support loading a plugin from a Qt configured for static linking. That means that, if you want to use dynamically-linked plugins you have to go whole-hog and use dynamically-linked Qt, too. Welcome to dll-hell. Now, instead of releasing an executable with maybe a few optional plugins for extending your application, you have to release your application, your plugins, and every single thrice-darned Qt dll as well! Not an acceptable solution.

By doing a little searching on the internet, it appears to be programmer lore that:

Thou shalt not mix explicit- with static-linking.

Certainly the people at Trolltech believe this; their tech support told me so. Qt’s documentation does not explicitly say it, but apparently it is such common knowledge that programmers just know in their bones not to do it.

Why not?

The fear seems to be that an explicitly-linked library will potentially load code that is incompatible with code loaded by the executable. Even granting this could happen, how is this a compatibility problem a programmer doesn’t have to worry about anyway within a dynamically-linked application? Furthermore, if the load-time library (the .dll) and the executable both statically-link to all the libraries they have in common, where is the problem?

I don’t deny there could be some problems having to do with incompatible states, shared resources, or some bizarre subtleties I don’t know about, but that’s just it; if there are problems, I don’t know about them. So I ran a test.

The only thing in Qt’s source code preventing plugins from working within an executable statically-linked to Qt is a macro conditional at line #236 of src/corelib/plugin/qpluginloader.cpp (in release 4.3.4). I changed the line

#if defined(QT_SHARED)

#if 1

Then I recompiled Qt (in the static configuration) and linked into my application. My application attempted to load a plugin through QPluginLoader according to the techniques in Qt’s documentation. I created a plugin .dll, linked to Qt’s static libraries, and moved it to where my executable could find it.

I executed the application. It ran. It found the plugin, linked to it, resolved it, executed it, and never complained. So much for lore.

Now, my plugin isn’t doing anything exotic: it’s launching no new threads, sending no signals, capturing no events, sharing no static memory. Maybe if I try those things it will break, but as far as I can tell it is working for me. I’ll post the source code for my application (a news feed and blog feed text scraper) later this week.

Update (27 March 2008)

The feed text-scraper application is released and described here.

Posted by brainrack

Filed in Development Environment ·Tags: dynamic, explicit, implicit, link, plugin, static

3 Comments »

XEmacs SLIME and Corman Common Lisp

March 17, 2008

SLIME and Corman Common Lisp

[An updated version of this post is available here: http://brainrack.wordpress.com/2008/04/04/slime-swank-and-corman-lisp-v301/. The update describes how to get SLIME working with ccl.]

I spent some time trying to get xemacs (version 21.4.21) slime (version 2.0 changelog dated 2006-04-20) working with corman common lisp (version 3.01). I am able to get SLIME socket communication working, but there are still problems with corman lisp which are preventing a functional integration.

As of today (17 March 2008) here is what I have:

Roger Corman suggested on 09 January 2008 using a trimmed-down lisp console for SLIME. As things currently stand, don’t bother. Even after making the changes outlined below this lisp console will only connect sporadically with SLIME. About 90% of the time it throws an exception because it can’t open a communication socket.
You need to make the changes to [CORMAN LISP]/Sys/sockets.lisp and [CORMAN LISP]/Sys/winsock.lisp suggested by ungil on 19 Sep 2006. With apologies, I am copying ungil’s changes verbatim here:

$ diff -b sockets.lisp Program\ Files/Corman\ Technologies/Corman\ Lisp\ 3.0/Sys/
469c469
< (winsock::listen (socket-descriptor s) SOMAXCONN))))
—
> (listen (socket-descriptor s) SOMAXCONN))))$ diff -b winsock.lisp Program\ Files/Corman\ Technologies/Corman\ Lisp\ 3.0/Sys/
14,15d13
< (shadow ‘(cl:listen))
<

In the slime distribution file [XEmacs]/xemacs-packages/lisp/slime-2.0/swank.lisp you need to comment out a series of assertions. Open the file and search for ‘test-print-arglist’. Then comment out the following progn:

#-cormanlisp(progn
(assert (test-print-arglist ‘(function cons) “(function cons)“))
(assert (test-print-arglist ‘(quote cons) “(quote cons)“))
(assert (test-print-arglist
‘(&key (function #’+)) “(&key (function #’+))“))
(assert (test-print-arglist ‘(&whole x y z) “(y z)“))
(assert (test-print-arglist ‘(x &aux y z) “(x)“))
(assert (test-print-arglist ‘(x &environment env y) “(x y)“)))
;; Expected failure:
;; (assert (test-print-arglist ‘(&key ((function f))) “(&key ((function f)))”))

The corman lisp definition for #’user-homedir-pathname is incorrect. It wrongly omits a trailing backslash.
Suppose your home directory is C:\Users\Name\. Corman lisp gives:

LISP> (user-homedir-pathname)
#P”C:\Users\Name” ; should be #P”C:\Users\Name\”
LISP> (merge-pathnames “subdir\\file.ext” (user-homedir-pathname))
#P”C:\Users\dir\file.ext”
; should be #P”C:\Users\Name\dir\file.ext”

To fix this, replace the definition for #’user-homedir-pathname with the following:

(defun cl:user-homedir-pathname (&optional host)
(flet ((getenv (name)
           (let ((buffer (ct:malloc 1))
                 (cname (ct:lisp-string-to-c-string name)))
             (let* ((needed-size
                  (win:getenvironmentvariable cname buffer 0))
                    (buffer1 (ct:malloc (1+ needed-size))))
        (prog1
          (if (zerop
          (win:getenvironmentvariable cname buffer1 needed-size))
            nil
           (ct:c-string-to-lisp-string buffer1))
           (ct:free buffer)
           (ct:free buffer1))))))
   (cond ((or (stringp host)
          (and (consp host) (every #’stringp host)))
         nil)
         ((or (eq host :unspecific)
              (null host))
          (let ((homedrive (getenv “HOMEDRIVE”))
                (homepath (getenv “HOMEPATH”)))
            (parse-namestring
             (if (and (stringp homedrive)
                       (stringp homepath)
                       (= (length homedrive) 2)
                       (> (length homepath) 0))
       (concatenate ’string homedrive homepath “/”) ; GNH 03/2008
              “C:\\”))))
          (t
           (error
“HOST must be a string, list of strings, NIL or :UNSPECIFIC”)))))

Start Swank on the corman lisp side via the following steps:
1. Start [Corman Lisp]/CormanLisp.exe;
2. (load “Sys/winsock.lisp”)
3. (load “Sys/sockets.lisp”)
4. Load the new definition of #’user-homedir-pathname
5. Change directory to the slime distribution. For example (setf (cormanlisp:current-directory) “C:\\Program Files\\XEmacs\\xemacs-packages\\lisp\\slime-2.0\\“)
6. (load “swank-loader.lisp”) ; this will take some time
7. (in-package :swank)
8. (create-server)
Assuming everything works, you should see a notice that Swank started at port: 4005.
Start xemacs.
Initiate slime via
1. Alt-x slime-connect<return>
2. Host: 127.0.0.1<return>
3. Port: 4005<return>
Assuming everything works, you should see a “Connected” notice.
The *slime-repl* in xemacs seems to work, but it is not possible to edit a lisp file. The first problem to solve is that the corman common lisp type system does not understand generic functions. We have the following:

LISP> (type-of #’defun ‘GENERIC-FUNCTION)
;; throws error “Unknown type specifier GENERIC-FUNCTION”
LISP> (typep #’print-object (type-of #’print-object))
;; absurdly throws an error

These are only the problems I currently know about. There may be others.

A Note on XEmacs SLIME

Browsing through the SLIME code is a disheartening experience. Given that a lisp development environment under emacs has been a functional reality for at least twenty years, it is sad to realize that history is sputtering to its end.

The core functionality of SLIME is the following:

Create a communication link between emacs and a lisp;
Emacs reads some text to send to lisp;
Emacs sends the text to lisp, which interprets it, and sends the result back to emacs;
Emacs then displays the result.

Everything else is gravy. If need be, disable the debugger, disable the stepper, just get the core functionality running. Then, and only then, should the code contemplate steppers and profilers and debuggers and inspectors and… The primary interface file slime.lisp is almost 5,000 lines long — this amount of code can hide far too many bugs to be portable.

This Web Page Created with PageBreeze Free Website Builder

Posted by brainrack

Filed in Development Environment ·Tags: corman common lisp, slime, xemacs

Qt User Interface Sample Software

March 11, 2008

Sample Source Code

I am herewith distributing the source code and executables for a series of application templates developed using C++ and Trolltech’s Qt. The virtue of these templates is that they are free, they function, and they illustrate a number of features of Qt that, absent working source-code, it might take a person some significant time to get working. There is nothing particularly novel about them.

This software is intended for use under Microsoft Visual Studio 2005. It is programmed in C++ using Qt version 4.3.4. It has been tested under Vista and XP using the commercial version of Qt. To run it you will need a recent commercial version of Qt, Microsoft Visual Studio 2005, and a machine running either Vista or XP. I have included Visual Studio 2005 .sln and .vcproj files for compilation — you are on your own if you run the open source version of Qt or use an earlier version of Visual Studio.

Installation

In order to install this software, pick a directory for the installation and create the environment variable DIR4APPS with a value equal to the installation directory path. Copy all the directories included in the distribution into the DIR4APPS directory. You should have a number of sub-directories: e.g., DIR4APPS/Build++, DIR4APPS/UtilitiesLib, DIR4APPS/SimpleUserInterface, etc.

The projects use the static link version of all Qt libraries. [Software maintenance is hard enough without adding dll hell to your list of woes.] The projects assume you have defined an environment variable QTDIR with value equal to the Qt installation directory. If you have not built Qt for static linking, it is best to reinstall Qt (version 4.3.4 is current as of this documentation) then configure Qt at the command prompt (with Qt environment variables) via:

“configure –static –qt-sql-sqlite –qt-libmng –qt-libtiff –qt-libjpeg –qt-libpng”

“nmake”

When compiling the application, you must make sure that the utility Build++.exe is findable within the Visual Studio executables search path. To inspect the Visual Studio search path, visit /Tools/Options/Projects and Solutions/VC++ Directories. Show Directories For: Executable Files. The utility Build++.exe is distributed at $(DIR4APPS)/Build++/Release.

Contents of the Distribution

This solution contains a number of progressively more complicated application templates. In total, the distribution is approximately 6,000 lines of code.

The short road-map to the interfaces is the following:

1. SimpleUserInterface – A simple application template with a display pane, a menu/navigation pane, a text output pane, context-sensitive help, an application icon, progressively incremented build number, application style sheet (a skin);

2. UserInterfaceWithDatabase – The SimpleUserInterface with a Sqlite database attached;

3. UIwDatabaseDrop – A template illustrating drag-and-drop — with associated database.

4. LoadedUserInterfaceA – The user interface of UserInterfaceWithDatabase, with the addition of application licensing, login/logout capability, hardware keying;

5. LoadedUserInterfaceB – The LoadedUserInterfaceA, with an additional capability for minimization to the system tray.

Each project contains a description of its specific capabilities in a file named Documentation.txt. A short description of how to derive a new application from the samples is included — since Visual Studio does not help in this common task.

One stand-alone utility, Build++, is included. It is the utility used to increment the project build number for every new compilation. Consult the Usage Notes.txt file in the Build++ project directory for further information.

Two application libraries are included. UtilitiesLib contains some basic utilities for non-GUI tasks like error handling and such. It is a hodge-podge of utilities that has grown by accretion, not design. UserInterfaceLib is the library containing the description of the user interface.

While it is possible to download, install, compile, and run every single sample application in this solution within a single day, it would be wise to assume you’ll need a week to get reasonably fluent with the software. The best way to learn the software is to run the applications for awhile and play around with them. Get a feel for how they function before diving into the code. Then start with the SimpleUserInterface.

Documentation

Each project is described in a Documentation.txt file. A series of subsequent blog posts will describe the projects in a bit more detail.

Posted by brainrack

Filed in Development Environment ·Tags: application templates, Qt User Interface, source code

2 Comments »

Loaded User Interface B

March 11, 2008

Design

Here’s a screen shot from the application template

Figure 1

When first run, the application template creates an application license that will expire in a few days. In order to login, you will need to navigate to ‘Create Account’.

Features

This application template is derived from the LoadedUserInterfaceA application by adding support for minimization to the system tray. Note that in Figure 1 the application icon appears in the system tray.

Source Code

The C++ source code is contained in the LoadedUserInterfaceB subdirectory from the distribution.

Posted by brainrack

Filed in Development Environment ·Tags: application template, Qt source code

1 Comment »

Loaded User Interface A

March 11, 2008

Design

Here’s a screen shot from the interface

Figure 1

When first run, the application template creates an application license that will expire in a few days. In order to login, you will need to navigate to ‘Create Account’.

Features

This application template is derived from the UserInterfaceWithDatabase by adding the following features

Support for user accounts (login, logout, encrypted passwords);
Support for application licenses (encrypted licenses keyed to a particular machine);
Support for hardware keying (application only functions on a particular machine).

A security weakness in this demo code is that the database itself is not encrypted.

Source Code

The C++ source code is contained in the LoadedUserInterfaceA subdirectory from the distribution.

The UserInterfaceLib contains definitions for the ApplicationLicense, LockToHardware, Security, and UserAccount classes.

Posted by brainrack

Filed in Development Environment ·Tags: application templates, Qt, source code

2 Comments »

User Interface with Database Drop

March 11, 2008

Design

Here’s a screen shot from the application template

Figure 1

Any URI which is dropped into the application will be placed in the database.

Source Code

This application template is derived from the UserInterfaceWithDatabase by adding drag-and-drop support.

The C++ source code is contained in the UIwDatabaseDrop subdirectory from the distribution .

Posted by brainrack

Filed in Development Environment ·Tags: application template, Qt, user interface source code

1 Comment »

User Interface with Database

March 11, 2008

Design

Here’s a screen shot from the application template

Figure 1

This application template is derived from the simple user interface described above by adding database support for a sqlite database. My experience has been that sqlite is perfectly suited to single-user applications: it requires no configuration and runs as advertised. I have used it for databases several hundred megabytes in size with no problems in multi-threaded applications.

Source Code

The C++ source code is contained in the UserInterfaceWithDatabase subdirectory from the distribution .

The UserInterfaceLib contains definitions for the Database class. The sqlite database driver is used.

Posted by brainrack

Filed in Development Environment ·Tags: application templates, Qt, sample source code

2 Comments »

March 31, 2008

Letter to Trolltech

March 29, 2008

Pitfalls in Deriving the Expectation-Variance Model for Investment Value

G. Neil Haven Liberty Reach, Inc. Clearwater, Idaho

28 March 2008

ABSTRACT

Statement of the Hypothesis

Plausibility of the Hypothesis — Taylor Series Expansion

Illegitimate Derivation — Truncation of the Taylor Series Expansion

Illegitimate Derivation — Quadratic Utility Function

Legitimate Derivation — Gaussian Distribution

Counterexample to the Hypothesis

Restatement

Comments

March 27, 2008

Intro to the Feed Manager Application

News Feed Scraper

Function of the Feed Manager Application

Screen Shots

The Feed Server

The Feed Client

Status of the Application

March 19, 2008

Trolltech’s Qt

Plugins with Qt Static Builds

Update (27 March 2008)

March 17, 2008

SLIME and Corman Common Lisp

A Note on XEmacs SLIME

March 11, 2008

Sample Source Code

Installation

Contents of the Distribution

Documentation

March 11, 2008

Design

Features

Source Code

March 11, 2008

Design

Features

Source Code

March 11, 2008

Design

Source Code

March 11, 2008

Design

Source Code

Pages

Archives

Categories

Meta

Blogroll

Pitfalls in Deriving
the Expectation-Variance Model
for Investment Value

G. Neil Haven
Liberty Reach, Inc.
Clearwater, Idaho