It all started with a compiler warning. I like to compile my gcc projects with:

-ansi -pedantic -W -Wall

and I like my projects to compile with no warnings. My personal feelings on this subject are that if you cannot find a way to reconcile the warning, then odds are you don’t understand what it is trying to warn you about. Anyway, on to the issue at hand. I added a macro call of QT_REQUIRE_VERSION to the main function in one of my projects, the goal of course is to give a run-time error if the program is run with a version of Qt that is too old, simple enough. Unfortunately, this caused a warning in my project which I found entirely unacceptable.

main.cpp: In function 'int main(int, char**)':
main.cpp:8: warning: format not a string literal and no format arguments

I’m sure that many of you can relate to the horror I felt after seeing this in my usually clean build. First I checked, then double checked that the usage of the macro was the cause. Yup, commenting out the simple one liner use of the macro made it go away. Did I use it right? well  the call basically looks like this:

QT_REQUIRE_VERSION(argc, argv, "4.0.2")

Not much room for error there, and the only string parameter is passing a string literal. So what’s going on here? I figured it’s a pretty minor issue so I’d be patient. After waiting a while for someone to notice this, I finally decided to submit a bug. Two or so weeks went by then I finally saw that my bug had be resolved as “invalid”. Fine, my gut reaction was to check if I had misused the macro, so I’ll check out what they had to say.

I believe the warning is due to there being no semicolon after the call to QT_REQUIRE_VERSION. Note that the warning message specifies line 8 – the line after the QT_REQUIRE_VERSION call. The QT_REQUIRE_VERSION macro, like most macros, does not end with a semi-colon, so you have to use one after calling it.

I was thrilled that it was such a simple fix. How could I have been so dumb as to miss a semi-colon. An amateur mistake! So I thanked the bug assignee, mentioned that the docs didn’t have a semicolon and went on my way. Of course this wouldn’t be much of a post if things were that simple…

As you may have guessed the proposed solution had nearly zero effect on the presence of the warning. I have to say “nearly” there because it did change the line number it occurred on to be the line of the macro, instead of the line after. But the warning still remains, mocking me.

Let’s revisit the response I got (i’m adding emphasis).

I believe the warning is due to there being no semicolon after the call to QT_REQUIRE_VERSION…

Ahh, I see, so he just guessed! I think it’s time for a basic rule of bug handling to be established.

1. Actually test the bug and then test the solution before any resolution is given!

This may sound simple, but clearly it didn’t happen here. He did not test the issue or its solution, he simply looked at my example code and made a wild guess. It would have taken him all of 1 minute to copy and paste my code into a test.cpp file and compile it with the flags I gave to see that the warning would remain.

I looked for way to re-open the bug, but could not, so I submitted a second one. This time I made it clear that this was a followup to the other bug report and that I had added the semicolon, which did not help. If you are wondering how clear I was, here’s a quote from my bug report:

Note: I’ve added a semicolon as recommended by the previous bugs resolution. Which did change the warning’s line number to 7 from 8. But the warning still stands.

Later that day I saw on my phone an email from the Qt bug tracker saying the status had changed. Less than 24 hours, that’s great turn around time, maybe I was in luck. Nope, not this time, the same assignee still didn’t do a good job . He assigned it to someone in documentation with the following comment (remember my bug report was about a compile time warning, the documentation was a side issue).

The problem was a missing semi-colon after the macro call, but that what the doc example leads you to do.

The example in the docs should be updated to have a semi-colon after the macro call, otherwise following the example produces code that fails to compile and the compiler error message is not very helpful.

Great! I’m glad we could get the documentation squared away, but how about fixing the actual bug I reported!

2. Read the actual contents of the bug report. Don’t just glaze over it and assume you know what I meant. Read it.

I don’t think it is too much to ask that the assignee of the bug actually read the bug report before drawing conclusions. This was a complete disregard for the reported issue. At this point I was annoyed, so I decided to look into the issue myself. Here’s the macro in question:

#define QT_REQUIRE_VERSION(argc, argv, str) { QString s = QString::fromLatin1(str);\
QString sq = QString::fromLatin1(qVersion()); \
if ((sq.section(QChar::fromLatin1('.'),0,0).toInt()<<16)+\
(sq.section(QChar::fromLatin1('.'),1,1).toInt()<<8)+\
sq.section(QChar::fromLatin1('.'),2,2).toInt()<(s.section(QChar::fromLatin1('.'),0,0).toInt()<<16)+\
(s.section(QChar::fromLatin1('.'),1,1).toInt()<<8)+\
s.section(QChar::fromLatin1('.'),2,2).toInt()) { \
if (!qApp){ \
    new QApplication(argc,argv); \
} \
QString s = QApplication::tr("Executable '%1' requires Qt "\
 "%2, found Qt %3.").arg(qAppName()).arg(QString::fromLatin1(\
str)).arg(QString::fromLatin1(qVersion())); QMessageBox::critical(0, QApplication::tr(\
"Incompatible Qt Library Error"), s, QMessageBox::Abort, 0); qFatal(s.toLatin1().data()); }}

First thing to note is that the whole thing is wrapped in brackets, meaning that the semicolon isn’t strictly required and thus a complete non-issue. Sure there is the subtle issues involving things like this:

if (<condition>) MACRO(a); else foo(a);

But this macro is meant to be used unconditionally as the first statement in the program, nothing major. Personally I’d recommend wrapping the macro in a do {/*…*/ } while(0), but that’s a different discussion. Anyway, did you catch the source of the warning? Took me a bit but the issue is the very last statement:

qFatal(s.toLatin1().data());

Basically, they’ve constructed a QString, getting the equivalent c-string and passing that to qFatal, seems harmless enough. Let’s look at the definition for qFatal:

Q_CORE_EXPORT void qFatal(const char *, ...) /* print fatal message and exit */
#if defined(Q_CC_GNU) && !defined(__INSURE__)
    __attribute__ ((format (printf, 1, 2)))
#endif
;

Aha! qFatal is a printf style function, it even has the printf attribute when compiled with gcc. Which basically means we can expect that values passed will likely be given to something like vsprintf, followed by an abort call. This function is called qFatal after all. We all know that it is good security practice to always have the first parameter of a printf function be a string literal to prevent format string vulnerability (at least we all should know…), so the warning is well founded.

Out of curiousity, I tried something like this:

QT_REQUIRE_VERSION(argc, argv, "4.7.0%n%n%n%n");

Boom! Segmentation fault (instead of the expected abort). That means that those %n’s are trampling memory, which is simply not good. Fortunately the odds of this macro being passed a value provided by a user is pretty much zero, so there is no practical vulnerability here, but it is still an issue, one which has a simple solution. If we simply change the last statment to look like this:

qFatal("%s", s.toLatin1().data());

Then all of these issues go away. No warning, no format string crashes when you do naughty things, it just works correctly. I’ve added all of these details to my bug report, so far no response. I still love Trolltech and thank them for their wonderful product which I will continue to recommend to people and be a happy user of. But someone over there really just wasn’t doing their job well. Once I got frustrated enough to look, it took me all of 15 minutes to figure out exactly what was causing the warning and come up with a functional solution. Instead my report has been assigned to documentation where it will likely sit for days until someone realizes it doesn’t belong there. Wish me luck, I’ve got my fingers crossed.

I’ve recently discovered that Flash player’s settings system stores settings for every site which references flash player. This is a nice usability feature. It allows your flash related settings to work across browser sessions, even different browsers as long as you remain in the same account. But, the filing system itself is also a history of all flash enabled sites you’ve visited.

On my Linux box, I have the following directory:

~/.macromedia/Flash_Player/macromedia.com/support/flashplayer/sys

I also found a similar directory on my windows 7 box, so this is not just a Linux phenomenon. In this directory is a bunch of directories whose names are things like “#mail.google.com” and “#viddler.com”, each of which has a settings.sol file containing the flash settings for that particular domain. These get created and updated regardless of privacy mode and any history clearing you ask your browser to do.

Fortunately, these files are safe to delete (they’ll just get recreated if needed). If you go to: http://www.macromedia.com/support/documentation/en/flashplayer/help/settings_manager.html, there are settings there that allow you to manage the offline storage settings, but from my quick tests it seems that the folders are created even when you set the allowed storage to 0Kbs.

Obviously this doesn’t give a complete list of all sites that were visited, but lets face it, much of the questionable content on the internet is served by flash.

Yet another reason to use noscript.

All of this works great, except for the fact that graphviz decided to change some of the structures used to represent the layed out graph.

The most annoying issue I found is that the older versions of the graphviz headers did not have any defines for versioning. This means that there is no good way to detect at compile time which version you are trying to compile against. Later one was introduced, which means that you can coarsely determine if you are compiling against a version before or after that macro was introduced. But this isn’t really good enough given than there have been several changes over time before and after this macro was introduced.

So rule #1: If you plan to change anything in your publicly installed headers, ever, have a version macro which is unique for each and every version of structures and API.

Of course, the fun doesn’t stop there, otherwise I wouldn’t have a whole lot to write about here. In addition to that, they decided to make the version macro they did create resolve to a string! This is great and all for displaying the version to the user. But is not easily comparable at compile time at all. The most creative solution I can think of is to have a configure script which will emit run a program  which includes the header and outputs the version if available, then finally the configure script will parse the result and produce a compile time comparable macro. This is some of what the autotools scripts do to produce things like config.h. But is less than fun to deal with.

Rule #2: Make the version macro numeric, and encode things in a way that x >= y always means that x is newer.

Something like this feels ideal.
#define VERSION 0x00010203 /* version 1.2.3 */
Here I am suggesting to use a 32-bit constant for the version, dividing on byte boundaries for trivial decoding and ease of human readability.  There is no need to stick to the tradition versioning schemes either. Lately, I’ve been preferring using the year and release number like this:
#define VERSION 0x00200901 /* version 2009.1 */
Both work equally well. But wait there is more nonsense! The graphviz people decided to use macros to access the members of the primary structures (node, edge, graph). This way they can change the implementation details of these structures but the code using these structures can remain the same. This was a good idea. Unfortunately, they didn’t do it for all of the structures. Only those three. But there are a whole bunch of structures representing text labels, coordinates and other fun stuff which didn’t use this system.

Rule #3: Be consistent with your API.

Finally, they used to have both floating point and non floating point versions of certain data members. At some point they decided to switch to only floating point. The problem is that some things changed type, but not the name. So if I wanted to store the value of one of these, I can’t conveniently. For example:
struct T { point *pos; };
became
struct T { pointf *pos; };
This makes code such as this unfortunately broken across versions.
point *p = x.pos;
If I had a usable version macro, I could make the type depend on the version. Even worse some things changed names and types. For example:
ND_coord_i /* returns "struct point " */
was replaced by:
ND_coord /* returns "struct pointf" */
The former no longer exists. At least I can test for the existence of the ND_coord macros to figure out which to use, but the net result is messier code.

Rule #4: Do everything possible to avoid changing datatypes of structure members and return types of functions. It makes storing the result ugly and convoluted.

One could argue that not all of the structures and functions are part of there public API. After all, it is pretty rare that someone implements there own rendering engine for something as big, complex as graphviz. But these structures are in the headers that every distribution installs into /usr/include/graphviz/.

Rule #5: If it is in a header which gets installed for the user, it is part of the API. If you have internal details you wish to hide use the PIMPL pattern.

So the end of the story is that I’ve got my Qt based graphing widget I sought after. But the deployment is less than fun. Because I distribute sources I need to test that it compiles on all sorts of distributions. All of this could have been so much simpler if they just planned ahead and had a version friendly API.

The main reason for this is that I had no real practical way to display this information in a useful way to users. A list of blocks wouldn’t do, there would simply be to many for anyone to make sense of them. Some other tools have a nice solution though…graphs .

So I figured that it’s time EDB got a graphing plugin of sorts. And that’s where my focus has been lately. Finally after playing with the awesome QGraphicsView API, I have some results that are worth noting.

As you can see, it is looking pretty nice. All of the nodes are draggable so if you don’t like how it is layed out, you can adjust it. Of course it isn’t quite ready yet, otherwise I would have just tossed it into EDB and bumped the version. There are two major things I need to figure out.

1. How big do I make the whole scene? Right now I just go with “very big”, but eventually, I’ll have to figure this out based on the content. This is very dependant on issue #2 though…
2. I have no automatic layout engine. Currently in my test code, I’ve manually placed each node. This is likely to prove very difficult, but I’ll get there (Obviously if any graph experts out there want to help, feel free to email me ).

Beyond that, I’m feeling pretty good about how it works.

You may be wondering why I didn’t just use GraphViz. Well, to be honest, I looked into it. But there didn’t seem to be any nice and simple way to get the results into a Qt widget. As far as I can tell, the API revolves around programatically generating a .dot file, and sending that through one of the graphviz renderer’s. Then taking the output file and finally displaying it. I don’t like all the temporary files and the dependency on external programs. I’m not a fan of “front-ends” in general, otherwise EDB would just be another GDB front end.

Though if I could do a pure library solution, I could work with the temporary files. Obviously I could have missed something in graphviz, if so, let me know .

I hope to have things nice and ready for release soon.

This isn’t a problem with stackoverflow.com, it’s a problem with people who claim things to be the case without actually testing if what they are saying is true. It happens time and time again.

You may be wondering why the title of this blog entry is “Micro-optimization is stupid” when I haven’t even mentioned that yet, well wait no more, here it is. Someone had asked a question which had a simple answer on SO. He was trying to insert a pointer to an object into a list which contained actual objects. I am guessing he is usually a java programmer (this isn’t an attack of java programmers, just that java’s syntax for creating references to objects looks very much like the broken c++ he was trying to use).

Someone took it upon themselves to not only give advice on how to correct this trivial error, but also gave some optimization advice for iterating over the list. He recomended something like this:

for(std::list<Point>::iterator it = l.begin(), end = l.end(); it != end; ++i) {
/* whatever */
}

as apposed to this:

for(std::list<Point>::iterator it = l.begin(); it != l.end(); ++i) {
/* whatever */
}

The reality is that there isn’t a huge amount of difference between the two. The goal of the posters code is to avoid std::list::end() from being called after every iteration of the loop. I felt compelled to point out to the poster that the optimization recomendation was misguided as any sane compiler will nicely optimize that loop so the std::list::end() is only called once. In addition to that std::list::end() is an O(1) operation as mandated by the standard, so even if it were called multiple times, it wouldn’t make a difference. You’re trading one memory fetch for another, no real difference.

Of course he argued, people who don’t know what they are talking about usually do . He started claiming that since the compiler doesn’t know that the size is constant,  it has to recalculate the size each time, and that this extra function call would cause cache misses and such. See the problem here? There is enough techno-babble to sound convincing! But don’t worry folks, I was determined to demonstrate my point.

So I responded with a few corrections. Most importantly,  size() != end(), so even if it did need to recalculate size() each time, the size has nothing to do with it.  In addition, certainly if the loop only uses member functions which are const it certainly can know that the list hasn’t changed.  (he of course retorted that while the end()/size() function is const, it’s result is not, which is just silliness, since end() won’t change the list, if all other functions don’t change the list, well then end() will be the same next iteration!). Finally, I sent him a link to some code which looks like this:


// program 1
#include <list>
#include <iostream>

std::list<int>::iterator end = l1.end();
for(std::list<int>::iterator it = l1.begin(); it != end; ++it) {
printf("%d\n", *it);
}
}


// program 2
#include <list>
#include <iostream>

for(std::list<int>::iterator it = l1.begin(); it != l1.end(); ++it) {
printf("%d\n", *it);
}
}


And asked that he compile them both with g++ -O3 and look at the results. Not just assume what some other jackass told him was correct. I had already done this, and knew that the result on the latest g++ is identical. At this point I had a good feeling, knowing I had prevailed… until he responded with this:

GCC 4.01 with g++ -O3 -S produces different assembly outputs and different binaries with vectors and lists. GCC 4.2 produces identical output for lists but different output for vectors. The OP uses a vector, my comment stands.

Seriously?! This guy just couldn’t admit being even slightly wrong. To be clear, the OP used a list, not a vector. So at this point he was just making things up without taking the time to read. In addition, I checked and the “difference” between the two was more or less cosmetic. Doing the same work in slightly different order (but not repeatedly calling end(), which was the original concern).

Finally, justice came and people started to downvote his answer.  Finally he edited his post to remove the incorrect advice he gave and the downvotes were removed. None of this would have happened if he hadn’t gotten fixated on silly little micro-optimizations. When will people learn?

Tricks like this may have made a difference in the early days of c++, but modern compilers are suprisingly good at optimization. Let them do their job so that you can do your job!

At this point in time, every compitent developer should know that it is best to do things the simplest, most obvious way first. This is better in many ways, most importantly in maintainability. Everyone can agree that simple and obvious code is easier to understand. Then if after profiling you find that a particular segment of code is not performant enough, feel free to go bananas with optimizing the little things in that particular section.

I know this sounds crazy, and I know that the game itself centers around violent and criminal behavior…but hear me out. To give you some background, here’s how the wanted level system works in GTA. You can get stars, each of which on a scale of 0 through 6 which roughly translates to “how pissed are the police.” The more overt your criminal activity, the more stars you get. One or two stars are pretty easy to evade. At three it starts to get hard. Four or more, and I wish you luck getting away alive.

Throughout the game there are plenty of “kill the bad guys, steal their stuff, get away clean” missions. Usually the police will show up shortly after the “steal the stuff” with two stars. Here’s the thing, the moment you do anything violent towards the police, the stars jump up to three or four. This makes escape much more annoying. The path of least resistance is nearly always to just get in a vehicle and get away as fast as possible. Forget shooting the police, just get away and get away fast.

The bottom line of this is that you tend to be much more successful in the game if you just avoid the police instead of trying to brute force your way through them and killing all in your way.

The primary place that comes to mind with trimming the fat is backwards compatibility. One of Windows strengths in the past has been that upgrades usually keep your programs running. But at this point, after all these versions, I think it has become a weakness. Not to fret, I have an idea .

In essence, I think that compatibility should be far more modular and optional. A good way to do this would be to run older programs in something that functions like the UNIX "chroot." Microsoft could create a few new folders, something like "C:\WindowsXP\" and "C:\Windows2K", etc. Each of these would function like a virtual "C:\" drive for each compatibility module. Inside of each of these would be an entirely functional file structure which matches the previous versions of windows. This would be complete with older versions of various dlls so programs can’t tell the difference between the fake XP mode and the original.

A few things such as user document folders could be handled with something that can function like symbolic links. In addition to this, each compatibility mode would probably need to have it’s own registry tree.

Actual installation could be handled a few ways with various levels of compatibility.

1. Have a "Run in Windows XP mode" right click menu similar to "Run as Administrator."
2. Have a new flag in the PE header to indicate newest supported version of Windows. Lack of this flag would imply some common denominator.
3. Have the system detect writes to program folders and registry keys, and provide a user prompt asking which mode to run it in.
4. Have a program database of some sort.

Installation is the toughest part of this system, but I think can be handled. This idea is actually inspired by two things I’ve already seen. Wine and how Apple handled the migration of OS9 to OSX by being able to virtually run OS9 in OSX.

Basically, in essence, what I am talking about is a Win32 port of Wine that would be supported by the system on a kernel level. This would allow the compatibility to be optional which could be a huge saver of resources and space on the system if you intend to be up to date on your programs and only run applications intended for the newer OS. Best part, if you don’t need it, you don’t have it. It can be that simple!

All in all, I can see few downsides to this concept. It’s already in practice on linux systems which run wine. Since Windows XP is an iterative build on top of NT and 2K, the PE flag idea would work well (compatibility mode would be either on or off). In the end you have a virtual Windows XP environment which is sandboxed and let’s face it, who is better equipped to build this system than the designers of the original?

You can improve things a little bit by playing with X resources, in fact on nedit’s site, there is an article about it. This works ok, but on my KDE or Gnome desktops which use modern GUI toolkits it still stands out, it simply isn’t a pretty app. The nedit developers have actually expressed some interest in a port to other toolkits in the FAQs, but feel it is too large of a task, and to be honest, I agree. Rewriting all of that code just to update the toolkit is a bit much to ask for, especially since it functions so well.

What I propose is something different, which I feel is a bit more practical. Rewrite older UI toolkits on top of newer ones! This has several benefits. First and foremost, it allows applications to get a face lift simply by installing a new library, you probably wouldn’t even need to recompile your apps! Related to this, all of the “common dialogs” will be updated as well. This will provide a more cohesive user experience. More subtly, there would be a lot more code reuse. If I write my hypothetical “GTKTiff” library which implements a binary and source compatible API for Motif on top of GTK, all of the core functionality is already done. All of these toolkits implement the same core features; buttons, frames, tabs, file dialogs, etc. So this “GTKTiff” would likely be smaller and more maintainable than a full “from the ground up” implementation such as OpenMotif or LessTiff.

For the more traditionalist users, well pretty much every major toolkit provides theming, and there could easily be a “Motif Classic” theme.

Don’t get me wrong, I think that the OpenMotif and LessTiff projects are wonderful, they let me run my favorite editor on GNU/Linux after all! But I do think that there is an opportunity to consolidate some code and provide a better user experience. The current state of the UNIX or GNU/Linux desktop is not just a mish-mash of applications, but instead is a collection of applications designed to share a look and feel in order to provide a good user experience.

Having many GUI APIs has its pros and cons. Operating Systems such as GNU/Linux are founded on the concept of choice, which is a wonderful thing. I just think that older APIs shouldn’t lock applications in the “stone age.”

I do a lot of reverse engineering work and thus the lack of anything like Ollydbg spawned off my EDB project. It’s a debugger designed to focus on applications at a machine code level. This project is coming along nicely but there is one thing that I really wish I could change…ptrace sucks, and it sucks a lot.

First of all, it has no inherent support for threads. This is a huge problem as many modern applications are multi threaded. Instead, since Linux treats threads as independent processes which happen to share the the majority of there address spaces, you are supposed to ptrace each thread individually. So, you need to attach to all “pids” found in “/proc/<pid>/task/”. On top of this, you have to set an option in ptrace to attach to new threads as they are created with the clone system call. It is entirely undocumented whether you need to do this on a per process basis or a per-thread basis. Finally, you need track threads exiting so you know to stop looking for events on those. That’s an awful lot of effort just to be able to debug threads. By the way, the information necessary to know which bits of the status code tell you if it was a clone event (new thread) is also entirely undocumented.

Did i mention the potential race condition with attaching to all threads in the /proc/<pid>/task/ directory? Since the threads could be spawning more threads while you are enumerating the directories. Threads could even exit during this time. So you have to loop continually trying to attach until the you are sure the thread count is stabilized and all are attached to. The only saving grace here is that attaching stops a thread so it is possible to get them all if you think it through.

Next, I wish that the PTRACE_PEEK* and PTRACE_POKE* request types had support for non-word width granularity. It makes setting a breakpoint more annoying than it has to be since you really only want to read/write a single byte in that case. Not only that, but reading/writing from the edges of region boundaries is equally annoying. A much better interface would have been similar to the file API where you can specify and address and a length. In addition to this, you need to pay careful attention to the various gotchas due to the fact that the return value is both an error code and a result. So if it returns (long)-1, then you need to check errno just to make sure that it isn’t an error.

The usage of wait for debug events is just awkward. It works great for single threaded command line debuggers like gdb, but for a GUI, where you want things to be interactive while the debugger is waiting for the next event, it is a disaster. Sure you can use a separate thread to capture events and deliver them to the GUI, but then you have issues properly shutting down that thread, since it will pretty much always be blocked! Also, wait has no timeout, so if you aren’t careful it is possible to get hung forever waiting for an even that will never happen. There is SIGCHLD, which sounds promising at first, but the fun part is that without sigprocmask trickery, you can’t predict which of your threads will get the signal. Grr!

Finally, there is lots of information that would be better suited being part of the debugging API. A great example of this is x86 segments. This really should be in the user area. You can get the segment values from the user area or even a PTRACE_GETREGS request. But the segment values are nearly worthless without being able to look at things like the segment base and limits. I understand not all platforms have this data, and x86-64 has much less usage of segments, but that’s why it should be in the user area.

A better API would first of all, be at a process level. I don’t care how it works under the hood, I want to attach to “processes.” There should also be a function to enumerate threads, this would only be valid when the process is stopped. This way you could get/set the context of each thread by passing a tid. Just these changes would make things much easier.

Overall, the user space API provided by ptrace could use a large overhaul. I understand the desire to be consistent with other unix’s debugging APIs, but this should not get in the way of making something usable.

utrace sounds ok, but as far as I know, it is designed to be kernel level changes. In fact, it appears there are plans to have ptrace implemented on top of utrace in the future. That’s great and all, but the user space API needs an update! I can only hope that utrace bring along a new user space API as well.