I know this sounds crazy, and I know that the game itself centers around violent and criminal behavior…but hear me out. To give you some background, here’s how the wanted level system works in GTA. You can get stars, each of which on a scale of 0 through 6 which roughly translates to "how pissed are the police." The more overtly your criminal activity, the more stars you get. One or two stars are pretty easy to evade. At three it starts to get hard. Four or more and I wish you luck getting away alive.

Throughout the game there are plenty of "kill the bad guys, steal their stuff, get away clean" missions. Usually the police will show up shortly after the "steal the stuff" with two stars. Here’s the thing, the moment you do anything violent towards the police, the stars jump up the three or four. This makes escape much more annoying. The path of least resistance is pretty much always to just get in a vehicle and get away as fast as possible. Forget shooting the police, just get away and get away fast.

The bottom line of this is that your tend to be much more successful in the game if you just avoid the police instead of trying to brute force your way through them and killing all in your way.

The primary place that comes to mind with trimming the fat is backwards compatibility. One of Windows strengths in the past has been that upgrades usually keep your programs running. But at this point, after all these versions, I think it has become a weakness. Not to fret, I have an idea :).

In essence, I think that compatibility should be far more modular and optional. A good way to do this would be to run older programs in something that functions like the UNIX "chroot." Microsoft could create a few new folders, something like "C:\WindowsXP\" and "C:\Windows2K", etc. Each of these would function like a virtual "C:\" drive for each compatibility module. Inside of each of these would be an entirely functional file structure which matches the previous versions of windows. This would be complete with older versions of various dlls so programs can’t tell the difference between the fake XP mode and the original.

A few things such as user document folders could be handled with something that can function like symbolic links. In addition to this, each compatibility mode would probably need to have it’s own registry tree.

Actual installation could be handled a few ways with various levels of compatibility.

1. Have a "Run in Windows XP mode" right click menu similar to "Run as Administrator."
2. Have a new flag in the PE header to indicate newest supported version of Windows. Lack of this flag would imply some common denominator.
3. Have the system detect writes to program folders and registry keys, and provide a user prompt asking which mode to run it in.
4. Have a program database of some sort.

Installation is the toughest part of this system, but I think can be handled. This idea is actually inspired by two things I’ve already seen. Wine and how Apple handled the migration of OS9 to OSX by being able to virtually run OS9 in OSX.

Basically, in essence, what I am talking about is a Win32 port of Wine that would be supported by the system on a kernel level. This would allow the compatibility to be optional which could be a huge saver of resources and space on the system if you intend to be up to date on your programs and only run applications intended for the newer OS. Best part, if you don’t need it, you don’t have it. It can be that simple!

All in all, I can see few downsides to this concept. It’s already in practice on linux systems which run wine. Since Windows XP is an iterative build on top of NT and 2K, the PE flag idea would work well (compatibility mode would be either on or off). In the end you have a virtual Windows XP environment which is sandboxed and let’s face it, who is better equipped to build this system than the designers of the original?

You can improve things a little bit by playing with X resources, in fact on nedit’s site, there is an article about it. This works ok, but on my KDE or Gnome desktops which use modern GUI toolkits it still stands out, it simply isn’t a pretty app. The nedit developers have actually expressed some interest in a port to other toolkits in the FAQs, but feel it is too large of a task, and to be honest, I agree. Rewriting all of that code just to update the toolkit is a bit much to ask for, especially since it functions so well.

What I propose is something different, which I feel is a bit more practical. Rewrite older UI toolkits on top of newer ones! This has several benefits. First and foremost, it allows applications to get a face lift simply by installing a new library, you probably wouldn’t even need to recompile your apps! Related to this, all of the “common dialogs” will be updated as well. This will provide a more cohesive user experience. More subtly, there would be a lot more code reuse. If I write my hypothetical “GTKTiff” library which implements a binary and source compatible API for Motif on top of GTK, all of the core functionality is already done. All of these toolkits implement the same core features; buttons, frames, tabs, file dialogs, etc. So this “GTKTiff” would likely be smaller and more maintainable than a full “from the ground up” implementation such as OpenMotif or LessTiff.

For the more traditionalist users, well pretty much every major toolkit provides theming, and there could easily be a “Motif Classic” theme.

Don’t get me wrong, I think that the OpenMotif and LessTiff projects are wonderful, they let me run my favorite editor on GNU/Linux after all! But I do think that there is an opportunity to consolidate some code and provide a better user experience. The current state of the UNIX or GNU/Linux desktop is not just a mish-mash of applications, but instead is a collection of applications designed to share a look and feel in order to provide a good user experience.

Having many GUI APIs has its pros and cons. Operating Systems such as GNU/Linux are founded on the concept of choice, which is a wonderful thing. I just think that older APIs shouldn’t lock applications in the “stone age.”

I do a lot of reverse engineering work and thus the lack of anything like Ollydbg spawned off my EDB project. It’s a debugger designed to focus on applications at a machine code level. This project is coming along nicely but there is one thing that I really wish I could change…ptrace sucks, and it sucks a lot.

First of all, it has no inherent support for threads. This is a huge problem as many modern applications are multi threaded. Instead, since Linux treats threads as independent processes which happen to share the the majority of there address spaces, you are supposed to ptrace each thread individually. So, you need to attach to all "pids" found in "/proc/<pid>/task/". On top of this, you have to set an option in ptrace to attach to new threads as they are created with the clone system call. It is entirely undocumented whether you need to do this on a per process basis or a per-thread basis. Finally, you need track threads exiting so you know to stop looking for events on those. That’s an awful lot of effort just to be able to debug threads. By the way, the information necessary to know which bits of the status code tell you if it was a clone event (new thread) is also entirely undocumented.

Did i mention the potential race condition with attaching to all threads in the /proc/<pid>/task/ directory? Since the threads could be spawning more threads while you are enumerating the directories. Threads could even exit during this time. So you have to loop continually trying to attach until the you are sure the thread count is stabilized and all are attached to. The only saving grace here is that attaching stops a thread so it is possible to get them all if you think it through.

Next, I wish that the PTRACE_PEEK* and PTRACE_POKE* request types had support for non-word width granularity. It makes setting a breakpoint more annoying than it has to be since you really only want to read/write a single byte in that case. Not only that, but reading/writing from the edges of region boundaries is equally annoying. A much better interface would have been similar to the file API where you can specify and address and a length. In addition to this, you need to pay careful attention to the various gotchas due to the fact that the return value is both an error code and a result. So if it returns (long)-1, then you need to check errno just to make sure that it isn’t an error.

The usage of wait for debug events is just awkward. It works great for single threaded command line debuggers like gdb, but for a GUI, where you want things to be interactive while the debugger is waiting for the next event, it is a disaster. Sure you can use a separate thread to capture events and deliver them to the GUI, but then you have issues properly shutting down that thread, since it will pretty much always be blocked! Also, wait has no timeout, so if you aren’t careful it is possible to get hung forever waiting for an even that will never happen. There is SIGCHLD, which sounds promising at first, but the fun part is that without sigprocmask trickery, you can’t predict which of your threads will get the signal. Grr!

Finally, there is lots of information that would be better suited being part of the debugging API. A great example of this is x86 segments. This really should be in the user area. You can get the segment values from the user area or even a PTRACE_GETREGS request. But the segment values are nearly worthless without being able to look at things like the segment base and limits. I understand not all platforms have this data, and x86-64 has much less usage of segments, but that’s why it should be in the user area.

A better API would first of all, be at a process level. I don’t care how it works under the hood, I want to attach to "processes." There should also be a function to enumerate threads, this would only be valid when the process is stopped. This way you could get/set the context of each thread by passing a tid. Just these changes would make things much easier.

Overall, the user space API provided by ptrace could use a large overhaul. I understand the desire to be consistent with other unix’s debugging APIs, but this should not get in the way of making something usable.

utrace sounds ok, but as far as I know, it is designed to be kernel level changes. In fact, it appears there are plans to have ptrace implemented on top of utrace in the future. That’s great and all, but the user space API needs an update! I can only hope that utrace bring along a new user space API as well.

I’ll start out with what I don’t like in order to be fair. I really am annoyed that there is no longer a “navigate up” button in windows explorer. I find myself missing it all the time. I understand that they are trying to go with the “breadcrumb” navigation scheme, but with long path names, it just takes more work.

Also, Vista tends to be a slug on a lot of hardware, particularly if you are short on ram. This is likely a combination of a few things, the aggressive ram usage for caching, the pretty GUI, etc. You just don’t get the same performance out of that Pentium4 with 1GB of RAM as you did with XP. However, on brand new hardware loaded with RAM I find it more responsive than XP. The bottom line here is that there seems to be a tipping point where given enough room to play, Vista starts to feel faster.

Backwards compatibility is a tough thing here. Too compatibility and you lose the ability to add new useful features. I have only encountered one program which I simply could not get to work in Vista, and it was an older game. Programs I actually care about work just fine. I like that fact that running as a less privileged user is the default. However, I would have liked to see the option to run old applications in a more compatible sandbox of sorts. I am thinking of something where you right click and choose “Install in compatibility mode” and it would copy a minimal directory tree structure into a special directory and run the program “chrooted” there. It would have it’s own registry, it’s own filesystem, etc. Under the hood file system hardlinks could reduce the wasted filesystem usage. Almost how wine works with a virtual drive c, so could a compatibility mode.
Finally, I don’t like how some things are different just for the sake of being different. I used to be able to get at my network settings by just right clicking on an icon in the system tray and choosing “Status.” Now that information is hidden behind at least 2 extra dialogs. I understand that they are trying to provide a unified look at networking even on systems with many network cards…but when I just want to know the IP of an interface, clicking through several new (and confusing) dialogs or opening a command prompt and typing ipconfig is simply slower. I nice middle ground would have been to have right click have a submenu for each interface each of which would have a status menu. This would be simple, functional, and fast.

On to what I like

First of all, as everyone points out, it’s pretty…very pretty. The eye candy is done well and is usually functional from a usability perspective (the 3D flip is an exception I can think of) . The default look of XP just looks like a Fisher Price toy to me. Of course if you are running Windows Vista Basic edition then you are probably going to be missing out on this. But that’s OK, but it’s not what makes the OS good

Vista is also much more stable on average than XP. My new system had some issues, but it turned out to be faulty hardware. After Dell replaced my motherboard and CPU, haven’t had a single crash.

The most useful addition I found is the way that search/running is just a single button push away. You simply hit the windows key and start typing. Immediately the start menu will filter out the entries which don’t match the keywords. In addition to that, it’ll start displaying files it found. Finally if you hit enter when done it will try to execute the search string as a command if found. So the run dialog, desktop search and start menu have all be integrated in a way which work really well. I know a lot of people are going to say “well OSX had something like this for a while now.” This is true, but just because you didn’t do it first, doesn’t mean it isn’t a good idea, it just isn’t innovative. Someone was first to offer airbags in cars, I don’t see people giving the other car companies a hard time because they also offer this feature.

The software compatibility wizard also is very nice. Whenever Vista detects that an installer tried to do something that may or may not play nice with the limited privilege concept, it will ask “did that work correctly.” If you say no, it will try again with more backwards compatible settings and potentially store information about it in it’s issue manager for later resolution.

Many features exist under the hood where people don’t see any visible difference, mostly in the name of security. There is Address Space Layout Randomization (ASLR), which is very effective in reducing the severity of buffer overflows because it makes finding a viable “offset” much more difficult. It also makes return-into-libc attacks a lot less likely to succeed. The net result here is that while code execution is technically possible, it is more likely going to be a DoS which is bad, but a lot less bad. The user space heap has been hardened beyond what was done in XP SP2. This once again tends to turn what was arbitrary execution into a DoS instead. These two things combined with the non-executable stack capabilities effectively make standard buffer overflow attacks much harder to pull off. In addition to these direct approaches, they have also made many improvements to the compilers used. Modern versions of Visual Studio will warn the developers about the usage of potentially unsafe functions. Not only that, but Visual C++ by default will disable the “%n” functionality of the printf family of functions. Once again this turns a vulnerability which would have the possibility of arbitrary code execution into a DoS. In the format string case, there is no longer any possibility of execution using existing techniques because there is no way to corrupt memory.

Finally the much debated UAC. People seem to hate this, but you know what, it’s a good feature. Coming from the UNIX world where I have to type “sudo <command>” whenever I need to do something priviledged, this was a welcomed change. I understand that it is inconvenient, but the fact of the matter is that most users can’t be trusted to be administrator all of the time, particularly in a corporate environment.I feel that it is in the business world that UAC will shine. Simply set the UAC to require the system password instead of just a prompt and the vast majority of Trojan horse installs will disappear.

The real problem is that both users and developers are too used to being administrator, this has created horribly bad habits. Microsoft has tried to meet halfway here by virtualizing reads and writes to files in what are now protected locations to a local store. This is done seamlessly and improves backwards compatibility.

In the end, I feel that Vista is an improvement over XP, if you have a new machine. I would not recommend upgrading anything with less than 2GB of RAM to it. Certainly there is no need to upgrade from XP, but if you have the horse power it runs like a champ.

Here’s the thing, in the x86 architecture, there are a handful of instructions that make virtualization hard, 17 of them to be exact. What I mean by this is that these instructions don’t throw an exception when run in user mode (ring 3), but either act differently than in system mode (ring 0) or expose sensitive information.

VMware and other virtualization technologies on x86 address this by using dynamic translation techniques. This isn’t a new idea, if you are from the emulation world then this is just a combination of an emulator and a just in time compiler. People writing emulators tend to call this technique “dynamic recompilation.” This way you can put a break on any instruction you like, and you can run the code at nearly native speed.

However, to get the real speed gains, user mode code is run unchanged, natively. This means that if user mode executes a sensitive instruction that doesn’t throw an exception, no way to intercept it. Software can leverage this fact to find out information that you would normally want hidden from the guest OS.

A classic example of this is the “redpill” code. What it basically does is call the sidt instruction from user mode. In kernel mode, VMware would trap this, emulate the desired result and continue with the OS none-the-wiser. However, when executed in user mode, there is no trap, it just works. In fact, it should provide the caller with the real IDT of the host system. Aside from the fact that if an attacker ever figures out a way to write to memory in the host this would be useful information, the discrepancy between the two results is a clear cut sign of virtualization. This is just one of the many instructions that can be used to detect virtualization.

So what’s the solution? According to both Intel and AMD the solution is a new mode of operation which is more privileged than ring 0. A much more simple approach would be to add a new flag to a system register that when set, makes all sensitive operations cause an exception in user mode. This would allow the guest kernel code to run unmodified in user mode. No more dynamic translation, making the whole virtualization concept a lot simpler.

If this is done, there are only a few challenges left. The first would be protecting the guest OS code from the guest user code. Since they are both running at user mode, page level protection don’t work. If we are looking at x86 only and not x86-64, then segmentation is a viable solution. (I did hear that certain models of the AMD bring back segmentation limits). The rest of the concerns are fairly usual when it comes to virtualization such as properly emulating devices and such. For these the usual strategies of page fault trapping and I/O port trapping should work well as they have in the past.

I’m not the only one to think of this, the folks at pagetable.com came to a very similar conclusion. The question is, why wouldn’t Intel/AMD think of this, they are pretty smart. And even so, if I worked for VMware I’d probably push for a technology like this.

To my surprise Windows only saw 3GB of RAM! Since I do a little hobby OS development, I immediately had an idea of what could cause this. I jumped to the conclusion that PAE was not properly enabled, and started sifting through different sites to find the proper way to enable PAE. Of course if this were the solution, I wouldn’t really have much to say here… It turns out that PAE is in fact enabled by default on Windows Vista if your system supports it, after all, you need to in order to make use of the no-execute bit on x86.

At this point, I should probably explain what PAE is and why it should make 4GB available without a problem. Basically there are two ways to look at memory on an x86 system. There is “physical memory,” which is your system memory start to finish in order. Note, this includes memory mapped devices. This is how the memory would look if you used an OS which does not use paging. Then there is “virtual memory.” The idea here is that your physical memory is viewed as blocks (”pages”) that can be mapped to any location you wish in your virtual memory space. For example, as the OS writer, I could ask the system to map the 4K of memory at physical address 0×11223000 to address 0×00000000. In this case, any reads and writes that programs do in first 4K of memory will occur at the physical address associated with it. Virtual memory is also what allows modern OSes to protect one application from another. It does this by switching the virtual memory layout during task switch so that each process has a unique view of what memory looks like.

The problem is that memory mapped devices occupy physical address space as well, so if you have a 512MB video card, then that’s half a gig of of that 4GB physical memory space which can’t be RAM. Here’s where PAE comes in. Before PAE, your physical memory was limited to 4GB and your virtual memory was limited to 4GB (per process) . PAE changes this to be 64GB (36-bits) of physical address while keeping the 4GB of virtual addresses. Sure, no single process can use more than 4GB at a time, but all the RAM could be put to use. There is one catch though, your memory mapped devices must still be below the 4GB mark because they use 32-bit addressing when doing DMA. So the natural solution is to relocate the RAM that is displaced by devices to above the 4GB mark. Most modern motherboards support this.

It turns out that for “compatibility reasons” Microsoft has opted to simply ignore any RAM it sees above 4GB. Some people are convinced it is a hardware problem, saying:

To be perfectly clear, this isn’t a Windows problem– it’s an x86 hardware problem. The memory hole is quite literally invisible to the CPU, no matter what 32-bit operating system you choose. The following diagram from Intel illustrates just where the memory hole is:

This simply isn’t the case (Sorry Jeff, I love your blog BTW). It is a design choice by the windows engineers to take the easy way out. A perfectly viable solution is to divide your memory up into types, namely “suitable for DMA” and “not suitable for DMA.” I know this works, because Linux does it. In fact, here’s a screen shot of my shiny new dell using all 4GB of my RAM.

This isn’t a 64-bit build, it’s 32-bit (arch reports “i686″ not “x86-64″) with the 64GB support config option used (which basically means enable PAE). There are plenty of people saying online that all 32-bit operating systems have this problem. This isn’t true.

Jeff gets it right with this statement though:

As far as 32-bit Vista is concerned, the world ends at 4,096 megabytes. That’s it. That’s all there is. No más.

Addressing more than 4 GB of memory is possible in a 32-bit operating system, but it takes nasty hardware hacks like 36-bit PAE extensions in the CPU, together with nasty software hacks like the AWE API. Unless the application is specifically coded to be take advantage of these hacks, it’s confined to 4 GB. Well, actually, it’s stuck with even less– 2 GB or 3 GB of virtual address space, at least on Windows.

Except that PAE isn’t a nasty hack by any stretch, in fact, Vista uses it already as previously mentioned. User space software doesn’t need to be specially programed to take advantage of the extra memory since it will only see 4GB at a time anyway (minus kernel land). Also AWE-API is used to address the 4GB of virtual memory limitation not the physical limitations! What AWE does is allow an application to selectively map physical RAM locations to user space virtual locations. A program can thus can access much more than the 2GB user space that windows will give it by default, just not all at the same time.

Microsoft of course does support upwards of 4GB on it’s x86-64 builds, that’s all fine and dandy, but my Dell didn’t come with that. And to my knowledge Dell (as a company, not the hardware) doesn’t support x86-64 officially yet. So maybe I’ll take that up with them and demand a 64-bit copy.

All in all, it’s a little lame that Windows doesn’t support all the RAM that it could on 32-bit builds. It really wouldn’t be hard, but it would likely require that driver writers start passing a flag to the allocator specifying that the memory be OK for DMA. I see that this is a problem since there are simply tons of drivers. But at the least, the extra RAM could have been used for things where DMA is clearly not involved (pretty much all user space uses since only drivers should be doing DMA). Also Microsoft could have done something clever like add a new flag to the driver’s PE header which when not present would make the allocator only return addresses below 4GB and if set would allow the driver to use a more robust allocator API.

I hope this shed some light on the subject, because there is unfortunately a lot of mis-information out there.

Evan