Hard to C

Reordering Arguments

Mon, 07 Nov 2016 22:04:16 +0100

When a C program runs, it usually receives any command-line arguments through the parameters argc (argument count) and argv (argument vector). It is up to the program to interpret the contents of this array of strings.

There are many ways to do this, one of the simpler solutions is the getopt function (in the following, getopt will refer to both getopt and getopt_long). One extension some getopt implementations offer, is that they reorder the contents of argv as they process it, resulting in an array where all the options appear first followed by the nonoptions.

This reordering partitions the array. And we want a stable partition, so the relative order of all the options, and of all the nonoptions is the same.

Last year I released parg, a simple parser for argv that works similarly to getopt. It has a separate function, parg_reorder(), that performs this reordering. I used a simple algorithm for this – allocate a temporary array the size of argv, parse the arguments, and fill in the options from the start and nonoptions from the end of the new array. Then copy them back, reversing the nonoptions to their original order in the process. In hindsight, I could have moved the options into their final place in argv right away.

This runs in $O(n)$ time, but also requires $O(n)$ additional space. The space requirement could be a problem (for instance on embedded systems), so let’s see if we can do better.

The code examples in the following are pseudo-code, and glance over some details. Please see the examples on GitHub for working C++ code.

We note that if we already partitioned the two halves of an array, $L_{1}R_{1}L_{2}R_{2}$ , we can compute the partition of both by swapping $R_{1}$ and $L_{2}$ . Swapping adjacent blocks in an array is sometimes called rotating, and can be done in linear time, for instance using reversal (observing that $LR = (R^{R}L^{R})^{R}$ ).

So we can use divide and conquer. With a recursive function that splits the array at the middle, this runs in $O(n \log n)$ time and requires $O(\log n)$ stack space.

stable_partition_recursive(first, n, pred)
{
	if (n == 1) {
		return pred(first) ? first + 1 : first;
	}

	left   = stable_partition_recursive(first, n / 2, pred);
	middle = first + n / 2;
	right  = stable_partition_recursive(middle, n - n / 2, pred);

	rotate(left, middle, right);

	return left + (right - middle);
}

Much better, but ideally we would like to use only constant extra space. To achieve this, we can use the same technique as in bottom-up merge sort. We first process the array in blocks of size 2, then again in blocks of size 4 (whose two halves of size 2 are already partitioned), and so on, until we process the entire array as one block.

stable_partition_bottomup(first, n, pred)
{
	for (width = 1; width < n; width *= 2) {
		next = first;

		for (i = 0; i < n; i += 2 * width) {
			limit = n - i < 2 * width ? n - i : 2 * width;

			if (limit > width) {
				middle = next + width;
				left   = std::find_if_not(next, middle, pred);
				next   = middle + width
				right  = std::find_if_not(middle, next, pred);

				rotate(left, middle, right);
			}
		}
	}

	return left + (right - middle);
}

This runs in $O(n \log n)$ time and uses constant extra space.

In a sense, the price we pay to avoid the recursion is that we do not remember the partition points of the two halves, and need to find them again. This means we use the predicate $O(n \log n)$ times instead of $O(n)$ .

However, since the two halves are already partitioned, we can use binary search to find the partition points. This reduces the number of predicate applications to $O(n)$ (since $\sum_{k=1}^{\infty}\frac{n}{2^k}\log 2^k = 2n$ ), which means we have the same time complexities as the recursive algorithm, but using only constant extra space.

stable_partition_bsearch(first, n, pred)
{
	for (width = 1; width < n; width *= 2) {
		next = first;

		for (i = 0; i < n; i += 2 * width) {
			limit = n - i < 2 * width ? n - i : 2 * width;

			if (limit > width) {
				middle = next + width;
				left   = std::partition_point(next, middle, pred);
				next   = middle + width
				right  = std::partition_point(middle, next, pred);

				rotate(left, middle, right);
			}
		}
	}

	return left + (right - middle);
}

If the array is almost partitioned, these algorithms still go through every step. We can do something similar to natural merge sort, and repeatedly look for the next place where a rotate is needed, stopping if we find none. We must be careful though, if we simply look for $RL$ areas and rotate them, we get quadratic time on alternating patterns. Instead we can look for $L_{1}R_{1}L_{2}R_{2}$ and rotate the middle two, just like above.

stable_partition_natural(first, last, pred)
{
	do {
		next = first;
		change = false;

		do {
			left   = std::find_if_not(next, last, pred);
			middle = std::find_if(left, last, pred);
			right  = std::find_if_not(middle, last, pred);
			next   = std::find_if(right, last, pred);

			if (left != middle && middle != right) {
				rotate(left, middle, right);
				change = true;
			}
		} while (next != last);
	} while (change);
}

At best this runs in $O(n)$ , while the worst-case complexity is the same as the bottom-up version. Since the widths are no longer fixed size, we have to perform a linear search for the partition points, so the number of predicate applications is back to $O(n \log n)$ . Also, we have to use the predicate to find the middle and next starting point, so we may use more predicate applications than the bottom-up version.

There is an algorithm that can perform stable partitioning in $O(n)$ using constant extra space (PDF), but it is somewhat more involved, and not practical given the constraints of the specific task.

Let us return to the problem of reordering arguments. One issue here is that we cannot tell if any given element is an option, an option argument, or a nonoption, without parsing the entire array up to that point (i.e. we cannot assume random access).

This is because any given element could be preceded by an option that uses it as its option argument (looking at the previous element is not enough, since that might be an option argument instead of an option). Or there could be a -- somewhere, which means the remaining elements are nonoptions (unless that -- is in fact the option argument of a preceding option).

This makes the natural algorithm a good fit, since it applies the predicate linearly from left to right in each loop.

So how does this all compare to getopt implementations? The two I checked perform the reordering in each call by scanning over any nonoptions from the current position, and then rotating them to the end of the array. This effectively builds the array of nonoptions at the end, while moving down the part that has not yet been processed.

This requires constant extra space, but takes worst-case $O(n^{2})$ time. While this could theoretically be used in an algorithmic complexity attack, most systems limit the size of the command-line arguments in some way. As an example of the worst-case behavior, the following line runs ls with 200,000 nonoptions (redirecting the error messages for missing file), and takes about 0.3 seconds (Fedora 24 running in a VM):

time ls `perl -e "print 'a a ' x 100000"` 2>/dev/null

Whereas this line runs ls with the same number of arguments, but alternating nonoptions and options (the -a option enables showing files that start with a period, and has no effect in this case), and takes 11 seconds:

time ls `perl -e "print 'a -a ' x 100000"` 2>/dev/null

Moving to Jekyll

Wed, 11 May 2016 11:56:16 +0200

I am moving this blog to Jekyll. While WordPress has worked quite well for the past seven years, I feel it is too much effort for a site that might as well be static.

This is pretty much the stock Jekyll generated site using the Solarized light color theme, Source fonts, and a few icons from Font Awesome. Source is here.

One sad thing is that support for comments is gone. I could add something like Disqus, but quite frankly I don’t think it is worth it. You can always send me an email, or find me in #donationcoder on freenode.

Not Quite Monokai

Wed, 03 Sep 2014 09:24:34 +0200

TextMate is a popular text editor for OS X. Since the first release 10 years ago, a lot of people have contributed color themes, many of which have been ported to other editors.

If you are using a Windows or Linux text editor with such a conversion, you might be looking at something slightly different than what the designer intended.

This quote from the Wikipedia article on RGB sums up the problem:

RGB is a device-dependent color model: different devices detect or reproduce a given RGB value differently, since the color elements (such as phosphors or dyes) and their response to the individual R, G, and B levels vary from manufacturer to manufacturer, or even in the same device over time. Thus an RGB value does not define the same color across devices without some kind of color management.

So, an RGB color like #272822 does not necessarily look the same on all computers, at least not without color management. When we combine a color model like RGB with a color space, we get device independent colors, which can be used to produce consistent results on calibrated equipment.

A simple way to think of this is that an RGB color contains the relative level of the red component, but does not specify what red means. An RGB color space describes what exactly red, green, blue, and white are.

The color spaces used on Windows and Mac OS computers more or less reflect the hardware each platform was using at the time. This had the advantage that existing media matched the default profiles.

HP and Microsoft created the sRGB color space, which was later adopted as the default color space for the web, and used in many consumer products.

On the Apple side, Adobe created Apple RGB for use in their software. With Mac OS X, Apple introduced a number of profiles, and Generic RGB was the default assumed for untagged media up to OS X 10.7 Lion, 2011.

The color differences between sRGB and Generic RGB are relatively subtle, the biggest difference is gamma correction, which used a gamma value of 1.8 on Apple computers, while Windows uses 2.2.

This means that colors from OS X Generic RGB will look darker on Windows, and colors from Windows sRGB will look light and washed out on OS X. This has long been a known issue, and today it is much less of a problem due to color management, browser support for images tagged with color profiles, and Apple changing the default gamma from 1.8 to 2.2 in OS X 10.6 Snow Leopard, 2009.

So how does this all relate to TextMate themes?

TextMate was first released in 2004, and a number of popular color themes were developed for it on Mac OS X, with the Generic RGB default color profile and a gamma of 1.8.

These themes have since been ported to other editors on Windows and Linux (some even support the same color theme file format that TextMate uses).

The colors are stored as RGB hex codes, like #272822, which means we will not necessarily get the same color on Windows as on Mac OS. If the RGB color values are used directly on Windows, the themes will look darker. In fact, even OS X after the gamma change may render the colors darker outside TextMate.

The same problem has also occurred the other way around, where for instance the Solarized palette was developed in sRGB, and using the RGB values directly in a TextMate theme would result in lighter colors.

As an example, let us look at the Monokai color theme by Wimer Hazenberg. It was made in 2006 for TextMate, so we assume it was designed under the default Generic RGB profile, with a gamma of 1.8.

Monokai has been ported to many editors, including Atom, Notepad++, and Sublime Text. In these three editors (and presumably many others), the RGB values from the original TextMate theme are used directly, which means they all appear darker on Windows.

In 2013 TextMate added a key to the theme files that allows choosing sRGB color space (without it, colors should be treated as Generic RGB). This is great for new themes, but you cannot simply put a sRGB tag on old themes without converting the colors into sRGB.

Converting from one RGB color space to another involves removing the source gamma, doing a linear transformation, and applying the destination gamma. If you are interested in more details on color spaces and the math involved in converting between them, Lindbloom and Pascale (PDF) are both good resources.

I wrote a Python script to convert the RGB values in a tmTheme file from Generic RGB to sRGB. Here is a screenshot comparing Monokai original and sRGB in Sublime Text 3 on Windows:

You may object that the screenshot on the Monokai blog matches the dark appearance of the theme on Windows, but this is because the image is a GIF file without embedded color profile, and thus suffers the same fate as the theme, when its RGB values are assumed to be sRGB by your browser.

Time Difference

Sat, 29 Dec 2012 20:09:04 +0100

This post is based on a discussion about Progress Bars of Life, where I was foolish enough to claim that printing a text string representing the difference between two times could not be that hard in C. It is not hard, but turned out not to be entirely trivial either.

The problem we will consider is; given two tm structs, compute the difference in time between them, in such a way that we can easily format a string that gives a textual representation of it. We want years, months, days, hours, minutes, seconds.

The first idea you might get is to use difftime() to get the difference in seconds between the two times, and then compute the quantities we want by simple arithmetic. So,

start_time = mktime(&start);
end_time = mktime(&end);
secdiff = difftime(end_time, start_time);

years   = secdiff / SEC_IN_YEAR;
months  = (secdiff % SEC_IN_YEAR) / SEC_IN_MONTH;
days    = (secdiff % SEC_IN_MONTH) / SEC_IN_DAY;
hours   = (secdiff % SEC_IN_DAY) / SEC_IN_HOUR;
minutes = (secdiff % SEC_IN_HOUR) / SEC_IN_MINUTE;
seconds = (secdiff % SEC_IN_MINUTE);

You often see something like this in timing code – it works great for showing elapsed time in seconds, minutes, even hours. Do you see any problems with this approach?

How many seconds are there in a month? That depends on which month of course. And even worse, it also depends on which year, due to leap years.

For instance, the difference between Jan 31st and Mar 1st is sometimes 29 days, sometimes 30 days, but always 1 month 1 day. The difference between Jul 2nd and Aug 1st is 30 days, but not a month.

So once the difference in time exceeds hours, we somehow need to use the additional information about where this period of time starts and ends, in order to get answers that make sense to humans. Luckily this information is already available in the tm structs.

If we accept the convention that the time between the 1st of a month and the 1st of the following is a “month” regardless of how many days are between, then one approach is to use elementary school subtraction on the tm structs. We subtract one field at a time, starting at the lowest, borrowing from the next one if required.

For instance, we get the difference in seconds by subtracting the tm_sec fields. If the result is negative, we have to borrow a minute.

/* Difference in the seconds */
seconds = end.tm_sec - start.tm_sec;

/* If negative, we have to borrow a minute */
if (seconds < 0) {
        seconds = 60 + seconds;
        min_borrow = 1;
}

But how do we borrow a month? Since we already know the number of days in the end month (end.tm_mday), and all months in between are full calendar months, we only need to figure out how many days are in the start month.

/* Returns 1 if y is a leap year, 0 otherwise */
static int leap(int y)
{
        return (y % 400 == 0 || (y % 4 == 0 && y % 100 != 0)) ? 1 : 0;
}

const int md[] = { 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31 };

/* ... */

/* Difference in the days of month */
days = end.tm_mday - start.tm_mday - day_borrow;

/* If negative, we have to borrow a month */
if (days < 0) {
        int start_mon_days;

        /* Get number of days in start month */
        start_mon_days = md[start.tm_mon];

        /* If february, correct for leap year */
        if (start.tm_mon == 1) {
                start_mon_days += leap(1900 + start.tm_year);
        }

        days = start_mon_days + days;
        mon_borrow = 1;
}

So, if we borrow a month, then the difference in the days of the month is the number of days left in the start month, plus the number of days in the end month, minus any borrow from the calculation of hours of the day. This is the same as the total number of days in the start month plus the negative difference we already computed.

This method gives the desired results on examples like the ones given above – months are what we perceive as months on the calendar.

There is one more detail we may have to consider – how many hours are there in a day? Usually 24, but due to daylight saving time (summer time) it might be 23 or 25.

As an example, consider the difference between 01:45 and 03:15, which is 1 hour 30 minutes in most cases. But the night DST starts, it will be only 30 minutes (from 01:45 STD to 03:15 DST). Even worse, the night DST ends, it will first be 2 hours 30 minutes (from 01:45 DST to 03:15 STD), then 1 hour 30 minutes (from 01:45 STD to 03:15 STD) as the clock runs through the extra hour.

Much like the relationship between months and days, I think it makes sense to interpret the difference between the same time on consecutive dates as a “day”, no matter if it takes 23, 24, or 25 actual hours. Once the difference goes below 1 day however, it is more logical to use the actual difference in clock time.

So we would like the difference between 12:00 STD the day before DST starts, and 12:30 DST the day DST starts to be 1 day 30 minutes even though it only takes 23 hours 30 minutes. And we want the difference between 12:30 DST the day before DST ends, and 12:00 STD the day DST ends to be 24 hours 30 minutes but not a day.

Handling of DST is implementation defined in C, so to keep things simple, we can let mktime() handle this for us. We just have to adjust the hours of the day in case the difference is below one day.

/* If difference is below one calendar day and there was a DST difference
   we adjust hour difference to clock time */
if (years + months + days == 0
 && hours != (secdiff % SEC_IN_DAY) / SEC_IN_HOUR) {
        int oldhours = hours;

        hours = (secdiff % SEC_IN_DAY) / SEC_IN_HOUR;

        /* Handle the special case where DST increased hours past 24 */
        if (oldhours - hours > 11) {
                hours += 24;
        }
}

In many cases you can safely ignore these details, and I am sure there are better ways to handle it – but I hope this has given some indication of how it may not be quite as simple as it appears. The code I wrote while playing around with this problem is available here.

Move Semantics Podcast

Mon, 16 May 2011 11:07:47 +0200

The Visual C++ Weekly had a link to a podcast with Scott Meyers from ACCU 2011 which I thought was great: link.

Long Division, Part 3

Mon, 11 Oct 2010 14:49:41 +0200

I ended part two of this series with an open question:

And of course I can’t help but wonder if the CLR is compiled with Visual C++, so doing arithmetic on 64-bit numbers in C# and other .NET languages ends up at the same runtime functions?

I don’t have a lot of experience in debugging the CLR myself, so I asked Brian Rasmussen if he might be interested in taking a look at it. He was kind enough to take the time to point me in the right direction.

A little digging showed that the CLR does in fact call some of these functions from the C runtime, but with a twist.

The C# function we looked at was this:

[MethodImpl(MethodImplOptions.NoInlining)]
private static Int64 div64(Int64 x, Int64 y) {
    return x / y;
}

The JIT compiler produced the following 32-bit code:

    push    ebp
    mov     ebp,esp
    push    dword ptr [ebp+14h]
    push    dword ptr [ebp+10h]
    push    dword ptr [ebp+0Ch]
    push    dword ptr [ebp+8]
    call    clr!JIT_LDiv
    pop     ebp
    ret     10h

We note the call to JIT_LDiv, which is a helper function in clr.dll (the CLR runtime, formerly mscorwks.dll). Let’s have a look at it:

INT64 JIT_LDiv(INT64 dividend, INT64 divisor)
{
    ...
    if (Is32BitSigned(divisor))
    {
        if ((INT32)divisor == 0)
        {
            ehKind = kDivideByZeroException;
            goto ThrowExcep;
        }

        if ((INT32)divisor == -1)
        {
            if ((UINT64) dividend == UI64(0x8000000000000000))
            {
                ehKind = kOverflowException;
                goto ThrowExcep;
            }
            return -dividend;
        }

        // Check for -ive or +ive numbers in -2**31 to 2**31
        if (Is32BitSigned(dividend))
            return((INT32)dividend / (INT32)divisor);
    }

    // For all other combinations fallback to int64 div.
    return(dividend / divisor);
    ...
}

This excerpt is from the Shared Source CLI, but the same code is present in clr.dll from .NET Framework 4.0.

The division in the return statement at line 28 generates a call to _alldiv from the C runtime library.

Now, the twist is that this helper function, besides checking for division by zero and overflow, also optimizes division of two values that both fit in 32 bits (the division with casts at line 24).

This is one of the optimizations I used in WCRT as well, so I was interested in seeing how the CLR code performed. I removed the exception handling along with associated checks and ended up with this function, which I tested using the timing application from part 2:

__int64 JIT_LDiv(__int64 dividend, __int64 divisor)
{
    if (Is32BitSigned(divisor))
    {
        if ((int)divisor == -1)
        {
            return -dividend;
        }

        // Check for -ive or +ive numbers in -2**31 to 2**31
        if (Is32BitSigned(dividend))
            return((int)dividend / (int)divisor);
    }

    // For all other combinations fallback to __int64 div.
    return(dividend / divisor);
}

Here is a summary of the results from my Athlon 64:

Function          WCRT        VC       Diff
-------------------------------------------
(CLR runtime function)
HH JIT_LDiv       2125      4222     +49.7%
HL JIT_LDiv       4104      5128     +20.0%
LL JIT_LDiv       1312      1312      +0.0%

(C runtime function)
HH _alldiv        1709      3907     +56.2%
HL _alldiv        3260      4309     +24.3%
LL _alldiv        1204      2161     +44.2%

HH = 64-bit op 64-bit
HL = 64-bit op 32-bit
LL = 32-bit op 32-bit

In the WCRT column, JIT_LDiv is compiled with _alldiv from WCRT, and in the VC column with _alldiv from the VC CRT. So the first three rows compare the speed of the CLR helper function when using those as fallback.

We can see that there is a noticeable improvement on HH and HL, even though the check for two values that both fit in 32 bits is done twice (once in JIT_LDiv, and once in the WCRT version of _alldiv).

The next three rows compare the C runtime function _alldiv from WCRT and the VC CRT. These results are the same as in part 2.

The special handling of two values that both fit in 32 bits is a big improvement. On LL, JIT_LDiv is almost as fast as the WCRT version of _alldiv. For HL and HH, JIT_LDiv is slightly slower than the regular VC _alldiv due to the extra checks.

So, in conclusion, the CLR runtime is compiled with Visual C++, and when code runs under 32-bit, the CLR runtime will fall back to the 64-bit arithmetic functions from the C runtime library.

The CLR runtime contains optimizations for certain values (usually when both operands fit in 32 bits), but for other cases optimizing the C runtime functions appears to provide a direct improvement for the CLR as well.

Move to Parent

Sat, 02 Oct 2010 00:04:56 +0200

This post is inspired by a discussion regarding a coding snack done by lanux128 on DonationCoder. A user requested a tool to add a context menu entry that would copy selected files and folders to the parent folder.

Moving folders is one of those tasks that appear trivial. But as always, the devil is in the details.

To keep it simple, we will assume regular files and folders in a filesystem where paths are unique. The problem we will look at is: Given absolute paths Src and Dst (where Dst is not equal to, or a subfolder of Src), move the contents of Src to Dst.

The first solution that comes to mind is to move files and folders recursively. In pseudocode it would be something along the lines of:

MoveSimple(Src, Dst)
{
    // Loop through files/folders in Src folder
    for each Name in Src {
        if Name is a folder {
            // Create folder in Dst
            mkdir Dst\Name

            // Recursively move contents of subfolder
            MoveSimple(Src\Name, Dst\Name)

            // Remove subfolder, which is now empty
            rmdir Src\Name
        }
        else {
            // Move file to Dst
            move Src\Name Dst\Name
        }
    }
}

This works – well, almost.

Consider the following folder structure (please forgive the tree output, it was an easy way to get a consistent representation):

\---parent
    \---foo
        |   log.txt  ; contains "first"
        |
        \---foo
                log.txt  ; contains "second"

We are in \parent\foo and want to move the contents (the subfolder foo and the file log.txt) to \parent. The correct result would be:

\---parent
    |   log.txt  ; contains "first"
    |
    \---foo
            log.txt  ; contains "second"

The problem is that if we call MoveSimple(\parent\foo, \parent), and we process the subfolder foo first, then we will overwrite \parent\foo\log.txt before it gets moved out to \parent.

You might suggest that we handle files first, and then recurse on subfolders. But that won’t work either:

\---parent
    \---foo
        +---foo
        |   \---zeb
        |           log.txt
        |
        \---zeb
                log.txt

If we recurse into foo before zeb, we will overwrite \parent\foo\zeb\log.txt before it gets moved out.

By now it should be clear that the culprit is the foo subfolder. With that, we end up writing to parts of the folder structure we are in the process of moving.

More generally, we may run into problems if any destination path touches the source tree.

The sledge hammer fix is to make sure there is no way we can overwrite anything in the source folder, by first moving it to someplace safe, and then to where we want it:

MoveSafeButSlow(Src, Dst)
{
    Tmp = random unique folder name

    // Create temporary folder in Dst
    mkdir Dst\Tmp

    // Move contents of Src to Tmp
    MoveSimple(Src, Dst\Tmp)

    // Remove Src which is now empty
    rmdir Src

    // Move contents of Tmp to Dst
    MoveSimple(Dst\Tmp, Dst)

    // Remove Tmp which is now empty
    rmdir Dst\Tmp
}

This works, but now we are moving every file and folder twice. Besides the performance impact, it would also make it hard to interrupt the operation and handle errors – the user could end up with files lost in a temporary directory.

Let’s go back to MoveSimple and try to figure out what exactly went wrong. It worked fine as long as we did not write to parts of the source tree that we had not moved yet.

The only way we can end up writing into the source tree, is if Src contains a special subfolder with a relative path that exactly matches the path from Dst to Src. Such a path would be unique.

Moving that special folder is only a problem if there is still something else left in Src that has not yet been moved. If we moved everything else, there is nothing it can overwrite.

So if we check if we run into such a special folder, and make sure we move everything else before it, it should work:

MoveSafeHelper(Src, Dst, OrgSrc)
{
    Special = 0

    // Loop through files/folders in Src folder
    for each Name in Src {
        if Name is a folder {
            // Check if we are about to overlap original Src
            if Dst\Name == OrgSrc {
                // Remember Name, but skip for now
                Special = Name
            }
            else {
                // Create folder in Dst
                mkdir Dst\Name

                // Recursively move contents of subfolder
                Tmp = MoveSafeHelper(Src\Name, Dst\Name, OrgSrc)

                // If we found a point of overlap inside Name,
                // set Special to the relative path to it
                if Tmp != 0 {
                    Special = Name\Tmp
                }
                else {
                    // Otherwise remove subfolder which is now empty
                    rmdir Src\Name
                }
            }
        }
        else {
            // Move file to Dst
            move Src\Name Dst\Name
        }
    }

    return Special
}

MoveSafe(Src, Dst)
{
    Special = MoveSafeHelper(Src, Dst, Src)

    // If the call returned a relative path to a folder that touches Src,
    // then we have moved everything else in Src now, and can safely
    // move that folder
    if Special != 0 {
        MoveSafe(Src\Special, Dst\Special)
    }
}

I wrote a simple Perl script that implements MoveSafe, and it works for all the test cases in this post, but of course standard disclaimers apply.

There are other ways to implement safely moving a folder, depending on what features you desire. For instance you can make it safe to move a folder to a child folder, which this function does not handle. The aim with this particular method was to get (roughly) the same speed as the simple recursive method.

At this point I started wondering if I was overcomplicating things – if this was in fact all rather trivial. So I set out to check how some popular file managers would fare.

The five I tried were: Directory Opus 9.5.5.0, Total Commander 7.55a, xplorer² 1.8.0.13, XYplorer 9.50, and Windows Explorer (WinXP SP3).

I made two test cases, which are similar to the examples above, but contain a few more files to make sure sorting order does not help any application get it right by chance.

I used a script to generate the test cases to make sure they were identical, and used any internal move command where possible. I answered yes to any dialogs that asked if I wanted to overwrite anything.

The first test case is this (as usual we are in \parent\foo and want to move the contents to \parent):

\---parent
    \---foo
        |   a.txt ; "first"
        |   z.txt ; "first"
        |
        \---foo
            |   a.txt ; "second"
            |   z.txt ; "second"
            |
            \---foo
                    a.txt ; "third"
                    z.txt ; "third"

And the second test case is:

\---parent
    \---foo
        +---ccc
        |       a.txt
        |       x.txt
        |
        +---foo
        |   +---ccc
        |   |       b.txt
        |   |       y.txt
        |   |
        |   \---zzz
        |           b.txt
        |           y.txt
        |
        \---zzz
                a.txt
                x.txt

None of the five file managers tested handled these two test correctly. I realize this special condition is not very common, but I still found it interesting that none of them, not even Windows Explorer, appear to be able to move any folder to a parent folder.

Quoting Command-line Arguments

Fri, 24 Sep 2010 16:22:26 +0200

Raymond Chen recently blogged about the way CommandLineToArgvW treats quotes and backslashes. Parsing the command-line into argv[] is something I have had to fight with as well, so besides pointing to Raymond’s excellent post, I wanted to add a few comments of my own here.

We are examining how command-line arguments with spaces and quotes are handled. Part of the problem comes from the fact that DOS/Windows uses backslash as separator in paths. On systems like Unix, where forward slash is used instead, using backslash to escape special characters is less of a problem. But if you ever put a Windows path in a C string literal, you may have run into LTS – the situation where a string becomes unreadable due to escape characters.

Microsoft fixed this in C# with verbatim string literals. C# also implements a simpler method of escaping a quote inside a quoted string – doubling it – which is used in languages like Pascal and BASIC, and is what Raymond’s second hypothetical set of rules suggest.

The compromise we get for parsing command-line arguments in the C runtime library (and CommandLineToArgvW()) is documented on MSDN. What the MSDN documentation does not tell you is that there is a second mechanism for inserting a literal quote in a quoted string – or at least there might be, depending on which version of the C runtime library.

In Visual C++ 2.0 from 1994, the command-line parser handles two quotes inside a quoted string as an escape. What it does is that it copies the second quote and switches to unquoted mode. Thus you can insert a literal quote inside a quoted string by putting three quotes (two to insert the literal quote, and one more to put you back into quoted mode).

if (*p == DQUOTECHAR) {
    ...
    if (inquote) {
        if (p[1] == DQUOTECHAR)
            p++;    /* Double quote inside quoted string */
        else        /* skip first quote char and copy second */
            copychar = 0;
    } else
        copychar = 0;       /* don't copy quote */

    inquote = !inquote;
    ...
}

This code is present until Visual C++ 8.0 from 2005, where it is changed so the two quote escape sequence no longer switches mode. That means that to insert a literal quote inside a quoted string, you now only need two quotes.

if (*p == DQUOTECHAR) {
    ...
    if (inquote && p[1] == DQUOTECHAR) {
        p++;    /* Double quote inside quoted string */
    } else {    /* skip first quote char and copy second */
        copychar = 0;       /* don't copy quote */
        inquote = !inquote;
    }
	...
}

This change is probably what is causing some confusion in the comments on Raymond’s post – the twelve literal quotes in foo""""""""""""bar will result in five quotes with the current parser (opening quote, five pairs, closing quote), but only four with the older method (opening quote, three triples, and a double quote).

The implementation of CommandLineToArgvW() in current versions of shell32.dll use the older method. The same goes for the command-line parser used in GCC on Windows (I checked 4.5.1).

Looking at the two code excerpts, one could wonder if the old handling was in fact just a bug introduced by the else part not being in braces, and left in for over ten years. Or maybe it was intentional, and the change in Visual C++ 8.0 was to support the way two quotes are used as escape in other languages.

Either way, I find it interesting that this functionality was added (it wasn’t supported in Visual C++ 1.52), and then changed in the C runtime library but not in the API function, and not mentioned in the documentation of either.

Long Division, Part 2

Mon, 20 Sep 2010 09:24:06 +0200

In part one I talked about the support functions in the C standard libraries of various x86 32-bit compilers that perform arithmetic operations when you use 64-bit integers in your code.

While updating WCRT to work with the latest Visual C++ compilers, I was writing my own implementations of these functions, and naturally I tested them against the versions supplied in the VC CRT to verify they worked.

To my surprise, I found the GCD test I wrote for Long Division ran faster when compiled with WCRT.

C:\gcdtest>gcdtest_gcc-4.5.1.exe
32-bit GCD: 314
64-bit GCD: 851

C:\gcdtest>gcdtest_vc2010.exe
32-bit GCD: 300
64-bit GCD: 835

C:\gcdtest>gcdtest_vc2010-wcrt.exe
32-bit GCD: 300
64-bit GCD: 656

This naturally piqued my curiosity.

It turns out that the 64-bit arithmetic functions in VC have remained unchanged since Visual C++ 4.2 from 1996. The compiler and optimizer have evolved over the years, adapting to new processors, and improving the speed of your code. But if you happen to divide two 64-bit integers you are relying on code that is 14 years old.

This was of course more than enough incentive to spend a little time trying to improve my implementations to see if I could beat the VC CRT.

Here are parts of the result from running this test on an Athlon 64:

Function          WCRT        VC       Diff
-------------------------------------------
HH _alldiv        1709      3907     +56.2%
HL _alldiv        3260      4309     +24.3%
LL _alldiv        1204      2161     +44.2%
...
HH _allmul         293       337     +13.0%
HL _allmul         631       739     +14.6%
LL _allmul         279       291      +4.1%
...
<32 _allshl        657       745     +11.8%
<64 _allshl        568       566      -0.3%
<96 _allshl        654       537     -21.7%
...

As you can see, the difference ranges from a few percent to over 50% in the most favorable test.

The only operations that are slower in general are shifts of more than 64 bits. The shift functions do not contain a lot of code to work with in the first place, but I did choose to sacrifice some speed in shifts above 64 bits to get a small improvement in shifts below 32 bits, because they seem more likely to occur in actual code.

Now of course these improvements do come with some reservations. For one, the CRT functions have to perform well across all processors from 386 up to the cutting edge, while I only wrote mine with Intel Core and AMD Athlon in mind. Also, the test functions I use do not necessarily reflect the average use, so while some of the optimizations I did were favorable to the tests used, they may not be for other cases.

That being said, I think this does show that it is possible to get a non-trivial improvement by optimizing these functions. And if critical parts of your code deal with 64-bit numbers it is certainly worth investigating alternatives.

And of course I can’t help but wonder if the CLR is compiled with Visual C++, so doing arithmetic on 64-bit numbers in C# and other .NET languages ends up at the same runtime functions?

Long Division

Wed, 08 Sep 2010 08:35:41 +0200

Integer types with at least 64 bits have been a part of the C standard for a while now (they were added in C99, and were a standard extension in many 32-bit compilers before that). But have you ever wondered what exactly happens when you use them?

Consider the following function (substitute long long with __int64 if you are using an older version of Visual C++):

long long div64(long long x, long long y)
{
        return x / y;
}

let’s first have a look at what the VC 64-bit compiler gives us:

div64:
    mov   r8, rdx
    mov   rax, rcx
    cdq
    idiv  r8
    ret

Pretty much what you would expect, a little setup and an idiv instruction to perform the division. Now let’s try the VC 32-bit compiler:

div64:
_x$ = 8
_y$ = 16
    mov   eax, _y$[esp]
    mov   ecx, _y$[esp-4]
    mov   edx, _x$[esp]
    push  eax
    mov   eax, _x$[esp]
    push  ecx
    push  edx
    push  eax
    call  __alldiv
    ret

A little setup and .. a call?

The thing is, 32-bit x86 only has instructions for mixed mode 32/64 bit operations – with one assembly instruction you can multiply two 32-bit integers to get a 64-bit result, or you can divide a 64-bit integer by a 32-bit as long as the quotient fits in 32 bits. But to divide two 64-bit integers, you have to emulate the operation in software.

That is where the _alldiv function comes in. It is a support function in the C runtime library that performs a 64-bit divide. In fact there is a whole family of such functions to do multiply, divide and shifts on 64-bit numbers (addition and subtraction are inlined since they only require two assembly instructions).

Other compilers do likewise, for instance GCC calls __divdi3 to perform 64-bit division.

It is important to be aware of this, because using long long instead of int means that arithmetic operations you probably expect to be single instructions, all of a sudden become library calls who’s execution time can depend on the arguments.

As an example, I timed a simple implementation of the extended Euclidean algorithm with 32- and 64-bit integers. Compiling with the VC 32-bit compiler, the 64-bit version was roughly 3 times slower (depending on the CPU).