Benchmarking Microsoft BASICs

I decided it’d be an interesting project to try to benchmark the Microsoft BASICs – I’ve always seen the claim that QBASIC was slower than QuickBASIC, and I thought I’d put that to the test.

So, my test methodology is to use the PCW Benchmarks (might want to make sure you’ve got a good adblocker handy here), which are relatively simple, and make no attempt to benchmark graphics performance, but they do a decent job of testing the interpreter itself. I’ve made the following modifications:

• Calculating elapsed time by comparing the value of TIMER before and after each run, and maintaining a rolling average over 1000 runs. The resolution on TIMER is (14318180 / 12) / 65536 Hz, or slightly over 18.2 Hz, or approximately 55 ms, so I need a lot of runs to get something useful out of a fast benchmark. The only impact on benchmark time should be the variable operations involving TIMER – everything else isn’t timed. GOTO may also be affected slightly by the longer program, but eh, I’m not that worried.
• Benchmark 8 calls for the use of the LN() function on line 45, which is LOG() on Microsoft BASIC dialects.
• I’m explicitly ending the program, but the END occurs outside of the timed loop. PCW Benchmark runs which rely on subroutine calls have the subroutine (line 700) located after the END statement, as QuickBASIC dialects do not handle RETURN without GOSUB. This may actually have a performance impact, as there’s more code that GOSUB has to search through to find the subroutine. For Visual Basic, I’m using Exit Sub.
• For QuickBASIC and QBASIC, all lines that I’ve added are not numbered, as no QuickBASIC dialect needs that. For GW-BASIC, they are numbered of course.
• All compiled dialects will first use default/recommended settings, and then easily-available optimizations will be tried. (For QuickBASIC 4.5, this means disabling debug builds, really.)
• For Visual Basic, instead of using DIM m(5) for the array, I’m using ReDim m(5).
• Also for Visual Basic, I decided to put everything into one file, due to the nature of VB making that make sense. Fun fact: line numbers have to be unique, so I prepended them with the digit of the benchmark being run. This may affect performance ever so slightly (I did start out with one project/form per benchmark), but I’m not seeing much of a difference.

Hardware is an IBM ThinkPad 365XD with a Pentium 133 and no L2 cache (but 70 ns EDO RAM). The system has Windows 98 SE installed, and I’ll be running in MS-DOS mode for DOS-based BASICs.

And, ultimately, here’s what I want to find out:

• How much of a performance difference is there between QBASIC and QuickBASIC?
• There’s rumors that QBASIC is using QuickBASIC 4.0’s interpreter instead of 4.5’s… might performance metrics shed light on this?
• How close to the QuickBASIC engines is Visual BASIC? We know that Embedded BASIC is derived from QuickBASIC 4.0… I’m most interested in Visual BASIC 1.0 here, but if there were significant performance improvements in later releases, how do they look?
• For that matter, how do PDS and Visual BASIC for DOS compare?
• How do pre-4.0 releases, and pre-QuickBASIC releases, compare?

I won’t be deep-diving much with Visual Basic tonight, but I’ll come back and edit this post once I’ve done that.

And, a quick rundown of what the benchmarks actually do, if you didn’t click the link – these tests:

1. Test 1000 iteration FOR loop execution speed
2. Test 1000 iteration IF/GOTO loop execution speed
3. Test arithmetic calculation (division, multiplication, addition, subtraction) using variables inside of benchmark 2’s loop
4. Test arithmetic calculation using constants inside of benchmark 2’s loop
5. Test subroutine calling inside of benchmark 4’s loop
6. Test array dimensioning at the beginning, and a 5 iteration FOR loop inside of benchmark 5’s main loop (this result isn’t interesting)
7. Test array value assignment in benchmark 6’s inner FOR loop (this result needed benchmark 6 to put it into context)
8. Test array dimensioning at the beginning, and raising numbers to a power, natural logarithms, and sines inside of benchmark 2’s loop

So, in decreasing order of execution time, all times in seconds…

	BM 1	BM 2	BM 3	BM 4	BM 5	BM 6	BM 7	BM 8
QBASIC 1.1	.029	.050	.105	.111	.113	.306	.417	.631
GW-BASIC 3.23	.004	.026	.057	.057	.061	.108	.179	.194
QB 4.5 EXE	.012	.013	.018	.018	.019	.092	.136	.071
QB 4.5 EXE, no dbg	.011	.011	.016	.016	.016	.083	.120	.063
QB 4.5 IDE	.004	.012	.017	.018	.018	.051	.086	.056
VBWin 1.0 IDE	.001	.003	.005	.005	.005	.013	.022	.012
VBWin 1.0 EXE	.001	.002	.004	.005	.005	.013	.022	.014
QB 4.0 IDE	.001	.002	.003	.004	.004	.008	.014	.013
QB 4.0 EXE	.000	.000	.001	.001	.001	.003	.005	.009

What conclusions can we draw from this?

• First off, QBASIC is slow as shit. Even GW-BASIC beats it. Of course, QBASIC is using IEEE instead of Microsoft Binary number format (note that Microsoft ended up putting Microsoft Binary as an option back into their compilers starting with 6.0), and it’s doing a lot more under the hood. And, it doesn’t support the x87 FPU.
• QuickBASIC 4.0 is fast. Like seriously fast. It is, however, taking advantage of the x87, so I may want to bring these benchmarks over to a system without a FPU, and see how it compares there.
• QuickBASIC 4.5 takes a big speed hit compared to 4.0, and the EXEs are oddly slow compared to the IDE’s performance. Note that it’s taking advantage of the x87, too. A couple things that make me wonder, though… Microsoft BASIC 6.0 became the professional product above QuickBASIC 4.5, and QuickBASIC 4.5 builds significantly smaller executables than 4.0 – could Microsoft have crippled performance in QuickBASIC 4.5, or tweaked things to optimize for size over speed? (I noticed no difference when I linked NOEM.OBJ in, which should remove x87 emulation code, but I’m not sure it worked properly.)
• Visual Basic 1.0 is nearly as fast as QuickBASIC 4.0 – a touch slower, but not bad.

What next?

• I think I’ll re-run these tests on a machine without a FPU (a much slower machine, an HP OmniBook 425 with a Cyrix 486SLC 25 MHz), to see how that compares.
• I definitely need to add Microsoft BASIC 6.0, as well as the later DOS-based BASICs, to these tests.
• I also want to go backwards to QuickBASIC 2.0 and 3.0 – from before the threaded p-code interpreter/compiler architecture of QuickBASIC 4.0 – as well as the various BASIC compilers from the olden days. Similarly, I’ve got BASIC-86 5.28, and I’d like to figure out how to use it for this (one huge problem is that it’s not well-integrated into DOS, so I’ll likely need to implement TIMER myself by directly checking INT 21h AH=2Ch).

Edit: And now it’s time for the OmniBook 425’s results. Three digits of precision, because some of these will be over a second. OS is MS ROM DOS 5.0, with ROM Windows 3.1 in Standard mode (386 Enhanced mode isn’t available in execute-in-place Windows) for the VB tests. (The OmniBook is a bit of an oddball.) I’m also not going to test as much, only 100 iterations – this thing is far, far slower, which means the timer error isn’t as much of a problem, and I don’t want to wait for it – and I won’t bother with making EXEs. Except for the revised iteration count, same source code, though.

	BM 1	BM 2	BM 3	BM 4	BM 5	BM 6	BM 7	BM 8
VBWin 1.0 IDE	.289	.512	1.15	1.33	1.39	3.02	4.30	5.81
QBASIC 1.1	.194	.303	.699	.821	.833	2.07	2.86	5.70
QB 4.5 IDE	.167	.260	.618	.710	.720	1.79	2.49	5.38
QB 4.0 IDE	.142	.208	.568	.569	.578	1.35	1.77	4.12
GW-BASIC 3.23	.085	.371	.799	.796	.863	1.51	2.37	2.09

OK, so these results show off what not having an x87 does for you. GW-BASIC is actually a bit slower than the QuickBASICs (or even QBASIC) in looping constructs, but math helps it catch up, and advanced math is vastly faster, thanks to the Microsoft Binary number format, and possibly smaller code fitting in cache better for that.

QBASIC is definitely detuned relative to QuickBASIC 4.5, but the general trends in performance follow it, not QuickBASIC 4.0 (the big giveaway is that benchmark 4 is noticeably slower than 3, which isn’t true for QuickBASIC 4.0).

VBWin’s results strike me as an outlier, possibly caused by the low amount of cache on the CPU (only 1 kiB), and Windows causing cache flushes? Disabling cache may be informative. However, it’s also interesting to note that it follows the QuickBASIC 4.5 pattern, not the 4.0 pattern.