24 March 2020

Converting a float to an integer

Without giving it another thought we often convert a floating-point value to an integer by simply assigning a float to an integer data type variable. We assume the compiler knows what’s best and we trust the compiler does the proper conversion for us. Time for a look behind the scenes.

The FPU and the control register
The FPU is independent of the main processor and contains its own set of registers to perform its task. The FPU registers include eight 80-bit data registers (ST0-ST7), and three 16-bit registers called the control, status, and tag registers. We will ignore the status and tag registers and focus on the control register which is used to access the features of the FPU. The control register controls the floating-point functions within the FPU. The control register uses a 16-bit register where each bit defines a specific setting such as the exceptions to produce, the precision the FPU uses to calculate floating-point values, and the method used to round the floating-point results. The bits are shown below:

Control bits      Description
0-5 Exception masks
8-9 Precision control
10-11 Rounding control                
12Infinity control

For the conversion from float to integer we are only interested in bits 10 and 11. The x87 FPU implements four rounding methods in hardware. The possible settings of the rounding control bits are as follows:

00 - Round to nearest (even)
01 - Round down, towards negative infinity
10 - Round up, towards positive infinity
11 - Round toward zero

The "Round to nearest (even)" method is used by default by GFA-BASIC 32, so there's a high chance you're already using it. The current rounding mode of the x87 can be obtained using the fstcw assembler command, as shown in the following sample:

' Status of bits 10 and 11 of control word
Dim cf As Word
. fstcw [cf]
Debug Bin(cf %& %110000000000, 2)  ' = 00

By default, GB clears both rounding bits and converts floating-point values to integers using the “Round to nearest (even)”. For this to happen, GB initializes the FPU with the value 0x372, which clears the rounding bits. The way of rounding can be changed and set to a new value by setting  the control word to a new value using the fldcw instruction. GB changes the rounding bits when it rounds a floating-point value using one of the truncation/rounding functions like Int, Trunc, Floor, Ceil, etc. as we’ll see later in this post.

By default, the rounding control is set to rounding to the nearest (even) which seems to be correct for most calculations. However, this might cause an unexpected behavior, for instance 2.5 is not rounded to 3, but rounded to 2. The FPU prefers the nearest even value when the decimal part is exactly .5. For the same reason 3.5 is rounded to 4. This is useful in case of statistical analyzing where you want to spread numbers evenly when they are exactly in the middle of two integers. However, this might not always be what you want. For instance, by default C/C++ compilers always convert floating point values by truncating them down towards zero (cast to Int).

If you don’t care and if you are happy with the GB’s default rounding, you can easily convert a floating point value to integer by simply assigning the one to the other:

Dim f As Float = 2.5, i As Int
i = f     ' assign float to int
Debug i   ' = 2

This is the fastest possible conversion, it takes only one assembler instruction (fistp) to convert a floating point as shown in the disassembly (without the Debug command). See for more details of disassembling GB-code the blogpost Anatomy of a procedure.

--------  Disassembly -----------------------------------
0 - (Sub Main) (Lines=2)
0559C350: B8 2E 00 00 00                 mov     eax,0x0000002E
0559C355: FF 15 40 1A 4D 00              scall   INITPROC0 ; Ocx: $180277CB
0559C35B: F8                             clc    
0559C35C: FF 55 B4                       call    dpt -76[ebp] ; @Tron
0559C35F: C7 05 10 A7 5C 06 00 00 20 40  mov     dpt [0x065CA710],0x40200000
0559C369: FF 55 B4                       call    dpt -76[ebp] ; @Tron
0559C36C: D9 05 10 A7 5C 06              fld     dpt [0x065CA710]
0559C372: DB 1D 14 A7 5C 06              fistp   dpt [0x065CA714]

0559C378: 8B 4D F0                       mov     ecx,dpt -16[ebp]
0559C37B: 64 89 0D 00 00 00 00           mov     dpt fs:[0x00000000],ecx
0559C382: 8B E5                          mov     esp,ebp
0559C384: 5D                             pop     ebp
0559C385: 5B                             pop     ebx
0559C386: 5F                             pop     edi
0559C387: 5E                             pop     esi
0559C388: C3                             ret

The interesting commands are fld and fistp. First fld loads the floating-point value into ST0 – the top of the stack - and then the fistp instruction pops the value off the floating-point stack. fistp converts it to an integer, and then stores it at the address specified. This instruction uses the rounding control settings to determine how they will convert the floating point data to an integer during the store operation.

There is one other assembler instruction that rounds to integer. The frndint instruction rounds the value in ST0 (the top of the stack) to the nearest integer using the rounding algorithm specified in the control register. The result remains in ST0 as a floating point value, it simply does not have a fractional component. GB uses the frndint instruction for its truncation functions Int(), Fix() and others.

The truncation functions
The function Trunc (or Fix) round towards zero. The Int (or Floor) function round towards negative infinity and Ceil rounds to positive infinity. Let’s start with the most often used truncation function Int(), or its synonym Floor(). We’ll use this simple program to look at it’s disassembly.

Dim f As Float = 2.5, i As Int
i = Int(f)   ' truncate down to negative infinity

The disassembly of the Int() function (without the surrounding commands):

0559C4EC: D9 05 10 A7 5C 06   fld     dpt [0x065CA710]
0559C4F2: D9 2D 1C 1A 4D 00   fldcw   V_RNDMINUS
0559C4F8: D9 FC               frndint
0559C4FA: D9 2D 14 1A 4D 00   fldcw   V_RNDNEAR
0559C500: DB 1D 14 A7 5C 06   fistp   dpt [0x065CA714]

First ST0 is loaded with the value in variable f, then the control word of the FPU is loaded with the value from a variable called V_RNDMINUS, which is 0x77. This value sets the rounding bits of the control word to %01. By executing frndint the value in ST0 is rounded to an integer using the setting “Round down towards negative infinity”. Int() rounds 2.5 to 2 and –2.5 to -3. After rounding, the control word is reset to the default value 0x372, which is stored in the runtime variable V_RNDNEAR. Finally, the value in ST0 is moved to the variable i with fistp,

Note Technically fld and fistp aren’t part of the Int() function. How a value ends up in ST0 depends on the code of the program. Similarly, fistp is only inserted if the result of Int() is to be stored in a variable. If Int() is used inside an expression the value remains in ST0.

In the same way you can create samples for the other truncation functions Trunc(), Fix(), and Ceil() and then examine their disassembly output. Trunc() and its synonym Fix() load the control word with the value stored in V_RNDZERO ( = 0xF72). This value sets the rounding bits to %11 to round towards zero. For Ceil() the rounding bits are set to %10, the value for the control word is obtained from the variable V_RNDPLUS ( = 0xB72), and frndint then rounds to positive infinity.

The QRound function
The QRound function is an addition to the truncation/rounding functions. The compiler generates only one instruction for this function: the frndint instruction to round the value in ST0. The compiler does not load the control word prior to executing the rounding.  QRound uses the current control word setting (0x372) and “Round to nearest (even)”. QRound is useful inside a mathematical expression where some interim outcome needs to be rounded (converted) to integer using the current control word setting. Since the interim outcome remains in ST0 (without the fractional part) a (complex) expression can be evaluated more quickly. In short the steps for variable = QRound(float) are:

fld float
fistp variable

Again, fld and fistp are not part of the function itself. Since QRound is mostly used with the default control word setting rounding is the same as when a float is assigned to an integer variable directly, as demonstrated at the beginning of this blogpost.

Note that using the assembler instruction fldcw prior to QRound you can determine your own float-to-integer conversion. Do not forget to return the value of the control word back to 0x372 afterwards.

Use Round for proper rounding
The Round function generates the same assembler instructions as Int(). However before frndint is executed the value in ST0 is increased with 0.5. The value in ST0 is then rounded towards negative infinity. In short, these steps are:

fld value
fadd 0.5 / fldcw 0x772 / frndint / fldcw 0x372
fistp variable

If your program wants “proper rounding” it should use Round() to convert a floating-point value to an integer.