{"id":926,"date":"2014-04-03T12:52:19","date_gmt":"2014-04-03T12:52:19","guid":{"rendered":"http:\/\/notonbluray.com\/blog\/?p=926"},"modified":"2022-09-02T08:50:03","modified_gmt":"2022-09-02T08:50:03","slug":"python-ieee-floats-for-decimal-conversion","status":"publish","type":"post","link":"https:\/\/notonbluray.com\/blog\/python-ieee-floats-for-decimal-conversion\/","title":{"rendered":"Python: Pulling Apart IEEE Floats for Decimal conversion"},"content":{"rendered":"

While attempting to convert floats precisely to Decimals the other day I was caught out by the fact that Python 2.6’s print function defers to whatever the underlying C compiler’s printf function does. This meant that on Linux I got one result when I asked for ‘%.31f’ % math.pi, and on Windows – something completely different. This lack of portability is a pain, and I had to work around it.<\/p>\n

<\/p>\n

‘Why not use repr ?’ I hear you ask. Because it doesn’t give you a precise representation – witness:<\/p>\n

>>> 1.1 * 1.1\r\n1.2100000000000002\r\n>>> from decimal import Decimal\r\n>>> Decimal( repr(1.1) ) * Decimal( repr(1.1) )\r\nDecimal('1.21')\r\n>>> Decimal( repr( 1.1 * 1.1 ) )\r\nDecimal('1.2100000000000002')<\/pre>\n
As you note, repr provides the shortest representation that you are likely to want within the granularity of a floating point number, not the exact midpoint value. Which is what I wanted<\/p>\n
There are various very good reasons for Decimal<\/a> not supporting direct conversion, but if the only way in is float to string conversion that varies by platform, then it’s kind of crappy. Note this has been fixed in later versions of Python (2.7 and 3.x support from_float) but for various reasons I’m stuck with 2.6 for now.<\/p>\n
As it turns out there are two ways to do it: an easy way, and a hard way.<\/span><\/p>\n
The Easy Way – Integer Ratios<\/h2>\n
import decimal\r\n\r\ndef floatToDecimalViaRatio( f ):\r\n     '''Given a floating point number, \r\n        return the exact representation of that number as a decimal '''\r\n     top, bottom = f.as_integer_ratio()\r\n     return decimal.Decimal( top ) \/ decimal.Decimal( bottom )<\/pre>\n
Obviously this doesn’t handle NaN, +Inf, -Inf but it gets the job done. There is one minor issue with it if you are operating inside hugely precise Decimal contexts: Divide is a bit slow.<\/p>\n
Besides this was a bit boring. Is it possible to pull a float apart and get at the mantissa and exponent directly ?<\/p>\n
The Hard Way – Extracting Mantissa and Exponent<\/h2>\n
In C this would be a straightforward cast to a 64 bit long and a bit of shifting. In Python you have to cheat. One way I’ve seen to do this is using struct.pack and unpack. But testing this I found to be very slow.<\/p>\n
So after experimenting a bit, I came up with using the C interface library to do a cast:<\/p>\n
mantissa_mask = (2 ** 52) - 1\r\nexponent_mask = ((2 ** 63) - 1) - mantissa_mask\r\nsign_mask = (2 ** 63)\r\n\r\ndef float_to_mantissa_exponent_negative( f ):\r\n    val = cast((c_double * 1)(f), POINTER(c_ulonglong)).contents.value\r\n    exponent = ((val & exponent_mask) >> 52L) - 1023\r\n    mantissa = val & mantissa_mask\r\n    negative = bool(val & sign_mask)\r\n    return exponent, mantissa, negative<\/pre>\n
Now the first line casts the 64bit floating point value (as an array of c_double of size 1) to a c_longlong, and then unpacks the contents into val.<\/p>\n
The rest is some straightforward masking and shifting. The minus 1023 comes from the fact that the exponent is stored unsigned, and makes for easier comparisons in hardware.<\/p>\n
Now we have the various bits of the floating point value – can we bring it all together ?<\/p>\n
import ctypes\r\nimport decimal\r\n\r\nzero = decimal.Decimal( 0 )\r\ntwo  = decimal.Decimal( 2 )\r\ntwofiftytwo = two ** decimal.Decimal( 52 )\r\n\r\nmantissaMSB  = 2L ** 52L\r\nmantissaMask = ( 2L ** 52L ) - 1L\r\nexponentMask = ( ( 2L ** 63L ) - 1L ) - mantissaMask\r\nsignMask = 2L ** 63L\r\n\r\n# @context_memo - add memoization here - don't forget to include the decimal.context into the args...\r\ndef _decimal_exponent(exponent, negative):\r\n    \"\"\" When passed an exponent, returns the corresponding decimal value, divided by 2 ** 52\r\n    \"\"\"\r\n    res = (two ** decimal.Decimal(exponent)) \/ two_pow_fifty_two\r\n    if negative:\r\n        return negative_one * res\r\n    return res\r\n\r\ndef float_to_decimal(f):\r\n    \"\"\" Convert a passed floating point number to a Decimal, by directly accessing the mantissa and exponent of\r\n     the floating point value. Depends on decimal.context to provide sufficient precision for this operation \"\"\"\r\n    val = cast((c_double * 1)(f), POINTER(c_ulonglong)).contents.value  # c-style cast of float to long\r\n    exponent = ((val & exponent_mask) >> 52L) - 1023\r\n    mantissa = val & mantissa_mask\r\n    full_mantissa = mantissa + mantissa_msb  # 1.1 instead of .1\r\n    negative = bool(val & sign_mask)\r\n\r\n    if exponent not in (1024, -1023):  # Most common case\r\n        if exponent > 55:  # No fractional part - rely on Pythons arbitrary length longs and shifting\r\n            if negative:\r\n                return decimal.Decimal(-(full_mantissa << (exponent - 52)))\r\n            else:\r\n                return decimal.Decimal(full_mantissa << (exponent - 52))\r\n        else:\r\n            return decimal.Decimal(full_mantissa) * _decimal_exponent(exponent, negative)\r\n    elif exponent == 1024:  # (i.e. all ones: +\/- inf, NaN)\r\n        return decimal.Decimal(str(f))\r\n    elif exponent == -1023 and mantissa == 0:\r\n        return zero  # No negative zero\r\n    elif exponent == -1023:\r\n        # Denormal number, no leading one, special case mantissa 0.1 or 0.01 instead of 1.1\r\n        return decimal.Decimal(mantissa) * _decimal_exponent(-1022, negative)\r\n    else:\r\n        raise RuntimeError('Unexpected error during conversion')<\/pre>\n
So why on earth would you want to do it the hard way ? Given the extra complexity ?<\/p>\n
Two reasons: It’s slightly faster on typical cases, and where you are operating inside a decimal context with extreme precision (>50 digits), it may be much much faster.<\/p>\n
So why is it faster? In essence the ‘hard way’ spends most of it’s time bit shifting Native Longs, and then right at the end it does a single Decimal multiply. The exponent part is likely to have been memoized, so is already constructed and costs nothing.<\/p>\n
Comparing Performance of Float To Python Decimal Techniques<\/h2>\n
I wrote a simple benchmark that converts math.e and math.pi two hundred times in a loop to compare the methods. As you can see Repr is very fast – but as discussed isn’t suitable for my needs. Using Fraction instead of decimal is comparable to the casting method (it uses as_integer_ratio under the covers).<\/p>\n
Now don’t get too excited. There is some clever magic in the new from_float factory method that makes it faster still, but I was quite impressed that I managed to come close in pure Python.<\/p>\n
\"A<\/a>
A performance comparison of techniques to convert a floating point number to a Python Decimal. from_float is comparable to repr (but slightly slower)<\/p><\/div>\n<\/div>","protected":false},"excerpt":{"rendered":"
While attempting to convert floats precisely to Decimals the other day I was caught out by the fact that Python 2.6’s print function defers to whatever the underlying C compiler’s printf function does. This meant that on Linux I got one result when I asked for ‘%.31f’ % math.pi, and on Windows – something completely […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2,35],"tags":[66,65,84],"class_list":["post-926","post","type-post","status-publish","format-standard","hentry","category-essay","category-python","tag-decimal","tag-float","tag-python"],"yoast_head":"\nPython: Pulling Apart IEEE Floats for Decimal conversion - Not on Blu-ray<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/notonbluray.com\/blog\/python-ieee-floats-for-decimal-conversion\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Python: Pulling Apart IEEE Floats for Decimal conversion - Not on Blu-ray\" \/>\n<meta property=\"og:description\" content=\"While attempting to convert floats precisely to Decimals the other day I was caught out by the fact that Python 2.6’s print function defers to whatever the underlying C compiler’s printf function does. This meant that on Linux I got one result when I asked for ‘%.31f’ % math.pi, and on Windows – something completely […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/notonbluray.com\/blog\/python-ieee-floats-for-decimal-conversion\/\" \/>\n<meta property=\"og:site_name\" content=\"Not on Blu-ray\" \/>\n<meta property=\"article:published_time\" content=\"2014-04-03T12:52:19+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-09-02T08:50:03+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/notonbluray.com\/blog\/wp-content\/uploads\/2014\/03\/Float-To-Decimal-in-Python-performance-of-different-techniques.png\" \/>\n<meta name=\"author\" content=\"Link\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Link\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/notonbluray.com\\\/blog\\\/python-ieee-floats-for-decimal-conversion\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/notonbluray.com\\\/blog\\\/python-ieee-floats-for-decimal-conversion\\\/\"},\"author\":{\"name\":\"Link\",\"@id\":\"https:\\\/\\\/notonbluray.com\\\/blog\\\/#\\\/schema\\\/person\\\/bd31ad218b684e12e90bbc2db3f6e650\"},\"headline\":\"Python: Pulling Apart IEEE Floats for Decimal conversion\",\"datePublished\":\"2014-04-03T12:52:19+00:00\",\"dateModified\":\"2022-09-02T08:50:03+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/notonbluray.com\\\/blog\\\/python-ieee-floats-for-decimal-conversion\\\/\"},\"wordCount\":644,\"image\":{\"@id\":\"https:\\\/\\\/notonbluray.com\\\/blog\\\/python-ieee-floats-for-decimal-conversion\\\/#primaryimage\"},\"thumbnailUrl\":\"http:\\\/\\\/notonbluray.com\\\/blog\\\/wp-content\\\/uploads\\\/2014\\\/03\\\/Float-To-Decimal-in-Python-performance-of-different-techniques.png\",\"keywords\":[\"Decimal\",\"Float\",\"Python\"],\"articleSection\":[\"Essay\",\"Python\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/notonbluray.com\\\/blog\\\/python-ieee-floats-for-decimal-conversion\\\/\",\"url\":\"https:\\\/\\\/notonbluray.com\\\/blog\\\/python-ieee-floats-for-decimal-conversion\\\/\",\"name\":\"Python: Pulling Apart IEEE Floats for Decimal conversion - Not on Blu-ray\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/notonbluray.com\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/notonbluray.com\\\/blog\\\/python-ieee-floats-for-decimal-conversion\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/notonbluray.com\\\/blog\\\/python-ieee-floats-for-decimal-conversion\\\/#primaryimage\"},\"thumbnailUrl\":\"http:\\\/\\\/notonbluray.com\\\/blog\\\/wp-content\\\/uploads\\\/2014\\\/03\\\/Float-To-Decimal-in-Python-performance-of-different-techniques.png\",\"datePublished\":\"2014-04-03T12:52:19+00:00\",\"dateModified\":\"2022-09-02T08:50:03+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/notonbluray.com\\\/blog\\\/#\\\/schema\\\/person\\\/bd31ad218b684e12e90bbc2db3f6e650\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/notonbluray.com\\\/blog\\\/python-ieee-floats-for-decimal-conversion\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/notonbluray.com\\\/blog\\\/python-ieee-floats-for-decimal-conversion\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/notonbluray.com\\\/blog\\\/python-ieee-floats-for-decimal-conversion\\\/#primaryimage\",\"url\":\"http:\\\/\\\/notonbluray.com\\\/blog\\\/wp-content\\\/uploads\\\/2014\\\/03\\\/Float-To-Decimal-in-Python-performance-of-different-techniques.png\",\"contentUrl\":\"http:\\\/\\\/notonbluray.com\\\/blog\\\/wp-content\\\/uploads\\\/2014\\\/03\\\/Float-To-Decimal-in-Python-performance-of-different-techniques.png\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/notonbluray.com\\\/blog\\\/python-ieee-floats-for-decimal-conversion\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/notonbluray.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Python: Pulling Apart IEEE Floats for Decimal conversion\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/notonbluray.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/notonbluray.com\\\/blog\\\/\",\"name\":\"Not on Blu-ray\",\"description\":\"Reviews of streaming media\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/notonbluray.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/notonbluray.com\\\/blog\\\/#\\\/schema\\\/person\\\/bd31ad218b684e12e90bbc2db3f6e650\",\"name\":\"Link\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Python: Pulling Apart IEEE Floats for Decimal conversion - Not on Blu-ray","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/notonbluray.com\/blog\/python-ieee-floats-for-decimal-conversion\/","og_locale":"en_US","og_type":"article","og_title":"Python: Pulling Apart IEEE Floats for Decimal conversion - Not on Blu-ray","og_description":"While attempting to convert floats precisely to Decimals the other day I was caught out by the fact that Python 2.6’s print function defers to whatever the underlying C compiler’s printf function does. This meant that on Linux I got one result when I asked for ‘%.31f’ % math.pi, and on Windows – something completely […]","og_url":"https:\/\/notonbluray.com\/blog\/python-ieee-floats-for-decimal-conversion\/","og_site_name":"Not on Blu-ray","article_published_time":"2014-04-03T12:52:19+00:00","article_modified_time":"2022-09-02T08:50:03+00:00","og_image":[{"url":"http:\/\/notonbluray.com\/blog\/wp-content\/uploads\/2014\/03\/Float-To-Decimal-in-Python-performance-of-different-techniques.png","type":"","width":"","height":""}],"author":"Link","twitter_misc":{"Written by":"Link","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/notonbluray.com\/blog\/python-ieee-floats-for-decimal-conversion\/#article","isPartOf":{"@id":"https:\/\/notonbluray.com\/blog\/python-ieee-floats-for-decimal-conversion\/"},"author":{"name":"Link","@id":"https:\/\/notonbluray.com\/blog\/#\/schema\/person\/bd31ad218b684e12e90bbc2db3f6e650"},"headline":"Python: Pulling Apart IEEE Floats for Decimal conversion","datePublished":"2014-04-03T12:52:19+00:00","dateModified":"2022-09-02T08:50:03+00:00","mainEntityOfPage":{"@id":"https:\/\/notonbluray.com\/blog\/python-ieee-floats-for-decimal-conversion\/"},"wordCount":644,"image":{"@id":"https:\/\/notonbluray.com\/blog\/python-ieee-floats-for-decimal-conversion\/#primaryimage"},"thumbnailUrl":"http:\/\/notonbluray.com\/blog\/wp-content\/uploads\/2014\/03\/Float-To-Decimal-in-Python-performance-of-different-techniques.png","keywords":["Decimal","Float","Python"],"articleSection":["Essay","Python"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/notonbluray.com\/blog\/python-ieee-floats-for-decimal-conversion\/","url":"https:\/\/notonbluray.com\/blog\/python-ieee-floats-for-decimal-conversion\/","name":"Python: Pulling Apart IEEE Floats for Decimal conversion - Not on Blu-ray","isPartOf":{"@id":"https:\/\/notonbluray.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/notonbluray.com\/blog\/python-ieee-floats-for-decimal-conversion\/#primaryimage"},"image":{"@id":"https:\/\/notonbluray.com\/blog\/python-ieee-floats-for-decimal-conversion\/#primaryimage"},"thumbnailUrl":"http:\/\/notonbluray.com\/blog\/wp-content\/uploads\/2014\/03\/Float-To-Decimal-in-Python-performance-of-different-techniques.png","datePublished":"2014-04-03T12:52:19+00:00","dateModified":"2022-09-02T08:50:03+00:00","author":{"@id":"https:\/\/notonbluray.com\/blog\/#\/schema\/person\/bd31ad218b684e12e90bbc2db3f6e650"},"breadcrumb":{"@id":"https:\/\/notonbluray.com\/blog\/python-ieee-floats-for-decimal-conversion\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/notonbluray.com\/blog\/python-ieee-floats-for-decimal-conversion\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/notonbluray.com\/blog\/python-ieee-floats-for-decimal-conversion\/#primaryimage","url":"http:\/\/notonbluray.com\/blog\/wp-content\/uploads\/2014\/03\/Float-To-Decimal-in-Python-performance-of-different-techniques.png","contentUrl":"http:\/\/notonbluray.com\/blog\/wp-content\/uploads\/2014\/03\/Float-To-Decimal-in-Python-performance-of-different-techniques.png"},{"@type":"BreadcrumbList","@id":"https:\/\/notonbluray.com\/blog\/python-ieee-floats-for-decimal-conversion\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/notonbluray.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Python: Pulling Apart IEEE Floats for Decimal conversion"}]},{"@type":"WebSite","@id":"https:\/\/notonbluray.com\/blog\/#website","url":"https:\/\/notonbluray.com\/blog\/","name":"Not on Blu-ray","description":"Reviews of streaming media","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/notonbluray.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/notonbluray.com\/blog\/#\/schema\/person\/bd31ad218b684e12e90bbc2db3f6e650","name":"Link"}]}},"_links":{"self":[{"href":"https:\/\/notonbluray.com\/blog\/wp-json\/wp\/v2\/posts\/926","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/notonbluray.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/notonbluray.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/notonbluray.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/notonbluray.com\/blog\/wp-json\/wp\/v2\/comments?post=926"}],"version-history":[{"count":10,"href":"https:\/\/notonbluray.com\/blog\/wp-json\/wp\/v2\/posts\/926\/revisions"}],"predecessor-version":[{"id":949,"href":"https:\/\/notonbluray.com\/blog\/wp-json\/wp\/v2\/posts\/926\/revisions\/949"}],"wp:attachment":[{"href":"https:\/\/notonbluray.com\/blog\/wp-json\/wp\/v2\/media?parent=926"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/notonbluray.com\/blog\/wp-json\/wp\/v2\/categories?post=926"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/notonbluray.com\/blog\/wp-json\/wp\/v2\/tags?post=926"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}