Home >> FAQs/Tutorials >> SQL Server FAQ

SQL Server FAQ - Precision and Rounding on REAL and FLOAT Literals

By: FYIcenter.com

(Continued from previous topic...)

How REAL and FLOAT Literal Values Are Rounded?

By definition, FLOAT(n) should store the mantissa of the floating number in n bits. For example, FLOAT(16) should have a precision one-byte less than FLOAT(24).

However, SQL Server 2005 only supports two precisions for floating numbers:

  • Single Precision: FLOAT(24) or REAL, stored in 4 bytes, giving about 7 digits of precision, covering all types from FLOAT(1) to FLOAT(24),
  • Double Precision: FLOAT(53), stored in 8 bytes, giving about 15 digits of precision, covering all types from FLOAT(25) to FLOAT(53).

The tutorial exercise below shows you some different precision and rounding examples:

-- FLOAT(1) works like FLOAT(24) 
DECLARE @x FLOAT(1)
SET @x = 9.234567890E+10;
SELECT @x;
GO
9.234568E+10 -- 7 digits precision

-- Single precision with rounding 
DECLARE @x REAL; -- FLOAT(24)
SET @x = 9.234567890E+10;
SELECT @x;
GO
9.234568E+10 -- 7 digits precision

-- FLOAT(25) works like FLOAT(53)
DECLARE @x FLOAT(25);
SET @x = 9.2345678901234567890E+100;
SELECT @x;
GO
9.23456789012346E+100 -- 15 digits precision

-- Double precision with rounding
DECLARE @x FLOAT(53);
SET @x = 9.2345678901234567890E+100;
SELECT @x;
GO
9.23456789012346E+100 -- 15 digits precision

(Continued on next topic...)

  1. What Is a Constant or Literal?
  2. How To Write Character String Constants or Literals?
  3. What Is a Collation?
  4. How To Specify the Collation for a Character Data Type?
  5. What Happens If Strings Are Casted into Wrong Code Pages?
  6. How To Find Out What Is the Default Collation in a Database?
  7. How Fixed Length Strings Are Truncated and Padded?
  8. How To Enter Unicode Character String Literals?
  9. How To Enter Binary String Literals?
  10. How To Enter Date and Time Literals?
  11. Why I Can Not Enter 0.001 Second in Date and Time Literals?
  12. What Happens If Date-Only Values Are Provided as Date and Time Literals?
  13. What Happens If Time-Only Values Are Provided as Date and Time Literals?
  14. What Are Out-of-Range Errors with Date and Time Literals?
  15. What Happens If an Integer Is Too Big for INT Date Type?
  16. How Extra Digits Are Handled with NUMERIC Data Type Literals?
  17. How REAL and FLOAT Literal Values Are Rounded?
  18. What Are the Underflow and Overflow Behaviors on FLOAT Literals?

Related Articles:

More...


Other Tutorials/FAQs:

More...


Related Resources:

More...


Selected Jobs:

More...