Precision (Computer Science)

From Wikipedia, the free encyclopedia

In computer science, the precision of a numerical quantity is a measure of the detail in which thequantity is expressed. This is usually measured in bits, but sometimes in decimal digits. It is related toprecision in mathematics, which describes the number of digits that are used to express a value.

Precision is often the source of rounding errors in computation. The number of bits used to store anumber will often cause some loss of accuracy. An example would be to store "sin(0.1)" in IEEEsingle precision floating point standard. The error is then often magnified as subsequentcomputations are made using the data (although it can also be reduced).

Arbitrary-precision arithmeticExtended precisionIEEE754 (IEEE floating point standard)Integer (computer science)Significant figuresTruncation

Retrieved from "https://en.wikipedia.org/w/index.php?title=Precision_(computer_science)&oldid=708962480"

Categories: Computer data

This page was last modified on 8 March 2016, at 12:30.Text is available under the Creative Commons Attribution-ShareAlike License; additional termsmay apply. By using this site, you agree to the Terms of Use and Privacy Policy. Wikipedia® is aregistered trademark of the Wikimedia Foundation, Inc., a non-profit organization.

Precision (computer science) - Wikipedia, the free encyclopedia https://en.wikipedia.org/wiki/Precision_(computer_science)

1 of 1 2016/4/26 上午 12:18