IDE doing some stra...

Clear all

# IDE doing some strange rounding

10 Posts
6 Users
3 Likes
821 Views
(@fastrunner08)
Member
Joined: 2 years ago
Posts: 22
Topic starter

Hello everyone,

I have had a frustrating occurrence start happening that I'm hoping someone could point me in the direction of fixing since my googling of the problem hasn't led to an answer.

On several of my coding projects, i'm seeing some strange rounding of numbers occurring even mid-equation that is messing up results.  A few examples below.

I'm working on RPM with a rotary encoder, and followed the video/code Bill uploaded in the video.  I kept having RPM return a 0 result, so I added a lot of prints to the serial monitor to determine what is happening.  The problem is in this equation

rpm = encoderValue * (60/360);  --> returning 0 every time.  Serial monitor is showing me there is a value tied to encoderValue, so how in the world would this return 0?  after checking many things I decided to write this line of code.

myVariable = 60/360;  --> myVariable was declared as a long.   should return .167 or something there of but it returns 0.

This is making me assume that in my RPM calculation the encoderValue is being multiplied by 0 fwhich is the only way i can guess that is occurring.

On another project, i was using a 0-5 volt pressure transducer.  Again wrote code to covert analog input value to a 0-5 volt signal.  it would read 0 even though based on the analogRead, the value should have been .51.

It seems like all of a sudden, my equations are not using decimals anymore.   Is there some setting that changed, or I need to change, or am I going crazy?

(@will)
Member
Joined: 3 years ago
Posts: 2507

@fastrunner08

60/360 will probably result in zero because both 60 and 360 will be interpreted as integers and so the result will also be an integer --- zero.

Use something like float(60)/360 to force a floating point calculation (and result).

Anything seems possible when you don't know what you're talking about.

(@zander)
Father of a miniature Wookie
Joined: 3 years ago
Posts: 6893

@fastrunner08 Welcome to the world of conversion rules. It's not you, other than needing to understand the rules. It's been too long for me to tell you exactly, but some others will chime in.

First computer 1959. Retired from my own computer company 2004.
Hardware - Expert in 1401, and 360, fairly knowledge in PC plus numerous MPU's and MCU's
Major Languages - Machine language, 360 Macro Assembler, Intel Assembler, PL/I and PL1, Pascal, Basic, C plus numerous job control and scripting languages.
Sure you can learn to be a programmer, it will take the same amount of time for me to learn to be a Doctor.

(@will)
Member
Joined: 3 years ago
Posts: 2507

@fastrunner08

Generally speaking

- binary operations involving two values declared as integer (or implicitly integer like 12) will result in an integer value so 12/5 = 2, 60/360 is 0, 36/3 is 12 and so on.

- binary operations which operate on one (or both) floating point numbers results in a floating point value, so 12./5 = 12/5. =2.4, 60./360 = .16667  and 36/3. is 12.0

You can use the float() operator to change an integer into a floating value in order to force a result to be floating point.

so float(60)/360 = 60/float(360) = float(60)/float(360) = .166667

Anything seems possible when you don't know what you're talking about.

Inst-Tech reacted
Member
Joined: 3 years ago
Posts: 493

Or 60.0/360.0?

(@will)
Member
Joined: 3 years ago
Posts: 2507

Or 60.0/360.0?

That's equivalent to float(60)/float(360), so it's .166667 again 🙂

But where it gets freaky is that float(60/360) will probably be zero again. This is because the value 60/360 will be evaluated as an integer and so that result will be zero and the float will convert it into 0.0

Anything seems possible when you don't know what you're talking about.

Member
Joined: 3 years ago
Posts: 493

Posted by: @will

Or 60.0/360.0?

That's equivalent to float(60)/float(360), so it's .166667 again 🙂

But where it gets freaky is that float(60/360) will probably be zero again. This is because the value 60/360 will be evaluated as an integer and so that result will be zero and the float will convert it into 0.0

Yes, which is why I prefer to work directly with floats. Be explicit, which doesn't mean swearing when you get an unexpected answer! 😁

(@byron)
No Title
Joined: 5 years ago
Posts: 1121

@fastrunner08

Way back when I used to work with some financial software 'floating points are evil' was a mantra. 😎

But if you are interested in better results for floating point arithmetic then this link to a nice write up could be of interest.

Eliza reacted
(@fastrunner08)
Member
Joined: 2 years ago
Posts: 22
Topic starter

Thanks for the help!  That solved it.  I'm getting the right readings now.  This is the first time I needed to do that in my code.  I'm wondering if on pervious versions it defaulted to a float or something, because I remember reading a whole bunch of thermistors and never needing to declare a float mid equitation like that.

(@eliza)
Member
Joined: 2 years ago
Posts: 81

@fastrunner08 Integers are stored in memory as a binary pattern exactly representing the number.

Floating point values are stored as a normalized number between zero and one, (the mantissa) such as .2039 and an exponent such as 100.0 which would make this value 203.9. For every floating point operation, such as x * y the pairs of values are sent to the floating point arithmetic unit, (perhaps software, perhaps hardware) for processing.

The floating point processing will normalize both values relative to each other and then (say) multiply the mantissas and add the exponents, or normalize and then subtract one from the other.

Some floating point operations yield annoying/useless results because some numbers cannot be represented in memory at all. Some operations such as 10 million minus 1 might never produce any results. The 10 million never becomes 9999999 because the difference between the operands is too great such that when the mantissas are normalized relative to each other, one of the values becomes zero. Now neither addition nor subtraction has any effect.

Floating point ops in a small computer like arduino will always be fraught with problems. Larger systems, even raspberry pi, have ways (alternative numeric representations) to calculate with arbitrary precision, for astronomers and physicists etal.

This post was modified 2 years ago by Eliza