New ConKit (10077) eating up System resources!

[Spybot]

Hello!

I don't know if this is just happening to me... but I've noticed something strange on my laptop, the CPU fans go all the way up as soon as I open the ConKit program, you can also check this by opening task manager and under the processes tab, you'll find the percentage of CPU ConKit is consuming.
Is this normal?
Not to mention the battery life of my laptop due to this situation and that ConKit now crashes more frequently than it used to.

On the other hand, this is not happening with the virtual calculator (emulator).

[Dougggg]

I does use alot of CPU mine is at 25-28%. wonder what it is doing

[mandresve]

It happen in previous releases too. At least for me...

[Carlos295pz]

I spent sometimes with the virtual calculator, but never with the ConKit, is now inevitable when programming.

[eried]

It is probably updating the calculator screen projection... in a quite ineficient way?

[Carlos295pz]

(04-25-2016 05:20 AM)eried Wrote:  It is probably updating the calculator screen projection... in a quite ineficient way?

Apparently have this problem even if no connected calculators

[Thomas_Sch]

I also noted this behavior.
If i remember correctly, the last version didn't eat up cpu resources.
Should be an error.

[eried]

OK, after checking it I can confirm the issue. In my system it uses 15% to 20% constantly too.

Strangely it appears to be doing several things constantly, first something related with HID usb polling:


Also is polling the filesystem, the Conn kit content folder constantly, 12 times per minute, every 5 seconds:


And the registry, constantly, 30 times per minute, every 2 seconds:


Someone will have to tell the recently graduated from CS coding this part that there are better ways to notify about USB device insertions and filesystem changes than timers:

[Tim Wessman]

(04-25-2016 02:20 PM)eried Wrote:  Someone will have to tell the recently graduated from CS coding this part that there are better ways to notify about USB device insertions and filesystem changes than timers:

Not in a cross platform way unfortunately for USB...

Also, not using QT for filesystem stuff unfortunately as well (once you have network drives involved especially)...

Thanks for the pointers I guess... :-P (which program are you using for anlysis there? Not familiar with that one)

(we're working on the resource thing - it isn't actually any of the things you pointed at however. Those have already been eliminated as the cause. Also, remember that pretty much everyone here is looking at the connkit as a "i plug in my single calculator and do stuff with it" - we have to take into account much larger things like having 60+ calculators doing wireless, upcoming network connections to other computers/apps over TCPIP, etc. Some pieces don't make sense until you look at that side of things as well)

[eried]

(04-25-2016 03:59 PM)Tim Wessman Wrote:  Not in a cross platform way unfortunately for USB...

Also, not using QT for filesystem stuff unfortunately as well (once you have network drives involved especially)...

For USB it might be possible to add some little platform specific code to listen for device changes and throw the other routine, something like: https://ic3man5.wordpress.com/2012/04/29...ng-qt-4-8/

And for the filesystem there might be something for QT: http://doc.qt.io/qt-5/qfilesystemwatcher.html ?

(04-25-2016 03:59 PM)Tim Wessman Wrote:  Thanks for the pointers I guess... :-P (which program are you using for anlysis there? Not familiar with that one)

The diagnostics apps are process monitor and process explorer.

(04-25-2016 03:59 PM)Tim Wessman Wrote:  (we're working on the resource thing - it isn't actually any of the things you pointed at however. Those have already been eliminated as the cause. Also, remember that pretty much everyone here is looking at the connkit as a "i plug in my single calculator and do stuff with it" - we have to take into account much larger things like having 60+ calculators doing wireless, upcoming network connections to other computers/apps over TCPIP, etc. Some pieces don't make sense until you look at that side of things as well)

I am not sure how these majestic scenarios should impact a humble single calculator user constantly wasting a quarter of his cpu showing an empty window in the background. It is like if Excel worked sluggish when the user wanted to do his monthly accounts because Excel can handle multi-billion digit accounts.

[Tim Wessman]

(04-25-2016 04:57 PM)eried Wrote:  For USB it might be possible to add some little platform specific code to listen for device changes and throw the other routine, something like: https://ic3man5.wordpress.com/2012/04/29...ng-qt-4-8/

Unfortunately, it doesn't always work...

Quote:And for the filesystem there might be something for QT: http://doc.qt.io/qt-5/qfilesystemwatcher.html ?

Also doesn't work - especially with network drives.

Quote:calculator user constantly wasting a quarter of his cpu showing an empty window in the background.

It doesn't for everyone. It has proven difficult to track down as it isn't on every system. :-(

[Marcus von Cube]

I'm pretty sure that polling in intervals of seconds or even fractions of seconds is harmless and is definitely not the cause of the CPU hogging. This must be some thread which does not release its resources, something like a busy wait or some other (unintended) loop.

[jte]

(04-25-2016 04:57 PM)eried Wrote:  It is like if Excel worked sluggish when the user wanted to do his monthly accounts because Excel can handle multi-billion digit accounts.

Algorithms that scale better (e.g., have better asymptotic characteristics) often do have larger fixed (more precisely: larger constants involved in their resource-consumption bounds) costs. They're not just more work to implement correctly (which is a real limitation), but also require more computational resources for smaller / simpler problem instances. (This doesn't take into account the additional requirements placed on / features expected of software over time. What a word processor does on an 8-bit home computer of the 70s / 80s is quite different than what is done by a word processor on a 64-bit home computer of the 10s.)

[cyrille de brébisson]

Hello,

USB-HID scanning is done once every 2s, so this is most likely not the CPU hog. As Tim pointed out, on windows, the notification method does not always work well. Which is why we had to twitch to pooling.

For the QT file system threads, as Tim pointed, out, the method does not work on networked drives (and teachers often have network mounted drivers from servers)...

However, we are aware of the issue and will attempt to fix it. I hate wasting CPU cycles!

Cyrille

[jebem]

I had a look into this issue as well.
I have just installed the Connectivity Kit on a Windows 8.1 laptop having SSD drives and a Intel i7 dual core with hyper-threading enabled processor (total of 4 "cores" on Windows).

I have no virtual calc or psychical calc connected.
As soon a I run the Connectivity Kit exe, the Connectivityit.exe process is consuming 100% of the available CPU Core resources, running in a very tight program loop.

This is a single-threaded program, so if you have a multi-core processor you will not see 100% total processor usage because Windows will schedule the process threads, one at a time, to one of the cpu cores in a round robin fashion.

To see this 100% scenario, just set CPU affinity on this Connectivityit.exe process (in task manager or in process explorer): then you will be able to see that the process consumes 100% of this core.

I have to say that this is old news. I remember to see this behavior in older program versions as well. I just don't remember which ones.

While this behavior is annoying, it is nearly harmless as long as your Processor has adequate temperature monitoring and adequate refrigeration/ventilation.
Of course the electrical power consumption will go up as well, and this is important if you are running the program on a battery powered laptop.

Basically the process is looping doing some kind of I/O control functions (not the regular data read/write I/Os), checking the file system and the registry at a very high rate, depending on how fast is your PC system.

There are plenty of tools to monitor this, starting with the MS sysinternals.
(I remember Mark Russinovich presentation at one of the Microsoft Teched in Barcelona long time ago, just after he joined Microsoft. Really enjoyed and learned from his Windows debug presentation. What an expert he is!).

This very busy looping activity generates the following metrics:
- CPU average of 57% user time (non privileged, application level);
- CPU average of 43% privileged time
- 63 I/O other Operations/sec (I/O control functions)
- 1350 Interrupts/sec
- 118 DPCs Queues/sec
All the other metrics are close to zero activity. So no I/O data access to disk, no physical or virtual memory issues, no network activity, just a very busy program loop consuming as much CPU as possible from a single core.

As I see it, scalability doesn't need this program behavior in order to cope with all the extra work to manage the maximum number of calculators connected to it.

As others have remarked, polling should not cause this heavy CPU usage behavior IF the poll rate is low enough.

What we see here is that the Windows kernel returns from the system calls but the program is calling it back all the time, hence you have 43% usage in Kernel and 57% usage inside the program itself (user time).

Probably all here knows about this one, but here it goes:
This tight loop cpu usage behavior is easy to simulate in Windows. One just needs a text editor and type something like:
@echo off
:loop
go to loop
Then save the file as test.bat, run it and set the cpu affinity for the process. We will see about 80% or more of cpu usage.

This reminds me of the good old days of MS-DOS when the Command.com was consuming 100% of the CPU only because it was idling in a really tight loop (this was a MS-DOS design feature, not a bug).
How do I know and care about this?
Well, because some VMware customers used to run MS-DOS legacy applications, we consolidated them as VM's. Then we noticed this performance issue caused by the Command.com behavior. For those interested, I used a well know 3rd party TSR, but really, one just needs to set a cpu usage limit on those legacy VM's to fix it.

[eried]

(04-25-2016 10:16 PM)jte Wrote:  

(04-25-2016 04:57 PM)eried Wrote:  It is like if Excel worked sluggish when the user wanted to do his monthly accounts because Excel can handle multi-billion digit accounts.

Algorithms that scale better (e.g., have better asymptotic characteristics) often do have larger fixed (more precisely: larger constants involved in their resource-consumption bounds) costs. They're not just more work to implement correctly (which is a real limitation), but also require more computational resources for smaller / simpler problem instances. (This doesn't take into account the additional requirements placed on / features expected of software over time. What a word processor does on an 8-bit home computer of the 70s / 80s is quite different than what is done by a word processor on a 64-bit home computer of the 10s.)

I don't see the point of your comment. We are talking about sending things to a calculator or processing the LHC data from a particle collision?

It is just not really acceptable to waste a whole cpu core for an empty window, no matter how it is explained.

Anyhow, the issue is within just one thread of the app, maybe the timer is not properly implemented to go to idle?:


Here is how it looks:
https://cloudup.com/iC9aB0IpWX7

And after suspending/killing the thread, it all goes to normal:
https://cloudup.com/it8MvDOQle0

Via advanced decompilation methods I can see the code generating this issue:

Code:

while(true){
  int n; // TODO: Change to long to increase the delay
  for(;;){try{n++;}catch(){break;}} 
  checkUSB();checkFiles();
}

Hahaha just kidding... that code actually catches the memory leaks.

[jrozsas]

I performed a test on my notebook (Samsung I5) and there is also a large consumption of CPU simply opening the program
Initially CPU consumption is as follows:


After opening the program:

The program increase of 4 - 6% to 32% CPU usage, without opening the emulator or connect the calculator.

[jte]

(04-26-2016 02:43 PM)eried Wrote:  

(04-25-2016 10:16 PM)jte Wrote:  

(04-25-2016 04:57 PM)eried Wrote:  It is like if Excel worked sluggish when the user wanted to do his monthly accounts because Excel can handle multi-billion digit accounts.

Algorithms that scale better (e.g., have better asymptotic characteristics) often do have larger fixed (more precisely: larger constants involved in their resource-consumption bounds) costs. They're not just more work to implement correctly (which is a real limitation), but also require more computational resources for smaller / simpler problem instances. (This doesn't take into account the additional requirements placed on / features expected of software over time. What a word processor does on an 8-bit home computer of the 70s / 80s is quite different than what is done by a word processor on a 64-bit home computer of the 10s.)

I don't see the point of your comment.

The point of my comment was to state that I wouldn't assume that later version of Excel that are designed to better handle larger and more complex data sets (than earlier versions of Excel) are as efficient at handling basic data sets as older, simpler, versions of Excel. This effect would be masked by the steady increase in computational power available to the typical end user (which happens when they upgrade their hardware) - much of this happens within the noise floor (from the perspective of a typical user - I'm not talking about running precise timers / diagnostics etc.) nowadays (it doesn't invariably lead to observed sluggishness - but observed sluggishness certainly was the case as programs became more sophisticated decades back).

(04-26-2016 02:43 PM)eried Wrote:  It is just not really acceptable to waste a whole cpu core for an empty window, no matter how it is explained.

I wasn't trying to explain this as being acceptable. I didn't write this part of the code and haven't looked at it. I wouldn't, however, immediately assume that it is "wasting a whole core" - or at least wouldn't phrase it that way as that tends to be an oversimplification. It may, instead, be consuming a whole core as it is not needed (being requested) for anything else (its use of the core may drop dramatically if another process starts consuming more cycles; tight polling loops, e.g., can keep a core busy and prevent a reduction in energy consumption / heat production on modern CPUs - tight polling loops may thus waste energy but nevertheless step out of the limelight when another process starts to use more cycles). To be excessively pedantic: I'm not saying that this is ideal (or acceptable or whatever else: I'm just saying what I said…)