Excessive physics overhead while moving of model with big amount of beams/nodes

I noticed this issue in version 0.15, and it is still reproducible in 0.16, so I decided to post it there.

If you are driving a model with big amount of nodes (let's say - triple articulated bus), through an area with a big amount of rigid physics obljects, (trees, houses, bridges, etc) with some speed - physics graph starts to peak, while CPU and GPU are still partially idling.



All three factors are needed for reproducing the issue:
1. big number of nodes
2. big number of surroundings collidable objects
3. some conciderable speed of movement

Second part of the video shows that in case of regular model (one of vanilla vehicules) this issue is not reproducible (removing element #1).
There is no FPS drop while moving through area without big number of collidable objects (without element #2)
Stopping of movement (removing element #3) eliminates overhead as well.

That video was made with 0.15, but with 0.16 situation is still the same.
I've also got same effect on West Coast map, but there the problem was just more pronouced so I decided to use this map for demonstration of an issue.

I'm sure that there was no such issue in previous versions, because at a time of triple-articulated bus rame development I've driven a lot on all vanilla maps, West Coast included, and there were no such FPS lags.

 

Is the problem overhead or hitting limits of some thread of CPU?

 

Nope, workload distributed between cores nicely and evenly, but when peak appears, load on CPU decreases like that:

 

One increases though, having close to 90% in task manager means that in practice limit is reached as load is very fast spikes.

Try HWinfo sensors only with 200ms update rate, that might show bit better that fast spiking load, but it can be that physics spike you are seeing is from delay caused by something.

From experience I know that there are some things running mostly on one thread, which usually overloads in situations you describe.

That then makes it sometimes pretty hard to isolate which is doing what.

Technically single vehicle runs on single thread, but is articulated bus single vehicle or vehicle with trailers is of course again question.

Anyway you have single vehicle that runs on 1 thread, then you have LUA and some of graphics that often seem to share 1 thread, then static objects can be spread over several threads.
--- Post updated ---
If you have possibility to repeat exact same conditions with current clock speed and increased clock speed, you might see physics spikes disappearing, or maybe running game at lower framerate, like 30 as physics should always run at 2000Hz, but lowering framerate lowers graphics load on CPU, physics load should be same even at 30fps and issue should happen still, but if it does not happen, then it could be CPU thread hitting limits instead.

Just something to exclude other possibilities than claimed one, so it could be certain what causes the issue as these can be sometimes quite tricky to tell for certain.

 

I've made a simpler example with HWInfo running with minimum update interval as you suggested. Now it is clearly visible that there is no "peaking to 100%" thread(s), as well as that this "physics overhead" is tightly connected to model movement with some considerable speed. So I'd like to bet on some algorythmic issue there. Especially because there was clearly no such behavior in previous versions.

 

With this mod, you can find out that even in West Coast usa there is similar I believe:
https://www.beamng.com/resources/crashhard-8x8-bigrig-truck.8316/

That mod has lot of collision triangles, I was thinking maybe it is possible that collision triangles are very high in numbers with Articulated bus also.

However one thing that troubles me is that map you are using has very high graphics workload on CPU and would need to verify your findings with something that has practically no graphical workload, but has still lot of stuff for physics.

So, I came up with a test object, these are very easy to place on gridmap and graphical workload is very small, also it is very fast to build huge collision surface density if houses are piled on top of each other:


When .zip file is placed to mods folder, you should should be able to add fakehouses object to Gridmap level using F11 editor.

See on right panel, Library and meshes tab, also shows path you need navigate to:


Double click fakehouses file and click ok to dialog that appears.

Remember to choose Visible Mesh Final for collision type.

Then you can drag object to empty area outside gridmap loop and shift drag to make copy of houses so you can make as many of them as you like.

Pylons are there to increase collision surfaces.

Using that would remove graphics workload variable I believe and it could be easier to know what is happening. Also ssao and dynamic reflections off will allow more houses before GPU reaches limit, but CPU will not reach graphics thread limit with these.

What little I tested with three layers of houses, was that there were some spikes, but I got much bigger spike when I did press print screen key (not visible in screenshot):


Technically this house setup should already have much more static object collision surface that what is present within active range anywhere in Tennessee. Collision faces are only active when close enough, or should be, afaik. Anyway more testing is needed obviously to see what is really going on, but if it is number of static objects or collision surfaces, then with these houses on gridmap it should be possible to replicate severe physics spiking. Only other possibility that I can think of is that graphics workload is causing physics spike to happen, but I don't know too much of all that.

Anyway your findings are quite interesting.

 

I'm sure graphics workload is of no importance there, because peaking appears when bus arrives on the bridge, and dissipates at the moment when it stops. Also in latest video, there's 100+FPS in the beginning. But I'll check your test object later today, thanks.

 

I did try to replicate this, I had ssao off and dynamic reflections off, highest quality, AA on, DOF on, others off.

Also I did use gamepad to drive, instead of AI driving, but I think other than that I did manage to replicate test quite well, unless I missed something important. For me, there was no fps drop:


I did notice some spiking to 100% CPU / thread load, which source I don't really know, can be vehicle or graphics, imo.

Also testing with test houses on gridmap I could not get such fps drop to happen, but there can be some variable missing from my testing of course. Puzzling.

 

Well, first of all you have way better CPU (btw that 8086 index - is that an internal joke from Intel? ), which appears to crunch these red peaks before they are forming a "mountain" as in my case. I've made a fast retest with houses you provided and got a stable reproducing of the issue:


P.S. Okay, not a joke, but "Intel Core i7-8086K was released to celebrate the 40th anniversary of Intel's first ever x86 processor, the Intel 8086."

 

Very good, I think that excludes it being graphics thread that could be overloaded, Estama probably can understand much better why that is happening.

 

Somehow this thead is still ignored by devs... while I still think this issue worth investigating and fixing.

 

Well Estama is one of the Physics devs I think the other is @Diamondback the both of them could take a good hard look at this and tell you whats going on....

 

Lack of replies does not mean it is ignored, it is just that they don't have too much time to write replies here.

 

Diminishing returns applies to everything I guess, up to a point until more cannot be extracted simply by optimizations, more computing power is needed to go towards higher complexity.

@slysenko Oh yes, 8086k definitely a joke from Intel, with this silicon they could of done pretty mad CPU, instead they made just the same as 8700k which is better only on paper, no difference in practical use, but with bit of tuning one of the best for complex vehicles in BeamNG, that is correct. Also graphics, lua etc. limits are not easily manifested when running 8086k in overclock, with stock clocks still bit of those can be witnessed.

Right now I have set this to 4Ghz and 4 cores no HT for testing purposes, such flexibility makes it easy to do experiments and find out what is happening and why. 8700K would do the same and many others for sure.

 

Hello Estama!
I'm almost certain that this behaviour appeared recently (at least at this saturation), because I've used to drive in that triple-articulated bus a lot while trying to finish it and did not notice such FPS drops before.

The "test case" with fufsgfen's buildings shown the picture more clearly, especially at 09:30, when driving through same buildings at speed as low as 40 km/h did not cause any peaking at all, while with higher speeds it is there.

Is there a way to downgrade to previous version(s) so that I can retest this test case and pinpoint exact version where it "appeared"?

 

From Steam, you can choose last DX10 compatible version 0.14 I believe (right click BeamNg.drive on list, choose properties, then betas page), but other than that there is no way unless you have saved old backups or something.

 

Okay, I've installed the build from 0.14.0.5 branch and repeated the test with houses:

There are totally four passes trough buildings:
1-st pass - through "wide" gap between "blocks" of buildings, no change of FPS whatsoever
2-nd pass - through one of standard "narrow" gap between buildings, FPS dropped as low as 17
3-rd.pass - again through "wide" gap between "blocks", to confirm result of 1-st pass
4-th pass - again through "narrow" gap, repeating 2-nd pass, FPS dropped to 18, and also I tried to stop to verify that zero velocity eliminates a lag

All is all, based on "white box testing apprach" I can only assume the somewhere between 0.14.0.5 and 0.15.x there was a change of distance to geometry data to be accounted for simulation, which in turn caused this issue to be more "saturated" in recent versions.

I'm also going to try to retest "driving through bridge" testcase now.

 

You can get colors to houses if you choose option force update materials.cs during import in dialog box, I believe, or copy materials.cs from zip to same folder as test object is. I don't know if it affects to anything related to performance though.

I did disable 2 cores so my CPU had 4 cores and 8 threads, then I set maximum clock speed to 3Ghz and I had swapped GPU to 1050Ti earlier for other testing purposes.
I got most of the red stuff on graph from video recorder actually, even I was using nvsync in OBS studio, somehow there is more CPU load and it starts to show up in graph, but even hitting 100% for some threads pretty much there is not much slowdown:


Then narrow direction without video recording software running so there is about 96fps when running between houses, but then you all of sudden hit like a tar and lowest I saw was 18fps while only one thread were working maximum amount it could (Core# 3 Thread #1 is maxing out despite shows 94% there), GPU load dropped a lot:

With 3.5Ghz it drops only to 50fps or so, with 4Ghz there is really no slowdown that can be noticed in same situation.


Have you tested in 1080p resolution using normal graphics, ssao and reflections off so those CPU graphic stuff don't interfere?


I find effect much more noticeable with reduced graphics settings.
--- Post updated ---
Just for fun, here is regular bus from vanilla game with CPU at 1.5Ghz 4c/8t it amazingly does perfectly fine over 100fps until driving fast to narrow gap of test object.