Xbox Elite Series 2 intermittently but consistently locks up #247
Comments
|
Welcome to our community! This is an excellent example of a bug report and well done. This report has a few interesting aspects but I'm not quite sure when we are looking at xpadneo alone and can be sure of it. But the interesting part is that you found the left stick being involved. What about the right stick? Thanks to a donation from the community, I'm having the XBE2 controller, too. The current believing is that turning off rumble seems to work around the firmware lockup of the controller. But even back during development of XB1S controller support, this bug looked like the stick may be involved because I was sometimes seeing only the sticks freezing on some value while the buttons continued to work for some seconds. About your battery reporting issue: Is this with xpadneo? Because to me it looks like the battery is correctly identified as Play'n'Charge kit. Your dmesg also shows it working: "battery detected" means that we received the first battery report from the controller and could identify the bits correctly. "battery registered" means we have identified the battery type and registered it with the power supply subsystem of Linux. From here on it's up to user space to make any use of it. "shutting down" is also a battery report and will be sent when the controller automatically turns off due to depleted battery or idle timeout (no input in 10-15 minutes), or because you forced it to shut down by holding the Xbox logo button. It will (and can) never be shown if Bluetooth disconnected because we wouldn't read a battery event then. So if your controller input froze between the last two lines, it'll mean that Bluetooth is still functional. That would also explain why the Bluetooth stack doesn't disconnect because of transmit/receive timeout. I've already seen two different pictures:
I've also seen the following behavior: Run To clear some things up: If you're using xow, you're not using the Bluetooth connection. xpadneo won't be involved here, it cannot see the controller. The same is true vice-versa: xow doesn't handle Bluetooth but only the dongle connection. If you're using USB, you will be using the xpad driver. Neither xow nor xpadneo support the USB wired connection of the controller: xow may probably support it because the wireless dongle protocol and wired USB protocol are very similar but the Bluetooth protocol is completely different: Bluetooth uses HID, while dongle and USB use very similar variants of GIP - if not identical protocols. So first question would be: Did your games use rumble effects? If yes, could you disable rumble and see if behavior changes? You can use the module parameter After all, we are back again at some very weird timing bug or firmware race of the controller which only occurs in wireless mode (either one) but not in wired mode. For XB1S this was fixed by keeping intervals between rumble commands above 10ms. But that doesn't seem to be a work-around for XBE2: For me it usually crashes at the first rumble packet no matter which timing is used. I wonder what the problem is because I don't think it happens with Windows or Android. Both systems use a different kernel-side Bluetooth implementation. So the actual Bluetooth protocol implementation may matter here (which would still be a bug in the controller firmware: we should not be able to crash it by sending packets in some different order or timing). At least the Windows driver also keeps rumble command timings above intervals of 10ms, which by the way is the timing resolution of the rumble motor programming interface. Next question would be: Does xow show the same problem - tested with and without rumble? I don't expect the USB connection to show any problems, so we probably don't need to test that. Also, it's a different implementation of the protocol and we would be comparing apples and oranges. By that, it even doesn't work as a reference to compare. For xow it would but not for xpadneo. About the paddle buttons: As far as I found out, these are mapped to events in hardware. While we can see 4 bits that show the paddle state, we cannot tell them apart from A,B,X,Y because in default profile 0 (all profile LEDs are off), it'll be mapped to A,B,X,Y in hardware: The protocol shows P1-P4 bits set to 1 along with A,B,X,Y bits set to 1, so we cannot know the real state of A,B,X,Y when P1-P4 shows up. When switching to a different profile, we don't see any of those bits. So it seems like P1-P4 are only indicated by bits if they are programmed. The HID descriptor shows that there's a programming command to upload macros to the controller. I didn't yet decode that or record any programming from a Windows VM to reverse engineer it. But that's planned for v0.10 or v0.11. Currently, we will detect that the profile switch button was used and which profile number it activated. Current master branch has the proper commits in place for that. My next commit will be to detect the position of the pressure limiter for the triggers. So yes, the paddles will map to A,B,X,Y and that's implemented in hardware for profile 0, it's not a driver thing. If you switch to profile 1-3, the paddles will become invisible to xpadneo currently, not sure about xow because that is using GIP instead of HID. |
|
Thank you for the kind words, I try to submit solid bug reports so I'm glad to hear it was helpful! With that said, I may not have correctly captured the issue, looking at it. When it occurs, as far as I can tell, both the computer and the controller continue to believe they're connected until I manually disconnect the controller. I might be wrong about that, though, so I think I need to re-test. I'll do that and attach here, though it's difficult to get a reproduction due to the intermittent nature of the issue. The issue only occurs while connected through Bluetooth, I believe xow does have some degree of Bluetooth support as I've connected through Bluetooth and reproduced while only xow is installed and not xpadneo. I could double check with xow and no xpadneo, though. I'm not seeing this behavior when using the wireless dongle or USB and I agree we probably don't need to test those. I'll uninstall xow from my system and go forward exclusively with xpadneo for now to reduce confusion. I haven't tried with rumble disabled. The games do both have rumble effects, so I'll give disabling it a try. Keep in mind that may require over an hour of play time so the btmon output will probably be pretty large and may take me a while to collect. With that said, the rumble DOES work at least sometimes and doesn't always result in this behavior. I'll try to generate some rumble events, though. So, the reproduction this time is done with the following: Several minutes later the issue occurred and the controller stopped responding. Side note: I checked and the controller may actually still be working on some level. The profile select button still works--I can press the button and the profile lights change as expected. However, the jstest does not change. The final line is where it remains regardless of how long I let it sit: Axes: 0:-32767 1: 10283 2:-32767 3: 0 4: 0 5:-32767 6: 0 7: 0 8: 0 Buttons: 0:off 1:off 2:off 3:on 4:off 5:off 6:off 7:off 8:off 9:off I manually powered off the device by pressing and holding the button on the controller, then waited for that to time out on my system. Note that where it ended up in the btmon.txt is where it was when I killed the monitoring. I gave it several minutes to see if it would gracefully disconnect or something, but even after I manually powered off the controller it stayed at this final line. xpadneo-btmon.txt I killed the original monitoring and restarted with a fresh file, then reconnected the controller by again pressing the xbox button on it. I gave it some inputs on the left analog, then powered the controller off again by holding the xbox button. That's this file: So, next: despite the fact that rumble does work some times, it's possible a rumble event is connected to the issue anyway. I'm going to disable rumble. I could also try disconnecting the bluetooth from the computer side and seeing what happens. I'll try that this time though I don't think it'll work because the controller usually tries to reconnect. Might get some interesting behavior, though. |
|
Alright, trying with rumble disabled. Starting from where I ended the last comment. Ran btmon and output to file. Ran jstest and output to file. And... I'm convinced it works! Rumble had actually worked at least a little, but turning it off keeps the controller working consistently, so that's a solid work-around. I wonder if I could just turn off the trigger rumble or if it needs to be the whole thing. I mean, I'm actually fine without rumble for most games but now I'm curious. I'd upload the files I created by it's >200MB for the both of them so that's probably not great. Regardless, thank you for the assistance. If you have something you'd like me to test further, please let me know. (Edit: Nope, gotta be 100,0. Setting it to 0,100 still gets the freeze.) |
No, it actually doesn't. If you don't use xpadneo, the built-in kernel modules will take over: both The kernel built-in drivers 200+ MB logs are not very useful. In fact, If you're still seeing battery reporting problems, the output of I'm still curious why you were seeing no problems with using the dpad alone: Did that game support rumble at all? But even if, I'm not sure how we could use that to work-around the problem because we have no control over when the controller sends data to us, we won't be requesting axes and button states, it just sends it to us. The only thing I can throttle is sending rumble commands to the controller. We cannot even control what Bluetooth does because that's in a different layer but I'm sure if we could fix anything, it would be needed there. Luckily, the xow author is currently working on an in-kernel driver for the dongle. Plans are that we migrate xpadneo over to using the dongle at some later time - probably merging efforts of xow and xpadneo into one project. That's currently not possible because xow lives purely outside of the kernel. |
Yeah, that's somewhat expected: Then it's some firmware component in the controller that crashes. We already found that the firmware seems to be several autonomous parts running independently from each other, and one of them may crash and the controller doesn't always automatically recover by restarting itself. The profile LEDs are actually purely a hardware implementation: We do not control which LEDs light up. So unless you're seeing the profile announced at the same time in dmesg, some important part of the controller crashed. |
|
I see, all very interesting to know. As far as upower -d: The game I was using the dpad on (Streets of Rage 4) does have rumble in it, but I'm not certain I did enough testing to trigger the bug to occur. Neither SoR4 nor Trials of Mana feature a significant amount of rumble in them. (Edit: I can't do this right now because of other ongoing things, but I should probably just restart the computer. I've been messing around with stuff and something might be confused.) There seems to be a lot of logic in this controller in particular (for example, the triggers reporting different values when the locks are set differently), so yeah, that makes sense that some parts would fail while other parts still work. That said, I don't appear to see profiles show up in dmesg at all? Maybe because I haven't set any and it's just the default. I'm not sure, I haven't been paying much attention to the profiles. On another note, you've been super helpful. I can't donate a whole controller (but I guess you already have one so that's good!) but I'd throw a few bucks into the project if you had a donation link somewhere. |
It probably needs the work-in-progress v0.9 branch, I'm not sure if I merged those patches into v0.8. The v0.9 branch lives in the master branch currently. In my fork of xpadneo, there may be some proposed v0.9 patches not yet merged. I think currently I merged all patches for v0.9. I've seen the profile messages in your dmesg logs: These lines show up when you press the center button below the Xbox logo button: It switches the LEDs, and thus the profile. But the driver is just receiving this event, the LEDs itself are switched inside the controller firmware, we cannot control that (at least, I don't know how if we could). |
Yeah, I've already got this controller. There's a donation button somewhere but that currently goes to @atar-axis who kindly has given me write access to this repository. He also suggested that he would share donations if you'd attribute it to some feature I'm implementing. Last time, I denied his sharing offer because the feature I'm working on wasn't about the actual feature that was requested for these donations. I've set up some projects in the projects tab which you may attribute donations to. I'm not sure, tho, how closely @atar-axis is currently watching this. I'm currently the main maintainer here. |
kakra
added
0 | type: hardware support
|
May be a duplicate of #243 |
Looks fine, battery is detected properly and shows "fully charged". But I must admit that I didn't test the battery indicator yet because I wasn't using the controller much except for some tests, and put it back onto its charging pad otherwise. But I've just tested it just now and something is wrong there: While I see "battery detected", it no longer showed "battery registered" and thus it doesn't show up in upower... Hmm, that's strange. It looks a bit fishy, like hid-generic may interfere here - but that makes no sense. I'll let you know about any new findings on my behalf, I'm lacking some time today. |
|
Interesting that it shows sometimes in dmesg. When I fired it up and specifically hit the button to test, I didn't see it. Well, so it goes. Regarding upower: I may be overestimating what it's doing. When I first hooked the controller up (fresh out of the box) it was reporting 70% battery life, then I charged it and it went up to 100% so I was like "oh neat, it's reporting the current battery". However, that may have just been incidental. |
|
Yeah, the controller doesn't tell us the battery level while charging. It also doesn't tell us a percentage, it just has 4 levels. So there's a lot of guessing work. |
|
Hi, @kakra above you say this was solved for the XB1S controller, but I'm having the same issue with that controller (model 1708). I already updated the controller to the latest firmware. Also I'm running Arch Linux Should I provide more debugging information? Or handle it as a separate issue? Or am I just doing something wrong? :) |
|
@maharifu It is possible that you are using a current version of SDL2 which can bypass the driver for rumble when raw HID permissions are available. You may want to check the permissions (including ACLs) of You could also try exporting If this proves to fix the problem, we're going to need a patch for SDL2 which I'd be willing to create. Otherwise, there's still a chance that your kernel is running NO_HZ_FULL or at a very low HZ frequency which can make the timer unstable (it should be 300 Hz at least). In the current master branch I've changed the workqueue for this to run on a dedicated high-priority queue. So you may also want to try that: d55e6d4 I'm not sure how to check the model of my XB1S... But since XBE2 still has the rumble problem, there may be another issue not yet uncovered. |
@solystm I think your donation reached me. Thanks. |
Hi, thank you for your time. :) I checked the model of the controller under the batteries, just included it to make sure it's actually an XB1S controller. I double checked and I'm running the latest commit from the master branch already. Checking the permissions: I'm not sure how they're supposed to be, so I also tested after adding which makes them: Also: My kernel is compiled with: so it should be set for 300Hz, right? Finally, I tried setting |
|
After some research, I found that this is not a Linux specific bug, it also happens when used in Windows with a Bluetooth connection: |
|
I found some more funny bugs while debugging an issue with the XBE2 profile switcher: MS fixed the HID reports with the latest firmware updates and we no longer see a lot of duplicated buttons and axes. Apparently that broke some assumptions we were making over the packet format. That firmware is a real mess. No wonder that it took Windows 95 until Windows 10 to work mostly reliably. ;-) @solystm Originally, I intended to implement a delay pool for rumble. You may want to try #253 which also includes all the fixes I made on the way. It doesn't help on my computer but maybe it changes something for you. OTOH, I may have just made some stupid mistake and took a too easy route. Please try and report back. BTW: If the delay pool kicks in, it will log the incident to dmesg. |
|
I tried #253 but it didn't actually work at all, for whatever reason. The controller was recognized (I could see it in jstest) but Trials of Mana, Streets of Rage 4, and Asetto Corsa Competizione all refused to recognize it even after a restart. I swapped back to the mainline version and that worked without issue, so I'm not clear what was going on there. |
|
Does any of those games have a free demo version (preferably for Steam) so I could try? |
|
Yeah, Trials of Mana does: https://store.steampowered.com/app/924980/Trials_of_Mana/ |
|
Without revisiting the complete report, it may be a duplicate of #272 if rumble is involved. But even with ERTM enabled, I see strange problems with the XBE2 controller where it would sometimes stop sending data even without rumble involved. This may still be an issue in the Bluetooth stack and needs further investigation but there's nothing that xpadneo could do about it. With ERTM disabled, the controller crashes when it receives rumble commands but streaming of input reports works flawlessly. So as long as you don't need rumble, you could disable rumble and disable ERTM and you should be fine. With ERTM enabled, the controller rumbles just fine and as it should, but it eventually stops sending input reports if you constantly stream reports from it (like circling one of the thumb sticks smoothly and constantly around). It usually backs up again as soon as you stop sending data - but that may take seconds, or even lag input events behind. If this is purely caused by rumble, please close this report. It will be fixed by upcoming kernels (probably 5.12) and by enabling ERTM then. |
|
Interesting, good to know! I'll look out for 5.12. |
Version of xpadneo
v0.8
(I also have xow 0.5-17-gaf5b9d4 installed, but the behavior is the same with xpadneo alone, xow alone, and xpadneo with xow)
Severity / Impact
It completely prevents the controller from being used through Bluetooth for more than between 5 minutes and an hour, depending on some factors I'm not sure on. After that period it locks up and needs to be powered off and then reconnected again. Games generally need to be restarted in order to re-detect it.
Describe the bug
The controller connects fine and works well, reporting battery, all buttons working, even the underpad triggers (though that may be xow's doing?) I start playing a game (tested with Trials of Mana and Streets of Rage 4 primarily) and it's detected and works as expected in the game. However, after a period of time somewhere between 5 minutes (at the earliest) and about an hour (at the latest) the controller will lock up and stop working until it's powered off (hold Xbox button) and then powered back on again.
By "lock up" I mean the controller's input 'sticks' on whatever it was doing when the bug occurred. So, for example, if i was holding right on the left analog, it would continue holding right even if i let go entirely. Other buttons cease to register at this time. For example, pressing ABXY gets no input in game or in a controller calibration utility, when both work fine normally. The controller calibration utility will report the last state (for example, the left analog being held to the right) and no longer update. The controller stays connected to bluetooth from what i can see.
Steps to Reproduce
Connect up, start a game, and play. Eventually, it will drop the connection. With that said, it seems to be associated more with left analog stick inputs. It's never actually locked up while the left analog was neutral. That includes an extended session of Streets of Rage 4 only using the dpad, but when I switched to the left analog it locked up pretty much immediately. Given the wide range in the amount of time that passes it's difficult to say if it's related but it sure seems that way.
Expected behavior
It should continue to work normally without doing that :)
Screenshots/Gifs
I could take a video of me giving the pad inputs while nothing is registered by a game/controller calibration tool, but I'm not sure that would be terribly useful.
System information
Controller and Bluetooth information
xpadneo-btmon.txt
xpadneo-dmesg.txt
xpadneo-lsusb.txt
Additional context
Issue was reproduced on a desktop using Asus X99 Deluxe built-in bluetooth also reproduced on a Dell XPS13, both computers on Fedora 32 and roughly up to date with updates. Attempted to reproduce issue on a Windows 10 laptop, it didn't happen there but given the intermittent nature I wasn't certain it wouldn't.
I saw some other strange behavior out of the controller, including it not reporting the battery level at all even in Windows, so i ended up returning it for a second controller which works better (reports battery for instance) but exhibits this same behavior. I THINK it behaved the same way both on the firmware the controller had out of the box as well as the most up-to-date firmware (as per the Xbox Accessories app on Windows) but it definitely happens on the up-to-date one.
This only occurs on Bluetooth, I haven't had it happen with USB direct connection or the wireless adapter dongle. (I may not have given it enough time with the dongle, though.)
I'm not sure the best option for this, but capturing data while the issue occurs might help. I can't make it happen on command so I'd need to be using the controller over a period of time while logging.
The text was updated successfully, but these errors were encountered: