VOICE Scheduled Testing Session #3
Saturday 20 Oct, 2019, 8:00 UTC
Previous Session: Test Session #2
Comments by: @strypey@mastodon.nzoss.nz
VOICE (VOICE Organized Investigation of Chat Engines) is an informal app testing group, trialing free code apps to see how well they handle voice chat, especially with groups. We aim to have a group chat testing session at least once a month, usually during the weekend. Our next session is scheduled for Sunday 2 November, starting at 12:00 UTC. We currently use a Matrix room to confirm the timing of testing sessions, as well as for discussion about available apps and related topics: #voicechat:matrix.org
For our third group test, we had three testers, two on GNU/Linux, and one on Android/Linux. One in China and two in Europe.
Wire group chat
Wire is a centralized (client/server) system, like Signal or Mumble. One of the testers on GNU/Linux stress tested the Wire client (Electron) by running a number of other internet services at the same time, including a Riot (Electron), Tox client, Jami client, and BitTorrent client. The other GNU/Linux tester tried both the Electron client and the web client in Firefox. The Android/Linux tester tried both the native app and the web client in Chrome fork.
Although our original goal for the session was to test Tox, two of our testers happened to be trying out a one-to-one voice chat using Wire when the session began, so we decided to give that a quick test first. But we quickly discovered that in Wire you can't invite a third person into a one-to-one chat.
You first have to create a text chat group, then start a voice call from inside that group, using the usual phone button. Once a voice call is initiated this way, any user who is a member of that chat group can join the call at any time, by clicking a green 'join' button. The first time we tried this we ended up with one tester being able to hear the other two, but those two were not able to hear each other. On subsequent tests it worked fine, so it may have been a race conditions bug, perhaps caused by the stress testing mentioned above.
The sound quality and consistency of the voice call was pretty good, but overall it didn't seem any better than Jami. Again, this may have been because of the stress testing. We could hear each other fine most of the time, even we starting to speak over each other, but the sound did occasionally glitch out, create situations where one tester could hear the others but not be heard. Wire voice chat has excellent echo cancellation. It's one of the few free code voice chat apps I've been able to use comfortably without headphones. But this may also contribute to a loss of audio quality in group chats.
Tox chat
Tox is a distributed (peer-to-peer) chat system, like Jami. But unlike Jami, Tox is a protocol and a core library (Toxcore), implemented in a number of independently developed clients. One of our testers tried qTox on GNU/Linux, one using TRIfa on Android/Linux, and the third tested with both, using different accounts.
After a bit of tinkering, we came to the conclusion that TRIfa doesn't to support group chat at all. On qTox, there's a button for creating a text chat group, and it seems like that's necessary to create a group voice chat. Sadly, our Android/Linux tester didn't have a net connected desktop system available, so we weren't able to get more than two people into a text chat group, or test a group voice chat.
That said, one-to-one text chat performed better in both the Tox clients we tested than in Jami, the messages arriving more reliably, and providing a smoother overall UX. It will be interesting to see how group text chat compares to this, when we get a chance to test it. Two of us had previously tested a one-to-one voice call between qTox and TRIfa, including a period of video chat, and were impressed with the call quality. Since we couldn't get all three test participants into a group, we ended up testing a one-to-one call between two qTox clients for more than 1 ½ hours, with only two or three minor glitches.
This bodes well for group voice chat over Tox, but next time we try to test that, we need to make sure all the participants have at least one client that supports both group text chat and voice chat.