Support thread with Zoom on closed-captioning API design issue

1 May 2021
Tagged
Updates

11-minute read

Zoom allows using third-party captioning services in meetings, but unfortunately due to an issue with the design of their API, it is not possible for Web Captioner or other third-party services to reliably integrate with Zoom meetings. This is my support thread with Zoom about the issue.

After integrating Web Captioner with Zoom using Zoom’s third-party captioning API, I began to receive messages from users who experienced issues with captions not appearing in Zoom for particularly longer meetings. After further investigation, I discovered the design issue with Zoom’s API that I have detailed in the support thread with Zoom below. Zoom says that the API is working as intended, so unfortunately Web Captioner is not able to reliably provide captions to Zoom meetings with Zoom’s current API design. Zoom users can try Zoom’s native closed captioning and live transcription features instead of using Web Captioner to send captions to Zoom.

Because of this design issue and because Zoom now has captioning features integrated into their product, I recommend against developers trying to integrate with Zoom’s closed captioning API at this time.

Support ticket #7787300: Closed caption endpoint successfully accepts sequence numbers out of order

2020-09-17 - First message to Zoom Support

Short description: Closed caption endpoint successfully accepts sequence numbers out of order, but this causes closed captions to freeze up in the Zoom meeting for about 30 seconds and makes it hard to guess which sequence number the meeting is expecting next

Long description: Hello - thank you for providing a closed caption API in Zoom. I operate a speech-to-text application and I use the closed caption API to send captions to Zoom. I am seeing an issue where the Zoom meeting’s closed captions are almost in an unrecoverable state after sending a sequence number out of order. This makes it hard to know which sequence number the meeting is expecting next.

I attached videos for two tests. If the attachments didn’t go through, I also have links below.

Minimal reproduction used in both tests: https://jsfiddle.net/rskwnu0x/10/show (in Chrome, must be run with the –disable-web-security to allow CORS: https://alfilatov.com/posts/run-chrome-without-cors/ )

I think there are a couple issues:

  1. The /closedcaption endpoint returns with 200 success even if the sequence number I provided is out of order. If it returned with an error and a hint at what sequence number it is instead expecting, and then did not show those captions sent out of order at all, I think that would help.
  2. Once I’m out of sequence order, and if I don’t keep track very well of what sequence numbers I’ve sent, it’s very hard for me to recover for the rest of that meeting and show captions again. I am working with some organizations that use the same meeting for a period of months (so they don’t need to distribute the same link to participants), and if I do not keep track of what sequence number I last used for those meetings, as far as I can tell there isn’t a way to reset the sequence number or know what number the meeting is expecting next. Right now the best solution I can offer is to ask people to start a new meeting.

Test 1: Send a duplicate sequence number
Video: https://www.dropbox.com/s/3qe9vilkcmooyr9/zoom%20caption%20test%201.mp4?dl=0
Meeting ID: 971 5116 2410
Date: 9/17/20 1:40 pm

  1. I start a new Zoom meeting and copy the caption API token
  2. I paste it into the minimal reproduction above and have it send six captions with sequence numbers 1, 2, 3, 4, 5 and 6, with a pause of a second between each. Captions show as expected in Zoom up until this point.
  3. I then have it send sequence number 6 again and start a timer (red text).
  4. I pause for a second, and then I send sequence numbers 7, 8, 9, and 10. Zoom stays stuck on the 6 caption.
  5. Consistently after about 30 seconds, any captions that have failed to display in Zoom so far (the duplicate 6, and then 7, 8, 9 and 10), will all show at once.
  6. In the minimal reproduction, I then change the sequence number to 11 and can successfully send more captions for 11, 12, 13, 14, 15 and 16. These show up in Zoom immediately
  7. I send a duplicate caption with sequence number 16.
  8. I then send 17, 18, 19 and 20. The Zoom window is still stuck on 16.
  9. Again, after about 30 seconds, the captions that failed to send after the 16 are all sent at once.

Test 2: Skip a sequence number
Video: https://www.dropbox.com/s/trvfgoi3njh5zwv/zoom%20caption%20test%202.mp4?dl=0
Meeting ID: 961 6440 5842
Date: 9/17/20 1:42 pm

  1. I start a new Zoom meeting and copy the caption API token
  2. I paste it into the minimal reproduction above and have it send captions with sequence numbers 1, 2, 3, 4, 5, 6, 8, 9, and 10 with a pause of a second between each. Captions show as expected in Zoom up until sequence number 6, where it stays stuck on 6 in the Zoom window. I start a red timer at this time.
  3. After about 30 seconds, the captions that failed to display in the Zoom window so far (everything after the skipped number 7) now show in the Zoom window all at once.
  4. I then manually send captions with sequence number 11 - 23 in order without any issues and those captions correctly display in the Zoom window.

I know a solution here could also be that I just should keep better track of the sequence numbers, but I’m hoping there is some way that I can know when I’ve sent a sequence number out of order and correct the issue on my end. Currently the only advice I can offer is to start a brand new meeting. Any help would be appreciated.

Thanks,
Curt

2020-09-17 - Message from Mary Julyanne Ocampo with Zoom Support

Hi Curt,

Thanks for contacting Zoom Support!

In regards to your concern about your close caption freezing when you input the wrong sequence, I understand your concern and how troublesome this must have been for you. Let me help you out.

Upon going through the information you provided, The problem might be associated with your API, I suggest that you try to change this. But just to make sure kindly try reinstalling the app on your end too.

If you’re using a Windows operating system, kindly follow these steps to help get the old installation cleared, and get Zoom reinstalled and working for you.

Could you please run the CleanZoom application in the attachments, and then restart your computer when it’s finished?

Once done restarting please visit this link https://zoom.us/support/download this will automatically download the newest version for you.

Please install this version once downloaded.

I hope this helps answer your question. Please feel free to let me know if you need anything else!

Thanks,
Mary

2020-09-17 - My message to Zoom Support

Hello Mary,

Thank you for your response. The issue I wrote about above was discovered and tested on a MacOS computer. I have now installed the latest version of Zoom from the link you gave me on a Windows 10 computer that has never had Zoom installed on it before, and I reran the same tests above and can reproduce the same issue.

Test 1 video on Windows: https://www.dropbox.com/s/yo9sj5i7un8vfea/zoom%20test%201%20windows.mp4?dl=0
Meeting ID: 992 5250 0655
Date: 9/17/20 1:45 pm
The pause is about 3.5 minutes instead of the 30 seconds from MacOS test 1. The second time I try the duplicate sequence again, the pause is about 15 minutes before I see the text (the video ends before this). During this pause, there seems to be no caption event I can send that will cause captions to appear on the screen.

Test 2 video on Windows: https://www.dropbox.com/s/h9jn6ii1art7ng4/zoom%20test%202%20windows.mp4?dl=0
Meeting ID: 96631262429
Date: 9/17/20 2:10 pm
The pause is almost exactly 4 minutes before captions appear again. Then I can continue sending like normal. During this pause, there seems to be no caption event I can send that will cause captions to appear on the screen.

These videos are using the same proof of concept from my first message. The only difference is that they are running on a Windows system that has never had Zoom installed on it until now. It is running the latest version of Zoom.

I appreciate your time. Please let me know if I can provide any other details.

Thanks,
Curt

2020-09-18 - Message from Mary Julyanne Ocampo with Zoom Support

Hi Curt,

Thanks for reaching back to us. We appreciate your response and the information you provided in order for us to solve your concern.

I understand that you still experiencing an issue with your closed caption, and I know the urgency of this matter, so I would now escalate your issue to our engineering team to give you a proper resolution for the matter at hand. Right now we are having increased demand for our services, so I hope you would bear with us if we don’t get back to you at your expected time. Rest assured that we would be doing our best to give you quality service.

Have a great day!

Thanks,
Mary

2020-09-28 - My message to Zoom Support

Hello,

Any update?

Thank you,
Curt

2020-10-13 - Message from Tyler Spaeth with Zoom Support

Hi Curt,

Thank you for reaching out to Zoom Support. I appreciate you bringing this issue with closed captioning to our attention. I have forwarded this information to our dev team to resolve this issue. As soon as I have a response I’ll provide an update.

Thanks
Tyler

2020-10-16 - My message to Zoom Support

Thank you Tyler.

Curt

2020-11-03 - Message from Tyler Spaeth with Zoom Support

Hi Curt,

Thank you for your patience. I wanted to provide an update and let you know that our engineers are still working to address this issue properly. As soon as I know more about a fix release I will provide an update.

Thanks
Tyler

2020-11-03 - My message to Zoom Support

Thank you Tyler, I appreciate the update. Looking forward to hearing more about a possible resolution.

Thanks, Curt

2020-11-15 - Message from Tyler Spaeth with Zoom Support

Hi Curt,

I wanted to let you know that our engineers are still investigating the closed caption issue. As soon as I have more information I will provide an update.

Thanks
Tyler

2020-11-16 - My message to Zoom Support

Thanks Tyler.

2020-11-29 - My message to Zoom Support

Hi Tyler,

Are you able to tell me if the engineers can acknowledge there is an issue after looking at the information I provided in my first message? Is a fix planned?

I currently am maintaining a monthly Zoom subscription only so I can keep in contact through this support ticket. If I cancel my subscription, will I lose access to this ticket?

Thank you, Curt

2020-11-29 - Message from Tyler Spaeth with Zoom Support

Hi Curt,

Our engineers are working on a resolution. If you did cancel your subscription as long as you didn’t delete your basic account you would still have access to this ticket.

Thanks
Tyler

2020-11-29 - My message to Zoom Support

Sounds good, thank you very much for the update, Tyler. Looking forward to hearing more.

Curt

2020-12-07 - Message from Tyler Spaeth with Zoom Support

Hi Curt,

I want to let you know that while we are working on this issue. I don’t currently have an update. As soon as I have more information or a fix in place I will let you know.

Thanks
Tyler

2020-12-08 - Message from Tyler Spaeth with Zoom Support

Hi Curt,

So it looks like that’s actually by design. If you supply a sequence out of order it will unify after 30 seconds. Our engineers are still reviewing to improve the design. I’ll update you as those improvements are made.

Thanks
Tyler

2020-12-08 - My message to Zoom Support

Hi Tyler,

Thank you for the update. The issue I’m experiencing is that while Zoom will reconcile it after a period of time, I have no way of knowing there was an issue on my end because Zoom will accept sequence numbers out of order without any indication that something went wrong. If the closed captioning API could give some indication when it’s received a sequence out of order, that would be a huge help in keeping things in sync. I’m doing the best I can on my end to throttle requests appropriately and always send sequence numbers in order and incrementally, but I feel like I should not be able to put the closed caption API in a state where there’s nothing I can send to it for a period of time that will cause captions to show.

Thank you for keeping me updated. If there’s any other information I can provide please let me know.

Thanks,
Curt

2020-12-14 - My message to Zoom Support

Hi Curt,

I’ll request that as a feature. As of right now, it’s working as intended, but I definitely understand your use case for receiving an alert. Please let me know if you have any other questions.

Thanks
Tyler

Recent Posts