Transcribing audio using Whisper AI

[WilliamChew]

Kevin Stratvert describes how to use whisper AI to run on Google Colaboratory (a free service) to transcribe audio files to text.

 

The Results:

Transcribing a 40 min talk: "2011-MemorialTalk-HarfmannMartin.mp3" took about 15 mins.

The results were excellent. There were very few errors in transcription. I'm not sure if its because the audio quality was good and the speaker also spoke clearly.

 

Best FREE Speech to Text AI - Whisper AI

 

 

Instructions for transcribing are summarised here:
https://kevinstratvert.com/2023/01/19/best-free-speech-to-text-ai-whisper-ai/

 

[Dages]

I tested with a song in French... that isn't great  

Now a normal video, and it translated it in English, but it's weird.  

Edited February 13, 2023 by Dages

[WilliamChew]

2 hours ago, Dages said:

I tested with a song in French... that isn't great  

Now a normal video, and it translated it in English, but it's weird.  

 

Do you mean to say Whisper AI is also able to translate French audio to English text?

[Dages]

14 minutes ago, WilliamChew said:

Do you mean to say Whisper AI is also able to translate French audio to English text?

it's a very weird output

[luisenriquereyes]

ThioJoe describes how he created a tool called "Auto Synced and Translated Dubs", to create YouTube videos that will let you switch the audio track to one of several languages, and hear a dubbed spoken version, instead of just having to read translated subtitles like you'd usually do.

 

 

[WilliamChew]

21 hours ago, luisenriquereyes said:

ThioJoe describes how he created a tool called "Auto Synced and Translated Dubs", to create YouTube videos that will let you switch the audio track to one of several languages, and hear a dubbed spoken version, instead of just having to read translated subtitles like you'd usually do.

 

 

 

At 12:37, ThioJoe talks about how impressed he is with whisper AI:

 

Now, if you're wondering about my whole workflow, like if you're looking to do this yourself, to transcribe the videos, I actually use OpenAI's "Whisper" model, which is free and you can run it locally on your own GPU if you have a powerful enough one. And I have found this model to be way more accurate than anything else I've seen,even Google's transcription API.

 

The Whisper thing even does punctuation pretty decently. I mean the first time I tried it, I put in a 4,500 word video that was 22 minutes long and it missed three words. I was astounded.

 

It's not always that good, but it's a really good starting point from which you can just play the video and then correct any minor changes you see as you go through it.

 

[rockieboi]

I have been using WhisperAI for months.  I've converted most English talks and even some Tagalog ones from jwstream to text.  It is incredible.

[WilliamChew]

6 hours ago, rockieboi said:

I have been using WhisperAI for months.  I've converted most English talks and even some Tagalog ones from jwstream to text.  It is incredible.

 

Do you use the google colaboratory method or do you run whisper AI on your windows desktop?

[rockieboi]

I have been using google collab.  I do not know how to run it on my own computer yet.  Is it much faster?  I have an RTX2060 GPU.

[WilliamChew]

1 hour ago, rockieboi said:

I have been using google collab.  I do not know how to run it on my own computer yet.  Is it much faster?  I have an RTX2060 GPU.

 

I've been using google colaboratory too. My GPU is an NVIDIA GTX 1650. Entry level GPU.

[rockieboi]

I have been using Google collab so much that it came to a point that I can't use their free GPUs anymore.  I was using it to transcribe 10 meetings per day.  I have almost 1000 recordings from JW stream to transcribe and I have done around 200 already, and now I am only doing around 10-15 per week so that it won't limit my usage anymore.

[WilliamChew]

I'm now running whisper locally using my GTX 1650 Ti 4GB graphics card.

I find the tiny.en model good enough for english transcriptions and its very fast.

The largest model I could run was the small.en which took 3.6GB of GPU memory.

 

The Details:

Spoiler

Install Python

Installed Python 3.10.9

 

Install CUDA

CUDA 12.0 - cuda_11.7.0_516.01_windows.exe

 

To install PyTorch:
pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu117
pip show torch (to enquire version installed)
torch 1.13.1+cu117, torchvision 0.14.1+cu117, torchaudio 0.13.1+cu117

 

Install ffmpeg

ffmpeg 5.1.2

 

Install GIT

winget install --id Git.Git -e --source winget
Installed Git for Windows 2.39.2

 

TroubleChute's Youtube Link of how to do it:
https://youtu.be/XX-ET_-onYU 

The FREE & OFFLINE Audio to Text | Whisper: Install Guide | OpenAI Whisper | ASR
(by TroubleChute)

 

 

Installation Instructions in text can be found here:
https://hub.tcno.co/ai/whisper/install/

[rockieboi]

I have not used the tiny model yet, but I used the small, base, medium, and large.  I find that medium is the most accurate for almost all of my meeting audios including the non english ones.  I tried the small and base models to make it faster, but there are a lot that I need to correct.

Thanks for the instructions by the way.  I might try it if I can run it overnight with my computer.😁

[WilliamChew]

whisper-desktop gives a simple interface to transcribe audio to text.

I've had good results with the medium model. The tiny model doesn't work.

 

The 2023 RC Fri AM 1 720P file with audio of 1:51:16 took 34 mins 48 sec using the medium model.

 

My PC has an AMD Ryzen 5-5600G CPU with integrated Radeon graphics. 

 

 

 

Example of the debug console while transcribing.

 

 

 

Task manager showing how the Radeon graphics GPU is heavily loaded while the CPU is idling when transcribing:

 

 

 

 

[WilliamChew]

Subtitleedit now integrates speech to text transcribing.

It allows using either the whisper or vosk/kaldi transcribing methods.

 

This is the link to the subtitleedit help file on audio to text:
https://www.nikse.dk/subtitleedit/help#audio_to_text

 

The following Const-me option gives good results:
A GPU optimized version called Whisper Const-me (written in C++)

 

 

 

 

[WilliamChew]

This makeuseof.com article talks about whisper desktop and also give instructions on how to install and use it.

 

How to Turn Your Voice to Text in Real Time With Whisper Desktop
BY ODYSSEAS KOURAFALOS
PUBLISHED MAY 19, 2023

 

Whisper Desktop is a third-party GUI for Whisper, made for everyone who'd prefer to click buttons instead of typing commands.

 

Whisper Desktop is a standalone solution that doesn't rely on an existing Whisper installation. As a bonus, it uses an alternative, optimized version of Whisper, so it should perform better than the standalone version.