DAEMLABS
When we started experimenting with speech enhancement, the challenge was clear: real-world audio is rarely recorded in perfect conditions. Hospital environments, busy streets, crowded rooms, and industrial settings all introduce noise that makes speech difficult to understand. Traditional filtering techniques struggle with complex, non-stationary noise, so we built a deep learning-based denoising system using a U-Net architecture trained directly on spectrograms.
This is the story of how we built an end-to-end speech denoising pipeline — the architectural decisions, the training strategy, and the lessons learned along the way.
Noise removal sounds straightforward until you encounter real-world recordings.
Simple approaches such as low-pass filters or spectral subtraction can remove certain frequencies, but they often distort speech in the process. Human speech overlaps heavily with environmental noise, making traditional signal processing techniques insufficient for many scenarios.
We needed a system that could:
Rather than attempting to directly predict clean waveforms, we chose to operate in the frequency domain using spectrograms, where speech and noise patterns are easier for neural networks to distinguish.
Audio signals in the time domain are difficult for convolutional models to interpret directly.
Using the Short-Time Fourier Transform (STFT), we convert audio into a time-frequency representation where each pixel represents energy at a specific frequency and time.
The resulting spectrogram resembles an image:
Time →
┌─────────────────────┐
│ │
│ Speech Harmonics │
│ │
│ Noise Floor │
│ │
└─────────────────────┘
↑
Frequency
This transformation allows us to leverage computer vision architectures for audio processing.
Instead of predicting clean speech directly, the model learns an Ideal Ratio Mask (IRM) — a mask indicating how much of each frequency bin belongs to speech versus noise.
The clean spectrogram can then be approximated as:
Clean Speech = Noisy Spectrogram × Predicted Mask
This framing significantly simplifies the learning problem.
The core of the system is a 2D U-Net.
U-Net was originally developed for biomedical image segmentation, but its encoder-decoder structure maps extremely well to spectrogram enhancement tasks.
The encoder progressively compresses the spectrogram into higher-level feature representations:
Input Spectrogram
↓
Encoder
↓
Bottleneck
↓
Decoder
↓
Predicted Mask
Each downsampling stage captures larger contextual patterns, while the decoder reconstructs fine-grained frequency details.
The most important architectural component is the skip connection.
Without skip connections, high-frequency speech information tends to disappear during downsampling. By directly passing encoder features into corresponding decoder layers, the network retains the detailed structure required for natural speech reconstruction.
A denoising model is only as good as the diversity of audio it sees during training.
For speech data we used the LJ Speech Dataset, containing over 13,000 professionally recorded speech clips.
For noise sources we combined multiple ambient noise datasets, including:
The challenge was generating realistic training examples.
Rather than storing pre-mixed noisy samples, we implemented dynamic noise mixing during training.
For every batch:
1. Load a clean speech sample
2. Load a random noise sample
3. Adjust volume levels
4. Mix them together
5. Generate the target Ideal Ratio Mask
This effectively creates a new dataset on every epoch and significantly improves generalization.
Training begins with standardizing all audio inputs.
Every file is:
The model receives the log-scaled magnitude spectrogram as input.
The target is the Ideal Ratio Mask computed from the clean and noisy spectrogram pair.
Training uses Mean Squared Error (MSE) loss between the predicted mask and the target mask.
Loss = MSE(predicted_mask, ideal_ratio_mask)
The model gradually learns which regions of the spectrogram correspond to speech and which belong to noise.
One practical feature we added was emergency checkpointing.
Long training sessions are expensive, and interruptions happen. If training is stopped manually, the current model state is automatically saved, allowing training to resume without losing progress.
Once training is complete, denoising a file becomes a straightforward process.
The inference workflow is:
Noisy Audio
↓
STFT
↓
Magnitude Spectrogram
↓
U-Net
↓
Predicted Mask
↓
Apply Mask
↓
Reconstruct Spectrogram
↓
iSTFT
↓
Clean Audio
A key detail is phase preservation.
The model predicts only the magnitude mask while reusing the original phase information from the noisy recording. This allows us to reconstruct a waveform without requiring the network to learn phase estimation, which is a significantly harder problem.
The final reconstruction is performed using the inverse STFT (iSTFT).
Several decisions contributed significantly to the final performance.
Creating new noisy examples every epoch prevented overfitting to specific noise recordings and improved robustness across unseen environments.
Operating in the frequency domain simplified the separation task and allowed us to leverage mature image-processing architectures.
The U-Net architecture preserved speech detail that would otherwise be lost through repeated downsampling.
Predicting a ratio mask instead of a waveform reduced complexity and stabilized training.
MSE is simple and effective, but it does not always correlate with perceived audio quality.
Future iterations would incorporate perceptual metrics such as SI-SDR, PESQ, or STOI into the training objective.
Recent architectures such as Demucs and transformer-based speech enhancement networks have shown impressive results and may outperform traditional U-Nets.
While the current datasets cover many scenarios, adding industrial, transportation, and crowd environments would further improve generalization.
The current implementation processes files offline. Supporting streaming inference would enable live denoising for calls, meetings, and voice assistants.
A deep learning-powered speech enhancement system capable of separating speech from ambient noise using a spectrogram-based U-Net architecture.
The model learns to predict Ideal Ratio Masks, dynamically trains on mixed speech-noise pairs, reconstructs clean audio through inverse STFT, and provides a practical foundation for speech enhancement applications ranging from healthcare recordings to customer support calls and voice assistant systems.
The project demonstrates how modern computer vision architectures can be successfully adapted to audio processing problems, turning noisy recordings into significantly cleaner and more intelligible speech.
Need a custom audio enhancement or speech processing solution? We build production-ready AI systems for audio analysis, denoising, transcription, and real-time voice applications.