Latent Space

Minimizing audio latency with latent space.

An AI-powered, zero-lag video calling platform.

Using AI to Decrease Latency

Using recurrent autoencoders, Latent Space compresses audio streams with embeddings specifically for music, eliminating latency and allowing for smooth real-time virtual collaboration.

Latent Space reduces audio payloads by 78%, enabling musicians and others to synchronize as closely as possible.

Want to learn more? Watch our video demo.

ClefNet

Latent Space is a TensorFlow implementation of the ClefNet paper:

Ramesh, V.; Wang, M. ClefNet: Recurrent Autoencoders with Dynamic Time Warping for Near-Lossless Music Compression and Minimal-Latency Transmission. Preprints 2021, 2021030360 (doi: 10.20944/preprints202103.0360.v1).

The main contributions of this paper can be summarized as follows:

  1. We propose an architecture for a platform with end-to-end, near-lossless audio transmission tailored specifically for online joint music production, called Latent Space.
  2. We discuss the usage of a recurrent autoencoder with sequence-aware encoding (RAES) and a 1D convolutional layer for audio compression, which we dub ClefNet.
  3. We propose a new evaluation metric for naive autoencoders (AEs), MSE-DTW loss, which combines the traditional mean square error (MSE) loss function with dynamic time warping (DTW) to prevent an increase in loss when the target sequence predicted by the AE is strictly a temporal variation of the source sequence.
  4. We detail the logistics of a live system implementation which uses the Web Audio API to extract raw audio samples in real-time to feed into our client-side model before relaying the traffic using peer-to-peer WebRTC technology.

Technical Advantages

  Smart Audio Compression

  • Uses RNAEs to compress audio into unstructured genre-based data.
  • Transfers and then decodes the data back into audio, all with real-time client-focused architecture.

  Decentralized Calls

  • Peer-to-peer WebRTC tech means your data goes directly to the other person without a server, further lowering latency.

  Complete Privacy

  • Data never leaves the browser.
  • WebRTC end-to-end state of the art encryption.

Pick a name.
Share the URL.

Simple as that.