#tacotron

[aefensteorrra]

all of the audio stuff... it’s done... I have 7GB free on my laptop still.... incredible! now tomorrow I make my python script do what it’s supposed to do and from there I delve deep into working with Tacotron which was once just a funny name in a reading for my speech synthesis class but is now something I’m spending the next 6 weeks with 

[lyingbard]

Why Nothing Sounds Quite Like LyreBird

Fans of RTVS may remember Joshua, beautiful baby boy of wayneradiotv. If you're like me, you might be wondering why Joshua stayed dead after LyreBird shut down. Why couldn't he be brought back with a different TTS? The fact is that LyreBird was a product of a very specific time in AI TTS. In March 2017, Google released a paper on Tacotron [1], one of the first AI TTS's with real success. In April 2017, LyreBird began showing off their TTS buisness [2]. As AI bros are wont to do they took that shit. LyreBird is a version of Tacotron. It incorporates technologies that would be published in the next few Tacotron papers [3] including multi-speaker, prosody encoding, and prosody prediction. And in February 2018, Tacotron 2 came out [4].

Tacotron 2 is better in every way. It's faster, better at imitation, and simpler. This makes it much more economical to run and fine-tune on a specific speaker, so every subsequent AI TTS is based off of Tacotron 2.

If you read the paper, Tacotron 1 has a lot of arbitrary and untested choices. It's clear that they published it in a hurry to prove that it could be done, but they hadn't refined it to cut the unnecessary fluff.

This brings me to why I'm writing this. I hope it's clear that I did a lot of research for this. That's because I did my best to recreate LyreBird, named LyingBard, and I've put it up for you to play with here.

You may notice though that it's not quite right. The main reason is that I had to go with a low quality version (reduction factor 5 for those who read the paper). A high quality version would take too long to train with my current set up and I'm almost certain that's what they used.

If I got about $100 in donations, I'm pretty sure I could get a high quality version trained in about a month. It still wouldn't sound exactly the same. Due to the chaotic nature of training a neural network, anything short of getting the actual files off LyreBird (now Descript's) servers won't make it sound exactly the same.

Regardless, LyingBard is here to stay. It's hosted on a free server so I have no reason to take it down. I'll be posting about updates here on this blog. I'm working towards getting custom voices ready at the moment and I've got some ideas for new features and fun toys for the future.

Thanks for reading!

Here's some sources if you wanna learn more about stuff I mentioned:

[1] https://arxiv.org/abs/1703.10135

[2] https://www.pcmag.com/news/lyrebird-can-listen-and-copy-any-voice-in-one-minute

[3] https://google.github.io/tacotron/

[4] https://arxiv.org/abs/1712.05884

[pastila-krim-top]

Голосовое Сопровождение: Ключ к Успешным Карточкам Товаров на Ozon и Wildberries

В современном мире маркетплейсов, таких как Ozon и Wildberries, конкуренция за внимание покупателей требует инновационных решений. Одним из таких решений является использование голосового сопровождения для видео в карточках товаров. Этот инструмент помогает выделить ваш продукт, привлечь внимание и увеличить продажи. Давайте разберемся, почему озвучка видео так важна и как она может помочь вашему бизнесу.

Преображение Видео: Добавьте Жизнь Вашим Карточкам Товаров

Голосовое сопровождение превращает обычное видео в живую историю, которая захватывает внимание. Современные технологии генерации синтетической речи, такие как WaveNet и Tacotron, соз��ают натурально звучащие голоса, которые передают эмоции и интонации. Это делает ваше видео более привлекательным и запоминающимся, что особенно важно на платформах с высокой конкуренцией, таких как Ozon и Wildberries.

Вовлечение Аудитории: Задержите Внимание Покупателей

Озвученные видео мгновенно привлекают внимание и удерживают его дольше. Голос, передающий эмоциональные нюансы, делает информацию более интересной и доступной. Это увеличивает вероятность того, что потенциальный клиент досмотрит видео до конца, узнает все ключевые преимущества вашего продукта и сделает покупку.

Создание Доверия: Подчеркните Профессионализм и Надежность

Качественное голосовое сопровождение демонстрирует ваш профессионализм и внимание к деталям. Когда клиент слышит приятный и уверенный голос, он начинает ��ольше доверять вашему бренду. Голосовое сопровождение помогает создать ощущение личного общения, что укрепляет лояльность клиентов и стимулирует повторные покупки.

Простота и Удобство: Доступность Информации для Всех

Видео с озвучкой делает информацию более доступной и понятной для широкой аудитории. Независимо от возраста или уровня грамотности, голосовая информация воспринимается легче и быстрее. Это особенно важно на маркетплейсах, где покупатели часто принимают решения в условиях ограниченного времени. Качественная озвучка помогает донести ключевые преимущества вашего продукта быстро и эффективно.

Экономия Времени и Ресурсов: Быстрая и Эффективная Озвучка

Использование технологий генерации синтетической речи значительно упрощает процесс создания озвучки для ваших видео. Вам не нужно искать профессиональных дикторов и организовывать студийные записи. Современные системы TTS (Text-to-Speech) быстро и качественно преобразуют текст в речь, что экономит ваше время и ресурсы. Это позволяет оперативно обновлять контент и быстро реагировать на изменения рынка.

Повышение Конверсии: Превратите Внимание в Продажи

Конечная цель любого маркетингового инструмента – это увеличение конверсии. Озвученные видеоролики делают информацию более понятной и запоминающейся, что повышает вероятность покупки. Голосовое сопровождение помогает не только привлечь внимание, но и удержать его, что ведет к росту продаж. Инвестируя в качественную озвучку, вы создаете конкурентное преимущество, которое приводит к увеличению прибыли.

Инвестируйте в Голосовое Сопровождение для Вашего Успеха

Использование голосового сопровождения для видео в карточках товаров на Ozon и Wildberries – это мощный инструмент для привлечения внимания, повышения доверия и увеличения конверсии. Инвестируя в качественную озвучку, вы демонстрируете свою приверженность к высокому качеству и вниманию к деталям, что укрепляет позиции вашего бренда на рынке.

Не упустите возможность вывести свои карточки товаров на новый уровень. Закажите профессиональную озвучку уже сегодня и убедитесь в ее эффективности на практике. Ваши клиенты оценят ваше внимание к деталям и стремление предоставить лучший сервис.

[educationtech]

How is deep learning used in speech recognition?

Deep learning speechsynthesis:-Application of deep learning models to generate natural-sounding human speech from text

Key Techniques:-Utilizesdeep neural networks (DNN) trained with a large amount of recorded speech and text data

BreakthroughModels:-WaveNet by DeepMind, char2wav by Mila, Tacotron , and Tacotron2 by Google, VoiceLoop by Facebook

AcousticFeatures:-Typically use spectrograms or mel-spectrograms for modeling raw audio waveforms

Speech recognition is afield that involves converting spoken language into written text, enabling various applications such as voice assistants, dictation systems, and machine translation. Deep learning has significantly contributed to theadvancement of speech recognition, offering various architectures and techniques to improve accuracy and robustness.

Deep learning architectures for speech recognition include Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), and Transformers. RNNs are particularly suited for speech recognition tasks due to their ability to handle sequential data. Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU) are popular variants of RNNs that address the vanishing gradient problem, enabling them to learn long-term dependencies in speech data.

Convolutional Neural Networks (CNNs) are another deep learning architecture successfully applied to speech recognition tasks. CNNs are particularly effective in extracting local features from spectrogram images, commonly used as input representations in speech recognition.

Transformers are a morerecent deep learning architecture with promising results in speech recognition tasks. Transformers are particularly effective in handling long-range dependencies in speech data, which is a common challenge in speech recognition tasks.

Deep learning techniquesfor speech recognition include Connectionist Temporal Classification (CTC), Attention Mechanisms, and End-to-End Deep Learning. CTC is a popular technique for speech recognition that allows for the direct mapping of input sequences to output sequences without the need for explicit alignment. Attention Mechanisms are another deep learning technique that has been successfully applied to speech recognition tasks, enabling models to focus on relevant parts of the input sequence for each output. End-to-end deep Learning is a more recent technique that involves training a single deep learning model to perform all steps of the speech recognition process, from feature extraction to decoding.

Deep learning hassignificantly improved the accuracy and robustness of speech recognition systems, enabling various applications such as voice assistants, dictation systems, and machine translation. However, there are still challenges to be addressed, such as handling noisy environments, dealing with different accents and dialects, and ensuring privacy and security.

In summary, deep learninghas revolutionized speech recognition, offering various architectures and techniques to improve accuracy and robustness. Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), and Transformers are popular deep learning architectures for speech recognition tasks, while Connectionist Temporal Classification (CTC), Attention Mechanisms, and End-to-End Deep Learning are popular deep learning techniques for speech recognition. Despite the significant progress made in speech recognition, there are still challenges to be addressed, such as handling noisy environments, dealing with different accents and dialects, and ensuring privacy and security.

There are some courses inmany courses like this in which one of the Top Engineering college in Jaipur Which is Arya College of Engineering & I.T.

[ardhra2000]

Speech Synthesis

Speech Synthesis, also known as text-to-speech (TTS), is the artificial production of human speech. It's a technology that converts written information into spoken words, allowing computers and other devices to communicate information out loud to a user or audience.

The purpose of speech synthesis is to create a spoken version of text information so devices can communicate with users via speech rather than just display text on a screen. This aids in accessibility and improves user experience by enabling a more natural form of communication.

Automated customer service is another common application of speech synthesis. By synthesizing human-like speech, customer service bots can create a more personal experience for the customer.Allow users to adjust the voice and speaking rate to suit their listening preferences. Providing a choice of voices, including different genders and accents, can greatly enhance user experience.

Voice-powered assistants like Amazon’s Alexa and Apple’s Siri use speech synthesis to interact with users, providing a seamless hands-free experience.

Deep learning-based speech synthesis methods are gaining popularity. Models like Tacotron and WaveNet have shown promising results in generating high-quality speech.

Modern speech synthesis leverages deep learning techniques such as recurrent neural networks and convolutional neural networks for generating natural-sounding speech.

Text-to-Speech engines convert written text into spoken words, providing the backbone for speech synthesis by generating audible speech from raw text inputs.

[kevin-roozrokh]

Unraveling the Differences: ChatGPT, DALL·E, Google’s PALM2 AI, and Google BARD AI | Guide to Popular AI APIs

Exploring AI Tools: A Comprehensive Guide https://medium.com/@kroozrokh/unraveling-the-differences-chatgpt-dall-e-googles-palm2-ai-and-google-bard-ai-guide-to-567535ef0e57

Introduction: Artificial Intelligence (AI) has revolutionized various industries, enabling innovative applications and solutions. In this blog post, we’ll delve into the differences between ChatGPT, DALL-E, Google BARD AI, and Google’s Palm2 AI. We’ll also explore popular AI tools such as OpenCV, spaCy, NLTK, CoreNLP, YOLO, TensorFlow, DeepSpeech, Tacotron 2, and Apache Mahout. Additionally, we’ll highlight companies that have developed applications using these AI tools.

Understanding ChatGPT, DALL-E, Google BARD AI, and Google’s Palm2 AI: 1. ChatGPT: ChatGPT, developed by OpenAI, is an advanced language model based on the GPT (Generative Pre-trained Transformer) architecture. It can generate human-like text responses given a prompt. ChatGPT excels in natural language understanding and has been trained on a vast corpus of text data.

2. DALL-E: DALL-E, also developed by OpenAI, is a groundbreaking AI model that generates unique and creative images from textual descriptions. It combines elements of GPT and generative adversarial networks (GANs) to produce visually stunning and conceptually novel images.

3. Google BARD AI: Google BARD AI (Basic AI for Research and Development) is an AI platform developed by Google. It offers a suite of tools and services for researchers and developers, allowing them to build and deploy AI models with ease. It provides access to pre-trained models, tools for data preprocessing, and scalable infrastructure for training and inference.

4. Google’s Palm2 AI: Google’s Palm2 AI is an AI model developed by Google that focuses on multimodal learning, combining text and image understanding. It leverages advanced techniques like self-supervised learning to achieve state-of-the-art performance in various tasks, such as image classification and text understanding.

Exploring Popular AI Tools: 1. OpenCV: OpenCV (Open Source Computer Vision) is a widely-used open-source library for computer vision tasks. It provides a comprehensive set of tools and functions for image and video processing, object detection, facial recognition, and more.

2. spaCy: spaCy is a popular natural language processing (NLP) library. It offers efficient text processing capabilities, including tokenization, named entity recognition, part-of-speech tagging, and dependency parsing. spaCy is known for its ease of use and performance.

3. NLTK (Natural Language Toolkit): NLTK is a Python library that provides a wide range of tools and resources for NLP. It includes functionalities for text classification, sentiment analysis, stemming, tokenization, and more. NLTK is often used for research and educational purposes.

4. CoreNLP: CoreNLP is a Java-based NLP library developed by Stanford University. It provides robust and accurate NLP capabilities, including tokenization, part-of-speech tagging, named entity recognition, sentiment analysis, and coreference resolution.

5. YOLO (You Only Look Once): YOLO is an object detection algorithm that stands for “You Only Look Once.” It is known for its real-time object detection capabilities, allowing for efficient and accurate detection of objects in images and videos.

6. TensorFlow: TensorFlow is a powerful and widely-used open-source framework for machine learning. It provides a flexible platform for building and deploying various AI models, including deep neural networks, for tasks such as image recognition, natural language processing, and more.

7. DeepSpeech: DeepSpeech is an open-source automatic speech recognition (ASR) system developed by Mozilla. It uses deep learning techniques to convert spoken language into written text, enabling

applications like transcription services, voice assistants, and more.

8. Tacotron 2: Tacotron 2 is an AI model for generating human-like speech from text input. It leverages deep learning techniques to synthesize natural-sounding speech, making it useful for applications like text-to-speech systems and voice assistants.

9. Apache Mahout: Apache Mahout is a scalable machine learning library built on top of Apache Hadoop and Apache Spark. It provides various algorithms and tools for clustering, classification, and recommendation systems, making it suitable for large-scale data processing.

Exploring Companies and Their AI Tool Applications: While it is beyond the scope of this blog post to provide an exhaustive list, here are some notable companies that have created applications using AI tools:

1. OpenCV: Companies like Microsoft, Intel, and Adobe have integrated OpenCV into their software and products for computer vision tasks.

2. spaCy: Leading companies like Explosion AI, Rasa, and IBM Watson have utilized spaCy for NLP-related projects and services.

3. TensorFlow: Google, Airbnb, Uber, and many other companies have employed TensorFlow for a wide range of machine learning tasks.

4. DeepSpeech: Mozilla has utilized DeepSpeech in their Common Voice project, which aims to create open datasets for speech recognition research.

5. Tacotron 2: Companies like NVIDIA, Baidu, and OpenAI have used Tacotron 2 for generating high-quality synthetic speech.

6. Apache Mahout: Major companies such as Amazon, LinkedIn, and Twitter have leveraged Apache Mahout for developing recommendation systems and large-scale data analysis.

Conclusion: AI tools play a pivotal role in various domains, empowering developers and researchers to build cutting-edge applications. In this blog post, we explored the differences between ChatGPT, DALL-E, Google BARD AI, and Google’s Palm2 AI. We also discussed popular AI tools like OpenCV, spaCy, NLTK, CoreNLP, YOLO, TensorFlow, DeepSpeech, Tacotron 2, and Apache Mahout. Furthermore, we highlighted some companies that have successfully incorporated these AI tools into their applications, showcasing the widespread adoption and impact of AI in the industry.

Written by Kevin K. Roozrokh Follow me on the socials: https://linktr.ee/kevin_roozrokh Portfolio: https://KevinRoozrokh.github.io Hire me on Upwork: https://upwork.com/freelancers/~01cb1ed2c221f3efd6?viewMode=1

[aialgorithmicartuofw]

March 23-28 Readings

GPT-4 Creator Ilya Sutskever (Prediction Is Compression)

https://www.youtube.com/watch?v=SjhIlw3Iffs

Yan Le Cun

https://www.youtube.com/watch?v=mBjPyte2ZZo

AI and the Limits of Language

https://www.noemamag.com/ai-and-the-limits-of-language/

A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learning

https://arxiv.org/abs/2208.07860?utm_campaign=The%20Batch&utm_medium=email&_hsmi=251337027&_hsenc=p2ANqtz-9v1_R1jzaE0WAYvudEa5pguSrmrjBmT6N57OTbvZb2A1xvX0hk40aY0gAVyXpZOyPeiWDd9imdXpZwOFXqoFMQ5gDc4g&utm_content=251335039&utm_source=hs_email

Limits of Language https://www.noemamag.com/ai-and-the-limits-of-language/

Brain Controlled Attack Robots https://researchcentre.army.gov.au/rico/robotic-and-autonomous-systems/robotic-autonomous-systems-ras-strategyresearchcentre.army.gov.au \

https://www.youtube.com/watch?v=ldezLFCH9UMYouTube 

Jaron Lainer on the Dangers of AI https://www.theguardian.com/technology/2023/mar/23/tech-guru-jaron-lanier-the-danger-i[…]AR0BEumj9-Rct3gNyTLfJ74hRQW0evqGsGxDE9xR9ONvmmHjRzou0zXzc9g 

Leonardo AI

https://leonardo.ai/

Luma video to 3D

https://captures.lumalabs.ai/luma-api

Group 1

Audio samples from "Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis"

https://google.github.io/tacotron/publications/speaker_adaptation/

Vall-e Github

https://github.com/enhuiz/vall-e

PlayHT's new text to speech models that are capable of cloning any voice and generating expressive speech from text.

https://playground.play.ht/

Foucault - Power Is Everywhere

https://www.powercube.net/other-forms-of-power/foucault-power-is-everywhere/

Foucault’s Theory of Power and Knowledge

https://www.powercube.net/other-forms-of-power/foucault-power-is-everywhere/

Resemble app

https://www.resemble.ai

Resemble github  Resemblyzer allows you to derive a high-level representation of a voice through a deep learning model (referred to as the voice encoder). Given an audio file of speech, it creates a summary vector of 256 values (an embedding, often shortened to "embed" in this repo) that summarizes the characteristics of the voice spoken.N.B.: this repo holds 100mb of audio data for demonstration purpose. To get the package alone, run pip install resemblyzer (python 3.5+ is required). https://github.com/resemble-ai/resemblyzer

Resemble localization tool

https://www.engadget.com/resemble-ai-localize-voice-translation-artificial-intelligence-193517036.html

Group 3

The first movie ever made - Muybridge 1878

https://www.studiobinder.com/blog/what-was-the-first-movie-ever-made/

The Zoopraxisscope

https://www.youtube.com/watch?v=aG5erS2GNG0&t=2s

In 1888, Le Prince produced a few short films (the first of their kind). And when I say short, I mean short. Like as in two seconds short. https://www.youtube.com/watch?v=F1i40rnpOsA&t=2s

Leger Le Ballet Mecanique 1924

https://www.youtube.com/watch?v=wi53TfeqgWM&t=796s

https://antono.notion.site/How-to-make-videos FFMPG ad327577aa584642a3ec26c4896afb62

Online Video Enhancement Powered by AI

https://neural.love/video

PNG vs JPG

https://www.reddit.com/r/StableDiffusion/comments/102k4bg/png_vs_jpg_format_for_the_highest_quality/

https://huggingface.co/spaces/stabilityai/stable-diffusion/commit/3f24cd0fa4d92f6bd3cd535234ba36dd23073d99

Stable Diffusion Experimental Compression

https://arstechnica.com/information-technology/2022/09/better-than-jpeg-researcher-discovers-that-stable-diffusion-can-compress-images/

https://colab.research.google.com/drive/1Ci1VYHuFJK5eOX9TB0Mq4NsqkeDrMaaH?usp=sharing

Default for Automatic 111 is JPG not PNG

https://github.com/deforum-art/deforum-for-automatic1111-webui/blob/07ff466e36f996[…]4a57e116fb1d7d/scripts/deforum_helpers/video_audio_utilities.py

def vid2frames(video_path, video_in_frame_path, n=1, overwrite=True, extract_from_frame=0, extract_to_frame=-1, out_img_format='jpg', numeric_files_output = Fa

You So Done - Noga Erez

https://www.youtube.com/watch?v=Xn813NKlhzI

[auspicious-voice]

auspicious voice Voicebank Roadmap

Hi guys~

Spring has already arrived, and yet I am still busy... ^^ I still need to finish a lot of work before I take a huge break that should last for months!

And speaking of that, I'll be explaining what plans I have in store for this wonderful dumb vocal synth project of mine under the cut. It's mostly voicebank plans for the most part.

Please keep reading for more info!

UTAU Voicebank Plans

This one's the main part of the roadmap that remains as the highest priority for me to complete. Considering that I have a lot of plans for developing UTAU voicebanks, I'll infodump a lot of tidbits about them.

Fuwa Maria AUSPICE + & Fuwa Mario OPULENCE+

So! About these updates, they've been in development since early December of 2021. I never really intended to work on them until I realized how bad their current voicebanks were, which prompted me to make new updates completely from scratch with a new recording setup save for the microphone.

I've been spending these past few months fixing these voicebanks since I first recorded them. It was mostly rerecording faulty samples when rendered in UTAU, fixing incorrect oto.ini entries, and so on. I will say that the quality of these voicebanks have been significantly improved since then, but there's still a lot of room for improvement.

On the other hand, I am definitely considering distributing these voicebanks privately for beta testing purposes before the final release, but I may limit myself to close friends in case that happens. I feel like I am at a point where I've mostly fixed all of the errors and issues I've encountered when using these voicebanks, so I might rely on some friends to catch any issues and errors I may have overlooked and report them as feedback.

That being said, Maria and Mario's WIP updates will be their final Japanese voicebanks in UTAU for the foreseeable future. I intended these updates to be comprehensive and that it will suit anyone's needs when making content with them, so I am doing my best to make them the best they can be! So yeah, many appends in once packages along with a plethora of add-ons, with the icing on top being OpenUtau compatibility.

Plans for voicebanks in other languages

I might get around to record voicebanks in other languages ONCE I release Maria and Mario's Japanese updates. But I'll be taking a huge break once I release them, as I want to focus on other projects.

I am interested in recording Korean CVC voicebanks for Maria and Mario, but I cannot find the reclist for the life of me. So if anyone knows the link to that reclist, please let me know!! I also plan on recording English Arpasing (still have troubles finding a comprehensive reclist that won't beat the shit out of me) and potentially Tagalog voicebanks.

Potential updates for Junka Meteo & Suiden Zero

Considering that Meteo and Zero are secondary UTAU voicebanks, I don't think I will be giving them any updates? I mean they are there, but I'm getting burnt out from recording so many new voicebanks that it's not too much of a priority anymore.

I might work on them in the future depending on the circumstances, but so much has been going on lately that I can't keep track. Zero is also hard to voice act, so he might not even get a new update at all.

DeepVocal Voicebanks

I've actually made voicebanks specifically for DeepVocal! This was in 2019 I believe, and I made separate designs for my UTAUs. Not sure if DeepVocal is still used today, but I kind of like the program.

Status on Maria & Mario's DeepVocal voicebanks

So Maria and Mario's DeepVocal voicebanks have been sitting around on my computer for a long time, but because I recorded these voicebanks in 2019, the recording quality is very subpar. So I've been considering porting Maria and Mario's UTAU updates, specifically their normal voicebanks, into DeepVocal since I don't want to record from scratch. Plus there are already utilities for porting UTAU voicebanks to DeepVocal.

What about Meteo and Zero?

They're probably not going to get DeepVocal voicebanks anytime soon.

AI Voicebank Possibilities

AI voicebanks have been the hot topic of the vocal synth community, and a lot of hobbyists have been making their own AI voicebanks! The process is rather complicated as you need to record a lot of lines, label them, and let some deep learning shit do the magic, but it's worth it.

English AI

Fun fact: I have recorded an English AI text-to-speech database for Mario (using Tacotron)! You can find it here. It's pretty barebones and I'd like to update it in the future. Other than that, I might record an English AI text-to-speech database for Maria as well.

Also, is singing English databases a thing in the vocal synth community? Like is there a demand for that sort of thing? I've never really heard of databases like those so I would like to hear your opinions on it ^^

Japanese AI

I am interested in developing Japanese AI databases for Maria and Mario! I do have a corpus ready to use when I record the talking database, but I have NO idea where to start for the singing databases, because I would like to develop them for ENUNU or NNSVS or whatever it's called. This might be the first thing I'll start on once I am done with my break from recording voicebanks (which should start when I release Maria and Mario's UTAU updates).

Anything else?

Life's been pretty hectic, I'll say that. It always was to be honest, but I suppose that in late April things will calm down and that I can work on things normally. For now, I have temporarily suspended all vocal synth activities so that I can focus on work. I wrote this post as a sneak peak into what I'll be doing for the auspicious voice project, but it's mostly voicebank plans.

I'll see you guys in a while I suppose ^^

[soda-ghost]

👌 I'm this f*****g close to make Danny Phantom say f*ck.

[c0rrupt3dsp1r1t]

Calling Benny fans:

I’m not doing BStober right now because of a different project. I’m using the AI speech-to-text Whisper to transcribe Bernice Summerfield scripts for accessibility to folks hard of hearing or with audio processing issues. But I can’t do it all alone! They need corrections and formatting, and even with the AI helping (which without having an insane GPU or wanting more than 30 seconds requires making the audio file a video, uploading it unlisted to youtube and praying it doesn’t get blocked for copyright before taking it down as soon as the program is done so my account isn’t struck) it’s very tedious.

I’m doing some of the fixing myself, and gathering raw text files in word documents. They have most of the grammar already, and it’s phonetic so some words that aren’t in a typical dictionary work- but it can still easily mishear, so there are a lot of mistakes. First they need to be run through a sentence splitter so the whole thing isn’t running on as one massive paragraph, then they need to get actually fit into a script with the names, sound effects, and such, then fixes. This is still WAY better than needing to type it out manually and faster than any remotely accurate speech-to-text I’ve seen as well.

If anyone is willing to volunteer, please DM me. :)

[dilfslayer1080p]

So I've been messing with AI generated text to speech and all of this crusty ass dialogue was generated with Tacotron pinching from around 160 samples. Most of it is an absolute mess but it's pretty good for a test run. Oh and Benrey follower mod coming eventually I guess.

[gslin]

Google 發表新的 TTS (Text-to-Speech) 技術 Tacotron 2

Tacotron 是 Google 發表的 TTS 技術 (i.e. 輸入文字，請電腦發音)，而前一版的 Tacotron 的錄音可以參考「Audio samples from “Tacotron: Towards End-to-End Speech Synthesis”」，論文則是在「Tacotron: Towards End-to-End Speech Synthesis」這邊可以看到。 這一版的則是在 Twitter 上看到有人提到： Wow! I can no longer distinguish between a computer generated voice and recording of a person. #TTS #generative #DeepLearningTry the samples then the Turing test:…

[mostlysignssomeportents]

Bill Clinton sings "Baby Got Back"

Back in 2017, Google Research published a paper on using machine learning to create vocal synthesis models - just feed the system samples of someone's speech and it then hand it a script and it would read that speech in the target's voice.

https://ai.googleblog.com/2017/12/tacotron-2-generating-human-like-speech.html

Like so many of ML's greatest party-tricks, the amazing thing about Vocal Synthesis is its low barrier to entry - it's easy for amateurs to get involved and play with the system and get it to do amazing things. There's a whole subreddit devoted to it:

https://www.reddit.com/r/VocalSynthesis/

Periodically, the community there puts out a video showingcasing their work. In March, they released "Bill Clinton reads 'Baby Got Back' by Sir Mix-A-Lot."

It does EXACTLY what it says on the tin.

https://youtu.be/Jt7iFD_USwc

I'm no Clinton expert, but if you played this for me, my first reaction would be, "How did they get Clinton to recite Baby Got Back" and NOT "That is some impressive machine learning sorcery."

[acommonrose]

So this is a thing I found in an actual academic paper.

[itsbydesign]

(via Tucker Carlson reads the Book of Genesis (Speech Synthesis) - YouTube)

the real award goes to https://ai.googleblog.com/2017/12/tacotron-2-generating-human-like-speech.html

[jkottke]

Audio Deepfakes Result in Some Pretty Convincing Mashup Performances

Have you ever wanted to hear Jay Z rap the "To Be, Or Not To Be" soliloquy from Hamlet? You are in luck:

What about Bob Dylan singing Britney Spears' "...Baby One More Time"? Here you go:

Bill Clinton reciting "Baby Got Back" by Sir Mix-A-Lot? Yep:

And I know you're always wanted to hear six US Presidents rap NWA's "Fuck Tha Police". Voila:

This version with the backing track is even better. These audio deepfakes were created using AI:

The voices in this video were entirely computer-generated using a text-to-speech model trained on the speech patterns of Barack Obama, Ronald Reagan, John F. Kennedy, Franklin Roosevelt, Bill Clinton, and Donald Trump.

The program listens to a bunch of speech spoken by someone and then, in theory, you can provide any text you want and the virtual Obama or Jay Z can speak it. Some of these are more convincing than others -- with a bit of manual tinkering, I bet you could clean these up enough to make them convincing.

Two of the videos featuring Jay Z's synthesized voice were forced offline by a copyright claim from his record company but were reinstated. As Andy Baio notes, these deepfakes are legally interesting:

With these takedowns, Roc Nation is making two claims:

1. These videos are an infringing use of Jay-Z's copyright. 2. The videos "unlawfully uses an AI to impersonate our client's voice."

But are either of these true? With a technology this new, we're in untested legal waters.

The Vocal Synthesis audio clips were created by training a model with a large corpus of audio samples and text transcriptions. In this case, he fed Jay-Z songs and lyrics into Tacotron 2, a neural network architecture developed by Google.

It seems reasonable to assume that a model and audio generated from copyrighted audio recordings would be considered derivative works.

But is it copyright infringement? Like virtually everything in the world of copyright, it depends-on how it was used, and for what purpose.

Celebrity impressions by people are allowed, why not ones by machines? It'll be interesting to see where this goes as the tech gets better.