The same design pattern can be used to create any talking sensor, with applications abounding around home, school, work, shop, factory, industrial site, mass-transit, public space, or interactive art\/engineering\/museum display.<\/p>\n
![\"\"](\"https:\/\/mathscitech.org\/articles\/wp-content\/uploads\/2021\/04\/03-TalkingPIRsensor_06b_2021-04-03-300x186.jpg\")
<\/a>Bringing Junk Model Robots to life with Talking Motion Sensors (April Fools Prank, 2021)<\/p><\/div>\n
<\/p>\n
\n
1. The rise of low-cost voice controlled hardware<\/h3>\n
Three technology streams have converged in the past 10 years to enable low-cost voice-controlled devices:<\/p>\n
(1) An explosion of highly capable silicon for sensors and data processing. Mass-produced application-specific integrated circuits (ASICs) and ultra low-cost microcontrollers (including multimedia processors required for voice generation) now cost 3-30 cents apiece at scale (typically 10k+ pieces). This has led to the creation of dozens of niche sensors available for 50p to \u00a35.00 ea<\/a> at retail cost. <\/p>\n (2) Full stack WiFi has been added to hobbyist microcontrollers while retaining a cost-per-unit between $3-$5. Integrated WiFi allows devices to transfer input data into the cloud and let cloud servers handle the computationally intensive DSP\/AI tasks, collecting and executing the outputs. This computational load sharing allows sophisticated voice capability on low-cost devices that are able to maintain continuous connection.<\/p>\n (3) Advances in statistical learning\/machine learning\/AI have enabled more effective approaches to voice processing and speech understanding including using deep learning models to boost training\/accuracy. Digital voice assistants have been embedded into a slew of devices, initially phones and computers, and then smart speakers, TVs, home appliances, and even cars<\/a>: Apple’s Siri (2011), Amazon’s Alexa and Microsoft’s Cortona (2014), Google’s Assistant (2016), and Samsung’s Bixby (2017). We’ve gotten used to interacting with technology through voice.<\/p>\n These trends above have been amplified by four additional developments: (a) the rise of the Maker movement since 2005 (Dale Dougherty), (b) the widespread adoption of Arduino since 2005 to become the world’s largest open source hardware platform, (c) the acceleration of e-commerce in consumer electronics and components (Amazon and Ebay), and (d) the flattening of global supply chains and greater pricing efficiency with manufacturers selling directly to consumer (B2C) via the e-commerce outlets. The result is that voice integration and voice control are now within reach using low-cost devices. <\/p>\n Let’s take a look at how this has come about.<\/p>\n The artificial generation of speech has gone through five stages: (0) mechanical generation of sounds pre 1930s, (1) electromechanical generation through combination of hisses and buzzes passed through various filters and pitch adjustors from 1939 to 1970s, (2) the digital combination of pre-recorded phonemes stored in databases and combined to form speech, kicking off a genre of toys-that-talk, alongside other talking devices (3) formant synthesis<\/a> in the late 1990s in which 3 frequencies are mixed to create speech elements and to control pitch, prosody, and duration (formants are the spectral patterns making up the sounds of human speech), (4) machine learning \/ AI \/ neural networks \/ deep learning models applied to speech technology. <\/p>\n Below are some highlights:<\/p>\n 1939 – The Voder, Bell Labs, Schematic Illustrating how the synthetic modal of the human vocal tract were to be preserved, with only the mechanism for operating them changing. On the Left, the human brain causes the muscles in the human vocal tract to adjust and sound comes out of the mouth, which is analyzed into constituent parts, and the reconstructed on the right with switches and pedals.<\/p><\/div>\n 1939 – The Voder, Bell Labs, an operator would train for a year or more, and then could create 20 sounds from which, with perfect timing, could form words, and then sentences, and songs.<\/p><\/div>\n The next phase was using Linear Predictive Coding<\/a> and then in the 1990s Formant synthesis<\/a> which built upon earlier FM (frequency modulated) synthesis<\/a> and other synthesis models.<\/p>\n 1978 – Texas Instruments TMS5100 chip that modelled speech using phonemes to be strung together to form words and sentences.<\/p><\/div>\n The current stage is focused on statistical learning techniques (neural networks, AI, machine learning, deep learning), and the delivery of voice interactive digital assistants.<\/p>\n Vocal response to trigger events can be added to an existing sensor for less than \u00a35 using off-the-shelf modules that offer record\/playback functionality in either single-track (e.g. YS41F<\/a>) or multi-track (e.g. ISD1760<\/a>) mode. There are also modules to which pre-recorded MP3s can be uploaded for play<\/a>, again single-track or multi-track variants (e.g. WT2003S multi-message MP3 chip<\/a>). Typical cost for record\/playback modules is \u00a33-\u00a34 ea., with prices dropping to 50p ($0.85) ea at scale (100,000+ units)<\/a>. There are also OTP (one-time programmable) systems commonly found in talking plush toys (e.g. this one for \u00a38.75, 20s record time)<\/a>, or 2pc for \u00a312.25 (\u00a36.12 ea)<\/a>), cars, action figures, dolls, and greeting cards, and feature pre-recorded material that can be played back following a trigger event. The price at scale for OTP chips drops to $0.15-$0.22 per unit<\/a> (500-10,000 units).<\/p>\n Record\/playback chips offer single-track and multi-track modes which can be used to turn ordinary sensors into talking sensors for less than \u00a35 ea (less than 50p at scale).<\/p><\/div>\n Vocal Response<\/strong> If we needed to play multiple tracks, we could use the ISD1760 chip (pictured above right) with a programmer module that has microphone and buttons to individually record the tracks into onboard Flash memory. The programmer board conveniently has a socket (DIP-28) so that an ISD1760 chip can be physically inserted just for the recording to sounds, and then removed for installation as a bare chip into the application circuit, which allows the chips to be bought independently (\u00a33 ea in single quantities, 40% cheaper than with the programming module). The special capability of the ISD1760 provides an SPI interface for a microcontroller to select at runtime which of the multiple tracks will be played. See Nuvoton’s ISD ChipCorder range<\/a> and NuVoice range<\/a>.<\/p>\n Speech Generation on the Cheap<\/strong> Text-to-Speech parsing front-ends<\/strong> Formant Synthesis for ultra-low memory requirements<\/strong> Formant Synthesis – using 3 or more frequencies passed through band-pass filters of varying widths (Q factors) to shape sound and create vowels (left). Adding control over gain (volume) and Q (filter width) makes the sound more human (right).<\/p><\/div>\n Speech Understanding through the Cloud<\/strong> Offline Speech Understanding<\/strong> One especially interesting example is Peter Balch’s speech understanding experiments using an Arduino Nano<\/a>. The Nano is an 8-bit microcontroller with up to 32kB of FLASH RAM and processing of up to 10 MIPS, whichi s akin to that of a 1970s mainframe computer (2-32KB, 0.5 – 1 MIPS). Using this, he was able to obtain a recognition rate of 90-95%<\/a> on simple single words uttered by a single speaker against a training set of the same words from the same speaker. This is comparable to capabilities of 1970s speech recognition.<\/p>\n Several microcontroller design and manufacturing companies are bringing to market voice solutions on chip<\/a>. With the price of custom ASICs continuing to fall, the next few years may see these state-of-the-art devices that currently retail from \u00a315-\u00a340 follow the same trend and reach the sub \u00a310 and sub \u00a35 price points.<\/p>\n In the last part of this article, we’ll look at implementing the first step: pre-recorded vocal response to event triggers. Let’s build!<\/p>\n What we’re after is a talking version of a Passive Infra-Red (PIR) motion detection sensor. Here’s what we need:<\/p>\n Bill of Materials (catalog here<\/a>)<\/strong> Notice that, at scale, the price could be reduced by \u00a33 (on the record\/playback module), and \u00a31 (on the boost converter), dropping the prototype price to \u00a34.55.<\/p>\n Making and testing the talking PIR motion sensor should take approx. 2 hours (excluding tool setup) assuming intermediate electronics experience (breadboarding components, some light soldering<\/a>, and a little Arduino programming<\/a>).<\/p>\n Schematics and Prototype layout.<\/strong> [link]<\/p>\n Let’s step through the key components for the build.
\n2. A Short History of Speech Synthesis<\/a><\/h3>\n
\n
![\"\"](\"https:\/\/mathscitech.org\/articles\/wp-content\/uploads\/2021\/04\/screenshot.2043-800x253.png\")
<\/a>![\"\"](\"https:\/\/mathscitech.org\/articles\/wp-content\/uploads\/2021\/04\/screenshot.2042-800x299.png\")
<\/a>![\"\"](\"https:\/\/mathscitech.org\/articles\/wp-content\/uploads\/2021\/04\/screenshot.2044.png\")
<\/a>
\n3. Converting Ordinary Sensors to Talking Sensor for under \u00a35<\/h3>\n
![\"\"](\"https:\/\/mathscitech.org\/articles\/wp-content\/uploads\/2021\/04\/screenshot.2080-800x373.png\")
<\/a>4. Examining the Value Chain, from Vocal Response to Voice Generation, Voice Recognition and Voice Understanding<\/h3>\n
\nThe YS41F module can record up to 30s of high quality audio recording and play it back with hi-fidelity on an 8R 0.5W speaker integrated into the module. What enables the sophisticated functionality cheaply is the ultra-low cost 16-bit multimedia processor Shenzhen TR16F064B (datasheet)<\/a>, which functions as a mini voice recorder with the ability to reprogram the recording. This highly capable microprocessor costs $0.25-$0.35 per unit at scale (500-10,000 units)<\/a> and provides built-in DSP operations, 10-bit ADC, I2C\/SPI, Audio PWM output support, 64K of SRAM memory (for data) and 1MB of Flash ROM memory (for program). (What’s cool about this controller chip is that it’s a von Neumann, not Harvard architecture, meaning all memory spaces are visible to program counter, which means it is possible to implement a real 3-instruction Forth without wearing out Flash memory, or requiring pre-compiled instructions as with my 3iForth for Arduino. But of course, all of the low cost MCUs have more complicated development systems requiring specialized hardware for their programming.)<\/p>\n
\nThe ISD1760 multi-track recorder with SPI playback interface is perhaps the simplest approach to speech synthesis\/generation using a technique called concetenative speech synthesis. The user pre-records phonemes and the microcontroller creates speech by seleting dynamically which phonemes to play, at what pitch, and for how long. With more memory space, one could in principle record whole words or phrases. Examples are talking clocks and other number readers. <\/p>\n
\nA step-up in capability would have a Text-to-speech front-end parsing user inputted text into phonemes and annotating for pitch, intonation (prosody) and duration, and then generating speech waveforms using the pre-recorded phonemes. This is how the more expensive EMIC2 module (c.\u00a365)<\/a> works (demo video<\/a>), or the latest S1V30120 from Epson, at \u00a339<\/a><\/p>\n
\nTo get away from pre-recorded phonemes, the next step is so-called Formant Synthesis, which creates the phonemes of human speech through mixing frequencies and judiciously selecting the band-pass filters. Vowels and consonants can be created. More sophisticated control is required to generate the speech, but only frequency data and lookup tables need to be stored instead of waveforms, significantly reducing the amount of memory needed.<\/p>\n![\"\"](\"https:\/\/mathscitech.org\/articles\/wp-content\/uploads\/2021\/04\/screenshot.2045.png\")
<\/a>
\nThe above sections have been about speech generation. But what about the other end of the value chain, speech understanding? The most advanced approach is to leverage the cutting edge statistical\/machine learning\/AI available in the cloud which underpins high performance voice processing, speech understanding, and speech generation capabilities (think Alexa’s vocal clarity, exceptional speech recognition and understanding<\/a>). This is now accessible using low-cost WiFi-enabled microcontrollers that can be continuously connected to the cloud, e.g. Espressif’s ESP family of WiFi capable microcontrollers<\/a> intended for internet-of things devices. Building in Alexa integration<\/a> via Alexa’s cloud-based Skills API<\/a> gives true voice-controlled hardware. Example: Microchip’s Alexa Development Kit<\/a> sold for \u00a3480ea (MOQ20)<\/a><\/p>\n
\nCan acceptable speech understanding be performed offline? There are a few off-the-shelf low-cost ASICs that do offline speech recognition and speech understanding and appear to have reasonable performance. Examples:<\/p>\n\n
\n5. Building a Talking PIR Motion Detection Sensor<\/h3>\n
\nTotal: \u00a38.55<\/strong><\/p>\n\n
\n<\/a>
\nDesign Video<\/strong>
\n