Realtime Audio Processing with LiCoRICE

Sabar Dasgupta

What is realtime audio processing?

Realtime

  • realtime =! fast
  • realtime == reliable; timing guarantees
  • hard vs soft realtime
  • Simulink, QNX, Zephyr (RTOS)
  • RT Linux kernel patch

“Realtime systems must guarantee a response within a specified time constraint [or] deadline” -Wikipedia

Audio processing

  • Audio signals (compressive sound waves)
  • converted to electronic signals
    • analog: represented by voltage levels
    • digital: voltage levels discretized and stored as floating point values

Audio processing

  • Nyquist rate
  • filtering
  • time domain vs. frequency domain
  • source separation

Audio Processing Intro + Examples slides

Tools and technologies

  • analog synths
    • manipulate voltage signals directly via circuitry
    • expensive, lots of gear
  • digital signal processing (DSP)
    • manipulate digital representations of signals
    • specialized instruction sets require high level of expertise
  • Python script
    • manipulate audio with the help of numpy, scipy, librosa, etc.
    • memory and timing issues
  • LiCoRICE
    • Python environment
    • realtime guarantees

Demo

  • take MIDI input from keyboard
  • generate audio
  • manipulate audio
  • output audio over default channel
  • split audio to oscilloscope and speaker

LiCoRICE Overview

Documentation

Github repository

LCTES 2018 Conference paper, contact Sabar for PDF

Model-based deisgn

  • directed acyclic graph
  • sources, modules, and sinks
  • internal numpy signals
  • external inputs and outputs

Model file

# matrix_multiply
#
# A model that demonstrates the math capabilities of LiCoRICE
# The math operations performed by the matmul module are statically compiled against BLAS/LAPACK using numba's pycc
#
# Note: model configuration is still in alpha stage and subject to cosmetic changes
#

config:
  tick_len: 10000
  num_ticks: 30
  source_init_ticks: 10

signals:

  m1:
    shape: (4, 4)
    dtype: float64
    history: 1

  m2:
    shape: (4, 4)
    dtype: float64
    history: 1

  matmul_out:
    shape: (4, 4)
    dtype: float64
    history: 1
    log: True

modules:

  matmul_init:
    language: python
    constructor: true       # this permits the use of constructor code to intialize the module
                            # constructors only execute once, before the ticks have begun
    out:
      - m1
      - m2

  matmul:                   # define the module
    language: python        # language the module is written in
    numba: true             # compile module using numba.pycc (if true, only a subset of python is permitted, but code is statically compiled with BLAS/LAPACK support)
    in:
      - m1
      - m2
    out:
      - matmul_out

  matmul_print:
    language: python

    in:
      - matmul_out

Module file

matmul_init_constructor.py

# import numpy as np # numpy already imported in module template

m1_init = np.random.rand(4, 4)
m2_init = np.random.rand(4, 4)

matmul_init.py

# if init only happens on first tick, only writes to first slot in history
m1[:, :] = m1_init[:, :]
m2[:, :] = m2_init[:, :]

matmul.py

matmul_out[:, :] = np.dot(m1, m2)[:, :]
# the [:,:] notation must be used or Python will create a new matmul_out
# variable instead of copying the data into the array mapped by matmul_out

matmul_print.py

if pNumTicks[0] >= 0 and not pNumTicks[0] % 5:

    print(matmul_out.shape, flush=True)
    print(matmul_out[:, :], flush=True)

Timings

  • ticks
  • measuring jitter
  • synchronization

Setup

LiCoRICE documentation

Install Python (if not installed):

curl https://pyenv.run | bash
pyenv install 3.8.12 
pyenv virtualenv 3.8.12 licorice-audio

Let me know if you’re having issues!

Install LiCoRICE

Install latest version from source:

pip install git+https://github.com/bil/licorice@main

Install versioned release:

pip install licorice


Set permissions (Linux)

Create a file /etc/security/limits.d/licorice.conf with the following contents, replacing with your system username:

<user> - rtprio 95
<user> - memlock unlimited
<user> - nice -20

Advanced setup

  • Kernel hacking

Testing your setup

Print out the help options:

licorice -h

Run sine_wave demo

  1. Install git if not installed

  2. Clone LiCoRICE examples

cd <folder-to-clone-licorice>
git clone https://github.com/bil/licorice
  1. Run sine wave example
cd licorice
./examples/sine/run.sh

Making updates

How do you want to modify the audio signal?

Some examples:

  • new filters; changing filter parameters
  • introduce noise
  • audio effects libraries
  • autotune
  • in-built driver: read in microphone data
  • custom drivers

Thank you!

Engaging with LiCoRICE:

  • issues
  • contributing
  • sharing drivers