Version 1.2

WebGL and Audio Limitations

It’s well documented that WebGl does not support some audio features used within a Unity build for other platforms. It is noticeably thin when coding audio source and clip data functionality. Some of the audio enhancements to the game were made to improve immersion and performance.

1. WebGl and Audio Limitations

The GetData() method from an audio source returns an array of samples at a certain point in the audio. These were needed to calculate the motion of a volume dial and light intensity on the radio. WebGl doesn't support the GetData() method so a solution was needed to calculate peak values during the track playing.

The solution for this was to essentially bake the audio peak levels into a file and retrieve them alongside the audio at runtime.

An audio to float array class was developed to stream the audio in unity and save the peak values to a resource file in the project. To save space and time the audio was buffered every 100ms and the peak value was then calculated. These peak values were then stored in the array. This allowed a reasonable approximation of the file whilst keeping memory and file size low. 

On start the audio data was pre-loaded depending on which 2 out of 5 radio samples were used. When played each 100ms the peak value was used to move the dial and provide a light intensity. Fluid motion was provided by leaping the angle every frame the new peak wasn’t being calculated which provided a smooth dial rotation.

——

Sampling Method

A class was written to sample the audio. The class then stores the data in a file and then plays the file back. This way each radio source could have its amplitude baked into a file and then the results tested.

To get the best samples whilst also conserving space the audio was sampled every 0.1 seconds (100ms) which meant for the audio being encoded at 44100 an array of 4410 samples was taken. The amplitude was based on two different collection methods: Max Sample Value vs. Average Sample Value.  

As audio oscillates in both negative and positive ranges a Abs function was used to make sure only the positive values were summed when calculating an average. However during playback I noticed the dial tended to be out of sync with the audio. I tried to change the offset by changing the sample starting value but it still did not align up effectively enough.

I found the Max sample value produced results which were more in sync overall. This method did not need to calculate an absolute value.

——

Rendering

When playing the audio peak levels are stored in a float[] and then accused by a audioCounter as the track plays. The peak levels are then used in the rotation of the dial and the intensity of light.

Whilst testing out the rendering I used an amplitude multiplier for the dial to get more motion, but after switching to Max Sample Value it wasn’t necessary.

I also found by offsetting the sample starting value by 1 a greater syncing occurred.

——

Further Improvements

So overall, it works, but isn't great.

The audio is far from perfect and in some cases doesn’t seem to be in sync, a refinement of the sample capture perhaps with a smaller sample rate and different sample taking algorithm. Also worth exploring is writing to the file in one go rather than continuously, the frame rate and storing to a single .json file.

It's worth noting the audio source itself. Using old radio broadcasts with a high level of peak limiting does effect the audio peak level. I noticed the best results came from slow, well recorded speech.

Other options are baking an animation or finding a better way to use a DAW to digitally extract the audio more precisely - perhaps using Reaper to extract peak information using ReaScript.

 2. WebGl and Safari

While Testing the game in various web browsers Safari presented the most issues. No sound was the common problem and the error was generated in the Development build: "Loading FSB failed for audio clip 'X'". While most sites seem to present various workarounds there appears to be no obvious fix so for now safari browser will not work with the current audio. Other options to explore are more advanced audio tools such as FMOD or another third party plugin.

 3. Various Audio Import Settings

To make sure the game runs as smoothly as possible it was possible to make some audio importing optimisations. Force to mono is really useful when working with audio in a 3D space as you only really need a mono source in a space anyway, the second track is not necessary.

https://blog.theknightsofunity.com/wrong-import-settings-killing-unity-game-part...

Note: Radio Sources needed to be in Decompress On Load for their data in clip.GetData() to be accessed. Not sure if it makes a different but their Compression format was changed to PCM as the GetData() method uses PCM samples according to Unity Docs.