Close Menu
Contact us
AI Innovations

AI Innovations: Brain-Inspired Design for Better, More Sustainable Technology

digitalaiinsight.comBy 06 Mins Read
Diagram illustrating four common patterns of neural connectivity in biological neural networks: Feedforward, Mutual, Lateral, and Feedback. Each pattern is composed of circles representing neurons and arrows representing synapses.

As AI research and technology development continues to advance, it also becomes necessary to consider the energy resources and infrastructure needed to manage large data sets and perform complex computations. When we look to nature for models of efficiency, the human brain stands out, ingeniously handling complex tasks. Inspired by this, Microsoft researchers are looking to understand the brain’s efficient processes and replicate them in AI.

HAS Microsoft Asia Research (opens in new tab)in collaboration with Fudan University (opens in new tab), Shanghai Jiao Tong University (opens in new tab)and the Okinawa Institute of Technology (opens in new tab)There are three notable projects underway. One introduces a neural network that simulates how the brain learns and processes information; another improves the accuracy and efficiency of predictive models for future events; and a third improves AI’s proficiency in language processing and pattern prediction. These projects, highlighted in this blog post, aim not only to improve performance but also to significantly reduce energy consumption, paving the way for more sustainable AI technologies.

CircuitNet simulates brain-like neural patterns

Many AI applications rely on artificial neural networks, designed to mimic the brain’s complex neural patterns. These networks typically replicate only one or two types of connectivity patterns. In contrast, the brain propagates information using a variety of neural connection patterns, including feedforward excitation and inhibition, mutual inhibition, lateral inhibition, and feedback inhibition (Figure 1). These networks contain densely interconnected local areas with fewer connections between distant regions. Each neuron forms thousands of synapses to perform specific tasks in its region, while some synapses connect different functional groups—that is, groups of interconnected neurons that work together to perform specific functions.

Diagram illustrating four common patterns of neural connectivity in biological neural networks: Feedforward, Mutual, Lateral, and Feedback. Each pattern is composed of circles representing neurons and arrows representing synapses.
Figure 1: The four patterns of neuronal connectivity in the brain. Each circle represents a neuron and each arrow represents a synapse.

Inspired by this biological architecture, the researchers developed CircuitNeta neural network that reproduces multiple types of connectivity patterns. CircuitNet’s design features a combination of densely connected local nodes and fewer connections between distant regions, allowing for improved signal transmission through circuit pattern units (CMUs)—small, recurring patterns of connections that help process information. This structure, shown in Figure 2, supports multiple signal processing cycles, which could advance how AI systems process complex information.

Diagram illustrating the CircuitNet architecture. On the left, the diagrams labeled “Model Inputs” and “Model Outputs” show that CircuitNet can handle various forms of input and produce corresponding outputs. The middle section, labeled “CircuitNet,” depicts several interconnected blocks called Circuit Motif Units (CMUs for short), which maintain locally dense communications via direct connections and globally sparse communications via their input and output ports. On the right, a detailed view of a single CMU reveals densely interconnected neurons, demonstrating how each CMU models a universal circuit motif.
Figure 2. CircuitNet architecture: A generic neural network performs various tasks, accepts different inputs, and generates corresponding outputs (left). CMUs maintain most local connections with few long-distance connections, which promotes efficiency (middle). Each CMU has densely interconnected neurons to model universal circuit models (right).

The evaluation results are promising. CircuitNet outperformed several popular neural network architectures in function approximation, reinforcement learning, image classification, and time series prediction. It also achieved comparable or better performance than other neural networks, often with fewer parameters, demonstrating its effectiveness and strong generalization capabilities in various machine learning tasks. Our next step is to test CircuitNet’s performance on large-scale models with billions of parameters.

Spiking Neural Networks: A New Framework for Time Series Prediction

Spiking neural networks (SNNs) are emerging as a powerful type of artificial neural network, known for their energy efficiency and potential application in fields such as robotics, edge computing, and real-time processing. Unlike traditional neural networks, which process signals continuously, SNNs fire neurons only when they reach a specific threshold, generating spikes. This approach mimics how the brain processes information and conserves energy. However, SNNs are not very good at predicting future events based on historical data, a key function in industries such as transportation and energy.

To improve the predictive capabilities of the SNN, researchers proposed a SNN Framework Designed to predict trends over time, such as electricity consumption or traffic patterns, this approach exploits the efficiency of spiking neurons in processing temporal information and synchronizes time series data (collected at regular intervals) and SNNs. Two encoding layers transform the time series data into sequences of spikes, allowing SNNs to process them and make accurate predictions, as shown in Figure 3.

Diagram illustrating a novel framework for SNN-based time series prediction. The image shows the process starting with the time series input, which is encoded into spikes by a novel spike encoder. These spikes are then fed into different SNN models: (a) Spike-TCN, (b) Spike-RNN, and (c) Spike-Transformer. Finally, the learned features are input into a projection layer for prediction.
Figure 3. A novel framework for SNN-based time series prediction: Time series data are encoded into spikes using a novel spike encoder (middle, bottom). The spikes are then processed by SNN models (Spike-TCN, Spike-RNN, and Spike-Transformer) for training (top). Finally, the learned features are fed into the projection layer for prediction (bottom right).

Tests show that this SNN approach is very effective for time series prediction, often matching or outperforming traditional methods while significantly reducing energy consumption. SNNs successfully capture temporal dependencies and model time series dynamics, providing an energy-efficient approach that closely matches how the brain processes information. We plan to continue exploring ways to further improve SNNs based on how the brain processes information.

Refining SNN sequence prediction

While SNNs can help models predict future events, research has shown that their reliance on spike-based communication makes it difficult to directly apply many techniques from artificial neural networks. For example, SNNs struggle to effectively handle the rhythmic and periodic patterns found in natural language processing and time series analysis. In response, researchers have developed a new approach to SNNs called CPG-EPwhich combines two techniques:

  1. Central pattern generators (CPGs): Neural networks in the brainstem and spinal cord that autonomously generate rhythmic patterns, controlling functions such as movement, breathing, and chewing
  1. Positional coding (PC): A process that helps artificial neural networks discern the order and relative positions of elements within a sequence.

By integrating these two techniques, CPG-PE helps SNNs discern the position and timing of signals, thereby improving their ability to process temporal information. This process is illustrated in Figure 4.

Diagram illustrating the application of CPG-PE in an SNN. It shows three main components: an input spike matrix labeled “X”, a transformation process involving positional encoding and a linear transformation to produce “X’”, and the output of a spike neuron layer labeled “X_output”. The input matrix “X” has multiple rows corresponding to different channels or neurons, each containing spikes over time. The transformation process maps the dimensionality of (D + 2N) to D. The spike neuron layer takes the transformed input “X’” and produces the output spike matrix “X_output”.
Figure 4: Application of CPG-PE in a SNN. X, X′ and X-output are peak matrices.

We evaluated CPG-PE using four real-world datasets: two covering traffic patterns, and one each for electricity consumption and solar energy. The results demonstrate that SNNs using this method significantly outperform those without positional encoding (PE), shown in Table 1. Moreover, CPG-PE can be easily integrated into any SNN designed for sequence processing, making it adaptable to a wide range of neuromorphic chips and SNN hardware.

Table showing experimental results of time series forecasting on two datasets, Metr-la and Pems-bay, with prediction lengths of 6, 24, 48, and 96. The table compares the performance of various models, including different configurations of SNN, RNN, and Transformers. Performance metrics such as RSE and R^2 are shown. The best SNN results are highlighted in bold, and upward arrows indicate higher scores, representing better performance.
Table 1: Results of evaluating time series forecasts on two benchmarks with prediction lengths of 6, 24, 48, 96. “Metr-la” and “Pems-bay” are traffic pattern datasets. The best SNN results are in bold. Upward arrows indicate a higher score, representing better performance.

Ongoing research on AI for greater capacity, efficiency and sustainability

The innovations presented in this blog demonstrate the potential to create AI that is not only more powerful, but also more effective. Moving forward, we are excited to deepen our collaborations and continue to apply insights from neuroscience to AI research, continuing our commitment to exploring ways to develop more sustainable technologies.

Related Posts

Add A Comment
Leave A Reply