A Path Not Taken

I’ve been a software engineer for nearly thirty years, primarily working on large-scale high-performance distributed systems. The complexity of these systems is both amazing and frustrating. They are typically a collection of highly specialized, custom-built modules or a complex integration of disparate COTS/FOSS products. Sometimes they are both.

As these systems grow and evolve, so does the institutional knowledge that is required to maintain, fix and enhance them. Shifting markets, political fluctuations and employee attrition/movement result in a loss of this institutional knowledge. Corporations typically back-fill positions with fresh-talent without realizing or acknowledging that it takes most people months or years to master the complex systems.

Much of the complexity in these systems can be attributed to microprocessor architecture. Microprocessors operate sequentially; they load/execute/store and branch/return billions of times per second, but always follow the instruction counter. Consequently, programming languages function in the same fashion. Modern microprocessor architectures can process instructions in parallel using multiple cores, but the code itself is still sequential. Modern operating systems also support concurrent/parallel processing using processor cores and software threading, but again the code is still sequential.

I’ve noticed a limiting pattern over the years: Our code repeatedly collects data elements to be used as input arguments, calls a function and inspects the results upon return. The code collects the data elements by possibly calling other functions, and we must collect all of the input data elements before we can invoke the function. This is sufficient in many cases, but what if the input data elements arrive asynchronously in an event-driven system? Then our code repeatedly polls until the asynchronous event occurs. This polling impacts efficiency and defeats parallelism. For example, if we must poll for multiple independent data elements, we are stacking these polling loops and not taking advantage of the intermediate time for other processing. Sure, we can put the polling on a separate thread, but then we need additional concurrency controls and IPC to coordinate the threads.

My colleagues and I started imagining what programming would look like if:

• Data elements could arrive at any time, in any order.


• The same data element could arrive at different functions at the same time.


• Functions could get executed asynchronously when all data elements were available.


• The location and number of function instances were dynamic.


• A standard and consistent architecture made software easier to understand and maintain.

Although we arrived at these questions purely from experience, it turns out they weren’t entirely novel. For example, Kahn Process Networks (KPNs) were introduced in the 1970’s for signal processing. Flow-Based Programming (FBP), also introduced in the 1970’s, is another example where asynchronous processes consume data chunks that arrive over data streams.

Our notion was centered around Cells that process Stimuli arriving over Synapses. Cells could be connected, even multiply-connected, in any fashion by synapses. The synapses would transmit stimuli locally or remotely, and cell location would be transparent to the system. Stimuli would traverse the vast network of cells and synapses, triggering Activators in interested cells. Cells could host multiple activators that could be independent or interrelated. We referred to the construction of cellular systems as Cellular Programming.

While my colleagues chose to focus on a cellular operating system, I wanted to prove (or disprove) that the idea is generally useful. I could imagine the applications in distributed, event-driven, real-time systems, but could it be applied to general computing? There was only one way to find out; I had to build a framework for cellular programming. Over many years I would experiment, discuss and re-write my framework while overcoming the challenges in solving general computing tasks. Performance in a cellular system is paramount, so sophisticated protocols for discovery, subscription and routing were created and honed. Finally, cells were designed to consume minimal resources, especially while waiting for stimuli.

I would finally present the framework called NeuPaths to the software community in 2025. I suffered from a “Field of Dreams” fantasy, as if just publishing the framework would result in its discovery. Unfortunately there has been little interest so far. I shouldn’t be surprised. There is no single, tangible definition of what the framework is. People ask me what it does; I can only respond “anything you want.” It doesn’t solve a particular problem in a particular domain. It presents a new way to solve problems in any domain. This is why I invite everyone to experiment with cellular programming. I am hoping for collaboration, creativity and invention.

I invite you to explore and create with NeuPaths, available on GitHub: https://github.com/commence-software/NeuPaths