Flow-Based Programming

Suppose you were told that an obscure programming technology that has been in continuous production use at one of Canada's major banks since the 1970s provides an amazingly simple solution to a number of the major challenges facing today's programmers, including multicore machines and distributed computing,  while providing improved maintainability,  as well as a much more seamless transition from structured design to running code, would you reject this as a complete and utter fairy tale... or just think your informant has been misled?!

It's true though!  An early implementation of these concepts, called Flow-Based Programming (FBP), has enabled the applications written using it to respond to changes in business needs and systems environment, while still exhibiting the high reliability required of banking applications,  Plus these programs actually ran in less elapsed time than programs written using conventional techniques.  FBP to a large extent frees the programmer from von Neumann thinking, and the modern implementations written in Java and C# make it a snap to take advantage of all the processors on your machine. Of course, there is a paradigm change!

FBP was actually invented/discovered back in the late '60s, but it took us several decades to understand its potential.   During that period we came to realize that this technology solves many of the old problems that have always bedeviled conventional programming, but it also has the potential to simplify many of the new problems that we are just now starting to run into.  It's now starting to catch on in a big way!   "Go with the flow!"

Herakleitos, in about 500 BC, said

usually translated as "Everything changes".  What he actually said was: "Everything flows".

Second edition now available (15 years later) from CreateSpace eStore and Amazon.com!

A second edition of my book is now available from CreateSpace's eStore, and at Amazon.com. Take a look and see how FBP has changed in 15 years!

The new edition contains information about the Java and C# implementations, as well as the diagramming tool (DrawFBP), several sections contributed by Mike Beckerle, CTO of Oco Inc., on their experience using FBP concepts on truly humongous amounts of data, plus a whole new chapter on software projects that have appeared in the last 15 years and either build on FBP, or have strong similarities to it. And, of course, FBP is the obvious technology for application development involving multi-processor computers and networks of computers, so this is addressed in several of the chapters.

For obvious reasons, the new sections are not available on this web site - but just think, you could have a shiny new copy in your hands, printed with loving care by CreateSpace - and IMHO it looks just as good as the original edition (if not better)!

For the Wikipedia article, see Flow-Based Programming. 

For late-breaking news, see the FBP Wiki News.
A Google Group has been started for FBP - see FBP Google Group.

Information about the author: J. Paul Morrison.

Flow-Based Programming is a new/old approach to application development, based on a completely different way of thinking about building applications. Some of its roots can be traced all the way back to the early days of computing, yet it offers solutions to many of the most pressing problems facing application development today.

Not surprisingly parts of the answer have been presaged many times over in the past, but my 1994 book on the subject, "Flow-Based Programming", van Nostrand Reinhold, 1994, ISBN 0-442-01771-5 was the first book to tie together many of these different efforts, and to point towards a very practical, powerful, yet perfectly realizable future of application development. This book went out of print some years ago, but the second edition is now available from CreateSpace (via Print On Demand) - see above.

In "Flow-Based Programming" (FBP), applications are defined as networks of "black box" processes, which exchange data across predefined one-way connections. These black box processes can be reconnected endlessly to form different applications without having to be changed internally. It is thus naturally component-oriented. To describe this capability, Raoul de Campo of IBM Research coined the term "configurable modularity", and it was later adopted and developed by the distinguished IBM engineer, Nate Edwards, who described it as "the basis of all true engineered systems."

When using FBP, the application developer works with flows of data, being processed asynchronously, rather than the conventional single hierarchy of sequential, procedural code. This means that the precise timing of events can usually not be predicted exactly, which tends to make old-style programmers nervous. However, it turns out that this isn't really necessary (and never has been!), and in fact shifts the emphasis from sequences of actions to transformations performed on streams of data.

FBP is thus a good fit with multiprocessor computers, and also with modern embedded software. In many ways, an FBP application resembles more closely a real-life factory, where items travel from station to station, undergoing various transformations.  Think of a soft drink bottling factory, where bottles are filled at one station, capped at the next and labelled at yet another one.  FBP is therefore highly visual: it is quite hard to work with an FBP application without having the picture laid out on one's desk, or up on a screen!  For an example, see a portion of a batch program (you have to zoom in to see some of the connections). This diagram, which was built using the earlier C++ DrawFBP diagramming tool, only shows part of an extremely complex program, which was running almost daily for over 30 years, processing millions of transactions a day (it is now in process of being replaced). Now imagine building the same function using a conventional programming language!

Strangely though, in spite of being at the leading edge of application development, it is also simple enough that trainee programmers can pick it up, and it is a much better match with the primitives of data processing than the conventional primitives of procedural languages.

In recent years, the concept seems to have started to take off - this seems to be coinciding with a feeling that Object-Oriented isn't the perfect solution to all our problems that it was originally believed to be. More and more projects and papers use the term Flow-Based Programming (often in lower case) - to the point where it has started to be a generic term (like Bandaid in N. America!). For a fairly lengthy list of projects, papers, and even companies, that seem to be using related concepts, see the Wiki page called "Flowlike Projects".

As you might guess from the logo at the top, a project has been registered on SourceForge. Source code for Java and C# implementations of the concepts has been uploaded to SourceForge, as well as a picture-drawing tool that supports many of its concepts (see DrawFBP).

An article on FBP has recently been added to Wikipedia - please feel free to extend it in any direction that you feel would enhance its usefulness.

The earliest implementations of FBP concepts are described in the Introduction to the book, and were built for IBM mainframes. 

For PCs, there was an early implementation called THREADS, also described in the book, followed by Java (JavaFBP) and C# (C#FBP) implementations (developed after the book was published).  The network syntax and component API are described in THREADS API and Network Specification .

The latest version of JavaFBP is available on SourceForge and also on the FBP web site in non-executable jar file format - JavaFBP jarfile.  Its network syntax and component API are described in "JavaFBP Network Syntax and Component API" .

The latest version of  C#FBP is available on SourceForge and on the FBP web site - C#FBP zipfile.  Its network syntax and component API are described in "C#FBP Network Syntax and Component API" .

A new feature of both of these releases is the addition of metadata (or "attributes") to components, giving port names and characteristics, and the "must run" attribute. This is intended to allow diagramming tools to display port names and use this information for various types of checking. If you are currently working with a previous version of JavaFBP, the metadata will have to be added to your existing components, but this only has to be done once, and hopefully will not be too onerous.

A new, much more powerful, Java version of the  DrawFBP diagramming tool is now available.

Last but not least, I am starting to have more time available, and I hope to move more into research on Flow-Based Programming (of course). As the signs are that FBP is finally starting to take off, I am hoping that interested individuals will start to contact me about ideas and possibilities in this area. In particular, I am looking for organizations that are willing to share their experience (and even componentry) with other organizations. If we could pool our resources, as is done in the Open Source community, we could bootstrap our productivity, and everyone benefits! Another interesting area where collaboration could take place is in the design and management of component repositories.

FBPWiki

I have pulled together a lot of material in a FBP Wiki, either dredged from the Internet by myself, or contributed by interested readers. It's not very organized (no, it's very unorganized), so it would be fantastic if some energetic person would undertake to do what Wikizens call "refactoring". To add stuff to it, please contact me for an edit password.

Book-related stuff

The book is now out of print, but is still listed at Amazon.com , and second-hand copies become available from time to time.

A copy is now on-line in PDF format - to access it, click on on-line copy . To download it, you may have to shift and click. The diagrams are not as elegant as in the original publication, but hopefully they will suffice to get the ideas across. Maybe at some point in the future, they will be upgraded. Please let Paul know if there are any obvious omissions or formatting errors (apart from the diagrams!).

Two reviews of "Flow-Based Programming" appeared a few years ago in one of the Australian computer journals, one by Tony Stevenson, then at Monash University, and the other by Andrew Wenn, of Victoria University of Technology (Footscray). The latter has a pretty complete description of FBP and of the contents of the book, as well as some (well-deserved) criticisms! To see them, click on Stevenson, and Wenn, respectively.

Ed Yourdon has kindly added my book to his list of "cool books".

Papers on FBP

There are some papers on FBP on my web site. Not surprisingly, there is some duplication as it is generally necessary to give a short description of the basic concepts in each one, but each paper has some unique sections. Here they are:

• "Asynchronous Component-Based Programming." (contains some stats on typical application component counts)

• "Flow-Based Programming" (contains some statistics on reuse)

• "Patterns in Flow-Based Programming"

Chapters from the Book

I have uploaded a slightly earlier version of my book to this web site. Unfortunately, as the book is out of print, this will be your only version, until we can arrange to have a paper version available again. As stated above, a copy is also now on-line in PDF format.

I should state that, as of 2009, the web site will start to diverge from the exact text of the book - in fact it always did! This is partly due to changes in the diagramming tools; also we will be starting to incorporate results from later Java and C# work as tools based on these languages evolve.

C Implementation of FBP Concepts (THREADS)

A few years ago I implemented a number of the concepts of FBP in C, calling it THREADS. It used the Borland C(++) compiler. It is described in my book, "Flow-Based Programming" - in particular, in the Appendix (see "THREADS API and Network Specification"). The code is available on this web site at THREADS. It is in the process of being upgraded to use Win32 and "fibers" - I have put it up on my web site, at THREADS_32, but I freely admit that testing has only just started. I am also uncertain how to implement component metadata, so this is currently unsupported.

Java Implementation of FBP Concepts (JavaFBP)

A Java implementation of Flow-Based Programming which we have called JavaFBP (formerly JFBP, formerly JavaFlow) is now available in open source form, and has been registered as a project on SourceForge, and the code has been uploaded onto the SourceForge File release System. It can be accessed by clicking here: SourceForge. To download the source code, click on "files", and then download the latest release. An earlier version was uploaded onto the CVS server on SourceForge, but it is not up-to-date.

The code is also available on this web site. To download it, click here: JavaFBP jar file (you may have to shift and click).

This implementation was written by John Cowan and tweaked by me. It allows a developer to specify an application as a network of nodes, which are long-running Java threads, connected by fixed-capacity connections, over which data objects travel.  We feel this approach lends itself to server applications where high throughput is required.  It is easy to multiplex components that are either I/O-bound or CPU-bound (in the latter case this allows us to take advantage of the multiple processors in current machines). The network is also specified as a Java program, which we feel is a more flexible approach than designing a special-purpose language for this purpose.

As JavaFBP did not exist when my book was written, a page has now been added to this site describing the syntax of a JavaFBP network definition and a sample JavaFBP component - information will be added to this page as time permits. To see it, click on JavaFBP Network Syntax and Component API.

The latest version, JavaFBP-2.4, has a number of new features, described on the FBP wiki.

JavaFBP also supports "static subnets" - an example can be found in the jar file, and there is a fairly complete description of how it works in Composite Components, particularly in the second half, including "substream sensitivity".

Please feel free to look at the code, and/or contact me for more information, or to give me feedback.

The following notice has been added to all the source files:

static final String copyright = 
"Copyright 2007, 2008, 2009, J. Paul Morrison.  At your option, you may copy, " +
"distribute, or make derivative works under the terms of the Clarified Artistic License, " +
"based on the Everything Development Company's Artistic License.  A document describing " +
"this License may be found at http://www.jpaulmorrison.com/fbp/artistic2.htm  " +
"THERE IS NO WARRANTY; USE THIS PRODUCT AT YOUR OWN RISK.";

As stated in the copyright notice, you can look at a document describing this license by clicking on Clarified Artistic License .

C# Implementation of FBP Concepts

A couple of years ago, Amanda Ge kindly converted the pre-5.0 Java version to C#, but did not have time to test it. With input from David Bennett in Melbourne, Australia, I have upgraded my C# implementation and it is now available on SourceForge and on my web site. There is a readme file describing the new features.

The new version of C#FBP can be found at C#FBP (you may have to shift and click), and on SourceForge. The source code is now available on SourceForge SVN. Please feel free to play with it - and let me know how you make out. I will of course post updates as required. If you have already taken a copy, please download the latest version!

A page has been added to this site describing the syntax of a C#FBP network definition and a sample C#FBP component - information will be added to this page as time permits. To see it, click on C#FBP Network Syntax and Component API.

DrawFBP

Several years ago, a picture-drawing tool which supports many of the concepts of FBP was written in C++ for Windows. It can still be obtained by clicking here: DrawFBP-C++,  but it has now been superseded by a Java version, which can be obtained by clicking here: DrawFBP Installer, and this is also available on SourceForge - look for the 2.3 version. The Java version is much more powerful, has fairly decent help facilities, and can generate runnable Java code, if it is given enough information.  You can now draw an application, click on "Generate Network", fill in a few missing bits of information (if any), and voilà - a running program! What's not to like?!

DrawFBP can also be downloaded and executed immediately by clicking on JWS version of DrawFBP, or downloaded as a jar file .

DrawFBP does not attempt to generate diagrams from text – we have seen too many failed attempts at this – instead it allows the designer to lay out the flow as desired, after which DrawFBP captures the information from the diagram, including x-y coordinates of blocks and line bends, in an XML file, with an extension of .drw. The XML contains sufficient information to enable the diagram to be redrawn, but it also captures the relationships between processes and their connections, so it has enough information to generate the lists of connections used by FBP (or FBP-like) schedulers.

DrawFBP now provides a font chooser function, which displays all available fonts on the user's machine, showing sample text and indicating which fonts support Chinese characters. DrawFBP maintains a user profile, which will retain, among other things, the names of the selected fonts (one for fixed format text, and one for variable).

DrawFBP supports stepwise decomposition by allowing the designer to associate either a code component, or a lower-level subnet, with a block on the diagram. If a subnet is specified, the higher level block is shown with a doubled outline. Double-clicking on a block will bring up the component code in a separate window in the former case, or the subnet – in the latter. Once you have associated a component or subnet with a block on the diagram, DrawFBP lets you check that all necessary ports have been properly connected. It also colours the connection lines if it can determine that the generating and consuming processes are working with the same packet types. It's getting pretty powerful!

A Help facility is also available, based on Java SE Desktop Technologies JavaHelp System – this function does not require access to the Internet, but requires the jhall jar file, which is included in the DrawFBP Installer zip file.

To give credit where credit is due, a number of usability ideas for the new tool are due to Oliver Brupbacher, of NumCom in Zurich, Switzerland. Thanks, Oliver!

Here is a snapshot of a simple diagram, showing an arrow in process of being drawn:

Biography of the author

J. Paul Morrison.

Contact Information

J. Paul Morrison,
8A Station Lane,
Unionville,
Ont., L3R 1R3
Canada
(905) 479-4056

Internet: