You are looking at the old version of the Flow-Based Programming index page - please go to the newer, more succinct (and generally handsomer) version of the index page, designed by Mayuresh Kathé, with some tweaks by J. Paul Rodker Morrison.
"Flow-Based
Programming", 2nd Edition, is now available on Kindle
and Lulu
(EPUB format), as well as in paperback form from Amazon
and CreateSpace.
For more information see below.
Suppose someone told you that an obscure programming technology, in continuous production use at one of Canada's major banks since the 1970s, provides an amazingly simple solution to a number of the major challenges facing today's programmers, including multicore machines and distributed computing, while providing improved maintainability, as well as a much more seamless transition from structured design to running code - would you reject this as a fairy tale, made up by impractical dreamers who know nothing about the application development business? You probably would!
It's true though! The first implementation of these concepts, now being called "Flow-Based Programming" (FBP), was used to implement a significant proportion of the offline applications used by a large N. American bank, servicing around 5,000,000 customers, going live in the mid-1970s. FBP has allowed these applications to respond to changes in business needs and the systems environment, over a period of almost 40 years (at least one of the applications which went live in 1975 is still in use today - 2013 - processing around 5 million accounts daily), while still providing the high reliability required of banking applications. As a bonus, these programs actually ran in less elapsed time than programs written using conventional techniques. FBP is (and was then) a brand new way of thinking about application development, freeing the programmer from von Neumann thinking, one of the major barriers to moving to the new multiprocessor world, and has been evolving steadily over the intervening years. It has now been implemented using both Java and C# multithreading. Using these implementations, you can now easily take advantage of the processing power of all your machine's processors. Of course, there is a paradigm change, but, once you have made the shift, you can't go back!
FBP was actually invented/discovered back in the late 1960s, but it took us several decades to understand its potential. During that period we came to realize that this technology solves many of the old problems that have always bedeviled conventional programming, but it also has the potential to simplify many of the new problems that we are just now starting to run into. FBP is now starting to catch on in a big way.
More and more programmers are now saying:
Go with the flow!
Herakleitos, in about 500 BC, said
Πάντα ῥεῖ (Panta rhei)
usually translated as "Everything changes". What he actually said was: "Everything flows". |
The second edition of the book, "Flow-Based Programming", appeared in 2010, 16 years after the first edition, and is available from CreateSpace eStore and Amazon.com - also in ebook format, from Kindle and Lulu. Take a look and see how FBP has changed over the past 16 years!
The new edition contains information about the Java and C# implementations, as well as the diagramming tool (DrawFBP), several sections contributed by Mike Beckerle, CTO of Oco Inc., on his experience using FBP concepts on truly humongous amounts of data, plus a whole new chapter on software projects that have appeared in the last 16 years and either build on FBP, or have strong similarities to it. And, of course, FBP is the obvious technology for application development involving multi-processor computers and networks of computers, so this is addressed in several of the chapters.
For obvious reasons, the new sections are not available on this
web site - but just think, you could have a shiny new copy in your
hands, printed with loving care by CreateSpace - and
IMHO it looks just as good as the original edition (if not
better)!
"Flow-Based Programming, 2nd
Edition" is also available in ebook format from:
You can download the book in seconds from either of these
sites. In neither case do you need e-reader hardware:
I should mention that ebook versions provide live links and
cross-references on your PC or appropriate readers (although some of the
tables aren't quite as legible as in the printed version -
apparently this is a general problem with ebooks).
A few years ago, Henri Bergius and his team started to combine FBP concepts with JavaScript, and have been creating a serious buzz world-wide with their product NoFlo.
In Aug. 1, 2013, they started a KickStarter project to raise $100,000 for future development, and had raised $115,677 by the end of the 45-day period. NoFlo has been recognized by Forbes, O'Reilly, FastCompany, and many other companies and publications. There are some great videos on this web site.
NoFlo is integrated with a graphics tool called NoFlo-UI, developed by The Grid.
FBP now seems to finally be taking off, with a solid boost from this young, talented and enthusiastic team!
Table of contents:
(*) You may have to enable Java via the Java Control Panel.
(**) You will get a message saying "Unknown publisher" - I am
afraid
I cannot afford to get a verified certificate for distributing
free apps. Maybe Verisign will ease up in the future!
A Google Group has been started for FBP - see FBP
Google Group. To send the group a message, click
on messages
to Google group. Don't forget the wiki about FBP - FBP Wiki - although
nobody seems to have contributed to it in quite a while, probably
due to changing fashions in the industry, there is still a lot of
good stuff on the wiki, contributed by a lot of very smart people!
I have recently set up a bulletin
board to manage discussions about FBP and its
implementations - the idea is to use it mainly for implementation
details at a lower level than those discussed in the Google
group. I would appreciate feedback on its usefulness,
usability, etc.
Justin Bozonier has written what Google called a "knol" (unit of
knowledge) on Flow-Based
Programming, but knols have been migrated to WordPress, and the
migrated version can be viewed on Flow-Based
Programming on WordPress (Annotum) - great job, Justin!
There is a Wikipedia article on Flow-Based Programming - please feel free to extend it in any direction that you feel would enhance its usefulness.
There is a wiki about FBP - FBP
Wiki - nobody seems to have contributed to it in quite
a while, probably due to changing fashions in the industry.
The active discussion seems to have shifted to the Google Group
mentioned above, but there is still a lot of good stuff on the
wiki, contributed by a lot of very smart people!
Information about the author of this web page: J. Paul Morrison.
Flow-Based Programming is a new/old approach to application development, that defines applications using the metaphor of a "data factory". As such, it embodies a way of thinking about building applications that is completely different from the way they have been done traditionally. Some of its roots can be traced all the way back to the early days of computing, yet it offers solutions to many of the most pressing problems facing application development today.
Not surprisingly parts of the answer have been presaged many times over in the past, but the 1st edition (1994) of my book on the subject, "Flow-Based Programming", van Nostrand Reinhold, 1994, ISBN 0-442-01771-5 was the first book to tie together many of these different efforts, and to point towards a very practical, powerful, yet perfectly realizable future of application development. The 2nd edition came out with the assistance of CreateSpace in 2010 - Flow-Based Programming, 2nd Edition: A New Approach to Application Development, CreateSpace, 2010, ISBN 1451542321 - and contains contributions from a number of people who have been working with the technology over the last 16 years. It is also available in ebook format from Amazon.com and Lulu.com.
Flow-based programming (FBP) is a programming paradigm that defines applications using the metaphor of a "data factory". An application is defined as a network of asynchronously executing processes, which communicate by means of streams of data chunks called "information packets", travelling over predefined connections. These connections are defined externally to the processes, so no process knows who its neighbours are. Connections and processes can therefore be reconnected endlessly to form different applications, without the processes having to be changed internally. It is thus naturally component-oriented. To describe this capability, Raoul de Campo of IBM Research coined the term "configurable modularity", and it was later adopted and developed by the distinguished IBM engineer, Nate Edwards, who described it as "the basis of all true engineered systems."
When using FBP, the application developer works with flows of data, being processed asynchronously, rather than the conventional single hierarchy of sequential, procedural code. This means that the precise timing of events can usually not be predicted exactly, which tends to make old-style programmers nervous. However, it turns out that this isn't really necessary (and never has been!), and in fact shifts the emphasis from sequences of actions to transformations performed on streams of data.
FBP is thus a good fit with multiprocessor computers, and also with modern embedded software. In many ways, an FBP application resembles more closely a real-life factory, where items travel from station to station, undergoing various transformations. Think of a soft drink bottling factory, where bottles are filled at one station, capped at the next and labelled at yet another one. FBP is therefore highly visual: it is quite hard to work with an FBP application without having the picture laid out on one's desk, or up on a screen! For an example, see a portion of a batch program (you have to zoom in to see some of the connections). This diagram, which was built using the earlier C++ DrawFBP diagramming tool, only shows part of an extremely complex program, which was running almost daily for over 30 years (it has since been replaced). Now imagine building the same function using a conventional programming language!
Strangely though, in spite of being at the leading edge of application development, it is also simple enough that trainee programmers can pick it up, and it is a much better match with the primitives of data processing than the conventional primitives of procedural languages.
In recent years, the concept seems to have started to take off - this seems to be coinciding with a feeling that Object-Oriented isn't the perfect solution to all our problems that it was originally believed to be. More and more projects and papers use the term Flow-Based Programming (often in lower case) - to the point where it has started to be a generic term (like Bandaid in N. America!). For a fairly lengthy list of projects, papers, and even companies, that seem to be using related concepts, see the Wiki page called "Flowlike Projects".
As you might guess from the logo, a project has been registered on SourceForge. Source code for Java and C# implementations of the concepts has been uploaded to SourceForge, as well as a picture-drawing tool that supports many of its concepts (see DrawFBP).
The earliest implementations of FBP concepts are described in the Introduction to the book, and were built for IBM mainframes.
For PCs, there was an early implementation called THREADS, also described in the book, which has been superseded by two C++ implementations, the first using Windows fibers, the second using the Boost package. THREADS was followed by Java (JavaFBP) and C# (C#FBP) implementations (developed after the book was published). The network syntax and component API are described in THREADS API and Network Specification .
The latest version of JavaFBP is available on SourceForge and also on the FBP web site in non-executable jar file format - JavaFBP jarfile. Its network syntax and component API are described in "JavaFBP Network Syntax and Component API".
The latest version of C#FBP is available on SourceForge and on the FBP web site - C#FBP zipfile. Its network syntax and component API are described in "C#FBP Network Syntax and Component API".
A new feature of both of these releases is the addition of metadata (or "attributes") to components, giving port names and characteristics, and the "must run" attribute. This is intended to allow diagramming tools to display port names and use this information for various types of checking. If you are currently working with a previous version of JavaFBP, the metadata will have to be added to your existing components, but this only has to be done once, and hopefully will not be too onerous.
A new, much more powerful, Java version of the DrawFBP diagramming tool is now available.
I note that there is now a web site called www.flowbasedprogramming.com - this is in fact the web site for Marcus Tomlinson's DSPatch - a very interesting C++-based implementation of the FBP paradigm that allows the user to create and route (or "patch") what the author calls "high performance data processing circuits".
Last but not least, I am starting to have more time available, and I hope to move more into research on Flow-Based Programming (of course). As the signs are that FBP is finally starting to take off, I am hoping that interested individuals will start to contact me about ideas and possibilities in this area. In particular, I am looking for organizations that are willing to share their experience (and even componentry) with other organizations. If we could pool our resources, as is done in the Open Source community, we could bootstrap our productivity, and everyone benefits! Another interesting area where collaboration could take place is in the design and management of component repositories.
The first edition of the book is now out of print, but is still listed at Amazon.com , and it is also on-line in two formats: a copy in PDF format, and individual browsable chapters - listed below. The second edition is now available from CreateSpace eStore and Amazon.com., plus, as stated above, in ebook format from Kindle and Lulu (EPUB format).
Two reviews of "Flow-Based Programming" appeared a few years ago in one of the Australian computer journals, one by Tony Stevenson, then at Monash University, and the other by Andrew Wenn, of Victoria University of Technology (Footscray). The latter has a pretty complete description of FBP and of the contents of the book, as well as some (well-deserved) criticisms! To see them, click on Stevenson, and Wenn, respectively.
Ed Yourdon has kindly added my book to his list of "cool books".
There are some papers on FBP on my web site. Not surprisingly, there is some duplication as it is generally necessary to give a short description of the basic concepts in each one, but each paper has some unique sections. Here they are:
"Asynchronous Component-Based Programming." (contains some stats on typical application component counts)
"Flow-Based Programming" (contains some statistics on reuse)
"Flow-Based Programming" in "Application Developers' News" - Journal for Developers of Heterogeneous Computing Systems
"J. Paul Morrison video on the origins of FBP" - made for the NoFlo Kickstarter project, Summer 2013
"J. Paul Morrison video on his history with FBP" - presented at the first Toronto FBP Meetup, 23 Oct. 2013
The first implementation of the concepts of FBP for the PC was written in C, and was called THREADS. It used the Borland C compiler. It is described in the first edition of my book, "Flow-Based Programming", specifically in the Appendix (see "THREADS API and Network Specification"). The code is the base for an implementation using Win32 and "fibres", and can be found on SourceForge. As it is fibre-based, it cannot make use of multiple processors, but initial testing suggests that, on the other hand, it is very fast!
The original C implementation is in process of being upgraded to use Boost multithreading. This version is now on the "trunk" of the CppFBP code repository on SourceForge. Not all of the capabilities of the old THREADS implementation are in place as yet, but we have done some performance testing, and it definitely uses all the processors on your machine.
PS I am also uncertain how to implement component metadata, which I consider an important feature of the Java and C# implementations, so this is currently unsupported.
An earlier reimplementation of THREADS, using Windows fibers, is available on a branch of the CppFBP code repository on SourceForge - it is pretty fast, but uses green threads, and so only uses one processor. It has been kept in case, at some point in the future, we decide to explore the idea of a hybrid green thread/red thread C++ implementation.
A component has been built for CppFBP, tentatively called ThLua, which allows looper components to be written in the Lua scripting language. We are now able to write multithreaded networks whose infrastructure is CppFBP, and where as many processes as you like are implemented using Lua scripts - in this environment Lua scripts may even be multithreading with themselves, with no problems!
As Lua scripts communicate by way of character strings, if you need to pass more complex data structures through the network, you can also intermix C++ components in the same network - the API is the same as for THREADS - or even other higher-level languages which play well with C++!
Here is an example of a ThLua-supported Lua process which concatenates input streams from successive elements of array port IN, and sends the result to port OUT:
-- concat.lua
i = dfselct("IN"); -- get element count for port "IN"
for i = 0, i - 1 do
value, IPaddr, size, type = dfsrecv("IN["..i.."]") -- receive from element i of port "IN"
while value == 0 do
value = dfssend("OUT", IPaddr) -- send IP to port "OUT"
value, IPaddr, size, type = dfsrecv("IN["..i.."]") -- receive from element i of port "IN"
end
end
|
A Java implementation of Flow-Based Programming which we have called JavaFBP (formerly JFBP, formerly JavaFlow) is now available in open source form, and has been registered as a project on SourceForge, and the code has been uploaded onto the SourceForge File release System. It can be accessed by clicking here: SourceForge. To download the source code, click on "files", and then download the latest release. An earlier version was uploaded onto the CVS server on SourceForge, but it is not up-to-date.
The code (version 2.6) is also available on this web site. To download it, click here: JavaFBP jar file (you may have to shift and click).
This implementation was written by John Cowan and tweaked by me. It allows a developer to specify an application as a network of nodes, which are long-running Java threads, connected by fixed-capacity connections, over which data objects travel. We feel this approach lends itself to server applications where high throughput is required. It is easy to multiplex components that are either I/O-bound or CPU-bound (in the latter case this allows us to take advantage of the multiple processors in current machines). The network is also specified as a Java program, which we feel is a more flexible approach than designing a special-purpose language for this purpose.
There is a web page describing the syntax of a JavaFBP network definition and showing a sample JavaFBP component - information will be added to this page as time permits. To see it, click on JavaFBP Network Syntax and Component API.
JavaFBP also supports "static subnets" - an example can be found in the jar file, and there is a fairly complete description of how it works in Composite Components, particularly in the second half, including "substream sensitivity".Please feel free to look at the code, and/or contact me for more information, or to give me feedback.
As of version 2.5, the following features were added (also described in JavaFBP Network Syntax and Component API) :
As of version 2.6, JavaFBP now checks that components only receive and send packets containing data of the type specified in the class annotations. This uses the Java isAssignableFrom() method - e.g. if the port annotations specify Object (the default), this port may send or receive packets containing anything.
Input ports and input port array elements are automatically
optional - this has not changed since earlier releases.
The following notice has been added to all the source files (updating the dates as required):
static final String copyright =
"Copyright 2007, ... 2012, J. Paul Morrison. At your option, you may copy, " +
"distribute, or make derivative works under the terms of the Clarified Artistic License, " +
"based on the Everything Development Company's Artistic License. A document describing " +
"this License may be found at http://www.jpaulmorrison.com/fbp/artistic2.htm " +
"THERE IS NO WARRANTY; USE THIS PRODUCT AT YOUR OWN RISK.";
As stated in the copyright notice, you can look at a document
describing this license by clicking on Clarified
Artistic License .
Several years ago, Amanda Ge kindly converted the pre-5.0 Java version to C#, but did not have time to test it. With input from David Bennett in Melbourne, Australia, I have upgraded my C# implementation and it is now available on SourceForge and on my web site. There is a readme file describing the new features.
The new version of C#FBP can be found at C#FBP (you may have to shift and click), and on SourceForge. The source code is now available on SourceForge SVN. Please feel free to play with it - and let me know how you make out. I will of course post updates as required. If you have already taken a copy, please download the latest version!
A page has been added to this site describing the syntax of a C#FBP network definition and a sample C#FBP component - information will be added to this page as time permits. To see it, click on C#FBP Network Syntax and Component API.
A picture-drawing tool which supports many of the concepts of FBP can be obtained by clicking here: DrawFBP Installer, and is also available on SourceForge - look for the 2.8 version. This version supports a number of icons, has fairly decent help facilities, and can generate runnable Java, C# or NoFlo network definitions, if given enough information (i.e. component and port names). You can now draw an application, click on "Generate Network", fill in a few missing bits of information (if any), and voilà - a running program! What's not to like?!
For more information about DrawFBP, click on this link.
