FlowBasedProgramming | RecentChanges | Preferences

FBP Comments and Responses

Dave Bennett's Feedback

DB's comments in normal font; PM's in italics....

In my view, the following gets to the heart of the matter:

This raises a very interesting and perhaps critical issue: the division of labour between nodes and network, and between reusable code and custom code. I can see two approaches, which seem to me to be rather different.

Method CC is component-centric. It gives the network a relatively minor role, more or less confined to making sure that the outputs generated by component X are correctly passed on as input to component Y. Most details of the business logic are encoded into components, and as the logic becomes more complex so the components become larger and more specific to a particular task. Certainly the network allows a component used in one sequence of operations to be reused in another, but only where very similar business logic is needed in both situations.

Method NC is network-centric. The components are relatively small, atomic and highly reusable. Many of their operations are driven by parameters, and the parameters may even form a kind of mini-language interpreted by the component to carry out its simple task. By contrast, the network becomes richer and more complex, with much of the business logic expressed directly in terms of routing between generic components and specific parameters sent to those components.

In method CC the design process will allow a network to be sketched out quite early on, and the hard work goes into designing the individual components and the packets they use for communication. The tools for network design can be quite simple, but designing and testing the components will be the challenge.

In method NC the design of the network is a major part of the task. Many of the components will be drawn from standard libraries, but exactly which to use and how to stitch them together is a significant challenge. The components are easily tested in standard frameworks, but the network is a kind of programming language in its own right and will require good design tools and good support for testing.

As I read your book and look at how you’ve approached this subject, I’m left with the clear impression that you have favoured method CC. As I have constructed my particular sample, my thought pattern has quite definitely leaned towards method NC. I have found myself trying to design a network and functional decomposition down to quite small, atomic and highly reusable components, and I’ve been trying to push the specifics of a particular application up into the network layer. I can understand why you took the approach you did, but at least some of the problems you had to solve are no longer as pressing as they once were. Now I think we would place a higher priority on building an application very quickly using a visual designer and a library of pre-written components.

What I find fascinating is that I strongly lean to NC, but apparently I didn't give that impression in my book. What may have happened was that a) most of the work was done when using the technology intensively for the first time, so most of the components were new, and b) for an application that was new for most of us, so we did not have a base of reusable components to build on... Maybe c) is the point you yourself make - namely that at least some of the problems we had to solve are no longer as pressing as they once were. --pm

Absolutely. The problem is that once you take the NC view, the network language is not powerful enough. Yes, you could have mini languages but (a) it doesn’t seem the right answer to have a different mini language for each component and (b) since quite a bit of application logic will go into the mini language, it should be readable ie programmer friendly.

You need a more powerful language to describe the network itself. I’ve talked a bit about connections that work differently without the component knowing about it.

1. Normal pipe

2. Close before open pipe (avoids deadlock).

3. Storage pipe (allows data to be placed in suspension for later processing)

How are 2 and 3 different from

where the red blobs represent "automatic" ports ("auto out" means send a signal when the component terminates; "auto in" means defer start until a packet or close-down arrives)?

Another point is that you can use different write/read pairs, e.g. SQL, DL/I, XML write/read, etc.

The network in that diagram is intended to carry out a simple function: read a bunch of records into a temporary holding file, then at end of file start up another function to read the temporary file and pass it along. That network contains an abstract idea that could be useful in many places.

I agree – in this case they have exactly the same effect. However, I would like to capture that concept into a single connection that does both the synchronization and the temporary file stuff, with several purposes in mind.

1. To raise the level and power of the network definition language.

2. So it’s easier to reuse.

3. Hopefully for that concept to be applicable in other areas. For example, I hope it might handle a recursive looped call, like a parts explosion.

4. Perhaps it will suggest other kinds of “smart connection” for other network problems eg recursion, stacks, stores.

I think I see what you're driving at! Let's just say that I lean away from adding a new connection type unless we absolutely have to. That said, we could handle what I call the "infinite queue" as a subnet along the lines of the diagram I sent, or as a single component as it is a perfectly sequential concept (we only need one thread to execute it). If we handled it this way (or as a subnet) we could do the bill of materials explosion nicely as a pattern of: A feeds B (the new component) feeds A. You know, we don't even have to add a construct to the definition language if we have a nice diagramming icon for it - we could simply generate a component or subnet instance.

In AMPS we allowed a connection to have both ends attached to the same component, so it became a FIFO store, but as we have noted before you have to make sure it's big enough - which may be non-obvious! So the A feeds B feeds A structure would handle that nicely too!

As I mentioned before, in my view stacks fit very nicely with the processing requirements for nested substreams. If you look at the application logic described in http://www.jpaulmorrison.com/fbp/substrs.htm - look for Figure 9.4 - you see that we need to be able to push and pop on demand, which suggests to me we need push and pop methods on Component. Actually I thought we had them, as the Stack declare is there, but they seems to have been dropped! I am attaching some sample code below. Peek can be handled by pop followed by push - or it could have its own method. I don't see that a stack component would work as nicely...

So I think we are pretty much on the same wavelength - it's just a gut feeling I have about wanting to avoid multiple connection types. The basic question is whether it makes the mental model more or less complex! Your gut may tell you different!

  * Pop from stack. Return null if empty.
     Packet pop (){
         Packet p;
           try {p = stack.pop();}
           catch (EmptyStackException e) {
               p = null;
            return p;
   * Push onto stack.
     void push (Packet p){

You also need something to configure generic components. Here are some possibilities.

1. Sort.

Field “SURNAME” ascending

Field “DOB” descending

2. Filter.

Field “SEX” equals “M”

Field “DOB” less than “19570101” or Field “DOB” is null

3. Directory listing

Path “r:\main\” pattern “*.txt”

Path “s:\archive\” pattern “*.txt”

There are ways to do all those things with Unix pipes using sort, grep and find, but command line arguments are not a nice mini language. I really like the idea that a packet contains a dictionary and all the data is in the dictionary, so you can refer to it by name as in the examples above.

Re the Sort, filter and directory listing, they seem pretty straightforward, but I am beginning to agree with you that the IIP concept needs some expansion... --pm

A couple of thoughts…

You have defined 6 “standard” ports: IN, OUT, IN*, OUT*, IN[], OUT[].

Minor quibble: IN and OUT are conventions, not standard; *IN and *OUT are standard because the components don't know about them. --pm

I think there are 2 others.

OUTN is a standard “null” output. This is for packets NOT written to the OUT port or some other specified output. It is firstly for packets excluded by a filter, so you can use a filter as a splitter, or complement the filter by connecting the OUTN port instead of the OUT port. Secondly, it can be used for all dropped packets. Then you don’t need DropPacket?. By default it is connected to a bit bucket.

I like it! In the last implementation of FBP that we worked on at IBM (non-Java), we had up to 3 network-level ports for undelivered, error and trace packets, that could then be attached to any desired components, e.g. print, compare, etc. This sounds pretty close to what you have in mind. --pm

CONFIG is the configuration port. It is normally connected to an IIP, but may connect to some other OUT port.

Also, I think IIPs need to do more. A single packet is not really enough. I think it’s more useful to be able to define a mini language as a series of lines of text, with one line per packet. Then a mini language looks the same whether read from an IIP or from a text file.

Finally, the logic of when to start and MustRun needs to change.

1. A component that opens a CONFIG port where the CONFIG data is not ready yet (not an IIP) should not be started. It doesn’t have its instructions yet.

2. A component that does not open an IN port is started immediately (subject to rule 1). It is assumed to be a generator.

3. A component that opens an IN port is started when the data is ready, not before (IIP or not). It is assumed to depend on having input.

4. A component that opens a CONFIG port and has data (IIP or not) is started depending on rules 2 or 3. You need instructions and either have input data or be a generator.

Interesting: I need to think about this! --pm

On diagramming

pm wrote: have sketched out your concordance application (see the attachment - sorry it's a bit wiggly), and it started me wondering... Originally I designed the file icon as annotation only - i.e. not to generate any code - but what if we used it as shorthand for a Read or Write component, plus its associated parametrization? Currently the diagramming tool allows double-clicking on a component to drill down to a lower level, but there is no reason why other icons can't be double-clicked on to yield more detail.

So 1) double-clicking on a File icon could bring up a subnet with an I/O component and associated IIP.

2) another way to slice the pie would be to specify the component name underneath the component (as DrawFBP does currently for Java components, once you decide on them), and support double-clicking to show parameters expressed as a string, e.g. XML.

3) Still another alternative would be to add parametrization to the list of functions available when you right-click on a File icon.

We should try to make diagramming at least as easy as typing in text strings - it is just a lot more natural for human beings. However, while it's relatively easy to draw a high-level diagram, it's the question of how to add enough information to allow a working network to be generated that gets interesting!

Here's the diagram:

David Bennett's answer:

CMSP and PowerShell? got me thinking again about how simple things are when you just read the command line and write the console.

Then your diagram convinces me how much more I personally would like to be able to pull up a diagram, plug in a couple of parameters and just click Run!!

For debugging a network, I’d like to be able to (a) single step (b) inspect the contents of any connector (c) inspect/modify/insert packets at will. I don’t want to save/compile/run. I want to run the diagram piece by piece, right where it is.

Wouldn’t you?

Re parameters again: assume they are a list of keywords with optional values. Introspection on the component tells you what keywords it understands, and what value types. The Filter component understands (say) the following parameters:

Sort understands:

And so on.

Yup, that's what I want! --pm

FlowBasedProgramming | RecentChanges | Preferences
This page is read-only - contact owner for a password | View other revisions
Last edited December 16, 2007 1:24 pm by PaulMorrison (diff)