Information on Nial
Brief Lecture on Nial
Simple Examples
Advanced Examples
Uses of Nial
Simple Syntax
Hybrid Language
Extending the Language
Bibliography

NIAL stands for the Nested Interactive Array Language. It is a multi-paradigm language that combines aspects of both functional and procedural languages. The story of its name is found at The Name of the Language.

Nial is a hybrid language combining a functional array language based on Trenchard More's mathematical treatment of nested arrays called Array Theory, with a procedural language with familiar control structures. It has a rich set of language primitives that make it easy to rapidly develop loop-free data-driven algorithms.

Nial and its implementation were designed by a team led by Mike Jenkins while he worked at Queen's University in Canada and refined within the company after his retirement.

Q'Nial has been used for applications in decision support, knowledge based systems, scientific computing, and data analysis. Mike Jenkins continues to use it to support rapid prototyping of web based applications.

Brief Lecture on Nial

Click on the link Lecture on Nial to view a 3 unit lecture on Nial prepared for a talk at NYU in April 2009.

Simple Examples

Nial is designed to allow easy construction and manipulation of data structures. For example, the statement

	A := 3 18 27 45 7 23;

	sum A

	sum A / tally A

	average IS OPERATION Numbers {
            sum Numbers / tally Numbers }

	average A

	Record1 := ['Helen Jones', "345-1467 , 87 76 92 57 65 88];

     Record
+-----------+--------+-----------------+
|Helen Jones|354-1467|87 76 92 57 65 88|
+-----------+--------+-----------------+

A pair of student records would be displayed as:

     [Record1, Record2]
+----------------------------------------+-------------------------------------
|+-----------+--------+-----------------+|+------------+---------+-------------
||Helen Jones|354-1467|87 76 92 57 65 88|||Adam Trudeau|216 -2975|56 85 42 13 7
|+-----------+--------+-----------------+|+------------+---------+-------------
+----------------------------------------+-------------------------------------

-----+
----+|
7 56||
----+|
-----+

The language contains many operations that manipulate data structures in one operation: For example, the the operation "pack" combines the above two records item by item into a list of pairs:

     pack [Record1, Record2]
+--------------------------+--------------------+------------------------------
|+-----------+------------+|+--------+---------+|+-----------------+-----------
||Helen Jones|Adam Trudeau|||354-1467|216 -2975|||87 76 92 57 65 88|56 85 42 13
|+-----------+------------+|+--------+---------+|+-----------------+-----------
+--------------------------+--------------------+------------------------------

-------+
------+|
 77 56||
------+|
-------+

Programming is done by defining operations that manipulate the data structures. For example, we saw the definition of "average" above. Another example is:

	get_averages IS OPERATION Records {
		Nms Stnos Marks := pack Records;
		Avgs := EACH average Marks;
		pack Nms Avgs }

     get_averages [Record1, Record2]
+------------------+----------------------+
|+-----------+----+|+------------+-------+|
||Helen Jones|77.5|||Adam Trudeau|54.8333||
|+-----------+----+|+------------+-------+|
+------------------+----------------------+

For programmers who wish to express algorithms succinctly, Nial supports several functional combinators that build operations from simpler components. For example average can be defined by

	average is / [sum, tally]

	[sum, tally] 3 18 27 45 7 23
123 6

 	/ 123 6
20.5

Advanced Examples

In this section we present three examples that illustrate the power of Nial as a language for data manipulation.

Finding the position and text of tags in an HTML file.

Tags in an HTML file are surrounded by "<" and ">" characters. For simplicity in the example we will assume no other uses of these symbols are present. The algorithm we will use is:

• Read the file into memory as one string,
• Find the positions of the markers.
• Using address arithmetic, select out the fields corresponding to the text of the tags,
• Return the posns and text for the tags as a pair of lists.

The following two routines do the work:

# routine to look for html tags in text stored as a long string.

findtagtext IS OPERATION Text {
   Hdposns := `< findall Text; 
   Tlposns := `> findall Text;
   Lengths := Tlposns - Hdposns + 1;
   Tags := Hdposns EACHBOTH + EACH tell Lengths EACHLEFT choose Text;
   Hdposns Tags }

  test is op fnm {
    findtagtext readfield fnm 0 (filelength fnm) }

Here is an example of the use of the test operation:

     Posns Tags := test "intro.htm;

     tally Posns
1978

     first Tags
<!DOCTYPE HTML PUBLIC "-//SQ//DTD HTML 2.0 HoTMetaL + extensions//EN">

There are situations in dealing with textual documents where there is a need to do computation on tabular data stored in textual form. We make the assumption that the lines of the document that contain the file have been isolated into a separate file by a previous step. The computation we want to do to the table is to add two extra rows holding the sums and averages of the columns. The algorithm we use is:

• Read in the rows of the table as a list of strings,
• For each row find the numbers in the line by tokenizing the row using scan, selecting the tokens that correspond to numbers, and converting the tokens to real numbers,
• Check that the number of numbers in each row is the same, otherwise return a fault,
• Use pack to get the columns of numbers,
• Compute the sums and averages, and
• Extend the table with the new data and compute its picture as the result.

The following routines implement the algorithm.

# operation to select the numbers from a line of text.

linetonums IS OPERATION Line {
   Tkns := lo cutall rest scan Line;
   Numtkns := EACH first Tkns EACHLEFT in  16 18 sublist Tkns;
   EACH (1.0 * tonumber string second) Numtkns }

# routine to read in data compute sums and average of columns and display it.

tblcomp IS OPERATION Filenm {
   Svtrigger := settrigger o;
   Svfmt := setformat '%5.2f';
   Lines := getfile Filenm;
   IF isfault Lines THEN
      Res := Lines;
   ELSE
      Nums := EACH linetonums Lines;
      IF not equal EACH tally Nums THEN
         Res := fault '?tbl not valid for tblcomp';
      ELSE
         C := pack Nums;
         Sums := EACH sum C;
         Avgs := Sums / tally Nums;
         Res := picture mix (Nums link Sums Avgs);
      ENDIF;
   ENDIF;
   settrigger Svtrigger;
   setformat Svfmt;
   Res }

We use the file tab.data to test the example:

# show the data in the file:

     getfile "tbl.dat
+----------------+---------------+--------------+
|3.5,2.7,22,18,24|18,5,16.2,19,30|13,45,23,17,14|
+----------------+---------------+--------------+

     tblcomp "tbl.dat
 3.50  2.70 22.00 18.00 24.00
18.00  5.00 16.20 19.00 30.00
13.00 45.00 23.00 17.00 14.00
34.50 52.70 61.20 54.00 68.00
11.50 17.57 20.40 18.00 22.67

Efficient field selection from legacy system data files

A common need in data processing is to access data stored in files that have been created by a legacy information system. These are usually stored as flat files with fixed length records made up of preassigned fields. The record layout is known and can be described in terms of a sequence of quadruples: [Field_name,Offset,Length,Type]. Several kinds of selection might be wanted:

• selecting one field from one record
• selecting all fields from a block of records
• selecting one field from a block of records

The routines that follow provide support for doing the selections.

# support routines for accessing fixed length records

# record descriptor file format:
  Record_Name
  Record_size (in bytes including space for 1 or 2 newline
               characters if they are present)
  A field descriptor for each selected field in the format:
     Field_Name StartPos Length Datatype

# the above field descriptor is deliberately redundant so that
 fields may be omitted. The Datatype is either A for alphanumeric
 or N for numeric.

# routine to read a record descriptor from a file and store
the information internally as numbers or strings.

get_record_descriptor IS OPERATION Fname {
  % routine to split one field descriptor into its parts;
  convert_field_descriptor IS OPERATION Str {
    Tokens := EACH (string second) (lo cutall rest scan Str);
    Field_Name := phrase first Tokens;
    StartPos := tonumber second Tokens;
    Length := tonumber third Tokens;
    Datatype := 3 pick Tokens;
    Field_Name StartPos Length Datatype } ;
  % get the lines of the file;
    Lines := getfile Fname;
  % select the record name from the first line;
    Record_Name := second scan first Lines;
  % select the record size from the second line;
    Record_size := tonumber second Lines;
  % split the field descriptors in the remaining lines;
    Field_descriptors := EACH convert_field_descriptor (2 drop Lines);
  Record_Name Record_size Field_descriptors }

# routine to use the field descriptors to produce a list of list of
  indices for the items of the fields for a record. The list can be
  used by EACHLEFT choose to select the list of fields.
  
field_selectors IS OPERATION Field_descriptors {
  Posns  := EACH second Field_descriptors;
  Lengths := EACH third Field_descriptors;
  Posns EACHBOTH + EACH tell Lengths }
  
# routine to get a block of N records from a file Fname
   starting at record J as a list of lists of fields.

get_block IS OPERATION Fname Record_descriptor N J {
  % decompose the record descriptor;
    Record_Name Record_size Field_descriptors := Record_descriptor;
  % get the block of data for the N records;
    Datablock := readfield Fname (J*Record_size) (N*Record_size);
  % compute all the field selectors for all the records in the block;
    All_selectors :=  (field_selectors Field_descriptors
      EACHRIGHT + (Record_size * tell N));
  % choose all the fields of the selected records;
  All_selectors EACHLEFT EACHLEFT choose Datablock }

# routine to get one field from a block of N records starting at
  record J.
  
get_field IS OPERATION Field_Name Filename Record_descriptor N J {
 % split the recod descriptor into its parts;
    Record_Name Record_size Field_descriptors := Record_descriptor;
 % find the field descriptor for the named field;
    Field_desc := Field_Name find EACH first Field_descriptors pick 
      Field_descriptors;
 % compute the indices to select all the fields;
    Startpos length := [second, third] Field_desc;
    Selectors := tell N * Record_size + Startpos EACHLEFT + tell Length;
 % read the block of records and select the fields;
    Datablock := readfield Filename (J*Record_size) (N*Record_size);
    Fields := Selectors EACHLEFT choose Datablock;
 % convert the field to numbers if numeric;
    IF Field_desc@3 = 'N' THEN
       EACH tonumber Fields
    ELSE
       Fields
    ENDIF }
 

We illustrate the effectiveness of this approach by doing a single record selection and by timing large selections from a file with 10,000 records with record format:

   Name 0 - 19
   Age  20 - 22
   unused 23 - 24
   Job 25 - 44
   unused 45 - 49

# The record descriptor for this file is:
     Record_Foo
     50
     Name 0 20 A
     Age 20 3 N
     Job 25 20 A

# get record description

recdesc := get_record_descriptor "foorec.dsc;

# test to read and decompose one record;

     write first get_block "lgfile Recdesc 1 0;
+--------------------+---+--------------------+
|Sam                 | 45|Steward             |
+--------------------+---+--------------------+

timeit get_block "lgfile Recdesc 10000 0;
2.1

timeit get_field "AGE "lgdata Recdesc 10000 0;
3.43

Uses of Nial

Nial is convenient for rapid design of data manipulation programs because it builds into its predefined operations many of the loops that are required in a conventional programming language. For example, you have seen in the simple examples that "sum" adds all the items in a list, and that "pack" reorganizes the data in a list of lists.

This automatic looping is also done for defined operations by using built-in second order functions (called transformers in Nial) that apply an operation to each item of a list. The use of the transformer EACH is demonstrated in the definition of "get_averages" above.

Nial is an effective tool for doing data intensive computations on large amounts of data. Using the file access operations of the language, large data arrays can be loaded into the working memory and various computations performed. Because the primitive operations are implemented by compiled "C" code, the running time of such data manipulations is comparable with doing the same computations directly in a C program (usually within a factor of two or so), and hence using Nial for such problems allows for rapid development with only a small speed penalty.

Nial has been used in a number of commercial applications in a variety of fields. These include: insurance underwriting, statistical computations, database construction, graphics animation and machine learning in protein crystallography knowledge bases.

Nial has been used as a prototyping tool in research projects in AI, databases, and textual information systems at several universities. At Queen's it has been used for teaching concepts in AI and functional programming.

Simple Syntax

The syntax of Nial been designed to be easy to use. It is based on a few principles:

• The syntax of expressions uses juxtaposition rules to provide a straightforward functional notation that matches the semantic intent.
• All operations are inherently unary, but infix use is permitted and results in a left-to-right interpretation.
• There is no precedence rules for operations in infix usage (a consequence of the interpretation that it is the unary function applied to a pair)
• All valid program constructs denote either an array expression, an operation or a transformer.
• All the control constructs denote expressions and have a corresponding value.
• All the control constructs are bracketed.
• Parentheses, "(" and ")", are only used for grouping and can be placed around any valid program construct.

Juxtapositional Syntax of Expressions

The following juxtapositions are supported:

• List formation (array expression): 3 (4+5) 18 'abc'
• Operation application (array expression): f A
• Left Currying (operation): A f
• Operation composition (operation): f g
• Transformer application (operation): T f

Bracket Comma Lists of Expressions and Operations

The syntax is extended with an alternative list forming notation. The expression

    [3,4+5,18,'abc'] 

    [3,4 5,7 8 9]

A similar notation can be used for a list of operations, which is called an atlas . For example, in the expression:

     [first,rest] count 10
+-+------------------+
|1|2 3 4 5 6 7 8 9 10|
+-+------------------+

User Definitions

The language is extended by naming program fragments using the keyword IS. The definition mechanism can name either array expressions, operations or transformers. Examples of each of these are:

     Getname IS { readscreen 'Please enter your name: ' }

     median IS OPERATION A {  tally A quotient 2 pick sortup A }

     TWICE IS TRANSFORMER f OPERATION A { f f A }

The body of the operations and array expression defined above are blocks consisting of a single expression. In general a block can have local definitions followed by an expression sequence of one or more expressions.

User defined names can be used in exactly the same way as predefined ones. For example, the operation median can be applied to a list of numbers Nums by

     median Nums

     EACH median Lists_of_nums

Nial is a hybrid language in that it supports both both functional programming using the array expressions, operations and transformers discussed above and imperative programming, in which assignments can be made to variables and control structures for selection and iteration are used for flow of control.

This hybrid nature of the language shows up in the advanced examples above. Consider the definition:

findtagtext IS OPERATION Text {
   Hdposns := `< findall Text; 
   Tlposns := `> findall Text;
   Lengths := Tlposns - Hdposns + 1;
   Tags := Hdposns EACHBOTH + EACH tell Lengths EACHLEFT choose Text;
   Hdposns Tags }

The fourth line of the definition uses the powerful functional notation to select all the Tags in one step. The final line of the definition is the result expression, in this case a pair of lists.

The control structures provided in Nial are:

• if-then-else conditional expressions (with elseif subclauses)
• case selection expression
• for-loop definite iteration expression
• while-do-loop indefinite iteration expression
• repeat-until-loop indefinite iteration expression

In most programming languages there is a core language and extensions are achieved using a separate procedure or function call mechanism. In Nial, user definitions that extend the capability of the system are invoked in exactly the same way as predefined ones. Thus, Nial is inherently extensible in a simpler way than most languages. A library of commonly used definitions is provided with the system.

A second area of extensibility is the connection of Nial to programs written in another language. Different versions of Nial provide this in different ways. In the Windows versions of both Q'Nial and the Data Engine, a facility to invoke arbitrary routines in a dynamic load library (DLL) has been added.

Bibliography

Further information on Nial and Q'Nial can be found in the following references:

Franksen, O.I., Jenkins, M.A., " On Axis Restructuring Operations for Nested Arrays", Proceedings of the 2nd International Workshop on Array Structures ATABLE-92, Montreal, June 1992.

Glasgow, J., Jenkins, M., Blevis, E., Feret, M.,"Logic Programming With Arrays", IEEE Transactions on Knowledge and Data Engineering, 3 3 307-319 1991.

Glasgow, J.I., Lawson, D., Jenkins, M.A., Feret, M., "An Architecture for Real-Time Diagnostics Systems", Proceedings of The Third International Conference on Industrial and Engineering Applications of Artificial Intelligence and Expert Systems IEA/AIE-90, August 1990.

Glasgow, J.I., Jenkins, M.A., McCrosky, Carl, Meijer, H, "Expressing Parallel Algorithms in Nial", Parallel Computing Journal, 11 3 331-347 1989.

Jenkins, M.A., "Q'Nial: A Portable Interpreter for the Nested Interactive Array Language, Nial", Software Practice & Experience, 19 2 111-126 1989.

Jenkins, M.A., Glasgow, J.I., "A Logical Basis For Nested Array Data Structures", Computer Languages Journal, 14 1 35-51 1989.

Jenkins, M.A., Glasgow, J.I., Blevis, E., Hache, E., Lawson, D., The Nial AI Toolkit, Proceedings of the Avignon 8th International Workshop on Expert Systems and their Applications, June 88. Available as Tech Rept 88-212.

Chau, R., Glasgow, J.I. and Jenkins, M.A., "Fuzzy Information Management Using the Roster Model", " Proceedings of 21st Hawaii International Conference on System Sciences," (HICSS-21), January 1988.

Chau, R., Glasgow, J.I., Jenkins, M.A., "A Framework for Knowledge Based systems in Nial", 1987 Phoenix Conference on Computers and Communications Feb. 1987.

Glasgow, J.I., Jenkins, M.A., Hendren, L.J., "A programming language for learning environments", Computational Intelligence, 2 1 68-75 1986.

Jenkins, M.A., Glasgow, J.I., McCrosky, Carl, "Programming Styles in Nial", IEEE Software, January 1986.

Glasgow, J.I., Jenkins, M.A., Blevis, E., Chau, R., Hache, E., Whybra, J., "Experience with A.I. Programming in Nial", " Proceedings of STeP-86 Finnish Artificial Intelligence Symposium," Epsola, August 1986.

Glasgow, J.I., Jenkins, M.A., McCrosky, Carl, User Defined Parallel Control Strategies in Nial, Proceedings of the Symposium on Logic Programming pp 22-28, Boston, July 1985.

McCrosky, Carl, Glasgow, J.I., Jenkins, M.A., Nial: A Candidate Language for Fifth Generation Computing Systems, Proceedings of the ACM Annual Conference, San Francisco, October 1984.