Gds Machines Overview

Goto: Documentation Home

Contents

About

• Overview
• Examples of Use

Technical

• Reading Diagrams
• Creating Machines
• Connecting Data

Deep Technical

• Mesh Processing
• Publishing Data
• Creating Components
• DataNodes
• Datatypes
• Predicates
• Chips
• Environment
• Machines
• Optomisation
• IPO vs Events
• Distributed Processing
• Javascript Machines
• English Machines

Gds Machines allow you to quickly define complex processing tasks using a graphical representation.

Business users typically define reports and requirements using flow diagrams, with Gds Machines these diagrams can be executed by assembling building blocks together, the same way a child constructs a crane out of levers and pulleys

(@@picture of spam filter)

• Machines allow rapid prototyping, even by end consumers
• Information can be pulled from many sources, such as databases, emails, spreadsheets, internet.
• Programmers can create machines for specific high level tasks in isolation, and end consumers do not need to understand the complex internals of an operation.
• Machines replace the need to create complex SQL select statements and are suited to reports where SQL cannot realistically be used.

Machines execute runtime decision and query processing, much like SQL in a database. machines define a number of tiny highly focused engines (machines) which perform a single task, such as fetching values, or searching for single scalers or intersecting two sets. Each machine can only communicate with other machines over a MachineWire, passing MachineDataNodes. This specialisation allows for highly efficient algorithms to be implemented, and also permits relatively easy multi threading and grid operation. Machines are very much like electronic circuits, where each discrete chip performs a highly specialised function and the way in which these chips are arranged produces the overall outcome.

Gds machines create a series of primitive steps that can be applied to data and processing. These primitive steps (or "chips") are similar to electronics integrated circuits; each chip performs a specialised task, but without much knowledge of the overall environment. These chips can be dynamically assembled to create large processing machines in much the same way as electronic components are assembled together to create larger machines.

Gds machines are often quite high level in action. There is a machine that reads CSV format files and produces output, others provide a range of filtering and sorting, while still others perform dictionary lookups.

Gds machines were primarily designed to allow compotent but non technical users to describe a process, such as creating a report, without having to learn a programming language. As machines are dynamic, users can experiment with actual data and see results immediately. With machines being high level, much of the technical detail is abstracted away, allowing reasonably simple graphical depections of the process.

Machines have the following characteristics

1. Machines are created in high level languages to perform a specific function, such as calculating a formula
2. Machines are dynamically created and called via a machine executor
3. Operating machines can be very fast to run, the example above with dictionary searching will often run faster than the equivalent in a database engine.
4. As machines are performing a single task, with control on definition, input and output, they can be automatically tested
5. The machine executor is able to abstract away network operation, and machines can be running in a networked grid environment without knowing it.
6. The machine executor is able to launch multiple alternative techniques to solve the problem, running them in parallel. This is especially useful in searching type queries where different algorithms may excel with certain types of input patterns.
7. An machine optimisation phase will try and replace common user machine designs with more optimal designs, for example replacing a "fetch"->"filter" pair with a "fetch-filter" machine where it exists.
8. The description of machines (STEM and STEP blocks) defines the interconnections between machines, but not the realm of machines available. This allows machines to be translated to a range of different runtime languages, including Javascript.

Internals

Each operating machine has a "base" definition. This defines the actual code stream to execute, its input and output buses and constants. In OO programming terms, this is an instance of an object.

Machines are wired to each other via a BUS. Multiple machines can share the same input bus, it is not limited to 1-1 linking. In electronic terms, the fan-in is 1, the fan-out is unlimited.

Each bus, carries "data-nodes", which are objects holding data from previous steps. Each machine is free to add, edit, delete, hold etc the data nodes. Individual machines can only communicate via data-nodes, if there is a need for other types of communication for some reason, then combine the routines and create an uber-machine.

Machines do not need to buffer explicitly or flow control. The machine executor will select machines for execution and manage the flow control between machines.

DataNodes

Creation, stack, from machine only use DN, not external signals

	var m = new Machine;
	var p = new ChipListProducts(m);
	var s = new ChipCheckHaveSales(m);
	s.SetRequiredYear(2012);
	p.WireOutput(s);

	// Sync processing
	m.RunSync();
	while (! s.IsEOF()) {
		var record = s.GetOutput();
		// Process single record of product,  which has sales in 2012
	}

	// Or Async processing
	s.WireOutput(SingleRecordCallbackFunction);
	m.RunAsync();