\subsection{Experiment server}% -   -   -   -   -   -   -   -   -   -   -   -
\label{sec.server}

One source of rigidity in our previous software was its hierarchical tree architecture, with a single block of code orchestrating the whole proceedings (see Figure~\ref{fig.arch_tree}).
This setup allows well organized control over the experiment, but also imposes two restrictions.
First, any functionality that requires user-set parameters must have those parameters passed to it down the chain of command.
Therefore, if a low-level function needs more information, a method for passing the additional parameters down must be added to every function between the altered and root functions.
Second, the experimenter needs to predict and prepare for many possible situations before the experiment actually begins.
Many problems aren't recognized until the experiment is actually underway, and restarting the experiment after extending the software to cope with the errors is a time consuming process.

% Some searching around for alternative architectures led us to EPICS\cite{epics}, a control-software infrastructure mostly used in large accelerators and telescopes.
% EPICS uses Channel Access, a message passing layer, for communicating between user-side clients and hardware-side servers.
% In EPICS, each variable has its own name (like a URL) that allows clients to get and set its value.
%The resulting network of is also extremely flexible and robust, because all the nodes are well isolated from each other, and failure in one section of the network will not spread to other regions.

% Channel Access is designed for large, distributed systems, and using their code directly seemed overly complicated for our simple setup.
% We decided to construct a similar modular, message passing network for our situation.
%The network architecture allows direct communication between any two nodes on the network as well as hot-pluggable extensions (see Figure~\ref{fig.arch_net}), allowing the development of nimble control systems.
\begin{figure}[bt]
 \centering
 \sbox{\tempbox}{\includegraphics[width=0.4\textwidth]{arch_net}}
 \subfloat[Tree]{\label{fig.arch_tree}\raisebox{0.5\ht\tempbox-0.5\height}[0pt][0pt]{\includegraphics[width=0.5\textwidth]{arch_tree}}}
 \hspace{\stretch{1}} 
 \subfloat[Network]{\label{fig.arch_net}\usebox{\tempbox}}
 \caption{A comparison of (a) tree and (b) network software architectures.
   While the tree architecture favors structure and organization, the network architecture favors flexibility.}
\end{figure}

%\subsubsection{Evolution}
%\label{sec.evolution}

%We decided to develop our experiment network in a Linux/Unix (*nix) environment in order to leverage tools already developed for these systems and our experience developing for such environments.
%We initially envisioned the nodes as simple command-line utilities, and the actual experiment software as a collection of connecting scripts.
%After some thought, we noticed a few difficulties with this design.

%The difficulty of piping together the few processes needed to preform a task involving much inter-node communication is daunting.
%The number of connections can be expected to grow quadratically with the number of processes, which quickly makes hand-assembly difficult.
%% Nodes   max possible connections
%% S_n = (n-1) + S_{n-1}
%% dS/dn = n-1
%% S = 1/2 n^2 - n + C ~->~ n(n-1)/2 + C
%% S(1) = 0, so C=0
%% S(n) = n(n-1)/2      in english, each node needs to connect to (n-1) others, divide by 2 because link is non-directional
%%   check:
%% S(n) - S(n-1) = n(n-1)/2 - (n-1)(n-2)/2 = (n-1)/2 [n-n+2] = (n-1)  good.
%%
%% Nodes   up to 2 connections per new node
%% S = 2n - 1       for n >= 1
%There is also the difficulty of opening the pipes or sockets in the correct order to avoid locking (the starving philosophers problem).
%We decided that the simplest solution would be to have a single central server handling the connections.
%While the addition of a server would make the overall system more complicated, the server would be experiment-independent and only need to be written well once.
%Future experimenters could focus on simple client processes and ignore the mostly transparent server layer.

We decided to replace our tree architecture with a network architecture based on a central experiment server (see Figure~\ref{fig.arch_net}).
Besides simplifying inter-process communication, the server would allow hot-pluggable processes, simple logging and debugging, and easily maintained client state information.
Individual processes could be taken offline, adjusted, and reattached to the server without any change in real-world output.
Diagnostic processes could also take over control of the real-world interface of a temporarily paused experiment processes to assist troubleshooting.
An example of what the experiment-server could look like in operation is given in Appendix~\ref{sec.cmmdline_example}.

% The simple logging is a side benefit to having a single central server.
% Dumping all the messages passed through the server into a log file allows you to reconstruct the state of the entire system at any point during the experiment.
% This is not the most efficient way to monitor system status, but is excellent for troubleshooting when you don't want to take the time to write a logger for the specific situation, or don't know which parameters to focus on.
% Even complicated logs of this nature can be easily parsed using standard *nix tools like \texttt{grep} and \texttt{sort}.

% Easy debugging comes from a combination of the quick-and-dirty logging described above, and the enforced modularity of the network architecture.
% The logging allows quick isolation of the node exhibiting nonstandard behavior.
% The modular node can then be tested independently to pinpoint the flawed code.
%%The modularity allows each node's functionality to be tested independently and simply using scripts to check that all valid requests of a node generate proper replies.
%%Programming errors resulting in program crashes are also isolated to small nodes, and not occuring somewhere inside a single huge program.

% To understand the benefits of maintaining client state information, consider a stepping motor control node.
% In order to properly handle commands like ``take one step closer'', the process must remember its current position.
% This information could be saved to a parameter file and reloaded upon each invocation, but with the constantly running central server, the stepper node could also run constantly and simply retain its current position in memory, allowing simpler and more intuitive code.

% As a final benefit, the network design allows several nodes to share access to a single low-level node, easing the management of limited resources like input and output lines.
% Our input/output card has only two independent clocks for timing input and output, and several different tasks need different kinds of operations.
% Having a single node managing the situation makes setting up the card more centralized and allows the other nodes to focus on generating output and interpreting input, without worrying about the ugly details.
% An example of what the experiment-server could look like in operation is given in Appendix~\ref{sec.cmmdline_example}.

% \subsubsection{Server design}

% The server was designed to handle the addition and deletion of new clients and to robustly pass messages between them.
% The current version began as an example server from Lewis\cite{lewis98} and has evolved as my understanding increased.

The network uses a request-response message passing paradigm where a node that wishes to initiate an action or retrieve some information sends a request to another node and then waits for a response.
To get an idea of the type of behavior we expect the server to handle, consider the example conversation in Table~\ref{tab.command_trace}.
% This is roughly equivalent to the function-return paradigm of our old setup.
% One added benefit is that new commands (such as \texttt{cancel-action}) can passed in while another action is being processed.
% This behavior can be approximated in a functional paradigm using pass-by-reference, but the message-passing paradigm frees you of this need to prepare for all possible updates ahead of time.
\begin{table}[bt]
 \centering
 {\ttfamily
 \begin{tabular}{l|l|l}
  From & To & Message \\
  \hline
  Cmmdline & Stepper & Set:AbsPos 3 \\
  Stepper & Dig. Out & Set:Output 1 \\
  Dig. Out & Stepper & DONE \\
  Stepper & Dig. Out & Set:Output 2 \\
  Dig. Out & Stepper & DONE \\
  Stepper & Dig. Out & Set:Output 3 \\
  Dig. Out & Stepper & DONE \\
  Stepper & Cmmdline & DONE
 \end{tabular}
 }
 \caption{Tracing the messages for a single command.}
 \label{tab.command_trace}
\end{table}

% The basic format for a message is
% \begin{center}
%  \begin{tabular}{r c l}
%   Request & : & ``\texttt{<Return ID>:<Target client>|<Message>}'' \\
%   Response & : & ``\texttt{<Server ID>|<Message>}'' \\
%  \end{tabular}
% \end{center}
% Where the \texttt{Return ID} is a tag for the requesting client to match the eventual response with the corresponding request, and \texttt{Server ID} is a similar tag for the server.
% The delimiting characters `\texttt{:}' and `\texttt{|}' were chosen as being unlikely to appear in common messages.
% The other parameters should be self explanatory.

% The current message handling logic is given in Figure~\ref{fig.msg_handler}.
% \begin{figure}[bt]
%  \centering
%  \includegraphics[width=\textwidth]{handler}
%  \caption{Message handling logic.  This example following a single request-response pair through the logic is explained in detail in the text.}
%  \label{fig.msg_handler}
% \end{figure}
% Following the path of a single response-request pair through the server will clarify the process and demonstrate its flexibility.
% The actual message handling procedure is complicated by the need to handle adding and removing clients from the registry, but we ignore those steps for clarity.
% \begin{packed_enum}
%  \item Receive input from client \texttt{Cmmdline}, ``\texttt{Cmmd1:Add|2+2}''.
%  \item Assign a unique \texttt{Server ID} to this message, ``572''.
%  \item Parse message into \texttt{Return ID}, \texttt{Target client}, and \texttt{Message}.
%  \item There is a target client \texttt{Add}, so the message is a request.
%  \item Search the registry for the client \texttt{Add}.
%  \item Send the request to \texttt{Add} with the \texttt{Server ID}, ``\texttt{572|2+2}''
%  \item Add the handled reply to the stack of replies waiting for responses.
%  \item Wait for a response\ldots
%  \item Receive input from client \texttt{Add}, ``\texttt{572|2+2=4}''.
%  \item Assign a unique \texttt{Server ID} to this message, ``573''.
%  \item Parse message into \texttt{Return ID}, \texttt{Target client}, and \texttt{Message}.
%  \item There is no target client, so the message is a response, and the \texttt{Return ID} ``\texttt{572}'' is actually a \texttt{Server\ ID}.
%  \item Search the stack of requests waiting for responses for request ``\texttt{572}''.
%  \item See that the original request was from client \texttt{Cmmdline}.
%  \item Send the reply with the original \texttt{Return ID} back to \texttt{Cmmdline}, ``\texttt{Cmmd1|2+2=4}''.
% \end{packed_enum}
% With this simple logic, the server should be able to handle asynchronous message passing between many clients efficiently and robustly, providing a firm foundation for a modular experiment control network.

% \subsubsection{Client design}

There are two paths towards adding functionality to the experiment control software: full fledged, experiment server clients and stand-alone, command line programs.

Many simple procedures are best implemented as stand-alone, command line programs.
These programs can call server clients through the \texttt{Cmmdline} client, which allows easy text access to the server.
Stand alone programs can in turn be called by server clients through the \texttt{Exec} client, which simply executes the command passed to it in the \texttt{bash} shell and returns the results.
An example stand alone script for automatically approaching the sample surface is given in Appendix~\ref{sec.approach_surface}.

More complicated procedures requiring preserved state information, complicated initialization, user feedback, or continuous monitoring are best implemented as full-fledged clients, so they can plug into the server and remain running for the duration of the experiment.
The clients can be written in any language capable of handling sockets or linking to our library of socket routines, so the user can chose whichever language seems most suited to the task at hand.

% When they are required to support user feedback or changing needs (\texttt{cancel}, \texttt{abort}, etc.), asynchronous messages can add a fair bit of complication to a client.
% Care must be taken to properly check for and handle new requests, while continuing to process any outstanding ones.

% \subsubsection{Conclusions}

%As discussed in section~\ref{sec.evolution}, the server software design will encourage robust modularization of code, allow hot-pluggable diagnostic tools, and provide simple logging.
%To summarize, the server software design will encourage robust modularization of code, allow hot-pluggable diagnostic tools, and provide simple logging.
The enforced modularization of code will allow easy testing of individual modules and let new users learn the software in manageable chunks.
The hot-pluggable client-server architecture will allow diagnostic tools to be loaded on the fly.
The common text interface will encourage the construction and reuse of flexible analysis and diagnostic tools, providing easier troubleshooting, as well as easing the transition between different software versions.
Finally, simple logging by saving all messages passing through the server will increase the volume and quality of record-keeping, allowing easier forensic analysis of experiments-gone-wrong and providing sufficient material to reproduce interesting effects.

The experiment server is currently stable under light loads, and has been used to successfully run the surface approach script given in Appendix~\ref{sec.approach_surface}.
% as well as acquire the data presented in Figures~\ref{fig.stepper_stability} and~\ref{fig.step_size}.
