Hardware pages: where the bits meet the metal.

Back in 2007 I submitted a patch to Comedi to support triggering analog output off the analog input start signal. The patch was accepted, and is now old news, but I just ran across my notes from developing it, and thought I'd publish them for posterity (and as a reminder to myself in case I have to dig into the internals of Comedi or NI drivers in the future).

Patch notes

To understand the patch, you should understand some portion of National Instrument's Data Acquisition Systems Timing Controller (NI DAQ-STC, see “Understanding NI DAQ-STC” below and the manual for more details).

We are trying to set up the AO timing so it triggers off of the AI start signal. In NI terminology, that means we want AI_START1 to be the source of AO_START1. The possible sources are listed in the DAQ-STC manual in table 5-2, p6-7 (380), where we see that AO_START1's 19th input is AI_START1. To implement this, we need to write to the card's AO_Trigger_Select register (Appendix B, register 67, pB-21 (470)).

Sounds fairly simple, we should just be able to set the appropriate start_arg. However, looking into source of comedi/driver/ni_mio_common.c, we see that no external triggering information is written to the AO_Trigger_Select_register. Looking back through the STC manual, we come across their AO trigger example (3.6.1.4, p3-23 (185)) which shows how to set up AO triggering. Comparing to ni_mio_common.c, we see that the AO triggering configuration in ni_ao_cmd() looks just like the “software triggered” case in NI's example. So it seems impossible to run externally triggered AO.

The patch edits ni_mio_common.c to make it more like NI's example and ni_mio_common.c's own AI code.

Understanding NI DAQ-STC

Consider the timing sequence shown in the Comedi manual. For output, all the same timings apply, but data is flowing the other way.

Here is a translation table between NI's terminology and Comedi's (NI references from STC manual):

Comedi AI NI AO Reference
seq_start AO_START1Fig 3-10, p3-12 (174)
scan_beginUPDATE Fig 3-2, p3-6 (168)
convert TMRDACWR Fig 3-2, p3-6 (168)

There are references to an AO_START as distinct from AO_START1, but it appears to be unsupported:

  • It does not show up in the manual's AO feature list (section 3.2, p3-2 (164), compare to the AI feature list, section 2.2, p2-2 (31))
  • It shows up in some contexts as unsupported (e.g. section 3.6.5, p3-36 (198))
Posted Thu Dec 9 12:26:57 2010 Tags: hardware

Comedi is a Linux COntrol and MEasurement Device Interface. We use Comedi in our lab to contol experiments via a NI PCI-6052E. The basic architecture for a Comedi application is:

kernel space   |   user space
               |
device         |
  |            |
kernel --------|-- libcomedi - application
  |            |
comedi module  |

To make matters more complicated, there are two version of the kernel module, one in Git at comedi.org and another in the kernel itself.

Luckily, many distributions now package versions of the in-kernel staging driver (e.g. Debian and Ubuntu). I was feeling left out on Gentoo, so I've just revived the old Gentoo packages and added them to my Gentoo overlay.

Posted Wed Dec 8 15:17:48 2010 Tags: hardware

Relying only on the measured deflection voltage to measure the AFM tip-surface interaction is risky, as the loss of the signal may not be obvious. In order to validate incoming photodiode signals, you should also monitor the total light hitting the photodiode. This is especially true in fluid cell measurements, as bubbles of air or buffer solution can deflect the laser enough to miss the photodiode completely.

As far as I can tell, the total photodiode voltage is not carried on one of the DB-25 lines connecting the MultiMode to the NanoScope controller (see the MultiMode internals page). It is displayed on the MultiMode LCD, so I monitored the lines connecting the main board to the LCD board to isolate the total photodiode line (LCD connector 2). I build a small connector to tap out that line (despite appearances, none of the huge solder blobs actually short anything out :p).

TODO: image of connector.

We sent the tapped line out through Auxilliary D (DB25 line 4), so we could tap it again at out main break-out box. Comparing this signal to the photodiode ground (DB25 line 12), the signal is actually inverted (-7V signal when there's lots of light, 0V when there's none).

Posted Fri Nov 19 12:54:05 2010 Tags: hardware

Here are my current guesses about the role of various connector lines inside the MultiMode II

DB-25 line descriptions

DB-25 line Role To
1 Stepper A Main 3
2 Stepper B Main 20
3 +5 Volts Main 7 (also 1,2,18,19 through 20 Ohms)
4 Auxiliary D (role?) Jumper 2
5 HV to piezo (-Y) Piezo 5
6 NCAFMJM pin 2 (role?) Main 8
7 HV to piezo (-X) Piezo 9
8 Piezo/head chassis ground Peizo 1; Head 13; Jumper 8
9 -15 Volts Main 26
10 Analog 2 (gain select) Main 17, Jumper 6
11 Vertical cantilever deflection Main 15
12 Green loop (photodiode ground?) Main 27
13 BB2107 pin 5 (2+) though R (role?) Main 9
14 Manual switch mode AFM/LFM (+5 Volts?) Switch 10
15 Stepper C Main 4
16 Stepper D Main 21
17 Head Gnd? Head 1
18 HV to piezo (+Y) Piezo 4
19 Main board ground Main 5
20 HV to piezo (+X) Piezo 8
21 Horizontal deflection Main 33
22 HV to piezo (+Z) Piezo 6
23 NCAFMJM pin 3 (role?) Main 25
24 +15 Volts Main 10
25 AD620 pin 2 (in -) through R (role?) Main 31

Main board line descriptions

Main line Role To
1 Stepper C, from DS3658N pin 6 Stepper Black
2 Stepper A, from DS3658N pin 1 Stepper Brown
3 Stepper A? in DB-25 1
4 Stepper B? in DB-25 15
5 Main board ground (black loop) DB-25 19
6 Manual stepper switch direction Switch Purple
7 +5 Volts DB-25 3
8 NCAFMJM pin 2 DB-25 6
9 through R to BBOPA2107 pin 5 (+2) DB-25 13
10 +15 Volts DB-25 24, Head 5 though 10 Ohms
11 Photodiode D Head 11
12 Photodiode A Head 9
13 DG403 pin 10  
14 internal plane, but not ground loop?  
15 Vertical cantilever deflection 620 pin 6 (out) DB-25 11
16 DG403DJ pin 6 DB-25 11
17 Gain select DG 419pin 6 (through R) DB-25 10
18 Stepper D, from DS3658N pin 8 Stepper Orange
19 Stepper B, from DS3658N pin 3 Stepper Yellow
20 Stepper C? in DB-25 2
21 Stepper D? in DB-25 16
22 AD827(far) pin 7 (out) (Tapping-piezo voltage) Head 15, tipholder 6
23 Green loop Head 14
24 Manual stepper switch enable Switch grey
25 NCAFMJM pin 3 DB-25 23
26 -15 Volts DB-25 9, Head 7 through 10 Ohms
27 Green loop DB-25 12
28 Photodiode ground (green loop) Head 2
29 Photodiode C Head 10
30 Photodiode B Head 8
31 AD620 pin 2 (in -) through R DB-25 25
32 BBOPA2107 pin 1(out1) through R(total photodiode)  
33 DG403 pins 1 and 3 (drains from 1)(Horizontal deflection) DB-25 21
34    

Chip list

(following this numbering)

Index Chip Description Role
1 DS3658N Current sink. Stepper motor current sink
2 H7555IBA CMOS timer. Manual stepper motor control clock
3 "NCAFMJM" ? Stepper motor
4 AD827JN Dual op amp. Far side amplifies tapping piezo signal (out Main 22)
5 Scratched out ? Stepper motor?
6 Scratched out ? Stepper motor?
7 AD534KD Difference multiplier (X1-X2)(Y1-Y2)/(10V)+Z2. Photo-diode?
8 AD734BQ Identical to AD534, but with control over denominator. Photo-diode?
9 SG531P 25.0000 M 25 MHz crystal oscillator. Photo-diode?
10 BBOPA 2107AU Dual op amp. Photo-diode?
11 DG419DJ CMOS analog switch. Photo-diode?
12 DG403DJ CMOS analog switch. Photo-diode?
13 AD534JD Identical to AD534KD, but with slightly more error. Photo-diode?
14 AD637JD RMS-to-DC converter. Photo-diode?
15 AD620AR Instrument amplifier. Photo-diode?
16 BBOPA 2604AU Op amp. Photo-diode?
17 DG411DJ Quad CMOS analog switch ?

Main board wiring

Vertical deflection (Main line 15) comes from pins 6 and 8 (shorted together) of the DG403 switch. Pin 6 drains pin 5 and pin 8 drains pin 9 when the switch is turned on. I'm not sure what pin 5 is attached to. Pin 9 is connected (through a resistor) to the output (pin 12) of AD534a.

Manual switch connector line descriptions

See my switch notes for details on understanding the stepper control switch.

Line Color Role To
1 Orange Laser LED ?  
2 White-red STM mode LED DB-25 17
3 Green Mode TM Main 13
4 Blue Mode STM 120 Ohms to DB-25 17
5 Purple Stepper direction Main 6
6 White-brown Stepper switch +5 Volts Main 7
7 White Laser LED? DB-25 17
8 Yellow Laser LED?  
9 White-brown STM mode LED 160 ohms to DB-25 17??
10 Red Mode AFM/LFM, +5 Volts DB-25 14
11 Grey Stepper enable Main 24
12 Black Stepper switch Gnd Main 5

AFM head micro-D 15 connector line descriptions

outside female from top-left: 8 7 6 5 4 3 2 1/15 14 13 12 11 10 9

inside male (on board) from bottom-right: 8 7 ... 1/9 10 11 ... 15 blank

Line Role To
1 Laser Ground DB-25 17
2 Photodiode ground Main 28
3 +5 Volts Laser Power no main, no DB-25 ??
4    
5 +15 Volts DB-25 24 through 10 Ohms
6 STM current? Jumper 1, tipholder 2
7 -15 Volts DB-25 9 through 10 Ohms
8 Photodiode B Main 30
9 Photodiode A Main 12
10 Photodiode C Main 29
11 Photodiode D Main 11
12 AFM tip Jumper 3, tipholder 4
13 Chassis ground DB-25 8, Jumper 8
14 Tapping-piezo ground (main board shorts to photodiode ground) Main 23, tipholder 5
15 Tapping-piezo voltage Main 22, tipholder 6

(see, e.g. Glenair for Micro-D-Subminiature connector descriptions.)

AFM head to tipholder line descriptions

Labeled looking down on tipholder along laser, with MultiMode LCD at bottom from top-left: 1 2 3/4 5 6

Line Role To
1 +12 Volts ?
2 STM current? AFM connector 4
3 -12 Volts ?
4 AFM tip? AFM connector 12
5 Ground AFM connector 14, main 23, main board shorts to photodiode ground
6 Tapping-piezo voltage AFM connector 15, main 22

On our tapping-in-air tipholders, removing the connector-pad plate shows lines 1-4 are not connected to anything. Lines 5 and 6 have wires heading over towards the piezo, as you would expect. On our tapping-in-fluid tipholders, the pad is not removable, but the only wires leaving the pad area connect to the oscillating piezo region, so both tipholders have identical interfaces to the head, with 5/6 providing piezo oscillation and 1-4 as non-contacts.

Since both tapping mode pins are shorted through the head to the main board, but not to the DB-25 connector to the NanoScope, we assume the tapping oscillation is being generated on the MulitMode main board itself.

Piezo connector line descriptions

outside male from top-left: 1 2 3 4 5/6 7 8 9

inside male (on board) from top-left: same + blank

DB-25 labeled HV: 5,7,18,20,22

Line Role To
1 Ground DB-25 8, Jumper 8
2   no db
3   no db
4 +Y piezo DB-25 18
5 -Y piezo DB-25 5
6 +Z piezo DB-25 22
7 Piezo cap Jumper 7
8 +X piezo DB-25 20
9 -X piezo DB-25 7

Jumper pin descriptions

inside male from top-left: 1 2 3 4 5 6 7 8

Roles from MultiMode manual, 14-9

Pin Role To
1 STM current Head 6
2 Auxiliary D DB-25 4
3 AFM tip Head 12
4 Analog 2 DB-25 10
5 Gain Select Main 17
6 Analog 2 DB-25 10
7 Piezo Cap Piezo 7
8 Ground DB-25 8, Piezo 1, Head 13

LCD connector descriptions

inside male from top-left: 1 2 3 4 5 / 6 7 8 9 10

Pin Role To
1 Manual switch AFM/LFM, +5 V DB-25 14
2 Total photodiode voltage Main 32
3 -15 Volts Main 26
4 +15 Volts Main 10
5 Manual switch STM mode Switch 9
6 Head Gnd? DB-25 17
7   Main 16
8 Green loop Main 27, DB-25 12
9   Main 13
10 Horizontal deflection Main 33

LCD board

inside male from top-left: 1 2 3 4 5 / 6 7 8 9 10

Pin Role To
1 +5 V Big regulator
2 Total photodiode voltage  
3 -15 Volts  
4 +15 Volts  
5 Manual switch STM mode  
6 Head Gnd?  
7   Main 16
8 Ground Black loop
9    
10 Horizontal deflection top center BBOPA though R
Posted Fri Nov 19 12:05:39 2010 Tags: hardware

You may want to see my MultiMode wiring summary.

The manual stepper switch is a double pole double throw (DPDT) switch. I've labeled the contacts below

In my MultiMode, the switch contacts are wired as follows:

  • Contacts 1, 4, and 6 are shorted together, and fed by a white-brown striped wire,
  • Contact 2 is a purple wire,
  • Contact 3 is a black wire, and
  • Contact 5 is a grey wire.

Using a multimeter to measure the average voltages I got:

Wire Up DownOff Role MainDB-25
Black 0.2V0.2V0.0VGround UL+4none
Purple5.0V0.2V0.0VDirectionUL+5none
Grey 5.0V5.0V0.0VEnable LL+6none
White 5.0V5.0V5.0VPower UL+63

So the white and black wires carry power and ground to the switch, and the purple and grey wires carry the switch information back to the electronics. The 'Main' column records the line's lead on the main circuit board in the MultiMode, and the 'DB-25' column records the line's lead on the DB-25 cable to the NanoScope controller.

From the main-connector, the grey and purple lines start spreading out, ending up at (partial list):

  • H7555IBA, a CMOS clock.
  • NCAFMJM, a top-secret something-or-other.

My guess is that the grey enable-wire from the switch turns on the H7555 timer, and the NCAFMJM is a stepper driver chip, which receives a direction input (the purple wire) and a pulse-to-step input (the H7555 output). The output of the NCAFMJM heads over to a DS3658N current driver which drives the motor.

Posted Fri Nov 19 08:50:29 2010 Tags: hardware

There is a good deal of drive-train backlash in the stepper mechanism.

Detection procedure

  1. Step towards surface until there is plenty of surface contact.
  2. Step away until you leave the surface.
  3. Repeat until you have too much data :p.

Data from 17 counterclockwise approach-retreat cycles

Conclusions

About 47 steps of backlash. Since we never touch the stepper while any critcal surface interactions are going on, simply replace step back 1 step with step back 60 steps, then forward 59.

Supporting evidence

Noise on reversal

When we reversing the stepper (by any means), the movement is “clearly audible” for “a bit” and then becomes “almost silent”.

To give a rough idea, I usually have keep my finger on the drive-shaft to tell if the motor is moving in “almost silent” mode, but can hear the motor from around 5 ft away in “clearly audible” mode. “A bit” is about 0.5 s when controlling the motor using the manual Up/Down switch.

Posted Fri Nov 19 08:32:36 2010 Tags: hardware

Why

When I joined Prof. Yang's group, the LabVIEW software controlling the experiment lacked control of the stepper motor which provides coarse z axis control. This made it difficult to set up the fluid cell and meant that leaving the system running unattended for over an hour could place unnecessary strain on the z piezo (as it would need to compensate for thermal drift on its own).

How

After a staring into the bowels of the MultiMode AFM and watching some of the more suspicious lines on an oscilloscope, I saw that lines 1, 2, 15, and 16 were the stepper motor drive lines and guessed that they were TTL compatible. A short while later (wink), the LabVIEW code was up and running. The LabVIEW computer (via a PCI-6052E card and BNC-2090 box) writes TTL stepper motor currents to the relevent lines on the Nanoscope↔MultiMode cable using our break-in box.

Computer <-> Breakout box <-> MultiMode
NanoScope <------^

Part list (outdated commercial parts linked to the current generation):

  • LabVIEW 7.1 (in-house software)
  • PCI-6052E
  • BNC-2090
  • Nanoscope IIIa (altered)
  • Homemade break in box
  • MultiMode II AFM

Breakout box

Build a DB-25 breakout box, such as the ugly hack job shown below. This gives us access to the stepper motor drive-lines.

Break in box for Nanoscope-MultiMode cable

Control the motor

First, you need something to write to your new motor control lines. We are using the digital I/O lines from this National Instruments PCI-6052E.

Next, you need to send the proper signals down the lines. Prof. Douglas Jones has an excellent stepping motor tutorial, which explains how to drive a unipolar stepping motor. For basic operation, you just need something like this pseudo-code:

step( up? ) {
  static int index=0; // saved between function calls, only initialized once
  int output[] = {sink 1a, sink 2a, sink 1b, sink 2b}; // the output array
  if (up? == TRUE) {
    index = (index + 1) mod 4;
  } else { // down
    index = (index - 1) mod 4;
  }
  write( output[index] );
}

For my MultiMode, the motor steps 'Up' when I equate:

DB-25Role
11a?
21b?
152a?
162b?

Problems

The stepper motor, like any other mechanical linkage, can have problems with backlash.

Figuring it out

The interior of the MultiMode chassis with the main PCB removed

The main board of the MultiMode

General stepper motor wiring

If you look back at the MultiMode interior and main board (above), you can see that there are wires coming from the stepper motor, going into the connector board, and (presumably) ending up at the main MultiMode board. Opening up your own MultiMode (until I get some better pictures up), you will see that the motor has 6 leads: green, brown, black, red, yellow, and orange. A few seconds with your ohm-meter, and you'll find the resistances between the leads are either 20 or 40 ohms. I've recorded the resistances in the following table:

BrownGreenBlackYellowRed Orange
Brown 20 40 Inf. Inf.Inf.
Green 20 Inf. Inf.Inf.
Black Inf. Inf.Inf.
Yellow 20 40
Red 20

Browsing through Prof. Jones' tutorial, we see that our motor is probably unipolar with a total winding resistance of 40 Ohms on each winding.

Looking closely at the stepper motor, you can see (upside down, near the right hand screw) that it is labeled 5 V DC, and 5 V over a 20 Ohm half-winding gives a current of I = V/R = 0.25 Amps. Following the motor leads back up the the main board (using the ohm-meter guess-and-check method :p), we find that they come from the DS3658N (chip 1). This chip takes care of all the details of sinking the large motor currents given a TTL driving pattern.

WARNING! I strongly suggest you don't do this on your own. The high voltage lines for driving the piezo are potentially dangerous.

The control for the DS3658N was too difficult for me to trace out on the board, so I put the main board back in the MultiMeter (leaving the base-plate off), connected the Multimeter to our NanoScope IIIa, and started clicking on the ‘raise’ and ‘lower’ tip buttons. At the same time, I watched the various DB-25 lines on the oscilloscope. DB-25 lines 1, 2, 15, and 16 oscillated, but only when the motor was turning, so I figured they must be direct TTL controls getting (somehow) to the DS3658N. I built a DB-25 breakout box to take control of those lines, and started writing software.

Line roles

Knowing the stepper control lines and how to control a unipolar motor, it was only a matter of matching the lines to the roles. I arrived at the matches that I pointed out in the How section through trial and error. It was fairly easy to get the motor moving macroscopically, and backlash testing convinced me that the microscopic motion is reproducible and smooth.

Stepsize

We measured the stepper stepsize by stepping the AFM tip closer to surface and sweeping the piezo in after each step. This produced the data shown below.

Measuring the stepsize

As the motor steps in, we record consecutive traces a, b, c, d, e, and f. Because we can calibrate the piezo by imaging a calibration sample, we can convert our piezo voltage into the distance shown on the x axis. Measuring the x distance between to traces, we see that the sample moves ~170 nm closer with each step.

Switch

There is a separate post with some details on the manual stepper switch.

Posted Thu Nov 18 14:56:43 2010 Tags: hardware

The MultiMode product line has been passed through several companies over the years. It started out at Digital Instruments, which was aquired by Veeco in February 1998 before being passed off to Bruker in August 2010.

I've done a good deal of troubleshooting to keep our ancient MultiMode II stumbling along. If you're in a similar situation, check out my notes on:

An assembled MultiMode microscope

Posted Thu Nov 18 13:37:18 2010 Tags: hardware

Using

Our cluster runs the open source Torque/Maui portable batch scheduling system (PBS). A batch scheduler takes user submitted jobs, and distributes them across the the cluster in an intelligent manner, so users don't need to worry about sharing resources fairly or sshing into compute nodes to start their jobs. Users submit jobs to the queue using qsub. I've compiled my own brief intro to qsub, and there are lots more floating about the internet.

While PBS queues are great for distributing embarassingly parallel jobs across the cluster, your application may need processes running on seperate compute nodes to share data. A common approach is to use the Message Passing Interface (MPI). Our cluster uses the mpich2 implementation. Cluster-aware applications written in MPI can be started through Torque using an alternate mpiexec from the Ohio Supercomputer Center. There is a nice, brief introduction by Kristina Wanous at the University of Northern Iowa.

Managing

Our cluster (9 dual-core nodes) runs Debian. The compute nodes all boot to NFS roots off the server node. Once that hurdle was passed, setting up Torque, Maui, mpich2, and mpiexec was pretty simple, mostly the usual:

wget ...
tar ...
configure ...
make
make install

with a bit of configuring for our setup. I'll put up some more detailed notes and our config options when I get the time.

Posted Thu Nov 18 07:53:17 2010 Tags: hardware

GPGPU utilizes the number crunching speed and massive parallelization of your graphics card to accelerate general-purpose tasks. When your algorithm is compatible with GPU hardware, the speedup of running hundreds of concurrent threads can be enormous.

There are a number of ways to implement GPGPU, ranging from multi-platform frameworks such as OpenCL to single-company frameworks such as NVIDIA's CUDA. I've gotten to play around with CUDA while TAing the parallel computing class, and its lots of fun.

With NVIDIA (other vendors are probably similar, I'll update this as I learn more), each GPU device has a block of global memory serving a number of multi-processors, and each multi-processor contains several cores which can execute concurrent threads.

Specs on NVIDIA's GeForce GTX 580:

  • 512 cores (16 (MP) ⋅ 32 (Cores/MP))
  • 1.5 GB GDDR5 RAM
  • 192.4 GB/sec memory bandwidth
  • 1.54 GHz processor clock rate
  • 1.58 TFLOPs per second

Zoom.

The FLOPs/s computaton is cores⋅clock⋅2, because (from page 94 of the CUDA programming guide) each core can exectute a single multiply-add operation (2 FLOPs) per cycle. Also take a look at the graph of historical performance on page 14, the table of device capabilities that starts on page 111, and the description of warps on page 93.

Posted Sat Nov 13 10:18:44 2010 Tags: hardware